# Language detection

When one field holds text in several languages and you do not know which language each document
is in, the `multilingual` analyzer detects the language of each value and adds that language's
stems. Every word is indexed as written, and, when the language is detected reliably, its stem at
the same position. A French product description gets French stems and a German one German stems.

```json
{"fields": {"description": {"type": "text", "analyzer": "multilingual"}}}
```

`multilingual` covers English, French, German, Spanish, Italian, Portuguese, Dutch and Russian. To
choose the languages, use the `language_stem` filter in your own chain:

```json
{
  "type": "text",
  "analyzer": {
    "tokenizer": "unicode_word",
    "filters": ["nfkc_cf", {"name": "language_stem", "params": {"languages": ["english", "french", "polish"]}}]
  }
}
```

| Param | Default | Meaning |
|---|---|---|
| `languages` | | Required. The languages to stem: any of arabic, armenian, basque, catalan, danish, dutch, english, estonian, finnish, french, german, greek, hindi, hungarian, indonesian, irish, italian, lithuanian, nepali, norwegian, portuguese, romanian, russian, serbian, spanish, swedish, tamil, turkish and yiddish. |
| `mode` | `index` | `index` detects the language of the value; `query` adds the stem in every listed language. |
| `min_bytes` | `48` | Letters the detector must see before its answer counts. Shorter values are indexed as written. |
| `min_percent` | `70` | Share of the value the language must hold. |

The filter expects lowercase input, so put `nfkc_cf` or `lowercase` before it. For Thai, Chinese,
Japanese and other scripts without spaces, use `icu_tokenizer` in place of `unicode_word`.

## Searching

A query is too short to detect its language, so it needs the other mode: `multilingual_query`
adds the stem of each query word in every language. Name it as the query's analyzer and use the
`or` operator:

```json
POST /v1/search/products
{
  "language": "es",
  "query": {"match": {"description": {"query": "chaussures rouges", "analyzer": "multilingual_query", "operator": "or"}}}
}
```

A query word then finds the documents that contain it as written, and those whose detected
language stems it the same way. The `and` operator and phrase queries require every stem of a
word, in every language, so they match only the same inflection; use `or`.

The other setup is to index with `multilingual_query` too. Every word is then indexed with its
stem in every language, so a plain `match` with `or` finds other inflections, at the cost of a
larger index and of words from different languages meeting on the same stem.

## Trade-offs

- Detection is per value. A value in two languages gets the stems of the larger one, or none
  when no language reaches `min_percent`.
- Short values are not stemmed, so they are found only by the words as written.
- A language that is detected but not listed is indexed as written. Close languages are
  sometimes confused, such as Serbian and Croatian, or Indonesian and Malay.
- Detection reads the first 64 KB of a value; the rest is stemmed in the language detected there.
  A stop filter placed before `language_stem` removes words the detector relies on.
- Each listed language adds a term per word to a query, and more accidental matches.
- Changing `languages` or a threshold needs a reindex.

When you know each document's language, one field per language with that language's analyzer
(see [Snowball stemming](snowball-stemming.md)) matches more precisely.
