For AI agents: this page is also available as Markdown at /docs/language-detection.md. Every docs page is: add .md to its URL. llms.txt lists them all.
Language detection
When one field holds text in several languages and you do not know which language each document is in, the multilingual analyzer detects the language of each value and adds that language's stems. Every word is indexed as written, and, when the language is detected reliably, its stem at the same position. A French product description gets French stems and a German one German stems.
{"fields": {"description": {"type": "text", "analyzer": "multilingual"}}}
multilingual covers English, French, German, Spanish, Italian, Portuguese, Dutch and Russian. To choose the languages, use the language_stem filter in your own chain:
{
"type": "text",
"analyzer": {
"tokenizer": "unicode_word",
"filters": ["nfkc_cf", {"name": "language_stem", "params": {"languages": ["english", "french", "polish"]}}]
}
}
| Param | Default | Meaning |
|---|---|---|
languages | Required. The languages to stem: any of arabic, armenian, basque, catalan, danish, dutch, english, estonian, finnish, french, german, greek, hindi, hungarian, indonesian, irish, italian, lithuanian, nepali, norwegian, portuguese, romanian, russian, serbian, spanish, swedish, tamil, turkish and yiddish. | |
mode | index | index detects the language of the value; query adds the stem in every listed language. |
min_bytes | 48 | Letters the detector must see before its answer counts. Shorter values are indexed as written. |
min_percent | 70 | Share of the value the language must hold. |
The filter expects lowercase input, so put nfkc_cf or lowercase before it. For Thai, Chinese, Japanese and other scripts without spaces, use icu_tokenizer in place of unicode_word.
#Searching
A query is too short to detect its language, so it needs the other mode: multilingual_query adds the stem of each query word in every language. Name it as the query's analyzer and use the or operator:
POST /v1/search/products
{
"language": "es",
"query": {"match": {"description": {"query": "chaussures rouges", "analyzer": "multilingual_query", "operator": "or"}}}
}
A query word then finds the documents that contain it as written, and those whose detected language stems it the same way. The and operator and phrase queries require every stem of a word, in every language, so they match only the same inflection; use or.
The other setup is to index with multilingual_query too. Every word is then indexed with its stem in every language, so a plain match with or finds other inflections, at the cost of a larger index and of words from different languages meeting on the same stem.
#Trade-offs
- Detection is per value. A value in two languages gets the stems of the larger one, or none when no language reaches
min_percent. - Short values are not stemmed, so they are found only by the words as written.
- A language that is detected but not listed is indexed as written. Close languages are sometimes confused, such as Serbian and Croatian, or Indonesian and Malay.
- Detection reads the first 64 KB of a value; the rest is stemmed in the language detected there. A stop filter placed before
language_stemremoves words the detector relies on. - Each listed language adds a term per word to a query, and more accidental matches.
- Changing
languagesor a threshold needs a reindex.
When you know each document's language, one field per language with that language's analyzer (see Snowball stemming) matches more precisely.