For AI agents: this page is also available as Markdown at /docs/language-detection.md. Every docs page is: add .md to its URL. llms.txt lists them all.

Language detection

When one field holds text in several languages and you do not know which language each document is in, the multilingual analyzer detects the language of each value and adds that language's stems. Every word is indexed as written, and, when the language is detected reliably, its stem at the same position. A French product description gets French stems and a German one German stems.

{"fields": {"description": {"type": "text", "analyzer": "multilingual"}}}

multilingual covers English, French, German, Spanish, Italian, Portuguese, Dutch and Russian. To choose the languages, use the language_stem filter in your own chain:

{
  "type": "text",
  "analyzer": {
    "tokenizer": "unicode_word",
    "filters": ["nfkc_cf", {"name": "language_stem", "params": {"languages": ["english", "french", "polish"]}}]
  }
}
Param Default Meaning
languages Required. The languages to stem: any of arabic, armenian, basque, catalan, danish, dutch, english, estonian, finnish, french, german, greek, hindi, hungarian, indonesian, irish, italian, lithuanian, nepali, norwegian, portuguese, romanian, russian, serbian, spanish, swedish, tamil, turkish and yiddish.
mode index index detects the language of the value; query adds the stem in every listed language.
min_bytes 48 Letters the detector must see before its answer counts. Shorter values are indexed as written.
min_percent 70 Share of the value the language must hold.

The filter expects lowercase input, so put nfkc_cf or lowercase before it. For Thai, Chinese, Japanese and other scripts without spaces, use icu_tokenizer in place of unicode_word.

Searching

A query is too short to detect its language, so it needs the other mode: multilingual_query adds the stem of each query word in every language. Name it as the query's analyzer and use the or operator:

POST /v1/search/products
{
  "language": "es",
  "query": {"match": {"description": {"query": "chaussures rouges", "analyzer": "multilingual_query", "operator": "or"}}}
}

A query word then finds the documents that contain it as written, and those whose detected language stems it the same way. The and operator and phrase queries require every stem of a word, in every language, so they match only the same inflection; use or.

The other setup is to index with multilingual_query too. Every word is then indexed with its stem in every language, so a plain match with or finds other inflections, at the cost of a larger index and of words from different languages meeting on the same stem.

Trade-offs

When you know each document's language, one field per language with that language's analyzer (see Snowball stemming) matches more precisely.

View this page as Markdown

Cookie settings

Optional tracking stays off until you choose it. Your choice applies to this browser for 180 days. Read the details.

Essential storage keeps you signed in, protects forms, and remembers this choice. It is always available.