For AI agents: this page is also available as Markdown at /docs/more-languages.md. Every docs page is: add .md to its URL. llms.txt lists them all.
More languages
These analyzers follow the chains of Lucene's language analyzers, the ones Elasticsearch ships under the same names, with Lucene's stop words. Their normalizers and stemmers produce the same words as Lucene's, so an index migrated from Elasticsearch matches the same queries.
{
"fields": {
"title_fa": {"type": "text", "analyzer": "persian"},
"title_hi": {"type": "text", "analyzer": "hindi"},
"title_tr": {"type": "text", "analyzer": "turkish"}
}
}
| Analyzer | Language | Chain after unicode_word |
|---|---|---|
persian | Persian (Farsi) | persian_zwnj char filter, lowercase, decimal_digit, arabic_normalize, persian_normalization, stop words, persian_stem |
sorani | Sorani (Central Kurdish) | sorani_normalization, lowercase, decimal_digit, stop words, sorani_stem |
hindi | Hindi | lowercase, decimal_digit, indic_normalization, hindi_normalization, stop words, hindi_stem |
bengali | Bengali | lowercase, decimal_digit, indic_normalization, bengali_normalization, stop words, bengali_stem |
telugu | Telugu | decimal_digit, indic_normalization, telugu_normalization, stop words, telugu_stem |
bulgarian | Bulgarian | lowercase, stop words, bulgarian_stem |
latvian | Latvian | lowercase, stop words, latvian_stem |
galician | Galician | lowercase, stop words, galician_stem |
brazilian | Brazilian Portuguese | lowercase, stop words, brazilian_stem |
turkish | Turkish | apostrophe, turkish_lowercase, stop words, Snowball Turkish |
Persian text often joins word parts with a zero-width non-joiner. The persian_zwnj char filter turns it into a space, so میخواهم is split the way Persian readers see it, and decimal_digit turns Persian and Arabic-Indic digits into 0 to 9, so ۱۴۰۲ matches 1402.
For Turkish, turkish_lowercase lowercases I to ı and İ to i, and apostrophe cuts the suffix after an apostrophe on proper names, so Türkiye'nin is indexed as türkiye.
#Filters
Use these in your own chains. None takes parameters except elision.
| Filter | Does |
|---|---|
elision | Removes a leading article and its apostrophe (' or ’): l'avion to avion. articles lists them (default: the French ones), articles_case: true matches them in any case. |
apostrophe | Cuts a token at its first apostrophe. |
decimal_digit | Every Unicode decimal digit to its ASCII digit. |
indic_normalization | Normalizes the Unicode spellings of Indic scripts. |
hindi_normalization, bengali_normalization, telugu_normalization | Spelling normalization for the language. |
persian_normalization, sorani_normalization | Spelling normalization for the language. |
german_normalization | Folds ä, ö, ü and ß the way German readers spell them without umlauts, so Häuser matches Haeuser and Hauser. |
turkish_lowercase | Turkish case rules for dotted and dotless i. |
hindi_stem, bengali_stem, telugu_stem, persian_stem, sorani_stem | Light stemmers. |
bulgarian_stem, latvian_stem, galician_stem, brazilian_stem | Stemmers. |
A German chain that matches words written with and without umlauts:
{
"type": "text",
"analyzer": {
"tokenizer": "unicode_word",
"filters": ["lowercase", "german_normalization", {"name": "snowball", "params": {"language": "german"}}]
}
}
A French chain with elision and Snowball stemming:
{
"type": "text",
"analyzer": {
"tokenizer": "unicode_word",
"filters": [{"name": "elision", "params": {"articles": ["l", "d", "qu", "j"], "articles_case": true}},
"lowercase", {"name": "snowball", "params": {"language": "french"}}]
}
}
The normalizations and elision also apply to prefix and fuzzy terms, so häu* finds häuser. A wildcard is folded one character at a time, so rules that look at neighbouring characters do not apply to it: hae* does not find häuser, though the prefix query hae does. Stemmers do not apply to prefix, wildcard and fuzzy terms.
#Differences from Lucene
- A token that a filter empties is dropped, where Lucene indexes an empty term.
- In your own chains, the
stopfilter passes its words through the filters before it, so an entry the chain would rewrite still matches; Lucene's does not. The analyzers above remove the same words as Lucene's. - The
turkishanalyzer stems with Snowball 3.2, where Lucene bundles an older Snowball.
#Through the compatible APIs
Elasticsearch's persian, sorani, bengali, hindi, bulgarian, latvian, galician, brazilian and turkish analyzers map to the analyzers of the same names. The stemmer filter with these languages, elision, apostrophe, decimal_digit, the normalization filters and lowercase with language: turkish map to the filters above. Solr's text_fa, text_ckb, text_hi, text_bg, text_lv, text_gl and text_tr types and the matching factories and analyzer classes map the same way. See Through the compatible APIs.