For AI agents: this page is also available as Markdown at /docs/icu.md. Every docs page is: add .md to its URL. llms.txt lists them all.
Multilingual text and ICU
A field that holds several languages, or names written in several scripts, needs analysis that does not assume one language. These components use the ICU Unicode library, as the Elasticsearch ICU plugin does.
| Component | Kind | Does |
|---|---|---|
icu_tokenizer | tokenizer | Splits text into words by the Unicode rules, with dictionary segmentation for Thai, Lao, Khmer, Burmese, Chinese and Japanese, which do not separate words with spaces. Param locale (optional). |
icu_folding | filter | Unicode case folding and normalization, and removes Latin, Greek and Cyrillic accents (Ångström to angstrom). Param unicode_set_filter exempts characters, for example "[^ñ]" to keep ñ. |
icu_normalizer | char filter or filter | As a char filter, normalizes the whole value before tokenizing; as a filter, each token. Param name: nfkc_cf (default), nfc, nfkc, nfd or nfkd; mode: compose or decompose; unicode_set_filter. |
icu_transform | filter | Applies an ICU transform named by the required param id, such as "Any-Latin; Latin-ASCII" or "Traditional-Simplified". direction: forward (default) or reverse. |
#Several languages in one field
Without stemming, accent and case folding is the safest common ground:
{
"type": "text",
"analyzer": {"tokenizer": "icu_tokenizer", "filters": ["icu_folding"]}
}
If you know each document's language, one field per language with that language's analyzer matches more word forms. Search all of them with a multi_match or several field:term clauses.
#Names in several scripts
A field that matches both a transliteration and its source script, so Москва and moskva find the same document:
{
"type": "text",
"analyzer": {
"char_filters": ["icu_normalizer"],
"tokenizer": "unicode_word",
"filters": [{"name": "icu_transform", "params": {"id": "Any-Latin; Latin-ASCII"}}, "icu_folding"]
}
}
Traditional-Simplified does the same for Chinese, so traditional and simplified characters match each other.
#Notes
icu_foldingis close to Elasticsearch's but not identical: it does not fold letters such asø,đandł, and it keeps Indic, Arabic and Kana marks.- A transform can depend on context, so prefix, wildcard and fuzzy terms skip
icu_transformunless you set its"normalizing": true. Set it when the transform maps each character on its own, asLatin-ASCIIdoes. - ICU collation (
icu_collation_keyword) is not available. - Elasticsearch's
icu_tokenizer,icu_folding,icu_normalizerandicu_transform, thethaianalyzer, and Solr's ICU factories map to these components. See Through the compatible APIs.