# Multilingual text and ICU

A field that holds several languages, or names written in several scripts, needs analysis that
does not assume one language. These components use the ICU Unicode library, as the Elasticsearch
ICU plugin does.

| Component | Kind | Does |
|---|---|---|
| `icu_tokenizer` | tokenizer | Splits text into words by the Unicode rules, with dictionary segmentation for Thai, Lao, Khmer, Burmese, Chinese and Japanese, which do not separate words with spaces. Param `locale` (optional). |
| `icu_folding` | filter | Unicode case folding and normalization, and removes Latin, Greek and Cyrillic accents (`Ångström` to `angstrom`). Param `unicode_set_filter` exempts characters, for example `"[^ñ]"` to keep `ñ`. |
| `icu_normalizer` | char filter or filter | As a char filter, normalizes the whole value before tokenizing; as a filter, each token. Param `name`: `nfkc_cf` (default), `nfc`, `nfkc`, `nfd` or `nfkd`; `mode`: `compose` or `decompose`; `unicode_set_filter`. |
| `icu_transform` | filter | Applies an ICU transform named by the required param `id`, such as `"Any-Latin; Latin-ASCII"` or `"Traditional-Simplified"`. `direction`: `forward` (default) or `reverse`. |

## Several languages in one field

Without stemming, accent and case folding is the safest common ground:

```json
{
  "type": "text",
  "analyzer": {"tokenizer": "icu_tokenizer", "filters": ["icu_folding"]}
}
```

If you know each document's language, one field per language with that language's analyzer
matches more word forms. Search all of them with a `multi_match` or several `field:term` clauses.

## Names in several scripts

A field that matches both a transliteration and its source script, so `Москва` and `moskva` find
the same document:

```json
{
  "type": "text",
  "analyzer": {
    "char_filters": ["icu_normalizer"],
    "tokenizer": "unicode_word",
    "filters": [{"name": "icu_transform", "params": {"id": "Any-Latin; Latin-ASCII"}}, "icu_folding"]
  }
}
```

`Traditional-Simplified` does the same for Chinese, so traditional and simplified characters match
each other.

## Notes

- `icu_folding` is close to Elasticsearch's but not identical: it does not fold letters such as
  `ø`, `đ` and `ł`, and it keeps Indic, Arabic and Kana marks.
- A transform can depend on context, so prefix, wildcard and fuzzy terms skip `icu_transform`
  unless you set its `"normalizing": true`. Set it when the transform maps each character on its
  own, as `Latin-ASCII` does.
- ICU collation (`icu_collation_keyword`) is not available.
- Elasticsearch's `icu_tokenizer`, `icu_folding`, `icu_normalizer` and `icu_transform`, the
  `thai` analyzer, and Solr's ICU factories map to these components. See
  [Through the compatible APIs](text-analysis.md#through-the-compatible-apis).