For AI agents: this page is also available as Markdown at /docs/cjk.md. Every docs page is: add .md to its URL. llms.txt lists them all.

Chinese, Japanese and Korean

Chinese and Japanese do not separate words with spaces, and Korean compounds freely. The cjk analyzer indexes runs of Han, Hiragana, Katakana and Hangul as overlapping two-character pairs, so 東京都 is indexed as 東京 and 京都, and a query matches when its pairs appear in that order. It needs no dictionary. Latin words and numbers in the same text are kept as words.

{"fields": {"title_ja": {"type": "text", "analyzer": "cjk"}}}

The chain is unicode_word, cjk_width, lowercase, cjk_bigram and the CJK stop words, the same as Elasticsearch's cjk analyzer.

#Filters

A one-character query matches nothing by default, because only pairs are indexed. Set output_unigrams to true to also match single characters, at the cost of a larger index:

{
  "type": "text",
  "analyzer": {
    "tokenizer": "unicode_word",
    "filters": ["cjk_width", "lowercase", {"name": "cjk_bigram", "params": {"output_unigrams": true}}]
  }
}

#Tips

• Pairs match more than words do: a query for 京都 (Kyoto) also finds 東京都 (Tokyo Metropolis), which contains the same pair. Phrase queries and BM25 ranking keep the best matches on top.
• icu_tokenizer is the alternative: it splits Chinese and Japanese into dictionary words instead of pairs, which gives a smaller index and fewer partial matches, but misses words the dictionary splits differently. See Multilingual text and ICU.
• Elasticsearch's cjk analyzer, the cjk_width and cjk_bigram filters and Solr's text_cjk type map to these components. See Through the compatible APIs.

View this page as Markdown