For AI agents: this page is also available as Markdown at /docs/analysis-components.md. Every docs page is: add .md to its URL. llms.txt lists them all.

Tokenizers, filters and char filters

An analyzer is optional char_filters, one tokenizer and filters in order (see Analyzers). This page lists the general-purpose components and their parameters. The language, CJK, ICU, n-gram, synonym and phonetic components have their own pages under Languages and text analysis. The names and parameters are Elasticsearch's, so an analyzer written for Elasticsearch carries over.

Tokenizers

Tokenizer Splits Params
unicode_word Words by the Unicode rules (Elasticsearch's standard).
whitespace On whitespace.
keyword Not at all: the whole value is one token.
letter Runs of letters; everything else separates.
char_group On the characters you list. tokenize_on_chars (required): single characters, escapes such as \n, or the classes letter, digit, whitespace, punctuation, symbol; max_token_length (255).
pattern On matches of a regular expression, or keeps one group of each match. pattern (default \W+), group (-1 splits; 0 or more keeps that group), flags.
simple_pattern Each match of a pattern is a token. pattern.
simple_pattern_split Each match of a pattern separates tokens. pattern.
path_hierarchy A path into each of its prefixes: /usr/local/bin gives /usr, /usr/local, /usr/local/bin. delimiter (/), replacement (the delimiter), reverse (false: suffixes instead, for domain names), skip (0).
uax_url_email Like unicode_word, but keeps URLs and e-mail addresses whole. max_token_length (255).
edge_ngram Each word into its leading grams, for autocomplete. min_gram, max_gram, token_chars, custom_token_chars.

code, cmdline, phone, phone_search, icu_tokenizer, kuromoji_tokenizer, nori_tokenizer, jieba and trigram are described on the pages of their use cases.

A field of e-mail addresses and links, also searchable by domain:

{
  "type": "text",
  "analyzer": {"tokenizer": "uax_url_email", "filters": ["lowercase"]},
  "variants": {
    "words": {"type": "text", "analyzer": {"tokenizer": "unicode_word", "filters": ["lowercase"]}}
  }
}

Regular expressions

pattern and pattern_replace use RE2, so a pattern cannot backtrack and run away on a long value. RE2 reads the common subset of Java's syntax, which Elasticsearch uses, and a pattern that needs backtracking (lookahead, lookbehind, backreferences, possessive quantifiers) is rejected when the schema is saved. Named groups are written (?P<name>...). flags takes CASE_INSENSITIVE, MULTILINE, DOTALL, LITERAL, UNICODE_CASE and UNIX_LINES, separated by |. \w, \d and \s match ASCII only; write \pL or \p{Nd} for Unicode classes.

simple_pattern and simple_pattern_split read Lucene's regular expression syntax: ., ?, *, +, {n,m}, |, groups, character classes, \d, \s and \w work, and ^ and $ are ordinary characters. The complement ~, intersection & and numeric ranges <n-m> are not available.

Filters

Filter Does Params
lowercase, uppercase Change case. uppercase works per character, so ß stays.
nfkc_cf Unicode NFKC case folding.
fold Removes accents and diacritics.
trim Removes leading and trailing whitespace from each token, useful after keyword.
length Keeps tokens between min and max characters long. min (0), max (no limit).
truncate Cuts tokens to length characters. length (10).
unique Drops a token already seen in the value. only_on_same_position (false): drop only repeats at the same position.
keep Keeps only the listed words. keep_words (required), keep_words_case (false): compare in any case.
stop Removes the listed words. words (required).
limit Stops after a number of tokens. max_token_count (1), consume_all_tokens (false).
pattern_replace Replaces matches of a pattern in each token. pattern (required), replacement (empty; $1 and ${name} refer to groups), all (true: every match, else the first), flags.
word_delimiter_graph Splits words at punctuation, case changes and letter-digit changes. See below.
reverse Reverses each token, for suffix search. marker.

A removed token (by length, keep or stop) leaves a gap, so a phrase query does not match across it; unique leaves none, as in Elasticsearch.

Word delimiters

word_delimiter_graph splits PowerShot-500 into Power, Shot and 500, and wi-fi into wi and fi, so product codes and compound names match by their parts:

{
  "tokenizer": "whitespace",
  "filters": [{"name": "word_delimiter_graph", "params": {"preserve_original": true, "catenate_words": true}}, "lowercase"]
}
Param Default Meaning
generate_word_parts true Index the letter parts.
generate_number_parts true Index the number parts.
catenate_words, catenate_numbers, catenate_all false Also index the parts joined (powershot, 500, powershot500).
split_on_case_change true Split PowerShot at the case change.
split_on_numerics true Split j2se at letter-digit changes.
preserve_original false Also index the token as written.
stem_english_possessive true Remove a trailing 's.
protected_words Words left unsplit, such as c++.
type_table Rules that change a character's class, such as "$ => DIGIT". Classes: LOWER, UPPER, ALPHA, DIGIT, ALPHANUM, SUBWORD_DELIM.
adjust_offsets true Give the parts their own offsets, for highlighting.

word_delimiter takes the same params except adjust_offsets. It stacks joined forms on one position, as Lucene's older filter does; prefer word_delimiter_graph, which gives each part its own position. For source code identifiers, the code analyzer is the better fit.

Char filters

Char filter Does Params
html_strip Removes markup and decodes entities.
nfkc_cf Unicode NFKC case folding of the whole value.
mapping Rewrites text with rules such as "ph => f"; map to nothing to delete. mappings (required).
pattern_replace Replaces matches of a pattern before tokenizing; offsets still point at the original text, for highlighting. pattern (required), replacement, flags.

Char filters run in the order you list them. cjk_width, icu_normalizer, kuromoji_iteration_mark, persian_zwnj and stconvert are char filters too; see their pages.

Normalizers

A string field's normalizer is a list of filters applied to each whole value. It takes the filters that change characters but never split or remove a token: lowercase, uppercase, nfkc_cf, fold, trim, pattern_replace, elision, decimal_digit, cjk_width, arabic_normalize, the language normalizations, icu_folding, icu_normalizer and reverse.

{"type": "string", "normalizer": ["trim", "lowercase", "fold"]}

Through the compatible APIs

Elasticsearch's and OpenSearch's tokenizers, filters and char filters of the same names map to these with their parameters, and remove_duplicates maps to unique with only_on_same_position. The lowercase tokenizer is letter followed by lowercase. The Elasticsearch simple, stop and pattern analyzers become the same chains. Parameters that name a file (keep_words_path, protected_words_path, type_table_path, ...) cannot be read; give the entries inline. ignore_keywords on the word delimiters is ignored with a warning. Solr's factories map with Solr's parameter names: WordDelimiterGraphFilterFactory, PatternReplaceFilterFactory, PathHierarchyTokenizerFactory, LengthFilterFactory, TruncateTokenFilterFactory, LimitTokenCountFilterFactory and the others. See Through the compatible APIs.

View this page as Markdown

Cookie settings

Optional tracking stays off until you choose it. Your choice applies to this browser for 180 days. Read the details.

Essential storage keeps you signed in, protects forms, and remembers this choice. It is always available.