For AI agents: this page is also available as Markdown at /docs/analysis-components.md. Every docs page is: add .md to its URL. llms.txt lists them all.
Tokenizers, filters and char filters
An analyzer is optional char_filters, one tokenizer and filters in order (see Analyzers). This page lists the general-purpose components and their parameters. The language, CJK, ICU, n-gram, synonym and phonetic components have their own pages under Languages and text analysis. The names and parameters are Elasticsearch's, so an analyzer written for Elasticsearch carries over.
#Tokenizers
| Tokenizer | Splits | Params |
|---|---|---|
unicode_word | Words by the Unicode rules (Elasticsearch's standard). | |
whitespace | On whitespace. | |
keyword | Not at all: the whole value is one token. | |
letter | Runs of letters; everything else separates. | |
char_group | On the characters you list. | tokenize_on_chars (required): single characters, escapes such as \n, or the classes letter, digit, whitespace, punctuation, symbol; max_token_length (255). |
pattern | On matches of a regular expression, or keeps one group of each match. | pattern (default \W+), group (-1 splits; 0 or more keeps that group), flags. |
simple_pattern | Each match of a pattern is a token. | pattern. |
simple_pattern_split | Each match of a pattern separates tokens. | pattern. |
path_hierarchy | A path into each of its prefixes: /usr/local/bin gives /usr, /usr/local, /usr/local/bin. | delimiter (/), replacement (the delimiter), reverse (false: suffixes instead, for domain names), skip (0). |
uax_url_email | Like unicode_word, but keeps URLs and e-mail addresses whole. | max_token_length (255). |
edge_ngram | Each word into its leading grams, for autocomplete. | min_gram, max_gram, token_chars, custom_token_chars. |
code, cmdline, phone, phone_search, icu_tokenizer, kuromoji_tokenizer, nori_tokenizer, jieba and trigram are described on the pages of their use cases.
A field of e-mail addresses and links, also searchable by domain:
{
"type": "text",
"analyzer": {"tokenizer": "uax_url_email", "filters": ["lowercase"]},
"variants": {
"words": {"type": "text", "analyzer": {"tokenizer": "unicode_word", "filters": ["lowercase"]}}
}
}
#Regular expressions
pattern and pattern_replace use RE2, so a pattern cannot backtrack and run away on a long value. RE2 reads the common subset of Java's syntax, which Elasticsearch uses, and a pattern that needs backtracking (lookahead, lookbehind, backreferences, possessive quantifiers) is rejected when the schema is saved. Named groups are written (?P<name>...). flags takes CASE_INSENSITIVE, MULTILINE, DOTALL, LITERAL, UNICODE_CASE and UNIX_LINES, separated by |. \w, \d and \s match ASCII only; write \pL or \p{Nd} for Unicode classes.
simple_pattern and simple_pattern_split read Lucene's regular expression syntax: ., ?, *, +, {n,m}, |, groups, character classes, \d, \s and \w work, and ^ and $ are ordinary characters. The complement ~, intersection & and numeric ranges <n-m> are not available.
#Filters
| Filter | Does | Params |
|---|---|---|
lowercase, uppercase | Change case. uppercase works per character, so ß stays. | |
nfkc_cf | Unicode NFKC case folding. | |
fold | Removes accents and diacritics. | |
trim | Removes leading and trailing whitespace from each token, useful after keyword. | |
length | Keeps tokens between min and max characters long. | min (0), max (no limit). |
truncate | Cuts tokens to length characters. | length (10). |
unique | Drops a token already seen in the value. | only_on_same_position (false): drop only repeats at the same position. |
keep | Keeps only the listed words. | keep_words (required), keep_words_case (false): compare in any case. |
stop | Removes the listed words. | words (required). |
limit | Stops after a number of tokens. | max_token_count (1), consume_all_tokens (false). |
pattern_replace | Replaces matches of a pattern in each token. | pattern (required), replacement (empty; $1 and ${name} refer to groups), all (true: every match, else the first), flags. |
word_delimiter_graph | Splits words at punctuation, case changes and letter-digit changes. | See below. |
reverse | Reverses each token, for suffix search. | marker. |
A removed token (by length, keep or stop) leaves a gap, so a phrase query does not match across it; unique leaves none, as in Elasticsearch.
#Word delimiters
word_delimiter_graph splits PowerShot-500 into Power, Shot and 500, and wi-fi into wi and fi, so product codes and compound names match by their parts:
{
"tokenizer": "whitespace",
"filters": [{"name": "word_delimiter_graph", "params": {"preserve_original": true, "catenate_words": true}}, "lowercase"]
}
| Param | Default | Meaning |
|---|---|---|
generate_word_parts | true | Index the letter parts. |
generate_number_parts | true | Index the number parts. |
catenate_words, catenate_numbers, catenate_all | false | Also index the parts joined (powershot, 500, powershot500). |
split_on_case_change | true | Split PowerShot at the case change. |
split_on_numerics | true | Split j2se at letter-digit changes. |
preserve_original | false | Also index the token as written. |
stem_english_possessive | true | Remove a trailing 's. |
protected_words | Words left unsplit, such as c++. | |
type_table | Rules that change a character's class, such as "$ => DIGIT". Classes: LOWER, UPPER, ALPHA, DIGIT, ALPHANUM, SUBWORD_DELIM. | |
adjust_offsets | true | Give the parts their own offsets, for highlighting. |
word_delimiter takes the same params except adjust_offsets. It stacks joined forms on one position, as Lucene's older filter does; prefer word_delimiter_graph, which gives each part its own position. For source code identifiers, the code analyzer is the better fit.
#Char filters
| Char filter | Does | Params |
|---|---|---|
html_strip | Removes markup and decodes entities. | |
nfkc_cf | Unicode NFKC case folding of the whole value. | |
mapping | Rewrites text with rules such as "ph => f"; map to nothing to delete. | mappings (required). |
pattern_replace | Replaces matches of a pattern before tokenizing; offsets still point at the original text, for highlighting. | pattern (required), replacement, flags. |
Char filters run in the order you list them. cjk_width, icu_normalizer, kuromoji_iteration_mark, persian_zwnj and stconvert are char filters too; see their pages.
#Normalizers
A string field's normalizer is a list of filters applied to each whole value. It takes the filters that change characters but never split or remove a token: lowercase, uppercase, nfkc_cf, fold, trim, pattern_replace, elision, decimal_digit, cjk_width, arabic_normalize, the language normalizations, icu_folding, icu_normalizer and reverse.
{"type": "string", "normalizer": ["trim", "lowercase", "fold"]}
#Through the compatible APIs
Elasticsearch's and OpenSearch's tokenizers, filters and char filters of the same names map to these with their parameters, and remove_duplicates maps to unique with only_on_same_position. The lowercase tokenizer is letter followed by lowercase. The Elasticsearch simple, stop and pattern analyzers become the same chains. Parameters that name a file (keep_words_path, protected_words_path, type_table_path, ...) cannot be read; give the entries inline. ignore_keywords on the word delimiters is ignored with a warning. Solr's factories map with Solr's parameter names: WordDelimiterGraphFilterFactory, PatternReplaceFilterFactory, PathHierarchyTokenizerFactory, LengthFilterFactory, TruncateTokenFilterFactory, LimitTokenCountFilterFactory and the others. See Through the compatible APIs.