# Tokenizers, filters and char filters

An analyzer is optional `char_filters`, one `tokenizer` and `filters` in order (see
[Analyzers](schemas-and-fields.md#analyzers)). This page lists the general-purpose components and
their parameters. The language, CJK, ICU, n-gram, synonym and phonetic components have their own
pages under [Languages and text analysis](text-analysis.md). The names and parameters are
Elasticsearch's, so an analyzer written for Elasticsearch carries over.

## Tokenizers

| Tokenizer | Splits | Params |
|---|---|---|
| `unicode_word` | Words by the Unicode rules (Elasticsearch's `standard`). | |
| `whitespace` | On whitespace. | |
| `keyword` | Not at all: the whole value is one token. | |
| `letter` | Runs of letters; everything else separates. | |
| `char_group` | On the characters you list. | `tokenize_on_chars` (required): single characters, escapes such as `\n`, or the classes `letter`, `digit`, `whitespace`, `punctuation`, `symbol`; `max_token_length` (255). |
| `pattern` | On matches of a regular expression, or keeps one group of each match. | `pattern` (default `\W+`), `group` (`-1` splits; `0` or more keeps that group), `flags`. |
| `simple_pattern` | Each match of a pattern is a token. | `pattern`. |
| `simple_pattern_split` | Each match of a pattern separates tokens. | `pattern`. |
| `path_hierarchy` | A path into each of its prefixes: `/usr/local/bin` gives `/usr`, `/usr/local`, `/usr/local/bin`. | `delimiter` (`/`), `replacement` (the delimiter), `reverse` (`false`: suffixes instead, for domain names), `skip` (0). |
| `uax_url_email` | Like `unicode_word`, but keeps URLs and e-mail addresses whole. | `max_token_length` (255). |
| `edge_ngram` | Each word into its leading grams, for [autocomplete](autocomplete.md). | `min_gram`, `max_gram`, `token_chars`, `custom_token_chars`. |

`code`, `cmdline`, `phone`, `phone_search`, `icu_tokenizer`, `kuromoji_tokenizer`,
`nori_tokenizer`, `jieba` and `trigram` are described on the pages of their use cases.

A field of e-mail addresses and links, also searchable by domain:

```json
{
  "type": "text",
  "analyzer": {"tokenizer": "uax_url_email", "filters": ["lowercase"]},
  "variants": {
    "words": {"type": "text", "analyzer": {"tokenizer": "unicode_word", "filters": ["lowercase"]}}
  }
}
```

### Regular expressions

`pattern` and `pattern_replace` use RE2, so a pattern cannot backtrack and run away on a long
value. RE2 reads the common subset of Java's syntax, which Elasticsearch uses, and a pattern that
needs backtracking (lookahead, lookbehind, backreferences, possessive quantifiers) is rejected
when the schema is saved. Named groups are written `(?P<name>...)`. `flags` takes
`CASE_INSENSITIVE`, `MULTILINE`, `DOTALL`, `LITERAL`, `UNICODE_CASE` and `UNIX_LINES`, separated by
`|`. `\w`, `\d` and `\s` match ASCII only; write `\pL` or `\p{Nd}` for Unicode classes.

`simple_pattern` and `simple_pattern_split` read Lucene's regular expression syntax: `.`, `?`, `*`,
`+`, `{n,m}`, `|`, groups, character classes, `\d`, `\s` and `\w` work, and `^` and `$` are ordinary
characters. The complement `~`, intersection `&` and numeric ranges `<n-m>` are not available.

## Filters

| Filter | Does | Params |
|---|---|---|
| `lowercase`, `uppercase` | Change case. `uppercase` works per character, so `ß` stays. | |
| `nfkc_cf` | Unicode NFKC case folding. | |
| `fold` | Removes accents and diacritics. | |
| `trim` | Removes leading and trailing whitespace from each token, useful after `keyword`. | |
| `length` | Keeps tokens between `min` and `max` characters long. | `min` (0), `max` (no limit). |
| `truncate` | Cuts tokens to `length` characters. | `length` (10). |
| `unique` | Drops a token already seen in the value. | `only_on_same_position` (`false`): drop only repeats at the same position. |
| `keep` | Keeps only the listed words. | `keep_words` (required), `keep_words_case` (`false`): compare in any case. |
| `stop` | Removes the listed words. | `words` (required). |
| `limit` | Stops after a number of tokens. | `max_token_count` (1), `consume_all_tokens` (`false`). |
| `pattern_replace` | Replaces matches of a pattern in each token. | `pattern` (required), `replacement` (empty; `$1` and `${name}` refer to groups), `all` (`true`: every match, else the first), `flags`. |
| `word_delimiter_graph` | Splits words at punctuation, case changes and letter-digit changes. | See below. |
| `reverse` | Reverses each token, for [suffix search](suffix-search.md). | `marker`. |

A removed token (by `length`, `keep` or `stop`) leaves a gap, so a phrase query does not match
across it; `unique` leaves none, as in Elasticsearch.

### Word delimiters

`word_delimiter_graph` splits `PowerShot-500` into `Power`, `Shot` and `500`, and `wi-fi` into
`wi` and `fi`, so product codes and compound names match by their parts:

```json
{
  "tokenizer": "whitespace",
  "filters": [{"name": "word_delimiter_graph", "params": {"preserve_original": true, "catenate_words": true}}, "lowercase"]
}
```

| Param | Default | Meaning |
|---|---|---|
| `generate_word_parts` | `true` | Index the letter parts. |
| `generate_number_parts` | `true` | Index the number parts. |
| `catenate_words`, `catenate_numbers`, `catenate_all` | `false` | Also index the parts joined (`powershot`, `500`, `powershot500`). |
| `split_on_case_change` | `true` | Split `PowerShot` at the case change. |
| `split_on_numerics` | `true` | Split `j2se` at letter-digit changes. |
| `preserve_original` | `false` | Also index the token as written. |
| `stem_english_possessive` | `true` | Remove a trailing `'s`. |
| `protected_words` | | Words left unsplit, such as `c++`. |
| `type_table` | | Rules that change a character's class, such as `"$ => DIGIT"`. Classes: `LOWER`, `UPPER`, `ALPHA`, `DIGIT`, `ALPHANUM`, `SUBWORD_DELIM`. |
| `adjust_offsets` | `true` | Give the parts their own offsets, for highlighting. |

`word_delimiter` takes the same params except `adjust_offsets`. It stacks joined forms on one
position, as Lucene's older filter does; prefer `word_delimiter_graph`, which gives each part its
own position. For source code identifiers, the [`code`](source-code.md) analyzer is the better fit.

## Char filters

| Char filter | Does | Params |
|---|---|---|
| `html_strip` | Removes markup and decodes entities. | |
| `nfkc_cf` | Unicode NFKC case folding of the whole value. | |
| `mapping` | Rewrites text with rules such as `"ph => f"`; map to nothing to delete. | `mappings` (required). |
| `pattern_replace` | Replaces matches of a pattern before tokenizing; offsets still point at the original text, for highlighting. | `pattern` (required), `replacement`, `flags`. |

Char filters run in the order you list them. `cjk_width`, `icu_normalizer`,
`kuromoji_iteration_mark`, `persian_zwnj` and `stconvert` are char filters too; see their pages.

## Normalizers

A `string` field's `normalizer` is a list of filters applied to each whole value. It takes the
filters that change characters but never split or remove a token: `lowercase`, `uppercase`,
`nfkc_cf`, `fold`, `trim`, `pattern_replace`, `elision`, `decimal_digit`, `cjk_width`,
`arabic_normalize`, the language normalizations, `icu_folding`, `icu_normalizer` and `reverse`.

```json
{"type": "string", "normalizer": ["trim", "lowercase", "fold"]}
```

## Through the compatible APIs

Elasticsearch's and OpenSearch's tokenizers, filters and char filters of the same names map to
these with their parameters, and `remove_duplicates` maps to `unique` with
`only_on_same_position`. The `lowercase` tokenizer is `letter` followed by `lowercase`. The
Elasticsearch `simple`, `stop` and `pattern` analyzers become the same chains. Parameters that
name a file (`keep_words_path`, `protected_words_path`, `type_table_path`, ...) cannot be read; give
the entries inline. `ignore_keywords` on the word delimiters is ignored with a warning. Solr's
factories map with Solr's parameter names: `WordDelimiterGraphFilterFactory`,
`PatternReplaceFilterFactory`, `PathHierarchyTokenizerFactory`, `LengthFilterFactory`,
`TruncateTokenFilterFactory`, `LimitTokenCountFilterFactory` and the others. See
[Through the compatible APIs](text-analysis.md#through-the-compatible-apis).
