# Languages and text analysis

A `text` field's analyzer decides which words a document is indexed under and how a query is
split, so it decides what matches. The built-in `_t` suffix suits English. For other languages,
and for values that are not prose, such as source code, phone numbers or command lines, give the
field one of the analyzers in this section in its [schema](schemas-and-fields.md#analyzers).

The same analyzer runs at index time and at query time, so a query in any spelling the analyzer
folds together finds the document. Prefix, wildcard and fuzzy terms get only the analyzer's
normalizing steps (case, width and accent folding), not stemming or lemmatization.

## Languages

| Page | Analyzers | What it covers |
|---|---|---|
| English | the `_t` suffix, or `snowball_english` | Word splitting, case and accent folding, possessive removal and English stemming. This is the default for `_t` fields; see [Analyzers](schemas-and-fields.md#analyzers). |
| [Snowball stemming](snowball-stemming.md) | `snowball_<language>` | Stemming and stop words for 30 languages, covering most European languages (German, French, Spanish, Italian, Portuguese, Dutch, the Nordic and Baltic languages, Greek, Russian and more), plus Hindi, Indonesian and Turkish. |
| [Arabic](arabic.md) | `arabic` | Spelling normalization, stop words and light stemming. |
| [Hebrew](hebrew.md) | `hebrew`, `hebrew_query` | Dictionary lemmatization with prefix stripping and spelling without niqqud, plus exact-form matching. |
| [Chinese, Japanese and Korean](cjk.md) | `cjk` | Overlapping character pairs, with no dictionary. |
| [Multilingual text and ICU](icu.md) | `icu_folding`, `icu_normalizer`, `icu_transform`, `icu_tokenizer` | Accent and case folding across scripts, Unicode normalization and transliteration. |

## Use cases

| Page | Analyzers and queries | What it covers |
|---|---|---|
| [Source code](source-code.md) | `code` | Identifiers, dotted paths, stack traces and log lines, matched whole and by their parts. |
| [Phone numbers](phone-numbers.md) | `phone`, `phone_search` | Numbers matched in any format and by their trailing digits. |
| [Command lines](command-line-search.md) | `cmdline`, `cmdline_exact`, `cmdline_base64` | Process command lines, paths, URLs and registry keys for SIEM detection, including decoded PowerShell `-EncodedCommand` payloads. |
| [Suffix search](suffix-search.md) | `reverse`, `rwildcard` | Fast leading-wildcard and suffix matches (`*.exe`, `*\install_update.bat`) on large dictionaries. |
| [Substring search](substring-search.md) | `trigram`, `substring` | Exact substring, wildcard and regex matches anywhere in a value of any length, like the Elasticsearch `wildcard` field. |

## Use an analyzer

Name a ready-made analyzer as the field's `analyzer`, as a string or as `{"name": ...}`:

```bash
curl -X PUT https://api.warpsearch.io/v1/collections/articles \
  -H "Authorization: Bearer $WARPSEARCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "schema": {
      "fields": {
        "title_ar": {"type": "text", "analyzer": "arabic"},
        "title_he": {"type": "text", "analyzer": "hebrew"},
        "title_de": {"type": "text", "analyzer": "snowball_german"},
        "title_ja": {"type": "text", "analyzer": "cjk"},
        "source":   {"type": "text", "analyzer": "code"},
        "phone":    {"type": "text", "analyzer": "phone"}
      }
    }
  }'
```

Or build a chain from the components on these pages, as for any analyzer: optional
`char_filters`, one `tokenizer`, then `filters` in order. A component with parameters is
`{"name": ..., "params": {...}}`.

```json
{
  "type": "text",
  "analyzer": {
    "tokenizer": "unicode_word",
    "filters": ["nfkc_cf", {"name": "snowball", "params": {"language": "german"}}]
  }
}
```

A named analyzer combines only with `char_filters`, which replace the preset's own. To change
anything else, write out the chain.

An analyzer cannot change once a field has data. To switch an existing field, add a new field
(or a [variant](schemas-and-fields.md#variants)) with the new analyzer and reindex, or create a
new collection. Search the field the usual way, for example `"query": "title_de:haus"` in
Lucene syntax or a `match` query through the Elasticsearch-compatible API.

The Elasticsearch, OpenSearch and Solr compatible APIs use these analyzers too, from the analyzer
names and settings those clients already send. See [Through the compatible APIs](#through-the-compatible-apis).

## Ready-made analyzers

| Analyzer | For | Chain |
|---|---|---|
| `arabic` | [Arabic](arabic.md) | `unicode_word`, `nfkc_cf`, `arabic_normalize`, Arabic stop words, `arabic_stem` |
| `hebrew`, `hebrew_query` | [Hebrew](hebrew.md) | `unicode_word`, `nfkc_cf`, `fold`, `hebrew_lemmatizer` |
| `snowball_<language>` | 30 languages, see [Snowball stemming](snowball-stemming.md) | `unicode_word`, `nfkc_cf`, the language's stop words, `snowball` |
| `cjk` | [Chinese, Japanese, Korean](cjk.md) | `unicode_word`, `cjk_width`, `lowercase`, `cjk_bigram`, CJK stop words |
| `code` | [Source code](source-code.md), identifiers, paths, logs | `code` tokenizer, `code_split`, `lowercase` |
| `phone` | [Phone numbers](phone-numbers.md), matched in any format and by trailing digits | `phone` tokenizer, region `US` |
| `phone_search` | Phone numbers, matched as whole numbers only | `phone_search` tokenizer, region `US` |
| `cmdline` | [Command lines](command-line-search.md), paths, URLs, registry keys | `cmdline` tokenizer, `lowercase` |
| `cmdline_exact` | Command lines, case preserved | `cmdline` tokenizer |
| `trigram` | [Substring search](substring-search.md) over whole values | `trigram` tokenizer |

The `arabic`, `snowball_<language>` and `cjk` chains follow the Elasticsearch analyzers of the
same languages, and `hebrew` follows the HebMorph analyzer of the Elasticsearch Hebrew plugin, so
an index migrated from Elasticsearch matches the same words.

## Through the compatible APIs

Elasticsearch and OpenSearch mappings and Solr schemas are translated to the analyzers in this
section. When a translation is not exact, the response carries a warning (a `Warning` header, and
for Solr also a `warnings` list in the body) that names what differs.

### Elasticsearch and OpenSearch

A `text` field's `analyzer` can be a built-in analyzer or one defined in `settings.analysis`:

```bash
curl -X PUT https://api.warpsearch.io/compat/es/articles \
  -H "Authorization: Bearer $WARPSEARCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {"analysis": {
      "analyzer": {
        "folded_de": {"type": "custom", "tokenizer": "standard", "filter": ["lowercase", "german_snow"]}
      },
      "filter": {"german_snow": {"type": "snowball", "language": "German"}}
    }},
    "mappings": {"properties": {
      "title_ar": {"type": "text", "analyzer": "arabic"},
      "title_de": {"type": "text", "analyzer": "folded_de"},
      "phone":    {"type": "text", "analyzer": "phone"}
    }}
  }'
```

| Elasticsearch or OpenSearch | WarpSearch |
|---|---|
| Analyzers `arabic`, `cjk` | The `arabic` and `cjk` analyzers |
| Analyzers `hebrew`, `hebrew_query` (the Elasticsearch Hebrew plugin) | The `hebrew` and `hebrew_query` analyzers. `hebrew_exact` and `hebrew_query_light` map to them with a warning |
| Analyzer `english` | The same chain as Elasticsearch's: possessives, lowercase, English stop words, Porter stemming |
| Analyzers `armenian`, `basque`, `brazilian`, `catalan`, `danish`, `dutch`, `estonian`, `finnish`, `french`, `german`, `greek`, `hindi`, `hungarian`, `indonesian`, `irish`, `italian`, `lithuanian`, `norwegian`, `portuguese`, `romanian`, `russian`, `serbian`, `spanish`, `swedish`, `turkish` | The `snowball_<language>` analyzer of that language, with a warning: the stemmer and stop list can differ from Elasticsearch's |
| Analyzer `thai`, tokenizer `thai` | ICU word segmentation (`icu_tokenizer`), with a warning: no Thai stop words |
| Analyzers and tokenizers `phone`, `phone-search` (OpenSearch) | `phone` and `phone_search`; `phone-region` sets `default_region` |
| Tokenizer `icu_tokenizer` | `icu_tokenizer` |
| Filters `snowball`, `stemmer` | `snowball`. Snowball languages are exact; Elasticsearch's light and minimal stemmers (`light_german`, `light_french`, ...) use the Snowball stemmer of the language, with a warning. `stemmer` `arabic` is `arabic_stem` |
| Filters `arabic_normalization`, `cjk_width`, `cjk_bigram`, `icu_folding`, `icu_normalizer`, `icu_transform` | The same filters, with their parameters |
| Filter `reverse` | `reverse`, for a [reversed variant](suffix-search.md) |
| Filter `stop` | `stop` with your word list or `_english_`. Other predefined lists such as `_german_` are dropped with a warning; list the words instead |
| Char filters `html_strip`, `mapping`, `icu_normalizer` | The same char filters. `mapping` needs inline `mappings`, not `mappings_path` |

The WarpSearch names `code`, `phone_search`, `cmdline`, `cmdline_exact`, `trigram` and
`snowball_<language>` also work as analyzer names in a mapping. Bengali, Bulgarian, Czech,
Galician, Latvian, Persian and Sorani have no stemmer here and get the standard analysis, with a
warning.

### Solr

Use a language type from Solr's default configset in `add-field`: `text_en`, `text_ar`,
`text_cjk`, `text_th`, or `text_ca`, `text_da`, `text_de`, `text_el`, `text_es`, `text_eu`,
`text_fi`, `text_fr`, `text_ga`, `text_hi`, `text_hu`, `text_hy`, `text_id`, `text_it`,
`text_nl`, `text_no`, `text_pt`, `text_ro`, `text_ru`, `text_sv` and `text_tr`, which use the
`snowball_<language>` analyzer of their language.

Or define your own type with `add-field-type` and use it in the same request or a later one.
Name each component by class or by name, as in a Solr schema:

```bash
curl -X POST https://api.warpsearch.io/compat/solr/articles/schema \
  -H "Authorization: Bearer $WARPSEARCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "add-field-type": {
      "name": "text_fr_folded",
      "class": "solr.TextField",
      "analyzer": {
        "tokenizer": {"class": "solr.ICUTokenizerFactory"},
        "filters": [
          {"class": "solr.ICUFoldingFilterFactory"},
          {"class": "solr.SnowballPorterFilterFactory", "language": "French"}
        ]
      }
    },
    "add-field": {"name": "title_fr", "type": "text_fr_folded"}
  }'
```

The factories that carry over are the Solr counterparts of the Elasticsearch components above:
the ICU tokenizer, folding, normalizer and transform, `SnowballPorter`, the language stem filters,
Arabic normalization and stemming, CJK width and bigrams, `Stop`, `HTMLStrip`, `LowerCase`,
`ASCIIFolding`, `KStem`, `PorterStem` and `EnglishPossessive`. An analyzer can also be a Lucene
analyzer class, such as `org.apache.lucene.analysis.ar.ArabicAnalyzer`. WarpSearch cannot read
files from a Solr config set, so a stop or mapping filter that names a file is dropped with a
warning. The one exception is `lang/stopwords_en.txt`, which is the English list. A type with
separate `indexAnalyzer` and `queryAnalyzer` uses the index analyzer for both, with a warning.

`ReverseStringFilterFactory` maps to the `reverse` filter. `ReversedWildcardFilterFactory` is
dropped with a warning: Solr keeps the reversed terms in the same field and rewrites leading
wildcards onto them. In WarpSearch, add a reversed variant and query it with `rwildcard`; see
[Suffix search](suffix-search.md).

## Choosing an analyzer

| Your text | Use |
|---|---|
| English | The `_t` suffix, or `snowball_english` for more aggressive stemming |
| Another European language, Hindi, Indonesian, Turkish, ... | `snowball_<language>` |
| Arabic | `arabic` |
| Hebrew | `hebrew` |
| Chinese, Japanese, Korean | `cjk` |
| Several languages in one field | `unicode_word` with `icu_folding`, without stemming, or one field per language |
| Names in several scripts | `icu_transform` to Latin, then `icu_folding` |
| Code, identifiers, log lines | `code` |
| Phone numbers | `phone` |
| Process command lines, file paths, registry keys | `cmdline` |
| Values matched by their ending (`*.exe`) | A `reverse` variant and `rwildcard`, see [Suffix search](suffix-search.md) |
| Exact substrings or patterns anywhere in long values | A `trigram` variant and `substring`, see [Substring search](substring-search.md) |