For AI agents: this page is also available as Markdown at /docs/text-analysis.md. Every docs page is: add .md to its URL. llms.txt lists them all.

Languages and text analysis

A text field's analyzer decides which words a document is indexed under and how a query is split, so it decides what matches. The built-in _t suffix suits English. For other languages, and for values that are not prose, such as source code, phone numbers or command lines, give the field one of the analyzers in this section in its schema.

The same analyzer runs at index time and at query time, so a query in any spelling the analyzer folds together finds the document. Prefix, wildcard and fuzzy terms get only the analyzer's normalizing steps (case, width and accent folding), not stemming or lemmatization.

#Languages

#Use cases

#Use an analyzer

Name a ready-made analyzer as the field's analyzer, as a string or as {"name": ...}:

curl -X PUT https://api.warpsearch.io/v1/collections/articles \
  -H "Authorization: Bearer $WARPSEARCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "schema": {
      "fields": {
        "title_ar": {"type": "text", "analyzer": "arabic"},
        "title_he": {"type": "text", "analyzer": "hebrew"},
        "title_de": {"type": "text", "analyzer": "snowball_german"},
        "title_ja": {"type": "text", "analyzer": "cjk"},
        "source":   {"type": "text", "analyzer": "code"},
        "phone":    {"type": "text", "analyzer": "phone"}
      }
    }
  }'

Or build a chain from the components on these pages, as for any analyzer: optional char_filters, one tokenizer, then filters in order. A component with parameters is {"name": ..., "params": {...}}.

{
  "type": "text",
  "analyzer": {
    "tokenizer": "unicode_word",
    "filters": ["nfkc_cf", {"name": "snowball", "params": {"language": "german"}}]
  }
}

A named analyzer combines only with char_filters, which replace the preset's own. To change anything else, write out the chain.

An analyzer cannot change once a field has data. To switch an existing field, add a new field (or a variant) with the new analyzer and reindex, or create a new collection. Search the field the usual way, for example "query": "title_de:haus" in Lucene syntax or a match query through the Elasticsearch-compatible API.

The Elasticsearch, OpenSearch and Solr compatible APIs use these analyzers too, from the analyzer names and settings those clients already send. See Through the compatible APIs.

#Ready-made analyzers

The arabic, snowball_<language> and cjk chains follow the Elasticsearch analyzers of the same languages, and hebrew follows the HebMorph analyzer of the Elasticsearch Hebrew plugin, so an index migrated from Elasticsearch matches the same words.

#Through the compatible APIs

Elasticsearch and OpenSearch mappings and Solr schemas are translated to the analyzers in this section. When a translation is not exact, the response carries a warning (a Warning header, and for Solr also a warnings list in the body) that names what differs.

#Elasticsearch and OpenSearch

A text field's analyzer can be a built-in analyzer or one defined in settings.analysis:

curl -X PUT https://api.warpsearch.io/compat/es/articles \
  -H "Authorization: Bearer $WARPSEARCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {"analysis": {
      "analyzer": {
        "folded_de": {"type": "custom", "tokenizer": "standard", "filter": ["lowercase", "german_snow"]}
      },
      "filter": {"german_snow": {"type": "snowball", "language": "German"}}
    }},
    "mappings": {"properties": {
      "title_ar": {"type": "text", "analyzer": "arabic"},
      "title_de": {"type": "text", "analyzer": "folded_de"},
      "phone":    {"type": "text", "analyzer": "phone"}
    }}
  }'

The WarpSearch names code, phone_search, cmdline, cmdline_exact, trigram and snowball_<language> also work as analyzer names in a mapping. Bengali, Bulgarian, Czech, Galician, Latvian, Persian and Sorani have no stemmer here and get the standard analysis, with a warning.

#Solr

Use a language type from Solr's default configset in add-field: text_en, text_ar, text_cjk, text_th, or text_ca, text_da, text_de, text_el, text_es, text_eu, text_fi, text_fr, text_ga, text_hi, text_hu, text_hy, text_id, text_it, text_nl, text_no, text_pt, text_ro, text_ru, text_sv and text_tr, which use the snowball_<language> analyzer of their language.

Or define your own type with add-field-type and use it in the same request or a later one. Name each component by class or by name, as in a Solr schema:

curl -X POST https://api.warpsearch.io/compat/solr/articles/schema \
  -H "Authorization: Bearer $WARPSEARCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "add-field-type": {
      "name": "text_fr_folded",
      "class": "solr.TextField",
      "analyzer": {
        "tokenizer": {"class": "solr.ICUTokenizerFactory"},
        "filters": [
          {"class": "solr.ICUFoldingFilterFactory"},
          {"class": "solr.SnowballPorterFilterFactory", "language": "French"}
        ]
      }
    },
    "add-field": {"name": "title_fr", "type": "text_fr_folded"}
  }'

The factories that carry over are the Solr counterparts of the Elasticsearch components above: the ICU tokenizer, folding, normalizer and transform, SnowballPorter, the language stem filters, Arabic normalization and stemming, CJK width and bigrams, Stop, HTMLStrip, LowerCase, ASCIIFolding, KStem, PorterStem and EnglishPossessive. An analyzer can also be a Lucene analyzer class, such as org.apache.lucene.analysis.ar.ArabicAnalyzer. WarpSearch cannot read files from a Solr config set, so a stop or mapping filter that names a file is dropped with a warning. The one exception is lang/stopwords_en.txt, which is the English list. A type with separate indexAnalyzer and queryAnalyzer uses the index analyzer for both, with a warning.

ReverseStringFilterFactory maps to the reverse filter. ReversedWildcardFilterFactory is dropped with a warning: Solr keeps the reversed terms in the same field and rewrites leading wildcards onto them. In WarpSearch, add a reversed variant and query it with rwildcard; see Suffix search.

#Choosing an analyzer

View this page as Markdown