For AI agents: this page is also available as Markdown at /docs/full-text-search.md. Every docs page is: add .md to its URL. llms.txt lists them all.

Full-text search

Full-text search finds documents by the words they contain and ranks them by how well they match. A database filter answers yes or no; a full-text query answers "how relevant?" and returns the best matches first. This page is a map of the ideas, and each section links to the page that covers it in depth.

#How it works

When you write a document, WarpSearch splits each text field into words with an analyzer and stores them in an inverted index: for every word, the documents that contain it. A query goes through the same analyzer, so Running, running and, with a stemmer, runs reach the same index entry. Looking up a few words is fast whatever the size of the collection.

The analyzer decides what matches. The _t suffix gives you word splitting, case and accent folding and English stemming. For other languages, and for values that are not prose, such as source code or phone numbers, pick a ready-made analyzer: see Languages and text analysis and Schemas and field types. A string field is not analyzed and matches whole values, which suits ids and categories.

#The basic query types

Filters narrow the candidates without changing the score. Searching lists every query type, and Query languages shows how to write them in SQL, Lucene, the Elasticsearch Query DSL and others.

#BM25 and scoring

Every match gets a score, and results are sorted by it. Keyword matches are scored with BM25, the standard ranking function for full-text search. It rewards:

• Term frequency. More occurrences score higher, with diminishing returns.
• Rarity. A rare word such as mistborn counts for far more than the.
• Length. A match in a short field, such as a title, beats the same word in a long article.

Scores compare only within one query, so do not compare them across queries or collections. If you use a minimum score as a cut-off, calibrate it on your own data.

#Semantic and vector search

Keyword search matches words, not meaning: cheap flights will not find low-cost airfares. Semantic search compares meaning instead. An embedding model turns text into a vector so that similar meanings sit close together; you embed the query the same way and look for the nearest vectors.

Vector search is the retrieval half: store embeddings in a vector field and run an exact or approximate nearest-neighbor (kNN) query, with filters applied inside the search. You bring your own embeddings. See Vector search, Dimensions and storage trade-offs and Multi-vector fields.

Vectors handle paraphrases and differently worded questions well, and are weaker at exact identifiers, rare names and part numbers, where a keyword match is what you want.

#Hybrid search

Each approach fails where the other succeeds, so hybrid search runs both in one request and merges the ranked lists. "Waterproof jacket for hiking" matches on meaning, and XJ-4421 still finds the exact product.

BM25 scores and vector similarities are on different scales, so the usual merge is reciprocal rank fusion (RRF), which uses only each document's rank in each list and works without tuning. You can also weight the two sides or sum their boosted scores. See Hybrid search. It is the usual choice for retrieval-augmented generation (RAG) and for most product and document search.

#Relevance tuning

Tuning relevance covers each of these in detail. Work from the cheapest lever to the most involved:

1. Fix the analysis. Most "why didn't this match?" problems come from the wrong language, no stemming, or accents and punctuation handled differently from how users type. Choose the right analyzer first.
2. Search the right fields. Query title and body together and boost the title, since a title match is stronger evidence.
3. Filter what is not about relevance. Category, availability and dates belong in filters, which do not distort scores. Facets let users narrow further.
4. Set strictness. Require all words or a share of them with minimum_should_match, favor words that appear together with phrases and slop, allow fuzziness for typos, and drop weak matches with a minimum score.
5. Balance keywords and vectors. In a hybrid query, change the weights, boosts or fusion method, and measure the effect.
6. Rerank in your application. Send the top 50 to 100 fused results to your own reranker, such as a cross-encoder.

Tune against a set of real queries with the results you expect, change one thing at a time, and check the whole set. Hybrid search shows how to measure recall.

#Where to go next

1. Searching to run keyword queries over the REST API.
2. Languages and text analysis to match words the way your users write them.
3. Vector search and Hybrid search to search by meaning.
4. Geo search to combine text with location.

View this page as Markdown