# Full-text search

Full-text search finds documents by the words they contain and ranks them by how well they match.
A database filter answers yes or no; a full-text query answers "how relevant?" and returns the
best matches first. This page is a map of the ideas, and each section links to the page that
covers it in depth.

## How it works

When you write a document, WarpSearch splits each `text` field into words with an **analyzer** and
stores them in an inverted index: for every word, the documents that contain it. A query goes
through the same analyzer, so `Running`, `running` and, with a stemmer, `runs` reach the same index
entry. Looking up a few words is fast whatever the size of the collection.

The analyzer decides what matches. The `_t` suffix gives you word splitting, case and accent
folding and English stemming. For other languages, and for values that are not prose, such as
source code or phone numbers, pick a ready-made analyzer: see
[Languages and text analysis](text-analysis.md) and
[Schemas and field types](schemas-and-fields.md#analyzers). A `string` field is not analyzed and
matches whole values, which suits ids and categories.

## The basic query types

| Query | Finds |
|---|---|
| Words | Documents containing them after analysis. Documents with more of the words rank higher. |
| A phrase, `"way of kings"` | The words in order. A slop, `"way kings"~2`, allows words in between. |
| A prefix or wildcard, `kin*` | Indexed terms that fit a pattern, useful for search-as-you-type. |
| A fuzzy term, `kings~1` | Terms within one or two character edits, so typos still match. |
| A range, `year:[1960 TO 1980]` | Numbers, dates or terms between two bounds. |
| Boolean combinations | `must`, `should` and `must_not` clauses, or `AND`, `OR` and `-`. |

Filters narrow the candidates without changing the score. [Searching](searching.md#what-each-query-type-does)
lists every query type, and [Query languages](query-languages.md) shows how to write them in SQL,
Lucene, the Elasticsearch Query DSL and others.

## BM25 and scoring

Every match gets a score, and results are sorted by it. Keyword matches are scored with **BM25**,
the standard ranking function for full-text search. It rewards:

- **Term frequency.** More occurrences score higher, with diminishing returns.
- **Rarity.** A rare word such as `mistborn` counts for far more than `the`.
- **Length.** A match in a short field, such as a title, beats the same word in a long article.

Scores compare only within one query, so do not compare them across queries or collections. If you
use a minimum score as a cut-off, calibrate it on your own data.

## Semantic and vector search

Keyword search matches words, not meaning: `cheap flights` will not find `low-cost airfares`.
**Semantic search** compares meaning instead. An embedding model turns text into a vector so that
similar meanings sit close together; you embed the query the same way and look for the nearest
vectors.

**Vector search** is the retrieval half: store embeddings in a `vector` field and run an exact or
approximate nearest-neighbor (kNN) query, with filters applied inside the search. You bring your
own embeddings. See
[Vector search](vector-search.md), [Dimensions and storage trade-offs](matryoshka-and-dimensions.md)
and [Multi-vector fields](multi-vector-fields.md).

Vectors handle paraphrases and differently worded questions well, and are weaker at exact
identifiers, rare names and part numbers, where a keyword match is what you want.

## Hybrid search

Each approach fails where the other succeeds, so **hybrid search** runs both in one request and
merges the ranked lists. "Waterproof jacket for hiking" matches on meaning, and `XJ-4421` still
finds the exact product.

BM25 scores and vector similarities are on different scales, so the usual merge is
**reciprocal rank fusion** (RRF), which uses only each document's rank in each list and works
without tuning. You can also weight the two sides or sum their boosted scores. See
[Hybrid search](hybrid-search.md). It is the usual choice for retrieval-augmented generation (RAG)
and for most product and document search.

## Relevance tuning

[Tuning relevance](tuning-relevance.md) covers each of these in detail. Work from the cheapest lever to the most involved:

1. **Fix the analysis.** Most "why didn't this match?" problems come from the wrong language, no
   stemming, or accents and punctuation handled differently from how users type. Choose the right
   [analyzer](text-analysis.md) first.
2. **Search the right fields.** Query title and body together and boost the title, since a title
   match is stronger evidence.
3. **Filter what is not about relevance.** Category, availability and dates belong in filters,
   which do not distort scores. [Facets](facets-and-aggregations.md) let users narrow further.
4. **Set strictness.** Require all words or a share of them with `minimum_should_match`, favor
   words that appear together with phrases and slop, allow fuzziness for typos, and drop weak
   matches with a minimum score.
5. **Balance keywords and vectors.** In a hybrid query, change the weights, boosts or fusion
   method, and measure the effect.
6. **Rerank in your application.** Send the top 50 to 100 fused results to your own reranker,
   such as a cross-encoder.

Tune against a set of real queries with the results you expect, change one thing at a time, and
check the whole set. [Hybrid search](hybrid-search.md#measure-recall) shows how to measure recall.

## Where to go next

1. [Searching](searching.md) to run keyword queries over the REST API.
2. [Languages and text analysis](text-analysis.md) to match words the way your users write them.
3. [Vector search](vector-search.md) and [Hybrid search](hybrid-search.md) to search by meaning.
4. [Geo search](geo-search.md) to combine text with location.