# Hybrid search Keyword search and vector search fail in opposite ways. Keywords miss paraphrases and synonyms, and vectors miss exact identifiers, rare names and part numbers. Hybrid search runs both and merges the results, so a query like "waterproof jacket for hiking" benefits from meaning, and "XJ-4421" still finds the exact product. WarpSearch gives you two ways to do it: - the native `fusion` operation, which ranks each source on its own and merges the lists with reciprocal rank fusion (RRF) or weighted linear fusion, and - the Elasticsearch and OpenSearch style, where a `knn` section sits beside a text `query` and the scores are summed. ## Native fusion > **Coming soon:** the `fusion` operation (RRF and weighted linear fusion) is implemented in the search > engine but the public API does not accept it yet. Today, build hybrid queries with a `knn` section > beside a `query` ([Hybrid with the Elasticsearch-compatible API](#elasticsearch-and-opensearch-style)), > where scores are summed, or run the two searches and merge the lists in your application. The > details below describe the behavior you can expect. A `fusion` operation takes named sources. Each source is a ranked list from its own query, and fusion merges them. ```json { "ops": { "hybrid": { "fusion": { "sources": { "lexical": { "query": { "match": { "title_t": "dune" } }, "limit": 100 }, "semantic": { "query": { "knn": { "field": "embedding_v", "query": [0.75, 0.15, 0.10], "k": 100 } }, "limit": 100 } }, "filter": ["kind_s:fiction"], "rrf": { "k": 60 }, "limit": 10, "get_scores": true, "fields": ["id", "title_t"] } } } } ``` The result stays under the name you gave the operation, here `ops.hybrid`, with the fused documents and scores. A fusion has no shorthand form at the root of a request, so request and response both use named `ops`. | Field | Meaning | |---|---| | `sources` | Named ranked inputs. Each takes its own `query`, `filter`, `sort` and `limit`. | | `filter` | Shared by every source and ANDed with each source's own filter. | | `weights` | Per-source weights keyed by source name. A source without an entry has weight 1. | | `rrf.k` | RRF smoothing constant, at least 0. Defaults to 60. | | `linear.normalization` | Normalization for linear fusion. `min_max` (the default) is the only option today. | | `limit`, `offset`, `fields`, `get_scores` | The shape of the fused response. Set them on the fusion, not on a source. | | `ops` | Further operations, such as facets, over the fused candidates. | Choose exactly one of `rrf` or `linear`. ## Reciprocal rank fusion With RRF, each document scores `weight / (k + rank)` for every source that returned it, summed over sources, with ranks starting at 1. It uses only ranks, so BM25 scores and cosine similarities never need to share a scale. That makes it a robust default that needs no tuning. A document ranked first in both sources above scores `1/61 + 1/61`. A document found by only one source still appears, with a smaller score. ## Weights Weights scale a source's contribution without changing which candidates it brings: ```json { "fusion": { "sources": { "lexical": { "query": { "match": { "title_t": "dune" } }, "limit": 100 }, "semantic": { "query": { "knn": { "field": "embedding_v", "query": [0.75, 0.15, 0.10], "k": 100 } }, "limit": 100 } }, "weights": { "lexical": 1, "semantic": 2 }, "rrf": {} } } ``` Weights must be finite and at least 0. Every key must name a source, and at least one weight must be above 0. A weight of `0` switches that source off completely: it is not run, and its documents do not reach the result. That lets you toggle a source from configuration without rewriting the request. ## Linear fusion Linear fusion combines scores instead of ranks. Each source's scores are rescaled to the range 0 to 1 over its own result list, then summed with the weights: ```text normalized(d) = (score(d) - min) / (max - min) fused(d) = sum over sources of weight * normalized(d) ``` ```json { "fusion": { "sources": { "lexical": { "query": { "match": { "title_t": "dune" } }, "limit": 100 }, "semantic": { "query": { "knn": { "field": "embedding_v", "query": [0.75, 0.15, 0.10], "k": 100 } }, "limit": 100 } }, "weights": { "lexical": 0.3, "semantic": 0.7 }, "linear": { "normalization": "min_max" } } } ``` `min` and `max` are the lowest and highest scores in that source's list, so its best document gets 1 and its worst gets 0. A source that did not return a document contributes 0 to it. If all scores in a list are equal, every document in it gets 1. Prefer RRF when score distributions are unknown or unstable, or when a source is sorted by a field. Prefer linear when score magnitudes carry information you want to keep, such as a clear winner in one source beating a near tie in another, and you can tune weights against relevance judgments. Min-max is sensitive to outliers and to the list size, so retune weights after changing a source's `limit`. Every source under linear fusion must rank by relevance score. ## Pools, filters and facets - **Pool size.** A document outside a source's `limit` cannot contribute from that source. Size each `limit` for the recall your final `limit` needs. 50 to 200 is a common range. - **Paging.** Fusion `offset` pages within the union of the source pools and does not enlarge them. Deep paging needs larger source limits. A per-source `offset` is ignored. - **Filters.** The fusion-level `filter` applies to every source. A source can add its own on top. Filters apply inside the vector search too, not afterwards. - **Facets.** Operations placed in the fusion's `ops` see the whole candidate set, which is every document in any source's list after filters. The fusion's `limit` and `offset` do not change facet counts. For facets over the full keyword match set, add a sibling `top_docs` operation at the request root instead. - **Diversity.** A source can use `collapse` so its `limit` counts groups, and its best members enter fusion. `fusion.collapse` groups the fused list before paging. See [Nearest groups](vector-search.md#nearest-groups) and [Searching](searching.md). ## Elasticsearch and OpenSearch style If your application already speaks Elasticsearch, add a `knn` section beside the `query`. A document matches if either part matches, and its score is the sum of the scores times their boosts: ```bash curl -s https://api.warpsearch.io/compat/es/books/_search \ -H "Authorization: Bearer $WARPSEARCH_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "query": { "match": { "title_t": { "query": "dune", "boost": 0.4 } } }, "knn": { "field": "embedding_v", "query_vector": [0.75, 0.15, 0.10], "k": 50, "num_candidates": 200, "boost": 0.6, "filter": { "term": { "kind_s": "fiction" } } }, "size": 10 }' ``` A few differences from Elasticsearch to keep in mind. Vector scores are WarpSearch similarity scores (raw cosine or dot product), not Elasticsearch's rescaled values, so boosts that worked on one engine may need retuning. A `knn` section's own `filter` restricts only its candidates. To filter the keyword side too, put the filter inside the `query` as well. The OpenSearch `knn` query clause works the same way inside a `bool` query, beside `must` and `should` clauses. On Solr, `q={!knn f=embedding_v topK=50}[0.75, 0.15, 0.10]` runs a vector search, and `fq` filters apply before the neighbors are chosen. See [Elasticsearch compatibility](compat-elasticsearch.md), [OpenSearch compatibility](compat-opensearch.md) and [Solr compatibility](compat-solr.md). Summed scores are not rank fusion. If you want RRF or normalized linear combination, use the native `fusion` operation. ## Recipes | Goal | Setup | |---|---| | Sensible default | RRF with `k` 60, equal weights, 100 candidates per source. | | Semantic-heavy (natural-language questions) | RRF with `weights` such as `lexical: 1, semantic: 2`, or linear with `0.3 / 0.7`. | | Keyword-heavy (product search, codes, names) | Weights such as `lexical: 2, semantic: 1`, or linear with `0.7 / 0.3`. | | Restrict to a segment of data | Put the condition in the fusion `filter` so every source obeys it. | | Turn off vectors for exact-match lookups | Set the `semantic` weight to 0 from your own configuration. | | Boost a field only on the keyword side | Keep boosts in the lexical source's query, not in weights. | Start with the defaults, collect a handful of queries with known good answers, and change one thing at a time. ## Measure recall Approximate vector search can miss true neighbors. To see whether that hurts your hybrid results, run the same request twice and set `"exact": true` on the `knn` in one of them. If the fused lists barely differ, raise your source `limit` or `ivf.nprobe` only as much as needed. If they differ a lot, see the tuning recipes in [Vector search](vector-search.md#tune-recall-and-latency). ## Reranking > **Not supported:** cross-encoder and other reranker models, including Elasticsearch's > `text_similarity_reranker` retriever. WarpSearch does not host models. Do reranking in your application. Retrieve the top 50 to 100 fused results, send them to your reranker, and keep the best few. Because fusion already blends keywords and meaning, a reranker usually only has to reorder a short list. ## Not supported > **Not supported:** the Elasticsearch `retriever` syntax (including `rrf` and `linear` retrievers), and > OpenSearch `hybrid`, `neural` and `neural_sparse` queries with search pipelines. They are rejected > with an `unsupported_feature` error. Use the native `fusion` operation, or a `knn` section beside a > `query`. > **Coming soon:** sparse vectors for learned sparse retrieval (for example SPLADE) and `rank_features` > fields, which suit learned token weights and static signals such as popularity. They are in > development, with no release date. Once available, a sparse source can join the same fusion as the > keyword and dense sources. For the vector side of the story, see [Vector search](vector-search.md) and [Multi-vector fields](multi-vector-fields.md).