# Highlighting

Highlighting returns excerpts of a document with matching words marked. Add `highlight` to a
standard search request. You can retrieve only an ID and title while highlighting a longer body;
the highlighted field does not have to be in the returned `fields` list.

## Native API

Send a normal request to `POST /v1/collections/articles/search`:

```json
{
  "language": "lucene",
  "query": "body_t:climate",
  "fields": ["id", "title"],
  "highlight": {
    "fields": ["body_t"],
    "backend": "auto",
    "fragment_size": 150,
    "fragments": 2
  }
}
```

The native response puts snippets in each hit's `highlight` map, alongside `fields`:

```json
{
  "id": "123",
  "fields": {"title": "Weather report"},
  "highlight": {
    "body_t": ["Changing <em>climate</em> affects rainfall."]
  }
}
```

Fields without a match are omitted unless `no_match_size` requests a fallback prefix. The
`highlight` option also works with the other `/v1` query languages and the MCP `search` tool.
For a native WarpSearch expression, use the same options with `language: "warpsearch"`:

```json
{
  "language": "warpsearch",
  "query": "body_t:$term",
  "vars": {"term": "climate"},
  "fields": ["id"],
  "highlight": {"fields": ["body_t"], "backend": "analysis"}
}
```

[Native expression access rules](query-languages.md#native-queries-and-replays) still apply.
Highlighted searches currently have no signed `/translate` replay form; send the original
request again. Field aliases and field permissions apply to highlighting, including
`match_fields` and `highlight_query`. Wildcard field selectors are unavailable on collections
with restricted field lists.

If calling the underlying engine's `/collections/{collection}/_search` directly, omit the
`language` wrapper and put the query and `highlight` in the same request. Its result column is
named `_highlight_`; the gateway converts that column to the `highlight` map shown above.

## Choose a backend

| Backend | Index requirement | Work at search time |
|---|---|---|
| `analysis` | Stored source text | Analyze source with the matching field's analyzer and locate matching terms and phrases. |
| `positions` | Stored source and indexed token positions | Read matching postings positions, then analyze the source prefix needed to map positions to source offsets. |
| `stored_offsets` | A `highlight_text` field with `stored_offsets: true`, configured before indexing | Read persisted source offsets and token data without analyzing the text again. |
| `auto` (default) | Source available | Select stored offsets when configured; otherwise analysis. |

Explicit selections do not silently fall back. An unavailable backend returns an error.
Positions are unavailable on docs-only keyword fields and fields with `index: "none"`.
No backend depends on norms; there is currently no separate text-field norms toggle in the
native schema. `auto` keeps the positions optimization opt-in. Stored chunks must still be read
and decompressed, and character filters may process the full source; benchmark your workload
before choosing a backend for latency reasons.

Ordinary `text` fields store source by default. A `string` or `id` field needs stored source to
be highlighted; matching a keyword highlights the whole value. A field with `stored: false`
cannot be highlighted from its postings alone.

### Configure stored offsets

Create the field through the [native schema API](schemas-and-fields.md) before ingesting documents.
For example, `PUT /v1/collections/articles` with:

```json
{
  "schema": {
    "fields": {
      "body": {
        "type": "highlight_text",
        "params": {"analyzer": "english", "stored_offsets": true}
      }
    }
  }
}
```

Index `body` as a string or an array of strings. Named analyzer presets are supported, including
`standard` (the default), `english`, and enabled analysis plugin presets. Inline custom analyzer
chains are not supported by this field type.

Through the gateway, query and highlight the logical name `body`:

```json
{
  "language": "lucene",
  "query": "body:climate",
  "fields": ["id", "body"],
  "highlight": {"fields": ["body"], "backend": "stored_offsets"}
}
```

The gateway resolves lexical queries to the field's private text representation. Clients do not
name its internal `$text` or `$data` fields. Raw `warpsearch` expressions are opaque to the
gateway and do not get this field rewrite; use a translated language such as `lucene` or `es`
for queries on `highlight_text` fields.

`stored_offsets: false` keeps source and indexed positions, so analysis and positions remain
available. Enabling stored offsets or changing the analyzer requires reindexing. Existing plain
text fields do not gain offsets retroactively. Elasticsearch mapping options such as
`term_vector` and `index_options` do not configure this plugin storage; use the native field
configuration above.

## Fragment and rendering options

Use defaults at the top level and override them inside a field's object:

```json
{
  "language": "lucene",
  "query": "title:climate OR body_t:climate",
  "highlight": {
    "encoder": "html",
    "fields": {
      "title": {"backend": "analysis", "fragments": 0},
      "body_t": {"backend": "positions", "fragment_size": 150, "fragments": 2}
    }
  }
}
```

| Option | Default | Meaning |
|---|---|---|
| `fields` | Required | Field name, list of names, or map of names to options. `*` patterns select eligible fields on unrestricted collections. |
| `fragment_size` | `100` | Target fragment length in Unicode code points; matches stay whole. |
| `fragments` | `1` | Maximum fragments across the field's values; `0` returns whole matching values, subject to limits. |
| `pre_tag`, `post_tag` | `<em>`, `</em>` | Markup around matches. |
| `encoder` | `html` | Escape source text; `none` returns it verbatim. Tags are caller-supplied markup. |
| `boundary` | `word` | `word`, `sentence`, or `none`. Sentence boundaries use bounded punctuation scanning. |
| `boundary_max_scan` | `20` | Extra code points considered for sentence boundaries. |
| `order` | `position` | Return fragments in source order, or `score` order. |
| `phrase_spans` | `true` | Mark a matching phrase as one span; false marks its matched tokens individually. |
| `require_field_match` | `true` | Restrict matching to the highlighted logical field. False applies positive query terms to other highlighted fields too. |
| `match_fields` | Empty | Explicit same-source representations, such as variants. Cross-source offset mapping is rejected. |
| `no_match_size` | `0` | Return a source prefix when there is no lexical match. |
| `ellipsis` | Empty | Text inserted at clipped value edges. |
| `max_analyzed_chars` | `1000000` | Per-value code point limit; truncation produces a warning. |
| `format` | `tagged` | Strings, or `offsets` objects as shown below. |
| `highlight_query` | Ranking query | Request-level override using the request's filter/clause syntax: Elasticsearch DSL, Solr query strings, native expressions, or Lucene clauses for the other languages. It does not change matching or ranking. |

All queried variants of a logical field contribute source-offset matches. Fragments never join
separate array values. Phrase occurrences are verified, while negative clauses and separately
supplied filters do not contribute spans. Highlighting extracts positive lexical matches; it is
not a complete explanation of Boolean matching or vector similarity. For vector-only searches,
supply a lexical `highlight_query` if you want highlighted text.

### Source offsets

`format: "offsets"` returns unescaped source text and spans instead of injecting markup:

```json
{
  "text": "a cat",
  "value_index": 0,
  "start": 10,
  "end": 15,
  "spans": [[2, 5]]
}
```

`start` and `end` are half-open UTF-8 byte offsets into the original value. Each span is a
half-open UTF-8 byte range relative to `text`. `value_index` identifies the original array value.
Fragment sizes and analysis limits count code points, not bytes or JavaScript UTF-16 units.

## Elasticsearch and OpenSearch

Use the normal `highlight` object on
`POST /compat/es/articles/_search` or `POST /compat/opensearch/articles/_search`:

```json
{
  "query": {"match": {"body_t": "climate"}},
  "_source": ["title"],
  "highlight": {
    "type": "unified",
    "encoder": "html",
    "pre_tags": ["<mark>"],
    "post_tags": ["</mark>"],
    "fields": {
      "body_t": {"fragment_size": 150, "number_of_fragments": 2}
    }
  }
}
```

Each entry in `hits.hits` carries a `highlight` map. It is independent of `_source` and is
returned even with `_source: false`. `_msearch`, hybrid/fusion searches, collapsed inner hits,
and `top_hits` aggregations support highlighting too. Grafana's Elasticsearch/OpenSearch
requests use the same compatibility endpoints.

| Elasticsearch / OpenSearch setting | WarpSearch behavior |
|---|---|
| `type: "unified"` (default) | `auto` backend. |
| `type: "plain"` | Analysis backend. |
| `type: "fvh"` | Stored-offset backend; requires the native `highlight_text` configuration above. |
| `fields` | Object or array of one-field objects; per-field options supported. |
| `number_of_fragments` | `fragments`; default `5`, and `0` returns whole matching values. |
| `fragment_size`, `no_match_size`, `order`, `encoder` | Corresponding native options. Compatibility encoder defaults to `none`; set `html` when rendering source as HTML. |
| `pre_tags`, `post_tags` | Exactly one string per tag array. |
| `require_field_match` | Supported; default true. |
| `matched_fields` | Same-source `match_fields`. |
| `highlight_query` | A request-level Elasticsearch Query DSL clause, authorized like the search query. |
| `boundary_scanner` | `word` or `sentence`; default sentence. |
| `boundary_max_scan` | Native bounded boundary scan. |
| `max_analyzed_offset` | Positive values set the code point limit; `-1` uses the native default. |

These type names select WarpSearch backends, not Lucene implementations. Fragment boundaries,
scoring, and character-limit semantics can differ. Semantic highlighters, multiple tag pairs,
`tags_schema`, `phrase_limit`, `fragmenter`, `fragment_offset`, custom boundary characters,
locale-specific boundary scanners, and per-field `highlight_query` are rejected with
`unsupported_feature`, rather than silently ignored. Highlighting does not add support for
otherwise unsupported query types such as nested-document queries.

## Solr and SolrJ

Use standard Solr highlighting parameters on `/compat/solr/articles/select`:

```bash
curl 'https://api.warpsearch.io/compat/solr/articles/select?q=body_t:climate&fl=id,title&hl=true&hl.fl=body_t&hl.snippets=2&hl.fragsize=150&hl.encoder=html' \
  -H "Authorization: Bearer $WARPSEARCH_API_KEY"
```

Solr responses contain the usual separate `highlighting` object, keyed by document ID:

```json
{
  "highlighting": {
    "123": {"body_t": ["Changing <em>climate</em> affects rainfall."]}
  }
}
```

The JSON Request API accepts the same settings inside `params`. Form-encoded requests and
`wt=javabin` are supported; SolrJ receives the corresponding named-list structure.

| Solr parameter | Behavior |
|---|---|
| `hl=true` | Enable highlighting. |
| `hl.fl` | Comma/space-separated or repeated fields; defaults to `df` or collection default fields. |
| `hl.method=unified` | Auto backend (default). |
| `hl.method=original` | Analysis backend. |
| `hl.method=fastVector` | Stored-offset backend; requires `highlight_text`. |
| `hl.snippets`, `hl.fragsize` | Fragment count (default 1) and size (default 100). `hl.fragsize=0` requests whole matching values; otherwise `hl.snippets=0` disables that field. |
| `hl.tag.pre`, `hl.tag.post` | Tags; take precedence over `hl.simple.pre` / `hl.simple.post`. |
| `hl.encoder` | `none` by default, or `html`. |
| `hl.requireFieldMatch` | False by default, so terms may highlight other requested fields. |
| `hl.q`, `hl.qparser` | Separate highlighting query and its parser. |
| `hl.maxAnalyzedChars` | Code point limit; default 51200. `-1` uses the native maximum, still subject to resource limits. |
| `hl.bs.type`, `hl.bs.maxScan` | `WORD` / `SENTENCE` boundaries and maximum scan distance. |
| `hl.defaultSummary=true` | Return a prefix when no matching fragment exists. |

Per-field forms such as `f.body_t.hl.fragsize=200` are supported. Grouped document lists can
carry highlights. `hl.usePhraseHighlighter` and `hl.highlightMultiTerm` must remain true.
Alternate fields, custom fragmenters/formatters, scoring tunables, and other unmapped `hl.*`
settings are rejected. Solr method names select the backends above; they do not reproduce
Solr's fragment scoring exactly.

## Other query languages and limits

SQL, PostgreSQL-style SQL, Lucene, Grafana models and LogQL searches can request native
`highlight` through `/v1`. PostgreSQL's `ts_headline()` SELECT expression remains unsupported;
use the request option instead. There is no Loki HTTP API, and the turbopuffer compatibility API
does not define a highlighting option.

The plugin requires matching gateway protobufs and a node build containing the highlighter.
It runs after paging and does not change document membership, order or scores. Normal plan
limits, collection permissions and timeouts still apply. Highlight queries count toward query
byte, clause and depth limits. At most 64 highlighted field selectors and 100 fragments per
field are allowed; combined tags are limited to 4096 bytes. Engine-side expansion can also
reject a wildcard selecting more than 64 eligible fields.

Work is bounded by source, token, phrase and passage-selection limits. Request-level native
`max_expansions` (default 4096) and `max_terms_examined` (default 100000) bound dictionary work.
Exceeding these limits fails explicitly. Source truncation returns a warning and never invents
a match from a partial token. See [Limits and errors](limits-and-errors.md).