For AI agents: this page is also available as Markdown at /docs/source-code.md. Every docs page is: add .md to its URL. llms.txt lists them all.
Source code
Code, stack traces, file paths and log lines break the assumptions of a prose analyzer: it splits std::vector and a.b.c apart and keeps getHTTPResponse as one word nobody types. The code analyzer keeps getHTTPResponse_v2, std::vector, a.b.c, foo-bar, 0xFF and 1.2.3 together as tokens and also indexes their parts, so the whole identifier and each word in it match: getHTTPResponse finds the document, and so do http and response.
{"fields": {"source": {"type": "text", "analyzer": "code"}}}
The chain is the code tokenizer, code_split and lowercase.
#Components
| Component | Kind | Does |
|---|---|---|
code | tokenizer | Splits on whitespace, brackets, /, \, = and other punctuation, keeping identifiers, dotted and :: paths, kebab-case names, numbers, hex and versions whole. A path or URL gives one token per component. Param keep_operators (default false) also emits operators such as == and ->. |
code_split | filter | Splits a token on _ - . : $, camelCase and letter/digit changes: HTTPServer2Go gives HTTP, Server, 2, Go. |
code_split params:
| Param | Default | Meaning |
|---|---|---|
preserve_original | true | Also keep the whole token. |
split_on_numerics | true | Split where letters and digits meet. |
min_part_length | 1 | Drop split-off parts shorter than this. |
stack_parts | false | Put every part at the whole token's position. By default the parts follow one another, so a phrase query such as "http response" matches getHTTPResponse. |
For example, to index operators and drop one-letter parts:
{
"type": "text",
"analyzer": {
"tokenizer": {"name": "code", "params": {"keep_operators": true}},
"filters": [{"name": "code_split", "params": {"min_part_length": 2}}, "lowercase"]
}
}
#Notes
- The tokenizer is not a lexer: strings and comments are split the same way as code.
code_splitdoes not run on prefix, wildcard and fuzzy terms, sogetHttp*matches whole identifiers that start withgethttp, after lowercasing.- For Windows command lines, executable paths and registry keys, the
cmdlineanalyzer splits on more punctuation and handles switches. For exact substrings of long values, see Substring search. codeworks as an analyzer name in an Elasticsearch or OpenSearch mapping too.