For AI agents: this page is also available as Markdown at /docs/consistency-and-durability.md. Every docs page is: add .md to its URL. llms.txt lists them all.

Writes, visibility and durability

WarpSearch separates accepting a write from making it searchable. Understanding that gap, and the few rules around it, is all you need to load data safely and to know what a search can and cannot see.

#Accepted, then committed, then searchable

A write goes through two steps:

1. Accepted. WarpSearch validates the documents and records them. The response tells you how many were indexed and which failed.
2. Committed. A commit publishes a new, immutable view of the collection. Searches see documents only after the commit that includes them.

You choose how the commit happens:

Do not commit after every small write. Each commit publishes a new view, and explicit refreshes are billed operations; see Pricing. For a bulk load, send the batches without commit=true and commit with the last one.

curl -X POST "https://api.warpsearch.io/v1/collections/books/documents?commit=true" \
  -H "Authorization: Bearer $WARPSEARCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"id": "b1", "title": "Dune", "year": 1965}]'

A document fetched by id (GET /v1/collections/{c}/documents/{id}) follows the same rule: it is visible once committed, not instantly.

A search never sees half of a commit. A commit is crash-safe: either the whole new view is published or the previous one stays.

#Replace is whole-document

A document is identified by its id. Writing a document with an existing id replaces the old one completely. Fields you leave out of the new version disappear.

# Before: {"id": "b1", "title": "Dune", "year": 1965}
curl -X POST "https://api.warpsearch.io/v1/collections/books/documents?commit=true" \
  -H "Authorization: Bearer $WARPSEARCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"id": "b1", "title": "Dune"}]'
# After: {"id": "b1", "title": "Dune"}   (year is gone)

String and numeric ids are accepted, and 123 and "123" are the same id. A document with no id can be searched but cannot be replaced or deleted by id, so always send one.

Not supported: partial or field-level updates, and delete by query. Read the document, change it in your application and write the whole document back. To delete many documents, delete them by id.

Deleting an id that is not there is a successful no-op, so deletes are safe to repeat.

#Retrying safely

Because a write with an id replaces the previous version, retrying the same batch is harmless: you end up with one copy of each document. Make your loader retry-friendly:

• Always send stable ids, derived from your source data.
• Read the response body. A request can return 200 and still list per-document failures, and 207 when part of a large request was not written; the errors list gives each failed document's index and id. Retry exactly those. See Ingesting documents.
• After a timeout or a dropped connection, assume the write may have been applied and send it again. WarpSearch does not retry writes for you, so a timeout can mean either outcome.
• On 429 (overload) or 503 warming_up, wait for Retry-After and retry.
• Use all_or_none=true when a batch must apply entirely or not at all. It covers one request and must fit in a single backend update, so keep such batches small.

If you pass allow_dups through a compatible API (for example Solr's overwrite=false) to speed up a load, retries can create duplicates. Leave it off unless every id is new.

#Durability

WarpSearch keeps collection data in durable storage, and the compute that serves a collection is disposable. A commit makes your documents durable and visible, and WarpSearch copies committed changes to durable storage shortly afterwards. An idle collection is saved there too and reloaded on its next request.

What this means in practice:

• A write is guaranteed only after its commit. Writes accepted but not yet committed can be lost if the infrastructure serving the collection fails at that moment.
• A very recent commit may not have reached durable storage yet when such a failure happens, so the newest minutes of a heavy load are the exposed ones.
• Do not treat WarpSearch as your system of record while a load is in flight. Keep the source data until the load has finished and a count or a sample search confirms the documents are there, and load with stable ids so you can replay.
• Collection deletion is permanent. There is no undo and no backup you can restore from yet, so keep your source of truth and reload by replaying it.

Schema changes follow the same model: they apply to documents written after the change and do not rewrite old ones. See Schemas and field types.

#Reads see one consistent view

A single search request, including its hits, facets and metrics, is computed over one consistent view of the collection, so a facet count never disagrees with the hit total because a commit landed in between. Two separate requests can see different views if a commit lands between them.

#Cold starts

A collection with no recent requests is unloaded and reloaded on demand. The first request after that can answer 503 warming_up with Retry-After. No data is lost and nothing is billed for the wait. See Collections.

#Quick reference

View this page as Markdown