Streaming ingest · low-latency reads
Tickets, logs, feeds, real-time content
Continuous writes from webhooks, message buses, or user activity. New content must be searchable within seconds, without rebuilding an index every time.
Keep your search index current without rebuilding it. Live documents appear in results immediately, no batch job required.
01
Stream into the live index
REST or streaming ingest accepts batches and individual writes. The live index makes them queryable immediately.
02
Background compaction
Periodically merges into the compressed base index. Reads continue uninterrupted; queries see a unified view.
03
Two-level search hides the seam
Each query searches both the live tier and the compacted base, then merges results. Fresh and historical content rank together.
Best fit when
- Writes are continuous and recall must reflect the last few seconds
- You can tolerate eventual compaction (read latency stays flat)
- Per-document metadata varies and you want it queryable
- You don't want to operate Kafka + a vector DB + a rebuild scheduler
Wrong choice when
- Writes need sub-100ms search visibility (not seconds)
- You need ACID multi-document transactions on ingest
- You need cross-region active-active writes today
~seconds
ingest-to-searchable
Online
compaction
Two-tier
live + base reads