The storage problem underneath a tool like this is unusual enough that the obvious answers are all slightly wrong. Here is the shape of it, and the three designs we went through.
The requirements
- Append-mostly. Data arrives in time order, mostly. Late arrivals happen and must not corrupt anything.
- Wildly heterogeneous. One recording holds a million-point cloud, a scalar logged at 1 kHz, and a text log. These have nothing in common except a timestamp.
- Queried by time, read by column. The dominant query is "for this entity, at this timestamp, give me the latest value of each component." You almost never want a whole row.
- Bigger than memory. Routinely.
Attempt one: row store
A row per logged event, with a blob payload. Simple, and wrong for the dominant query: reading positions for a million points meant touching a million rows, each carrying colour and radius bytes we did not want. Cache behaviour was as bad as it sounds.
Attempt two: per-component time series
Split each component into its own time-ordered array. Much better for reads. The problem was
write amplification and bookkeeping: a single log call fans out into five or six
independent appends, each needing its own index, and a crash mid-call could leave components at
different timestamps. Reconstructing a consistent view got complicated fast.
Attempt three: columnar chunks
What we landed on. Data is batched into chunks; within a chunk, each component is a contiguous typed column. A chunk covers a bounded time range for one entity, and carries its own min/max timestamps.
This makes the dominant query cheap. Latest-at for a component becomes: find candidate chunks by time range, binary search within the chunk's timestamp column, read one value. You touch only the column you asked for.
// Latest-at reduces to a range scan plus a binary search, per component. let chunks = index.chunks_overlapping(entity, ..=query_time); let latest = chunks.rev().find_map(|c| c.column(component)?.at_or_before(query_time));
Why Arrow
Having arrived at columnar chunks with typed columns and a schema, we had reinvented most of Apache Arrow, badly. Adopting it directly gave us the memory layout, zero-copy IPC between the SDK and the viewer, and the ability to hand a recording to pandas or Polars without a conversion step.
The dataframe API exists because of this decision. It was not planned — it fell out of the storage format being something other tools already understood.
What it costs
Chunking means a tunable latency between logging and visibility, because a chunk is not queryable until it is sealed. We flush on size or on a timer, whichever comes first. For live debugging that shows up as a small lag, and it is the main thing people notice.