0023 - Packed object storage for request bodies and audit batches

• Status: Accepted
• Date: 2026-07-29
• Deciders: nexus core team
• Tags: storage, ingest, query, audit, retention

Context and Problem Statement

Request and response bodies, and sealed audit batches, live in the object store rather than in ClickHouse or Postgres. Each body was written as its own object.

Object count therefore scaled one-to-one with request count, and the bodies that dominate that count are small — a few hundred bytes of JSON each. Per-object overhead governed both write cost and retention cost: a sweep had to enumerate objects to find what had aged out, and every body cost a separate PUT.

Decision Drivers

• Write amplification must not scale with request count when most bodies are small.
• A body must remain individually readable, with its integrity verifiable, without fetching unrelated bodies.
• Retention must operate on a key prefix rather than an object walk.
• A body reference must not be forgeable across organizations.
• Audit batch storage must not weaken the tamper-evidence guarantees in 0007 - Tamper-evident audit via per-tenant Merkle chains.

Considered Options

• One object per body — Simplest read path, and the status quo. Write and retention cost scale with request count.
• Packed objects addressed by locator — Ingest concatenates many compressed bodies into one object per batch and records an offset and length per body; reads are ranged GETs.
• Bodies inline in ClickHouse — Removes the object store from the read path, but puts unbounded blobs in analytics rows, ties body retention to the analytics table, and conflicts with keeping bodies out of the tier-1 analytics store.

Decision Outcome

Chosen option: packed objects addressed by locator.

Ingest packs the bodies of a batch into a single zstd-compressed object and records one locator per body of the form pack-v1:<bucket>/<key>#<offset>:<length>:zstd (services/ingest/src/batcher.rs in the source repository). Audit batches are likewise sealed as a single object per batch.

Query resolves a locator with a ranged GET and verifies the body hash before returning it (services/query/src/store.rs in the source repository). Locator parsing is strict: the pack-v1: prefix is required, zstd is the only accepted codec, a zero length is rejected, path traversal in the key is rejected, and the key must sit under the requesting row’s organization prefix, so a locator pinned outside that organization is refused.

Pack keys carry a dated prefix, so a retention sweep addresses a prefix instead of enumerating objects.

There is no dual-read path. A body reference that is not a pack-v1: locator does not parse, and the body proxy answers 502 (services/query/src/bodies_http.rs in the source repository).

Sealing changes the container an audit batch is stored in, not its canonical encoding, its Merkle chain, or the signature over the batch root, so the verifier’s guarantees are unchanged.

Consequences

• Write volume and object count are governed by batch count rather than request count, and retention operates on a dated prefix.
• Reads cost one ranged GET per body, and a body is verified against its recorded hash on every read.
• A locator cannot be used to read across organizations, and a forged one is refused rather than served.
• The cutover is not backwards compatible. References written before this change are unreadable and their body reads answer 502. There is no feature flag, and ingest and query must be deployed together rather than rolled independently, because a locator written by one is only readable by the other.
• The audit archive keeps the layout the verifier expects — meta-roots/*.json and batches/<tenant_id>/<batch_id>/events.ndjson.zst — so out-of-band export and verification are unaffected.
• End-to-end coverage of the pack write-and-read path is integration-tests/tests/ingest_body_packing.rs in the source repository, which runs against a real object store in its own compose project. It is #[ignore]d and gated behind the compose-it feature, so it runs on request rather than in the default test run.