Request Analytics Pipeline: Performance Baseline

The request-analytics pipeline stores request and response bodies as independently compressed zstd frames inside organization-isolated pack objects. Ingest flushes at 500 rows, 64 MiB of compressed body data, or 500 ms after the first buffered row — whichever comes first, so single-row low-traffic workloads see the full 500 ms of queueing delay while high-volume workloads see near-zero. Query serves bodies through an authenticated proxy that performs one exact-range object read and verifies the decoded byte count and BLAKE3 hash.

The pre-pack latency and throughput figures previously recorded here measured a pipeline that wrote one object per body and returned pre-signed URLs; they no longer describe the tree and have been retired. No packed-pipeline latency or throughput measurement is recorded yet — re-measure with the harnesses below before relying on absolute numbers.

Verification coverage

The repository exercises pack planning, tenant isolation, exact locators, packed range reads, truncated ranges, hash mismatches, org-prefix pinning, and authenticated proxy URL construction in Rust tests. The compose-backed request analytics test (integration-tests/tests/request_analytics_e2e.rs) covers inline and chunked body round trips through ingest, ClickHouse, and query when a Docker runtime is available. Live-traffic measurement is available through test-suite/python nexus-py-test smoke|stream|latency.

Cardinality of the requests table

The narrow requests schema (ordering key (org_id, project_id, created_at, request_id)) holds well under the following query patterns:

Query	Plan	Rows scanned for 1 M row table
WHERE org_id = ? ORDER BY completed_at DESC LIMIT 50	primary-key prefix scan	~ 50 K (single org)
WHERE org_id = ? AND project_id = ? AND created_at >= ?	primary-key range scan	< 10 K
WHERE request_id = ? without org_id	full table scan	1 M — disallowed by gRPC contract

The query service rejects ListRequests / GetRequest calls without org_id, so the disallowed full-scan path is unreachable from the tenant API.

Metrics

Ingest exports:

• nexus_ingest_body_packs_total
• nexus_ingest_body_pack_frames
• nexus_ingest_body_pack_bytes
• nexus_ingest_minio_put_duration_seconds
• nexus_ingest_batch_insert_duration_seconds
• nexus_ingest_insert_retries_total

Audit exports nexus.audit.batch.events, which records the number of events sealed into each Merkle batch.

These distributions provide the object-rate, bodies-per-pack, upload-latency, insert-latency, retry, and audit-batch inputs used for deployment characterization.

Methodology Notes

• Re-measure when any of: ClickHouse version, object-store backend, JetStream storage mode, the batcher MAX_BATCH_* / pack-size constants, the mpsc channel capacity, or the body inline threshold changes.
• Operators get time-series visibility from the Prometheus scrape metrics (counters and flush-duration/pack histograms) and the OTLP export through vortex-telemetry into the compose collector; the dev stack’s Grafana dashboards chart the ingest pipeline. Alerting rules are operator-supplied.