Nexus Log Storage

How Nexus persists audit events, request/session analytics, opt-in request bodies, and native agentic traces.

These are separate pipelines. Request analytics are not the audit log.

Canonical companions:

• Tracing and Auditing — what is always-on vs opt-in
• Data Models — ClickHouse and Postgres table shapes
• Services — service ownership and ports
• Telemetry Tiering — Tier 1 tenant analytics vs Tier 2 operator OTLP

---

Quick map

Kind	Hot store	Cold / payload store	Bus
Audit	Postgres nexus_audit	Audit archive (object store)	nexus.audit.events.*
Request / session metadata	ClickHouse requests (+ usage/cost lines)	—	nexus.requests.*
Full request / response bodies	Refs on requests	Bodies packs (object store)	Same request envelope
Agentic traces	ClickHouse trace_*	Trace artifacts (object store, opt-in)	nexus.traces.*
Operator spans / logs	Tempo / Loki (dev LGTM)	—	OTLP Collector

---

1. Audit logs

Purpose: Tamper-evident accountability (who did what), not request debugging.

Path

1. Emitters (gateway, control-plane, auth, ingest, and others) publish to NATS nexus.audit.events.<tenant_id>.
2. Control-plane and auth also write to Postgres staging: audit_events_staging in nexus_control / nexus_auth.
3. The audit service consumes NATS and inserts into sealed Postgres nexus_audit (audit_events, batch roots, signing keys, tenant counters).
4. A sealer builds Merkle batches (about 1,000 events), signs them, and archives meta-root manifests plus zstd batch files to the audit object store (S3/MinIO or filesystem).

Local exception

Gateway-only / CLI flows can also write JSONL to --state-dir/audit.log. That file is operational evidence only; it is not the cryptographic chain.

Reads

Audit HTTP/gRPC facade → console proxies under /api/audit/**. Viewing an event emits data_access.audit_viewed.

---

2. Request / session logs

Purpose: Always-on tenant analytics — identity, routing, status, latency, tokens, cost, and safe tool summary.

Path

1. Gateway publishes completed-request envelopes on NATS nexus.requests.<org>.* (nexus_REQUESTS stream).
2. ingest materializes into ClickHouse:
   • requests — primary row (ReplacingMergeTree)
   • request_usage_lines / request_cost_lines
   • rollups such as metrics_by_minute and usage_monthly_rollup
3. After a durable ClickHouse write, ingest publishes data.write.request_logged into the audit pipeline (deterministic event id), then acks NATS.

Sessions

Sessions are not a separate table. session_id, session_path, and session_name live on requests. Query aggregates them (GROUP BY session_id) for list, timeline, and live SSE.

Default without tracing

Metadata plus byte counts, BLAKE3 hashes, and tool_summary_json. Body refs stay empty / *_body_omitted = true.

Reads

Query service /api/requests, /api/sessions/** → console.

---

3. Request data (when enable-trace is on)

Purpose: Full prompt/completion (and tool artifacts) for debugging — opt-in.

Enabled by any of

• Request header nexus-enable-tracing: 1
• API-key trace logging setting
• Project trace logging setting
• Organization trace logging setting

Path

1. Gateway encodes bodies as inline or chunked NATS payloads when trace_enabled; otherwise BodyEnvelope::Omitted (hash + size only).
2. Ingest packs bodies into org-isolated zstd frames in the bodies object store:
   • Object key pattern: <org_id>/packs/<ulid>.pack
   • Refs on the ClickHouse row: pack-v1:<bucket>/<key>#<offset>:<length>:zstd
3. ClickHouse requests stores request_body_ref / response_body_ref (plus hashes, sizes, and trace_enabled).

Reads

Query body proxy /api/body/by-request/:request_id range-reads one frame, verifies BLAKE3, and returns JSON. Clients do not receive raw object-store URLs.

---

4. Trace ingest (agentic / OTLP)

Purpose: Native OpenTelemetry / OpenInference agent traces — spans, events, and artifact refs.

Path

1. trace-ingest (POST /v1/traces, port 14458) authenticates with project-scoped trace-ingest keys (Postgres).
2. It normalizes payloads to TraceRecords and publishes to NATS nexus.traces.> (nexus_TRACES stream).
3. ingest writes ClickHouse tables trace_spans, trace_events, and trace_artifact_refs.
4. Large or sensitive payloads become artifact refs. Full bytes are stored only when capture policy allows (payload_state: summary_only | full | omitted | redacted). Otherwise only summary metadata is kept.

trace-ingest itself stores nothing — NATS → ingest → ClickHouse (+ object store for artifacts when full).

Not the same as

• Gateway POST /v1/traces/ingest (OpenAI Agents SDK shape): acknowledged, not persisted.
• Tier-2 operator OTLP → Collector → Tempo/Loki (dev LGTM): service health only; no tenant payloads.

Reads

Query /api/traces/**, /api/sessions/:id/traces, joined to request rows by request_id where present.

---

Durability contract

Analytics ingest and audit consumers share one rule: a JetStream message is acked only after its write is confirmed durable.

• Analytics ack follows body-pack upload (when applicable), ClickHouse insertion, and JetStream persistence of the related audit event.
• Audit ack follows the Postgres event insert.

On write failure the message stays unacked and JetStream redelivers. Poison messages that cannot be decoded, or that exhaust the redelivery budget, go to a dead-letter stream rather than being discarded.