0010 - NATS JetStream as the internal async event bus

• Status: Accepted
• Date: 2026-07-03
• Deciders: nexus core team
• Tags: messaging, nats, jetstream, ingest, audit, gateway

The gateway hot path must not block on analytics or audit persistence: adding a synchronous database write to every proxied request would put ClickHouse and Postgres availability in the request latency budget. The gateway therefore publishes events and returns; downstream services persist them with at-least-once delivery, acking only after their write is durable. Publish latency must stay in the single-digit-millisecond range, subjects must carry tenant scoping so consumers can filter per tenant, and the whole bus must be operable by self-hosters as a single lightweight process.

• Decouple request completion from analytics/audit persistence with a fast, fire-and-forget publish.
• At-least-once delivery with durable consumers and explicit acks, paired with the ack-after-durable-write contract.
• Subject hierarchies that encode tenant identity for scoped filtering and fan-out.
• Operational weight matters: nexus ships to self-hosters and air-gapped environments.

• NATS JetStream — chosen.
• Kafka — rejected: durable and battle-tested, but its operational weight (brokers, controller quorum, JVM tuning, partition management) is disproportionate for self-hosters running the whole stack on a small footprint.
• Redis streams — lighter, but weaker durability guarantees, no subject-hierarchy filtering, and consumer-group semantics that fit this workload worse than JetStream pull consumers.
• Direct DB writes on the hot path — rejected outright: couples gateway latency and availability to ClickHouse/Postgres and loses the replay buffer that lets consumers restart without data loss.

Chosen option: NATS JetStream with tenant-scoped subject hierarchies and durable pull consumers.

• The gateway publishes request.completed analytics envelopes on nexus.requests.<org_id>.<request_id> (services/gateway/src/events.rs), inline or chunked for large bodies.
• Audit events flow on nexus.audit.events.<tenant_id>; the subject root is defined once in crates/nexus-runtime (AUDIT_SUBJECT_ROOT) and reused by publishers.
• Trace ingestion publishes on nexus.traces.* subjects (services/trace-ingest/src/events.rs).
• Registry catalog-sync notifications are pushed on nexus.registry.sync so gateway registry snapshots refresh without polling delay.
• The ingest service bootstraps its streams and durable pull consumers in services/ingest/src/bootstrap.rs, including the dead-letter stream nexus_REQUESTS_DLQ (subject nexus.dlq.requests) with 30-day retention for envelopes that cannot be decoded.
• Consumers ack only after their durable write succeeds; unacked messages are redelivered per the contract in the architecture data model documentation.

• Stream and consumer bootstrap is owned by the consuming service, not by a central provisioning step; each consumer creates or opens what it needs at startup.
• At-least-once delivery means every consumer must be idempotent under redelivery — the ReplacingMergeTree keying in ClickHouse (ADR-0003) and idempotent audit appends exist because of this.
• NATS is a hard runtime dependency: the dev stack and deploy manifests carry it, and publish failures surface as sink errors rather than being silently absorbed.
• Message size limits force a chunking protocol for large request/response bodies rather than unbounded single messages.