Observability (Local Stack)

The Nexus dev compose stack includes a Grafana LGTM observability stack for inspecting, debugging, and validating all Nexus services during local development.

Component	Role	Image (default)
Prometheus	Scrapes /metrics from all Rust services + OTel Collector	prom/prometheus:v3.11.3
Tempo	Receives OTLP traces from the OTel Collector	grafana/tempo:2.10.3
Loki	Receives OTLP logs from the OTel Collector	grafana/loki:3.5.12
Grafana	Datasource UI + Explore / Drilldown apps	grafana/grafana:13.0.1

License boundary

Grafana, Tempo, and Loki are licensed under AGPL-3.0; Prometheus is Apache-2.0. All four images are dev-only tooling and must never be included in release artifacts. See docs/legal/license-policy.md.

Quick start

just dev          # brings up the full stack including LGTM
just status       # verify core services and show LGTM readiness warnings

To run without the collector, set OTEL_EXPORTER_OTLP_ENDPOINT to an empty value (OTEL_EXPORTER_OTLP_ENDPOINT= just dev, or edit it in .env). The services install no exporter and log JSON to stdout instead; Prometheus keeps scraping /metrics, so the dashboards below still work, while Tempo and Loki stay empty.

just status treats the LGTM stack as dev-only diagnostics. Grafana, Prometheus, Tempo, and Loki are still probed, but their readiness failures are warnings rather than core stack failures.

Access

Direct (localhost ports)

Service	URL
Gateway	http://127.0.0.1:14450
Ingest	http://127.0.0.1:14457
Query	http://127.0.0.1:14461
Console	http://127.0.0.1:14449
Grafana	http://127.0.0.1:14470
OTel Collector metrics	http://127.0.0.1:18888/metrics

Via Traefik (system-wide reverse proxy)

Traefik routing is opt-in. The base compose file does not reference the external traefik network at all, so the stack starts on a host with no Traefik; add deploy/compose/docker-compose.traefik.yml when you want it.

The network must already exist — compose will not create an external network — and its name must match NEXUS_TRAEFIK_NETWORK if you have overridden it:

docker network create "${NEXUS_TRAEFIK_NETWORK:-traefik}"

docker compose --env-file .env \
               -f deploy/compose/docker-compose.yml \
               -f deploy/compose/docker-compose.dev.yml \
               -f deploy/compose/docker-compose.traefik.yml up -d

Setting NEXUS_COMPOSE_FILES to that same list makes the recipes that honour it use the overlay — just dev, up, down, logs, status, cli. It has no effect on just dev-release, dev-release-down or redis-it, which use COMPOSE_BASE and always run the base file alone.

deploy/scripts/reset-data.sh and deploy/scripts/bootstrap-registry.sh also restart the stack without Traefik. Both select a single overlay through NEXUS_COMPOSE_OVERLAY, which holds one file path rather than a -f list, so they cannot layer this overlay on top of the dev one. Re-add it by hand after running either script if you need the routes back.

With the overlay added, gateway, console, admin-console, trace-ingest, audit and Grafana attach to the network with hostname-based routing:

Service	Default hostname / rule	Override env var
Gateway	nexus.127.0.0.1.nip.io	NEXUS_HOSTNAME
Console	console.nexus.127.0.0.1.nip.io	NEXUS_CONSOLE_HOSTNAME
Admin console	admin.nexus.127.0.0.1.nip.io	NEXUS_ADMIN_CONSOLE_HOSTNAME
Trace ingest	Host(NEXUS_HOSTNAME) && Path(/v1/traces) (priority over gateway)	NEXUS_HOSTNAME
Audit facade	Host(NEXUS_HOSTNAME) && PathPrefix(/api/audit) (priority over gateway)	NEXUS_HOSTNAME
Grafana	grafana.nexus.127.0.0.1.nip.io	NEXUS_GRAFANA_HOSTNAME

The direct localhost ports above keep working with or without the overlay.

Note the audit read facade is exposed on the gateway hostname; it requires a bearer token with the audit:read scope.

Additional overrides:

Variable	Default	Purpose
NEXUS_TRAEFIK_NETWORK	traefik	External Docker network name
NEXUS_TRAEFIK_ENTRYPOINT	http	Traefik entrypoint (HTTP; no TLS)

The nip.io domain resolves any *.127.0.0.1.nip.io to 127.0.0.1, so no /etc/hosts editing is required.

Grafana datasources

All datasources are provisioned automatically on first start:

Name	Type	URL (internal)
Prometheus	prometheus	http://prometheus:9090
Tempo	tempo	http://tempo:3200
Loki	loki	http://loki:3100
ClickHouse	grafana-clickhouse-datasource	http://clickhouse:8123 (db: vortex)

Grafana starts with anonymous admin access enabled so dashboards are immediately editable in dev. No login required.

Readiness Semantics

Tempo and Loki expose strict /ready endpoints. Those endpoints can return non-200 while the containers are running, especially during startup, WAL replay, ring settling, or persistent-volume recovery. Use just logs tempo or just logs loki when you are debugging the observability stack itself. Only remove tempo-data or loki-data with just dev-clean when you intentionally want to discard local dev observability data.

Drilldown Apps

Grafana 13+ bundles the Logs Drilldown and Traces Drilldown apps (formerly plugins grafana-lokiexplore-app and grafana-exploretraces-app). Access them from the Grafana sidebar or directly:

App	URL path
Logs Drilldown	/a/grafana-lokiexplore-app/explore
Traces Drilldown	/a/grafana-exploretraces-app/explore

Loki is configured with pattern_ingester, discover_log_levels, volume_enabled, and allow_structured_metadata to support the full Drilldown feature set. Tempo is configured with the local-blocks metrics generator processor to support TraceQL metrics queries (rate(), quantile_over_time(), etc.).

Dashboards

Grafana is provisioned with Prometheus, Tempo, Loki, and ClickHouse datasources, plus six committed dashboards from deploy/compose/grafana/dashboards/ provisioned into a “Nexus” folder: gateway, ingest, ClickHouse, infra, OTLP pipeline, and service fleet. Use Explore, Logs Drilldown, and Traces Drilldown for ad-hoc queries; local dashboard edits persist in the grafana-data volume.

Telemetry data flow

Rust services ──/metrics──▶ Prometheus ──▶ Grafana
Rust services ──OTLP gRPC──▶ OTel Collector ──▶ Tempo (traces)
                                           ──▶ Loki  (logs)
                                           ──▶ debug (container logs)

• Prometheus scraping is the primary metrics path today. Every Rust service and the OTel Collector expose /metrics.
• OTLP traces and logs flow through the collector to Tempo and Loki. vortex-telemetry emits Tier 2 resource attributes, JSON logs, HTTP spans, gRPC interceptors, and the gateway/ingest request path spans.
• vortex-telemetry emits one startup OTLP log record per service ("vortex telemetry initialized"), which provides a stable Loki smoke signal.
• Basic HTTP request spans are emitted from the shared service routers via TraceLayer, and gateway request spans propagate through the NATS analytics event so ingest can continue the same trace.
• The Collector applies memory limits, a development resource override, forbidden-tenant-attribute filtering, header/attribute scrubbing, and 100% head sampling before export.
• The collector retains the debug exporter, so just logs otel-collector shows raw OTLP payloads for quick verification.

Validating a gateway request in observability

1. Send a request:

   curl http://127.0.0.1:14450/v1/chat/completions \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"model":"openai/gpt-5.4-mini","messages":[{"role":"user","content":"hi"}],"max_tokens":4}'

2. Open Grafana → Explore → Prometheus and query nexus_gateway_attempt_total. For a stack-level scrape check, query up or vortex_service_starts_total.
3. Open Grafana → Explore → Tempo and search recent traces. A completed request should include gateway.request, gateway.attempt, nats.publish, ingest.consume, and clickhouse.insert spans when the analytics pipeline is enabled.
4. In the Grafana Traces Drilldown app (or Explore → Tempo), search for recent traces. Use Logs Drilldown (or Explore → Loki) to browse service logs with automatic level detection.

Ingest durability and the request dead-letter queue

The ingest consumer acks a nexus_REQUESTS message only after its body pack, ClickHouse insert, and deterministic audit event are confirmed durable. On a failure the message is left unacked and JetStream redelivers it after the consumer’s ack_wait; re-insertion is idempotent because requests, request_cost_lines, and usage_monthly_rollup are ReplacingMergeTree deduplicated by their sorting key (which includes request_id, derived deterministically from the event), and the spend-cap/billing path reads usage_monthly_rollup FINAL. The per-minute dashboard rollup metrics_by_minute is a SummingMergeTree and can therefore double-count on a redelivery; treat it as best-effort, not as a billing source.

Metrics to watch on the ingest /metrics endpoint:

Metric	Meaning
nexus_ingest_insert_retries_total	Insert failures left unacked for redelivery (transient).
nexus_ingest_dlq_total{reason}	Messages dead-lettered instead of redelivered. reason="invalid_event" (undecodable envelope), reason="malformed_row" (ClickHouse rejected the isolated row as malformed), reason="process_error" (body-store/materialization failure exhausted the redelivery budget), reason="body_pack_exhausted" (the body-pack upload exhausted the budget), reason="insert_exhausted" (the ClickHouse insert exhausted the budget), reason="audit_emit" (the row was inserted durably but its audit event could not be published before the budget was exhausted).
nexus_ingest_clickhouse_errors_total{kind}	ClickHouse insert/stream errors by kind.
nexus_ingest_batch_insert_duration_seconds{table,result}	Batch flush latency, labelled by `result=success
nexus_ingest_body_packs_total	Successfully uploaded body packs.
nexus_ingest_body_pack_frames	Distribution of independently readable body frames per pack.
nexus_ingest_body_pack_bytes	Distribution of compressed bytes per pack.
nexus_ingest_clickhouse_durable_writes_total{table}	Line-table inserts whose rows read back present. table is request_usage_lines or request_cost_lines.
nexus_ingest_clickhouse_suppressed_writes_total{table}	Line-table inserts ClickHouse accepted while recognizing the block as a duplicate, leaving at least one request’s rows absent. Any nonzero value is lost cost evidence.
nexus_ingest_priced_requests_without_cost_lines_total	Priced requests charging a non-zero amount that carry no qualified cost line.
nexus_ingest_durability_read_failures_total	Durability read-backs that could not be completed, leaving the durability of an accepted line-table insert unknown. The insert outcome is unaffected.
nexus_ingest_durability_reads_skipped_total	Durability read-backs the flush sampler declined to run. Nonzero means sampling is active, so a zero nexus_ingest_clickhouse_suppressed_writes_total covers only the sampled flushes.

A sustained nonzero nexus_ingest_insert_retries_total with no DLQ growth means the analytics store is degraded but no data is being lost yet; DLQ growth means messages have outlasted the redelivery budget and need operator replay.

A reason="audit_emit" dead-letter is special: the analytics row is already committed durably, but its data.write.request_logged audit event is still pending. Replaying that payload is mandatory to restore the audit record; replay re-inserts the row idempotently and re-emits the audit event.

Durability read-back sampling

The durability read-back runs on the write path, once per line table per flush. NEXUS_INGEST_DURABILITY_SAMPLE_STRIDE verifies one flush in N and skips the rest. It defaults to 1, which verifies every flush; 0 is treated as 1, and an unparsable value falls back to the default. The decision is taken once per flush, so request_usage_lines and request_cost_lines written by the same flush are either both verified or both skipped.

Raising the stride trades detection latency for read load: a suppression is still detected, but only once a suppressed flush is sampled. Read nexus_ingest_durability_reads_skipped_total alongside nexus_ingest_clickhouse_durable_writes_total to know how much of the traffic the durability signal covers.

Dead-letter stream and replay

Undecodable envelopes and messages that exhaust their redelivery budget are published to the dead-letter stream nexus_REQUESTS_DLQ on subject nexus.dlq.requests (deliberately outside nexus.requests.> so replays are not re-captured by the main stream) and retained for 30 days.

To replay after resolving the outage, republish each dead-lettered payload back onto the request subject for its org and request id:

# Inspect the dead-letter stream
nats stream info nexus_REQUESTS_DLQ

# Replay one payload (the envelope JSON carries org_id/request_id)
nats stream view nexus_REQUESTS_DLQ
nats pub "nexus.requests.<org_id>.<request_id>" "<envelope-json>"

Replays are idempotent against already-materialized rows thanks to the ReplacingMergeTree dedup described above.

Image overrides

All observability images can be pinned or mirrored via .env:

NEXUS_IMAGE_GRAFANA=grafana/grafana:13.0.1
NEXUS_IMAGE_PROMETHEUS=prom/prometheus:v3.11.3
NEXUS_IMAGE_TEMPO=grafana/tempo:2.10.3
NEXUS_IMAGE_LOKI=grafana/loki:3.5.12

Volumes

Persistent named volumes keep data across just down / just dev cycles:

• prometheus-data — TSDB storage (7 d retention)
• tempo-data — trace storage
• loki-data — log storage
• grafana-data — Grafana state (user prefs, annotations)

Wipe everything with just dev-clean (removes all compose volumes).