Observability (Local Stack)

The Nexus dev compose stack includes a Grafana LGTM observability stack for inspecting, debugging, and validating all Nexus services during local development.

Component	Role	Image (default)
Prometheus	Scrapes /metrics from all Rust services + OTel Collector	prom/prometheus:v3.11.3
Tempo	Receives OTLP traces from the OTel Collector	grafana/tempo:2.10.3
Loki	Receives OTLP logs from the OTel Collector	grafana/loki:3.5.12
Grafana	Datasource UI + Explore / Drilldown apps	deploy/compose/grafana

License boundary

Grafana, Tempo, and Loki are licensed under AGPL-3.0; Prometheus is Apache-2.0. All four images are dev-only tooling and must never be included in release artifacts. See docs/legal/license-policy.md.

Quick start

just dev          # brings up the full stack including LGTM
just status       # verify core services and show LGTM readiness warnings

just status treats the LGTM stack as dev-only diagnostics. Grafana, Prometheus, Tempo, and Loki are still probed, but their readiness failures are warnings rather than core stack failures.

Access

Direct (localhost ports)

Service	URL
Gateway	http://127.0.0.1:14450
Ingest	http://127.0.0.1:14457
Query	http://127.0.0.1:14461
Console	http://127.0.0.1:14449
Grafana	http://127.0.0.1:14470
OTel Collector metrics	http://127.0.0.1:18888/metrics

Via Traefik (system-wide reverse proxy)

If you have a Docker Traefik instance running on an external traefik network, the compose stack attaches gateway, console, trace-ingest, audit, and Grafana to it with hostname-based routing:

Service	Default hostname / rule	Override env var
Gateway	nexus.127.0.0.1.nip.io	NEXUS_HOSTNAME
Console	console.nexus.127.0.0.1.nip.io	NEXUS_CONSOLE_HOSTNAME
Trace ingest	Host(NEXUS_HOSTNAME) && Path(/v1/traces) (priority over gateway)	NEXUS_HOSTNAME
Audit facade	Host(NEXUS_HOSTNAME) && PathPrefix(/api/audit) (priority over gateway)	NEXUS_HOSTNAME
Grafana	grafana.nexus.127.0.0.1.nip.io	NEXUS_GRAFANA_HOSTNAME

Note the audit read facade is exposed on the gateway hostname; it requires a bearer token with the audit:read scope.

Additional overrides:

Variable	Default	Purpose
NEXUS_TRAEFIK_NETWORK	traefik	External Docker network name
NEXUS_TRAEFIK_ENTRYPOINT	http	Traefik entrypoint (HTTP; no TLS)

The nip.io domain resolves any *.127.0.0.1.nip.io to 127.0.0.1, so no /etc/hosts editing is required.

Grafana datasources

All datasources are provisioned automatically on first start:

Name	Type	URL (internal)
Prometheus	prometheus	http://prometheus:9090
Tempo	tempo	http://tempo:3200
Loki	loki	http://loki:3100
ClickHouse	grafana-clickhouse-datasource	http://clickhouse:8123 (db: vortex)

Grafana starts with anonymous admin access enabled so dashboards are immediately editable in dev. No login required.

Readiness Semantics

Tempo and Loki expose strict /ready endpoints. Those endpoints can return non-200 while the containers are running, especially during startup, WAL replay, ring settling, or persistent-volume recovery. Use just logs tempo or just logs loki when you are debugging the observability stack itself. Only remove tempo-data or loki-data with just dev-clean when you intentionally want to discard local dev observability data.

Drilldown Apps

Grafana 13+ bundles the Logs Drilldown and Traces Drilldown apps (formerly plugins grafana-lokiexplore-app and grafana-exploretraces-app). Access them from the Grafana sidebar or directly:

App	URL path
Logs Drilldown	/a/grafana-lokiexplore-app/explore
Traces Drilldown	/a/grafana-exploretraces-app/explore

Loki is configured with pattern_ingester, discover_log_levels, volume_enabled, and allow_structured_metadata to support the full Drilldown feature set. Tempo is configured with the local-blocks metrics generator processor to support TraceQL metrics queries (rate(), quantile_over_time(), etc.).

Dashboards

Grafana is provisioned with Prometheus, Tempo, Loki, and ClickHouse datasources, plus six committed dashboards from deploy/compose/grafana/dashboards/ provisioned into a “Nexus” folder: gateway, ingest, ClickHouse, infra, OTLP pipeline, and service fleet. Use Explore, Logs Drilldown, and Traces Drilldown for ad-hoc queries; local dashboard edits persist in the grafana-data volume.

Telemetry data flow

Rust services ──/metrics──▶ Prometheus ──▶ Grafana
Rust services ──OTLP gRPC──▶ OTel Collector ──▶ Tempo (traces)
                                           ──▶ Loki  (logs)
                                           ──▶ debug (container logs)

• Prometheus scraping is the primary metrics path today. Every Rust service and the OTel Collector expose /metrics.
• OTLP traces and logs flow through the collector to Tempo and Loki. vortex-telemetry emits Tier 2 resource attributes, JSON logs to stderr, HTTP spans, gRPC interceptors, and the gateway/ingest request path spans.
• Service tracing events at info and above (OTLP_LOG_FILTER), and admitted by RUST_LOG, are exported as OTLP log records, so Loki holds each service’s warnings, errors, and lifecycle messages. Every service logs "vortex telemetry initialized" at info on startup, a stable Loki smoke signal while RUST_LOG admits info.
• Each HTTP request except /healthz, /readyz, and /metrics produces an http.request span and an http request completed log record labelled with the route template. The gateway also logs gateway request completed per model request with its request id, provider, model, status, latency, and tokens. Gateway request spans propagate through the NATS analytics event so ingest can continue the same trace. Field details are in ../security/telemetry-tiering.md.
• The Collector applies memory limits, a development resource override, forbidden-tenant-attribute filtering, header/attribute scrubbing, and 100% head sampling before export.
• The collector retains the debug exporter, so just logs otel-collector shows raw OTLP payloads for quick verification.

Validating a gateway request in observability

1. Send a request:

   curl http://127.0.0.1:14450/v1/chat/completions \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"model":"openai/gpt-5.4-mini","messages":[{"role":"user","content":"hi"}],"max_tokens":4}'

2. Open Grafana → Explore → Prometheus and query nexus_gateway_attempt_total. For a stack-level scrape check, query up or vortex_service_starts_total.
3. Open Grafana → Explore → Tempo and search recent traces. A completed request should include gateway.request, gateway.attempt, nats.publish, ingest.consume, and clickhouse.insert spans when the analytics pipeline is enabled.
4. In the Grafana Traces Drilldown app (or Explore → Tempo), search for recent traces. Use Logs Drilldown (or Explore → Loki) to browse service logs with automatic level detection.

Ingest durability and the request dead-letter queue

The ingest consumer acks a nexus_REQUESTS message only after its body pack, ClickHouse insert, and deterministic audit event are confirmed durable. On a failure the message is left unacked and JetStream redelivers it after the consumer’s ack_wait; re-insertion is idempotent because requests, request_cost_lines, and usage_monthly_rollup are ReplacingMergeTree deduplicated by their sorting key (which includes request_id, derived deterministically from the event), and the spend-cap/billing path reads usage_monthly_rollup FINAL. The per-minute dashboard rollup metrics_by_minute is a SummingMergeTree and can therefore double-count on a redelivery; treat it as best-effort, not as a billing source.

Metrics to watch on the ingest /metrics endpoint:

Metric	Meaning
nexus_ingest_insert_retries_total	Insert failures left unacked for redelivery (transient).
nexus_ingest_dlq_total{reason}	Messages dead-lettered instead of redelivered. reason="invalid_event" (undecodable envelope), reason="malformed_row" (ClickHouse rejected the isolated row as malformed), reason="process_error" (body-store/materialization failure exhausted the redelivery budget), reason="body_pack_exhausted" (the body-pack upload exhausted the budget), reason="insert_exhausted" (the ClickHouse insert exhausted the budget), reason="audit_emit" (the row was inserted durably but its audit event could not be published before the budget was exhausted).
nexus_ingest_clickhouse_errors_total{kind}	ClickHouse insert/stream errors by kind.
nexus_ingest_batch_insert_duration_seconds{table,result}	Batch flush latency, labelled by `result=success
nexus_ingest_body_packs_total	Successfully uploaded body packs.
nexus_ingest_body_pack_frames	Distribution of independently readable body frames per pack.
nexus_ingest_body_pack_bytes	Distribution of compressed bytes per pack.
nexus_ingest_clickhouse_durable_writes_total{table}	Line-table inserts whose rows read back present. table is request_usage_lines or request_cost_lines.
nexus_ingest_clickhouse_suppressed_writes_total{table}	Line-table inserts ClickHouse accepted while recognizing the block as a duplicate, leaving at least one request’s rows absent. Any nonzero value is lost cost evidence.
nexus_ingest_priced_requests_without_cost_lines_total	Priced requests charging a non-zero amount that carry no qualified cost line.
nexus_ingest_durability_read_failures_total	Durability read-backs that could not be completed, leaving the durability of an accepted line-table insert unknown. The insert outcome is unaffected.
nexus_ingest_durability_reads_skipped_total	Durability read-backs the flush sampler declined to run. Nonzero means sampling is active, so a zero nexus_ingest_clickhouse_suppressed_writes_total covers only the sampled flushes.

A sustained nonzero nexus_ingest_insert_retries_total with no DLQ growth means the analytics store is degraded but no data is being lost yet; DLQ growth means messages have outlasted the redelivery budget and need operator replay.

A reason="audit_emit" dead-letter is special: the analytics row is already committed durably, but its data.write.request_logged audit event is still pending. Replaying that payload is mandatory to restore the audit record; replay re-inserts the row idempotently and re-emits the audit event.

Durability read-back sampling

The durability read-back runs on the write path, once per line table per flush. NEXUS_INGEST_DURABILITY_SAMPLE_STRIDE verifies one flush in N and skips the rest. It defaults to 1, which verifies every flush; 0 is treated as 1, and an unparsable value falls back to the default. The decision is taken once per flush, so request_usage_lines and request_cost_lines written by the same flush are either both verified or both skipped.

Raising the stride trades detection latency for read load: a suppression is still detected, but only once a suppressed flush is sampled. Read nexus_ingest_durability_reads_skipped_total alongside nexus_ingest_clickhouse_durable_writes_total to know how much of the traffic the durability signal covers.

Dead-letter stream and replay

Undecodable envelopes and messages that exhaust their redelivery budget are published to the dead-letter stream nexus_REQUESTS_DLQ on subject nexus.dlq.requests (deliberately outside nexus.requests.> so replays are not re-captured by the main stream) and retained for 30 days.

To replay after resolving the outage, republish each dead-lettered payload back onto the request subject for its org and request id:

# Inspect the dead-letter stream
nats stream info nexus_REQUESTS_DLQ

# Replay one payload (the envelope JSON carries org_id/request_id)
nats stream view nexus_REQUESTS_DLQ
nats pub "nexus.requests.<org_id>.<request_id>" "<envelope-json>"

Replays are idempotent against already-materialized rows thanks to the ReplacingMergeTree dedup described above.

Image overrides

Grafana is built locally from deploy/compose/grafana/Dockerfile (grafana/grafana:13.1.0 with the ClickHouse data source installed at build time, so it loads without reaching grafana.com). The Drilldown apps Grafana installs at startup live in the container, so they download again when it is recreated. The other observability images can be pinned or mirrored via .env:

NEXUS_IMAGE_PROMETHEUS=prom/prometheus:v3.11.3
NEXUS_IMAGE_TEMPO=grafana/tempo:2.10.3
NEXUS_IMAGE_LOKI=grafana/loki:3.5.12

Volumes

Persistent named volumes keep data across just down / just dev cycles:

• prometheus-data — TSDB storage (7 d retention)
• tempo-data — trace storage
• loki-data — log storage
• grafana-data — Grafana state (user prefs, annotations)

Wipe everything with just dev-clean (removes all compose volumes).