The Nexus dev compose stack includes a Grafana LGTM observability stack for inspecting, debugging, and validating all Nexus services during local development.

Grafana, Tempo, and Loki are licensed under AGPL-3.0; Prometheus is Apache-2.0. All four images are dev-only tooling and must never be included in release artifacts. See License Policy.

just dev          # brings up the full stack including LGTM
just status       # verify core services and show LGTM readiness warnings

just status treats the LGTM stack as dev-only diagnostics. Grafana, Prometheus, Tempo, and Loki are still probed, but their readiness failures are warnings rather than core stack failures.

If you have a Docker Traefik instance running on an external traefik network, the compose stack attaches gateway, console, trace-ingest, audit, and Grafana to it with hostname-based routing:

Note the audit read facade is exposed on the gateway hostname; it requires a bearer token with the audit:read scope.

Additional overrides:

The nip.io domain resolves any *.127.0.0.1.nip.io to 127.0.0.1, so no /etc/hosts editing is required.

All datasources are provisioned automatically on first start:

Grafana starts with anonymous admin access enabled so dashboards are immediately editable in dev. No login required.

Tempo and Loki expose strict /ready endpoints. Those endpoints can return non-200 while the containers are running, especially during startup, WAL replay, ring settling, or persistent-volume recovery. Use just logs tempo or just logs loki when you are debugging the observability stack itself. Only remove tempo-data or loki-data with just dev-clean when you intentionally want to discard local dev observability data.

Grafana 13+ bundles the Logs Drilldown and Traces Drilldown apps (formerly plugins grafana-lokiexplore-app and grafana-exploretraces-app). Access them from the Grafana sidebar or directly:

Loki is configured with pattern_ingester, discover_log_levels, volume_enabled, and allow_structured_metadata to support the full Drilldown feature set. Tempo is configured with the local-blocks metrics generator processor to support TraceQL metrics queries (rate(), quantile_over_time(), etc.).

Grafana is provisioned with Prometheus, Tempo, Loki, and ClickHouse datasources, plus five committed dashboards from deploy/compose/grafana/dashboards/ provisioned into a “Nexus” folder: gateway, ClickHouse, infra, OTLP pipeline, and service fleet. Use Explore, Logs Drilldown, and Traces Drilldown for ad-hoc queries; local dashboard edits persist in the grafana-data volume.

Rust services ──/metrics──▶ Prometheus ──▶ Grafana
Rust services ──OTLP gRPC──▶ OTel Collector ──▶ Tempo (traces)
                                           ──▶ Loki  (logs)
                                           ──▶ debug (container logs)

• Prometheus scraping is the primary metrics path today. Every Rust service and the OTel Collector expose /metrics.
• OTLP traces and logs flow through the collector to Tempo and Loki. vortex-telemetry emits Tier 2 resource attributes, JSON logs, HTTP spans, gRPC interceptors, and the gateway/ingest request path spans.
• vortex-telemetry emits one startup OTLP log record per service ("vortex telemetry initialized"), which provides a stable Loki smoke signal.
• Basic HTTP request spans are emitted from the shared service routers via TraceLayer, and gateway request spans propagate through the NATS analytics event so ingest can continue the same trace.
• The Collector applies memory limits, a development resource override, forbidden-tenant-attribute filtering, header/attribute scrubbing, and 100% head sampling before export.
• The collector retains the debug exporter, so just logs otel-collector shows raw OTLP payloads for quick verification.

1. Send a request:

   curl http://127.0.0.1:14450/v1/chat/completions \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"model":"openai/gpt-5.4-mini","messages":[{"role":"user","content":"hi"}],"max_tokens":4}'

2. Open Grafana → Explore → Prometheus and query nexus_gateway_attempt_total. For a stack-level scrape check, query up or vortex_service_starts_total.
3. Open Grafana → Explore → Tempo and search recent traces. A completed request should include gateway.request, gateway.attempt, nats.publish, ingest.consume, and clickhouse.insert spans when the analytics pipeline is enabled.
4. In the Grafana Traces Drilldown app (or Explore → Tempo), search for recent traces. Use Logs Drilldown (or Explore → Loki) to browse service logs with automatic level detection.

The ingest consumer acks a nexus_REQUESTS message only after its ClickHouse insert is confirmed durable (ack-after-durable-insert). On an insert failure the message is left unacked and JetStream redelivers it after the consumer’s ack_wait; re-insertion is idempotent because requests, request_cost_lines, and usage_monthly_rollup are ReplacingMergeTree deduplicated by their sorting key (which includes request_id, derived deterministically from the event), and the spend-cap/billing path reads usage_monthly_rollup FINAL. The per-minute dashboard rollup metrics_by_minute is a SummingMergeTree and can therefore double-count on a redelivery; treat it as best-effort, not as a billing source.

Metrics to watch on the ingest /metrics endpoint:

A sustained nonzero nexus_ingest_insert_retries_total with no DLQ growth means the analytics store is degraded but no data is being lost yet; DLQ growth means messages have outlasted the redelivery budget and need operator replay.

A reason="audit_emit" dead-letter is special: the analytics row is already committed durably, but its data.write.request_logged audit event is still pending. Replaying that payload is mandatory to restore the audit record; replay re-inserts the row idempotently and re-emits the audit event.

Undecodable envelopes and messages that exhaust their redelivery budget are published to the dead-letter stream nexus_REQUESTS_DLQ on subject nexus.dlq.requests (deliberately outside nexus.requests.> so replays are not re-captured by the main stream) and retained for 30 days.

To replay after resolving the outage, republish each dead-lettered payload back onto the request subject for its org and request id:

# Inspect the dead-letter stream
nats stream info nexus_REQUESTS_DLQ

# Replay one payload (the envelope JSON carries org_id/request_id)
nats stream view nexus_REQUESTS_DLQ
nats pub "nexus.requests.<org_id>.<request_id>" "<envelope-json>"

Replays are idempotent against already-materialized rows thanks to the ReplacingMergeTree dedup described above.

All observability images can be pinned or mirrored via .env:

NEXUS_IMAGE_GRAFANA=grafana/grafana:13.0.1
NEXUS_IMAGE_PROMETHEUS=prom/prometheus:v3.11.3
NEXUS_IMAGE_TEMPO=grafana/tempo:2.10.3
NEXUS_IMAGE_LOKI=grafana/loki:3.5.12

Persistent named volumes keep data across just down / just dev cycles:

• prometheus-data — TSDB storage (7 d retention)
• tempo-data — trace storage
• loki-data — log storage
• grafana-data — Grafana state (user prefs, annotations)

Wipe everything with just dev-clean (removes all compose volumes).