0004 - OTLP for system telemetry (Tier 2)

• Status: Accepted
• Date: 2026-05-13
• Deciders: nexus core team
• Tags: observability, telemetry, operations, privacy
• Consulted: Aegis core team (shared vortex-telemetry consumer)

Context and Problem Statement

nexus has two observability audiences with different data boundaries. Tenants need request analytics, bodies, model usage, cost, and request detail; that data lives in the Tier 1 ClickHouse and object-store path. Operators need service health, latency, error rates, and dependency visibility; that data must not contain tenant identifiers, request bodies, prompts, completions, API keys, or classification labels.

The system needs a standard operator telemetry channel that works in local development and can be pointed at an operator-selected backend in production. The shared vortex-telemetry crate is the right boundary because every Rust service already initializes it and Aegis consumes the same conventions through the shared-crate contract.

Decision Drivers

• Preserve a hard privacy boundary between Tier 1 tenant analytics and Tier 2 operator telemetry.
• Use vendor-neutral OpenTelemetry Protocol so operators can choose Grafana LGTM, SigNoz, Datadog, Splunk, or another OTLP-compatible backend.
• Keep local development observable with the compose LGTM stack.
• Enforce low-cardinality labels and forbidden-key checks in code and tests.
• Reuse tracing, metrics, axum, and tonic rather than adding a second instrumentation stack.

Considered Options

• OTLP through an OpenTelemetry Collector — chosen.
• Prometheus-only metrics — useful for scrape compatibility but insufficient for distributed traces and logs.
• Direct vendor SDKs — rejected because they lock deployments to one backend and make air-gap packaging harder.
• Tenant ClickHouse as the operator telemetry store — rejected because it mixes audiences and increases the risk that tenant payloads leak into operator views.
• Structured logs only — insufficient for span correlation and dependency timing.

Decision Outcome

Chosen option: all Nexus Rust services emit Tier 2 operator telemetry through OTLP to an OpenTelemetry Collector.

Concrete constraints:

• vortex-telemetry owns initialization of tracing, Prometheus scrape support, OTLP trace/log export, resource attributes, cardinality helpers, and propagation helpers.
• Every Nexus service emits service.name, service.version, service.instance.id, deployment.environment, and nexus.tier = "2" resource attributes.
• Tier 2 attributes follow the telemetry contract: no tenant_id, org_id, project_id, user_id, api_key_id, session_id, request/response bodies, prompt/completion text, provider secrets, raw object keys, raw SQL parameters, or classification labels.
• The gateway and ingest pipeline propagate W3C Trace Context across the NATS request-analytics envelope so one chat completion can be traced through gateway work, provider call, NATS publish, ingest consume, body materialization, and ClickHouse insert.
• The compose stack ships a reference Collector plus Grafana LGTM for development. Production operators may replace the Collector exporters without service changes.
• Prometheus /metrics remains available for scrape-based environments; OTLP is the canonical trace/log path.

Consequences

• Work that touches vortex-telemetry must follow the shared-crate review rules in the shared-crate contract.
• Feature teams must treat Tier 2 as an operator channel, not a debugging shortcut for tenant-specific data. Use request IDs to correlate to Tier 1 data through authorized tenant-facing APIs.
• The Collector configuration becomes part of the privacy boundary. It scrubs forbidden attributes as defense in depth, but services are still responsible for not emitting them.
• Cardinality violations are build failures, not dashboard tuning tasks.
• Local development has a heavier compose stack, but the stack is dev-only tooling and is not included in release artifacts.

More Information

Telemetry Conventions

The telemetry contract: resource attributes, naming rules, and forbidden Tier 2 fields.

Read the contract →

Telemetry Tiering

Contributor rules for keeping Tier 1 tenant data and Tier 2 operator data separated.

Learn the tiers →

Observability (Local Stack)

The dev observability guide: reference Collector plus Grafana LGTM in the compose stack.

Set up observability →

The shared-crate contract lives at vortex-common-crates/contracts/shared-crates.md.