Telemetry Tiering

Nexus separates tenant analytics from operator telemetry so operational visibility does not expose tenant data. The rules in this document describe the behavior implemented by the current vortex-telemetry crate, service instrumentation, and local OpenTelemetry Collector configuration.

Channels

Channel	Storage	Audience	Data Boundary
Tier 1 tenant analytics	ClickHouse and MinIO	Tenant users through authorized APIs and console pages	Request metadata, bodies, usage, cost, and tenant-scoped identifiers are allowed.
Tier 2 operator telemetry	OpenTelemetry Collector and operator APM	Operators and SREs	Tenant identifiers, payloads, API keys, classification labels, prompts, and completions are forbidden.

Tier 2 exists for service health, dependency timing, request rates, error rates, and trace correlation. It is not a tenant-debugging data store. Use a request ID to correlate operator traces with tenant-visible request details through the authorized query and console paths.

Tier 2 Resource Attributes

Every Nexus Rust service initialized through vortex-telemetry emits these resource attributes on OTLP traces and logs:

• service.name
• service.version
• service.instance.id
• deployment.environment
• nexus.tier = "2"

Prometheus scrape metrics remain available through each service’s /metrics endpoint. The OpenTelemetry Collector also exposes its own metrics for collector health.

Forbidden Tier 2 Data

Tier 2 telemetry must not contain:

• Tenant or subject identifiers: tenant_id, org_id, project_id, user_id, api_key_id, session_id.
• Nexus-prefixed tenant attributes: nexus.org_id, nexus.project_id, nexus.user_id, nexus.api_key_id, nexus.classification.
• Request or response bodies, including prompt text, completion text, message arrays, tool arguments, and tool results.
• Provider credentials, Nexus API keys, bearer tokens, cookies, or upstream object keys.
• Raw SQL parameters or object-store paths that identify tenant records.
• Classification labels.

vortex-common-crates/vortex-telemetry/src/cardinality.rs enforces this boundary in debug assertions and metric-label helpers. integration-tests/tests/telemetry_tier2.rs covers forbidden-key detection, allowed gateway metric labels, baggage scrubbing, and Collector privacy processors. The tier-separation-test CI workflow runs the Tier 2 payload and cardinality tests.

Allowed Tier 2 Data

Tier 2 spans and metrics may include low-cardinality operational attributes:

• HTTP method, route template, and status class.
• gRPC service, method, and status code.
• Provider name.
• Result class or error kind enum.
• Static table or bucket class labels such as requests, tenant, or system.
• Non-sensitive stream or consumer names.

Do not emit raw request paths when the path can contain IDs. Use route templates such as /v1/chat/completions instead.

Metric Label Allowlist

The vortex-telemetry cardinality guard validates labels for known metric families. Examples of allowed labels include:

• nexus_gateway_request_duration_seconds: provider, route, status_class, result.
• nexus_gateway_attempt_total: provider, result, virtual_key_id, routing_rule_id, fallback_index.
• nexus_ingest_batch_insert_duration_seconds: table, result.
• nexus_ingest_clickhouse_durable_writes_total: table.
• nexus_ingest_clickhouse_suppressed_writes_total: table.
• nexus_object_store_operation_duration_seconds: operation, result, error_class.

Labels outside a metric’s allowlist are rejected by debug assertions and test helpers.

Monitoring Metric Contracts

Object-store metrics are nexus_object_store_operations_total, nexus_object_store_operation_duration_seconds, and nexus_object_store_bytes_total. Their bounded labels are operation, result, and error_class where applicable.

Ingest JetStream gauges are nexus_ingest_jetstream_pending_messages, nexus_ingest_jetstream_ack_pending_messages, nexus_ingest_jetstream_outstanding_messages, nexus_ingest_jetstream_redelivered_messages, nexus_ingest_jetstream_collection_success, and nexus_ingest_jetstream_last_collection_unixtime_seconds, each labeled only by consumer. The ingest dead-letter gauges are nexus_ingest_dlq_depth, nexus_ingest_dlq_collection_success, and nexus_ingest_dlq_last_collection_unixtime_seconds. outstanding is the sum of pending and ack-pending; redelivered remains an instantaneous gauge. nexus_ingest_requests_persisted_total has no labels, initializes to zero, and increments only after ClickHouse persistence, a server-confirmed JetStream PubAck for the request audit event, and a server-confirmed source-consumer acknowledgement all succeed. Request audit retries derive both the audit event id and Nats-Msg-Id from tenant, event kind, and request identity, while retaining the request occurrence time, so source redelivery is idempotent. nexus_ingest_clickhouse_persistence_failures_total has no labels and increments once for each ClickHouse JSONEachRow persistence operation that fails in transport or returns a non-success response, including malformed-row rejections. It does not count serialization, consumer-handler, object-store, audit-publication, or JetStream failures; batch bisection attempts count separately because each is a distinct ClickHouse operation. nexus_ingest_clickhouse_durable_writes_total and nexus_ingest_clickhouse_suppressed_writes_total are labeled only by table, whose domain is closed to the two line tables request_usage_lines and request_cost_lines. Both series initialize to zero for both label values, so a zero reading is distinguishable from an absent series. The durable counter increments when the rows of an accepted line-table insert read back present; the suppressed counter increments when at least one request’s rows are absent, which is lost cost evidence. nexus_ingest_priced_requests_without_cost_lines_total has no labels, initializes to zero, and counts priced requests charging a non-zero amount that carry no qualified cost line. nexus_ingest_durability_read_failures_total has no labels, initializes to zero, and counts durability read-backs that could not be completed. The read never fails the insert it verifies, so this counter is separate from nexus_ingest_clickhouse_errors_total. nexus_ingest_durability_reads_skipped_total has no labels, initializes to zero, and counts durability read-backs the flush sampler declined to run, so an operator can tell a table that read back complete from one that was not read at all. The sample decision is taken once per flush, so the line tables written by one flush are either both verified or both skipped. nexus_ingest_object_store_probe_failures_total has no labels, initializes to zero, and counts failed, timed-out, or cancelled ingest object-store readiness probes. Each probe uses a unique key; independently owned bounded cleanup tasks retain timed-out PUT and DELETE work long enough to finish or retry cleanup without racing a later probe.

Audit JetStream uses the corresponding nexus_audit_jetstream_* gauge names without labels. Audit operation failures use nexus_audit_operation_failures_total{operation}. Audit freshness uses nexus_audit_last_success_unixtime_seconds{operation} with the bounded operations seal, archive, and anchor; each series initializes to zero at process startup. nexus_audit_process_start_unixtime_seconds records the audit process start time. nexus_audit_archive_pending is 1 when unanchored batches exist and 0 when none exist; nexus_audit_archive_pending_known is 1 after a successful pending-state query and 0 when that state is not yet known or its query failed. Audit archive readiness has a startup grace of the greater of two anchor intervals or 60 seconds. It remains healthy without aging while no unanchored batches exist. While work is pending, readiness requires either that bounded grace or a completed real archive publication no older than the greater of three anchor intervals or 120 seconds.

Trace Propagation

Gateway spans propagate W3C Trace Context through the request analytics envelope. Ingest extracts that context so a single request can be traced across gateway work, provider attempts, NATS publish, ingest consume, body materialization, and ClickHouse insert.

Baggage is scrubbed before propagation. Tenant-bearing baggage values are not forwarded into Tier 2 spans or logs.

Collector Defense In Depth

The local Collector configuration applies:

• memory_limiter for process safety.
• filter/tenant_data to drop forbidden tenant attributes.
• attributes/scrub to redact sensitive headers and values.
• probabilistic_sampler for head sampling.
• batch before export.

These processors are defense in depth. Services must still avoid emitting forbidden Tier 2 attributes in the first place.

Contributor Guidance

When adding telemetry:

1. Prefer route templates, status classes, provider names, and result enums.
2. Do not add IDs, request bodies, headers, prompts, completions, classification labels, object keys, or raw error messages that can contain tenant content.
3. Add new metric labels to the vortex-telemetry allowlist only when the value has bounded cardinality and no tenant-specific meaning.
4. Add or update telemetry_tier2 coverage for new metric families, propagation behavior, or Collector privacy processors.
5. Use request IDs for correlation; inspect tenant details through the query service and console authorization path.
6. Check Telemetry Conventions before adding a new span, metric, or log field.