Telemetry Conventions

This document is the source of truth for telemetry naming, resource attributes, span structure, metric types, and cardinality budgets across Aegis and nexus. It is consumed by the vortex-telemetry crate.

nexus operates three distinct telemetry channels with different privacy boundaries. Aegis operates only the first two (it has no tenant-export channel).

The Three Channels

Channel	Where stored	Privacy tier	Who sees it	Owner
Tier 1 tenant analytics	nexus ClickHouse	Tenant payloads + PII allowed, per-tenant isolation	Tenant	nexus analytics
Tier 2 operator OTLP	Operator’s OTel Collector / APM	No tenant identifiers, no payloads, no PII	Operator only	Shared telemetry
Tier 1 tenant export	Tenant’s OTLP/Prometheus (nexus only)	CH aggregates with attribute allowlist	Tenant	nexus tenant export

The initial telemetry skeleton in both products initializes the Tier 2 channel only. Tier 1 and Tier 1 export are built by product-specific telemetry features.

flowchart LR
  subgraph nexusServices [nexus services]
    gw[gateway]
    api[control plane]
    ingest[ingest]
    audit[audit]
  end
  subgraph aegisServices [Aegis services]
    edge[aegis-edge]
    engine[aegis-engine]
  end
  subgraph stores [Stores]
    ch[(ClickHouse<br/>Tier 1)]
    otel[OTel Collector<br/>Tier 2]
  end
  subgraph tenant [Tenant-side]
    tenantOtel[Tenant OTLP/Prom<br/>Tier 1 export]
  end
  gw -->|Tier1_tenant_events| ch
  api -->|Tier1_tenant_events| ch
  ingest -->|Tier1_tenant_events| ch
  gw -.->|Tier2_ops_OTLP| otel
  api -.->|Tier2_ops_OTLP| otel
  ingest -.->|Tier2_ops_OTLP| otel
  audit -.->|Tier2_ops_OTLP| otel
  edge -.->|Tier2_ops_OTLP| otel
  engine -.->|Tier2_ops_OTLP| otel
  ch -->|Tier1_tenant_export_allowlisted| tenantOtel

Tier 2 (operator OTLP) rules

Tier 2 is the channel initialized by the vortex-telemetry skeleton in both products.

Forbidden in Tier 2

tenant_id, org_id, project_id, user_id, api_key_id, session_id.
Request or response bodies (including any prompt / completion text).
Provider-side secrets.
Model-specific identifiers that could fingerprint a tenant (e.g. a custom fine-tuned model id).

Allowed in Tier 2

service.name, service.version, service.instance.id.
deployment.environment.
host.name (operator-side host).
HTTP routing labels: method, route template (not raw path), status code class.
Model family (e.g. "gpt-4o", "llama-3-8b-instruct") — not custom tenant aliases.
Provider name (e.g. "openai", "anthropic", "bedrock").
Error kind enum (not error message when the message can embed request content).

Cardinality budget per service

Total active label combinations per service: ≤ 10 000. Violating this is a bug, not a warning.

A CI test asserts this bound by simulating synthetic load.

Span structure

Minimum spans every request-handling service MUST emit (Tier 2):

http.request at the edge: root span for an inbound request.
engine.generate (Aegis) / gateway.attempt (nexus): the core operation span.
Optional child spans for provider calls, DB queries, etc.

Attribute naming follows OpenTelemetry semantic conventions where one exists; otherwise prefix with the product name: aegis.* or nexus.*.

Metric types

Concept	Type	Reason
Request count	`Counter`	Monotonic.
In-flight requests	`UpDownCounter`	Increases and decreases.
Latency end-to-end	`Histogram`	Distribution matters.
Time-to-first-token (TTFT)	`Histogram`	Distribution matters.
Observed tokens-per-second per request	`Histogram` recording seconds elapsed and token count per request, derived at query time	A histogram of instantaneous TPS is meaningless; record the ingredients.
Queue depth	`UpDownCounter`
Error count by kind	`Counter`

Do not define a histogram for derived rates. Record the numerator and denominator; compute the rate in the dashboard query.

Reference machine (for performance acceptance)

When product acceptance criteria include a latency or throughput threshold, it is measured against the reference machine:

Linux x86_64, 8 vCPU, 32 GB RAM.
Local SSD.
No GPU.
cargo build --release with the workspace default features.

Deviations (GPU-gated features, tactical-edge targets) MUST specify an alternative reference machine explicitly.

Skeleton vs full

Skeleton: vortex-telemetry::init(config) wires up an OTLP tracer and a Prometheus metric registry; emits http.request root spans and a single request counter. No histograms, no cardinality guard, no dashboards.
Full OTLP: extends vortex-telemetry with histograms, the cardinality guard, Grafana LGTM dashboard JSON, and the Tier 1 / Tier 2 attribute allowlist enforcement.

No product telemetry feature before full OTLP may include a histogram or a cardinality-guard acceptance criterion. The cross-check pass enforces this.