Skip to content
Nexus

0018 - Nexus-native agentic tracing

  • Status: Accepted
  • Date: 2026-05-26
  • Deciders: nexus core team
  • Tags: tracing, observability, sdk, clickhouse, opentelemetry, openinference

Context and Problem Statement

Section titled “Context and Problem Statement”

Nexus already records tenant-visible request analytics, opt-in full payload trace artifacts, live session request streams, and separate audit events. That model is request-centric: a tenant can inspect gateway calls and sessions, but not the full agentic execution tree around those calls.

Agent applications need richer traces. A single user turn can contain prompt rendering, retrieval, reranking, multiple LLM calls, tool calls, tool results, guardrails, application logs, and final response synthesis. The trace system needs to show that execution live while preserving the current Nexus boundaries:

  • request analytics are not audit events;
  • full payload artifacts are opt-in;
  • tenant data is scoped by authenticated Nexus org/project state;
  • operator telemetry remains separate from tenant-visible trace artifacts;
  • SDK and external framework traces should be portable through OpenTelemetry-compatible conventions.

Reference systems provide useful patterns but do not fit wholesale. OpenInference provides a useful semantic convention layer over OpenTelemetry. Langfuse provides useful LLM trace explorer and append-event ideas, but its product internals and enterprise directories are not a Nexus dependency boundary. Logfire provides useful OpenTelemetry-first and SQL-queryable observability patterns, but its server/backend is closed-source. Phoenix provides a useful trace-inspector interaction model, but its server license is not allowed for Nexus runtime artifacts.

Decision Drivers

Section titled “Decision Drivers”
  • Tenant trace writes need a narrow, auditable authentication model.
  • Nexus needs live trace events before a gateway request has completed.
  • Nexus needs trace tree joins, request joins, session filters, cost/token analytics, and SQL/HQL query surfaces.
  • Full trace payload storage must remain policy-governed and opt-in.
  • SDKs should interoperate with OpenTelemetry/OpenInference ecosystems without delegating Nexus tenancy, authz, retention, or redaction to those ecosystems.
  • Third-party dependencies must comply with the Nexus license policy.
  • The existing internal ingest worker should not become an external API surface.

Considered Options

Section titled “Considered Options”
  • Nexus-native trace backend with OpenTelemetry/OpenInference boundary compatibility. Nexus stores normalized trace records in its data plane while SDKs, ingest adapters, and exporters speak OpenTelemetry/OpenInference-compatible shapes.
  • Store raw OpenTelemetry spans as the product source of truth. This maximizes generic observability compatibility but leaves Nexus to recover tenancy, request joins, full-payload policy, and trace UI semantics from arbitrary attributes.
  • Fork or embed an existing LLM observability product. This can accelerate UI and ingestion, but creates licensing, product-shape, and backend-coupling risks.
  • Use Loki as the canonical tenant trace store. Loki fits log streams, but Nexus needs trace-tree joins, request joins, cost/token analytics, and SQL/HQL-style analysis over promoted trace fields.
  • Expose the existing ingest service as the public trace API. This is simple operationally, but it changes an internal NATS consumer into an external edge and weakens the service boundary.

Decision Outcome

Section titled “Decision Outcome”

Chosen option: Nexus-native trace backend with OpenTelemetry/OpenInference boundary compatibility.

Nexus owns the tenant trace backend and console trace explorer. OpenTelemetry and OpenInference are compatibility layers at SDK, ingest-adapter, and export boundaries. SDKs emit OpenInference-compatible spans where practical, but Nexus owns the SDK helpers, privacy defaults, project scoping, trace-capture controls, transport, storage, read authorization, and live UI behavior.

Nexus uses a dedicated trace-ingest service as the external write surface for OTLP/OpenInference trace traffic. trace-ingest authenticates project-level trace ingest keys, normalizes incoming spans into Nexus trace records, and publishes tenant-scoped trace events to NATS JetStream. The existing ingest service remains an internal worker for NATS consumption and ClickHouse/MinIO writes.

Project-level trace ingest keys are write-only credentials. They scope accepted trace data to one project and are distinct from gateway API keys for model traffic and developer access tokens for read/control-plane APIs. Trace attributes that claim org or project identity are treated as hints and must match the authenticated project scope.

Canonical storage is:

  • NATS JetStream for live trace lifecycle events and internal fan-out.
  • ClickHouse for high-volume trace records, promoted query columns, and SQL/HQL analysis.
  • MinIO/S3 for large trace artifacts when full capture is enabled and policy allows storage.
  • Postgres for trace ingest key metadata, trace settings, retention metadata, saved views, and control-plane lifecycle state.

OpenInference-compatible packages are preferred where their licenses are allowed and their extension points fit Nexus. Package decisions are recorded separately with license evidence. Packages that are closed-source, license-incompatible, or product-coupled are used as references only or reimplemented as Nexus-native code.

Full payload capture remains opt-in and policy-governed. OpenInference or SDK-side masking is client-side minimization, not the compliance redaction authority. Nexus policy/redaction and artifact read authorization remain authoritative.

Consequences

Section titled “Consequences”
  • Nexus trace data has a stable tenant-native shape while remaining compatible with OpenTelemetry/OpenInference tooling at boundaries.
  • The new external trace write surface has a clear auth model and does not broaden the existing internal ingest worker.
  • ClickHouse remains the analytical store for tenant-visible high-volume data.
  • MinIO artifact storage remains aligned with existing request body storage and opt-in full trace capture.
  • Package adoption proceeds conservatively through explicit license and architecture review.
  • The console can keep the live session monitor as the primary workflow and add a trace-inspector drill-in without replacing request/session views.
  • Implementations that emit raw OTLP must pass through Nexus normalization before becoming product-visible trace records.

More Information

Section titled “More Information”