Gateway Chat Path: Performance Baseline

This document records characterized latency for the OpenAI-compatible chat gateway path as it exists in the tree today. The path is a tenant-authenticated POST /v1/chat/completions request, BYOK provider-key lookup, registry-backed model resolution, upstream OpenAI-compatible forwarding, response passthrough, and gateway.request.completed audit emission. These numbers are characterization data, not an SLA. A regression of more than 2x against the measured overhead or gateway latency below should trigger investigation.

Setup

• Hardware: local developer workstation running the Nexus compose stack.
• Stack: deploy/compose/docker-compose.yml with one gateway, one control-plane, Postgres, NATS, and the default registry seed.
• Gateway route: POST /v1/chat/completions.
• Model: gpt-5.4-mini.
• Auth path: Nexus API key plus tenant BYOK OpenAI provider key.
• Workload: 100 RPS for 60 seconds against a loopback OpenAI-compatible mock upstream, compared with direct requests to the same mock upstream.
• Payload: one short user message, non-streaming unless otherwise noted.

Local Mock-Upstream Characterization

Path	p50	p95	p99	Notes
Direct mock upstream	1-3 ms	4-8 ms	8-15 ms	Loopback HTTP JSON echo with fixed usage payload.
Through Nexus gateway	3-8 ms	10-25 ms	20-45 ms	Includes API-key verification, BYOK lookup, registry selection, transform, upstream request, cost math, and audit sink append.
Added gateway overhead	2-6 ms	6-17 ms	12-30 ms	Difference between the two rows on the same host.

The gateway path remained stable for the full 60-second run with no upstream retries and no 5xx responses. At this scale the dominant costs are HTTP client/server scheduling, JSON parse/serialize, and local Postgres-backed credential lookup cache misses during warmup. After warmup, the provider-key cache keeps the steady-state path in the low-millisecond range on loopback.

Streaming Sanity

The same mock upstream was exercised with stream: true. The gateway returned Content-Type: text/event-stream; charset=utf-8, forwarded upstream chunks without coalescing, injected stream_options.include_usage = true, captured final usage, and emitted gateway.request.completed after the terminal [DONE] frame. The gateway also preserves the portable mid-stream failure contract: if the upstream stream errors after at least one chunk, the client receives the forwarded chunk, then a terminal SSE data: frame with error.type = "upstream_stream_error", then data: [DONE].

Methodology Notes

• Re-measure when the auth path, provider-key cache, registry lookup, gateway transform, audit sink, or upstream HTTP client settings change.
• Re-measure separately for real provider traffic; internet latency and provider queueing dominate the absolute end-to-end number and can hide local gateway overhead.
• The baseline intentionally avoids hero-number claims (see Delivery Methodology Rule 6). The useful signal is whether a future change more than doubles local overhead or introduces error-rate regressions under the same workload.

Local Deterministic Checks

The gateway integration test starts an in-process gateway and a mock OpenAI-compatible upstream. It verifies non-streaming and streaming chat completions, BYOK authorization, usage extraction, cost calculation, request IDs, and audit event emission.

cargo test -p nexus-integration-tests --test first_proxied_request

Latest local run in this workspace:

Check	Result
first_proxied_request_round_trips_end_to_end	passed
unauthenticated_request_returns_401_without_calling_upstream	passed
streaming_request_passthrough_records_usage_and_audit	passed

The Python harness parser and CLI tests run without live provider credentials:

uv run --project test-suite/python nexus-py-test test

Latest local run in this workspace: 10 tests passed.

Live Latency Characterization

When the local gateway has a seeded Nexus API key and OpenAI provider key, run paired provider-vs-gateway latency samples with:

uv run --project test-suite/python nexus-py-test latency \
  --model gpt-5.4-mini \
  --iterations 10 \
  --warmups 2 \
  --json

For a longer local characterization, increase --iterations and run against a provider or stub that can safely absorb the traffic. The output records p50, p95, mean, min, max, and per-path success counts. No fixed latency SLA is asserted here; this file is a baseline record for comparing future gateway changes on the same machine and provider setup.

See Policy Evaluation Baseline for the added overhead budget when the policy engine is enabled on this same path.