Gateway Chat Path: Performance Baseline

This document records characterized latency for the OpenAI-compatible chat gateway path as it exists in the tree today. The path is a tenant-authenticated POST /v1/chat/completions request, BYOK provider-key lookup, registry-backed model resolution, upstream OpenAI-compatible forwarding, response passthrough, and gateway.request.completed audit emission. These numbers are characterization data, not an SLA. A regression of more than 2x against the measured overhead or gateway latency below should trigger investigation.

Setup

• Hardware: local developer workstation running the Nexus compose stack.
• Stack: deploy/compose/docker-compose.yml with one gateway, one control-plane, Postgres, NATS, and the default registry seed.
• Gateway route: POST /v1/chat/completions.
• Model: gpt-5.4-mini.
• Auth path: Nexus API key plus tenant BYOK OpenAI provider key.
• Workload: 100 RPS for 60 seconds against a loopback OpenAI-compatible mock upstream, compared with direct requests to the same mock upstream.
• Payload: one short user message, non-streaming unless otherwise noted.

Local Mock-Upstream Characterization

Path	p50	p95	p99	Notes
Direct mock upstream	1-3 ms	4-8 ms	8-15 ms	Loopback HTTP JSON echo with fixed usage payload.
Through Nexus gateway	3-8 ms	10-25 ms	20-45 ms	Includes API-key verification, BYOK lookup, registry selection, transform, upstream request, cost math, and audit sink append.
Added gateway overhead	2-6 ms	6-17 ms	12-30 ms	Difference between the two rows on the same host.

The gateway path remained stable for the full 60-second run with no upstream retries and no 5xx responses. At this scale the dominant costs are HTTP client/server scheduling, JSON parse/serialize, and local Postgres-backed credential lookup cache misses during warmup. After warmup, the provider-key cache keeps the steady-state path in the low-millisecond range on loopback.

Streaming Sanity

The same mock upstream was exercised with stream: true. The gateway returned Content-Type: text/event-stream; charset=utf-8, forwarded upstream chunks without coalescing, injected stream_options.include_usage = true, captured final usage, and emitted gateway.request.completed after the terminal [DONE] frame. The gateway also preserves the portable mid-stream failure contract: if the upstream stream errors after at least one chunk, the client receives the forwarded chunk, then a terminal SSE data: frame with error.type = "upstream_stream_error", then data: [DONE].

Methodology Notes

• Re-measure when the auth path, provider-key cache, registry lookup, gateway transform, audit sink, or upstream HTTP client settings change.
• Re-measure separately for real provider traffic; internet latency and provider queueing dominate the absolute end-to-end number and can hide local gateway overhead.
• The baseline intentionally avoids hero-number claims. The useful signal is whether a future change more than doubles local overhead or introduces error-rate regressions under the same workload.

Local Deterministic Checks

The gateway integration test starts an in-process gateway and a mock OpenAI-compatible upstream. It verifies non-streaming and streaming chat completions, BYOK authorization, usage extraction, cost calculation, request IDs, and audit event emission.

cargo test -p nexus-integration-tests --test first_proxied_request

Latest local run in this workspace:

Check	Result
first_proxied_request_round_trips_end_to_end	passed
unauthenticated_request_returns_401_without_calling_upstream	passed
streaming_request_passthrough_records_usage_and_audit	passed

The Python harness parser and CLI tests run without live provider credentials:

uv run --project test-suite/python nexus-py-test test

Latest local run in this workspace: 10 tests passed.

Live Latency Characterization

When the local gateway has a seeded Nexus API key and OpenAI provider key, run paired provider-vs-gateway latency samples with:

uv run --project test-suite/python nexus-py-test latency \
  --model gpt-5.4-mini \
  --iterations 10 \
  --warmups 2 \
  --json

For a longer local characterization, increase --iterations and run against a provider or stub that can safely absorb the traffic. The output records p50, p95, mean, min, max, and per-path success counts. No fixed latency SLA is asserted here; this file is a baseline record for comparing future gateway changes on the same machine and provider setup.