Gateway compatibility testing

This suite is the reproducible compatibility harness for Nexus’s OpenAI-compatible public API. It combines a pinned OpenAI OpenAPI snapshot with official SDK black-box tests against a running Nexus gateway.

Gold Truth

• spec/openapi.yaml is a pinned copy of the official OpenAI OpenAPI specification.
• Official SDKs are the runtime compatibility oracle. The first runtime suite uses Python’s openai package.
• Raw HTTP tests cover cases that SDK type systems cannot express, such as unknown field passthrough.

Running The Python Suite

Start Nexus with deterministic provider fixtures, then run from test-suite/compat/python:

just test

Or from nexus/:

just test-compat

Equivalent without just:

uv run --project test-suite/compat/python pytest test-suite/compat/python/tests -n auto

Environment:

• NEXUS_COMPAT_BASE_URL: defaults to http://127.0.0.1:14450/v1
• NEXUS_API_KEY: required for tests against a live Nexus gateway
• NEXUS_QUERY_BASE_URL: defaults to http://127.0.0.1:14461; used by image usage/cost analytics tests
• NEXUS_QUERY_TOKEN or NEXUS_SERVICE_TOKEN: required for query analytics tests (gateway API keys cannot read the query API)
• NEXUS_COMPAT_PG_CONTAINER: defaults to vortex-dev-postgres-1; the analytics test resolves the API key’s org by querying this control-plane Postgres container via docker exec
• NEXUS_INGEST_BASE_URL: defaults to http://127.0.0.1:14457; analytics tests skip when ingest is unhealthy
• NEXUS_COMPAT_REQUIRE_INGEST: set to 1 (certification/CI) to make an unhealthy ingest a hard failure instead of a skip
• Model selection reads GET /v1/models and runs the OpenAI-compatible chat, tools, Responses, embeddings, and image cases once per pinned provider (openai/..., anthropic/..., azure-foundry-gov/..., bedrock-gov/..., and any other <provider>/ prefix the catalog publishes). Each provider uses its cheapest model that advertises the capability. Direct OpenAI compare, session analytics, Agents SDK, and native Anthropic Messages stay on a single model so those suites do not fan out. Set NEXUS_COMPAT_PROVIDERS=openai,anthropic to restrict the matrix.
• NEXUS_COMPAT_MODEL: optional OpenAI chat pin. Replaces only the OpenAI slot; other chat providers still run.
• NEXUS_COMPAT_TOOL_MODEL: optional OpenAI tools pin for the OpenAI slot.
• NEXUS_COMPAT_REASONING_MODEL: optional OpenAI reasoning+tools pin for the OpenAI slot.
• NEXUS_COMPAT_RESPONSES_MODEL: optional OpenAI Responses pin for the OpenAI slot.
• NEXUS_COMPAT_BAND_MODEL: optional OpenAI long-context pin. When set (and present in the catalog), the expensive band-crossing probe runs against that row, or the cheapest openai/ row with more than one pricing band if the pin is stale. Set NEXUS_COMPAT_ENABLE_BAND_CROSSING=1 to auto-pick without a pin. The probe is skipped by default because it sends a 272k-token prompt.
• NEXUS_COMPAT_EMBEDDING_MODEL: optional OpenAI embeddings pin for the OpenAI slot. Always use a pinned id; an unprefixed text-embedding-3-small can match every embeddings-capable provider of that canonical model.
• NEXUS_COMPAT_EMBEDDING_MODELS: optional extra pinned ids merged into the embeddings listing check. They do not replace the per-provider matrix. Default coverage is the cheapest embeddings row for every provider that advertises capabilities.embeddings. OpenAI-schema extras (token ids, base64, dimensions) run for OpenAI and Foundry ids; Titan/Bedrock ids run the string/string-array contract plus token-id/base64 rejection.
• NEXUS_COMPAT_ANTHROPIC_MODEL: optional Anthropic chat pin for the Anthropic slot and the native Messages slice. Set NEXUS_COMPAT_ENABLE_ANTHROPIC=0 to omit native Anthropic files; OpenAI-compat Anthropic chat still runs when the catalog lists anthropic/ rows.
• NEXUS_COMPAT_ANTHROPIC_NATIVE_MODEL: optional native Messages pin. Default is the unprefixed cheapest Anthropic chat id.
• NEXUS_COMPAT_IMAGE_MODEL: optional OpenAI image-generation pin for the OpenAI slot.
• NEXUS_COMPAT_IMAGE_EDIT_MODEL: optional OpenAI image-edit pin for the OpenAI slot. gpt-image-1-mini is not selected even when catalogued as edits, because OpenAI images/edits rejects that id.
• NEXUS_COMPAT_IMAGE_VARIATION_MODEL: optional OpenAI image-variation pin for the OpenAI slot.
• NEXUS_COMPAT_REPORT_PATH: optional JSON report path for targeted certification runs
• OPENAI_API_KEY: required only for direct OpenAI comparison tests (including image token usage parity)
• OPENAI_COMPAT_MODEL: optional direct OpenAI chat id; defaults to the unprefixed Nexus chat id
• OPENAI_COMPAT_EMBEDDING_MODEL: optional direct OpenAI embedding id; defaults to the unprefixed Nexus embedding id
• OPENAI_COMPAT_IMAGE_MODEL: optional direct OpenAI image id; defaults to the unprefixed Nexus image id
• NEXUS_COMPAT_ENABLE_TRACING: set to 1 to collect the native OTLP trace-ingest module (test_trace_ingest.py)
• NEXUS_TRACE_INGEST_URL: trace-ingest endpoint for the trace suite, e.g. http://localhost:14458/v1/traces (a ?key= parameter is used as the bearer token)
• NEXUS_TRACE_INGEST_KEY: trace ingest key when not embedded in NEXUS_TRACE_INGEST_URL
• NEXUS_TRACE_QUERY_TOKEN: optional developer/manager token with trace:read for the trace read-back assertions (falls back to NEXUS_QUERY_TOKEN/NEXUS_SERVICE_TOKEN, which only carry query:read and cause the read-back to skip)

The direct OpenAI comparison tests are A/B shape checks. They send the same SDK call to OpenAI and to Nexus, then verify that both responses satisfy the same high-level OpenAI object contract.

Running The Anthropic Certification Slice

The Anthropic slice is a targeted provider-adapter certification for behavior implemented in the gateway today. It uses the official Python openai SDK against the Nexus OpenAI-compatible base URL, while selecting the cheapest Anthropic-backed model currently in the registry. The default just test-compat run includes this slice whenever GET /v1/models lists an anthropic/ row.

From nexus/:

just test-compat-anthropic

From test-suite/compat/python:

just test-anthropic

To write a machine-readable report:

NEXUS_COMPAT_REPORT_PATH=test-suite/compat/reports/anthropic-certification.json \
  just test-compat-anthropic

Expected provider coverage:

• pass: Anthropic chat completions, streaming chat completions, tool-call translation, usage shape, terminal finish reasons, and tenant-visible model listing.
• unsupported: Anthropic /v1/responses, /v1/embeddings, and /v1/images/generations.
• Provider-independent Nexus artifact coverage remains in the files/vector-stores suite and is not counted as Anthropic adapter coverage.

Running The Native Anthropic Messages Certification Slice

The native slice certifies the /anthropic/v1/messages edge with the official Python anthropic SDK (pinned in pyproject.toml and spec/anthropic-sdk-provenance.json).

From nexus/:

just test-compat-anthropic-native

TypeScript coverage with @anthropic-ai/sdk:

just test-node-anthropic-native

Claude Code launch configuration, error-parity matrix, and the manual session checklist live under claude-code/.

To write a machine-readable report:

NEXUS_COMPAT_REPORT_PATH=test-suite/compat/reports/anthropic-native-certification.json \
  NEXUS_COMPAT_REPORT_SUITE=anthropic-native-certification \
  just test-compat-anthropic-native

Expected native coverage:

• pass: non-streaming and streaming messages, tool loops, count_tokens, bearer and x-api-key auth, error envelopes (invalid key, unknown model, upstream 429/529), request-id header, unknown-route 404.
• live-only (requires ANTHROPIC_API_KEY): extended thinking replay, prompt caching, beta header forwarding, vision.
• NEXUS_COMPAT_ENABLE_THINKING=1: run extended-thinking certification cases against a stack that supports them.
• NEXUS_COMPAT_ENABLE_CACHING=1: run prompt-caching certification cases when cache token usage is observable.
• NEXUS_COMPAT_MOCK_MODEL: defaults to nxmock-tiny-fast for upstream 429/529 error-path tests (skipped when the model is absent).

Session Analytics

tests/test_sessions_analytics.py certifies that the session-correlation headers a client sends on a request flow through the ingest pipeline into ClickHouse and become readable through the query API.

The tests send chat completions tagged with nexus-session-id (plus optional nexus-session-path and nexus-session-name), then poll GET /api/sessions and GET /api/sessions/{id}/timeline until the rows are indexed.

These tests run as part of just test-compat. They require:

• NEXUS_QUERY_TOKEN or NEXUS_SERVICE_TOKEN: query API auth (gateway API keys cannot read the query API).
• A healthy ingest pipeline; set NEXUS_COMPAT_REQUIRE_INGEST=1 to fail instead of skip when ingest is unhealthy.

Native OTLP Trace Ingest

tests/test_trace_ingest.py certifies the native tenant trace pipeline end to end: an OpenTelemetry protobuf export posted to the trace-ingest edge is normalized, indexed into ClickHouse, and read back through the query API as a trace tree and a per-session trace summary. The tool-calling scenario links an LLM and a TOOL span to a real gateway request via the nexus.request_id attribute.

It is gated and opt-in (it needs the OpenTelemetry protobuf dependency). From nexus/:

just test-compat-tracing

Environment:

• NEXUS_COMPAT_ENABLE_TRACING=1: collect the trace-ingest module (set automatically by the just target).
• NEXUS_TRACE_INGEST_URL: trace-ingest endpoint, e.g. http://localhost:14458/v1/traces. A ?key= parameter is accepted and used as the bearer token.
• NEXUS_TRACE_INGEST_KEY: trace ingest key when not embedded in the URL.
• NEXUS_TRACE_QUERY_TOKEN: a developer/manager token with the trace:read scope. Trace reads require trace:read; the internal service token only carries query:read, so the read-back assertions skip cleanly without this token. The OTLP write path is always certified.

Full Gateway Certification

To run the complete BYOK certification against a live stack in one command from nexus/:

just test-gateway-cert

It runs, in order: just status, just test-compat, just test-compat-agentic, just test-compat-tracing, just test-compat-anthropic, just test-compat-anthropic-native, and just test-node-anthropic-native. It defaults NEXUS_COMPAT_AGENTIC_VECTOR_STORE_ID to the first id in NEXUS_VECTOR_STORE_IDS for the agentic file_search scenario.

Spec Refresh

Refresh spec/openapi.yaml from the official source:

curl -fsSL https://raw.githubusercontent.com/openai/openai-openapi/refs/heads/manual_spec/openapi.yaml \
  -o test-suite/compat/spec/openapi.yaml
shasum -a 256 test-suite/compat/spec/openapi.yaml

After refreshing, update spec/provenance.json with the source, retrieval time, and checksum.