Gateway compatibility testing

Run OpenAI and Anthropic compatibility suites and gateway certification checks

Last updated on: November 3, 2026

This suite is the reproducible compatibility harness for Nexus’s OpenAI-compatible public API. It combines a pinned OpenAI OpenAPI snapshot with official SDK black-box tests against a running Nexus gateway.

Gold Truth

spec/openapi.yaml is a pinned copy of the official OpenAI OpenAPI specification.
Official SDKs are the runtime compatibility oracle. The first runtime suite uses Python’s openai package.
Raw HTTP tests cover cases that SDK type systems cannot express, such as unknown field passthrough.

Running The Python Suite

Start Nexus with deterministic provider fixtures, then run from test-suite/compat/python:

just test

Or from nexus/:

just test-compat

Equivalent without just:

uv run --project test-suite/compat/python pytest test-suite/compat/python/tests -n auto

Environment:

NEXUS_COMPAT_BASE_URL: defaults to http://127.0.0.1:14450/v1
NEXUS_API_KEY: required for tests against a live Nexus gateway
NEXUS_QUERY_BASE_URL: defaults to http://127.0.0.1:14461; used by image usage/cost analytics tests
NEXUS_QUERY_TOKEN or NEXUS_SERVICE_TOKEN: required for query analytics tests (gateway API keys cannot read the query API)
NEXUS_COMPAT_PG_CONTAINER: defaults to vortex-dev-postgres-1; the analytics test resolves the API key’s org by querying this control-plane Postgres container via docker exec
NEXUS_INGEST_BASE_URL: defaults to http://127.0.0.1:14457; analytics tests skip when ingest is unhealthy
NEXUS_COMPAT_REQUIRE_INGEST: set to 1 (certification/CI) to make an unhealthy ingest a hard failure instead of a skip
Model selection reads GET /v1/models and runs the OpenAI-compatible chat, tools, Responses, embeddings, and image cases once per pinned provider (openai/..., anthropic/..., azure-foundry-gov/..., bedrock-gov/..., and any other <provider>/ prefix the catalog publishes). Each provider uses its cheapest model that advertises the capability. Direct OpenAI compare, session analytics, Agents SDK, and native Anthropic Messages stay on a single model so those suites do not fan out. Set NEXUS_COMPAT_PROVIDERS=openai,anthropic to restrict the matrix.
NEXUS_COMPAT_MODEL: optional OpenAI chat pin. Replaces only the OpenAI slot; other chat providers still run.
NEXUS_COMPAT_TOOL_MODEL: optional OpenAI tools pin for the OpenAI slot.
NEXUS_COMPAT_REASONING_MODEL: optional OpenAI reasoning+tools pin for the OpenAI slot.
NEXUS_COMPAT_RESPONSES_MODEL: optional OpenAI Responses pin for the OpenAI slot.
NEXUS_COMPAT_BAND_MODEL: optional OpenAI long-context pin. When set (and present in the catalog), the expensive band-crossing probe runs against that row, or the cheapest openai/ row with more than one pricing band if the pin is stale. Set NEXUS_COMPAT_ENABLE_BAND_CROSSING=1 to auto-pick without a pin. The probe is skipped by default because it sends a 272k-token prompt.
NEXUS_COMPAT_EMBEDDING_MODEL: optional OpenAI embeddings pin for the OpenAI slot. Always use a pinned id; an unprefixed text-embedding-3-small can match every embeddings-capable provider of that canonical model.
NEXUS_COMPAT_EMBEDDING_MODELS: optional extra pinned ids merged into the embeddings listing check. They do not replace the per-provider matrix. Default coverage is the cheapest embeddings row for every provider that advertises capabilities.embeddings. OpenAI-schema extras (token ids, base64, dimensions) run for OpenAI and Foundry ids; Titan/Bedrock ids run the string/string-array contract plus token-id/base64 rejection.
NEXUS_COMPAT_ANTHROPIC_MODEL: optional Anthropic chat pin for the Anthropic slot and the native Messages slice. Set NEXUS_COMPAT_ENABLE_ANTHROPIC=0 to omit native Anthropic files; OpenAI-compat Anthropic chat still runs when the catalog lists anthropic/ rows.
NEXUS_COMPAT_ANTHROPIC_NATIVE_MODEL: optional native Messages pin. Default is the unprefixed cheapest Anthropic chat id.
NEXUS_COMPAT_IMAGE_MODEL: optional OpenAI image-generation pin for the OpenAI slot.
NEXUS_COMPAT_IMAGE_EDIT_MODEL: optional OpenAI image-edit pin for the OpenAI slot. gpt-image-1-mini is not selected even when catalogued as edits, because OpenAI images/edits rejects that id.
NEXUS_COMPAT_IMAGE_VARIATION_MODEL: optional OpenAI image-variation pin for the OpenAI slot.
NEXUS_COMPAT_REPORT_PATH: optional JSON report path for targeted certification runs
OPENAI_API_KEY: required only for direct OpenAI comparison tests (including image token usage parity)
OPENAI_COMPAT_MODEL: optional direct OpenAI chat id; defaults to the unprefixed Nexus chat id
OPENAI_COMPAT_EMBEDDING_MODEL: optional direct OpenAI embedding id; defaults to the unprefixed Nexus embedding id
OPENAI_COMPAT_IMAGE_MODEL: optional direct OpenAI image id; defaults to the unprefixed Nexus image id
NEXUS_COMPAT_ENABLE_TRACING: set to 1 to collect the native OTLP trace-ingest module (test_trace_ingest.py)
NEXUS_TRACE_INGEST_URL: trace-ingest endpoint for the trace suite, e.g. http://localhost:14458/v1/traces (a ?key= parameter is used as the bearer token)
NEXUS_TRACE_INGEST_KEY: trace ingest key when not embedded in NEXUS_TRACE_INGEST_URL
NEXUS_TRACE_QUERY_TOKEN: optional developer/manager token with trace:read for the trace read-back assertions (falls back to NEXUS_QUERY_TOKEN/NEXUS_SERVICE_TOKEN, which only carry query:read and cause the read-back to skip)

The direct OpenAI comparison tests are A/B shape checks. They send the same SDK call to OpenAI and to Nexus, then verify that both responses satisfy the same high-level OpenAI object contract.

Running The Anthropic Certification Slice

The Anthropic slice is a targeted provider-adapter certification for behavior implemented in the gateway today. It uses the official Python openai SDK against the Nexus OpenAI-compatible base URL, while selecting the cheapest Anthropic-backed model currently in the registry. The default just test-compat run includes this slice whenever GET /v1/models lists an anthropic/ row.

From nexus/:

just test-compat-anthropic

From test-suite/compat/python:

just test-anthropic

To write a machine-readable report:

NEXUS_COMPAT_REPORT_PATH=test-suite/compat/reports/anthropic-certification.json \
  just test-compat-anthropic

Expected provider coverage:

pass: Anthropic chat completions, streaming chat completions, tool-call translation, usage shape, terminal finish reasons, and tenant-visible model listing.
unsupported: Anthropic /v1/responses, /v1/embeddings, and /v1/images/generations.
Provider-independent Nexus artifact coverage remains in the files/vector-stores suite and is not counted as Anthropic adapter coverage.

Running The Native Anthropic Messages Certification Slice

The native slice certifies the /anthropic/v1/messages edge with the official Python anthropic SDK (pinned in pyproject.toml and spec/anthropic-sdk-provenance.json).

From nexus/:

just test-compat-anthropic-native

TypeScript coverage with @anthropic-ai/sdk:

just test-node-anthropic-native

Claude Code launch configuration, error-parity matrix, and the manual session checklist live under claude-code/.

To write a machine-readable report:

NEXUS_COMPAT_REPORT_PATH=test-suite/compat/reports/anthropic-native-certification.json \
  NEXUS_COMPAT_REPORT_SUITE=anthropic-native-certification \
  just test-compat-anthropic-native

Expected native coverage:

pass: non-streaming and streaming messages, tool loops, count_tokens, bearer and x-api-key auth, error envelopes (invalid key, unknown model, upstream 429/529), request-id header, unknown-route 404.
live-only (requires ANTHROPIC_API_KEY): extended thinking replay, prompt caching, beta header forwarding, vision.
NEXUS_COMPAT_ENABLE_THINKING=1: run extended-thinking certification cases against a stack that supports them.
NEXUS_COMPAT_ENABLE_CACHING=1: run prompt-caching certification cases when cache token usage is observable.
NEXUS_COMPAT_MOCK_MODEL: defaults to nxmock-tiny-fast for upstream 429/529 error-path tests (skipped when the model is absent).

Session Analytics

tests/test_sessions_analytics.py certifies that the session-correlation headers a client sends on a request flow through the ingest pipeline into ClickHouse and become readable through the query API.

The tests send chat completions tagged with nexus-session-id (plus optional nexus-session-path and nexus-session-name), then poll GET /api/sessions and GET /api/sessions/{id}/timeline until the rows are indexed.

These tests run as part of just test-compat. They require:

NEXUS_QUERY_TOKEN or NEXUS_SERVICE_TOKEN: query API auth (gateway API keys cannot read the query API).
A healthy ingest pipeline; set NEXUS_COMPAT_REQUIRE_INGEST=1 to fail instead of skip when ingest is unhealthy.

Native OTLP Trace Ingest

tests/test_trace_ingest.py certifies the native tenant trace pipeline end to end: an OpenTelemetry protobuf export posted to the trace-ingest edge is normalized, indexed into ClickHouse, and read back through the query API as a trace tree and a per-session trace summary. The tool-calling scenario links an LLM and a TOOL span to a real gateway request via the nexus.request_id attribute.

It is gated and opt-in (it needs the OpenTelemetry protobuf dependency). From nexus/:

just test-compat-tracing

Environment:

NEXUS_COMPAT_ENABLE_TRACING=1: collect the trace-ingest module (set automatically by the just target).
NEXUS_TRACE_INGEST_URL: trace-ingest endpoint, e.g. http://localhost:14458/v1/traces. A ?key= parameter is accepted and used as the bearer token.
NEXUS_TRACE_INGEST_KEY: trace ingest key when not embedded in the URL.
NEXUS_TRACE_QUERY_TOKEN: a developer/manager token with the trace:read scope. Trace reads require trace:read; the internal service token only carries query:read, so the read-back assertions skip cleanly without this token. The OTLP write path is always certified.

Full Gateway Certification

To run the complete BYOK certification against a live stack in one command from nexus/:

just test-gateway-cert

It runs, in order: just status, just test-compat, just test-compat-agentic, just test-compat-tracing, just test-compat-anthropic, just test-compat-anthropic-native, and just test-node-anthropic-native. It defaults NEXUS_COMPAT_AGENTIC_VECTOR_STORE_ID to the first id in NEXUS_VECTOR_STORE_IDS for the agentic file_search scenario.

Spec Refresh

Refresh spec/openapi.yaml from the official source:

curl -fsSL https://raw.githubusercontent.com/openai/openai-openapi/refs/heads/manual_spec/openapi.yaml \
  -o test-suite/compat/spec/openapi.yaml
shasum -a 256 test-suite/compat/spec/openapi.yaml

After refreshing, update spec/provenance.json with the source, retrieval time, and checksum.