Nexus test suite

Developer-facing live and compatibility tests for Nexus. This directory contains the harnesses used to exercise a running gateway, compare routed model behavior, certify OpenAI and Anthropic SDK compatibility, and reconcile recorded usage against provider billing data.

Most tests in this directory are integration tests. They expect a Nexus stack to be running and reachable, and many of them call real model providers. Use focused targets while developing and reserve the full certification target for release checks or changes that touch provider adapters, routing, usage accounting, tracing, or public API compatibility.

Directory Layout

test-suite/
├── .env.example              # Client-side test configuration template
├── adopter/                  # Shell E2E flows for first-adopter scenarios
├── billing/                  # Provider admin cost reconciliation utilities
├── compat/                   # OpenAI/Anthropic SDK compatibility certification
├── harness/                  # Pinned coding-harness two-turn sessions (OpenCode, Codex, Claude Code)
├── node/                     # Vitest live model behavior and report harness
├── python/                   # Small Python CLI for ad hoc live checks
└── runs/                     # Generated run artifacts and reports

The suite is intentionally split by purpose:

• node/ is the main live behavior harness. It uses Vitest, the Vercel AI SDK, OpenAI-compatible requests, and a Gemini judge for quality-scored tasks.
• compat/python/ is the public API compatibility harness. It uses official SDKs as black-box clients against Nexus.
• python/ is a lightweight CLI for fast manual comparisons, latency probes, smoke checks, and stream checks.
• billing/ contains opt-in admin-key tooling for deriving and reconciling provider costs.
• adopter/ contains shell flows that exercise onboarding paths through the Nexus CLI.
• harness/ runs a pinned coding harness (OpenCode, Codex CLI, or Claude Code) through the gateway for a two-turn session on a small buggy repository and verifies the session by executing the code it leaves behind. Run it with just harness-gauntlet <harness> <model> [<switch model>]; see harness/README.md for keys, pinned versions, and exit codes.

Prerequisites

Install the repository-level developer tools before running the suite:

• Node.js >=26
• pnpm 9.x
• Python >=3.11
• uv
• just
• Docker, for local-stack and some analytics compatibility tests

For live gateway tests, start the local Nexus stack from nexus/:

just dev
just status

The default local gateway base URL is http://127.0.0.1:14450/v1. Some compatibility tests also read back analytics or traces through the query API and therefore need the ingest/query services to be healthy.

Environment Setup

Copy the client-side test environment template:

cp test-suite/.env.example test-suite/.env

Fill in the values needed for the tests you plan to run. The Node harness loads test-suite/.env. The Python harnesses load environment values from ancestor .env files without overwriting values already exported in the shell.

Common variables:

• NEXUS_API_KEY: API key used as the bearer token for gateway requests.
• INFERENCE_BASE_URL: Node harness target, defaulting to http://localhost:14450/v1.
• NEXUS_COMPAT_BASE_URL: Python SDK compatibility target, defaulting to http://127.0.0.1:14450/v1.
• GEMINI_API_KEY: required by the Node judge-backed tests.
• OPENAI_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY, XAI_API_KEY: provider keys used by the stack or direct comparison tests, depending on the target.
• MODELS: comma-separated provider/model ids used by smoke and compare tests.
• TIER1_MODEL, TIER2_MODEL, TIER3_MODEL: model ids used by tiered quality, reasoning, tool, and long-context tests.
• NEXUS_QUERY_TOKEN or NEXUS_SERVICE_TOKEN: query API read token for analytics assertions.
• NEXUS_TRACE_INGEST_URL and NEXUS_TRACE_INGEST_KEY: trace ingest target and token for native OTLP certification.
• NEXUS_TRACE_QUERY_TOKEN: query token with trace-read permission for trace read-back assertions.

Keep real secrets out of the example file. Use test-suite/.env for local client test configuration and the repository root environment files for stack configuration.

Quick Start

Run the default Node live suite, excluding the expensive compare matrix:

just test

Run one Node suite:

just test smoke
just test streaming
just test failover

Pass Vitest flags through after the suite name:

just test smoke -- -t "round-trips"
just debug streaming -- -t "first token"

Run the OpenAI SDK compatibility suite:

just test-compat

Run a fast Python smoke check:

just test-py smoke --path nexus --model openai/gpt-5.4-mini

Regenerate the HTML report from existing run artifacts:

just report

Main Commands

Run these commands from nexus/ unless noted otherwise.

Command	Purpose
just test	Default Node/Vitest live suite, with compare.test.ts excluded.
just test <suite>	Focus one Node suite, for example just test long-context.
just debug <suite>	Run a Node suite with verbose per-record traces.
just test-node <args>	Direct pass-through to the Node Vitest wrapper.
just compare	Live model comparison table across MODELS plus nexus/auto.
just test-all	Full Node/Vitest suite including compare.test.ts.
just test-py <args>	Run the lightweight Python CLI.
just test-compat	Official OpenAI SDK compatibility suite.
just test-compat-agentic	OpenAI Agents SDK and trace-ingest compatibility slice.
just test-compat-tracing	Native OTLP trace-ingest certification slice.
just test-compat-anthropic	Anthropic-backed OpenAI-compatible certification slice.
just test-compat-anthropic-native	Native Anthropic Messages Python certification slice.
just test-node-anthropic-native	Native Anthropic Messages TypeScript SDK certification slice.
just test-gateway-cert	Full live gateway certification sequence.
just harness-gauntlet <harness> <model>	Two-turn coding-harness session, verified by executing the result.
just regen-compare [artifact]	Regenerate compare markdown artifacts from a run JSON file.
just clean	Remove generated test artifacts and local Node build/dependency outputs.

Node Live Behavior Harness

The Node suite lives in test-suite/node. It is a Vitest harness with custom reporters that write structured artifacts under test-suite/runs.

You can run it through just:

just test
just test tool-calling
just test-node tests/structured-output.test.ts

Or directly:

cd test-suite/node
pnpm install
pnpm test
pnpm test smoke
pnpm test --debug streaming
pnpm typecheck

The wrapper node/bin/test.mjs accepts either a bare suite name such as smoke or a file path such as tests/smoke.test.ts. Remaining flags are forwarded to Vitest.

Node Suite Order

Vitest runs files serially in a deliberate order while allowing test cases inside a file to run concurrently. The order starts with cheaper routing and quality checks, then moves through feature coverage, long-context tasks, creative judging, failover behavior, and finally the compare matrix when it is enabled.

Primary suites:

• smoke: every MODELS entry can complete a basic request through the gateway.
• tiered: quality-scored prompts against the configured tier models.
• reasoning: provider-options and reasoning-effort forwarding.
• streaming: stream chunk integrity, time to first byte, and jitter.
• concurrency: parallel request behavior and p95 latency limits.
• conversation: multi-turn context and stable provider behavior.
• prompt-caching: cache-control and Anthropic beta header behavior.
• structured-output: JSON schema fidelity.
• tool-calling: positive and negative tool-call round trips.
• tool-selection: tool catalog behavior across tier models.
• long-context: needle-in-haystack checks at multiple depths.
• provenance: grounded synthesis with citations over a generated knowledge base.
• creative: open-ended judge-graded tasks.
• failover: bad model, unknown provider, timeout, and context overflow paths.
• compare: head-to-head model comparison matrix.
• anthropic-native-certification: TypeScript SDK checks for the native Anthropic Messages edge.

Node Configuration

Defaults are defined in test-suite/node/src/config.ts. The most commonly tuned variables are:

• PASS_THRESHOLD: minimum judge score, default 7.
• JUDGE_MODEL: Gemini judge model, default gemini-3-flash-preview.
• RESPONSE_TIMEOUT_MS: per-request timeout, default 60000.
• CONCURRENCY: in-file Vitest concurrency cap, default 5.
• CONCURRENCY_P95_MS: p95 latency limit for concurrency checks, default 15000.
• STREAM_TTFB_MAX_MS: streaming time-to-first-byte limit, default 5000.
• LONG_CONTEXT_TOKENS: generated long-context size, default 20000.
• CONTEXT_PROBE_TOKENS: context probe size, default 200000.
• FAIL_ON_REGRESSION: when truthy, flagged score or latency regressions fail the run.

Model ids should use the gateway’s provider/model grammar so requests resolve against the Nexus registry. For example:

MODELS=openai/gpt-5.4-mini,anthropic/claude-haiku-4-5-20251001
TIER1_MODEL=openai/gpt-5.4-mini
TIER2_MODEL=openai/gpt-5.5
TIER3_MODEL=anthropic/claude-sonnet-4-5-20250929
AUTO_MODEL=nexus/auto

If a model is reported as unknown, confirm that the reviewed registry artifact was applied to the target Postgres database and wait for the gateway’s registry poll interval. No gateway restart is required.

Compare Matrix

just compare is the developer-friendly runner for head-to-head model comparison. It runs fixed tasks against every model listed in MODELS plus nexus/auto, renders a live terminal table, and writes compare artifacts.

Examples:

just compare
just compare --task algebra
just compare --model claude
just compare --task kb-citations --concurrency 3

Artifacts include:

• test-suite/runs/compare.md
• test-suite/runs/compare-transcripts.md
• a model comparison section in test-suite/runs/report.html

Use just test-all when you specifically want the compare suite included in the unified Vitest/TAP stream.

Compatibility Harness

The compatibility harness lives in test-suite/compat. It treats official SDKs as the compatibility oracle and sends black-box requests to a running Nexus gateway.

Default OpenAI-compatible suite:

just test-compat

Equivalent direct command:

uv run --project test-suite/compat/python pytest test-suite/compat/python/tests -n auto -ra

The default suite covers OpenAI-compatible chat completions, responses, embeddings, files/vector stores, model registry shape, image paths, direct OpenAI shape comparison, pricing bands, and analytics read-back when the required services and tokens are available.

The default Python compat suite reads GET /v1/models and runs OpenAI-compatible cases against every pinned provider in that catalog, using the cheapest model each provider publishes for the capability under test. Direct OpenAI compare and native Anthropic Messages stay single-provider. Set NEXUS_COMPAT_PROVIDERS to a comma-separated allowlist to restrict the matrix.

Some slices stay gated so the default suite remains dependency-light:

• NEXUS_COMPAT_ENABLE_ANTHROPIC=0: omit Anthropic OpenAI-compatible and native Messages certification files. They otherwise collect whenever the registry lists an anthropic/ row.
• NEXUS_COMPAT_ENABLE_AGENTIC=1: collect OpenAI Agents SDK scenarios.
• NEXUS_COMPAT_ENABLE_TRACING=1: collect native OTLP trace-ingest scenarios.
• NEXUS_COMPAT_ENABLE_THINKING=1: include extended-thinking certification cases.
• NEXUS_COMPAT_ENABLE_CACHING=1: include prompt-caching certification cases when cache token usage is observable.
• NEXUS_COMPAT_REQUIRE_INGEST=1: fail analytics tests when ingest is unhealthy instead of skipping read-back assertions.

Focused targets:

just test-compat-anthropic
just test-compat-anthropic-native
just test-node-anthropic-native
just test-compat-agentic
just test-compat-tracing

The full live gateway certification target runs the major compatibility slices in order:

just test-gateway-cert

Use it before declaring provider-adapter, analytics, tracing, or public API compatibility changes complete.

Python Live CLI

The lightweight Python CLI lives in test-suite/python. It is useful when you want a quick manual probe without running the larger Vitest or SDK suites.

Examples:

just test-py compare \
  --model gpt-5.4-mini \
  --prompt "Say hello in one word." \
  --max-tokens 8

just test-py latency \
  --model gpt-5.4-mini \
  --iterations 10 \
  --warmups 2

just test-py smoke --path nexus

just test-py stream \
  --path nexus \
  --model openai/gpt-5.4-mini \
  --json

Use --json for machine-readable output. Use --allow-failures and --expect-status when intentionally probing error paths. Use --provider-options-json to verify provider-options passthrough.

Billing Reconciliation

The billing tools in test-suite/billing are opt-in admin-key scripts. They treat provider billing APIs as ground truth for pricing and cost checks.

Typical commands from test-suite/billing:

python derive_rates.py --days 31
python reconcile.py --window --days 31

Required keys depend on the provider and mode. See test-suite/billing/README.md and the billing section of .env.example before running these scripts. These commands can inspect real provider billing data, so use isolated projects/workspaces and dedicated keys when validating reconciliation changes.

Run Artifacts

Generated artifacts live under test-suite/runs. Important files include:

• current.jsonl: streaming per-record output for the current Node run.
• <timestamp>.json: structured Node run artifact.
• latest.json: copy of the latest structured Node run.
• node.tap: TAP stream from the Node Vitest run.
• report.html: HTML report across available run artifacts.
• compare.md: markdown compare table from just compare.
• compare-transcripts.md: prompt/answer transcripts from just compare.

Regenerate reports without re-running tests:

just report
just regen-compare
just regen-compare test-suite/runs/<timestamp>.json

just clean removes generated artifacts and local Node dependency/build outputs.

Debugging Failures

Start with the smallest failing slice. For Node tests, run the suite directly and enable debug traces:

just debug <suite> -- -t "<test name>"

Then inspect:

• terminal output from Vitest or pytest
• test-suite/runs/current.jsonl
• test-suite/runs/latest.json
• test-suite/runs/report.html
• test-suite/runs/compare-transcripts.md, for compare failures

Common issues:

• Missing NEXUS_API_KEY: most live requests skip or fail without a valid gateway key.
• Missing GEMINI_API_KEY: judge-backed Node tests cannot score model outputs.
• Unknown model: confirm the reviewed artifact is applied, then wait for the gateway’s Postgres registry poll.
• Ingest read-back skips: verify ingest health and set NEXUS_QUERY_TOKEN, NEXUS_SERVICE_TOKEN, or NEXUS_TRACE_QUERY_TOKEN as appropriate.
• Anthropic native Python tests skipped: confirm GET /v1/models lists an anthropic/ chat row, or use the focused just test-compat-anthropic-native target. The Node native suite still requires NEXUS_COMPAT_ENABLE_ANTHROPIC=1.
• Agentic or tracing tests not collected: use the focused targets so the required extras and gate variables are enabled.
• Docker-backed analytics failures: verify the active Docker context and that the configured Postgres container is running.

When a failure depends on a provider response, keep the run artifact. The structured records preserve the prompt, answer, returned model id, token metrics, latency, judge verdict, headers where captured, and extra diagnostic fields.

Adding Or Changing Tests

Choose the harness by the behavior you need to validate:

• Use node/tests/*.test.ts for live model behavior, routing outcomes, quality scoring, streaming, concurrency, tool calls, long-context behavior, and reportable metrics.
• Use compat/python/tests/*.py when validating public OpenAI-compatible or Anthropic-compatible API behavior through official SDKs.
• Use python/src/nexus_py_test when adding a reusable manual probe or CLI workflow.
• Use billing/*.py only for provider admin billing and cost reconciliation logic.

Prefer focused assertions with deterministic setup. Where a provider response is inherently variable, assert the protocol shape and use the judge only for tasks where semantic quality is the thing being tested.

For Node tests:

• Add a record(...) call when the result should appear in run artifacts.
• Keep prompts short unless the suite explicitly tests long context.
• Reuse shared helpers in src/client.ts, src/assertions.ts, src/expectations.ts, and src/judge.ts.
• Add new suite files to the ordering list in vitest.config.ts when their relative cost or dependency order matters.
• Run pnpm typecheck from test-suite/node after TypeScript changes.

For Python compatibility tests:

• Keep default collection focused on core OpenAI-compatible behavior.
• Gate expensive or dependency-heavy slices in conftest.py.
• Prefer SDK-level black-box calls over direct HTTP when the SDK can express the behavior.
• Use raw HTTP only for protocol details the SDK hides or prevents.
• Run uv run --project test-suite/compat/python pytest ... -ra for focused validation.

For billing tools:

• Do not require admin keys for normal test-suite imports or default commands.
• Keep live reconciliation opt-in.
• Prefer isolated provider projects/workspaces for tests that generate billable usage.

Cost And Safety Notes

Many tests call real providers and can incur cost. The compare matrix, long-context probes, creative judge-backed tasks, image tests, and billing reconciliation scripts are the most likely to create noticeable usage.

Before running broad suites:

• Narrow MODELS to the providers and model families relevant to your change.
• Lower CONCURRENCY when debugging provider rate limits.
• Use focused suite targets before just test-all or just test-gateway-cert.
• Confirm that .env points at the intended local or remote stack.
• Keep admin billing keys separate from ordinary provider inference keys.

The test-suite environment configures clients that drive Nexus. It does not configure the Nexus stack itself; stack configuration lives outside this directory.