Testing Nexus

This guide helps developers choose the right Nexus test capability for a change. It covers offline checks, local stack health checks, live gateway behavior tests, SDK compatibility certification, observability validation, provider billing reconciliation, and ad hoc probes.

See the test-suite reference for harness setup, the contributor guide for review expectations, and the pipeline guide for job-level checks.

Choose the right checks

Use just checks for every non-trivial change. It is the required offline gate and does not need a running stack or provider keys.

Then add the checks that match your change. Run these commands from the Nexus source checkout.

What you need to check	Command or guide
Local services and infrastructure after starting just dev	just status
Gateway routing, streaming, tools, usage recording, or model behavior	just test
OpenAI-compatible requests, client-library behavior, model listings, or analytics read-back	just test-compat
Anthropic-backed OpenAI adapter	just test-compat-anthropic
Native Anthropic Messages edge	just test-compat-anthropic-native and just test-node-anthropic-native
Broad gateway certification, including agentic clients and trace ingest	just test-gateway-cert
A single model, prompt, stream, or latency issue	just test-py
Console browser behavior	Console browser tests
Cost, pricing, or provider billing reconciliation	Billing reconciliation

Live gateway suites require the relevant services and credentials. See each suite’s setup instructions before running it.

Required offline checks

Run from nexus/:

just checks

just checks runs:

• fmt-check: Rust formatting check.
• lint: cargo clippy --all-targets --all-features -- -D warnings.
• test-workspace: cargo nextest run --workspace.
• licenses: Rust, Node, and image license checks.
• scripts/check-adr-references.sh: documentation reference linting.

Local hooks

Install hooks with pre-commit install. The repository’s default_install_hook_types installs both pre-commit and pre-push:

• Pre-commit: Secret checks, formatters (cargo fmt on staged Rust files, Prettier, and shfmt), and lightweight linters.
• Pre-push: Full-workspace Clippy, using the same command as just lint and the pipeline’s rust-lint job.

Focused commands

These checks are the default local gate before requesting review. They do not require just dev, provider API keys, or live model calls. GitLab CI also runs focused security and workspace gates; ci.md maps each job to a local command.

Focused offline commands:

just fmt-check
just lint
just test-workspace
just licenses
just proto-lint
just node-lint
just node-test

Run the separate console browser suite when verifying browser behavior; do not treat the workspace test command as a substitute.

Local Stack Health

Start the dev stack and verify health:

just dev
just status

just status checks the published gateway and console, internal Rust services, infrastructure containers, and dev observability readiness. Observability readiness failures are reported as warnings because the LGTM stack is diagnostic tooling.

Useful health probes:

curl -fsS http://127.0.0.1:14450/healthz
curl -fsS http://127.0.0.1:14449/healthz
curl -sS http://127.0.0.1:14450/metrics | rg vortex_service_starts_total

Related docs:

• onboarding.md: first-run setup, prerequisites, dev credentials, and day-to-day commands.
• end-to-end.md: clean local flow from signup/bootstrap to a proxied chat request.
• observability.md: Grafana, Prometheus, Tempo, Loki, ClickHouse, and request trace validation.
• ../reference/ports.md: published and internal service ports.

Rust Tests

Rust unit and integration tests run through cargo nextest:

just test-workspace

For focused crate work, run cargo nextest or cargo test directly:

cargo nextest run -p nexus-gateway
cargo test -p nexus-cli --test cli_e2e

For registry artifact generation and import:

cargo test -p nexus-registry-artifact
cargo test -p nexus-registry-generator
cargo test -p nexus-registry --test import_change_set_migration

These checks are offline. The ignored artifact_import_postgres integration test additionally requires a migrated Postgres database and verifies deterministic prepare, approval binding, atomic apply, quarantine recovery, pinned pricing, and full-replace retirement. Point TEST_DATABASE_URL at a throwaway database, never at a dev registry you care about: the suite imports its own fixture artifacts, and because an import replaces the whole provider catalog, it will overwrite that provider’s rows and leave fixture endpoints behind.

The compose-backed integration test target is:

just compose-it

compose-it runs the ignored nexus-integration-tests compose scenarios with the compose-it feature enabled. Use it when a Rust change needs more than ordinary unit tests but does not need the model-provider behavior suite.

The nexus-cache end-to-end cache-lookup tests in integration-tests/tests/first_proxied_request.rs are additionally gated behind the redis-it cargo feature (as well as #[ignore]), since they need a reachable Redis/Valkey at NEXUS_REDIS_URL (default redis://127.0.0.1:6379). The stock stack keeps Valkey on the Compose network only, so just redis-it layers deploy/compose/docker-compose.redis-it.yml over it to publish that port on loopback, brings the service up, and then runs the tests:

just redis-it

TypeScript And Console Checks

The repository-level TypeScript workspace uses pnpm and Turbo:

just node-install
just node-build
just node-lint
just node-test

Use these when changing the console, shared UI package, generated protobuf TypeScript package, or workspace tooling. The Next.js console has a separate Playwright suite with a development server and an isolated fake backend. See Console browser tests for setup and focused commands.

Protobuf And Generated Types

Run protobuf linting when changing files under proto/:

just proto-lint

Regenerate TypeScript protobuf outputs after schema changes:

just proto

Generated files are owned by their generators. Do not hand-edit crates/nexus-proto/src/generated/*.rs or packages/proto/src/gen/**/*.ts.

Live Gateway Behavior Suite

The main live behavior suite is under ../../test-suite/node. It uses Vitest with custom reporters and writes artifacts under test-suite/runs.

Run the default live behavior suite:

just test

Run one suite:

just test smoke
just test streaming
just test tool-calling
just test failover

Run with verbose per-record traces:

just debug streaming -- -t "first token"

The Node suite validates live behavior including smoke requests, model tier quality, reasoning options, streaming, concurrency, conversations, prompt caching, structured output, tool calling, tool selection, long context, grounded citations, creative tasks, failover paths, and native Anthropic TypeScript SDK behavior.

The default just test excludes the expensive compare matrix. Use just test-all when you need every Node test file in one Vitest run.

Configuration lives in ../../test-suite/.env.example and ../../test-suite/node/src/config.ts. Common requirements are NEXUS_API_KEY, MODELS, and a running gateway at INFERENCE_BASE_URL. GEMINI_API_KEY is needed only by tests that call the Gemini judge; without it those tests fail with a missing-variable error, and the artifact reporter still writes the run artifacts.

Model Compare Matrix

Use just compare when comparing model behavior across MODELS plus nexus/auto:

just compare
just compare --task algebra
just compare --model claude
just compare --task kb-citations --concurrency 3

The compare runner renders a live terminal table and writes:

• test-suite/runs/compare.md
• test-suite/runs/compare-transcripts.md
• a comparison section in test-suite/runs/report.html

Use this for model selection, routing behavior, regression investigation, and provider-adapter comparisons.

SDK Compatibility Certification

The OpenAI-compatible SDK suite lives under ../../test-suite/compat. It uses official SDKs as black-box clients against a running Nexus gateway.

Run the default compatibility suite:

just test-compat

It covers OpenAI-compatible endpoint families, SDK object shapes, files/vector stores, image behavior, embeddings, responses, direct OpenAI shape comparisons, pricing bands, model registry output, and analytics read-back when the required services and tokens are available. The supported endpoint families are summarized in ../compatibility-matrix.md.

Many tests run once per catalog provider. Such a test sends its requests, and reads usage back, with NEXUS_COMPAT_KEY_<PROVIDER> when that variable is set (the provider prefix upper-cased, - as _, e.g. NEXUS_COMPAT_KEY_BEDROCK_MANTLE_COMMERCIAL), and with NEXUS_API_KEY otherwise. When the gateway answers a request with no_provider_key, the test is skipped with the gateway’s message only if no NEXUS_COMPAT_KEY_<PROVIDER> is set for that provider (the test’s provider, or else the provider named in the error or the request’s model); with that key set, or with the provider unknown, the test fails. Arguments after the recipe name go to pytest, e.g. just test-compat -k anthropic_responses.

Focused certification targets:

just test-compat-anthropic
just test-compat-anthropic-native
just test-node-anthropic-native
just test-compat-agentic
just test-compat-tracing

Use these when changing provider adapters, SDK-compatible wire shapes, native Anthropic behavior, OpenAI Agents SDK scenarios, or trace ingest behavior.

The broad live certification target is:

just test-gateway-cert

It runs stack status, the default OpenAI compatibility suite, agentic compatibility, trace ingest compatibility, Anthropic OpenAI-compatible certification, native Anthropic Python certification, and native Anthropic TypeScript certification.

Coding-Harness Gate

The coding-harness tools live under ../../test-suite/harness; ../../test-suite/harness/README.md is the full reference. Use them when a change affects how the gateway serves real coding clients: chat tool calling, the Responses bridge, the Anthropic Messages edge, streaming, or cross-provider session history.

Run every registered job against the dev stack:

just harness-gauntlet all
just harness-gauntlet all --only bedrock-mantle --parallel 4
just harness-gauntlet --list

Each job drives a pinned harness through a two-turn session in a fresh fixture repository and verifies the code it leaves behind. The pinned versions are OpenCode 1.18.33, Codex CLI 0.161.0, and Claude Code 2.1.259; each is installed once into ${XDG_CACHE_HOME:-~/.cache}/nexus-harness. The job list covers all three harnesses on default client settings, one model per family per lane (OpenAI, Anthropic, Azure Foundry Gov, Bedrock, Bedrock GovCloud, Bedrock Mantle on both accounts, and xAI), plus one mid-session switch into each lane.

Keys:

• NEXUS_API_KEY is the default gateway key.
• NEXUS_HARNESS_KEY_<PROVIDER> overrides it for one provider prefix, e.g. NEXUS_HARNESS_KEY_BEDROCK_MANTLE and NEXUS_HARNESS_KEY_BEDROCK_MANTLE_COMMERCIAL. A switch job uses each model’s own key.
• The recipe sources .env (or NEXUS_ENV_FILE); set NEXUS_ENV_FILE=/dev/null to use only the shell’s keys.

If GET /v1/models fails for a key, the run stops with exit 2. A model with no key set is skipped; a model not listed for a key that is set is unavailable and fails the run unless --allow-skip is given.

Each attempt is classified:

• pass: both turns left working code. If a turn also printed a gateway error (for example a dropped stream the harness reconnected), the job is pass_with_gateway_errors and is listed as intermittent without failing the run.
• gateway_failure: code is broken and a turn timed out, printed a gateway error (an HTTP 4xx/5xx error, invalid request body, API Error, unexpected status, no candidate endpoint, stream disconnected, or a connection failure such as ECONNREFUSED or fetch failed), or exited non-zero without any recognised error.
• model_quality: code is broken with no gateway error. A harness’s failed to parse function arguments counts here, as does an upstream Expecting ',' delimiter or Expecting value rejection when the same attempt also shows failed to parse function arguments; lines with a 5xx status never do.

Failed attempts of either kind are re-run up to three times in total (--attempts). A gateway failure that passes on a later attempt is listed as intermittent rather than hidden, and a job whose attempts all fail as model quality is listed by harness and model without failing the run. The run exits non-zero when a job ends in a gateway failure or a model is unavailable.

Combinations with a known capability gap are registered as expected failures, each with a plain reason, a tracking id, and a pattern its upstream error must match:

• Codex on the Bedrock Mantle gpt-oss rows and xAI, which serve the Responses API natively and reject the namespace tool type Codex sends by default (namespace-passthrough).
• Claude Code on Mantle devstral, which rejects system messages placed after tool results in the middle of a conversation (mistral-mid-conversation-system).
• Claude Code on Mantle gemma, which rejects Claude Code’s tool schemas that use the propertyNames keyword (gemma-tool-schema-keywords).
• OpenCode on Nova Pro (Bedrock and Bedrock GovCloud), a model limitation rather than a gateway failure: Nova Pro can keep misusing OpenCode’s exact-match edit tool while reacting to every tool result, until the turn times out (nova-edit-loop). Only a timeout counts as this expected failure.

An expected job that fails with a different error, or a switch job whose first turn fails, is a gateway failure. An expected failure that passes is reported as an unexpected pass so its entry can be removed.

Each run writes results.json (every attempt, with its log file), summary.md, and one log per attempt under test-suite/runs/harness/matrix-<UTC timestamp>/.

The path-stress workload sends harness-shaped requests straight to the gateway for every scenario and model, and compares the verdicts with the committed baseline:

just harness-path-stress --baseline test-suite/harness/baseline.json
python3 test-suite/harness/gauntlet.py out.json --write-baseline test-suite/harness/baseline.json

A call that passed in the baseline and fails now is re-run once and counts as a regression only if it fails again; regressions exit 1, and improvements are listed.

Python Live CLI

The lightweight Python CLI lives under ../../test-suite/python. It is useful for quick manual checks while debugging.

Examples:

just test-py compare \
  --model gpt-5.4-mini \
  --prompt "Say hello in one word." \
  --max-tokens 8

just test-py latency \
  --model gpt-5.4-mini \
  --iterations 10 \
  --warmups 2

just test-py smoke --path nexus

just test-py stream \
  --path nexus \
  --model openai/gpt-5.4-mini \
  --json

Use --json when feeding results to another tool. Use --expect-status and --allow-failures for intentional error-path probes.

Adopter End-to-End Flows

The adopter E2E scripts live under ../../test-suite/adopter. They exercise CLI-driven onboarding and provider-key flows against a running stack.

Run the BYOK adopter path:

just test-adopter-e2e

Run the managed pass-through billing adopter path:

just test-adopter-ptb-e2e

Use these when changing CLI onboarding, credential management, project setup, provider key loading, or pass-through billing request setup.

Observability And Trace Validation

Use observability.md for manual validation in Grafana, Prometheus, Tempo, Loki, and ClickHouse.

For automated trace-ingest certification:

just test-compat-tracing

Trace read-back assertions require a query token with trace-read permission. Without that token, the write path is still exercised and read-back assertions skip cleanly.

For gateway request observability, a good manual loop is:

just dev
just status
just test smoke

Then open Grafana at http://127.0.0.1:14470 and inspect recent gateway metrics, logs, and traces.

Billing And Pricing Validation

Billing reconciliation tools live under ../../test-suite/billing. They use provider admin APIs and are opt-in because they inspect real billing data and can drive live provider traffic.

Run the pricing checks directly from nexus/:

cargo test -p nexus-registry-artifact
cargo test -p nexus-registry-generator
cargo test -p nexus-registry --test import_change_set_migration
python3 -m unittest discover -s test-suite/billing -p 'test_*.py'
python3 test-suite/billing/validate_pricing.py --provider both --bands both
python3 test-suite/billing/reconcile.py --sidecar test-suite/billing/reports/<run>.requests.json

The Rust gates validate the canonical artifact contract, fixture conversion, and persisted schema transition without provider access. The gateway gate (validate_pricing.py) routes live requests through Nexus API keys supplied by the environment (NEXUS_API_KEY, or the NEXUS_ISO_OPENAI_API_KEY / NEXUS_ISO_ANTHROPIC_API_KEY pair with --isolated) and reads exact expected rates and thresholds from curated contracts. Settlement (reconcile.py, opt-in via NEXUS_BILLING_RECONCILE=1) compares provider-isolated totals and exits with 0 pass, 1 fail, 2 pending, or 3 not configured.

Use these when changing pricing registry logic, cost attribution, billing settlement, provider cost API integration, or pass-through billing reconciliation. ../../test-suite/billing/README.md is the authoritative reference for these commands’ flags and credentials; read it before running them, and defer to it if it ever disagrees with this page.

Test Artifacts

Live Node runs write artifacts under test-suite/runs:

• latest.json: latest structured run artifact.
• <timestamp>.json: timestamped structured run artifact.
• current.jsonl: per-record stream for the current run.
• node.tap: TAP output from Vitest.
• report.html: generated HTML report.
• compare.md: model comparison table.
• compare-transcripts.md: prompts and outputs from compare runs.

Regenerate reports from existing artifacts:

just report
just regen-compare

Keep artifacts when debugging provider-specific failures. They include prompts, answers, returned model ids, latency and token metrics, judge verdicts, and extra diagnostic fields captured by the suite.

Choosing A Test Plan For A Change

For Rust-only library changes, run just checks and a focused cargo nextest run -p <crate> if the crate has meaningful local tests.

For CLI changes, run just checks, cargo test -p nexus-cli --test cli_e2e, and an adopter E2E when the change affects live onboarding behavior.

For gateway request-path changes, run just checks, just status, just test, and the focused compatibility target for any public API shape touched by the change.

For provider-adapter changes, run just checks, focused Node behavior suites, just test-compat, and the provider-specific certification target.

For OpenAI-compatible API changes, run just checks, just test-compat, and update ../compatibility-matrix.md when the supported surface changes.

For Anthropic native edge changes, run just test-compat-anthropic-native and just test-node-anthropic-native.

For analytics, ingest, query, or trace changes, run just checks, just test-compat, just test-compat-tracing, and inspect Grafana or ClickHouse when diagnosing data flow.

For console-only changes, run just node-build and just node-lint. Use the running console manually for UI flows because browser-level automated coverage is not present today.

For pricing or billing changes, run the focused unit tests, the relevant live behavior suite, and the billing reconciliation tools only with appropriate admin keys and isolated provider projects or workspaces.

Cost And Secret Safety

Live suites can call real model providers and consume API credits. The compare matrix, long-context tasks, creative judge-backed tasks, image tests, trace certification, and billing reconciliation scripts are the most expensive or sensitive paths.

Before broad live runs:

• Confirm .env and test-suite/.env point at the intended stack.
• Narrow MODELS to the providers and model families relevant to the change.
• Lower CONCURRENCY if you are debugging provider rate limits.
• Keep provider inference keys separate from provider admin billing keys.
• Keep generated registry bundles and test artifacts out of commits. Release automation retains designated generated evidence in CI/package storage rather than in the source tree.

Use just clean to remove generated test artifacts and local Node outputs when you want a clean workspace.