Testing Nexus

Guide for choosing the right Nexus test capability

Last updated on: September 21, 2026

This guide helps developers choose the right Nexus test capability for a change. It covers offline checks, local stack health checks, live gateway behavior tests, SDK compatibility certification, observability validation, provider billing reconciliation, and ad hoc probes.

For the full reference for live and compatibility harnesses, see ../../test-suite/README.md. For merge-request expectations, see ../../CONTRIBUTING.md. For GitLab CI tiers, security gates, release checks, and local reproduction commands, see ci.md.

Quick Decision Guide

Use just checks for every non-trivial change. It is the required offline gate and does not need a running stack or provider keys. Install hooks with pre-commit install and pre-commit install --hook-type pre-push so staged checks run on commit and the clippy gate runs on push. Rust commits run cargo fmt --all; Rust pushes run the same clippy gate as CI (--all-targets --all-features -- -D warnings), after applying machine-applicable clippy fixes.

Use just status after starting just dev when you need to confirm local services and infrastructure are healthy.

Use just test when your change affects gateway behavior, routing, model invocation, streaming, tools, usage recording, or model quality assertions.

Use just test-compat when your change affects OpenAI-compatible HTTP shapes, SDK behavior, endpoint semantics, model registry output, files/vector stores, images, embeddings, responses, or analytics read-back.

Use the Anthropic-focused targets when your change affects the Anthropic-backed OpenAI-compatible adapter or the native Anthropic Messages edge.

Use just test-gateway-cert when you need the broad live gateway certification sequence across SDK compatibility, agentic clients, trace ingest, and Anthropic native behavior.

Use just test-py for quick manual probes while debugging a single model, prompt, stream, or latency path.

Use test-suite/billing only when validating cost, pricing, provider billing reconciliation, or provider admin API behavior.

Required Offline Checks

Run from nexus/:

just checks

just checks runs:

fmt-check: Rust formatting check.
lint: cargo clippy --all-targets --all-features -- -D warnings.
test-workspace: cargo nextest run --workspace.
licenses: Rust, Node, and image license checks.
scripts/check-adr-references.sh: documentation reference linting.

These checks are the default local gate before requesting review. They do not require just dev, provider API keys, or live model calls. GitLab CI also runs focused security and workspace gates; ci.md maps each job to a local command.

Focused offline commands:

just fmt-check
just lint
just test-workspace
just licenses
just proto-lint
just node-lint
just node-test

The console package currently has a no-op test script, so just node-test verifies the TypeScript workspace test command wiring but does not provide browser or component coverage.

Local Stack Health

Start the dev stack and verify health:

just dev
just status

just status checks the published gateway and console, internal Rust services, infrastructure containers, and dev observability readiness. Observability readiness failures are reported as warnings because the LGTM stack is diagnostic tooling.

Useful health probes:

curl -fsS http://127.0.0.1:14450/healthz
curl -fsS http://127.0.0.1:14449/healthz
curl -sS http://127.0.0.1:14450/metrics | rg vortex_service_starts_total

Related docs:

onboarding.md: first-run setup, prerequisites, dev credentials, and day-to-day commands.
end-to-end.md: clean local flow from signup/bootstrap to a proxied chat request.
observability.md: Grafana, Prometheus, Tempo, Loki, ClickHouse, and request trace validation.
../reference/ports.md: published and internal service ports.

Rust Tests

Rust unit and integration tests run through cargo nextest:

just test-workspace

For focused crate work, run cargo nextest or cargo test directly:

cargo nextest run -p nexus-gateway
cargo test -p nexus-cli --test cli_e2e

For registry artifact generation and import:

cargo test -p nexus-registry-artifact
cargo test -p nexus-registry-generator
cargo test -p nexus-registry --test import_change_set_migration

These checks are offline. The ignored artifact_import_postgres integration test additionally requires a migrated Postgres database and verifies deterministic prepare, approval binding, atomic apply, quarantine recovery, pinned pricing, and full-replace retirement. Point TEST_DATABASE_URL at a throwaway database, never at a dev registry you care about: the suite imports its own fixture artifacts, and because an import replaces the whole provider catalog, it will overwrite that provider’s rows and leave fixture endpoints behind.

The compose-backed integration test target is:

just compose-it

compose-it runs the ignored nexus-integration-tests compose scenarios with the compose-it feature enabled. Use it when a Rust change needs more than ordinary unit tests but does not need the model-provider behavior suite.

The nexus-cache end-to-end cache-lookup tests in integration-tests/tests/first_proxied_request.rs are additionally gated behind the redis-it cargo feature (as well as #[ignore]), since they need a reachable Redis/Valkey at NEXUS_REDIS_URL (default redis://127.0.0.1:6379). The stock stack keeps Valkey on the Compose network only, so just redis-it layers deploy/compose/docker-compose.redis-it.yml over it to publish that port on loopback, brings the service up, and then runs the tests:

just redis-it

TypeScript And Console Checks

The repository-level TypeScript workspace uses pnpm and Turbo:

just node-install
just node-build
just node-lint
just node-test

Use these when changing the console, shared UI package, generated protobuf TypeScript package, or workspace tooling. The console app is built with Next.js and currently does not have browser-level automated UI coverage.

Protobuf And Generated Types

Run protobuf linting when changing files under proto/:

just proto-lint

Regenerate TypeScript protobuf outputs after schema changes:

just proto

Generated files are owned by their generators. Do not hand-edit crates/nexus-proto/src/generated/*.rs or packages/proto/src/gen/**/*.ts.

Live Gateway Behavior Suite

The main live behavior suite is under ../../test-suite/node. It uses Vitest with custom reporters and writes artifacts under test-suite/runs.

Run the default live behavior suite:

just test

Run one suite:

just test smoke
just test streaming
just test tool-calling
just test failover

Run with verbose per-record traces:

just debug streaming -- -t "first token"

The Node suite validates live behavior including smoke requests, model tier quality, reasoning options, streaming, concurrency, conversations, prompt caching, structured output, tool calling, tool selection, long context, grounded citations, creative tasks, failover paths, and native Anthropic TypeScript SDK behavior.

The default just test excludes the expensive compare matrix. Use just test-all when you need every Node test file in one Vitest run.

Configuration lives in ../../test-suite/.env.example and ../../test-suite/node/src/config.ts. Common requirements are NEXUS_API_KEY, GEMINI_API_KEY, MODELS, and a running gateway at INFERENCE_BASE_URL.

Model Compare Matrix

Use just compare when comparing model behavior across MODELS plus nexus/auto:

just compare
just compare --task algebra
just compare --model claude
just compare --task kb-citations --concurrency 3

The compare runner renders a live terminal table and writes:

test-suite/runs/compare.md
test-suite/runs/compare-transcripts.md
a comparison section in test-suite/runs/report.html

Use this for model selection, routing behavior, regression investigation, and provider-adapter comparisons.

SDK Compatibility Certification

The OpenAI-compatible SDK suite lives under ../../test-suite/compat. It uses official SDKs as black-box clients against a running Nexus gateway.

Run the default compatibility suite:

just test-compat

It covers OpenAI-compatible endpoint families, SDK object shapes, files/vector stores, image behavior, embeddings, responses, direct OpenAI shape comparisons, pricing bands, model registry output, and analytics read-back when the required services and tokens are available. The supported endpoint families are summarized in ../compatibility-matrix.md.

Focused certification targets:

just test-compat-anthropic
just test-compat-anthropic-native
just test-node-anthropic-native
just test-compat-agentic
just test-compat-tracing

Use these when changing provider adapters, SDK-compatible wire shapes, native Anthropic behavior, OpenAI Agents SDK scenarios, or trace ingest behavior.

The broad live certification target is:

just test-gateway-cert

It runs stack status, the default OpenAI compatibility suite, agentic compatibility, trace ingest compatibility, Anthropic OpenAI-compatible certification, native Anthropic Python certification, and native Anthropic TypeScript certification.

Python Live CLI

The lightweight Python CLI lives under ../../test-suite/python. It is useful for quick manual checks while debugging.

Examples:

just test-py compare \
  --model gpt-5.4-mini \
  --prompt "Say hello in one word." \
  --max-tokens 8

just test-py latency \
  --model gpt-5.4-mini \
  --iterations 10 \
  --warmups 2

just test-py smoke --path nexus

just test-py stream \
  --path nexus \
  --model openai/gpt-5.4-mini \
  --json

Use --json when feeding results to another tool. Use --expect-status and --allow-failures for intentional error-path probes.

Adopter End-to-End Flows

The adopter E2E scripts live under ../../test-suite/adopter. They exercise CLI-driven onboarding and provider-key flows against a running stack.

Run the BYOK adopter path:

just test-adopter-e2e

Run the managed pass-through billing adopter path:

just test-adopter-ptb-e2e

Use these when changing CLI onboarding, credential management, project setup, provider key loading, or pass-through billing request setup.

Observability And Trace Validation

Use observability.md for manual validation in Grafana, Prometheus, Tempo, Loki, and ClickHouse.

For automated trace-ingest certification:

just test-compat-tracing

Trace read-back assertions require a query token with trace-read permission. Without that token, the write path is still exercised and read-back assertions skip cleanly.

For gateway request observability, a good manual loop is:

just dev
just status
just test smoke

Then open Grafana at http://127.0.0.1:14470 and inspect recent gateway metrics, logs, and traces.

Billing And Pricing Validation

Billing reconciliation tools live under ../../test-suite/billing. They use provider admin APIs and are opt-in because they inspect real billing data and can drive live provider traffic.

Run the pricing checks directly from nexus/:

cargo test -p nexus-registry-artifact
cargo test -p nexus-registry-generator
cargo test -p nexus-registry --test import_change_set_migration
python3 -m unittest discover -s test-suite/billing -p 'test_*.py'
python3 test-suite/billing/validate_pricing.py --provider both --bands both
python3 test-suite/billing/reconcile.py --sidecar test-suite/billing/reports/<run>.requests.json

The Rust gates validate the canonical artifact contract, fixture conversion, and persisted schema transition without provider access. The gateway gate (validate_pricing.py) routes live requests through Nexus API keys supplied by the environment (NEXUS_API_KEY, or the NEXUS_ISO_OPENAI_API_KEY / NEXUS_ISO_ANTHROPIC_API_KEY pair with --isolated) and reads exact expected rates and thresholds from curated contracts. Settlement (reconcile.py, opt-in via NEXUS_BILLING_RECONCILE=1) compares provider-isolated totals and exits with 0 pass, 1 fail, 2 pending, or 3 not configured.

Use these when changing pricing registry logic, cost attribution, billing settlement, provider cost API integration, or pass-through billing reconciliation. ../../test-suite/billing/README.md is the authoritative reference for these commands’ flags and credentials; read it before running them, and defer to it if it ever disagrees with this page.

Test Artifacts

Live Node runs write artifacts under test-suite/runs:

latest.json: latest structured run artifact.
<timestamp>.json: timestamped structured run artifact.
current.jsonl: per-record stream for the current run.
node.tap: TAP output from Vitest.
report.html: generated HTML report.
compare.md: model comparison table.
compare-transcripts.md: prompts and outputs from compare runs.

Regenerate reports from existing artifacts:

just report
just regen-compare

Keep artifacts when debugging provider-specific failures. They include prompts, answers, returned model ids, latency and token metrics, judge verdicts, and extra diagnostic fields captured by the suite.

Choosing A Test Plan For A Change

For Rust-only library changes, run just checks and a focused cargo nextest run -p <crate> if the crate has meaningful local tests.

For CLI changes, run just checks, cargo test -p nexus-cli --test cli_e2e, and an adopter E2E when the change affects live onboarding behavior.

For gateway request-path changes, run just checks, just status, just test, and the focused compatibility target for any public API shape touched by the change.

For provider-adapter changes, run just checks, focused Node behavior suites, just test-compat, and the provider-specific certification target.

For OpenAI-compatible API changes, run just checks, just test-compat, and update ../compatibility-matrix.md when the supported surface changes.

For Anthropic native edge changes, run just test-compat-anthropic-native and just test-node-anthropic-native.

For analytics, ingest, query, or trace changes, run just checks, just test-compat, just test-compat-tracing, and inspect Grafana or ClickHouse when diagnosing data flow.

For console-only changes, run just node-build and just node-lint. Use the running console manually for UI flows because browser-level automated coverage is not present today.

For pricing or billing changes, run the focused unit tests, the relevant live behavior suite, and the billing reconciliation tools only with appropriate admin keys and isolated provider projects or workspaces.

Cost And Secret Safety

Live suites can call real model providers and consume API credits. The compare matrix, long-context tasks, creative judge-backed tasks, image tests, trace certification, and billing reconciliation scripts are the most expensive or sensitive paths.

Before broad live runs:

Confirm .env and test-suite/.env point at the intended stack.
Narrow MODELS to the providers and model families relevant to the change.
Lower CONCURRENCY if you are debugging provider rate limits.
Keep provider inference keys separate from provider admin billing keys.
Keep generated registry bundles and test artifacts out of commits. Release automation retains designated generated evidence in CI/package storage rather than in the source tree.

Use just clean to remove generated test artifacts and local Node outputs when you want a clean workspace.