Testing Nexus

This guide helps developers choose the right Nexus test capability for a change. It covers offline checks, local stack health checks, live gateway behavior tests, SDK compatibility certification, observability validation, provider billing reconciliation, and ad hoc probes.

For merge-request expectations, see Getting Started with Contributing. For GitLab CI tiers, security gates, release checks, and local reproduction commands, see CI/CD Pipeline.

Use just checks for every non-trivial change. It is the required offline gate and does not need a running stack or provider keys. Install pre-commit hooks with pre-commit install so fast staged checks run before each commit.

Use just status after starting just dev when you need to confirm local services and infrastructure are healthy.

Use just test when your change affects gateway behavior, routing, model invocation, streaming, tools, usage recording, or model quality assertions.

Use just test-compat when your change affects OpenAI-compatible HTTP shapes, SDK behavior, endpoint semantics, model registry output, files/vector stores, images, embeddings, responses, or analytics read-back.

Use the Anthropic-focused targets when your change affects the Anthropic-backed OpenAI-compatible adapter or the native Anthropic Messages edge.

Use just test-gateway-cert when you need the broad live gateway certification sequence across SDK compatibility, agentic clients, trace ingest, and Anthropic native behavior.

Use just test-py for quick manual probes while debugging a single model, prompt, stream, or latency path.

Use test-suite/billing only when validating cost, pricing, provider billing reconciliation, or provider admin API behavior.

Run from nexus/:

just checks

just checks runs:

• fmt-check: Rust formatting check.
• lint: cargo clippy --all-targets --all-features -- -D warnings.
• test-workspace: cargo nextest run --workspace.
• licenses: Rust, Node, and image license checks.
• scripts/check-adr-references.sh: documentation reference linting.

These checks are the default local gate before requesting review. They do not require just dev, provider API keys, or live model calls. GitLab CI also runs focused security and workspace gates; CI/CD Pipeline maps each job to a local command.

Focused offline commands:

just fmt-check
just lint
just test-workspace
just licenses
just proto-lint
just node-lint
just node-test

The console package currently has a no-op test script, so just node-test verifies the TypeScript workspace test command wiring but does not provide browser or component coverage.

Start the dev stack and verify health:

just dev
just status

just status checks the published gateway and console, internal Rust services, infrastructure containers, and dev observability readiness. Observability readiness failures are reported as warnings because the LGTM stack is diagnostic tooling.

Useful health probes:

curl -fsS http://127.0.0.1:14450/healthz
curl -fsS http://127.0.0.1:14449/healthz
curl -sS http://127.0.0.1:14450/metrics | rg vortex_service_starts_total

Related docs:

• Installation for Contributors: first-run setup, prerequisites, dev credentials, and day-to-day commands.
• Observability (Local Stack): Grafana, Prometheus, Tempo, Loki, ClickHouse, and request trace validation.
• Ports: published and internal service ports.

Rust unit and integration tests run through cargo nextest:

just test-workspace

For focused crate work, run cargo nextest or cargo test directly:

cargo nextest run -p nexus-gateway
cargo test -p nexus-cli --test cli_e2e

The compose-backed integration test target is:

just compose-it

compose-it runs the ignored nexus-integration-tests compose scenarios with the compose-it feature enabled. Use it when a Rust change needs more than ordinary unit tests but does not need the model-provider behavior suite.

The repository-level TypeScript workspace uses pnpm and Turbo:

just node-install
just node-build
just node-lint
just node-test

Use these when changing the console, shared UI package, generated protobuf TypeScript package, or workspace tooling. The console app is built with Next.js and currently does not have browser-level automated UI coverage.

Run protobuf linting when changing files under proto/:

just proto-lint

Regenerate TypeScript protobuf outputs after schema changes:

just proto

Generated files are owned by their generators. Do not hand-edit crates/nexus-proto/src/generated/*.rs or packages/proto/src/gen/**/*.ts.

The main live behavior suite is under test-suite/node in the source repository. It uses Vitest with custom reporters and writes artifacts under test-suite/runs.

Run the default live behavior suite:

just test

Run one suite:

just test smoke
just test streaming
just test tool-calling
just test failover

Run with verbose per-record traces:

just debug streaming -- -t "first token"

The Node suite validates live behavior including smoke requests, model tier quality, reasoning options, streaming, concurrency, conversations, prompt caching, structured output, tool calling, tool selection, long context, grounded citations, creative tasks, failover paths, and native Anthropic TypeScript SDK behavior.

The default just test excludes the expensive compare matrix. Use just test-all when you need every Node test file in one Vitest run.

Configuration lives in test-suite/.env.example and test-suite/node/src/config.ts in the source repository. Common requirements are NEXUS_API_KEY, GEMINI_API_KEY, MODELS, and a running gateway at INFERENCE_BASE_URL.

Use just compare when comparing model behavior across MODELS plus nexus/auto:

just compare
just compare --task algebra
just compare --model claude
just compare --task kb-citations --concurrency 3

The compare runner renders a live terminal table and writes:

• test-suite/runs/compare.md
• test-suite/runs/compare-transcripts.md
• a comparison section in test-suite/runs/report.html

Use this for model selection, routing behavior, regression investigation, and provider-adapter comparisons.

The OpenAI-compatible SDK suite lives under test-suite/compat in the source repository. It uses official SDKs as black-box clients against a running Nexus gateway.

Run the default compatibility suite:

just test-compat

It covers OpenAI-compatible endpoint families, SDK object shapes, files/vector stores, image behavior, embeddings, responses, direct OpenAI shape comparisons, pricing bands, model registry output, and analytics read-back when the required services and tokens are available.

Focused certification targets:

just test-compat-anthropic
just test-compat-anthropic-native
just test-node-anthropic-native
just test-compat-agentic
just test-compat-tracing

Use these when changing provider adapters, SDK-compatible wire shapes, native Anthropic behavior, OpenAI Agents SDK scenarios, or trace ingest behavior.

The broad live certification target is:

just test-gateway-cert

It runs stack status, the default OpenAI compatibility suite, agentic compatibility, trace ingest compatibility, Anthropic OpenAI-compatible certification, native Anthropic Python certification, and native Anthropic TypeScript certification.

The lightweight Python CLI lives under test-suite/python in the source repository. It is useful for quick manual checks while debugging.

Examples:

just test-py compare \
  --model gpt-5.4-mini \
  --prompt "Say hello in one word." \
  --max-tokens 8

just test-py latency \
  --model gpt-5.4-mini \
  --iterations 10 \
  --warmups 2

just test-py smoke --path nexus

just test-py stream \
  --path nexus \
  --model openai/gpt-5.4-mini \
  --json

Use --json when feeding results to another tool. Use --expect-status and --allow-failures for intentional error-path probes.

The adopter E2E scripts live under test-suite/adopter in the source repository. They exercise CLI-driven onboarding and provider-key flows against a running stack.

Run the BYOK adopter path:

just test-adopter-e2e

Run the managed pass-through billing adopter path:

just test-adopter-ptb-e2e

Use these when changing CLI onboarding, credential management, project setup, provider key loading, or pass-through billing request setup.

Use Observability (Local Stack) for manual validation in Grafana, Prometheus, Tempo, Loki, and ClickHouse.

For automated trace-ingest certification:

just test-compat-tracing

Trace read-back assertions require a query token with trace-read permission. Without that token, the write path is still exercised and read-back assertions skip cleanly.

For gateway request observability, a good manual loop is:

just dev
just status
just test smoke

Then open Grafana at http://127.0.0.1:14470 and inspect recent gateway metrics, logs, and traces.

Billing reconciliation tools live under test-suite/billing in the source repository. They use provider admin APIs and are opt-in because they inspect real billing data and can drive live provider traffic.

Run the pricing checks directly from nexus/:

cargo test -p nexus-registry --test pricing_contracts
cargo test -p nexus-registry live_ --lib -- --ignored
python3 -m unittest discover -s test-suite/billing -p 'test_*.py'
python3 test-suite/billing/validate_pricing.py --provider both --bands both
python3 test-suite/billing/reconcile.py --sidecar test-suite/billing/reports/<run>.requests.json

The offline gate uses fixtures only. The live-source gate has no inference cost and compares current public documents to independent curated contracts; ordinary test runs show its tests as ignored. The gateway gate (validate_pricing.py) routes live requests through Nexus API keys supplied by the environment (NEXUS_API_KEY, or the NEXUS_ISO_OPENAI_API_KEY / NEXUS_ISO_ANTHROPIC_API_KEY pair with --isolated) and reads exact expected rates and thresholds from curated contracts. Settlement (reconcile.py, opt-in via NEXUS_BILLING_RECONCILE=1) compares provider-isolated totals and exits with 0 pass, 1 fail, 2 pending, or 3 not configured.

Use these when changing pricing registry logic, cost attribution, billing settlement, provider cost API integration, or pass-through billing reconciliation.

Live Node runs write artifacts under test-suite/runs:

• latest.json: latest structured run artifact.
• <timestamp>.json: timestamped structured run artifact.
• current.jsonl: per-record stream for the current run.
• node.tap: TAP output from Vitest.
• report.html: generated HTML report.
• compare.md: model comparison table.
• compare-transcripts.md: prompts and outputs from compare runs.

Regenerate reports from existing artifacts:

just report
just regen-compare

Keep artifacts when debugging provider-specific failures. They include prompts, answers, returned model ids, latency and token metrics, judge verdicts, and extra diagnostic fields captured by the suite.

For Rust-only library changes, run just checks and a focused cargo nextest run -p <crate> if the crate has meaningful local tests.

For CLI changes, run just checks, cargo test -p nexus-cli --test cli_e2e, and an adopter E2E when the change affects live onboarding behavior.

For gateway request-path changes, run just checks, just status, just test, and the focused compatibility target for any public API shape touched by the change.

For provider-adapter changes, run just checks, focused Node behavior suites, just test-compat, and the provider-specific certification target.

For OpenAI-compatible API changes, run just checks, just test-compat, and update the compatibility matrix when the supported surface changes.

For Anthropic native edge changes, run just test-compat-anthropic-native and just test-node-anthropic-native.

For analytics, ingest, query, or trace changes, run just checks, just test-compat, just test-compat-tracing, and inspect Grafana or ClickHouse when diagnosing data flow.

For console-only changes, run just node-build and just node-lint. Use the running console manually for UI flows because browser-level automated coverage is not present today.

For pricing or billing changes, run the focused unit tests, the relevant live behavior suite, and the billing reconciliation tools only with appropriate admin keys and isolated provider projects or workspaces.

Live suites can call real model providers and consume API credits. The compare matrix, long-context tasks, creative judge-backed tasks, image tests, trace certification, and billing reconciliation scripts are the most expensive or sensitive paths.

Before broad live runs:

• Confirm .env and test-suite/.env point at the intended stack.
• Narrow MODELS to the providers and model families relevant to the change.
• Lower CONCURRENCY if you are debugging provider rate limits.
• Keep provider inference keys separate from provider admin billing keys.
• Keep generated artifacts out of commits unless a report is intentionally part of a reviewed change.

Use just clean to remove generated test artifacts and local Node outputs when you want a clean workspace.