Python live tests

uv-managed Python CLI for live checks against OpenAI-compatible providers and Nexus. The package loads .env with python-dotenv before reading environment variables and uses Typer + Rich for subcommands and terminal reports.

Setup

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export NEXUS_API_KEY=nxs_...

Defaults:

• Direct provider: PROVIDER_BASE_URL=https://api.openai.com/v1
• Anthropic direct provider: ANTHROPIC_BASE_URL=https://api.anthropic.com/v1
• Nexus: NEXUS_BASE_URL=http://127.0.0.1:14450/v1
• Provider key: PROVIDER_API_KEY or OPENAI_API_KEY
• Anthropic key: ANTHROPIC_API_KEY
• Nexus key: NEXUS_API_KEY, ROUTER_API_KEY, or TOKEN

Output token field (max_tokens vs max_completion_tokens)

OpenAI rejects max_tokens for some model families and expects max_completion_tokens instead. The harness defaults to auto: it picks max_completion_tokens for GPT-5 / o1 / o3 / o4 / GPT-4.1 style ids (by name), and max_tokens for older chat models (e.g. gpt-4o-mini). Override with:

• CLI: --output-cap max_tokens or --output-cap max_completion_tokens
• Env: NEXUS_PY_OUTPUT_CAP or OUTPUT_CAP

Nexus resolves the model from its Postgres-backed registry and refreshes its in-memory snapshot on a periodic poll. If compare shows unknown model, confirm that the reviewed registry artifact was applied to this environment and wait for the poll interval; a gateway restart is not required.

Modes

Compare one direct provider request with one Nexus request:

uv run --project test-suite/python nexus-py-test compare \
  --model gpt-5.4-mini \
  --prompt "Say hello in one word." \
  --max-tokens 8

Run repeated latency samples with warmups and randomized order:

uv run --project test-suite/python nexus-py-test latency \
  --model gpt-5.4-mini \
  --iterations 10 \
  --warmups 2

Smoke test one path:

uv run --project test-suite/python nexus-py-test smoke --path nexus

Stream test one path:

uv run --project test-suite/python nexus-py-test stream \
  --path nexus \
  --model openai/gpt-5.4-mini \
  --json

Run the local pytest suite for deterministic parser and CLI checks:

uv run --project test-suite/python nexus-py-test test

The just shortcuts are equivalent:

just test compare py
just test latency py --iterations 10
just test ratelimit py --expect-rpm 20 --expect-level project --concurrency 20 --requests 100

ratelimit fires concurrent requests at the Nexus path only and checks the scope request and token ceilings. Prefer a gateway built with the mock-provider feature (nexus_mock scripted responses) so a burst does not spend provider tokens. Against a real provider, pass a small --max-tokens. Optional --control-plane-url, --control-plane-token, --scope, and --scope-id PATCH the ceiling before the run and clear it after; the command then waits --refresh-wait-seconds (default 30, match the gateway’s NEXUS_SCOPE_RL_REFRESH_SECS) so the gateway snapshot picks the change up before the burst. --baseline measures the same burst with the ceiling cleared first. --duration-seconds keeps sending across a minute boundary and checks that requests succeed again after the window rolls.

just test-py compare --json

Use --json for machine-readable output, --allow-failures when intentionally comparing error responses, and --expect-status on smoke or stream for negative-path checks. Use --provider-options-json '{"openai":{"reasoningEffort":"high"}}' to verify provider-options passthrough.