Coding-harness gauntlet

harness_runner.py drives a real coding harness — OpenCode, OpenAI Codex CLI, or Claude Code — through a running Nexus gateway and checks the result by executing it.

Each run is one two-turn session in a fresh fixture repository:

1. Turn 1, on model A: read src/add.js, fix its bug, and run node check.mjs.
2. Turn 2 continues the same harness session on model B (model A unless a switch model is given) and adds mul(a, b).

After each turn the runner imports src/add.js with node and checks add(2, 3) === 5, plus mul(3, 4) === 12 after turn 2. A job passes only when both checks pass and neither turn’s output contains a fatal error marker, such as an upstream 4xx/5xx error, a stream error, or failed to parse function arguments. The harness’s own report of success is not consulted.

Prerequisites

• A running gateway (just dev, then just status).
• node, npm, git, and python3 on PATH. The runner uses only the Python standard library.
• A live gateway key whose project can reach the models under test and has budget left.
• npm registry access the first time a harness version is installed.

Running

From nexus/, one job:

just harness-gauntlet codex bedrock/gpt-oss-120b-1
just harness-gauntlet claude-code anthropic/claude-sonnet-4-6 bedrock-gov/gpt-oss-120b-1

Every registered job (see Matrix run):

just harness-gauntlet all
just harness-gauntlet all --only bedrock-mantle --parallel 4
just harness-gauntlet --list

The recipe sources .env (or NEXUS_ENV_FILE) like the other live-suite recipes, so a NEXUS_API_KEY in that file replaces one exported in the shell. To run with a key from the shell instead, point NEXUS_ENV_FILE at a file without it, for example:

NEXUS_API_KEY=... NEXUS_ENV_FILE=/dev/null just harness-gauntlet codex bedrock/gpt-oss-120b-1

Direct invocation:

python3 test-suite/harness/harness_runner.py <harness> <model> [<switch_model>]
    [--codex-disable FEATURE]... [--claude-no-thinking] [--out-dir DIR] [--timeout SECONDS] [--keep]
python3 test-suite/harness/harness_runner.py all [--parallel N] [--attempts N] [--only SUBSTR]
    [--out-dir DIR] [--timeout SECONDS] [--keep]
python3 test-suite/harness/harness_runner.py --list [--only SUBSTR]

• <harness> is opencode, codex, or claude-code. Any gateway model id is accepted.
• --codex-disable FEATURE passes --disable FEATURE to Codex. It is refused for the other harnesses.
• --claude-no-thinking runs Claude Code with MAX_THINKING_TOKENS=0, so its requests carry no thinking field. It is refused for the other harnesses.
• --timeout bounds each turn (default 600 seconds); a timed-out turn kills the harness’s whole process group.
• --keep leaves the fixture repository and the harness home in place and records their paths in result.json.
• --list prints the registered jobs, one command line each; expected failures carry [expected failure: <reason> (tracking: <id>; matches /<pattern>/)]. --only SUBSTR filters the list.

Registered jobs

JOBS in harness_runner.py is built from two lists, each run on all three harnesses with default client settings (no --codex-disable, no --claude-no-thinking):

• MATRIX_MODELS: one model per family per lane: OpenAI reasoning and non-reasoning, Anthropic budget-thinking and adaptive-thinking generations, Azure Foundry Gov, Bedrock Converse (Nova, gpt-oss, Qwen, Kimi, GLM, Mistral, Nemotron, MiniMax), Bedrock GovCloud, Bedrock Mantle open-weight and Claude rows on both accounts, and xAI.
• SWITCHES: one mid-session switch into each lane (turn 1 on OpenAI or Anthropic, turn 2 on the target lane).

Each entry is a Job(harness, model, switch=None, codex_disable=(), claude_no_thinking=False, expected_failure=None, tracking=None, expected_pattern=None, expected_on_switch=False). Jobs whose models need a lane key read it from NEXUS_HARNESS_KEY_<PROVIDER> when set, and from NEXUS_API_KEY otherwise; a switch job uses each model’s own key.

Expected failures

A job with expected_failure set is known to fail for a named capability gap; tracking is a short id for the work that resolves it, and expected_pattern is a regular expression the upstream error must match.

Jobs	What fails	Error pattern	Tracking
codex on bedrock-mantle/gpt-oss-120b, bedrock-mantle-commercial/gpt-oss-120b, and xai/grok-4.3, and the codex switches into bedrock-mantle/gpt-oss-120b and xai/grok-4.3	These rows are served on the Responses API natively, and their upstream rejects the namespace tool type Codex sends by default.	unknown variant `namespace`	namespace-passthrough
claude-code on bedrock-mantle-commercial/devstral-2-123b	Mistral rejects the system messages Claude Code places in the middle of a conversation, after tool results.	Unexpected role 'system' after role 'tool'	mistral-mid-conversation-system
claude-code on bedrock-mantle/gemma-4-31b	Mantle gemma rejects Claude Code’s tool schemas, which use the propertyNames keyword.	400 The task request was rejected by the target	gemma-tool-schema-keywords
opencode on bedrock/nova-pro-v1 and bedrock-gov/nova-pro-v1	A model limitation, not a gateway failure: Nova Pro can keep misusing OpenCode’s exact-match edit tool while reacting to every tool result, until the turn times out.	timed out	nova-edit-loop

A failed expected job counts as expected_failure only when an error line of the failing turn matches its pattern. For a switch job whose expected failure comes from the turn 2 model (expected_on_switch), turn 1 must also pass; otherwise the job is a gateway_failure. Any other failure of an expected job is a gateway_failure, so a new fault on the same lane is not hidden.

Matrix run

harness_runner.py all (just harness-gauntlet all) runs every registered job concurrently against the gateway.

• --parallel N sets the jobs in flight (default GAUNTLET_PARALLEL, else 8).
• --attempts N caps the attempts per job (default 3). A job with an expected failure runs once.
• --only SUBSTR keeps the jobs whose command line contains SUBSTR, for example --only gpt-4o-mini or --only "codex xai".
• --allow-skip records a job whose model is not listed for its key as skipped instead of unavailable.
• --out-dir DIR defaults to test-suite/runs/harness/matrix-<UTC timestamp>/. --timeout and --keep apply to every attempt.

Before any job runs, every needed harness is installed once, and GET /v1/models is sent once per distinct key.

• If that request fails (HTTP error or no answer), the run stops with exit 2.
• A job whose model has no key set is skipped.
• A job whose model is not listed for a key that is set is unavailable, which fails the run unless --allow-skip is given.
• If no job is left to run, the run stops with exit 2.

Classification

Each output line that matches one of ERROR_PATTERNS is an error line. The patterns cover HTTP 4xx/5xx errors, invalid request body, API Error, unexpected status, no candidate endpoint, stream disconnected, no_provider_key, invalid_request_error, Codex’s ERROR: lines, and connection failures (ECONNREFUSED, ECONNRESET, fetch failed, socket hang up, Connection error, connection refused, AI_RetryError, error sending request, Unable to connect).

An error line reports the model’s own malformed tool-call arguments, rather than a gateway fault, only when it carries no 5xx status and either:

• it contains failed to parse function arguments (the harness could not parse the arguments the model wrote), or
• it contains Expecting ',' delimiter or Expecting value, does not contain invalid request body, and some line of the same attempt contains failed to parse function arguments.

Every other error line is a gateway error. Each attempt is then classified:

Classification	Rule
pass	Both turns left working code and no turn has a gateway error.
pass_with_gateway_errors	Both turns left working code, but a turn has a gateway error (for example a stream the harness reconnected).
gateway_failure	Code is broken and a turn timed out, a turn has a gateway error, or a turn exited non-zero with no error line at all (harness exited <code> without a recognised error).
model_quality	Code is broken, no turn has a gateway error, and every turn that left broken code exited 0 or reported only malformed arguments.

An attempt that does not pass is re-run until one passes or --attempts is used up. Each job then gets one final status:

Status	Meaning	Fails the run
pass	The first attempt passed.	no
pass_after_retry	A later attempt passed. If an earlier attempt was a gateway failure the job is listed as intermittent.	no
pass_with_gateway_errors	The passing attempt had gateway errors; listed as intermittent.	no
model_quality	Every attempt failed and none was a gateway failure.	no
gateway_failure	Every attempt failed and at least one was a gateway failure, or an expected job failed some other way.	yes
runner_error	The runner itself raised (fixture or process setup).	yes
unavailable	A model is not listed by /v1/models for its key (without --allow-skip).	yes
expected_failure	The job has an expected failure and failed with an error matching its pattern.	no
unexpected_pass	The job has an expected failure and passed; reported so the expectation can be removed.	no
skipped	A model has no key, or is not listed and --allow-skip was given.	no

The run exits 1 when any job ends in gateway_failure, runner_error, or unavailable, and 0 otherwise. It exits 2 on a usage error, when GET /v1/models fails, or when no job can run, and 3 when a harness cannot be installed.

Matrix artifacts

• <tag>.attempt<N>.log: the command, exit code, elapsed time, stdout, and stderr of both turns of one attempt.
• results.json: run metadata (gateway, harness versions, attempt cap) and, per job, its command, models, final status, expected-failure reason, tracking id and pattern, skip reason, a note when an expected job failed differently, and every attempt record: per-turn exit code, elapsed time, verification result, error lines, gateway and model-output markers, first error line, classification, reason, and log file.
• summary.md: counts per status, then tables of gateway failures (log and first error line), unavailable models, intermittent gateway errors on jobs that passed, jobs that passed after a model-quality retry, model-quality failures (harness, model, every attempt), expected failures (reason, tracking, observed failure), unexpected passes, and skipped jobs.

The same summary is printed to stdout.

Environment

Variable	Purpose
NEXUS_BASE_URL	Gateway origin, default http://127.0.0.1:14450. A trailing /v1 is stripped. Codex and OpenCode use <origin>/v1; Claude Code uses <origin>/anthropic.
NEXUS_API_KEY	Gateway key for every model without a lane override.
NEXUS_HARNESS_KEY_<PROVIDER>	Key for models under one provider prefix: the prefix upper-cased with - as _, e.g. NEXUS_HARNESS_KEY_BEDROCK_MANTLE for bedrock-mantle/*.
NEXUS_HARNESS_CACHE	Pinned harness installs, default ${XDG_CACHE_HOME:-~/.cache}/nexus-harness.
GAUNTLET_PARALLEL	Default for all --parallel.

Before installing anything, a single-job run checks that every model’s key is set and that GET /v1/models with that key lists the model, and stops otherwise.

Keys reach the harnesses only through environment variables. Generated harness configuration refers to them by variable name, and any key value appearing in harness output is redacted before the log is written.

Pinned harness versions

Harness	npm package	Version	Binary
OpenCode	opencode-ai	1.18.33	opencode
Codex CLI	@openai/codex	0.161.0	codex
Claude Code	@anthropic-ai/claude-code	2.1.259	claude

Only the requested harness is installed, into <cache>/<harness>-<version> with npm install --prefix. An install whose package.json already reports the pinned version is reused.

Exit codes

For a single job:

Code	Meaning
0	The job passed.
1	The job failed: a turn left broken code or its output carried a fatal marker.
2	Usage, configuration, or preflight error (missing key, model not listed, no node).
3	The pinned harness could not be installed.

For all, see Classification.

Artifacts

Each single-job run writes to --out-dir, by default test-suite/runs/harness/<UTC timestamp>/ (ignored by git):

• <tag>.log: the exact command, exit code, and elapsed time of each turn, followed by its stdout and stderr. The tag names the harness, models, disabled Codex features, and --claude-no-thinking.
• result.json: harness version, models, disabled Codex features, whether Claude Code thinking was disabled, per-turn exit code, elapsed time, verification result, and error markers, plus the verdict and the matrix classification of the attempt.

Harness behaviour the runner accounts for

• All three harnesses resolve the project from $PWD rather than the process working directory, so each turn sets PWD to the fixture repository. OpenCode also gets --dir, and the first Codex turn gets -C.
• opencode run ignores a project opencode.json. Providers are written to the user configuration at $XDG_CONFIG_HOME/opencode/opencode.json inside the run’s isolated home, with every XDG_* directory pointed there.
• Codex runs with its own CODEX_HOME, approval_policy = "never", web_search = "disabled", and update checks off, with one [model_providers.<name>] entry per distinct key. Turn 2 is codex exec resume --last.
• Claude Code runs with HOME and CLAUDE_CONFIG_DIR inside the run’s isolated home (so it never reads the user’s ~/.claude.json or login), ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN pointing at the gateway, every model-tier variable set to the model under test, telemetry and auto-update off, and ANTHROPIC_API_KEY removed. Turn 2 uses --continue.
• OpenCode turn 2 uses --continue.

Path-stress workload

gauntlet.py sends the request shapes coding harnesses produce straight to the gateway, without installing a harness. It runs every scenario against every model in its MODELS list, concurrently: OpenCode-style chat completions (basic, streaming, forced tools, tool results followed by a user turn, an empty replayed assistant turn, harness hint fields, parallel tool results, JSON schema, system plus developer messages), Codex-style Responses (streaming and a function-call round trip), and Claude Code-style Anthropic Messages (basic, streaming with tools, and extended thinking).

just harness-path-stress
just harness-path-stress test-suite/runs/harness/path-stress.json --concurrency 32
just harness-path-stress out.json openai/gpt-4o-mini,anthropic/claude-sonnet-4-6
just harness-path-stress --baseline test-suite/harness/baseline.json

Direct invocation:

python3 test-suite/harness/gauntlet.py <out.json> [<model>[,<model>...]] [--concurrency N]
    [--baseline PATH] [--write-baseline PATH]

• <out.json> receives one record per call: model, scenario, verdict (pass, fail, or skip), note, HTTP status, latency, x-nexus-dropped-params, request id, a body excerpt on failure, and either transport_error or timeout when the call got no HTTP response. The recipe defaults it to test-suite/runs/harness/path-stress-<UTC timestamp>.json.
• The optional model list restricts the run to those entries of MODELS.
• --concurrency sets the number of calls in flight (default 16).
• --write-baseline PATH writes the run’s verdicts as {"<model>|<scenario>": "<verdict>"}, keys sorted. A call that was re-run against --baseline is written with its re-run verdict, and "flaky" entries already in the file are kept.
• --baseline PATH compares each verdict with that file (see Verdict baseline).

Keys and the gateway origin come from the same variables as harness_runner.py: NEXUS_BASE_URL, NEXUS_API_KEY, and NEXUS_HARNESS_KEY_<PROVIDER>. The recipe sources .env (or NEXUS_ENV_FILE) the same way.

Before the run, the script sends one GET /v1/models with the first model’s key and stops if the gateway does not respond. A non-200 answer is printed as a warning and the run continues.

The run ends with a summary of verdicts, HTTP status counts, client timeouts, and transport-level errors. A transport-level error is a call refused, reset, or closed before a complete response (connection_refused, connection_closed), or one that failed without a response in any other way (other). A call that hits the client timeout is recorded as "timeout": true and counted separately; timeouts do not change the exit code.

Verdict baseline

test-suite/harness/baseline.json holds the committed path-stress verdicts; regenerate it from a full run with --write-baseline when a change is meant to move a verdict. With --baseline, a call that passed in the baseline and fails now is re-run once; it is a regression only if it fails again. The run prints the regressions, the calls that failed once and passed on re-run, and the improvements (baseline fail, now pass). Calls missing from either side, and calls marked "flaky" (a verdict that depends on model output from run to run), are not compared, so a model subset can be checked against the full baseline; the run prints how many calls were not compared and why. out.json records rerun_verdict and rerun_note on each re-run call.

Code	Meaning
0	No transport-level errors and no baseline regressions. Scenario failures and timeouts do not change this code.
1	At least one call hit a transport-level error, or a call regressed against --baseline.
2	Usage or configuration error (missing key, unknown model), or the gateway is unreachable.