0017 - Retry policy for gateway upstream attempts

• Status: Accepted
• Date: 2026-05-13
• Deciders: nexus core team
• Tags: gateway, retries, fallback, streaming

BYOK routing may encounter transient upstream failures such as rate limits, temporary service errors, or short network interruptions. The gateway needs bounded retries that compose with multi-provider fallback without hiding persistent failures or duplicating streamed responses.

• Retries must be bounded and tenant-controllable without redeploying the gateway.
• Retry behavior must be identical for JSON and streaming execution until streaming bytes are sent.
• Cross-provider fallback must remain a separate attempt-level decision.
• Metrics must expose retry counts and fallback position for operator visibility.

• Header-driven retry budget per request — Let clients opt into a bounded retry count through nexus-retries, with gateway defaults and caps.
• Fixed gateway retry policy only — Simpler, but prevents SDKs and tenants from tuning retry behavior for different workloads.
• Fallback-only recovery — Avoids repeated calls to one provider, but treats transient single-provider failures as hard fallback events.

Chosen option: Header-driven retry budget per request.

The gateway honors nexus-retries on the inbound request and applies exponential backoff with jitter inside a single provider attempt. Retries only apply to retryable statuses and transport failures. Once a streaming response has begun delivering bytes to the client, the gateway does not cross to another fallback attempt. JSON and streaming executors share the same retry policy.

• Operational tuning is header-driven per tenant or SDK without redeploying.
• Metrics expose nexus_gateway_attempt_retries_total with Bifrost-oriented label keys such as virtual_key_id, routing_rule_id, and fallback_index.
• Aggregate attempt errors remain visible when retries and fallbacks are exhausted.