Ir al contenido
Nexus Docs
↑↓Navigate↵SelectescClose

0017 - Retry policy for gateway upstream attempts

Gateway retry policy decision for upstream failure handling

Last updated on:

Esta página aún no está disponible en tu idioma.

Context and Problem Statement

Section titled “Context and Problem Statement”

BYOK routing may encounter transient upstream failures such as rate limits, temporary service errors, or short network interruptions. The gateway needs bounded retries that compose with multi-provider fallback without hiding persistent failures or duplicating streamed responses.

Decision Drivers

Section titled “Decision Drivers”
  • Retries must be bounded and tenant-controllable without redeploying the gateway.
  • Retry behavior must be identical for JSON and streaming execution until streaming bytes are sent.
  • Cross-provider fallback must remain a separate attempt-level decision.
  • Metrics must expose retry counts and fallback position for operator visibility.

Considered Options

Section titled “Considered Options”
  • Header-driven retry budget per request — Let clients opt into a bounded retry count through nexus-retries, with gateway defaults and caps.
  • Fixed gateway retry policy only — Simpler, but prevents SDKs and tenants from tuning retry behavior for different workloads.
  • Fallback-only recovery — Avoids repeated calls to one provider, but treats transient single-provider failures as hard fallback events.

Decision Outcome

Section titled “Decision Outcome”

Chosen option: Header-driven retry budget per request.

The gateway honors nexus-retries on the inbound request and applies exponential backoff with jitter inside a single provider attempt. Retries only apply to retryable statuses and transport failures. Once a streaming response has begun delivering bytes to the client, the gateway does not cross to another fallback attempt. JSON and streaming executors share the same retry policy.

Consequences

Section titled “Consequences”
  • Operational tuning is header-driven per tenant or SDK without redeploying.
  • Metrics expose nexus_gateway_attempt_retries_total with Bifrost-oriented label keys such as virtual_key_id, routing_rule_id, and fallback_index.
  • Aggregate attempt errors remain visible when retries and fallbacks are exhausted.