A crash-test lab for LLM gateways and SDKs. One fake provider breaks on purpose in sixteen documented ways; every system below is pushed through the same sixteen faults; each cell is what the caller actually got.
Next scorecard: Monday 5 October 2026. Harness, methodology and raw logs: github.com/failoverbench/failoverbench.
| Scenario | Bifrost v2.1.0 (Docker) | No gateway (direct, no retries) | DeepSeek Harness + dsh-llm-fallbacks (SDK) | LangChain ChatOpenAI + with_fallbacks | LiteLLM proxy (Docker) | LiteLLM Router, tuned (allowed_fails + cooldown) | LiteLLM Router (in-process) | openai-python SDK (built-in retries) | Portkey Gateway (Docker, OSS) | Reference client (httpx) |
|---|---|---|---|---|---|---|---|---|---|---|
| S01 429 with Retry-AfterFirst attempt gets HTTP 429 with `Retry-After: 2`; every later attempt succeeds. | partial2p/0f · gap 0.6s · 1.5s | safe1p/0f · 0.0s | partial1p/1f · 2.6s | pass2p/0f · gap 2.0s · 3.0s | pass2p/0f · gap 2.4s · 3.4s | pass2p/0f · gap 2.8s · 4.1s | pass2p/0f · gap 2.7s · 4.0s | pass2p/0f · gap 2.0s · 3.3s | pass2p/0f · gap 2.0s · 3.0s | pass2p/0f · gap 2.0s · 3.0s |
| S02 429 without Retry-After, persistentEvery attempt gets HTTP 429 and no Retry-After header. | pass3p/1f · gap 0.4s · 2.3s | safe1p/0f · 0.0s | pass1p/1f · 1.9s | pass3p/1f · gap 0.4s · 2.3s | pass3p/1f · gap 1.1s · 5.8s | pass2p/1f · gap 1.1s · 2.1s | pass3p/1f · gap 0.6s · 5.3s | safe3p/0f · gap 0.5s · 1.4s | pass3p/1f · gap 1.0s · 4.0s | pass3p/1f · gap 0.5s · 2.4s |
| S03 500 once, then healthyFirst attempt gets HTTP 500; the second succeeds. | pass2p/0f · gap 0.4s · 1.4s | safe1p/0f · 0.0s | partial1p/1f · 1.9s | pass2p/0f · gap 0.5s · 1.5s | pass2p/0f · gap 0.8s · 1.8s | pass2p/0f · gap 1.3s · 2.2s | pass2p/0f · gap 0.7s · 1.7s | pass2p/0f · gap 0.5s · 1.5s | pass2p/0f · gap 1.0s · 2.0s | pass2p/0f · gap 0.5s · 1.5s |
| S04 503 persistentEvery attempt gets HTTP 503 (overloaded). | pass3p/1f · gap 0.6s · 2.4s | safe1p/0f · 0.0s | pass1p/1f · 2.0s | pass3p/1f · gap 0.5s · 2.4s | pass3p/1f · gap 0.8s · 5.5s | pass2p/1f · gap 1.0s · 2.0s | pass3p/1f · gap 0.9s · 5.8s | safe3p/0f · gap 0.5s · 1.4s | pass3p/1f · gap 1.0s · 4.0s | pass3p/1f · gap 0.4s · 2.2s |
| S05 TCP reset before headersFirst attempt: the connection is reset before any response bytes. Later attempts succeed. | pass2p/0f · gap 0.0s · 1.0s | safe1p/0f · 0.0s | pass1p/1f · 1.9s | pass2p/0f · gap 0.4s · 1.3s | pass2p/0f · gap 0.6s · 1.6s | pass2p/0f · gap 1.2s · 2.2s | pass2p/0f · gap 0.8s · 1.8s | pass2p/0f · gap 0.4s · 1.4s | pass2p/0f · gap 1.0s · 2.0s | pass2p/0f · gap 0.5s · 1.5s |
| S06 No response at allThe provider accepts the request and never answers (up to 120 s). | fail2p/0f · gap 120.0s · 130.0s | safe1p/0f · 30.0s | pass1p/1f · 31.9s | partial3p/1f · gap 30.4s · 92.3s | partial3p/1f · gap 30.8s · 95.5s | partial3p/1f · gap 31.2s · 96.8s | partial3p/1f · gap 30.8s · 96.2s | safe3p/0f · gap 30.5s · 91.6s | pass1p/1f · 31.0s | pass1p/1f · 11.0s |
| S07 Slow first tokenHeaders arrive at once; the first token arrives after 20 s; the rest is normal. | partial1p/0f · 21.0s | pass1p/0f · 21.0s | partial1p/0f · 22.0s | partial1p/0f · 21.0s | partial1p/0f · 21.0s | partial1p/0f · 21.0s | partial1p/0f · 21.0s | pass1p/0f · 21.0s | partial1p/0f · 21.0s | pass1p/1f · 11.0s |
| S08 Stream cut mid-answerThe stream dies after 40 of 60 tokens with no terminating chunk. | safe1p/0f · 0.7s | safe1p/0f · 0.6s | pass1p/1f · 2.6s | safe1p/0f · 0.6s | safe1p/0f · 0.7s | safe1p/0f · 0.7s | safe1p/0f · 0.7s | safe1p/0f · 0.6s | fail1p/0f · 0.7s · TRUNCATED | pass3p/1f · gap 1.2s · 4.3s |
| S09 Stream stallsAfter 20 tokens the connection stays open and goes silent for 30 s, then closes. | safe1p/0f · 30.3s | safe1p/0f · 30.3s | pass1p/1f · 32.3s | safe1p/0f · 30.3s | safe1p/0f · 30.3s | safe1p/0f · 30.3s | safe1p/0f · 30.3s | safe1p/0f · 30.3s | fail1p/0f · 30.3s · TRUNCATED | pass1p/1f · 11.3s |
| S10 Malformed chunk in the streamThe tenth SSE event is invalid JSON; the stream otherwise completes normally. | pass1p/0f · 1.0s | pass1p/0f · 1.0s | safe1p/0f · 1.2s | safe1p/0f · 0.2s | safe1p/0f · 0.2s | safe1p/0f · 0.2s | safe1p/0f · 0.2s | safe1p/0f · 0.2s | pass1p/0f · 1.0s | pass1p/0f · 1.0s |
| S11 Stream ends without [DONE]A complete stream, correctly terminated at the HTTP level, but the `data: [DONE]` sentinel is missing. | pass1p/0f · 1.0s | pass1p/0f · 1.0s | pass1p/0f · 1.9s | pass1p/0f · 1.0s | pass1p/0f · 1.0s | pass1p/0f · 1.0s | pass1p/0f · 1.0s | pass1p/0f · 1.0s | pass1p/0f · 1.0s | pass1p/0f · 1.0s |
| S12 Context length exceededHTTP 400 with code `context_length_exceeded` on every attempt. | pass1p/1f · 1.0s | safe1p/0f · 0.0s | pass1p/1f · 1.9s | pass1p/1f · 1.0s | pass1p/1f · 1.0s | pass1p/1f · 1.0s | pass1p/1f · 1.0s | safe1p/0f · 0.0s | pass1p/1f · 1.0s | pass1p/1f · 1.0s |
| S13 Content-filter rejectionHTTP 400 with code `content_policy_violation` on every attempt. | partial1p/1f · 1.0s | pass1p/0f · 0.0s | pass1p/0f · 1.0s | partial1p/1f · 1.0s | partial1p/1f · 1.0s | partial1p/1f · 1.0s | partial1p/1f · 1.0s | pass1p/0f · 0.0s | partial1p/1f · 1.0s | pass1p/0f · 0.0s |
| S14 Fallback target rejects prefillThe primary stream dies after 40 tokens. The fallback model rejects any request whose last message is an assistant prefill. | safe1p/0f · 0.6s | safe1p/0f · 0.6s | pass1p/1f · 2.7s | safe1p/0f · 0.6s | safe1p/0f · 0.7s | safe1p/0f · 0.7s | safe1p/0f · 0.7s | safe1p/0f · 0.6s | fail1p/0f · 0.7s · TRUNCATED | pass3p/1f · gap 1.2s · 4.6s |
| S15 Primary flaps: down, then healthyThe primary answers 503 for the first 60 s after reset, then is healthy. Twenty requests, one every 4 s. | pass15/15 served in outage · back on primary +0s | safe0/15 served in outage · back on primary +0s | pass15/15 served in outage · back on primary +0s | pass15/15 served in outage · back on primary +0s | pass11/11 served in outage · back on primary +0s | pass15/15 served in outage · back on primary +4s | pass11/11 served in outage · back on primary +2s | safe0/15 served in outage · back on primary +0s | pass15/15 served in outage · back on primary +0s | pass15/15 served in outage · back on primary +0s |
| S16 Fail-over accountingEvery primary attempt gets HTTP 503; the fallback answers. | pass3p/1f · gap 0.5s · 2.5s | safe1p/0f · 0.0s | pass1p/1f · 2.0s | pass3p/1f · gap 0.5s · 2.3s | pass3p/1f · gap 0.9s · 5.3s | pass2p/1f · gap 0.9s · 1.9s | pass3p/1f · gap 0.9s · 5.7s | safe3p/0f · gap 0.4s · 1.3s | pass3p/1f · gap 1.0s · 4.0s | pass3p/1f · gap 0.6s · 2.7s |
A subset of scenarios re-run under a different setting, shown next to the row they contrast with.
DSH, shared session — S15 fail 15/15 served in outage · back on primary never
| System | pass | partial | safe | fail | run |
|---|---|---|---|---|---|
| Bifrost v2.1.0 (Docker) | 9 | 3 | 3 | 1 | 275s |
| No gateway (direct, no retries) | 4 | 0 | 12 | 0 | 162s |
| DSH, shared session | 0 | 0 | 0 | 1 | 79s |
| DeepSeek Harness + dsh-llm-fallbacks (SDK) | 12 | 3 | 1 | 0 | 204s |
| LangChain ChatOpenAI + with_fallbacks | 9 | 3 | 4 | 0 | 238s |
| LiteLLM proxy (Docker) | 9 | 3 | 4 | 0 | 252s |
| LiteLLM Router, tuned (allowed_fails + cooldown) | 9 | 3 | 4 | 0 | 243s |
| LiteLLM Router (in-process) | 9 | 3 | 4 | 0 | 254s |
| openai-python SDK (built-in retries) | 6 | 0 | 10 | 0 | 232s |
| Portkey Gateway (Docker, OSS) | 11 | 2 | 0 | 3 | 184s |
| Reference client (httpx) | 16 | 0 | 0 | 0 | 136s |
Every check the runner scored, and what the fake provider saw. Advisory checks change a pass to partial; required checks change it to fail.
endpoint, fallback=yes, streaming=yesBifrost with a custom provider `fake` (base_provider_type openai, base_url = the wall, allow_private_network on). Baseline settings: max_retries 2 (vendor default is 0), request timeout 30 s, stream idle timeout 30 s, one fallback passed per request (`fallbacks: ["fake/fb-ok"]`). Bifrost does not honour Retry-After (backoff 500 ms → 5 s with jitter). Per docs, fallback covers HTTP-level errors and an error in the first SSE event; later stream errors are not retried.
params: {"base_url": "http://127.0.0.1:18080/v1", "api_key": "failoverbench", "model_format": "fake/{model}", "extra_body": {"fallbacks": ["{fallback}"]}}
S01 429 with Retry-After — partial rescued; advisory: retry_after_honoured
retried pass: 2 primary attempt(s)retry_after_honoured FAIL (advisory): gap 0.57sno_hammer pass: 2 attempt(s) in first 1.5sS02 429 without Retry-After, persistent — pass rescued
no_hammer pass: 3 attempt(s) in first 3sbounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 2.3s (success)S03 500 once, then healthy — pass rescued
no_hang pass: ended in 1.4s (success)same_provider_retry pass: 2 primary attempt(s)S04 503 persistent — pass rescued
bounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 2.4s (success)S05 TCP reset before headers — pass rescued
no_hang pass: ended in 1.0s (success)S06 No response at all — fail failed: within_budget
within_budget FAIL: 130.0sno_hang pass: ended in 130.0s (error)silent_route_not_retried FAIL (advisory): 2 primary attempt(s)S07 Slow first token — partial rescued; advisory: ttft_limit
no_hang pass: ended in 21.0s (success)ttft_limit FAIL (advisory): 21.0sS08 Stream cut mid-answer — safe not rescued — clean error: in-band error event after 40 chunk(s): provider closed the stream before sending
no_truncated_success pass: no success claimed (error)no_hang pass: ended in 0.7s (error)S09 Stream stalls — safe not rescued — clean error: in-band error event after 20 chunk(s): Error reading stream: stream idle timeout
bounded pass: 30.3sno_truncated_success pass: no success claimed (error)alive_after pass: healthy call succeeded afterwardsno_hang pass: ended in 30.3s (error)S10 Malformed chunk in the stream — pass rescued
alive_after pass: healthy call succeeded afterwardsno_garbage pass: complete answerno_hang pass: ended in 1.0s (success)recovered_in_place pass: complete answerS11 Stream ends without [DONE] — pass rescued
prompt_finalise pass: 1.0susage_reported pass: usage = {'prompt_tokens': 21, 'completion_tokens': 60, 'total_tokens': 81}S12 Context length exceeded — pass rescued
no_blind_retry pass: 1 primary attempt(s)bounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)S13 Content-filter rejection — partial answered by routing around the rejection
no_same_model_retry pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)fallback_attempted pass: 1 fallback attempt(s)S14 Fallback target rejects prefill — safe not rescued — clean error: in-band error event after 40 chunk(s): provider closed the stream before sending
no_prefill_masking pass: 0 prefill rejection(s) at the fallback; final outcome errorno_truncated_success pass: no success claimed (error)S15 Primary flaps: down, then healthy — pass rescued
no_hang pass: 0 hung request(s) of 20served_during_outage pass: 15/15 requests answered while primary was downrecovered pass: primary answered again +0.0s after recoveryprobe_discipline info: 45 primary attempt(s) for 15 request(s) during the outageS16 Fail-over accounting — pass rescued
reports_fallback_model pass: response.model = 'fb-ok'usage_present pass: usage = {'prompt_tokens': 21, 'completion_tokens': 60, 'total_tokens': 81}bounded_attempts pass: 3 primary attempt(s)direct, fallback=no, streaming=yesControl row: one request straight at the provider with no retries, no fallback, a 30 s read timeout. What an app gets with zero protection.
params: {"read_timeout_s": 30}
S01 429 with Retry-After — safe not rescued — clean error: HTTP 429: Rate limit reached for fb-s01-429-retry-after. Please retry after 2 se
retried FAIL (advisory): 1 primary attempt(s)retry_after_honoured FAIL (advisory): no second attemptno_hammer pass: 1 attempt(s) in first 1.5sS02 429 without Retry-After, persistent — safe not rescued — clean error: HTTP 429: Rate limit reached for fb-s02-429-persistent.
no_hammer pass: 1 attempt(s) in first 3sbounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 0.0s (error)S03 500 once, then healthy — safe not rescued — clean error: HTTP 500: The server had an error while processing your request.
no_hang pass: ended in 0.0s (error)same_provider_retry FAIL (advisory): 1 primary attempt(s)S04 503 persistent — safe not rescued — clean error: HTTP 503: The engine is currently overloaded. Please try again later.
bounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 0.0s (error)S05 TCP reset before headers — safe not rescued — clean error: ReadError:
no_hang pass: ended in 0.0s (error)S06 No response at all — safe not rescued — clean error: ReadTimeout:
within_budget pass: 30.0sno_hang pass: ended in 30.0s (error)silent_route_not_retried pass: 1 primary attempt(s)S07 Slow first token — pass rescued
no_hang pass: ended in 21.0s (success)ttft_limit n/a: n/a — system declares no fallbackS08 Stream cut mid-answer — safe not rescued — clean error: RemoteProtocolError: peer closed connection without sending complete message bod
no_truncated_success pass: no success claimed (error)no_hang pass: ended in 0.6s (error)S09 Stream stalls — safe not rescued — clean error: ReadTimeout:
bounded pass: 30.3sno_truncated_success pass: no success claimed (error)alive_after pass: healthy call succeeded afterwardsno_hang pass: ended in 30.3s (error)S10 Malformed chunk in the stream — pass rescued
alive_after pass: healthy call succeeded afterwardsno_garbage pass: complete answerno_hang pass: ended in 1.0s (success)recovered_in_place pass: complete answerS11 Stream ends without [DONE] — pass rescued
prompt_finalise pass: 1.0susage_reported pass: usage = {'prompt_tokens': 21, 'completion_tokens': 60, 'total_tokens': 81}S12 Context length exceeded — safe not rescued — clean error: HTTP 400: This model's maximum context length is 8192 tokens. However, your mess
no_blind_retry pass: 1 primary attempt(s)bounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 0.0s (error)S13 Content-filter rejection — pass rescued
no_same_model_retry pass: 1 primary attempt(s)no_hang pass: ended in 0.0s (error)fallback_attempted info: 0 fallback attempt(s)S14 Fallback target rejects prefill — safe not rescued — clean error: RemoteProtocolError: peer closed connection without sending complete message bod
no_prefill_masking pass: 0 prefill rejection(s) at the fallback; final outcome errorno_truncated_success pass: no success claimed (error)S15 Primary flaps: down, then healthy — safe not rescued — clean error: HTTP 503: The engine is currently overloaded. Please try again later.
no_hang pass: 0 hung request(s) of 20served_during_outage n/a: n/a — system declares no fallbackrecovered n/a: n/a — system declares no fallbackprobe_discipline pass: 15 primary attempt(s) for 15 request(s) during the outageS16 Fail-over accounting — safe not rescued — clean error: HTTP 503: The engine is currently overloaded. Please try again later.
reports_fallback_model n/a: n/a — system declares no fallbackusage_present n/a: n/a — system declares no fallbackbounded_attempts pass: 1 primary attempt(s)dsh, fallback=yes, streaming=yesContrast row for S15 only: identical to the DSH row except session_per_request: false, so all twenty requests share one harness session. Shows whether the primary is never probed again (plugin) or the replaced model sticks to the session (harness).
params: {"dsh_home": "gateways/dsh/home-default", "dsh_home_noprefill": "gateways/dsh/home-noprefill", "workspace": "gateways/dsh/workspace", "provider": "fake", "request_timeout_s": 60, "session_per_request": false}
S15 Primary flaps: down, then healthy — fail failed: recovered
no_hang pass: 0 hung request(s) of 20served_during_outage pass: 15/15 requests answered while primary was downrecovered FAIL: primary never used again after recoveryprobe_discipline pass: 1 primary attempt(s) for 15 request(s) during the outagedsh, fallback=yes, streaming=yesEXPERIMENTAL. DeepSeek Harness 0.1.2-rc.1 driven through the official Python SDK, with dsh-llm-fallbacks 0.4.2 (chain primary → fb-ok, cooldown 30 s, half-open recovery). Baseline: llm-retry maxRetries 2, stream idle timeout 30 s. The harness sends its own system prompt and tool roster on every call; the wall ignores both. Run gateways/dsh/setup.sh first.
params: {"dsh_home": "gateways/dsh/home-default", "dsh_home_noprefill": "gateways/dsh/home-noprefill", "workspace": "gateways/dsh/workspace", "provider": "fake", "request_timeout_s": 60, "session_per_request": true}
S01 429 with Retry-After — partial rescued; advisory: retried, retry_after_honoured
retried FAIL (advisory): 1 primary attempt(s)retry_after_honoured FAIL (advisory): no second attemptno_hammer pass: 1 attempt(s) in first 1.5sS02 429 without Retry-After, persistent — pass rescued
no_hammer pass: 1 attempt(s) in first 3sbounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 1.9s (success)S03 500 once, then healthy — partial rescued; advisory: same_provider_retry
no_hang pass: ended in 1.9s (success)same_provider_retry FAIL (advisory): 1 primary attempt(s)S04 503 persistent — pass rescued
bounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 2.0s (success)S05 TCP reset before headers — pass rescued
no_hang pass: ended in 1.9s (success)S06 No response at all — pass rescued
within_budget pass: 31.9sno_hang pass: ended in 31.9s (success)silent_route_not_retried pass: 1 primary attempt(s)S07 Slow first token — partial rescued; advisory: ttft_limit
no_hang pass: ended in 22.0s (success)ttft_limit FAIL (advisory): 22.0sS08 Stream cut mid-answer — pass rescued
no_truncated_success pass: complete answerno_hang pass: ended in 2.6s (success)S09 Stream stalls — pass rescued
bounded pass: 32.3sno_truncated_success pass: complete answeralive_after pass: healthy call succeeded afterwardsno_hang pass: ended in 32.3s (success)S10 Malformed chunk in the stream — safe not rescued — clean error: dsh finish_reason='error', final_response=empty
alive_after pass: healthy call succeeded afterwardsno_garbage pass: no success claimed (error)no_hang pass: ended in 1.2s (error)recovered_in_place FAIL (advisory): errorS11 Stream ends without [DONE] — pass rescued
prompt_finalise pass: 1.9susage_reported pass: usage = {'prompt_tokens': 810, 'completion_tokens': 60, 'total_tokens': 870}S12 Context length exceeded — pass rescued
no_blind_retry pass: 1 primary attempt(s)bounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 1.9s (success)S13 Content-filter rejection — pass rescued
no_same_model_retry pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (error)fallback_attempted info: 0 fallback attempt(s)S14 Fallback target rejects prefill — pass rescued
no_prefill_masking pass: 0 prefill rejection(s) at the fallback; final outcome successno_truncated_success pass: complete answerS15 Primary flaps: down, then healthy — pass rescued
no_hang pass: 0 hung request(s) of 20served_during_outage pass: 15/15 requests answered while primary was downrecovered pass: primary answered again +0.1s after recoveryprobe_discipline pass: 15 primary attempt(s) for 15 request(s) during the outageS16 Fail-over accounting — pass rescued
reports_fallback_model pass: response.model = 'fake/fb-ok'usage_present pass: usage = {'prompt_tokens': 810, 'completion_tokens': 60, 'total_tokens': 870}bounded_attempts pass: 1 primary attempt(s)langchain, fallback=yes, streaming=yeslangchain-openai ChatOpenAI(max_retries=2, timeout=30) with .with_fallbacks([ChatOpenAI(fallback)]). LangChain documents that streaming fallbacks only cover errors before the first chunk.
params: {"max_retries": 2, "timeout_s": 30}
S01 429 with Retry-After — pass rescued
retried pass: 2 primary attempt(s)retry_after_honoured pass: gap 2.00sno_hammer pass: 1 attempt(s) in first 1.5sS02 429 without Retry-After, persistent — pass rescued
no_hammer pass: 3 attempt(s) in first 3sbounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 2.3s (success)S03 500 once, then healthy — pass rescued
no_hang pass: ended in 1.5s (success)same_provider_retry pass: 2 primary attempt(s)S04 503 persistent — pass rescued
bounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 2.4s (success)S05 TCP reset before headers — pass rescued
no_hang pass: ended in 1.3s (success)S06 No response at all — partial rescued; advisory: silent_route_not_retried
within_budget pass: 92.3sno_hang pass: ended in 92.3s (success)silent_route_not_retried FAIL (advisory): 3 primary attempt(s)S07 Slow first token — partial rescued; advisory: ttft_limit
no_hang pass: ended in 21.0s (success)ttft_limit FAIL (advisory): 21.0sS08 Stream cut mid-answer — safe not rescued — clean error: RemoteProtocolError: peer closed connection without sending complete message bod
no_truncated_success pass: no success claimed (error)no_hang pass: ended in 0.6s (error)S09 Stream stalls — safe not rescued — clean error: ReadTimeout:
bounded pass: 30.3sno_truncated_success pass: no success claimed (error)alive_after pass: healthy call succeeded afterwardsno_hang pass: ended in 30.3s (error)S10 Malformed chunk in the stream — safe not rescued — clean error: JSONDecodeError: Expecting value: line 1 column 94 (char 93)
alive_after pass: healthy call succeeded afterwardsno_garbage pass: no success claimed (error)no_hang pass: ended in 0.2s (error)recovered_in_place FAIL (advisory): errorS11 Stream ends without [DONE] — pass rescued
prompt_finalise pass: 1.0susage_reported pass: usage = {'prompt_tokens': 21, 'completion_tokens': 60, 'total_tokens': 81}S12 Context length exceeded — pass rescued
no_blind_retry pass: 1 primary attempt(s)bounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)S13 Content-filter rejection — partial answered by routing around the rejection
no_same_model_retry pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)fallback_attempted pass: 1 fallback attempt(s)S14 Fallback target rejects prefill — safe not rescued — clean error: RemoteProtocolError: peer closed connection without sending complete message bod
no_prefill_masking pass: 0 prefill rejection(s) at the fallback; final outcome errorno_truncated_success pass: no success claimed (error)S15 Primary flaps: down, then healthy — pass rescued
no_hang pass: 0 hung request(s) of 20served_during_outage pass: 15/15 requests answered while primary was downrecovered pass: primary answered again +0.0s after recoveryprobe_discipline info: 45 primary attempt(s) for 15 request(s) during the outageS16 Fail-over accounting — pass rescued
reports_fallback_model pass: response.model = 'fb-ok'usage_present pass: usage = {'prompt_tokens': 21, 'completion_tokens': 60, 'total_tokens': 81}bounded_attempts pass: 3 primary attempt(s)endpoint, fallback=yes, streaming=yesLiteLLM proxy container configured by `python -m failoverbench litellm-config` (see gateways/litellm). The proxy owns retries and fallbacks.
params: {"base_url": "http://127.0.0.1:4000/v1", "api_key": "sk-failoverbench"}
S01 429 with Retry-After — pass rescued
retried pass: 2 primary attempt(s)retry_after_honoured pass: gap 2.37sno_hammer pass: 1 attempt(s) in first 1.5sS02 429 without Retry-After, persistent — pass rescued
no_hammer pass: 3 attempt(s) in first 3sbounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 5.8s (success)S03 500 once, then healthy — pass rescued
no_hang pass: ended in 1.8s (success)same_provider_retry pass: 2 primary attempt(s)S04 503 persistent — pass rescued
bounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 5.5s (success)S05 TCP reset before headers — pass rescued
no_hang pass: ended in 1.6s (success)S06 No response at all — partial rescued; advisory: silent_route_not_retried
within_budget pass: 95.5sno_hang pass: ended in 95.5s (success)silent_route_not_retried FAIL (advisory): 3 primary attempt(s)S07 Slow first token — partial rescued; advisory: ttft_limit
no_hang pass: ended in 21.0s (success)ttft_limit FAIL (advisory): 21.0sS08 Stream cut mid-answer — safe not rescued — clean error: in-band error event after 41 chunk(s): litellm.APIConnectionError: APIConnection
no_truncated_success pass: no success claimed (error)no_hang pass: ended in 0.7s (error)S09 Stream stalls — safe not rescued — clean error: in-band error event after 21 chunk(s): litellm.APIConnectionError: APIConnection
bounded pass: 30.3sno_truncated_success pass: no success claimed (error)alive_after pass: healthy call succeeded afterwardsno_hang pass: ended in 30.3s (error)S10 Malformed chunk in the stream — safe not rescued — clean error: in-band error event after 11 chunk(s): litellm.APIConnectionError: APIConnection
alive_after pass: healthy call succeeded afterwardsno_garbage pass: no success claimed (error)no_hang pass: ended in 0.2s (error)recovered_in_place FAIL (advisory): errorS11 Stream ends without [DONE] — pass rescued
prompt_finalise pass: 1.0susage_reported pass: usage = {'completion_tokens': 60, 'prompt_tokens': 21, 'total_tokens': 81, 'completion_tokens_details': {'reasoning_tokens': 0}}S12 Context length exceeded — pass rescued
no_blind_retry pass: 1 primary attempt(s)bounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)S13 Content-filter rejection — partial answered by routing around the rejection
no_same_model_retry pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)fallback_attempted pass: 1 fallback attempt(s)S14 Fallback target rejects prefill — safe not rescued — clean error: in-band error event after 41 chunk(s): litellm.APIConnectionError: APIConnection
no_prefill_masking pass: 0 prefill rejection(s) at the fallback; final outcome errorno_truncated_success pass: no success claimed (error)S15 Primary flaps: down, then healthy — pass rescued
no_hang pass: 0 hung request(s) of 20served_during_outage pass: 11/11 requests answered while primary was downrecovered pass: primary answered again +0.2s after recoveryprobe_discipline info: 33 primary attempt(s) for 11 request(s) during the outageS16 Fail-over accounting — pass rescued
reports_fallback_model pass: response.model = 'fb-ok'usage_present pass: usage = {'completion_tokens': 60, 'prompt_tokens': 21, 'total_tokens': 81, 'completion_tokens_details': {'reasoning_tokens': 0}}bounded_attempts pass: 3 primary attempt(s)litellm-router, fallback=yes, streaming=yesSame as the LiteLLM Router row plus allowed_fails=1 and cooldown_time=30, the settings LiteLLM needs before it will cool down a single-deployment model group (PR #8668). Shown so readers can see what one config line changes on S15.
params: {"num_retries": 2, "timeout_s": 30, "context_window_fallbacks": true, "allowed_fails": 1, "cooldown_time": 30}
S01 429 with Retry-After — pass rescued
retried pass: 2 primary attempt(s)retry_after_honoured pass: gap 2.76sno_hammer pass: 1 attempt(s) in first 1.5sS02 429 without Retry-After, persistent — pass rescued
no_hammer pass: 2 attempt(s) in first 3sbounded_attempts pass: 2 primary attempt(s)no_hang pass: ended in 2.1s (success)S03 500 once, then healthy — pass rescued
no_hang pass: ended in 2.2s (success)same_provider_retry pass: 2 primary attempt(s)S04 503 persistent — pass rescued
bounded_attempts pass: 2 primary attempt(s)no_hang pass: ended in 2.0s (success)S05 TCP reset before headers — pass rescued
no_hang pass: ended in 2.2s (success)S06 No response at all — partial rescued; advisory: silent_route_not_retried
within_budget pass: 96.8sno_hang pass: ended in 96.8s (success)silent_route_not_retried FAIL (advisory): 3 primary attempt(s)S07 Slow first token — partial rescued; advisory: ttft_limit
no_hang pass: ended in 21.0s (success)ttft_limit FAIL (advisory): 21.0sS08 Stream cut mid-answer — safe not rescued — clean error: APIConnectionError: litellm.APIConnectionError: APIConnectionError: OpenAIExcept
no_truncated_success pass: no success claimed (error)no_hang pass: ended in 0.7s (error)S09 Stream stalls — safe not rescued — clean error: APIConnectionError: litellm.APIConnectionError: APIConnectionError: OpenAIExcept
bounded pass: 30.3sno_truncated_success pass: no success claimed (error)alive_after pass: healthy call succeeded afterwardsno_hang pass: ended in 30.3s (error)S10 Malformed chunk in the stream — safe not rescued — clean error: APIConnectionError: litellm.APIConnectionError: APIConnectionError: OpenAIExcept
alive_after pass: healthy call succeeded afterwardsno_garbage pass: no success claimed (error)no_hang pass: ended in 0.2s (error)recovered_in_place FAIL (advisory): errorS11 Stream ends without [DONE] — pass rescued
prompt_finalise pass: 1.0susage_reported pass: usage = {'completion_tokens': 60, 'prompt_tokens': 21, 'total_tokens': 81, 'completion_tokens_details': {'accepted_prediction_tokens': None, 'audio_tokens': None, 'reasoning_tokens': 0, 'rejected_prediction_tokens': None, 'text_tokens': None, 'image_tokens': None, 'video_tokens': None}, 'prompt_tokens_details': None}S12 Context length exceeded — pass rescued
no_blind_retry pass: 1 primary attempt(s)bounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)S13 Content-filter rejection — partial answered by routing around the rejection
no_same_model_retry pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)fallback_attempted pass: 1 fallback attempt(s)S14 Fallback target rejects prefill — safe not rescued — clean error: APIConnectionError: litellm.APIConnectionError: APIConnectionError: OpenAIExcept
no_prefill_masking pass: 0 prefill rejection(s) at the fallback; final outcome errorno_truncated_success pass: no success claimed (error)S15 Primary flaps: down, then healthy — pass rescued
no_hang pass: 0 hung request(s) of 20served_during_outage pass: 15/15 requests answered while primary was downrecovered pass: primary answered again +4.0s after recoveryprobe_discipline pass: 4 primary attempt(s) for 15 request(s) during the outageS16 Fail-over accounting — pass rescued
reports_fallback_model pass: response.model = 'fb-ok'usage_present pass: usage = {'completion_tokens': 60, 'prompt_tokens': 21, 'total_tokens': 81, 'completion_tokens_details': {'accepted_prediction_tokens': None, 'audio_tokens': None, 'reasoning_tokens': 0, 'rejected_prediction_tokens': None, 'text_tokens': None, 'image_tokens': None, 'video_tokens': None}, 'prompt_tokens_details': None}bounded_attempts pass: 2 primary attempt(s)litellm-router, fallback=yes, streaming=yeslitellm.Router with one primary and one fallback deployment per scenario, num_retries=2, timeout=30 s, context_window_fallbacks on. Defaults otherwise.
params: {"num_retries": 2, "timeout_s": 30, "context_window_fallbacks": true}
S01 429 with Retry-After — pass rescued
retried pass: 2 primary attempt(s)retry_after_honoured pass: gap 2.74sno_hammer pass: 1 attempt(s) in first 1.5sS02 429 without Retry-After, persistent — pass rescued
no_hammer pass: 3 attempt(s) in first 3sbounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 5.3s (success)S03 500 once, then healthy — pass rescued
no_hang pass: ended in 1.7s (success)same_provider_retry pass: 2 primary attempt(s)S04 503 persistent — pass rescued
bounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 5.8s (success)S05 TCP reset before headers — pass rescued
no_hang pass: ended in 1.8s (success)S06 No response at all — partial rescued; advisory: silent_route_not_retried
within_budget pass: 96.2sno_hang pass: ended in 96.2s (success)silent_route_not_retried FAIL (advisory): 3 primary attempt(s)S07 Slow first token — partial rescued; advisory: ttft_limit
no_hang pass: ended in 21.0s (success)ttft_limit FAIL (advisory): 21.0sS08 Stream cut mid-answer — safe not rescued — clean error: APIConnectionError: litellm.APIConnectionError: APIConnectionError: OpenAIExcept
no_truncated_success pass: no success claimed (error)no_hang pass: ended in 0.7s (error)S09 Stream stalls — safe not rescued — clean error: APIConnectionError: litellm.APIConnectionError: APIConnectionError: OpenAIExcept
bounded pass: 30.3sno_truncated_success pass: no success claimed (error)alive_after pass: healthy call succeeded afterwardsno_hang pass: ended in 30.3s (error)S10 Malformed chunk in the stream — safe not rescued — clean error: APIConnectionError: litellm.APIConnectionError: APIConnectionError: OpenAIExcept
alive_after pass: healthy call succeeded afterwardsno_garbage pass: no success claimed (error)no_hang pass: ended in 0.2s (error)recovered_in_place FAIL (advisory): errorS11 Stream ends without [DONE] — pass rescued
prompt_finalise pass: 1.0susage_reported pass: usage = {'completion_tokens': 60, 'prompt_tokens': 21, 'total_tokens': 81, 'completion_tokens_details': {'accepted_prediction_tokens': None, 'audio_tokens': None, 'reasoning_tokens': 0, 'rejected_prediction_tokens': None, 'text_tokens': None, 'image_tokens': None, 'video_tokens': None}, 'prompt_tokens_details': None}S12 Context length exceeded — pass rescued
no_blind_retry pass: 1 primary attempt(s)bounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)S13 Content-filter rejection — partial answered by routing around the rejection
no_same_model_retry pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)fallback_attempted pass: 1 fallback attempt(s)S14 Fallback target rejects prefill — safe not rescued — clean error: APIConnectionError: litellm.APIConnectionError: APIConnectionError: OpenAIExcept
no_prefill_masking pass: 0 prefill rejection(s) at the fallback; final outcome errorno_truncated_success pass: no success claimed (error)S15 Primary flaps: down, then healthy — pass rescued
no_hang pass: 0 hung request(s) of 20served_during_outage pass: 11/11 requests answered while primary was downrecovered pass: primary answered again +2.2s after recoveryprobe_discipline info: 33 primary attempt(s) for 11 request(s) during the outageS16 Fail-over accounting — pass rescued
reports_fallback_model pass: response.model = 'fb-ok'usage_present pass: usage = {'completion_tokens': 60, 'prompt_tokens': 21, 'total_tokens': 81, 'completion_tokens_details': {'accepted_prediction_tokens': None, 'audio_tokens': None, 'reasoning_tokens': 0, 'rejected_prediction_tokens': None, 'text_tokens': None, 'image_tokens': None, 'video_tokens': None}, 'prompt_tokens_details': None}bounded_attempts pass: 3 primary attempt(s)openai-python, fallback=no, streaming=yesThe official SDK with its defaults except timeout (600 s → 30 s so a run stays bounded). No fallback concept, so fail-over checks are n/a.
params: {"max_retries": 2, "timeout_s": 30}
S01 429 with Retry-After — pass rescued
retried pass: 2 primary attempt(s)retry_after_honoured pass: gap 2.01sno_hammer pass: 1 attempt(s) in first 1.5sS02 429 without Retry-After, persistent — safe not rescued — clean error: RateLimitError: Error code: 429 - {'error': {'message': 'Rate limit reached for
no_hammer pass: 3 attempt(s) in first 3sbounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 1.4s (error)S03 500 once, then healthy — pass rescued
no_hang pass: ended in 1.5s (success)same_provider_retry pass: 2 primary attempt(s)S04 503 persistent — safe not rescued — clean error: InternalServerError: Error code: 503 - {'error': {'message': 'The engine is curr
bounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 1.4s (error)S05 TCP reset before headers — pass rescued
no_hang pass: ended in 1.4s (success)S06 No response at all — safe not rescued — clean error: APITimeoutError: Request timed out.
within_budget pass: 91.6sno_hang pass: ended in 91.6s (error)silent_route_not_retried FAIL (advisory): 3 primary attempt(s)S07 Slow first token — pass rescued
no_hang pass: ended in 21.0s (success)ttft_limit n/a: n/a — system declares no fallbackS08 Stream cut mid-answer — safe not rescued — clean error: RemoteProtocolError: peer closed connection without sending complete message bod
no_truncated_success pass: no success claimed (error)no_hang pass: ended in 0.6s (error)S09 Stream stalls — safe not rescued — clean error: ReadTimeout:
bounded pass: 30.3sno_truncated_success pass: no success claimed (error)alive_after pass: healthy call succeeded afterwardsno_hang pass: ended in 30.3s (error)S10 Malformed chunk in the stream — safe not rescued — clean error: JSONDecodeError: Expecting value: line 1 column 94 (char 93)
alive_after pass: healthy call succeeded afterwardsno_garbage pass: no success claimed (error)no_hang pass: ended in 0.2s (error)recovered_in_place FAIL (advisory): errorS11 Stream ends without [DONE] — pass rescued
prompt_finalise pass: 1.0susage_reported pass: usage = {'completion_tokens': 60, 'prompt_tokens': 21, 'total_tokens': 81, 'completion_tokens_details': None, 'prompt_tokens_details': None}S12 Context length exceeded — safe not rescued — clean error: BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum c
no_blind_retry pass: 1 primary attempt(s)bounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 0.0s (error)S13 Content-filter rejection — pass rescued
no_same_model_retry pass: 1 primary attempt(s)no_hang pass: ended in 0.0s (error)fallback_attempted info: 0 fallback attempt(s)S14 Fallback target rejects prefill — safe not rescued — clean error: RemoteProtocolError: peer closed connection without sending complete message bod
no_prefill_masking pass: 0 prefill rejection(s) at the fallback; final outcome errorno_truncated_success pass: no success claimed (error)S15 Primary flaps: down, then healthy — safe not rescued — clean error: InternalServerError: Error code: 503 - {'error': {'message': 'The engine is curr
no_hang pass: 0 hung request(s) of 20served_during_outage n/a: n/a — system declares no fallbackrecovered n/a: n/a — system declares no fallbackprobe_discipline info: 45 primary attempt(s) for 15 request(s) during the outageS16 Fail-over accounting — safe not rescued — clean error: InternalServerError: Error code: 503 - {'error': {'message': 'The engine is curr
reports_fallback_model n/a: n/a — system declares no fallbackusage_present n/a: n/a — system declares no fallbackbounded_attempts pass: 3 primary attempt(s)endpoint, fallback=yes, streaming=yesOpen-source Portkey AI Gateway. Everything is per request in x-portkey-config: a fallback strategy with two openai targets on custom_host = the wall (host must include /v1). Baseline: primary target retry attempts 2 on 429/5xx with use_retry_after_header, request_timeout 30 s; fallback on any non-2xx (strategy.on_status_codes omitted). Per source, a cut upstream stream is closed to the client with no error event and no retry.
params: {"base_url": "http://127.0.0.1:8787/v1", "api_key": "failoverbench", "headers_json": {"x-portkey-config": {"strategy": {"mode": "fallback"}, "request_timeout": 30000, "targets": [{"provider": "openai", "api_key": "failoverbench-primary", "custom_host": "http://host.docker.internal:8401/v1", "override_params": {"model": "{primary}"}, "retry": {"attempts": 2, "on_status_codes": [429, 500, 502, 503, 504], "use_retry_after_header": true}}, {"provider": "openai", "api_key": "failoverbench-fallback", "custom_host": "http://host.docker.internal:8401/v1", "override_params": {"model": "{fallback}"}}]}}}
S01 429 with Retry-After — pass rescued
retried pass: 2 primary attempt(s)retry_after_honoured pass: gap 2.02sno_hammer pass: 1 attempt(s) in first 1.5sS02 429 without Retry-After, persistent — pass rescued
no_hammer pass: 2 attempt(s) in first 3sbounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 4.0s (success)S03 500 once, then healthy — pass rescued
no_hang pass: ended in 2.0s (success)same_provider_retry pass: 2 primary attempt(s)S04 503 persistent — pass rescued
bounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 4.0s (success)S05 TCP reset before headers — pass rescued
no_hang pass: ended in 2.0s (success)S06 No response at all — pass rescued
within_budget pass: 31.0sno_hang pass: ended in 31.0s (success)silent_route_not_retried pass: 1 primary attempt(s)S07 Slow first token — partial rescued; advisory: ttft_limit
no_hang pass: ended in 21.0s (success)ttft_limit FAIL (advisory): 21.0sS08 Stream cut mid-answer — fail truncated answer presented as success
no_truncated_success FAIL: success with 0/60 wordsno_hang pass: ended in 0.7s (success)S09 Stream stalls — fail truncated answer presented as success
bounded pass: 30.3sno_truncated_success FAIL: success with 0/60 wordsalive_after pass: healthy call succeeded afterwardsno_hang pass: ended in 30.3s (success)S10 Malformed chunk in the stream — pass rescued
alive_after pass: healthy call succeeded afterwardsno_garbage pass: complete answerno_hang pass: ended in 1.0s (success)recovered_in_place pass: complete answerS11 Stream ends without [DONE] — pass rescued
prompt_finalise pass: 1.0susage_reported pass: usage = {'prompt_tokens': 21, 'completion_tokens': 60, 'total_tokens': 81}S12 Context length exceeded — pass rescued
no_blind_retry pass: 1 primary attempt(s)bounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)S13 Content-filter rejection — partial answered by routing around the rejection
no_same_model_retry pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)fallback_attempted pass: 1 fallback attempt(s)S14 Fallback target rejects prefill — fail truncated answer presented as success
no_prefill_masking pass: 0 prefill rejection(s) at the fallback; final outcome successno_truncated_success FAIL: success with 0/60 wordsS15 Primary flaps: down, then healthy — pass rescued
no_hang pass: 0 hung request(s) of 20served_during_outage pass: 15/15 requests answered while primary was downrecovered pass: primary answered again +0.1s after recoveryprobe_discipline info: 45 primary attempt(s) for 15 request(s) during the outageS16 Fail-over accounting — pass rescued
reports_fallback_model pass: response.model = 'fb-ok'usage_present pass: usage = {'prompt_tokens': 21, 'completion_tokens': 60, 'total_tokens': 81}bounded_attempts pass: 3 primary attempt(s)reference, fallback=yes, streaming=yesA small client written to pass: honours Retry-After, backs off with jitter, retries 5xx/resets/cut streams up to 3 times, fails over on timeouts and context-length errors, never retries content-filter 400s, restarts without prefill. It has no circuit breaker, so S15 shows what stateless retrying looks like.
params: {"max_attempts": 3, "base_backoff_s": 0.5, "stall_limit_s": 10, "ttft_limit_s": 10, "connect_timeout_s": 5}
S01 429 with Retry-After — pass rescued
retried pass: 2 primary attempt(s)retry_after_honoured pass: gap 2.00sno_hammer pass: 1 attempt(s) in first 1.5sS02 429 without Retry-After, persistent — pass rescued
no_hammer pass: 3 attempt(s) in first 3sbounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 2.4s (success)S03 500 once, then healthy — pass rescued
no_hang pass: ended in 1.5s (success)same_provider_retry pass: 2 primary attempt(s)S04 503 persistent — pass rescued
bounded_attempts pass: 3 primary attempt(s)no_hang pass: ended in 2.2s (success)S05 TCP reset before headers — pass rescued
no_hang pass: ended in 1.5s (success)S06 No response at all — pass rescued
within_budget pass: 11.0sno_hang pass: ended in 11.0s (success)silent_route_not_retried pass: 1 primary attempt(s)S07 Slow first token — pass rescued
no_hang pass: ended in 11.0s (success)ttft_limit pass: 11.0sS08 Stream cut mid-answer — pass rescued
no_truncated_success pass: complete answerno_hang pass: ended in 4.3s (success)S09 Stream stalls — pass rescued
bounded pass: 11.3sno_truncated_success pass: complete answeralive_after pass: healthy call succeeded afterwardsno_hang pass: ended in 11.3s (success)S10 Malformed chunk in the stream — pass rescued
alive_after pass: healthy call succeeded afterwardsno_garbage pass: complete answerno_hang pass: ended in 1.0s (success)recovered_in_place pass: complete answerS11 Stream ends without [DONE] — pass rescued
prompt_finalise pass: 1.0susage_reported pass: usage = {'prompt_tokens': 21, 'completion_tokens': 60, 'total_tokens': 81}S12 Context length exceeded — pass rescued
no_blind_retry pass: 1 primary attempt(s)bounded_attempts pass: 1 primary attempt(s)no_hang pass: ended in 1.0s (success)S13 Content-filter rejection — pass rescued
no_same_model_retry pass: 1 primary attempt(s)no_hang pass: ended in 0.0s (error)fallback_attempted info: 0 fallback attempt(s)S14 Fallback target rejects prefill — pass rescued
no_prefill_masking pass: 0 prefill rejection(s) at the fallback; final outcome successno_truncated_success pass: complete answerS15 Primary flaps: down, then healthy — pass rescued
no_hang pass: 0 hung request(s) of 20served_during_outage pass: 15/15 requests answered while primary was downrecovered pass: primary answered again +0.0s after recoveryprobe_discipline info: 45 primary attempt(s) for 15 request(s) during the outageS16 Fail-over accounting — pass rescued
reports_fallback_model pass: response.model = 'fb-ok'usage_present pass: usage = {'prompt_tokens': 21, 'completion_tokens': 60, 'total_tokens': 81}bounded_attempts pass: 3 primary attempt(s)