Reliability
Correctness, replay, and whether the endpoint stays up
Availability is not intelligence. Invalid JSON, broken tool calls, refusal spikes, and score variance are first-class. Provider-reported uptime is never mixed into GBU availability.
Canary detections
- POSSIBLE_MODEL_CHANGEGPT-5 · 2026-08-19
Same model string. Canary drift after an undeclared August deployment. Coding −2.8, science −4.0, refusals up.
Δcap -4.1Δlat 12%Δrefuse 2.2Δstyle 0.18 - NOISEGrok 4 · 2026-08-27
Within daily variance band. No adapter or tokenizer shift.
Δcap -0.6Δlat 4%Δrefuse 0.1Δstyle 0.04 - POSSIBLE_MODEL_CHANGEGemini 2.5 Pro · 2026-08-31
Long-context improved; tool-call latency dropped. Possible silent serving change.
Δcap 1.4Δlat -8%Δrefuse -0.4Δstyle 0.11 - CONFIRMEDClaude 4.1 Opus · 2026-09-02
Provider changelog matched canary. Computer-use stack bumped; TTFT slightly worse.
Δcap 0.8Δlat 6%Δrefuse 0.9Δstyle 0.07 - POSSIBLE_MODEL_CHANGEGemini 2.5 Flash · 2026-09-03
Structured JSON invalid-rate ticked up. Speed unchanged. Watching.
Δcap -1.2Δlat -3%Δrefuse 0.2Δstyle 0.22
| Endpoint | Avail | Error | Refusal | Invalid JSON | Broken tools | Replay match | Score σ | Evidence |
|---|---|---|---|---|---|---|---|---|
| Gemini 2.5 Flash Google API | 99.97% | 0.09% | 0.9% | 4.6% | 0.7% | 91.2% | 1.8 | LIVE_MEASURED |
| Grok 3 Mini xAI API | 99.96% | 0.12% | 0.9% | 4.6% | 0.7% | 91.2% | 1.8 | LIVE_MEASURED |
| Gemini 2.5 Pro Google API | 99.95% | 0.16% | 0.9% | 4.6% | 0.7% | 91.2% | 1.8 | LIVE_MEASURED |
| Grok 4 xAI API | 99.94% | 0.21% | 0.9% | 1.8% | 0.7% | 91.2% | 1.8 | LIVE_MEASURED |
| GPT-5 Mini OpenAI API | 99.93% | 0.18% | 1.6% | 4.6% | 0.7% | 91.2% | 1.8 | LIVE_MEASURED |
| Grok 4 Heavy xAI API | 99.91% | 0.34% | 0.9% | 1.8% | 0.7% | 96.4% | 1.1 | LIVE_MEASURED |
| GPT-5 OpenAI API | 99.91% | 0.28% | 1.6% | 1.8% | 0.7% | 91.2% | 1.8 | LIVE_MEASURED |
| Claude 4.1 Sonnet Anthropic API | 99.90% | 0.19% | 2.8% | 1.8% | 0.7% | 91.2% | 1.8 | LIVE_MEASURED |
| GPT-5 Azure OpenAI | 99.88% | 0.41% | 0.9% | 1.8% | 0.7% | 91.2% | 1.8 | LIVE_MEASURED |
| Claude 4.1 Opus Anthropic API | 99.86% | 0.24% | 2.8% | 1.8% | 0.7% | 91.2% | 1.8 | LIVE_MEASURED |
| Llama 4 70B Groq LPU | 99.84% | 0.14% | 0.9% | 4.6% | 2.4% | 91.2% | 1.8 | LIVE_MEASURED |
| Llama 4 405B Groq LPU | 99.81% | 0.27% | 0.9% | 1.8% | 2.4% | 91.2% | 1.8 | LIVE_MEASURED |
| Llama 4 70B Together | 99.75% | 0.22% | 0.9% | 4.6% | 2.4% | 91.2% | 1.8 | LIVE_MEASURED |
| o4-pro OpenAI API | 99.74% | 0.62% | 1.6% | 1.8% | 0.7% | 96.4% | 1.1 | LIVE_MEASURED |
| Llama 4 405B Together | 99.72% | 0.38% | 0.9% | 1.8% | 2.4% | 91.2% | 1.8 | LIVE_MEASURED |
| Qwen3-Max Together | 99.70% | 0.36% | 0.9% | 1.8% | 0.7% | 91.2% | 1.8 | LIVE_MEASURED |
| Llama 4 405B Fireworks | 99.68% | 0.33% | 0.9% | 1.8% | 2.4% | 91.2% | 1.8 | LIVE_MEASURED |
| Mistral Large 3 Mistral API | 99.63% | 0.31% | 0.9% | 1.8% | 0.7% | 91.2% | 1.8 | LIVE_MEASURED |
| DeepSeek V3.1 DeepSeek API | 99.41% | 0.55% | 0.9% | 1.8% | 0.7% | 91.2% | 1.8 | LIVE_MEASURED |
| Llama 4 405B Self-host H100 | 99.20% | 0.51% | 0.9% | 1.8% | 2.4% | 91.2% | 1.8 | REPRODUCED |
| Llama 4 405B Self-host B200 | 99.20% | 0.29% | 0.9% | 1.8% | 2.4% | 91.2% | 1.8 | REPRODUCED |
21 endpoints. Deterministic replay uses temperature 0 and a frozen tool policy.