Agents
Task consequence, not a chat transcript
Environments speak reset → step → close. We count success, invalid actions, tool calls, retries, wall time, tokens, cost, environment resets — and human_io, because a hidden operator corrupts an autonomy result.
Best episode success
76.8%
Claude 4.1 Opus
Useful action efficiency
0.046
success / (actions+tools+retries)
human_io
0
GBU runs are unattended
Env contract
reset/step
browser · terminal · CAD · sim
Environments on the actuator
Web browser
OSWorld-class desktop and browser tasks
Terminal
Isolated shells, package installs, build/test
Coding repo
SWE-bench style patch → hidden tests
Spreadsheet
Formula, join, and audit workflows
Scientific sim
CAD Science Lab + numeric oracles
Private world
Rotating vault layouts, never shipped to providers
| Model | Success | Actions | Invalid | Tools | Retries | Wall | Cost | UAE | human_io | Evidence |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude 4.1 Opus Anthropic API | 76.8% | 12.2 | 1.1 | 4.4 | 0.28 | 43 s | $1.01 | 0.046 | 0 | LIVE_MEASURED |
| Claude 4.1 Sonnet Anthropic API | 71.4% | 13.1 | 1.3 | 4.0 | 0.32 | 46 s | $0.197 | 0.041 | 0 | LIVE_MEASURED |
| Grok 4 Heavy xAI API | 71.2% | 13.2 | 1.3 | 2.8 | 0.30 | 46 s | $0.403 | 0.044 | 0 | LIVE_MEASURED |
| GPT-5 OpenAI API | 70.1% | 13.4 | 1.4 | 3.0 | 0.35 | 46 s | $0.504 | 0.042 | 0 | LIVE_MEASURED |
| Gemini 2.5 Pro Google API | 68.8% | 13.6 | 1.4 | 2.9 | 0.36 | 47 s | $0.098 | 0.041 | 0 | LIVE_MEASURED |
| Grok 4 xAI API | 67.5% | 13.8 | 1.4 | 3.2 | 0.40 | 48 s | $0.178 | 0.039 | 0 | LIVE_MEASURED |
| o4-pro OpenAI API | 66.2% | 14.1 | 1.5 | 2.2 | 0.25 | 49 s | $1.49 | 0.040 | 0 | LIVE_MEASURED |
| Mistral Large 3 Mistral API | 64.6% | 14.4 | 1.5 | 2.9 | 0.42 | 37 s | $0.115 | 0.037 | 0 | LIVE_MEASURED |
| DeepSeek V3.1 DeepSeek API | 63.4% | 14.6 | 1.6 | 2.7 | 0.45 | 50 s | $0.043 | 0.036 | 0 | LIVE_MEASURED |
| Qwen3-Max Together | 62.1% | 14.8 | 1.6 | 2.8 | 0.48 | 51 s | $0.062 | 0.034 | 0 | LIVE_MEASURED |
| Llama 4 405B Together | 61.2% | 15.0 | 1.6 | 2.6 | 0.70 | 39 s | $0.053 | 0.033 | 0 | LIVE_MEASURED |
| Gemini 2.5 Flash Google API | 59.4% | 15.3 | 1.7 | 2.2 | 0.55 | 40 s | $0.022 | 0.033 | 0 | LIVE_MEASURED |
| GPT-5 Mini OpenAI API | 58.6% | 15.5 | 1.7 | 2.4 | 0.50 | 41 s | $0.034 | 0.032 | 0 | LIVE_MEASURED |
| Grok 3 Mini xAI API | 54.1% | 16.3 | 1.9 | 2.1 | 0.60 | 43 s | $0.020 | 0.029 | 0 | LIVE_MEASURED |
| Llama 4 70B Groq LPU | 52.6% | 16.5 | 1.9 | 2.0 | 0.90 | 44 s | $0.0098 | 0.027 | 0 | LIVE_MEASURED |