public · agents
Agent Episodes
Browser, repo, spreadsheet, API, and simulator episodes. human_io tracked.
Items
96
Revision
2026.08
Contamination
clean
Scorer
scorer.agent.v3
- Dataset revision
- [email protected]
- Scorer
- task-consequence · scorer.agent.v3
- License
- GBU public suite
- Kind
- Public reproduction suite
| Model | Score | $/task | Median s | Tools | Evidence |
|---|---|---|---|---|---|
| Claude 4.1 Opus | 76.8 | $0.328 | 53.1 | 3.8 | LIVE_MEASURED |
| Claude 4.1 Sonnet | 71.4 | $0.058 | 36.0 | 3.4 | LIVE_MEASURED |
| Grok 4 Heavy | 71.2 | $0.150 | 69.1 | 2.8 | LIVE_MEASURED |
| GPT-5 | 70.1 | $0.081 | 39.8 | 3.0 | LIVE_MEASURED |
| Gemini 2.5 Pro | 68.8 | $0.029 | 26.8 | 2.9 | LIVE_MEASURED |
| Grok 4 | 67.5 | $0.048 | 31.7 | 3.2 | LIVE_MEASURED |
| o4-pro | 66.2 | $0.695 | 114.1 | 2.2 | LIVE_MEASURED |
| Mistral Large 3 | 64.6 | $0.022 | 30.9 | 2.9 | LIVE_MEASURED |
| DeepSeek V3.1 | 63.4 | $0.0051 | 25.0 | 2.7 | REPRODUCED |
| Qwen3-Max | 62.1 | $0.0070 | 29.6 | 2.8 | LIVE_MEASURED |
| Llama 4 405B | 61.2 | $0.0070 | 28.4 | 2.6 | REPRODUCED |
| Gemini 2.5 Flash | 59.4 | $0.0016 | 10.3 | 2.2 | LIVE_MEASURED |
| GPT-5 Mini | 58.6 | $0.0044 | 15.7 | 2.4 | LIVE_MEASURED |
| Grok 3 Mini | 54.1 | $0.0025 | 12.3 | 2.1 | LIVE_MEASURED |
| Llama 4 70B | 52.6 | $0.0016 | 5.9 | 2.0 | REPRODUCED |