public · agents
Terminal Bench
End-to-end terminal tasks in isolated sandboxes.
Items
80
Revision
2026.06
Contamination
watched
Scorer
scorer.term.v2
- Dataset revision
- [email protected]
- Scorer
- env-oracle · scorer.term.v2
- License
- Apache-2.0
- Kind
- Public reproduction suite
| Model | Score | $/task | Median s | Tools | Evidence |
|---|---|---|---|---|---|
| Claude 4.1 Opus | 58.6 | $0.328 | 50.4 | 3.8 | LIVE_MEASURED |
| Grok 4 Heavy | 53.1 | $0.150 | 65.6 | 2.8 | LIVE_MEASURED |
| Claude 4.1 Sonnet | 52.4 | $0.058 | 34.0 | 3.4 | LIVE_MEASURED |
| GPT-5 | 51.4 | $0.081 | 37.6 | 3.0 | LIVE_MEASURED |
| Gemini 2.5 Pro | 49.2 | $0.029 | 25.3 | 2.9 | LIVE_MEASURED |
| Grok 4 | 48.8 | $0.048 | 30.0 | 3.2 | LIVE_MEASURED |
| o4-pro | 44.6 | $0.695 | 107.0 | 2.2 | LIVE_MEASURED |
| Mistral Large 3 | 43.4 | $0.022 | 29.0 | 2.9 | LIVE_MEASURED |
| DeepSeek V3.1 | 41.2 | $0.0051 | 23.4 | 2.7 | REPRODUCED |
| Qwen3-Max | 40.8 | $0.0070 | 27.7 | 2.8 | LIVE_MEASURED |
| Llama 4 405B | 38.2 | $0.0070 | 26.5 | 2.6 | REPRODUCED |
| GPT-5 Mini | 36.8 | $0.0044 | 14.7 | 2.4 | LIVE_MEASURED |
| Gemini 2.5 Flash | 34.6 | $0.0016 | 9.6 | 2.2 | LIVE_MEASURED |
| Grok 3 Mini | 31.2 | $0.0025 | 11.5 | 2.1 | LIVE_MEASURED |
| Llama 4 70B | 28.6 | $0.0016 | 5.4 | 2.0 | REPRODUCED |