public · longContext
LongContext-128k
Needle, multi-hop, and argument tracking at 32k–128k.
Items
120
Revision
2026.07
Contamination
clean
Scorer
scorer.lc.v1
- Dataset revision
- [email protected]
- Scorer
- needle+reason · scorer.lc.v1
- License
- GBU public suite
- Kind
- Public reproduction suite
| Model | Score | $/task | Median s | Tools | Evidence |
|---|---|---|---|---|---|
| Gemini 2.5 Pro | 94.1 | $0.025 | 9.0 | 0.2 | LIVE_MEASURED |
| Gemini 2.5 Flash | 91.4 | $0.0011 | 3.5 | 0.2 | LIVE_MEASURED |
| GPT-5 | 90.2 | $0.063 | 13.1 | 0.2 | LIVE_MEASURED |
| Claude 4.1 Opus | 88.5 | $0.267 | 17.1 | 0.2 | LIVE_MEASURED |
| Grok 4 Heavy | 87.4 | $0.132 | 22.6 | 0.2 | LIVE_MEASURED |
| Claude 4.1 Sonnet | 86.8 | $0.047 | 11.7 | 0.2 | LIVE_MEASURED |
| Grok 4 | 86.0 | $0.038 | 10.4 | 0.2 | LIVE_MEASURED |
| Qwen3-Max | 85.6 | $0.0056 | 9.9 | 0.2 | LIVE_MEASURED |
| GPT-5 Mini | 84.0 | $0.0031 | 5.3 | 0.2 | LIVE_MEASURED |
| o4-pro | 82.8 | $0.638 | 37.4 | 0.2 | LIVE_MEASURED |
| Mistral Large 3 | 81.1 | $0.015 | 10.1 | 0.2 | LIVE_MEASURED |
| Llama 4 405B | 80.2 | $0.0043 | 9.4 | 0.2 | REPRODUCED |
| DeepSeek V3.1 | 79.2 | $0.0042 | 8.2 | 0.2 | REPRODUCED |
| Grok 3 Mini | 78.5 | $0.0016 | 4.1 | 0.2 | LIVE_MEASURED |
| Llama 4 70B | 74.8 | $0.00094 | 2.0 | 0.2 | REPRODUCED |