public · multimodal
MMMU
College-level multimodal understanding across disciplines.
Items
1730
Revision
2025.09
Contamination
watched
Scorer
scorer.mmmu.v3
- Dataset revision
- [email protected]
- Scorer
- choice+open · scorer.mmmu.v3
- License
- CC BY-NC 4.0
- Kind
- Public reproduction suite
| Model | Score | $/task | Median s | Tools | Evidence |
|---|---|---|---|---|---|
| Gemini 2.5 Pro | 88.4 | $0.025 | 8.8 | 0.2 | LIVE_MEASURED |
| GPT-5 | 84.8 | $0.063 | 13.0 | 0.2 | LIVE_MEASURED |
| Gemini 2.5 Flash | 82.6 | $0.0011 | 3.4 | 0.2 | LIVE_MEASURED |
| Claude 4.1 Opus | 79.4 | $0.267 | 16.7 | 0.2 | LIVE_MEASURED |
| GPT-5 Mini | 78.2 | $0.0031 | 5.2 | 0.2 | LIVE_MEASURED |
| Grok 4 Heavy | 77.0 | $0.132 | 22.0 | 0.2 | LIVE_MEASURED |
| Grok 4 | 76.2 | $0.038 | 10.2 | 0.2 | LIVE_MEASURED |
| Claude 4.1 Sonnet | 76.1 | $0.047 | 11.4 | 0.2 | LIVE_MEASURED |
| Qwen3-Max | 75.2 | $0.0056 | 9.6 | 0.2 | LIVE_MEASURED |
| o4-pro | 73.6 | $0.638 | 36.4 | 0.2 | LIVE_MEASURED |
| Mistral Large 3 | 73.4 | $0.015 | 9.9 | 0.2 | LIVE_MEASURED |
| Llama 4 405B | 71.6 | $0.0043 | 9.1 | 0.2 | REPRODUCED |
| Llama 4 70B | 66.8 | $0.00094 | 1.9 | 0.2 | REPRODUCED |
| Grok 3 Mini | 66.4 | $0.0016 | 4.0 | 0.2 | LIVE_MEASURED |
| DeepSeek V3.1 | 40.2 | $0.0042 | 7.3 | 0.2 | REPRODUCED |