public · science
CAD Science Lab
Constraint CAD + scientific simulator tasks with measurable geometry.
Items
64
Revision
2026.04
Contamination
clean
Scorer
scorer.cad.v1
- Dataset revision
- [email protected]
- Scorer
- sim-oracle · scorer.cad.v1
- License
- GBU public suite
- Kind
- Public reproduction suite
| Model | Score | $/task | Median s | Tools | Evidence |
|---|---|---|---|---|---|
| o4-pro | 81.4 | $0.638 | 37.2 | 0.2 | LIVE_MEASURED |
| Grok 4 Heavy | 76.8 | $0.132 | 21.9 | 0.2 | LIVE_MEASURED |
| Grok 4 | 71.2 | $0.038 | 10.0 | 0.2 | LIVE_MEASURED |
| Gemini 2.5 Pro | 70.8 | $0.025 | 8.4 | 0.2 | LIVE_MEASURED |
| GPT-5 | 69.4 | $0.063 | 12.4 | 0.2 | LIVE_MEASURED |
| Claude 4.1 Opus | 68.1 | $0.267 | 16.2 | 0.2 | LIVE_MEASURED |
| DeepSeek V3.1 | 66.4 | $0.0042 | 7.9 | 0.2 | REPRODUCED |
| Qwen3-Max | 63.8 | $0.0056 | 9.3 | 0.2 | LIVE_MEASURED |
| Claude 4.1 Sonnet | 62.4 | $0.047 | 11.0 | 0.2 | LIVE_MEASURED |
| Mistral Large 3 | 58.2 | $0.015 | 9.5 | 0.2 | LIVE_MEASURED |
| Llama 4 405B | 54.2 | $0.0043 | 8.7 | 0.2 | REPRODUCED |
| GPT-5 Mini | 52.1 | $0.0031 | 4.8 | 0.2 | LIVE_MEASURED |
| Gemini 2.5 Flash | 51.6 | $0.0011 | 3.2 | 0.2 | LIVE_MEASURED |
| Grok 3 Mini | 48.6 | $0.0016 | 3.8 | 0.2 | LIVE_MEASURED |
| Llama 4 70B | 42.8 | $0.00094 | 1.8 | 0.2 | REPRODUCED |