A lab, not a press table
Benchmark definition is separated from the model adapter. A run is an immutable tuple of hashes, not a screenshot of a score.
Effective work rate = correct completed tasks / (dollar · second). Useful action efficiency = successful task consequence / (actions + tool calls + retries).
Public suites establish comparability: GPQA Diamond, SWE-bench Verified, Terminal Bench, OSWorld, licensed FrontierMath, LiveCodeBench, MMMU, LongContext-128k, Agent Episodes, Structured JSON, CAD Science Lab. A rotating private vault ships new items, seeds, worlds, layouts, repositories, and factual probes. Providers never receive the answer key. Retired packs can be published for audit.
ARC is carried as a saturated reference, not a headline. If a number cannot be clicked into a run, it does not belong on a rank.
Immutable run
- benchmark_hash
- model_adapter_hash
- run_config_hash
- environment_hash
- timestamp · region
- → per-item transcripts → aggregate
Evidence badges — never merged silently
- LIVE_MEASURED
Captured by GBU workers against the live endpoint.
- REPRODUCED
Independent reproduction on the lab cluster or a second worker.
- OFFICIAL_PROVIDER
Published by the provider. Not independently merged.
- THIRD_PARTY
External lab or vendor figure. Shown separately.
- SELF_REPORTED
Vendor claim. Displayed, never mixed into GBU ranks.
- UNVERIFIED
Insufficient evidence. Do not treat as measured.
Backend shape
The gateway normalizes generate, stream, tool_call, embed, image, audio without erasing provider metadata. Temperature, reasoning setting, and token accounting live on the adapter, not the benchmark.
What we sample
DNS/connect, TLS, request accepted, TTFT, inter-token timing, total generation, input/output/reasoning tokens, HTTP status, provider request id, rate-limit events. Regions: US-East, US-West, Europe, Asia, Australia.