Which model actually delivers?
Every number below comes from the same 110 published one-shot rounds — you can open any round and read the raw answer behind each cell. We keep the two things a model can be good at as two separate axes: writing a correct short answer, and writing code that actually runs. We do not blend them into a fantasy index.
Axis 1 — Text rounds, blind rubric scores
Scored against a written gold answer and rubric, without knowing which model wrote which answer (the reveal happens after judging). 1–10 per round; the bracket count shows how thick the base is.
| Model | Avg | Scored | Knowledge | Language | Creative | Structure | Phys-Lab |
|---|---|---|---|---|---|---|---|
| gemma4:26b | 10.00 | 19 | 10.0 | — | — | 10.0 | — |
| qwq:32b | 10.00 | 5 | 10.0 | — | — | — | — |
| qwen3.5:9b | 9.22 | 23 | 9.4 | — | 10.0 | 9.0 | — |
| qwen3-coder:30b | 9.17 | 23 | 10.0 | — | 9.0 | 10.0 | — |
| qwen3:14b | 8.91 | 23 | 9.6 | — | 10.0 | 10.0 | — |
| qwen3:8b | 8.83 | 12 | 10.0 | — | — | 8.0 | — |
| mistral-small:24b | 8.76 | 17 | 7.9 | — | 10.0 | 8.0 | — |
| deepseek-r1:32b | 8.60 | 5 | 8.6 | — | — | — | — |
| deepseek-r1:14b | 8.00 | 17 | 8.4 | — | 6.0 | 7.0 | — |
| llama3.1:8b | 5.94 | 18 | 6.0 | — | 7.0 | 8.0 | — |
| command-r:35b | 3.20 | 5 | 3.2 | — | — | — | — |
Axis 2 — Code sims: does it run?
Share of code-round answers whose simulation is runtime-verified OK (renders, animates, no JavaScript error). Harness failures are excluded from the base — a broken cluster node is our fault, not the model's. The failure column shows what went wrong with the rest.
| Model | Runnable | OK / measured | Failure modes |
|---|---|---|---|
| qwen3.5:9b | 36% | 29/80 | ERR 25 · STAT 9 · BLANK 16 · HANG 1 |
| mistral-small:24b | 33% | 26/80 | ERR 24 · STAT 11 · BLANK 8 · HANG 3 · DNF 8 |
| qwen3:14b | 28% | 22/80 | ERR 37 · STAT 9 · BLANK 12 |
| deepseek-r1:32b | 26% | 21/80 | ERR 32 · STAT 7 · BLANK 18 · HANG 1 · DNF 1 |
| command-r:35b | 26% | 21/80 | ERR 29 · STAT 10 · BLANK 8 · HANG 3 · DNF 9 |
| qwen3-coder:30b | 25% | 20/80 | ERR 33 · STAT 8 · BLANK 18 · HANG 1 |
| llama3.1:8b | 14% | 11/80 | ERR 37 · STAT 16 · BLANK 8 · HANG 3 · DNF 5 |
| deepseek-r1:14b | 13% | 10/80 | ERR 53 · STAT 4 · BLANK 9 · HANG 1 · DNF 3 |
| qwq:32b | 9% | 7/80 | ERR 4 · STAT 41 · BLANK 15 · DNF 13 |
| qwen3:8b thin base | 50% | 1/2 | BLANK 1 |
| gemma4:26b thin base | 0% | 0/1 | BLANK 1 |
Trend — runnable share per round band
The permanent test keeps growing; this row shows each model's runnable-% over consecutive bands of ten code rounds (a dot means the model was not in the field for that band).
| Model | r56–r65 | r66–r75 | r76–r85 | r86–r95 | r96–r105 | r106–r115 | r116–r125 | r126–r135 | r136–r145 |
|---|---|---|---|---|---|---|---|---|---|
| qwen3.5:9b | 30 | 30 | 20 | 40 | 80 | 20 | 20 | 60 | 40 |
| mistral-small:24b | 10 | 50 | 60 | 20 | 30 | 40 | 10 | 20 | 60 |
| qwen3:14b | 10 | 40 | 50 | 10 | 20 | 30 | 30 | 40 | 20 |
| deepseek-r1:32b | 20 | 10 | 50 | 20 | 10 | 40 | 30 | 0 | 60 |
| command-r:35b | 40 | 30 | 20 | 20 | 30 | 50 | 20 | 0 | 0 |
| qwen3-coder:30b | 30 | 30 | 20 | 0 | 20 | 30 | 20 | 40 | 60 |
| llama3.1:8b | 0 | 20 | 10 | 10 | 20 | 20 | 20 | 0 | 20 |
| deepseek-r1:14b | 10 | 10 | 20 | 10 | 10 | 20 | 10 | 0 | 20 |
| qwq:32b | 0 | 0 | 0 | 10 | 20 | 20 | 20 | 0 | 0 |
Frontier reference — outside the blind field
Answered the identical prompts, collected through context-free one-shot agents rather than the cluster harness — deliberately unscored on text and shown separately here so they cannot inflate the field.
| Model | Runnable | OK / measured | Failure modes |
|---|---|---|---|
| Claude Sonnet 5 | 100% | 80/80 | — |
| Claude Fable 5 | 100% | 41/41 | — |
| Claude Opus 5 | 100% | 55/55 | — |
| GPT-5.5 (Codex) | 96% | 77/80 | ERR 1 · STAT 1 · BLANK 1 |
| Claude Haiku 4.5 | 80% | 63/79 | ERR 5 · STAT 8 · BLANK 2 · HANG 1 |
Generated 2026-07-31 from the published rounds. No votes, no private variants, no composite score — how we measure.