How the blind test works
blindtest.ai exists because model names carry a halo. The same answer reads smarter under a famous logo. So we remove the logo first and let you judge the substance.
The protocol
One identical prompt. Every round is a single task, sent verbatim to every model in the field through the same API harness — one shot, no retries, no per-model prompt tuning. Local open-weight models run on our own Brain cluster hardware; cloud models are called over their official APIs.
Everything stays on the page. Answers are published raw: typos, wrong answers, refusals and all. If a model times out or errors, that is recorded as a DNF (did not finish) and shown — infrastructure failures are labelled as such so a model isn't blamed for a broken node.
Two models are currently blocked by our own setup. On the 25 runtime-verified simulation rounds, qwen3:8b and gemma4:26b returned an answer zero times. We measured why, on 2026-07-28: both spend their entire token budget on internal reasoning and leave the answer field empty, and our gateway refuses the larger budgets that might let them finish. That is a limit of our harness, not a result about those models. Their cards still read DNF — we do not rewrite measurements — but each affected card is marked harness with the reason, and their model pages say so plainly. Read their DNFs as “we could not ask them properly”, not as “they could not do it”.
Why line counts alone would mislead. The simulation rounds ask for at most 120 lines. That limit can be satisfied by simply leaving out line breaks, and a handful of answers did exactly that — one packed 1,859 characters into two lines. Those carry a MINIFIED label, because otherwise a wall of compressed code would look more concise than a readable answer that honestly reports 130 lines. From R81 onwards the task states a character budget of 7,000 instead, which cannot be gamed by formatting. We did not re-judge the earlier rounds against it: those models were told “120 lines”, so that is the rule they are measured by. The budget is not arbitrary — every one of the 35 frontier reference answers fits inside it, the longest using 6,745 characters.
What "blind" means here. It describes the scoring, not the presentation: every answer was reviewed without knowing which model wrote it, and the rubric scores you see were fixed under those conditions. The published page then names every model on its own card, next to its raw answer, its score and its measured runtime. The card order is a deterministic shuffle per round, so the sequence carries no ranking. Until 2026-07-28 the names were hidden behind a reveal button — that turned the site into a guessing game for visitors, which was never the point.
Scoring
Where a task has an objectively checkable result, each round defines a gold answer and a written rubric before the answers are judged. Answers are scored 0–10 against that rubric, blind against the gold answer. Runtime tasks (the physics and algorithm rounds) are additionally verified by actually executing the generated code in a headless browser.
Creative and style tasks — slogans, register shifts, idiom translation — are deliberately unscored: several answers can be equally right, and pretending otherwise would be false precision. For those rounds the gold answer is a reference, not a verdict.
We do not publish synthetic aggregate benchmarks, and you will not find claims like "model X is 23% better" here. The comparisons are qualitative and task-by-task; the raw material for your own judgement is always on the page.
Limits, honestly
This is a small-sample, hand-curated test suite — not a substitute for large benchmarks. Single-shot results have variance; a model can flub a task it would usually solve. Many tasks are currently written in German (a deliberate stress test for models tuned on English); those rounds are labelled. Latency numbers compare our specific hardware and network paths, not the models in the abstract.
Transparency
Content on this site is created with the help of AI and reviewed editorially. The models being tested are also the kind of systems that help produce and score the write-ups — we consider that a feature of the format, and we disclose it on every page. Tasks, gold answers, rubrics and raw model output are kept as data files and artifacts, so rounds can be regenerated and checked.
Questions, corrections, or a task you'd like to see run blind? See the imprint for contact details.