Structure & Data · blind round

R44 · Appointment extraction to strict JSON

blind round 11/11 local models answered task written in German

Every answer below carries the name of the model that wrote it. Blind refers to the scoring: the review was done without knowing which model produced which answer — the published result names them.

A frontier reference answered the same task (Claude Opus 5) — outside the field and deliberately unscored, further down the page.

Show task

Extrahiere alle Termine aus folgendem Text als JSON-Array von Objekten mit exakt den Feldern datum (YYYY-MM-DD), uhrzeit (HH:MM, 24h) und titel (String). Gib NUR das JSON aus, ohne Erklärung.

Text: "Am Dienstag, den 12. August 2026, treffen wir uns um halb drei nachmittags zur Budgetplanung. Der Zahnarzttermin ist am 03.09.2026 um 9 Uhr morgens. Nicht vergessen: Silvesterparty am 31.12. um 20:30! Das Sommerfest wurde übrigens abgesagt."

The answers

Local models (Brain cluster) — identical prompt, shuffled order (the sequence is not a ranking)

mistral-small:24b

```json
[
  {"datum": "2026-08-12", "uhrzeit": "15:30", "titel": "Budgetplanung"},
  {"datum": "2026-09-03", "uhrzeit": "09:00", "titel": "Zahnarzttermin"},
  {"datum": "2026-12-31", "uhrzeit": "20:30", "titel": "Silvesterparty"}
]
```

qwen3-coder:30b

```json
[
  {
    "datum": "2026-08-12",
    "uhrzeit": "14:30",
    "titel": "Budgetplanung"
  },
  {
    "datum": "2026-09-03",
    "uhrzeit": "09:00",
    "titel": "Zahnarzttermin"
  },
  {
    "datum": "2026-12-31",
    "uhrzeit": "20:30",
    "titel": "Silvesterparty"
  }
]
```

gemma4:26b

```json
[
  {
    "datum": "2026-08-12",
    "uhrzeit": "14:30",
    "titel": "Budgetplanung"
  },
  {
    "datum": "2026-09-03",
    "uhrzeit": "09:00",
    "titel": "Zahnarzttermin"
  },
  {
    "datum": "2026-12-31",
    "uhrzeit": "20:30",
    "titel": "Silvesterparty"
  }
]
```

qwen3:8b

[
  {
    "datum": "2026-08-12",
    "uhrzeit": "15:30",
    "titel": "Budgetplanung"
  },
  {
    "datum": "2026-09-03",
    "uhrzeit": "09:00",
    "titel": "Zahnarzttermin"
  },
  {
    "datum": "2026-12-31",
    "uhrzeit": "20:30",
    "titel": "Silvesterparty"
  }
]

deepseek-r1:14b

[
  {
    "datum": "2026-08-12",
    "uhrzeit": "15:30",
    "titel": "Budgetplanung"
  },
  {
    "datum": "2026-09-03",
    "uhrzeit": "09:00",
    "titel": "Zahnarzttermin"
  },
  {
    "datum": "2026-12-31",
    "uhrzeit": "20:30",
    "titel": "Silvesterparty"
  }
]

qwen3:14b

```json
[
  {
    "datum": "2026-08-12",
    "uhrzeit": "14:30",
    "titel": "Budgetplanung"
  },
  {
    "datum": "2026-09-03",
    "uhrzeit": "09:00",
    "titel": "Zahnarzttermin"
  },
  {
    "datum": "2026-12-31",
    "uhrzeit": "20:30",
    "titel": "Silvesterparty"
  }
]
```

llama3.1:8b

[
    {
        "datum": "2026-08-12",
        "uhrzeit": "15:30",
        "titel": "Budgetplanung"
    },
    {
        "datum": "2026-09-03",
        "uhrzeit": "09:00",
        "titel": "Zahnarzttermin"
    },
    {
        "datum": "2026-12-31",
        "uhrzeit": "20:30",
        "titel": "Silvesterparty"
    }
]

qwen3.5:9b

```json
[
  {
    "datum": "2026-08-12",
    "uhrzeit": "15:30",
    "titel": "Budgetplanung"
  },
  {
    "datum": "2026-09-03",
    "uhrzeit": "09:00",
    "titel": "Zahnarzttermin"
  },
  {
    "datum": "2026-12-31",
    "uhrzeit": "20:30",
    "titel": "Silvesterparty"
  }
]
```

deepseek-r1:32b

[
  {
    "datum": "2026-08-12",
    "uhrzeit": "15:30",
    "titel": "Budgetplanung"
  },
  {
    "datum": "2026-09-03",
    "uhrzeit": "09:00",
    "titel": "Zahnarzttermin"
  },
  {
    "datum": "2026-12-31",
    "uhrzeit": "20:30",
    "titel": "Silvesterparty"
  }
]

command-r:35b

[
  {
    "datum": "2026-08-12",
    "uhrzeit": "14:30",
    "titel": "Budgetplanung"
  },
  {
    "datum": "2026-09-03",
    "uhrzeit": "09:00",
    "titel": "Zahnarzttermin"
  },
  {
    "datum": "2026-12-31",
    "uhrzeit": "20:30",
    "titel": "Silvesterparty"
  }
]

qwq:32b

[
  {
    "datum": "2026-08-12",
    "uhrzeit": "14:30",
    "titel": "Budgetplanung"
  },
  {
    "datum": "2026-09-03",
    "uhrzeit": "09:00",
    "titel": "Zahnarzttermin"
  },
  {
    "datum": "2026-12-31",
    "uhrzeit": "20:30",
    "titel": "Silvesterparty"
  }
]

Outside the Brain cluster — same task, same character budget, collected through their own tooling (response times not comparable)

GPT-5.5 (Codex)frontier reference

[
  {
    "datum": "2026-08-12",
    "uhrzeit": "14:30",
    "titel": "Budgetplanung"
  },
  {
    "datum": "2026-09-03",
    "uhrzeit": "09:00",
    "titel": "Zahnarzttermin"
  },
  {
    "datum": "2026-12-31",
    "uhrzeit": "20:30",
    "titel": "Silvesterparty"
  }
]

Claude Sonnet 5frontier reference

[
  {"datum": "2026-08-12", "uhrzeit": "14:30", "titel": "Budgetplanung"},
  {"datum": "2026-09-03", "uhrzeit": "09:00", "titel": "Zahnarzttermin"},
  {"datum": "2026-12-31", "uhrzeit": "20:30", "titel": "Silvesterparty"}
]

Claude Fable 5frontier reference

[
  {"datum": "2026-08-12", "uhrzeit": "14:30", "titel": "Budgetplanung"},
  {"datum": "2026-09-03", "uhrzeit": "09:00", "titel": "Zahnarzttermin"},
  {"datum": "2026-12-31", "uhrzeit": "20:30", "titel": "Silvesterparty"}
]

Claude Haiku 4.5frontier reference

[
  {
    "datum": "2026-08-12",
    "uhrzeit": "14:30",
    "titel": "Budgetplanung"
  },
  {
    "datum": "2026-09-03",
    "uhrzeit": "09:00",
    "titel": "Zahnarzttermin"
  },
  {
    "datum": "2026-12-31",
    "uhrzeit": "20:30",
    "titel": "Silvesterparty"
  }
]

Reference

Gold — the correct answerreference

[{"datum":"2026-08-12","uhrzeit":"14:30","titel":"Budgetplanung"},{"datum":"2026-09-03","uhrzeit":"09:00","titel":"Zahnarzttermin"},{"datum":"2026-12-31","uhrzeit":"20:30","titel":"Silvesterparty"}] — abgesagtes Sommerfest darf nicht erscheinen; 'halb drei nachmittags' = 14:30; Jahr für Silvester aus Kontext 2026.

Results at a glance— every model in this round with score, response time and status.

Who wrote what

Model	Blind score	Latency	Status
mistral-small:24b	8.0	31.7 s	OK
qwen3-coder:30b	10.0	12.3 s	OK
gemma4:26b	10.0	129.9 s	OK
qwen3:8b	8.0	12.4 s	OK
deepseek-r1:14b	8.0	31.2 s	OK
qwen3:14b	10.0	14.4 s	OK
llama3.1:8b	8.0	128.5 s	OK
qwen3.5:9b	8.0	12.1 s	OK
deepseek-r1:32b	—	79.9 s	OK
command-r:35b	—	93.9 s	OK
qwq:32b	—	106.3 s	OK

Verdict

8 of 8 answering local models matched the gold answer (blind score ≥ 8): gemma4:26b, mistral-small:24b, llama3.1:8b, qwen3-coder:30b, qwen3:14b, qwen3.5:9b, qwen3:8b, deepseek-r1:14b.

Frontier reference — outside the blind field

These models answered the identical prompt outside the blind field. They are deliberately not scored and not part of the blind comparison — they are there so you can see where a frontier model lands on the same task.

Collected differently from the blind field: the local models run through a plain API harness, the frontier references through a context-free agent, one shot, no retries. Response times are therefore not comparable and are not shown. On two English tasks the reference answer came back in German — an artefact of that harness, flagged on the affected rounds.

Claude Opus 5frontier reference

[{"datum": "2026-08-12", "uhrzeit": "14:30", "titel": "Budgetplanung"}, {"datum": "2026-09-03", "uhrzeit": "09:00", "titel": "Zahnarzttermin"}, {"datum": "2026-12-31", "uhrzeit": "20:30", "titel": "Silvesterparty"}]