Structure & Data · blind round

R149 · Extract meeting invite into a fixed JSON schema

blind round 11/11 local models answered

Every answer below carries the name of the model that wrote it. Blind refers to the scoring: the review was done without knowing which model produced which answer — the published result names them.

Show task

Extract the event information from the text below into JSON with EXACTLY these fields: title (string), date (YYYY-MM-DD), time (24h HH:MM), location (string), attendees (array of strings). Output ONLY the JSON object — no explanation.

Hey all - quick heads up about Thursday's meeting.
It's the Q3 Roadmap Kickoff, happening on August 18, 2026 at 10:30 AM.
We'll be in Conference Room B, at the Berlin office (3rd floor).
Sarah, Tom, and Priya really need to be there since we're deciding scope.
Let me know if anything conflicts - thanks!

The answers

Local models (Brain cluster) — identical prompt, shuffled order (the sequence is not a ranking)

qwen3:14b

```json
{
  "title": "Q3 Roadmap Kickoff",
  "date": "2026-08-18",
  "time": "10:30",
  "location": "Conference Room B, Berlin office (3rd floor)",
  "attendees": [
    "Sarah",
    "Tom",
    "Priya"
  ]
}
```

deepseek-r1:14b

{
  "title": "Q3 Roadmap Kickoff",
  "date": "2026-08-18",
  "time": "10:30",
  "location": "Conference Room B, Berlin office (3rd floor)",
  "attendees": ["Sarah", "Tom", "Priya"]
}

qwen3-coder:30b

```json
{
  "title": "Q3 Roadmap Kickoff",
  "date": "2026-08-18",
  "time": "10:30",
  "location": "Conference Room B, Berlin office (3rd floor)",
  "attendees": [
    "Sarah",
    "Tom",
    "Priya"
  ]
}
```

qwen3.5:9b

```json
{
  "title": "Q3 Roadmap Kickoff",
  "date": "2026-08-18",
  "time": "10:30",
  "location": "Conference Room B, Berlin office (3rd floor)",
  "attendees": [
    "Sarah",
    "Tom",
    "Priya"
  ]
}
```

qwen3:8b

{
  "title": "Q3 Roadmap Kickoff",
  "date": "2026-08-18",
  "time": "10:30",
  "location": "Conference Room B, at the Berlin office (3rd floor)",
  "attendees": ["Sarah", "Tom", "Priya"]
}

gemma4:26b

```json
{
  "title": "Q3 Roadmap Kickoff",
  "date": "2026-08-18",
  "time": "10:30",
  "location": "Conference Room B, Berlin office (3rd floor)",
  "attendees": [
    "Sarah",
    "Tom",
    "Priya"
  ]
}
```

mistral-small:24b

```json
{
    "title": "Q3 Roadmap Kickoff",
    "date": "2026-08-18",
    "time": "10:30",
    "location": "Conference Room B, at the Berlin office (3rd floor)",
    "attendees": ["Sarah", "Tom", "Priya"]
}
```

llama3.1:8b

```
{
    "title": "Q3 Roadmap Kickoff",
    "date": "2026-08-18",
    "time": "10:30",
    "location": "Conference Room B, 3rd floor at the Berlin office",
    "attendees": ["Sarah", "Tom", "Priya"]
}
```

command-r:35b

{
  "title": "Q3 Roadmap Kickoff",
  "date": "2026-08-18",
  "time": "10:30",
  "location": "Conference Room B, Berlin office (3rd floor)",
  "attendees": ["Sarah", "Tom", "Priya"]
}

qwq:32b

{
  "title": "Q3 Roadmap Kickoff",
  "date": "2026-08-18",
  "time": "10:30",
  "location": "Conference Room B, at the Berlin office (3rd floor)",
  "attendees": ["Sarah", "Tom", "Priya"]
}

deepseek-r1:32b

{
  "title": "Q3 Roadmap Kickoff",
  "date": "2026-08-18",
  "time": "10:30",
  "location": "Conference Room B, Berlin office (3rd floor)",
  "attendees": ["Sarah", "Tom", "Priya"]
}

Outside the Brain cluster — same task, same character budget, collected through their own tooling (response times not comparable)

Claude Opus 5frontier reference

{
  "title": "Q3 Roadmap Kickoff",
  "date": "2026-08-18",
  "time": "10:30",
  "location": "Conference Room B, Berlin office (3rd floor)",
  "attendees": ["Sarah", "Tom", "Priya"]
}

Claude Sonnet 5frontier reference

{"title": "Q3 Roadmap Kickoff", "date": "2026-08-18", "time": "10:30", "location": "Conference Room B, Berlin office (3rd floor)", "attendees": ["Sarah", "Tom", "Priya"]}

Claude Fable 5frontier reference

{
  "title": "Q3 Roadmap Kickoff",
  "date": "2026-08-18",
  "time": "10:30",
  "location": "Conference Room B, Berlin office (3rd floor)",
  "attendees": ["Sarah", "Tom", "Priya"]
}

Claude Haiku 4.5frontier reference

{
  "title": "Q3 Roadmap Kickoff",
  "date": "2026-08-18",
  "time": "10:30",
  "location": "Conference Room B, Berlin office (3rd floor)",
  "attendees": ["Sarah", "Tom", "Priya"]
}

Reference

Gold — the correct answerreference

{"title":"Q3 Roadmap Kickoff","date":"2026-08-18","time":"10:30","location":"Conference Room B, Berlin office (3rd floor)","attendees":["Sarah","Tom","Priya"]} — maschinell validiert: json.loads erfolgreich, Keys exakt {title,date,time,location,attendees}, attendees ist Array mit 3 Strings, Datum korrekt normalisiert (August 18, 2026 -> 2026-08-18), Zeit korrekt (10:30 AM -> 10:30).

Results at a glance— every model in this round with score, response time and status.

Who wrote what

Model	Blind score	Latency	Status
qwen3:14b	7.0	6.8 s	OK
deepseek-r1:14b	10.0	21.3 s	OK
qwen3-coder:30b	7.0	2.7 s	OK
qwen3.5:9b	7.0	10.8 s	OK
qwen3:8b	10.0	9.8 s	OK
gemma4:26b	7.0	85.8 s	OK
mistral-small:24b	7.0	53.1 s	OK
llama3.1:8b	7.0	43.1 s	OK
command-r:35b	10.0	76.9 s	OK
qwq:32b	10.0	76.3 s	OK
deepseek-r1:32b	10.0	41.1 s	OK

Verdict

5 of 11 answering local models matched the gold answer (blind score ≥ 8): qwen3:8b, deepseek-r1:14b, deepseek-r1:32b, qwq:32b, command-r:35b.