1. Maria -- finalize new homepage copy -- by this Thursday 2. Maria -- send finalized homepage copy to Ben for review -- by this Thursday 3. Ben -- review and provide feedback on new homepage copy -- by Friday 4. Chen -- push contact form fix -- by Monday 5. Maria -- chase client about new logo colors -- by today (July 29) 6. Ben -- send updated privacy policy text to legal -- by next Wednesday
Overview › summarization › R168
summarization · blind round
R168 · Extract action items as a checklist
Every answer below carries the name of the model that wrote it. Blind refers to the scoring: the review was done without knowing which model produced which answer — the published result names them.
Show task
Below is a summary of a scattered email thread in which several people mention things they will do, in passing, mixed in with other remarks. Extract ALL action items as a numbered checklist, one line per item, in the exact format "Who -- what -- by when". Do not invent any action item that is not in the text, and do not include anything that has already been completed (it is not an action item anymore). Output ONLY the numbered checklist -- no explanation. Email thread summary: Email thread summary -- Website Relaunch Project From Maria (Jul 28): Hi team, following up from yesterday's call. I'll finalize the new homepage copy and send it over to Ben for review by this Thursday. Also, quick reminder that the staging server credentials were already rotated last week, so no action needed there anymore. From Ben (Jul 28): Thanks Maria. Once I get the copy I'll review it within a day or two -- realistically I can have feedback back to you by Friday. Separately, has anyone looked at the broken contact form? It's been throwing errors since the last deploy. From Chen (Jul 29): I can take the contact form bug -- I'll have a fix pushed by Monday. On an unrelated note, does anyone know if the client ever confirmed the new logo colors? I still haven't heard back from them. From Maria (Jul 29): No word from the client yet on the logo colors, I'll chase them again today. Also, Ben, don't forget we still need the updated privacy policy text -- can you get that to legal by next Wednesday so they have time to review before launch? From Ben (Jul 29): Will do, I'll send the privacy policy draft to legal by next Wednesday.
The answers
Local models (Brain cluster) — identical prompt, shuffled order (the sequence is not a ranking)
1. Maria -- finalize and send new homepage copy -- by this Thursday 2. Ben -- review homepage copy and give feedback -- by Friday 3. Chen -- push fix for broken contact form -- by Monday 4. Maria -- chase client for logo color confirmation -- today 5. Ben -- send updated privacy policy draft to legal -- by next Wednesday
1. Maria -- finalize and send new homepage copy to Ben -- by this Thursday 2. Ben -- provide feedback on homepage copy -- by Friday 3. Chen -- push contact form fix -- by Monday 4. Maria -- chase the client for new logo colors confirmation -- today 5. Ben -- send updated privacy policy draft to legal -- by next Wednesday
1. Maria -- chase client for logo colors confirmation -- today 2. Chen -- fix contact form bug -- Monday 3. Ben -- get updated privacy policy text to legal -- next Wednesday
1. Maria -- finalize new homepage copy -- by this Thursday 2. Ben -- review homepage copy and provide feedback -- by Friday 3. Chen -- fix contact form bug -- by Monday 4. Maria -- chase client for logo color confirmation -- by today 5. Ben -- send updated privacy policy to legal -- by next Wednesday
1. Maria -- finalize and send new homepage copy for review -- by this Thursday 2. Ben -- review homepage copy and provide feedback -- by Friday 3. Chen -- push a fix for the contact form bug -- by Monday 4. Maria -- chase the client about new logo colors -- today 5. Ben -- send the updated privacy policy draft to legal -- by next Wednesday
1. Maria -- finalize new homepage copy and send to Ben for review -- by this Thursday 2. Ben -- review new homepage copy and provide feedback -- by Friday 3. Chen -- fix broken contact form bug -- by Monday 4. Maria -- chase client about confirming new logo colors -- today (no due date specified) 5. Ben -- send updated privacy policy text to legal for review -- by next Wednesday
1. Maria -- finalize the new homepage copy and send it to Ben -- by this Thursday 2. Ben -- review the homepage copy and provide feedback -- by Friday 3. Chen -- push a fix for the contact form bug -- by Monday 4. Maria -- chase the client regarding the new logo colors -- today 5. Ben -- send the privacy policy draft to legal -- by next Wednesday
1. Chen -- fix the broken contact form -- by Monday 2. Maria -- chase the client for confirmation on new logo colors -- by today 3. Ben -- get updated privacy policy text to legal -- by next Wednesday
1. Maria -- finalize the new homepage copy and send it over to Ben for review -- by this Thursday 2. Ben -- review the copy and provide feedback -- by Friday 3. Chen -- fix the broken contact form and push the fix -- by Monday 4. Maria -- chase the client for confirmation on the new logo colors -- by today 5. Ben -- send the updated privacy policy draft to legal -- by next Wednesday
1. Maria -- finalize new homepage copy and send to Ben for review -- by this Thursday 2. Ben -- review homepage copy and provide feedback -- by Friday 3. Chen -- fix broken contact form bug -- by Monday 4. Maria -- chase client for confirmation on new logo colors -- today 5. Ben -- send updated privacy policy text to legal -- by next Wednesday
Outside the Brain cluster — same task, same character budget, collected through their own tooling (response times not comparable)
GPT-5.5 (Codex)frontier reference
1. Maria -- finalize the new homepage copy and send it to Ben for review -- by this Thursday 2. Ben -- review the new homepage copy and send feedback to Maria -- by Friday 3. Chen -- push a fix for the contact form bug -- by Monday 4. Maria -- chase the client about the new logo colors -- today 5. Ben -- send the privacy policy draft to legal -- by next Wednesday
Reference
Gold — the correct answerreference
Sinngemaess muss enthalten/erfuellen: eine nummerierte Checkliste mit GENAU den 5 tatsaechlichen Action Items im Format 'Wer -- was -- bis wann': (1) Maria -- Homepage-Text fertigstellen und an Ben schicken -- bis Donnerstag; (2) Ben -- Text reviewen und Feedback an Maria schicken -- bis Freitag; (3) Chen -- Kontaktformular-Bug fixen -- bis Montag; (4) Maria -- beim Kunden wegen Logo-Farben nachhaken -- heute; (5) Ben -- Datenschutz-Entwurf an Legal schicken -- bis naechsten Mittwoch. VERBOTEN: den Distraktor (die bereits letzte Woche rotierten Staging-Zugangsdaten) als Action Item aufzunehmen, da diese Aufgabe schon erledigt ist; keine neuen, im Thread nicht vorkommenden Aufgaben erfinden. Musterbeispiel (maschinell verifiziert: genau 5 nummerierte Zeilen, jede Zeile enthaelt das 'Wer -- was -- bis wann'-Muster mit 2 Gedankenstrichen, alle 5 Personen/Termine als Substring vorhanden, 'credentials' NICHT vorhanden): 1. Maria -- finalize the homepage copy and send it to Ben -- by Thursday 2. Ben -- review the copy and send feedback to Maria -- by Friday 3. Chen -- fix the contact form bug -- by Monday 4. Maria -- follow up with the client about the logo colors -- today 5. Ben -- send the privacy policy draft to legal -- by next Wednesday
Results at a glance— every model in this round with score, response time and status.
Who wrote what
| Model | Blind score | Latency | Status |
|---|---|---|---|
| llama3.1:8b | 7.0 | 59.1 s | OK |
| qwen3:14b | 10.0 | 15.2 s | OK |
| qwen3.5:9b | 10.0 | 20.9 s | OK |
| mistral-small:24b | 1.0 | 67.7 s | OK |
| qwen3:8b | 7.0 | 16.2 s | OK |
| qwen3-coder:30b | 10.0 | 10.9 s | OK |
| deepseek-r1:14b | 7.0 | 20.9 s | OK |
| gemma4:26b | 10.0 | 92.3 s | OK |
| command-r:35b | 1.0 | 105.3 s | OK |
| qwq:32b | 10.0 | 104.7 s | OK |
| deepseek-r1:32b | 10.0 | 36.6 s | OK |
Verdict
6 of 11 answering local models matched the gold answer (blind score ≥ 8): gemma4:26b, qwen3-coder:30b, qwen3:14b, qwen3.5:9b, deepseek-r1:32b, qwq:32b.