Handwritten-edit benchmark: Fable 5 is #1, Opus 4.8 regresses 55% on miscounting

I'll pull the source article and HN thread so the paragraphs stick to verified details only.A handwritten-edit benchmark at dorrit.pairsys.ai ranks multimodal models on recovering editorial marks from scanned Little Dorrit pages. On that leaderboard, Fable 5 sits at number one. Opus 4.8 shows a 55 percent regression on miscounting. The run was flagged on Hacker News under Claude, Codex, and Fable discussion, with four points and one comment at the time of the summary.

The task is not plain OCR. Models get page images of printed text with handwritten marks and must emit structured JSON for every correction: edit type (insertion, deletion, replacement, punctuation, capitalization, or italicize), original text, corrected text, line number, and page id. Line 0 is reserved for titles or headings; body lines start at 1. Marks include carets for inserts, strikethroughs for deletes, circled or bracketed swaps, and underlines for italics. Claude-family runs use the same prompt but have images resized and recompressed to stay under Anthropic’s 5 MiB per-image limit. Most models were hit via OpenRouter; some GPT entries also ran on api.openai.com.

For engineers building document and vision pipelines, the hard part is joint visual and editorial reasoning under a strict schema. Miscounting—wrong line numbers or missing edits—breaks any pipeline that applies those JSON records as patches. A large regression on that axis means the model can still look fluent while producing unusable diffs. The benchmark’s failure notes also show models that return non-JSON or unparseable payloads, which is the same class of production break as a wrong edit list.

In competitive terms, a smaller or less public name topping Claude- and Codex-adjacent models on this slice matters because the task sits at the intersection of vision, layout, and structured output. Leaderboard position on Little Dorrit does not transfer automatically to general chat or code, but it does rank systems on a workflow that looks like real editorial and publishing tooling. The Claude image-size caveat also means cross-vendor comparisons are not fully input-identical, so ranking gaps need to be read with that constraint in mind.

Practical takeaway: if you score multimodal editors, measure edit-level F1 and line-number accuracy separately, and treat schema-valid JSON as a gate, not a nice-to-have. Watch whether Fable 5 holds the top slot as more vendors re-run the suite, whether Opus 4.8’s miscounting drop shows up on other annotation sets, and whether Claude results change when image budgets no longer force resize and recompress. Source: https://dorrit.pairsys.ai/ ; HN: https://news.ycombinator.com/item?id=49028771