Benchmarking Fable, Sol, and Kimi K3 on SlopCodeBench

I'll pull the source article so the paragraphs stick to its actual facts—names, mechanics, and any numbers it states—without inventing extras.A HumanLayer write-up from dexhorthy put Fable, Sol, and Kimi K3 through SlopCodeBench on a six-challenge subset totaling 30 checkpoints per model. Fable and Sol tied at 33.3 percent strict checkpoint passes (10 of 30 each). Kimi K3 on Modal scored 26.7 percent and Kimi K3 on Baseten scored 23.3 percent. The run was posted on Hacker News with 7 points and 1 comment. Modal and Baseten sponsored the Kimi inference; the author stresses the two Kimi runs were single-shot per provider and should not be treated as statistically significant.

SlopCodeBench, from Gabe Orlanski’s lab at UW Madison, is a long-horizon coding eval where each challenge is a sequence of checkpoints. The full problem is not shown up front; requirements land incrementally and the agent must evolve the same codebase. Defects are scored with held-out black-box tests against the produced entrypoint, and a strict pass requires the new work plus every inherited regression suite to stay green. This run used xjq, file_backup, dag_execution, circuit_eval, code_search, and etl_pipeline. Harnesses were Claude Code 2.1.219 for Fable, Codex CLI 0.145.0 for Sol, and OpenCode 1.18.0 for both Kimi K3 providers, with a fresh context window per checkpoint and identical prompts across models.

For builders, the useful signal is not a leaderboard gap of a few points but how agents fail over time. Every model collected some strict passes and still stacked defects as checkpoints advanced, so a broken earlier stage blocks later ones under the strict metric. Isolated passes split the Fable/Sol tie (Fable 16, Sol 14), and quality side metrics diverged: Fable drove the largest clone-share growth on circuit_eval (about 9x from checkpoint 1 to 8), Kimi posted the highest single-function complexity, and Sol left 1,318 SLOC of persistent Python tests while others leaned on shell scripts, fixtures, or temporary files. Slop-density averages were high across the board: Fable 86 percent, Sol 95 percent, Kimi K3 Baseten 82 percent, Kimi K3 Modal 79 percent.

On the competitive side, none of these frontier agents cleared the ceiling seen in the prior Opus 5 work: brand-new models still sat under 35 percent pass rate on this style of eval, roughly 2024 SWE-Bench territory from the Sonnet 3.5 era. The shared circuit_eval challenge showed more strict passes for the newer models than the earlier Opus 5 subset, but the previous Opus 5 run’s lower final-defect count is hard to compare cleanly because that suite had 557 tests versus 566 in the suite used for Fable, Sol, and Kimi. Cost versus final defect rate was only a vague inverse; Modal cost was estimated at $26.93 from token counts using Baseten-style rates of $3 per million input, $0.30 per million cache read, and $15 per million output. The single HN comment pushed on harness fairness for Kimi (preferring native Kimi code over OpenCode) and on n=1 provider variance.

Practical takeaway: treat these numbers as directional, not as a production agent ranking. Watch for the author’s next harness experiment—deterministic linters and LLM adversarial review after each checkpoint—to see whether intermediate review raises strict pass rates or just burns more tokens. Until then, long-horizon agent work still needs human gates on regression carry-forward, not lights-out autonomy on multi-stage product code.