Claude was faster and cheaper. We took the scenic route anyway

Verify, Then Vote checks the sources small models cite and then applies a fixed rule, as described in blog.grainulation.org's report. Five models ran through Ollama on an Apple M3 Pro with 36 GB of RAM and, left overnight, finished in about eight hours. Claude Sonnet 5.5 checked those citations as a separate hosted model. On 1,000 main questions, accuracy moved from 77.1% with a simple majority vote to 85.0%.

This piece is for people running a few small models on one machine who want a saved reason for every answer switch. It covers the checker, the Lean 4 guarantee, a missed 5% limit, a revision that passed on five different models, and a replay with no new model calls.

Claude was faster and cheaper. We took: what actually changed

Five models got the same question, looked things up, and often disagreed. Models can share a mistake, and a vote cannot catch that. The authors checked each cited source, then used a fixed rule to pick the answer, and called the paper Verify, Then Vote. Answering, checking, and choosing stayed separate, and each result was saved for reuse. Settings were tuned on a practice set. On 1,000 main questions the rule moved accuracy from 77.1% with a simple majority vote to 85.0%.

On 526 mixed questions the rule fixed 91 answers the vote got wrong and broke 12 it got right, from 71.5% to 86.5%. The vote broke ties at random, and the rule always broke them the same way, so part of the gain came from ties. Of 376 answers the vote got right, 12 became wrong (3.2%). Allowing for chance, the true rate could reach 5.5%, above the 5% limit set beforehand, so the original rule failed. The plan and analysis were written before collection and were not filed in a public registry.

Claude was faster and cheaper. We took: how it works

Each model wrote a search query and got five passages: Wikipedia for general questions, research abstracts for science claims. It answered, stated its confidence, and cited up to three passages. All of that was saved before any answer was picked. Claude Sonnet 5.5, a separate hosted model, saw only one sentence stating the answer and one cited passage, and it never saw the correct answer. It said whether the passage supported the answer, contradicted it, or did not say enough.

If a passage says Sydney is the capital of Australia, it supports "Sydney," even though the capital is Canberra. The check asks whether the passage backs the answer, and it does not ask whether the passage is true. The rule starts on the most-voted answer, breaks every tie the same way, and switches only when a cited passage supports another answer, keeping that passage and check. Lean 4, software that checks mathematical proofs, shows a correct-to-wrong swap needs a source check behind the wrong answer. It does not show the source was true, or that the checker read it correctly.

Claude was faster and cheaper. We took: why it matters

Cited text was removed, or swapped for unrelated text, and 76 of the 91 fixes disappeared either way. The passages mattered, though Sonnet may still lean on what it already knows when the right text is present. The authors then kept a starting answer that already had a supporting source. They wrote the change down before taking answers from five different models on the same questions, and they reused a check whenever the answer and the passage were the same.

On 547 mixed questions the revised rule fixed 73 and broke three, from 73.1% to 85.9%. Of 400 answers the vote got right, three became wrong (0.75%). The worst case was 2.2%, under the 5% limit, so the rule passed. The questions matched the first run, and the change was designed after those results, so the pass covers other models rather than new questions. Sonnet was right 93.9% of the time on the first 526 mixed questions, versus 86.5% for the rule, at about $0.014 per question versus $0.021. Neither number includes the models' writing cost.

Claude was faster and cheaper. We took: who is affected

Llama 3.2 3B, Mistral 7B, Gemma 2 2B, Phi-3 mini, and Qwen 2.5 3B ran through Ollama on one Apple M3 Pro with 36 GB of RAM. The machine ran overnight and finished in about eight hours. Each model wrote a query, read five passages, stated a confidence, and cited up to three. Five different models later answered the same questions, and a check was reused only when the answer and the passage matched. Claude Sonnet 5.5 scored one sentence against one passage and never saw the correct answer.

The pool is 1,100 questions, including a separate set of 100 from TruthfulQA, and the models disagreed on 702. A Python script dropped one model at a time and sent the remaining saved answers to a Rust program to choose again. The chosen answer changed 9.4% of the time. A judge's choice can be logged, but choosing again means another call. The paper, code, proofs, and saved runs are on GitHub, with scripts to reproduce the results and try other rules on the same data.

Claude was faster and cheaper. We took: what to watch

The next step the authors name is new questions. Saved runs let them open a decision, change the rule, and measure the effect before paying to collect more answers. Answering, checking, and choosing stay separate, so saved answers and checks remain while the chooser changes. New answers or new sources would need new checks. Five reruns on each of the 702 disagreements came to 3,510 decisions, took 23 seconds, and made no new model calls. Rerunning all 1,100 original decisions gave exactly the same output.

GitHub holds the paper and code, the results with the harm test and replay, the models and evaluation methods, a reproducible artifact, the local run design and hardware, a proof catalog and the Lean 4 implementation, the decision rule and its formal guarantee, and the recorded local collection session. A correct-to-wrong swap requires a source check that backed the wrong answer, which is what the Lean 4 proof shows. The proof does not show the passage was true. The under-5% pass used questions already seen, so it describes other models.