Write the eval before the fix: building a WhatsApp transcriber with LLM-TDD

Independent developer Waldo built WhatYouSay, a private voice-message transcription server linked to a WhatsApp account that transcribes incoming audio and automatically posts the text directly into the chat thread. Built as a privacy-focused alternative to the hosted service WhatScribe, the server keeps audio in volatile process memory and purges recordings once transcripts are delivered, as detailed in waldo.vanderlore.de's report.

This technical analysis examines how test-driven development was adapted for non-deterministic language models to solve Whisper transcription dropouts, translation edge cases, and conversational cleanup errors across English, German, and Dutch. The implementation provides practical architectural patterns for software engineers and systems builders designing multilingual speech pipelines under strict data-retention boundaries.

01Evaluation checks and benchmark reference loops

Development centered on LLM-TDD, an engineering methodology where human developers write evaluation checks before prompting an artificial intelligence model to implement code. Because outputs from large language models are non-deterministic, the evaluation pipeline checks draft transcripts against reference benchmarks rather than expecting exact string equality. Real voice notes recorded by active chat participants serve as test data, and comparison mismatches trigger revisions to prompts or application logic.

The pipeline operates as a multi-stage speech processing workflow where Whisper generates initial transcripts and a secondary model cleans up grammar, translates text, and condenses long audio notes. Validation relies on twenty real conversational voice messages recorded in English, German, and Dutch, scored against independent benchmark transcripts produced by ChatGPT. A successful comparison leads to production deployment, while unexpected failures detected in live operations are converted into permanent evaluation test cases.

02Twenty voice notes and cleanup prompt rules

Initial pipeline evaluations across the twenty benchmark voice notes achieved an accuracy score of only 69.9 percent against the reference transcripts. Most of the divergence originated in the secondary cleanup model, which systematically converted natural conversational speech into formal German prose. The model replaced regional words like gucken with the formal schauen, substituted the colloquial loanword matcht with entspricht, and stripped out natural German particles including ja, also, mal, and halt.

The cleanup model also translated English technical vocabulary that bilingual German speakers had intentionally spoken, treating natural code-switching as erroneous phrasing. To fix this over-correction, developers revised the prompt to list protected speech categories that the model must never modify, including colloquial phrasing, regional vocabulary, discourse particles, and embedded English words. When evaluated against the same twenty benchmark voice notes, the updated prompt lifted word-match accuracy from 69.9 percent to 86.4 percent.

03Language gates and misheard consonant repairs

Following prompt refinements, developers instituted specific word-drift gates for each supported language. German transcript drafts are permitted to alter at most 8 percent of words, while English drafts allow a maximum variation of 5 percent. When a draft exceeds those thresholds, the pipeline distrusts the text revision, retains Whisper's original words, adopts only the formatting structure, and preserves spoken English discourse markers such as like, you know, and I mean.

Dutch transcripts bypass percentage gates because spoken numbers are spelled out as words, which inflates word counts. The pipeline applies targeted phonetic repairs instead, permitting single-consonant adjustments when sentence context is clear, allowing grapjes to replace misheard prapjes over the unfitting praatjes. A dedicated rule protects Berlin's Spree river from becoming starlings (spreeuwen), while unresolved errors like gelaschten turning into gesendeten instead of gelöschten remain as red tests.

04Sender context and Whisper early truncation

Testing exposed a severe failure mode where Whisper abruptly stops transcribing midway through an audio file. In one benchmark case, a 32-second English recording yielded only 27 words because the server passed the sender's display name into Whisper as context. Whisper interprets prompt strings as audio it has already processed, matching the phonetic sound of the name within the message and terminating the transcription early.

Removing the sender name from context restored the full 86-word transcript, while lowercase names caused Whisper to drop 21 of 226 words. Sender names are now restricted to pass two, which is rejected unless it retains 80 percent of pass one words. If processing falls below 1.2 words per second across both passes, a fallback model takes over, dropping names to preserve transcript completeness.

05Multi-pass transcription and memory processing pipeline

Between 11 September and publication, WhatYouSay processed 230 voice notes totaling three hours of conversational audio, with the longest message running five minutes. Every audio file is downloaded directly into volatile process memory and processed across four modular pipeline stages before being purged. Audio files under 15 seconds, which accounted for 61 of the 230 processed messages, complete their processing after a single promptless transcription pass using Groq Whisper large-v3.

Longer recordings undergo forced-language secondary passes and Groq-hosted LLM cleanup, which skips notes under 30 words. If cleaned drafts cut over half the words from 50-word notes or return dense text walls for 80-word files, the engine retries once before reverting to raw Whisper output. Notes over 120 words receive summaries capped at 60 words, condensing messages like a 49-second note on oat milk into three sentences.