Article: LLM Reliability Layers Under Test: Accuracy, Yield

Architects Pushpdeep Mishra and Wejdan Bagais published an evaluation of language model reliability mechanisms under the online InfoQ Certified Architect Program, testing whether stacking verification layers actually repairs faulty model outputs. As detailed in InfoQ's report, the study benchmarked six verification strategies on medical adverse-event extraction and found that stacking checks does not incrementally improve accuracy. Instead of fixing mistakes, verification layers primarily filter answers by confidence, revealing that yield drops significantly when doubtful records are withheld.

The investigation evaluates open-weight language models parsing complex clinical records and explores why common mitigation patterns like voting, model judges, and self-correction fall short in production pipelines. It is aimed at software architects, data engineers, and machine learning practitioners who build structured extraction workflows where incorrect model assertions carry operational risk. By examining run determinism, vocabulary groundings, and extraction bounds, the research outlines where deterministic software succeeds and where model-based validation breaks down.

01Recognition menus over generative code recall

Direct generative extraction failed severely when models were tasked with recalling medical concept identifiers directly from memory. Asking the extraction model to produce standardized identifiers caused it to return null values on up to 38 percent of records. Furthermore, between 13 and 18 percent of the identifiers committed by the model did not exist in any SNOMED release.

To solve this, the pipeline was structured so that the model never directly invents an identifier. An extractor quotes the exact reaction span from prose, and an independent retriever searches 227,554 description rows from the SNOMED CT Australian edition of 31 July 2026. The search runs locally using granite-embedding:30m with 384 dimensions, presenting the model a menu of the 20 nearest concepts to choose from.

Testing this approach on 40 development documents showed that picking from a retrieved menu scored eight to twelve times better than recalling codes from memory. Grounding the process in an exact vocabulary database established the baseline pipeline on which all subsequent verification layers were tested.

02Noise floors across five model families

Identical runs of the extraction pipeline revealed meaningful differences in run-to-run consistency at temperature zero. Across three identical cold runs on 40 development documents, the primary extractor, gpt-oss:20b, exhibited a four-point variation in span-exact F1. Two runs were byte-identical, but the third diverged, finishing with 98 correct mentions compared to 87 in the matching pair.

Across all three executions, the system produced 233 distinct mentions. In 164 instances, representing 70 percent of mentions, all three runs agreed on both the span boundary and the assigned identifier. When all runs detected the same text span, they agreed on the identifier 84 percent of the time, demonstrating that instability stemmed primarily from text reading rather than code assignment.

This drift proved unique to the mixture-of-experts architecture of gpt-oss:20b. Four other model families tested under identical conditions—llama3.1:8b, mistral:7b-instruct, qwen3:8b, and granite4:micro-h—produced byte-identical outputs across every run. The engineering team retained gpt-oss:20b despite the variance because it delivered a 6.5-point F1 improvement over llama3.1:8b.

03Deterministic boundaries in the vocabulary check

The first free verification stage applied deterministic code rules to classify model outputs into three operational buckets without altering underlying text. The check marks an output as REJECT if the model selected a nonexistent concept identifier or quoted text absent from the input document. For instance, code 2714004 claimed for pale skin exists in no SNOMED release and gets dropped immediately.

Outputs receive an ACCEPT verdict when the extracted phrase matches the vocabulary's formal names verbatim, such as mapping chronic pain directly to the corresponding concept. The remaining entries fall into an intermediate BAND verdict when the extracted words do not match canonical vocabulary phrasing. A phrase like bit drowsy mapping to drowsy is clinically valid, while truck hit mapping to motor vehicle injury represents a fabrication, yet deterministic rules cannot distinguish between the two.

Deterministic code successfully catches provably impossible answers at zero token cost. However, because it cannot validate semantic plausibility without additional models or human reviewers, its utility is confined to filtering invalid syntax and corrupted spans.

04Paid verification layers and self correction limits

The benchmark evaluated three paid verification techniques: self-correction loops costing roughly 1,500 tokens, majority voting, and an independent evaluator model. Self-correction rarely triggered because it only engaged when an output received a REJECT verdict from deterministic checks. When it did run, asking the model to correct provable failures rarely yielded a valid downstream extraction.

Majority voting across repeated model samplings failed to recover missing accuracy, functioning instead as a filter that suppressed marginal records. While a second model acting as an evaluator generated validity verdicts, the primary system had no mechanism to translate those judgments into corrected extractions.

The findings demonstrated that verification layers sort outputs by trustworthiness rather than repairing underlying errors. Each successive layer decreased yield—the ratio of verified correct answers over total records—without lifting raw baseline accuracy.

05Extraction ceilings and annotated gold standards

The entire benchmarking harness evaluated performance against the CSIRO Adverse Drug Event Corpus, known as CADEC, comprising 1,250 forum posts covering adverse reactions across two medications. Against 875 fine-tuned training files, the supervised benchmark CONORM achieved an end-to-end lenient F1 score of 0.72. The zero-shot extraction setup achieved an F1 of 0.47 to 0.50 under lenient matching, and 0.39 to 0.43 under span-exact scoring.

Initial extraction establishes an unyielding performance ceiling across the entire architecture. If the initial model fails to locate an adverse reaction span in the text, no subsequent verification layer, voting scheme, or evaluator model can recover it.

Evaluating the held-out test split of 60 documents confirmed that gains came from aligning with corpus annotation rules rather than adding verification steps. Where gold annotations labeled extremely sick as generally unwell and the model selected illness, both interpretations were clinically reasonable, yet strict benchmark scoring counted the model's answer as incorrect.