Daily LLM content jobs fail silently without evals; build a pytest harness for facts, style, and regression gates before publishing. Full breakdown.
Why daily LLM content jobs need a hard gate
Scheduled generators that draft posts, pulses, or summaries can look fine in a log line while shipping wrong facts, thin structure, or off-brand tone. Without an automated check between generation and publish, failures show up as reader trust loss, broken SEO, or silent drift across weeks of runs. An eval harness treats each draft as a test subject: if it fails facts, style, or regression gates, the pipeline stops and stages nothing for human review or deploy.
The goal is not to prove the model is “smart.” It is to make quality failures loud, repeatable, and cheap to debug before anything reaches a public URL.
Shape the harness as ordinary pytest
Keep the harness in the same toolchain you already use for application tests. Each content artifact becomes a fixture or temporary file; each check is a plain test function that asserts a property of that artifact. Run the suite after generation and before any move into the publish tree. Exit non-zero on failure so cron, CI, or a writer script can refuse to continue.
Separate generation from evaluation. The writer produces a draft under a staging path; pytest loads that draft and runs independent suites. That split lets you re-run evals after a manual edit without regenerating, and it keeps model calls out of the assertion layer so flaky network behavior does not masquerade as a content bug.
Three gate families: facts, style, regression
Facts gates catch inventable or unverifiable claims. Assert that every external link resolves or is allowlisted, that version-like strings appear only when the source payload supplied them, and that required entities from the job brief (title topics, product names, dates already present in inputs) still appear in the body. Prefer structured checks over free-form “is this true” prompts when the input ledger or source JSON already encodes the truth you care about.
Style gates enforce the shape your site actually needs: minimum body length, required heading count, presence of a deck or byline pattern, forbidden filler phrases, and section order. Regression gates freeze a small gold set of past good outputs (or their fingerprints) and fail when a new draft drops below a similarity or checklist baseline you define—for example, missing a previously required section type or collapsing from multi-paragraph depth into a stub.
- Facts: source-backed claims, link integrity, no orphan numbers or versions.
- Style: structure, length, tone constraints, banned stock phrases.
- Regression: gold fixtures and checklist baselines that block quiet quality decay.
Wire it into the daily path and keep it maintainable
Hook the harness as a single command after staging: generate, eval, then publish only on green. Failures should write a short report next to the draft—which test failed and which snippet tripped it—so a human can fix or re-run without replaying the whole scrape. Parametrize tests by content type (standalone post vs daily pulse) so one suite does not force the same rules on every format.
Start with deterministic checks you can trust without another model call. Add LLM-as-judge only for judgments that are hard to encode as rules, and pin those prompts and scoring rubrics so the judge does not drift. Treat the harness as product code: review new assertions with the same care as app tests, and never let a flaky gate become a reason to skip the suite. Silent daily jobs stay silent until something fails hard; a pytest gate is how you make that failure happen before readers do.