Detecting Claude and ChatGPT using letter counting

What happened

Now I have all the source facts I need. Let me write the article.

---

Detecting Claude and ChatGPT using letter counting

Chris Greening, writing on his atomic14 blog on August 19, 2026, has extended an AI-text detector he built the day before to recognize a second model. Where the original tool could only flag text written by Claude Opus 5, the updated version now covers ChatGPT as well, and a third combined detector can answer the broader question of whether any robot wrote something. All three detectors are live at meatbag.atomic14.com.

How it works

This article explains how each of the three detectors was trained, what the accuracy numbers mean, where the method breaks down, and what builders who want to verify or replicate the approach should check before putting any confidence score to practical use. It is written for developers and researchers who are already comfortable with machine learning basics and want to understand the mechanism rather than just the headline results.

What happened

On August 19, 2026, Chris Greening published a follow-up to his August 18 post in which he had trained a linear classifier to detect Claude Opus 5 by counting four-letter runs of characters, called n-grams. That original detector reached around 0.91 ROC AUC on text it had never seen. The new post adds a parallel ChatGPT detector trained the same way, reports that ChatGPT is consistently easier to detect than Claude, and introduces a third model trained on data from both systems to answer whether any AI wrote a passage.

The ChatGPT data was collected by having the model rewrite 519 human passages three ways, producing 1,557 rewrites, plus 519 passages generated from scratch that were held out of training and used only as a test set. That comes to 2,076 total ChatGPT generations, mirroring the approach used for Claude. Greening notes the ChatGPT data was collected using the command-line Codex tool on an existing subscription rather than Anthropic's batch API, which cost about $30 for the Claude run.

Why it matters

How it works

The core technique is the same across all three detectors. Rather than looking at words or grammar, the system counts every four-character sequence, called a four-letter n-gram, in a passage. Those counts are fed into a linear support vector machine, which learns which n-gram patterns appear more often in AI-written text than in human-written text. The model never sees which words are present, only the letter-level frequencies. Because it is linear and operates on counts, it can run inside a browser without a server.

Each detector is a separate binary classifier trained from scratch on its own data. The Claude detector learned what Claude Opus 5 output looks like against human writing. The ChatGPT detector learned what ChatGPT output looks like against the same human passages. The third, general detector was trained on data pooled from both models, so it outputs a single score for the question of whether any robot produced the text. The three scores are independent; a passage can score high on all three or high on one and low on the others.

Why it matters

Who is affected

The performance gap between the two model-specific detectors is notable. On cross-validation the ChatGPT detector scores 0.930 versus 0.906 for Claude, and on held-out authors it scores 0.935 versus 0.925. At 1,000 letters the difference widens further, with the ChatGPT detector at 0.952 compared to 0.858 for the Claude detector. ChatGPT is also easier to detect on short text: at 100 letters the ChatGPT detector reaches 0.709 while the Claude detector sits at 0.645. These differences suggest the two models leave measurably different letter-level traces even when asked to do the same task.

The cross-detector test is also revealing. When the Claude detector is shown ChatGPT text it scores 0.792, and when the ChatGPT detector is shown Claude text it scores 0.800. Both are well above the 0.5 baseline a coin toss would produce. That means a single-model detector retains useful signal when pointed at a model it was not trained on, which matters practically when you do not know which model generated a piece of text.

Who is affected

Anyone relying on the original meatbag.atomic14.com tool to screen text should be aware of what was and was not updated. The original detector, which only knew Claude Opus 5, would call ChatGPT output "meatbag" because it had no ChatGPT class. The updated site now runs all three detectors in parallel, so a single submission gets a Claude score, a ChatGPT score, and a generic robot score. Users who were already dismissing low Claude scores as evidence of human authorship need to reconsider, because a low Claude score still leaves the ChatGPT and generic detectors to check.

What to watch next

Builders who want to integrate or replicate this detector should note a caveat Greening flags directly. The ChatGPT training data was generated through Codex, which is a coding agent and runs with its own system prompt. That system prompt likely shifts the statistical fingerprint of the output, meaning a detector trained on Codex generations may not transfer cleanly to text produced by the ChatGPT web app or API without a system prompt. Anyone deploying this in a real pipeline should collect or test against samples from the specific ChatGPT interface they care about.

What to watch next

The most significant limitation Greening calls out is coverage. The detector knows Claude Opus 5 and ChatGPT as generated through Codex. Gemini, Llama, DeepSeek, and any other model are unknown to it, so a low score from all three detectors is not evidence that a person wrote the passage. The system detects a learned style rather than a model identity, and that distinction matters: in testing, one held-out writing style that was never seen during training scored 0.153 on the Claude detector, meaning the model confidently rated that human as more human than the average human in the training set.

Greening has flagged that both new detectors train on all three instruction styles in the data for robustness, but says more data with more style variation is still needed. The practical question for a builder is how that generalizes when the Claude or ChatGPT output in the wild uses a custom system prompt or a fine-tuned variant. The next concrete thing to verify is whether gathering held-out generations from the ChatGPT web interface produces the same detector accuracy as the Codex-sourced training set, or whether the system-prompt difference causes a measurable drop.