OpenAI boosts Astra's eval metrics, and continues to change others

What shipped

I now have enough source material from the Fortune article. The article is behind a paywall but the HTML contains the key article content. Let me compile the key facts and write the article.

Key verifiable facts from the source: - OpenAI's model: GPT-6 Astra - Blog post was planned for 2 p.m. ET on Sept. 3 - OpenAI's X account tweeted the link at 3:32 p.m. - Sam Altman posted at 3:50 p.m. - First archive snapshot: 2:23 p.m. ET, Sept. 3 - Hallucination rate: originally 4.2%, changed to 2% in the 6th archive snapshot (5:20 p.m.), then back to 4.2% - GPT-5.6 Sol hallucination rate: 12.2% → 9.4% → back to 12.2% - ExploitBench (cybersecurity): GPT-5.6 Sol went from 5.5% to 11.5% (OpenAI investigating reverting) - FrontierMath Tier 4 (v2): Astra stays at 97.6% - Anthropic's Fable 5.1 FrontierMath score: 87.8% → 78% → back to 83% - GPT-5.6 Sol FrontierMath: 83% → 80.5% → back to 83% - ARC-AGI-3: pre-publication draft showed 98.6%, live blog shows 99.99% - Arc Prize Foundation independently found 99.9% with a powerful harness, 63% with standard harness - Author: Emily Forlini, Fortune, published September 4, 2026 - Post type: announcement

OpenAI released a blog post announcing GPT-6 Astra, its latest large language model, on September 3, 2026, but the rollout was anything but smooth. The company originally planned to publish the post at 2 p.m. ET. The post did briefly appear around that time, was then pulled for reasons OpenAI declined to fully explain, and did not become widely accessible until the early evening. When Fortune and other outlets compared the version that eventually loaded to archived snapshots from earlier that afternoon, they found that several evaluation scores had shifted — some in ways that made Astra appear stronger and rival models appear weaker.

What improved

This piece reconstructs what changed, in which metrics, and by how much. It is written for developers and product teams that use benchmark figures to guide model selection, and for anyone tracking how AI companies communicate model performance at launch.

OpenAI's X account shared a link to the Astra blog post at 3:32 p.m. ET on September 3, but the URL was returning an error for many readers at that moment. Sam Altman posted his own link at 3:50 p.m., noting that the team had hit a snag deploying the post. The company initially told Fortune the delay was caused by a bug in its content management system; it later said the cause was an internet outage. OpenAI added that whatever forced the retraction was unrelated to the benchmark performance figures in the post.

By the time the post was loading reliably, roughly two hours after the planned publication window, the numbers inside it had changed from the version that had briefly been live at 2:23 p.m. ET — the timestamp of the first internet archive snapshot. Additional changes have continued to appear even after that final version became publicly accessible, making the post a moving target rather than a fixed record of the model's capabilities at launch.

What you gain

The most striking revision involved Astra's reported hallucination rate. Across the first five archival snapshots, ending with one taken at 3:11 p.m. ET, the rate was listed as 4.2%. In the sixth snapshot, captured at 5:20 p.m., it had been halved to 2%. The figure for Astra's predecessor, GPT-5.6 Sol, dropped in parallel from 12.2% to 9.4%. As of the time Fortune published its report, both figures had reverted to the original 4.2% and 12.2%.

On the FrontierMath Tier 4 (v2) mathematics evaluation, Astra's own score held steady at 97.6% across all snapshots. However, the scores listed for Anthropic's Fable 5.1 and for GPT-5.6 Sol moved. Fable 5.1 went from 87.8% in the first snapshot down to 78% by 5:17 p.m., before climbing back to 83%. Sol went from 83% to 80.5% and then back to 83%. A separate shift involved the ExploitBench cybersecurity evaluation, where Sol's score jumped from 5.5% to 11.5%; OpenAI said it was investigating whether to revert that number because the 11.5% result reflected a reasoning level not commercially available for Sol.

Developers and researchers who benchmark models for production use have the most direct exposure to this kind of post-launch revision. When a benchmark figure on an official release page changes by several percentage points within hours of publication — and when the direction of the change happens to favor the announcing company — it creates a real problem for teams trying to make objective comparisons. If you pulled Astra's hallucination rate from the Fortune pre-publication draft, the 2 p.m. archive, the 5:20 p.m. archive, or the current live page, you would have seen four different numbers.

How to get it

Anyone citing OpenAI's ARC-AGI-3 score should also note this specific shift: an embargoed pre-publication draft given to Fortune listed Astra's score at 98.6%, but the live blog now shows 99.99%. OpenAI said that differences between draft and final version are normal because the company verifies evaluations before publication. The Arc Prize Foundation, which independently runs the benchmark, found that Astra scored 99.9% when given a powerful custom harness — a set of tools available to the model during evaluation — and 63% under the benchmark's standard harness. Both figures are still significantly above any other model currently in public release, but the conditions attached to a score matter as much as the score itself.

GPT-6 Astra is available through OpenAI's products and API as of the September 3 launch. The official blog post at openai.com is the company's primary source of record for its evaluation claims, though the figures there have continued to change since publication. OpenAI's disclaimers on the page note that evaluation scores represent the maximum at any effort level and may differ from results in production; individual metrics include additional caveats in footnotes.

For any team running their own assessments, the most defensible approach right now is to treat the published numbers as a starting point rather than a definitive reference. Verify the ARC-AGI-3 score against the Arc Prize Foundation's independent reporting, noting whether the result used the standard harness or a custom one, since that gap between 63% and 99.9% is large enough to significantly affect how you interpret the claim. For ExploitBench and hallucination rate, archive the page immediately when you read it and note the timestamp, since both figures have moved since launch.

What to watch next

OpenAI said it is currently investigating whether to revert the ExploitBench figure for GPT-5.6 Sol from 11.5% back to 5.5%, and the hallucination rates for both Astra and Sol have already cycled back to their original published values. Watching whether those numbers stabilize — and whether OpenAI publishes a changelog or correction notice on the post — will be the clearest signal of how the company plans to handle transparency around post-launch benchmark revisions going forward.

The broader issue that this episode surfaces is how companies report evaluation scores for frontier models. OpenAI itself acknowledged that metrics have noise within a few percentage points depending on the checkpoint, scaffold, and evaluation run used. That is a technically honest statement, but it also means two researchers reading the same benchmark name on the same page at different hours could be comparing measurements taken under different conditions. Watching how OpenAI, Anthropic, and the Arc Prize Foundation coordinate on independent reporting — and whether the industry moves toward time-stamped, versioned benchmark disclosures — is the structural question this launch raised that the metrics themselves have not answered.