GPT-5.6 vs. Claude Fable 5 for Physical AI, which performs best?

I'll pull the JuliaHub evaluation and the HN thread so the paragraphs stick to reported facts only.JuliaHub published a sealed Physical AI evaluation pitting OpenAI’s GPT-5.6 family (terra, sol, luna) against Anthropic’s claude-fable-5 inside the Dyad agent on five modeling and simulation problems. Difficulty-weighted scores ranked fable at 0.889 ($9.60 per trial, 16.1 min), sol at 0.814 ($1.74, 13.4 min), terra at 0.786 ($1.25, 12.6 min), and luna at 0.727 ($3.26, 25.0 min). The study ran 52 graded trials: three per model on four core problems and one long-horizon trial each on the fifth. Authors Chris Rackauckas, Anas Abdelrehim, and Anshul Singhvi held the harness, problems, reasoning effort (xhigh), 1M context, and 128k token budget fixed so the model was the only variable.

The pipeline is not “did the code compile.” Each trial packages a model, the Dyad harness, pinned config, and a sealed problem; the agent must derive physics, write Dyad, compile, simulate, and verify, then a grader compares full trajectories to sealed ground truth. Problems scale from constitutive and constrained consistency through relativistic dynamics and steady-state linearization to P5, NASA’s HL-20 lifting body: six-degree-of-freedom dynamics, roughly 170 wind-tunnel tables, standard atmosphere, and a control-surface mixer from two NASA technical memos, graded on eight physics scenarios. No model fully solved P5. Work-style fingerprints differed: fable averaged 28 tool calls and spent more time deriving and testing; luna averaged 59 calls and more hunting/editing; terra was cheapest and fastest but spent more of its clock editing.

For engineers and builders, the failure mode is specific: agents can pass every self-written check while the physics is wrong. Sol renormalized a pinned four-velocity and still satisfied mass-shell consistency, handing the particle 27% excess longitudinal momentum. Luna ran 22 self-checks on a constrained-consistency trial and still shipped a solution 30% off. Terra built a nonlinear reference for linearization validation but truncated both the required horizon and the reference (stop 5.0 to 3.0), blinding its own check. Fable’s edge showed up in adversarial verification—tolerance sweeps, independent reimplementations, deliberate sign flips—not in writing more code.

On the market and product axis, JuliaHub’s incentive is not to crown a single lab: Dyad ships multiple agent backends, so the firm wins when users pick the model that works. Fable led capability and was the only model to sweep all twelve core-problem trials, but at three to eight times GPT trial cost (~$125 for the full fable study). Sol was the value story: second on merit at roughly a fifth of fable’s price. A companion harness experiment matters as much as the leaderboard: the same frontier model scored 0.899 in Dyad AI versus 0.533 in stock Claude Code at nearly the same spend ($4.20 vs $4.65), a 0.366-point gap more than double the 0.162 gap between best and worst models in the main study.

Practical takeaway: choose the instrument first, then the model. If the requirement is highest probability that the committed model matches sealed physics, claude-fable-5 is the pick inside Dyad. If the requirement is physics per dollar for day-to-day work, gpt-5.6-sol is the rational default; terra is the cheapest credible path. Watch for JuliaHub’s full harness-versus-harness write-up and for whether other physical-AI stacks report trajectory-graded, sealed evaluations instead of self-authored unit tests—especially on unsolved long-horizon problems like HL-20, where every model still failed lateral-directional response and only fable tracked full-flight altitude within about a third of a meter of the reference.