Online Dense Reinforcement Learning with Planning-Grounding Agent for UI

In November 2026, researchers at Apple Machine Learning Research published a new framework for mobile navigation agents on live iOS devices. Authored by Peitong Duan, Seyed Khashaiar Gatmiry, Jeff Nichols, Emmanuel Abbe, and Samy Bengio, the work appears in Apple Machine Learning Research's report. The framework trains models directly on real operating systems, avoiding the brittle execution of static datasets.

This article examines Apple's decoupled agent architecture, the infrastructure managing hundreds of virtual iOS devices, and the dual-reward training mechanisms enabling error recovery. It is written for mobile developers and machine learning engineers assessing compact on-device automation models against costly frontier vision-language systems.

Online Dense Reinforcement Learning: what actually changed

Frontier vision-language models demonstrate capable navigation, but their substantial inference costs prohibit everyday consumer use. Conversely, smaller models trained via supervised fine-tuning or offline reinforcement learning often memorize static trajectories, causing brittle execution and leaving agents unable to recover from errors. Apple researchers altered this paradigm by training navigation models directly on live iOS applications. Operating inside live environments exposes the agent to dynamic layouts, latency shifts, and real interface failures during training rather than relying on curated, static demonstration logs.

To make live training feasible, the framework decouples the autonomous agent into two distinct functional components: a Planning agent and a UI Grounding model. The planning component evaluates high-level user instructions and manages overall task trajectory, while the UI grounding model translates those strategic steps into precise screen interactions. This separation prevents confusion between semantic reasoning and coordinate-level pixel mapping. By isolating tactical screen grounding from strategic planning, the system maintains stable execution even when third-party applications update their visual styling, component placement, or underlying navigation hierarchies.

Online Dense Reinforcement Learning: how it works

The reinforcement learning loop is driven by an LLM-as-a-Judge system that delivers a dual reward mechanism during task execution. Instead of relying exclusively on a sparse signal at the end of an attempt, the framework combines dense, step-level feedback to evaluate intermediate navigation progress with a terminal reward for total task completion. Policy updates are governed by Proximal Policy Optimization using step-level advantage estimation. This mathematical structure ensures the policy learns which discrete screen gestures contribute to forward progress, allowing the agent to correct mistaken screen transitions before exhausting its step budget.

To stabilize the critic network during visual rollouts, Apple introduced GiGPPO, a neighbor-conditioned value estimation technique. GiGPPO stabilizes critic estimations by referencing reward signals from visually similar states across concurrent trajectories, mitigating noisy value predictions when visually identical screens yield different outcomes. This training process is supported by custom infrastructure orchestrating hundreds of concurrent virtual iOS devices. Distributing rollouts across parallel virtualized instances generates the high-throughput environmental feedback required for stable online optimization, enabling continuous exploration across diverse application states without stalling model updates during interface delays.

Online Dense Reinforcement Learning: why it matters now

Empirical evaluations demonstrate that training directly on live applications accelerates convergence while surpassing traditional baseline methods. Tested against rigorous benchmarks including UINavBench, iOSWorld, and AndroidWorld, Apple's online reinforcement learning agents substantially outperformed both supervised fine-tuning baselines and terminal-only reinforcement learning models. Providing dense step-level rewards directly addressed the policy collapse commonly observed in sparse terminal setups, where agents struggle to identify which specific interaction caused overall task failure. By resolving credit assignment at every step, the framework reached target performance metrics significantly faster than standard offline training regimes.

The performance of compact models under this framework presents compelling efficiency metrics against industry baselines. In out-of-distribution evaluations on AndroidWorld environments, Apple's 8B parameter reinforcement learning agent achieved nearly a sevenfold improvement over the supervised fine-tuning baseline. Furthermore, on real iOS applications, the 8B agent outperformed Gemini-3.5-Flash and tied with Gemini-3.1-Pro. These results indicate that specialized online reinforcement learning allows an 8B model to rival or surpass frontier vision-language systems on mobile automation tasks, avoiding the prohibitive operational expenses and cloud latency associated with larger commercial models.

Online Dense Reinforcement Learning: who is affected

Mobile application developers and quality assurance engineers face direct operational impacts from these findings. Automated testing on mobile platforms has traditionally relied on brittle test scripts that break whenever layout hierarchies shift. Because Apple's decoupled agent architecture combines a planning agent with visual grounding trained across live apps, automated testing suites can explore interface paths autonomously. Development teams can use such models to detect navigational regressions, broken links, and unresponsive touch targets without manual script authoring, significantly reducing the labor required to certify complex applications across different operating system releases.

Device manufacturers and mobile platform engineers are also influenced by the validation of 8B parameter models. Demonstrating that an 8B model matches Gemini-3.1-Pro on iOS apps provides hardware makers with a viable path toward running local, on-device assistants. Rather than sending continuous screen captures over cellular networks to hosted cloud clusters, consumer devices can run compact models locally. This on-device execution profile improves response latency and user privacy while lowering server costs for device vendors seeking to deliver autonomous operating system assistance.

Online Dense Reinforcement Learning: what to watch

Infrastructure scalability remains a central technical dimension to watch as online reinforcement learning expands. Orchestrating hundreds of concurrent virtual iOS devices requires significant compute and virtualization management. Observers will track whether Apple provides open-source access or shared tooling for its iOS simulation environment to aid external academic research. Furthermore, the neighbor-conditioned value estimation mechanics of GiGPPO invite scrutiny regarding how well critic stabilization transfers to desktop operating systems and complex web interfaces, where dynamic DOM trees and varying screen aspect ratios introduce far larger visual state permutations.

Longer term, researchers will monitor how online-trained agents handle production app variance across multiple software update cycles. While nearly a sevenfold out-of-distribution gain on AndroidWorld confirms strong transfer capabilities across mobile operating systems, real-world deployment requires continuous handling of two-factor authentication flows, system permission prompts, and transient network errors. The ongoing trajectory from the research team—including Peitong Duan, Seyed Khashaiar Gatmiry, Jeff Nichols, Emmanuel Abbe, and Samy Bengio—will reveal whether Apple intends to embed this decoupled architecture into future consumer-facing operating system features across iOS.