A 149M model answers 70% of phone agent steps locally, Claude does the rest
Milo Pointer uses a 149M local model to answer 70% of phone agent actions on device, cutting Claude Opus 5.5 batch API costs to $2.82 per thousand steps.
By Dillip Chowdary • Nov 18, 2026 • Source: github.com

At a glance
- Product
- Milo Pointer
- Model size
- 149 million parameters
- License
- Apache 2.0
- Training compute
- 6.4 hours on one NVIDIA T4
- API evaluation cost
- $86.75 total Claude Batch costs
- Preprint date
- 2026 (DOI: 10.5281/zenodo.23818333)
In this post
Independent researcher Chaitanya Talapalli published Milo Pointer, an open-source 149M text encoder that resolves mobile actions locally before routing complex requests to frontier models. According to github.com's report, the system scores 86.1 percent on AndroidControl-Low under screenshot evaluation protocols. Each training run takes 6.4 hours on an Nvidia T4 GPU, while Claude API evaluation calls cost $86.75 in total at batch prices.
This report covers the hybrid architecture, benchmark scores, cost reductions, operational limits, and reproduction steps for software engineers building smartphone agents. System architects can evaluate how small on-device models reduce latency and cloud API expenses while maintaining task completion accuracy.
01Benchmark accuracy across one thousand offline steps
The local model handles single-step interactions by predicting bounding box click targets and text inputs directly from device states. Across 1,000 held-out AndroidControl steps, the standalone 149M encoder completed 869 steps accurately without external assistance. Pairing the local model with an escalated frontier model increased overall accuracy while offloading execution from remote servers.
| System | Run 1 | Run 2 | Steps on device | API $ per 1k steps (Batch) |
|---|---|---|---|---|
| Small model alone (149M) | 869 | 869 | 100% | 0 |
| Claude Opus 5.5, 16 retrieved examples | 882 | 879 | 0% | 9.54 |
| Small model, then Opus when unsure | 892 | 890 | 70.7% | 2.82 |
| Claude Sonnet 5.5, 16 retrieved examples | 857 | 862 | 0% | 4.80 |
| Small model, then Sonnet when unsure | 881 | 889 | 70.7% | 1.41 |
Routing unsure predictions to Claude Opus 5.5 delivered 892 correct actions in run one and 890 in run two. The combined setup answered 70.7 percent of steps on device, cutting API costs from $9.54 to $2.82 per thousand steps. Cascading to Claude Sonnet 5.5 achieved comparable gains while reducing expenses to $1.41.
02Confirmatory test scores and pre-registered hypotheses
The project registered forty experimental hypotheses with verifiable timestamps in project documentation before scoring test data. A dedicated confirmatory trial evaluated 650 steps sourced from test episodes excluded from earlier sample pools. This confirmatory run relied on a learned hand-off rule frozen beforehand to decide whether the local model should defer execution to Claude Opus 5.5.
The frozen routing strategy resolved 599 steps correctly compared to 581 steps achieved by standalone Claude Opus 5.5. That outcome established a statistically significant advantage of 2.8 percentage points with a p-value of 0.003 and a ninety-five percent confidence interval between 1.0 and 4.5 points. Across all 1,650 pooled test steps, the gain remained 1.5 points, exceeding the 85.2 percent benchmark reported for OS-Atlas-7B.
03On-device latency and planner accuracy gaps
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
Hardware evaluations reveal execution bottlenecks when running the text encoder without dedicated neural acceleration. Running the exported model on a Google Pixel 6a central processing unit required a median duration of 3.4 seconds per step. Deploying the model inside production applications demands optimization through runtime engines to prevent interface delays during interactive sessions.
A post-paper evaluation on October 30, 2026 examined a 1.5-billion-parameter Qwen planner trained with low-rank adaptation on human instructions. Across 1,804 in-domain steps, human instructions yielded 90.2 percent accuracy, whereas raw goal prompts managed 19.3 percent. The planner scored 53.0 percent, missing the registered ten-point margin from human guidance while confidence scores failed to detect invalid plans.
Note: The single-step cascade lost accuracy on live AndroidWorld tasks, and the small model alone does not outperform Opus when provided with identical examples. The architecture also relies on pre-segmented human step instructions rather than generating task trajectories independently.
04Verification commands and offline weight downloads
Researchers can replicate all reported headline evaluation statistics locally without triggering cloud model invocations or incurring billing charges. The repository contains automated test suites that validate claimed accuracy metrics against frozen evaluation outputs. Developers configure a local Python environment using standard packages before running validation scripts against curated AndroidControl benchmark data files.
- Validation execution Run the reproduce shell script or invoke pytest to assert headline numbers without cloud expenses.
- Environment configuration Install project dependencies from requirements files and configure the Anthropic API key environment variable.
- Weight retrieval Download the Milo Pointer model files from Hugging Face into the local release directory.
- Checkpoint staging Copy the clean retrain checkpoint file into the training directory prior to executing evaluation scripts.
Re-scoring the curated dataset requires unpacking the compressed text file built from the public AndroidControl repository. Running latency benchmarks evaluates local model checkpoints, whereas the distributed safetensors archive contains identical model parameters for standalone inference. Developers must supply Anthropic API credentials only when reproducing comparative Claude frontier benchmark branches.
05Python execution pipeline and Apache license
Integrating the pointer model into existing agent frameworks requires importing the primary interface class within Python scripts. Developers load model weights from the local release directory and pass screen dimensions, interactive UI elements, and goals into the prediction routine. The prediction output returns an explicit boolean flag indicating whether the agent should defer the step to a frontier model.
The underlying model weights represent a fine-tuned checkpoint derived from ModernBERT-base, trained on AndroidControl interaction data. Both the training code and released model weights are distributed under the permissive Apache 2.0 license. The research paper is archived on Zenodo under digital object identifier 10.5281/zenodo.23818333 for technical review by mobile interface developers.
Frequently asked questions
What is Milo Pointer?
Milo Pointer is an open-source 149M text encoder fine-tuned from ModernBERT-base that resolves Android UI actions locally and routes uncertain steps to frontier models.
How much does the hybrid routing approach reduce API costs?
The hybrid setup answers over 70 percent of steps locally, cutting Claude Opus 5.5 batch API expenses from $9.54 to $2.82 per thousand steps.
How fast does the model run on mobile hardware?
On a Google Pixel 6a central processing unit, the local model runs via ONNX Runtime with a median latency of 3.4 seconds per step.
Where can developers download the weights and code?
The code is available on GitHub under chayworks/milo-pointer and model weights are published on Hugging Face under coolnow/milo-pointer under the Apache 2.0 license.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement