Home / Blog / A 149M model answers 70% of phone agent steps locally,…
Tech News

A 149M model answers 70% of phone agent steps locally, Claude does the rest

Milo Pointer uses a 149M local model to answer 70% of phone agent actions on device, cutting Claude Opus 5.5 batch API costs to $2.82 per thousand steps.

By Dillip Chowdary • Nov 18, 2026 • Source: github.com

A 149M model answers 70% of phone agent steps locally, Claude does the rest
Image: github.com

At a glance

Product
Milo Pointer
Model size
149 million parameters
License
Apache 2.0
Training compute
6.4 hours on one NVIDIA T4
API evaluation cost
$86.75 total Claude Batch costs
Preprint date
2026 (DOI: 10.5281/zenodo.23818333)

Independent researcher Chaitanya Talapalli published Milo Pointer, an open-source 149M text encoder that resolves mobile actions locally before routing complex requests to frontier models. According to github.com's report, the system scores 86.1 percent on AndroidControl-Low under screenshot evaluation protocols. Each training run takes 6.4 hours on an Nvidia T4 GPU, while Claude API evaluation calls cost $86.75 in total at batch prices.

This report covers the hybrid architecture, benchmark scores, cost reductions, operational limits, and reproduction steps for software engineers building smartphone agents. System architects can evaluate how small on-device models reduce latency and cloud API expenses while maintaining task completion accuracy.

01Benchmark accuracy across one thousand offline steps

The local model handles single-step interactions by predicting bounding box click targets and text inputs directly from device states. Across 1,000 held-out AndroidControl steps, the standalone 149M encoder completed 869 steps accurately without external assistance. Pairing the local model with an escalated frontier model increased overall accuracy while offloading execution from remote servers.

SystemRun 1Run 2Steps on deviceAPI $ per 1k steps (Batch)
Small model alone (149M)869869100%0
Claude Opus 5.5, 16 retrieved examples8828790%9.54
Small model, then Opus when unsure89289070.7%2.82
Claude Sonnet 5.5, 16 retrieved examples8578620%4.80
Small model, then Sonnet when unsure88188970.7%1.41

Routing unsure predictions to Claude Opus 5.5 delivered 892 correct actions in run one and 890 in run two. The combined setup answered 70.7 percent of steps on device, cutting API costs from $9.54 to $2.82 per thousand steps. Cascading to Claude Sonnet 5.5 achieved comparable gains while reducing expenses to $1.41.

02Confirmatory test scores and pre-registered hypotheses

The project registered forty experimental hypotheses with verifiable timestamps in project documentation before scoring test data. A dedicated confirmatory trial evaluated 650 steps sourced from test episodes excluded from earlier sample pools. This confirmatory run relied on a learned hand-off rule frozen beforehand to decide whether the local model should defer execution to Claude Opus 5.5.

The frozen routing strategy resolved 599 steps correctly compared to 581 steps achieved by standalone Claude Opus 5.5. That outcome established a statistically significant advantage of 2.8 percentage points with a p-value of 0.003 and a ninety-five percent confidence interval between 1.0 and 4.5 points. Across all 1,650 pooled test steps, the gain remained 1.5 points, exceeding the 85.2 percent benchmark reported for OS-Atlas-7B.

03On-device latency and planner accuracy gaps

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

Hardware evaluations reveal execution bottlenecks when running the text encoder without dedicated neural acceleration. Running the exported model on a Google Pixel 6a central processing unit required a median duration of 3.4 seconds per step. Deploying the model inside production applications demands optimization through runtime engines to prevent interface delays during interactive sessions.

A post-paper evaluation on October 30, 2026 examined a 1.5-billion-parameter Qwen planner trained with low-rank adaptation on human instructions. Across 1,804 in-domain steps, human instructions yielded 90.2 percent accuracy, whereas raw goal prompts managed 19.3 percent. The planner scored 53.0 percent, missing the registered ten-point margin from human guidance while confidence scores failed to detect invalid plans.

Note: The single-step cascade lost accuracy on live AndroidWorld tasks, and the small model alone does not outperform Opus when provided with identical examples. The architecture also relies on pre-segmented human step instructions rather than generating task trajectories independently.

04Verification commands and offline weight downloads

Researchers can replicate all reported headline evaluation statistics locally without triggering cloud model invocations or incurring billing charges. The repository contains automated test suites that validate claimed accuracy metrics against frozen evaluation outputs. Developers configure a local Python environment using standard packages before running validation scripts against curated AndroidControl benchmark data files.

  1. Validation execution Run the reproduce shell script or invoke pytest to assert headline numbers without cloud expenses.
  2. Environment configuration Install project dependencies from requirements files and configure the Anthropic API key environment variable.
  3. Weight retrieval Download the Milo Pointer model files from Hugging Face into the local release directory.
  4. Checkpoint staging Copy the clean retrain checkpoint file into the training directory prior to executing evaluation scripts.

Re-scoring the curated dataset requires unpacking the compressed text file built from the public AndroidControl repository. Running latency benchmarks evaluates local model checkpoints, whereas the distributed safetensors archive contains identical model parameters for standalone inference. Developers must supply Anthropic API credentials only when reproducing comparative Claude frontier benchmark branches.

05Python execution pipeline and Apache license

Integrating the pointer model into existing agent frameworks requires importing the primary interface class within Python scripts. Developers load model weights from the local release directory and pass screen dimensions, interactive UI elements, and goals into the prediction routine. The prediction output returns an explicit boolean flag indicating whether the agent should defer the step to a frontier model.

The underlying model weights represent a fine-tuned checkpoint derived from ModernBERT-base, trained on AndroidControl interaction data. Both the training code and released model weights are distributed under the permissive Apache 2.0 license. The research paper is archived on Zenodo under digital object identifier 10.5281/zenodo.23818333 for technical review by mobile interface developers.

Frequently asked questions

What is Milo Pointer?

Milo Pointer is an open-source 149M text encoder fine-tuned from ModernBERT-base that resolves Android UI actions locally and routes uncertain steps to frontier models.

How much does the hybrid routing approach reduce API costs?

The hybrid setup answers over 70 percent of steps locally, cutting Claude Opus 5.5 batch API expenses from $9.54 to $2.82 per thousand steps.

How fast does the model run on mobile hardware?

On a Google Pixel 6a central processing unit, the local model runs via ONNX Runtime with a median latency of 3.4 seconds per step.

Where can developers download the weights and code?

The code is available on GitHub under chayworks/milo-pointer and model weights are published on Hugging Face under coolnow/milo-pointer under the Apache 2.0 license.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →