A 32K-window proxy that beat full context on long agent histories

Developer Niko1221 and open-source contributors released Mnemonic Proxy, a lightweight HTTP proxy designed to sit transparently between autonomous coding agents and local inference servers. Built using standard Python without external runtime dependencies, the proxy intercepts conversation history sent by coding tools such as Claude Code, Codex, Continue, and Hermes. Instead of letting context management truncate or discard operational histories, the service stores complete, verbatim message trails on disk while dynamically curating what the model actively ingests, according to github.com's report.

This report examines how the system manages large prompt accumulations, scales across different local runtime engines, and maintains accuracy on complex benchmarks without relying on vector databases. It is intended for software engineers, systems architects, and machine learning practitioners who deploy local language models for software development and need long-running agent workflows to operate stably within modest hardware constraints.

A 32K-window proxy that: what actually changed

Traditional context handling techniques for agentic workflows routinely rely on periodic lossy summarization or arbitrary message truncation once prompts exceed engine limits. Mnemonic Proxy alters this architecture by allowing coding agents to dispatch their entire, unmodified conversation logs while the proxy independently regulates the active prompt payload sent upstream. Every message, tool call, reasoning trace, and terminal output is archived verbatim to a local SQLite database on disk rather than condensed into ambiguous narrative summaries.

When older interactions are pushed out of active memory, they leave the prompt in stable blocks and are replaced with single-line receipts containing unique identifier tags. Because the prefix remains byte-identical between operational cycles, the underlying engine preserves its existing key-value cache rather than recalculating previous tokens. In single-run benchmark evaluations on standard testing hardware, this structural adjustment enabled a 32K context window to score 86 out of 100 on AgentLongBench, outperforming an LLM summarization setup that scored 47 out of 100 and a full-history baseline in an expanded window that scored 69 out of 100.

A 32K-window proxy that: how it works

The operational mechanics center on four internal stages: masking, recall, segmentation, and persistent state storage. During the masking phase, older tool executions and intermediate reasoning steps leave the immediate prompt and become compact receipt markers. When the language model requires historical specifics, an internal recall mechanism executes structured queries against the disk archive to retrieve exact file edits, command outputs, or historical decisions. The proxy handles these recall queries directly, ensuring downstream clients never see intermediate retrieval rounds even during token streaming.

When the masked context eventually nears maximum capacity, the active segment seals following a handoff note generated by the model, opening a clean interaction segment. Developers can mark vital context lines with persistent pins such as double exclamation marks or pin emojis to survive segment transitions, while temporary instructions can be prefixed with trash or tilde markers to signal automatic disposal once resolved. On backends supporting disk serialization, the system saves the engine state directly to disk slots, reducing restoration latency on a 110K-token conversation from roughly 50 seconds down to 1.4 seconds.

A 32K-window proxy that: why it matters now

Local language model inference frequently confronts rigid memory boundaries on workstation GPUs. During sustained development sessions, extensive context windows can cause substantial compute overhead, excessive memory consumption, or catastrophic prompt truncation. Hardware tests recorded a continuous 3-hour and 15-minute coding session run entirely on a single 12 GB GPU, maintaining an operational history spanning 448K tokens while capping active prompt overhead at no more than 105K tokens.

This approach bypasses common silent failure modes observed in consumer inference engines. Standard installations of Ollama and LM Studio often default to context limits of 4096 or 8192 tokens on personal computers, returning standard HTTP 200 success codes while quietly dropping portions of long prompts without alerting users. Mnemonic Proxy actively checks engine context through administrative endpoints before requests execute, refusing to boot if upstream capacity falls below 16384 tokens and enforcing automated warning flags whenever engines truncate payload volumes.

A 32K-window proxy that: who is affected

Engineers running local agent tooling on Linux, native Windows environments, and Windows Subsystem for Linux stand to see the most direct workflow adjustments. The software acts as an intermediary bridge for llama.cpp's llama-server, Strata versions 0.1.40 or higher, ds4-server, Ollama, LM Studio, and generic endpoints conforming to the OpenAI Chat Completions, Anthropic Messages, or OpenAI Responses wire standards. Users configuring agents like Claude Code, Codex, or Hermes must disable their clients' internal auto-compaction features to prevent conflicting truncation cycles.

Developers utilizing the Model Context Protocol can also deploy an included mnemonic-mcp utility to execute read-only archive searches across existing database records. For hardware testing, the software requires at least a 16K token upstream window, with 32K tokens formally advised for regular workloads. Deployments without the optional exact tokenizer package fallback to a character-ratio estimator with an automated 15 percent margin slack to avoid accidental boundary overflows on numeric or code-heavy prompts.

A 32K-window proxy that: what to watch

Field validation remains ongoing across diverse operating platforms and non-standard architectures. Native Windows operation relies on PowerShell configurations and packaged startup scripts verified on continuous integration runners rather than full desktop environments, while macOS setups remain untested and require manual compilation alongside Metal-accelerated runtimes. Experimental builds have run inside Android 13 Waydroid emulation environments using Termux, though native execution on ARM mobile chipsets has not been verified.

Engine compatibility also dictates which operational tiers are accessible. While llama-server and updated Strata installations support full slot-saving and sub-two-second state resumption from local storage, deployments using Ollama, LM Studio, or remote cloud-hosted APIs operate exclusively in base mode without disk-backed key-value acceleration. Teams adopting the software will need to track upstream releases for improved cross-platform coverage, tokenizer synchronization, and tighter integration across alternative local backends.