Infinity: A runtime for extreme agent concurrency

AWS and UC Berkeley released Infinity under the Apache 2.0 license as part of the Hydro project, introducing an asynchronous runtime that suspends software agents during tool calls. Rather than running blocking loops that hold operating system processes while waiting on tools, Infinity treats execution as stackless coroutines. According to aws.amazon.com's report, the runtime allows 75,000 concurrent agents to fit in the 8 GB memory of a Raspberry Pi 5.

This report analyzes how bounded execution slices, actor-style input queues, and non-blocking tools replace monolithic agent processes. The breakdown covers benchmark data from AMD EPYC server hardware, explains how synthetic tool calls deliver asynchronous events, and evaluates operational trade-offs for deploying agent workloads on serverless infrastructure. Engineers building multi-agent systems will find technical specifications covering memory footprints and execution guarantees.

A runtime for extreme agent concurrency: what actually changed

Conventional agent frameworks maintain active execution threads and call stacks throughout an agent's lifecycle, consuming compute capacity while idling on external tools. Infinity structures agent execution around stackless coroutines, similar to asynchronous functions in Rust. The runtime runs each agent as bounded slices. In each slice, it loads conversation history from a pluggable store, executes one model completion, saves the output, and dispatches at most one tool call before terminating. Tool calls act as await points, discarding in-memory state when a request begins.

To evaluate concurrency limits, AWS and UC Berkeley tested 80,000 agents against a simulated model on a single thread of an AMD EPYC 9R14 processor using in-memory stores. Each agent completed twenty turns, generating 3.2 million total completions. Every turn processed a 1.2 KB streaming completion, dispatched a tool, and handled the return message through an input queue. Resident memory scaled linearly at roughly 100 KB per idle agent. Retaining no task stack or connection, 75,000 concurrent agents fit in the 8 GB memory of a Raspberry Pi 5.

A runtime for extreme agent concurrency: how it works

Infinity manages concurrency through an agent system modeled on the actor pattern, where agent threads share tools, storage, and models without mutable state. Each conversation thread maintains a dedicated input queue that receives prompts and tool outputs tagged with a thread identifier. An internal scheduler manages active threads like an async executor schedules futures, running slices for different threads in parallel. The runtime guarantees sequential message processing within each thread to prevent corruption. When a thread has no pending messages, its task exits until new inputs arrive.

The programming interface exposes this flow through Rust components, including AgentSystemBuilder, InMemoryConversationStore, and InMemoryStateStore paired with models like Anthropic Claude Sonnet 4.6. Rather than blocking until work finishes, a tool's execute method launches background jobs through Tokio spawns and returns immediately to end the slice. When tasks complete, background logic creates a tool result message carrying the ToolCallId and ProviderCallId. The message returns to the agent's input queue using context message senders. Arriving messages trigger the runtime to launch a slice that restores history and continues execution.

A runtime for extreme agent concurrency: why it matters now

Modern agent workloads spend most of their operational duration idling while waiting for builds, pull request reviews, or human approvals. In conventional blocking harnesses, holding open server processes during extended waiting periods incurs significant cloud hosting costs and wastes resources. Because Infinity persists conversation history to external stores and terminates the slice upon tool dispatch, serverless functions shut down completely while an agent waits. Even if a tool operation takes several days, zero compute runs during the delay, drastically reducing the cost of hosting large agent swarms.

The architecture also addresses synchronization and coordination bottlenecks in multi-agent orchestration. Infinity introduces synthetic tool calls to deliver asynchronous events and inter-agent messages directly into existing model contexts without requiring specialized APIs. Messages directed across agent threads appear as structured tool events delivered to target input queues. This mechanism allows multiple autonomous agents to exchange data and coordinate without maintaining persistent network connections or blocking execution threads. Systems can scale to thousands of interacting agents while preserving message ordering within individual conversation threads and executing separate threads concurrently.

A runtime for extreme agent concurrency: who is affected

Software teams building code generation assistants, repository triage bots, and workflow automation tools are directly impacted by this architectural transition. Engineering groups operating fleets of agents can move away from resource-intensive container clusters toward ephemeral serverless functions backed by pluggable stores. Developers authoring custom agent tools must adjust their code from synchronous return patterns to decoupled asynchronous task dispatchers. Rather than returning outputs directly from tool calls, developers must write tools that initiate background tasks and transmit completion payloads to input queues, ensuring host execution slices terminate without blocking worker threads.

Infrastructure architects and database administrators supporting agent platforms will see operational demands shift from compute capacity to storage throughput. While idle agent memory drops to roughly 100 KB, maintaining tens of thousands of concurrent agents creates frequent read and write operations against persistence stores as slices start and finish. Organizations adopting the Apache 2.0 open source framework must verify that their database backends sustain high input and output rates when background tool calls complete simultaneously. Engineering teams must design persistence layers capable of handling rapid history serialization across large agent fleets.

A runtime for extreme agent concurrency: what to watch

Because Infinity is an early research release within the Hydro project, several operational aspects require ongoing development and community validation. The collaboration between AWS and UC Berkeley is seeking feedback on how the runtime performs when backed by distributed production databases rather than local in-memory stores. Observers should watch how the runtime manages persistence latency and state serialization overhead under enterprise traffic. Additionally, the stability of synthetic tool calls across large agent networks remains an open area of exploration, particularly regarding queue depth limits, scheduling contention, and backpressure handling.

Ecosystem integration represents another crucial milestone for the project. While the initial release showcases Rust implementations utilizing Tokio async primitives, the broader artificial intelligence engineering landscape remains heavily centered on Python frameworks. Future releases will show whether AWS and UC Berkeley provide native Python bindings or standardized deployment templates for serverless cloud environments like AWS Lambda. Platform engineers should track public updates to the Hydro repository to observe how the runtime handles network partition recovery, distributed state store connectors, and execution guarantees in production multi-agent deployments.