Presentation: Beyond the Prototype: Scaling Framework Agnostic AI Agent

Deepak Mohanakumar Chandramouli and Bhumik Vinodkumar Thakkar presented "Beyond the Prototype: Scaling Framework Agnostic AI Agent Infrastructure with Ray" in a 46-minute, 53-second QCon AI session. InfoQ's report records the talk. Chandramouli, Senior ML Engineering Leader at Apple with over 20 years in distributed systems and scalable data, compute, and ML infrastructure, said the focus is building an agent and running it at scale with Ray, an open-source distributed computing framework for training, fine-tuning, and inference. Thakkar, Senior Software Engineer at Apple, is an expert in artificial intelligence and large-scale distributed systems.

Engineers moving an agent from local Python to FastAPI and Kubernetes are the audience. The session covers the building blocks, a trip-planner, the lifecycle locally and remotely, the same lifecycle on Ray, and the production learnings that follow. QCon AI is a practitioner-led event on scaling these workloads safely, with direct access to architectural playbooks and failure metrics that peer organizations use in production.

Beyond the Prototype: what actually changed

Chandramouli started with agent building blocks, after sessions on prompts, context, and memory, then the developer lifecycle locally and remotely. Python comes first, Ray comes next, and production learnings close the session. The baseline composes tools, stitches a graph from the LLM to those tools, and adds a checkpointer for memory. That compiled graph is the agent inside FastAPI. A test starts the server, sends a RESTful request, and reads the trails as tools run, so the builders can course-correct the agent.

The app is containerized into a Kubernetes deployment from a standard spec. Chandramouli said this works well as a classic RESTful service. For a multi-turn agent that keeps state at scale and orchestrates many tool calls, the same pattern hits the ceiling. Thakkar said, "That was a quick ride to prod for the agent, but we are not done yet." Before the trip-planner accepts real user queries, enough capacity must be provisioned and scheduling decisions for that particular agent must be made. A bunch of things are still missing.

Beyond the Prototype: how it works

Four blocks make up the agentic runtime Chandramouli described. The LLM drives each request and controls planning, tool calls, summarizing, and guardrails. Tools access resources and take actions. Memory personalizes the user experience and seeds knowledge. The prompt sets a persona, as in "you're Claude, do this," passes the user's intent, and carries directives: guidelines, rules, and what to do and what not to do. Those directives control orchestration. Traces show the reasoning loop and the controls. Guardrails cover the incoming request, planning, orchestration, and the summarized result.

The worked example is a trip-planner. Search finds destinations from seasonal parameters, a weather tool looks up conditions for a city, and a synthesizer writes a day-to-day itinerary. On a three-day winter trip, the model plans, picks tools, and issues an execution. A system runs the tool call and returns the results. The model may then fetch weather, synthesize, and answer the user. A static graph is more deterministic, and an LLM-chosen plan is more non-deterministic. Chandramouli said each approach has its own pros and cons.

Beyond the Prototype: why it matters now

Chandramouli said popular agent frameworks compose graphs, let an LLM decide some steps, and still control the flow. They run tools at scale, handle multi-agent interactions, and manage memory, stateful checkpoints, and stream processing. He called that set table stakes for multi-turn, long-context agent interactions. Async flows matter when tool calls run in parallel and many agents multiply the load in production. A standard Kubernetes deployment then shows a lot of scaling limitations, because the multi-turn agent stays stateful and orchestrates many tool calls.

Operators must see how the reasoning logic performs at scale. Chandramouli said that means inspecting traces of the reasoning loop and its controls, so the agent behaves as desired and can be made more efficient. Guardrails apply when the user request arrives, during planning, during orchestration, and when results are summarized. Thakkar said enough capacity has to be provisioned, and scheduling decisions for this agent have to be made, before the trip-planner accepts real user queries. Those items still have to be accounted for in production.

Beyond the Prototype: who is affected

Chandramouli asked how many attendees were familiar with Ray, then defined it and set the session on agents at scale. The room includes people using popular agent frameworks and people on standard Python, with a compiled graph, a checkpointer, and a FastAPI server queried by a REST request. It includes teams that containerize that server as a standard Kubernetes deployment. That deployment meets the scaling limitations once the agent is multi-turn, stateful, and busy with tool calls. He described Ray through training, fine-tuning, and inference.

The trip-planner is the shared workload. Beside the LLM, it searches destinations by season, checks weather for a city, and synthesizes a day-to-day plan. The three-day winter trip runs plan, tool execution, results returned, a possible weather fetch, synthesis, and a reply. Prompt authors set the persona, the user's intent, and the directives. Memory owners personalize the experience and seed knowledge through the checkpointer. Capacity owners face the provisioning and scheduling Thakkar placed before real queries. Both speakers work at Apple.

Beyond the Prototype: what to watch

The production list the speakers share is concrete. It covers RayCluster topology, stateful memory handling, auto-scaling on dynamic traffic, and robust control plane designs. It also covers critical guardrails, multi-region setups, and LLM-as-a-judge evaluation pipelines. Those topics follow Chandramouli's order: a local build, a remote run, the same lifecycle on Ray, then learnings from production. Static graphs remain the more deterministic option. An LLM-chosen plan remains the more non-deterministic option. He said the two have their own pros and cons.

After the quick ride that puts the FastAPI agent on Kubernetes, Thakkar said a bunch of things are still missing. Enough capacity has to be provisioned before the trip-planner accepts real user queries, and scheduling decisions for this particular agent have to be made. Trails from those RESTful requests show the reasoning loop and let builders course-correct the agent. Guardrails stay on the incoming request, planning, orchestration, and the summary. The checkpointer on the compiled graph manages memory in that vanilla build.