Google announces New vLLM and SGLang Inference Engines on TPU

Google launched native PyTorch backends for open-source model serving engines vLLM and SGLang running directly on Tensor Processing Units. The release, detailed in Google Developers Blog's report, introduces runtime stacks engineered with open-source maintainers Inferact and RadixArk on top of TorchTPU. Rather than routing execution through JAX backends, the new engines treat TPUs as standard PyTorch devices, executing models using standard PyTorch tensors, eager execution, XLA compilation via torch.compile, and torch.distributed collectives. The architecture addresses developer demand for familiar PyTorch debugging and customization workflows on TPU silicon while maintaining production-grade serving performance.

This report details the architectural changes powering TorchTPU, public benchmark results, and current enterprise production deployments. Engineering teams operating AI infrastructure, machine learning platform engineers scaling agent pipelines, and open-source contributors evaluating accelerator alternatives will find concrete operational details on memory tiering, custom kernels, and framework integration. The analysis examines how custom Pallas operators, tiered key-value cache offloading, and TPU Sync enable low-latency inference on Ironwood TPU systems. It also covers production validation from enterprise adopters like Meta, alongside steps to access source repositories and apply for community compute allocations.

Google New vLLM and SGLang Inference Engines: the announcement

Google announced the new inference engines on October 20, 2026, authored by Principal Software Engineer Qi Zhou and Group Product Manager Chris Chan as part of broader PyTorch updates. The initiative targets the heavy demands of agentic and multimodal workloads, which require responsive multi-turn interactivity and efficient memory use. While vLLM and SGLang previously ran on TPUs through JAX backends, developers frequently encountered obstacles when trying to customize and debug stacks outside PyTorch. To address this, Google built TorchTPU to run PyTorch natively on TPU hardware, partnering with Inferact, founded by vLLM creators, and RadixArk, founded by SGLang creators.

The release combines framework software with dedicated hardware compute to support open-source development. Alongside code releases, Google and vLLM launched a community contribution TPU capacity pool providing free compute to developers who actively contribute to both engines. The software uses a platform-plugin architecture, ensuring core framework components like schedulers, API servers, and worker abstractions match upstream projects while hardware optimizations progress independently. Google confirmed that existing vLLM-Jax and SGLang-JAX runtimes will continue to be maintained and supported, giving engineering teams flexibility across both machine learning environments.

What actually changed with Google New vLLM and SGLang Inference Engines

TorchTPU makes the TPU a native PyTorch device, managing tensors directly on a tpu device without wrapper types. Code runs eagerly or compiles via torch.compile using XLA, while collective communication executes through standard torch.distributed primitives. Because each TPU device runs its own PyTorch process, vLLM-TorchTPU and SGLang-TorchTPU retain upstream schedulers, API servers, and worker models while adding TPU-specific layer implementations. Custom kernels written in Pallas execute directly from PyTorch as operators within compiled graphs. This allows models like Qwen, DeepSeek, GLM, Kimi, GPT-OSS, and Gemma4 to run standard upstream PyTorch model definitions on TPU hardware.

Serving multi-turn agentic workloads prompted the creation of TPU Sync, an open-source C++ library for key-value cache management. Because agents repeatedly send expanding conversations, recomputing context adds major latency. TPU Sync reads device buffers behind PyTorch tensors directly, avoiding staging copies on the host and running transfers on separate threads outside Python during prefill and decode. This enables disaggregated serving, where the prefill host streams cache data directly to the decode host. It also enables tiered offload, transferring cache beyond High Bandwidth Memory to host dynamic random-access memory and Google Cloud Managed Lustre secondary storage.

Who should care about Google New vLLM and SGLang Inference Engines

Infrastructure engineers running production inference receive a verified alternative to GPU clusters, supported by public data from the SemiAnalysis InferenceX benchmark. Evaluating Qwen 3.5 on Ironwood TPU, also called TPU 7x, Google reported a serving cost of 2.01 cents per million tokens. On the AgentX benchmark evaluating agentic long-context and multi-turn tasks, Ironwood TPU led across most of the Pareto frontier. Additionally, Inferact integrated a Pallas megakernel for the 2.8-trillion-parameter Kimi K3 model that runs the entire decode step as a single kernel, serving over 700 tokens per second to an individual user on Ironwood TPU.

Enterprise infrastructure operators have already deployed the new software stack into production. Meta runs SGLang-TorchTPU in live production, developed through collaboration among Meta, Google, and the SGLang team. Zongwei Zhou, PyTorch-TPU lead at Meta, stated that the native stack integrates into existing AI infrastructure while delivering necessary performance, reliability, and flexibility. Banghua Zhu, Chief Technology Officer and co-founder of RadixArk, reported that TorchTPU accelerated development velocity across silicon targets. Simon Mo, Chief Executive Officer and co-founder of Inferact, noted that bridging PyTorch capabilities into vLLM allows developers to run production workloads with high throughput and memory efficiency.

How to try Google New vLLM and SGLang Inference Engines

Developers can inspect and deploy the new backends via public GitHub repositories. Google published the core runtime under google-pytorch as torch_tpu, while framework implementations reside in upstream project organizations as vllm-project/vllm-torchtpu and sgl-project/sglang-torchtpu. These repositories supply installation guidelines, container definitions, and sample deployment scripts. Because both engines preserve upstream interfaces, client applications and OpenAI-compatible API setups connect with minimal modifications. In vLLM-TorchTPU, developers interact with TPU Sync through the standard key-value connector interface, whereas SGLang-TorchTPU uses the library as its disaggregation transfer backend and as the input-output engine powering HiCache.

Engineers evaluating hardware performance or contributing code can apply for compute through the community contribution TPU capacity pool. This program grants free accelerator capacity to developers adding kernel optimizations, testing new configurations, or introducing model definitions. After provisioning TPU hardware, engineers can execute PyTorch code eagerly for interactive debugging or compile with torch.compile for deployment. Teams operating distributed inference can configure disaggregated prefill and decode partitions across separate TPU hosts, tuning tiered offloading parameters with Google Cloud Managed Lustre to benchmark multi-turn conversations and agentic context handling under live production conditions.

What to watch after Google New vLLM and SGLang Inference Engines

The technical roadmap prioritizes expanding native TPU support across additional open model families while refining serving efficiency. Inferact and Google plan follow-up submissions to the SemiAnalysis InferenceX and AgentX benchmarks to demonstrate efficiency gains from larger-scale prefill and decode disaggregation on the 2.8-trillion-parameter Kimi K3 model. Development teams are also preparing Day 0 support for upcoming open models, building on current coverage for Qwen, DeepSeek, GLM, Kimi, GPT-OSS, and Gemma4. Beyond model support, engineering teams are inviting community collaboration through GitHub issues to enhance Pallas kernels and optimize disaggregation for complex agent loops.

Hardware diversification and software co-design will shape enterprise AI deployment strategies as organizations seek scalable alternatives to standard GPU instances. With Meta running SGLang-TorchTPU in production and preview customers operating vLLM-TorchTPU at scale, open-source serving runtimes are establishing competitive footprints on non-GPU silicon. The continued maintenance of vLLM-Jax and SGLang-JAX ensures stability for existing deployments while TorchTPU matures. Future indicators to monitor include community contributions through the free TPU pool, broader disaggregation adoption across cloud environments, and updated public benchmark results showing whether TPU-based PyTorch stacks maintain their cost advantage on long-context workloads.