Google has joined Amazon in attacking Nvidia’s stronghold with AI app

Google on Tuesday made TorchTPU available as an open-source project, its long-awaited version of PyTorch. TechCrunch's report by venture editor Julie Bort says Google is joining Amazon in an attempt to break Nvidia's stranglehold, because most AI apps are written by default for Nvidia GPUs. TorchTPU runs those apps on Google's Tensor Processing Units, or TPUs, and moves an existing PyTorch app to Google Cloud with a configuration change and no rewrite.

This piece is for AI app developers and startups who build, train, and run models in PyTorch, a toolkit used by 85% of AI apps by some accounts. It covers TorchTPU, the TPU agent suite, and Google's updated LiteRT tool for phones, laptops, and embedded devices, then sets them next to Amazon's TorchNeuron for the Trainium chip and AWS. Hugging Face, Inferact's vLLM, and RadixArk's SGLang are the partner tools named with TorchTPU.

Google has joined Amazon in attacking: what actually changed

Google announced TorchTPU on Tuesday as an open-source project, its version of the popular PyTorch toolkit. Existing apps can run on Google's TPUs and move to Google Cloud with a simple configuration change and no rewrite. Google also launched the TPU agent suite, a dedicated tool that helps developers move apps onto Google's chips and cloud and fix errors along the way. It updated LiteRT, the popular tool for phones, laptops, and embedded devices, so that tool works better with PyTorch AI apps.

On Monday Amazon released TorchNeuron, its own version of PyTorch. TorchNeuron ports AI apps to Amazon's homegrown chip, Trainium, and to AWS with no rewriting. Amazon said it is submitting code to the main PyTorch project that decouples various features from Nvidia's chips, so PyTorch is more open to any hardware by default. Google, a day later, says it worked directly with Meta, Hugging Face, Inferact, maker of vLLM, RadixArk, maker of SGLang, and others so TorchTPU's features and code libraries match original PyTorch.

Google has joined Amazon in attacking: how it works

Meta, then Facebook, developed PyTorch in 2016, long before LLMs were expected to change the world. Nvidia GPUs were already standard for AI research, so many PyTorch features were built for those chips. In 2022 Meta turned PyTorch over to the Linux Foundation, a nonprofit that manages many important open source projects. Google's version, TorchTPU, lets AI apps run on TPUs. A developer moves an existing app with a simple configuration change and does not rewrite it to reach TPUs or Google Cloud.

The TPU agent suite helps developers move apps and fix errors on the way to Google's chips and cloud. Google says it worked with Meta, Hugging Face, Inferact's vLLM, and RadixArk's SGLang so TorchTPU's features and code libraries behave as they do in original PyTorch. LiteRT, which runs AI on phones, laptops, and embedded devices, was updated to work better with PyTorch AI apps. TorchNeuron ports apps to Trainium and AWS with no rewriting, and Amazon is submitting decoupling code to the main PyTorch project.

Google has joined Amazon in attacking: why it matters now

Most AI apps are written by default for Nvidia GPUs. PyTorch, used to build, train, and run models, is by some accounts used by 85% of AI apps. Meta built it in 2016 around Nvidia hardware that was already the research standard, and many features were built for those chips. On Monday Amazon released TorchNeuron for Trainium and AWS. On Tuesday Google released open-source TorchTPU for TPUs and Google Cloud. Both keep the existing PyTorch app and do not require a rewrite.

Google may be one-upping Amazon. It worked with Meta, Hugging Face, Inferact (vLLM), RadixArk (SGLang), and others so those tools keep working as they do in original PyTorch. The TPU agent suite helps move apps onto Google's chips and cloud and fix errors along the way. Popular LiteRT, which runs on phones, laptops, and embedded devices, was updated to work better with PyTorch AI apps. Amazon's further step is code for the main PyTorch project, decoupling features from Nvidia's chips for any hardware by default. Developers will not automatically ditch Nvidia GPUs en masse. Both companies are working to end that advantage.

Google has joined Amazon in attacking: who is affected

AI app developers and startups that build, train, and run models in PyTorch are who both releases address. TorchTPU takes them to Google TPUs and Google Cloud with the same app, a configuration change, and no rewrite. Hugging Face, Inferact's vLLM, RadixArk's SGLang, Meta, and others are the tools Google says TorchTPU's features and code libraries should match, as original PyTorch does. LiteRT's audience is developers running PyTorch AI apps on phones, laptops, and embedded devices. Google updated that popular tool to work better with those apps.

Developers heading to Amazon have TorchNeuron, released Monday, which ports AI apps to Trainium and AWS with no rewriting. Developers who stay on the main PyTorch project are the audience for code Amazon said it is submitting. That code decouples features from Nvidia's chips so PyTorch is more open to any hardware by default. Nvidia's GPUs are still what most AI apps are written for, a default from the 2016 design. Meta gave PyTorch to the Linux Foundation in 2022.

Google has joined Amazon in attacking: what to watch

Google says a simple configuration change moves an existing PyTorch app to TPUs and Google Cloud with no rewriting. The TPU agent suite is supposed to help with that move and fix errors along the way. The libraries to watch are Meta, Hugging Face, Inferact, maker of vLLM, RadixArk, maker of SGLang, and others. Google says TorchTPU's features and code libraries work with those tools as original PyTorch does. The LiteRT update is the one to watch on phones, laptops, and embedded devices.

Amazon's code for the main PyTorch project is the other piece to watch. The company said it decouples various features from Nvidia's chips and makes PyTorch more open to any hardware by default, past TorchNeuron's no-rewrite port to Trainium and AWS. The Linux Foundation has managed that project since Meta turned it over in 2022. Tuesday's TorchTPU release and Monday's TorchNeuron release do not mean developers will automatically leave Nvidia GPUs en masse just because other chips and clouds are easier to reach.