Google's TPUs now serve AI in production at Meta through a new PyTorch-native SGLang engine
Meta is running production AI inference on Google's tensor processing units through SGLang-TorchTPU, a PyTorch-native version of the SGLang inference engine built by Google, Meta and SGLang-founded startup RadixArk and released at PyTorch Conference alongside Google's newly open-sourced TorchTPU backend. "SGLang-TorchTPU is running in production at Meta today," said Meta PyTorch-TPU lead Zongwei Zhou; previously the engines needed Google's JAX framework to run on TPUs. Google also released a TorchTPU version of vLLM, which partner Inferact says tops 700 tokens per second for a single user on Moonshot's Kimi K3 using 16 Ironwood TPUs, versus 452 on 16 Nvidia GB200s.
Source: Tom's Hardware