Tag: TPU

  • Google Splits Its TPU Chip in Two. Here’s Why That Decision Matters for Enterprise Buyers.

    Google Splits Its TPU Chip in Two. Here’s Why That Decision Matters for Enterprise Buyers.

    The AI chip acronym soup of CPUs, GPUs, TPUs, etc., shows how the computing landscape continued to expand and change over the past decade. At Google Cloud Next, the company released two distinct TPUs (Tensor Processing Units) instead of one — TPU-8t, built for training, and TPU-8i, built for inference and the emerging demands of agentic workloads. The launch highlights an architectural decision that reflects how AI workloads are diverging, with real implications for how enterprise buyers should think about AI infrastructure strategy.

    What Google Actually Announced

    During a press and analyst session at Google Cloud Next, Amin Vahdat, Google’s SVP and Chief Technologist for AI Infrastructure, introduced the eighth-generation TPUs — and emphasized the plural intentionally. Vahdat said the two chips were designed from the ground up separately.

    TPU-8t is the training workhorse. Compared to last year’s Ironwood generation, it delivers roughly three times the floating-point compute per pod, twice the network bandwidth per chip, and four times the bandwidth at scale-out — all with approximately the same pod size of 9,600 chips, but with denser, faster interconnects.

    TPU-8i is the inference and agent engine. It quadruples the pod size to 1,152 chips, delivers 10x the FP8 compute, 7x larger HBM memory capacity, and offers bidirectional scale-out bandwidth. The design priority is latency, not just throughput — a meaningful distinction as enterprises move from batch processing toward real-time agentic workloads.

    Vahdat put the pace of progress plainly: “2x, 4x, 8x, 10x all in one year — the rate of progress, the rate of advancement is just stunning.”

    That’s impressive on paper. The more important question for enterprise buyers is what it means for how they plan and procure AI infrastructure.

    The Specialization Signal

    The two-chip decision acknowledges that training and inference have different physics.

    Training is throughput-bound, which means you’re moving enormous amounts of data through interconnected chips in a coordinated, largely predictable batch process. Inference, especially for the new upcoming wave of agentic systems, is latency-bound.  For this use case, chips need to respond in near-real time as agents plan, act, evaluate, and route across multiple tools and workflows.

    To address the latency problem directly, Google and DeepMind collaborated on a new network “boardfly” topology for TPU-8i, designed to reduce the number of hops between any two chips, significantly cutting chip-to-chip latency. As Vahdat described it: “Our default way of connecting them didn’t support latency. It supported bandwidth. What you really care about in the age of agents is latency — the minimum time it takes to get the data.”

    This mirrors a trend Jensen Huang surfaced at NVIDIA, where chip-to-chip connectivity is increasingly central to total system performance, not just an afterthought to compute specs. The implication: network topology is now a first-class variable in AI infrastructure design, not just chip count or memory.

    Vahdat was direct about the broader trajectory: “The age of specialization is going to continue.” His prediction for the industry — not just Google — is that workloads will continue diverging, and two chips may eventually become more. General-purpose improvements, he noted, are now yielding roughly 5% annual performance gains normalized to cost. Specialization is how you get past that ceiling.

    What This Means for Enterprise Buyers

    Enterprise buyers don’t purchase TPUs. They consume AI services through public cloud, SaaS platforms running on cloud infrastructure, and increasingly through hybrid architectures spanning on-premises and cloud. There are at least three reasons why a chip announcement matters.

    1. AI infrastructure costs are becoming a material business decision. Google is running AI inference on TPUs across Search, YouTube, Gmail, and its enterprise Gemini services. The efficiency of that infrastructure directly affects the cost structure of AI-powered services. When Google cuts inference costs through better hardware, the economics of running AI at scale improve for Google and for its cloud customers. Citadel Securities, the securities trading firm, was cited as a TPU customer that reduced costs 30% and achieved two to four times efficiency improvement on trading systems. Specialized hardware scales well beyond its original design targets.
    2. Inference is where AI delivers the most value to most enterprise buyers. For several years, we’ve been discussing the shift from large-scale frontier model training and enterprise AI fine-tuning towards inferencing. It’s finally here, and we have multiple ways to improve inference, including new TPUs designed for inference.  As Vahdat noted, using a historical parallel to web search: the heavy lifting happens in training, but the value is created in serving. “Serving is where the value is created for Gemini enterprise and search, and ads and YouTube.” Enterprise AI budgets and infrastructure roadmaps need to weigh inference infrastructure proportionally to where value is actually produced.
    3. Reliability at scale is still an unsolved problem — and it matters. Vahdat was candid about a challenge the industry rarely advertises: at the scale of tens of thousands of chips working in coordination, at least one chip will fail several times per day. If human intervention is required to detect and recover from failures, the minimum response time is 30 minutes — enough to halt progress entirely. Google’s approach delivers over 97% of good computational throughput, enabling failures to be automatically detected and remediated. Still, Google Cloud said enterprises aren’t interested in any failure. For enterprises evaluating AI infrastructure providers, reliability and observability at scale are now table-stakes questions, not nice-to-haves.

    The Agentic Infrastructure Shift Is Already Here

    A surprising forward-looking element of Vahdat’s remarks was a prediction about CPUs. As agentic systems grow, general-purpose compute will make a comeback — not to replace specialized chips, but to orchestrate them. Agents require sandboxed environments, virtual machines, code execution, and dynamic routing across inference calls. He stated that it’s CPU work.

    Enterprise infrastructure planners should take note: agentic AI isn’t just an inference problem. It’s a systems design problem that spans specialized accelerators, general-purpose compute, network topology, and increasingly, identity and governance layers sitting above the hardware. The companies Google cited as running on TPUs today — from its own consumer services to financial services firms — are already thinking holistically about infrastructure.

    The infrastructure decisions enterprises make now will determine how quickly and cost-effectively they can deploy agentic systems at scale. Building on platforms engineered for latency, reliability, and specialization is a different starting point than building on platforms that aren’t.

    Google Cloud’s eighth-generation TPUs are a signal that the advancement of AI infrastructure is far from over.  

    This article was originally published on Forbes.

  • Google Cloud’s Ironwood TPU Forges Better Enterprise AI

    Artificial intelligence infrastructure has emerged as the critical battleground for cloud computing dominance. At this year’s Google Cloud Next conference, the company demonstrated its intensified commitment to AI infrastructure, unveiling strategic investments, such as the Ironwood Tensor Processing Units (TPUs), designed to transform enterprise AI deployment across industries.


    “We’re investing in the full stack of AI innovation,” stated Sundar Pichai, CEO of Google and Alphabet, who outlined plans to allocate $75 billion in capital expenditure toward this vision. This substantial commitment reflects the scale of investment required to maintain competitive positioning in the rapidly evolving AI infrastructure market. Innovating in AI requires courage and deep pockets.

    Google Cloud articulated a full stack strategy focused on developing AI-optimized infrastructure spanning three integrated layers: purpose-built hardware, foundation models, and tooling for building and orchestrating multi-agent systems. During the keynote presentation, Google Cloud introduced the Ironwood TPU its seventh-generation Tensor Processing Units (TPUs), representing a significant advancement in AI computational architecture.

    Optimizing Infrastructure for AI With TPUs

    Cloud Computing infrastructure started as a method of replacing and optimizing on-premise data centers. Today, cloud computing providers are adding specific infrastructure to support new computing requirements introduced with supporting AI. TPUs are specialized processors developed by Google specifically to accelerate AI and machine learning workloads—with particular optimization for deep learning operations. TPUs deliver superior performance-per-dollar compared to general-purpose GPUs or CPUs across numerous machine learning use cases, resulting in reduced infrastructure costs or increased computational capability within existing budget constraints.

    Ironwood TPUs represent a cornerstone component of Google Cloud’s AI Hypercomputer architecture, which integrates optimized hardware and software components for high-demand AI workloads. The AI Hypercomputer platform constitutes a supercomputing system that combines performance-optimized silicon, open software frameworks, machine learning libraries, and flexible consumption models designed to enhance efficiency throughout the AI lifecycle—from training and tuning to inference and serving.

    According to Google’s technical specifications, these specialized AI processors deliver computational performance that’s 3,600 times more powerful and 29 times more energy efficient than the original TPUs launched in 2013. Ironwood also demonstrates a 4-5x performance improvement across multiple operational functions compared to the previous version 6 Trillium TPU architecture.

    Ironwood implements advanced liquid cooling systems and proprietary high-bandwidth Inter-Chip Interconnect (ICI) technology to create scalable computational units called “pods” that integrate up to 9,216 chips. At maximum pod configuration, Ironwood delivers 24 times the computational capacity of El Capitan, currently ranked as the world’s largest supercomputer.


    To maximize this infrastructure’s utility, Google Cloud has developed Pathways, a machine learning runtime created by Google DeepMind that enables efficient distributed computing across multiple TPU chips. Pathways on Google Cloud simplifies scaling beyond individual Ironwood Pods, allowing for the orchestration of hundreds of thousands of Ironwood chips for next-generation AI computational requirements. Google uses Pathways internally to train advanced models such as Gemini and now extends these same distributed computation capabilities to Google Cloud customers.

    Marrying Business Impact With Economics

    While the industry has witnessed a proliferation of smaller, specialized AI models, significant AI chip innovation remains essential to deliver the performance requirements for supporting advanced reasoning and multimodal models.

    According to Amin Vahdat, VP/GM of ML, Systems & Cloud AI at Google Cloud, “Ironwood is designed to gracefully manage the complex computation and communication demands of ‘thinking models,’ which encompass Large Language Models (LLMs), Mixture of Experts (MoEs) and advanced reasoning tasks.” This architecture addresses the market requirement for modular, scalable systems that deliver improved performance and accuracy while optimizing both cost efficiency and energy utilization.


    For enterprises implementing large-scale AI initiatives, Google’s hardware advancements translate to quantifiable benefits across three dimensions:

    1. Economic Efficiency. Google’s specialized hardware substantially increases computational density per dollar, reducing the total cost of ownership for AI infrastructure. Organizations can deploy increasingly sophisticated AI models without corresponding linear increases in computing expenditures.
    2. Sustainability Metrics. As AI model complexity increases (across systems like Gemini, ChatGPT, and advanced image generators), the underlying computational infrastructure generates significantly more heat and power consumption. Liquid cooling technology, implemented in Ironwood, delivers substantially higher thermal efficiency compared to conventional air cooling, enabling chips to operate at higher frequencies without thermal throttling. This innovation addresses power consumption—a critical consideration for both cloud providers and enterprise buyers with sustainability commitments. The enhanced performance-per-watt metrics of these TPUs help organizations address environmental impact concerns while scaling their AI capabilities.
    3. Time-to-Market Acceleration. The exponential increase in processing capacity means that AI model training and inference workflows—previously requiring weeks or months of computation—can now be completed in days or hours. This compression of development timelines enables organizations to iterate more rapidly and operationalize AI solutions with significantly reduced deployment cycles.

    Why TPUs Matters to Enterprise Buyers

    Organizations are over the phase of interesting AI proof of concept trials that never make it to production-grade systems. 2025 is the year that organizations expect to deploy use cases with quantifiable business value while laying the foundation for what’s next. Google Cloud’s enhanced AI infrastructure enables practical enterprise applications today while supporting previously constrained by computational economics or performance limitations. Consider the impact of AI today and tomorrow in:

    • Financial Services Analytics. During the Google Cloud Next keynote, Deutsche Bank shared how it uses technology from Google Cloud to power an AI-powered research agent named DB Lumina for faster data analysis. Many banking and investment firms are investigating how to use enhanced AI infrastructure to process market data streams, detect complex pattern anomalies in real-time, and enable more responsive trading strategies and comprehensive risk management frameworks.
    • Customer Experience Transformation. Retail and service organizations can implement sophisticated recommendation engines and multimodal conversational AI systems that process customer interactions with minimal latency while incorporating rich contextual understanding. For example, Verizon uses Google Cloud’s Customer Engagement Suite to enhance its customer service for over 115 million connections with AI-powered tools, like the Personal Research Assistant which accurately answers 95% of questions, helping agents provide faster, more accurate, and personalized support. The next era focuses on cracking the code for personalization, advanced marketing assets, and empathetic conversational AI.
    • Computational Medicine. Today, healthcare organizations use AI to improve patient experiences with data gathering and summarization features. For example, Seattle Children’s Hospital used Google Cloud’s generative AI to create Pathway Assistant, an AI-powered agent that improves clinicians’ access to complex information and the latest evidence-based best practices needed to treat patients. As we advance, healthcare institutions can leverage advances in AI infrastructure to accelerate the analysis of complex imaging datasets, genomic sequences, and patient records, potentially enhancing diagnostic accuracy and treatment protocol optimization.

    Get Comfortable With Change

    As competition intensifies among cloud infrastructure providers, Google’s substantial investment in AI represents a strategic assessment that enterprise computing will increasingly prioritize AI-driven workloads—and that organizations will select platforms offering the optimal combination of performance, cost efficiency, and energy sustainability.


    The only constant in the AI market will be change. Business leaders must be comfortable with continuously adapting strategies to leverage AI advancements. For CIOs and technology leaders developing their AI implementation roadmaps, Google Cloud’s hardware innovations, such as the Ironwood TPU, present technical and economic justifications to reevaluate their infrastructure strategy as AI becomes increasingly central to operational excellence and competitive differentiation.