Beyond GPUs: How Inferentia2, TPUs, Groq LPUs, and Tenstorrent Actually Differ for LLM Serving

• Four chip families, four tradeoffs. GPUs trade peak efficiency for flexibility. Fixed-function accelerators (Inferentia2, TPU) trade flexibility for lower cost per token on one fixed model. Groq’s LPU trades capacity for predictable latency. Tenstorrent is an earlier-stage dataflow option.
• Most teams need GPU flexibility. Swap models, change precision, and adjust batching without recompiling. Specialized chips make you pay again for each of those changes.
• Groq wins on predictable latency. Its fixed schedule keeps p50 and p99 close together, which is important for real-time, single-request workloads.
• Inferentia2 and TPU win on cost per token for one fixed model at high volume. The catch: you recompile on every model, precision, or shape change.
• Memory, not FLOPS, sets the serving ceiling. Capacity limits model size. Bandwidth limits speed. Two commonly misquoted numbers are corrected below.

If you are new to inference hardware, you can refer to the below tables first. It defines the terms used in the article.

Core inference concepts

Latency and reliability

Chip families and products

Compilation, software, and flexibility

Memory, math, and why generation feels “slow”

Model size, precision, and quantization

Scaling across multiple chips

Benchmarks and optimization techniques

This article compares chip architectures using published vendor specs and public architecture papers. Every hardware number links to its source. These are not DigitalOcean benchmark results. When a claim is about how a design works, not something we measured in production, the text says so.

One recent event is worth noting. On December 24, 2025, NVIDIA licensed Groq’s inference technology (reported at about $20 billion), hired Groq’s founder and several senior staff, and kept the deal non-exclusive. Groq still runs GroqCloud on its own under new leadership. Source: Groq newsroom. The deal shows that predictable, on-chip-memory inference matters even to the largest GPU maker. It does not change the tradeoffs below.

Start with four chip categories. Each one optimizes for something different.

Four chip families, four different tradeoffs. The GPU is the flexible baseline everything else is measured against.

General-purpose GPU. NVIDIA H100 and H200, AMD MI300X and MI325X. Load a model and serve it with no compile step. Swap models, change precision, and adjust batching as traffic changes. This is the flexible baseline. Every other option trades some of that flexibility for a specific gain.

Fixed-function accelerator. AWS Inferentia2 and Google Cloud TPU v5e in serving mode. Built to run one fixed model cheaply at high volume. Both require a compile step that turns your model into chip-specific code. Inferentia2 uses the AWS Neuron SDK. TPU uses XLA. Performance depends heavily on the input shapes you compiled for.

Deterministic low-latency chip. The Groq LPU (Language Processing Unit). It stores model weights in fast on-chip memory and schedules every operation ahead of time, so each request takes the same number of clock cycles. You give up capacity and flexibility to get predictable speed. Groq documented the design in two peer-reviewed papers at ISCA: the original Tensor Streaming Processor in 2020, and multi-chip scaling in 2022. Source: Think Fast: A Tensor Streaming Processor for Accelerating Deep Learning Workloads, ISCA 2020.

Dataflow chip. Tenstorrent. Its serving software is less mature than the other three, and reliable production specs are harder to find. This article covers it at the category level only.

Pay close attention here. p99 latency is a day-to-day operations problem, and most articles skip the hardware reasons behind it.

For user-facing apps, median latency tells you little. What breaks an SLA is the slow tail, the p99 and p999. On a GPU, that tail comes from two design choices. First, the GPU decides what to run at runtime, so scheduling varies. Second, concurrent requests share the same memory channels. These are built into the architecture. They are not signs of a bad deployment.

On a GPU, slow requests sit far from the median. On a Groq LPU, latency stays tight by design.

Groq reduces that variation by deciding everything ahead of time. Its compiler maps the full model to the hardware, down to the clock cycle, including communication between chips. The same request with the same input shape takes the same number of cycles every time, so p50 and p99 sit close together. Groq’s ISCA 2020 paper describes this as a core design goal: remove reactive hardware like arbiters and caches so timing stays predictable. Source: Groq ISCA 2020 paper. That is Groq’s design claim from a peer-reviewed paper, not a production test DigitalOcean ran.

The tradeoff is rigidity. GPUs use continuous batching: the server groups a changing number of live requests into one pass. A fixed schedule cannot adapt as freely. Groq shines for single-stream, real-time serving. Its advantage shrinks when batch sizes swing widely.

Inferentia2 and TPU v5e behave differently because of compilation. Both compile for specific input shapes. A request that does not match gets padded to the nearest compiled shape. That padding shows up as p99 latency spikes even when typical requests look fine. Neuron and XLA let you choose which shapes to compile, so you can tune this, but it is a real tail-latency source that GPUs avoid.

GPUs win on operational flexibility. That is a real engineering advantage, not marketing.

Inferentia2 needs the model compiled to AWS Neuron format through the Neuron SDK. TPU v5e needs XLA compilation. Compilation is not a one-time step. You repeat it when the model changes, when precision changes, or when input size or batch size changes. If you ship a new model every week, or run several versions at once, that cost never goes away.

Same model change, two paths. A GPU hot-swaps weights. A fixed-function accelerator recompiles every time something changes.

We do not have verified compile times for representative model sizes, and we will not invent them. For the published version, pull current figures from the AWS Neuron documentation and Google’s XLA and TPU docs, run a compile on a real model, and report what you measure with the software version noted. Until then, the point stands: compilation is a recurring cost that grows with how often your setup changes.

The second constraint is operator coverage. On fixed-function hardware, you can only run what the compiler supports. If your model uses an attention variant or activation the compiler does not implement, it either fails to compile or runs on a slow fallback path, and you lose the speed you paid for. Check the AWS Neuron supported-operator list and the OpenXLA operation list against your model before you commit. This is the gap between a vendor’s “supported models” list and the models you actually serve at full speed.

The third constraint is precision. Precision options matter because quantization is how teams fit larger models into less memory.

Sources: AWS Inferentia for the cFP8, FP16, BF16, and INT8 list. SemiAnalysis for the TPU v5e BF16 and INT8 figures. Groq for TruePoint. Verify the INT4 and quantization rows against current SDK docs before publishing, because framework support changes between releases.

Choosing a GPU is a deliberate tradeoff. You give up Groq’s tight p99 latency and Inferentia2’s lowest cost per token on one fixed model at full utilization. In return, you swap models without recompiling, use whatever quantization your serving engine supports, and run continuous batching to keep hardware busy under changing load.

When an LLM generates tokens one at a time, speed is limited by how fast the chip reads model weights from memory, not by how fast it can do math. The chip reads the full model for every token. So memory bandwidth, not peak FLOPS, sets the serving ceiling. Each token also reads per-request state from memory (the KV cache), which adds more traffic. Google researchers formalized this in a peer-reviewed paper: generation is dominated by loading weights and cache from memory, while prompt processing is compute-heavy. The two phases hit different bottlenecks on the same chip. Source: Pope et al., Efficiently Scaling Transformer Inference, MLSys 2023.

Computer architects call the ratio of math to memory traffic arithmetic intensity. It comes from the roofline performance model:

Arithmetic intensity = Total FLOPs performed ÷ Total bytes moved from memory

Low arithmetic intensity means memory-bound: the chip waits on data. High arithmetic intensity means compute-bound: the math units stay busy. Prompt processing is compute-heavy. Token generation is memory-heavy. That is why the two phases feel so different on the same chip. Source: Williams, Waterman, and Patterson, Roofline: An Insightful Visual Performance Model for Multicore Architectures, Communications of the ACM, 2009.

Processing a prompt keeps the math units busy. Generating each new token mostly waits on memory. That is why memory bandwidth, not FLOPS, decides how fast a chip serves.

The H200 is the clearest proof. It uses the same compute die(dies per wafer) as the H100. The only change is more and faster memory, and that alone makes it much faster at inference. Source: NVIDIA H200.

Capacity decides the largest model that fits. Bandwidth decides how fast it serves. Groq’s tiny bar shows the SRAM tradeoff.

Two numbers in that chart get misquoted constantly.

Inferentia2 bandwidth. AWS lists 9.8 TB/s for its largest Inf2 server. That is the total across all 12 chips, not one chip. Per chip, it is about 820 GB/s. Quoting 9.8 TB/s as a single-chip number overstates bandwidth by more than ten times. Source: Amazon EC2 Inf2 instances.

Groq on-chip memory. Groq’s ISCA 2022 paper states that each chip contributes 220 MiB to a shared on-chip memory pool. Marketing rounds this to “about 230 MB,” but the paper is the precise source. There is no HBM on the chip. That single design choice explains most of how Groq behaves. Source: A Software-defined Tensor Streaming Multiprocessor for Large-scale Machine Learning, ISCA 2022.

Turning memory bandwidth into an actual millisecond number

The bandwidth claim is more useful as a number you can check against your workload:

Time per token, floor = Model weight size in bytes ÷ Memory bandwidth in bytes per second

This is a theoretical floor, not a measured result. It assumes the chip spends the whole step reading weights, with no contention, no KV cache reads, and no batching overlap, so real production latency runs higher. It still tells you the fastest a chip could possibly go. Use it as a sanity check on vendor latency claims.

Each bar divides a model’s weight size by that chip’s bandwidth. H200’s only change from H100 is memory, and the math shows the effect directly: about 41.8 ms per token on H100 versus about 29.2 ms on H200, for the same 70B model at FP16.

Worked out: a 70B model at FP16 needs 140 GB of weights. On an H100 at roughly 3.35 TB/s, that is 140 ÷ 3,350 ≈ 0.0418 seconds, about 41.8 ms per token at the floor. On an H200 at roughly 4.8 TB/s, the same 140 GB takes 140 ÷ 4,800 ≈ 0.0292 seconds, about 29.2 ms. The compute die is identical. Only memory changed, and that alone explains the gap. Run this formula for any chip and model size, as long as the model fits in that chip’s memory.

How big a model fits on one chip

You can estimate the largest model a single chip can hold from its memory size. The method works for any chip:

Weight size in bytes = Parameter count × Bytes per parameter

• FP16 or BF16: 2 bytes per parameter
• INT8: 1 byte per parameter
• INT4: about 0.5 bytes per parameter

So a 70B model needs about 140 GB at FP16, about 70 GB at INT8, and about 35 GB at INT4.

Apply that, weights only, before the working memory each request needs:

• H200, 141 GB: fits a 70B model at FP16 at the outer edge, with almost no room to spare.
• H100, 80 GB: fits 30B to 40B models at FP16, or a 70B model once you quantize it.
• Inferentia2, 32 GB per chip: about a 16B model at FP16.
• TPU v5e, 16 GB per chip: about an 8B model at FP16.
• Groq, 220 MiB per chip: a small slice of one model. A 70B model spreads across hundreds of chips.

These are weights-only ceilings. Each request also needs working memory that grows with batch size and context length, so real limits sit lower. Treat these as upper bounds, not guarantees. No authoritative public source states exactly how many Groq chips a given model needs in production. Groq describes racks of chips working together without always giving a per-model count, treat any “N chips per model” figure elsewhere as an estimate.

You can estimate a lower bound yourself:

Chips needed, floor ≈ (Parameter count × Bytes per parameter) ÷ 220 MiB per chip

A 70B model at INT8 needs about 70 GB of weights. Divide by 220 MiB per chip and you get roughly 303 chips, just to hold the weights. At FP16 (about 140 GB), that rises to roughly 607 chips. Both numbers ignore KV cache, activation memory, and networking overhead, so treat them as a floor, not an order spec.

A GPU holds a large model on one or two chips. Groq’s small per-chip memory means the same model spreads across a rack of chips, each holding a thin slice. The chip count shown is a capacity-only floor, worked out from the formula, not a published deployment spec.

The Groq tradeoff in one line: on-chip memory is very fast, but there is very little of it, so large models need many chips and a lot of rack space.

Scaling across chips

When one chip cannot hold a model, you spread it across several. Then the links between chips become the bottleneck.

Each architecture connects chips differently. GPUs use a full-speed mesh. TPUs route through a torus. Groq schedules chip-to-chip traffic at compile time.

• NVIDIA GPUs use tensor parallelism over NVLink. Fourth-generation NVLink on H100 and H200 provides 900 GB/s total bandwidth per GPU across 18 links. NVSwitch lets every GPU in a server reach every other GPU at full speed. Source: NVIDIA Hopper Architecture In-Depth.
• Inferentia2 scales over NeuronLink at 192 GB/s between chips inside an Inf2 instance. Source: Amazon EC2 Inf2 instances.
• TPU v5e scales over a 2D torus interconnect, up to 256 chips per pod. Source: SemiAnalysis.
• Groq uses a chip-to-chip network scheduled at compile time, the same approach as inside a single chip, so cross-chip communication stays deterministic. Groq’s ISCA 2022 paper describes this design, but we could not find a published bandwidth figure in GB/s. Treat any Groq interconnect bandwidth number you see elsewhere with caution until Groq publishes one.

The key question: does throughput scale roughly linearly as you add chips, and how much does communication overhead cost? Use published interconnect bandwidth where vendors disclose it, and note where they do not.

Everything above comes from vendor spec sheets. Specs show theoretical limits, not what a chip does under real traffic. This section gathers independently measured numbers we could verify. One caveat: these numbers come from different tests on different models, so you cannot rank all four chips from this data alone. Treat them as separate data points, not a leaderboard.

Real, independently measured numbers exist for some of these chips, but they were measured on different models with different methods. Read this as several separate data points, not one chart.

Groq, on Meta’s open Llama models, tested by the independent benchmarking firm Artificial Analysis: 241 tokens per second on Llama 2 70B in February 2024, 284 tokens per second on Llama 3 70B a few months later, and 276 tokens per second on Llama 3.3 70B by December 2024, with a separate speculative-decoding endpoint reaching 1,665 tokens per second on the same model. Artificial Analysis’s live tracking page later showed Groq tied for the fastest output speed on Llama 3.3 70B among 17 providers at 296.7 tokens per second. Sources: Groq newsroom, citing Artificial Analysis and Artificial Analysis, Llama 3.3 70B provider comparison. These are real independent measurements, though Groq’s own newsroom is the source for the historical figures, so treat the trend as directionally reliable and verify the current number directly on Artificial Analysis before citing it.

NVIDIA GPUs, on the MLPerf Inference benchmark, which is run and audited by MLCommons, a nonprofit industry consortium: on the Llama 2 70B benchmark, H200 delivered up to 1.5 times the throughput of H100, and the newer Blackwell architecture delivered up to 4 times H100’s throughput, in NVIDIA’s own MLPerf Inference v4.1 submission. Source: NVIDIA Blackwell Platform Sets New LLM Inference Records in MLPerf Inference v4.1. In a later round, CoreWeave’s own H200 submission to MLPerf Inference v5.0 reported over 33,000 tokens per second on the Llama 2 70B benchmark, a system-level number that depends heavily on how many GPUs were in that configuration. Source: HPCwire coverage of MLPerf v5.0.

Google TPU v5e has one directly comparable, audited result: in its MLPerf Inference 3.1 submission, four TPU v5e chips ran the GPT-J 6B benchmark at 2.7 times the performance per dollar of TPU v4, Google’s prior generation. Source: Google Cloud, cost-efficient AI inference with Cloud TPU v5e. This is a small-model benchmark, not a 70B-class comparison, and I could not find a TPU v5e MLPerf submission on a large Llama model to compare directly against the GPU numbers above.

AWS Inferentia2 is the gap in this section. We searched for a public MLPerf Inference submission using Inferentia2 and did not find one. AWS publishes its own claims, such as 40 percent better price-performance than comparable EC2 instances, but we could not verify an independently audited LLM throughput number the way Artificial Analysis and MLPerf provide for the other three chips. Check the current MLCommons results page directly; submissions change every round.

Bookmark this section. It is a decision guide, not a sales pitch.

Start at the top. Most paths lead to a GPU, because most workloads change more often than they stay frozen.

In short: one frozen model at high volume with no strict latency SLA → Inferentia2 or TPU may cost less per token. Strict latency SLA → Groq may fit best. Models that change often, many models, or unusual architectures → GPU.

A GPU is the flexible default, but not always the right answer. Specialized chips win in these cases:

• A hard, real-time p99 SLA. Voice agents and other interactive tools that need sub-second responses on single requests are where Groq’s predictable latency pays off. There is no universal millisecond threshold, test against what your users actually tolerate. A GPU’s latency tail can break a strict SLA under load in a way a fixed schedule does not.
• One model, frozen for a long time, at high volume. When the model rarely changes, you pay the compile cost once and spread it across billions of tokens. Cost per token can beat a GPU.
• A model that matches the chip’s compiled shapes. Fixed-function chips work best when your input sizes match what you compiled for, so padding stays small.
• Cost-sensitive batch inference with no latency pressure. Offline or queued jobs care about cost per token, not tail latency, fixed-function chips fit well.

If any of these describe your workload, a specialized chip is the rational pick. If your setup changes often, GPU flexibility is worth more than peak efficiency on a narrow benchmark.

1. Is a GPU always the best choice for LLM inference?

No. A GPU is the best default for teams that change models often, test quantization, or serve several models, because it needs no recompile. For one frozen model at high volume, Inferentia2 or TPU can cost less per token. For a strict real-time latency SLA, Groq can hold a tighter tail.

2. Why does memory matter more than FLOPS for inference?

Token generation reads the entire set of model weights from memory for every token, then does relatively little math. So serving speed is limited by memory bandwidth, and the largest model you can hold is limited by memory capacity. The H200 proves the point. It shares the H100’s compute die and is faster only because it has more and faster memory.

3. Why does Groq need so many chips?

Each Groq chip holds 220 MiB of on-chip SRAM, per Groq’s own peer-reviewed architecture paper, and no HBM. A large model does not fit on one chip, so it spreads across many chips, each holding a thin slice of the weights. That is the cost of the on-chip memory that makes Groq fast and predictable.

4. What is the real cost of compiling for Inferentia2 or TPU?

The cost is not the first compile. You repeat the compile every time the model, the precision, or the input shape changes. A team iterating weekly pays it again and again. A team running one frozen model pays it once.

5. Did NVIDIA buy Groq?

Not as a full acquisition. In December 2025 NVIDIA took a non-exclusive license of Groq’s technology and hired its founder and several staff. Groq still operates GroqCloud independently. The deal signals that deterministic inference matters, but the architectural tradeoffs in this article still hold.

Specialized inference chips are real, and each one wins at something specific. Groq wins on predictable latency. Inferentia2 and TPU win on cost per token for one fixed model at scale. GPUs win for teams that need to change their setup, which is most teams.

GPU serving stays the default not because no one built a faster chip. It stays the default because specialized silicon comes with real operational costs: recompiling on every change, compiler gaps for unusual models, and rigid scheduling that does not adapt to shifting load. When those constraints do not apply, specialized chips win on a specific dimension, and this article maps which dimension and when.

If GPU serving fits your workload and you need to model cost next, check out our article on dedicated GPU inference economics. DigitalOcean’s resources on GPU Inference, What is AI Inference, choosing the right GPU Droplet, and LLM inference benchmarking go deeper on running inference at scale.

Vendor hardware specifications

• NVIDIA H100
• NVIDIA H200
• NVIDIA Hopper Architecture In-Depth
• AMD Instinct MI300X Accelerators
• AMD Instinct MI325X Accelerators
• Amazon EC2 Inf2 instances
• AWS Inferentia
• Google Cloud TPU v5e documentation
• OpenXLA / XLA compiler
• Groq LPU architecture explained
• Tenstorrent

Peer-reviewed papers

• Samuel Williams, Andrew Waterman, and David Patterson, “Roofline: An Insightful Visual Performance Model for Multicore Architectures,” Communications of the ACM, Vol 52, No 4, April 2009.
• Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, et al., “Efficiently Scaling Transformer Inference,” MLSys 2023.
• Dennis Abts et al., “Think Fast: A Tensor Streaming Processor (TSP) for Accelerating Deep Learning Workloads,” ISCA 2020.
• Dennis Abts et al., “A Software-defined Tensor Streaming Multiprocessor for Large-scale Machine Learning,” ISCA 2022.

Independent benchmarks

• Artificial Analysis, Llama 3.3 70B provider comparison
• Groq newsroom, first independent LLM benchmark via Artificial Analysis
• NVIDIA Blackwell Platform Sets New LLM Inference Records in MLPerf Inference v4.1
• HPCwire, MLPerf v5.0 results coverage
• Google Cloud, performance per dollar of GPUs and TPUs for AI inference
• MLCommons, MLPerf Inference results

Other sources

• Groq and NVIDIA licensing agreement
• SemiAnalysis: TPU v5e benchmark