Kai Tanaka

The frontier war is being fought with tiered pricing architectures, not model weights. Every major provider now runs a three-tier stack: a flagship that anchors perception, a mid-tier workhorse that handles 80% of production volume, and a routing layer that compresses cost on high-throughput paths.

Anthropic's March 2024 launch pattern is the template [5]: Opus sets the capability ceiling and justifies premium rates ($15 input / $75 output). Sonnet undercuts GPT-4 Turbo by 20% while matching 90% of capability. Haiku routes the long tail at $0.25 input — fifteen times cheaper than the flagship it launched alongside.

Then Sonnet 3.5 compressed the entire stack from above [8]: it beat Opus on coding and graduate-level reasoning while priced at the old Sonnet rate ($3 / $15). The premium tier became obsolete by its own successor at one-fifth the cost. That's not a product roadmap. That's deliberate margin destruction.

OpenAI's GPT-4o mini did the same thing from below [2]: $0.15 / $0.60 per million tokens — 60% cheaper than GPT-3.5 Turbo, which was already the volume play. The mini tier now handles routing, batching, and high-frequency calls that used to go to the previous generation's flagship.

The pricing tiers aren't segmenting customers by willingness to pay. They're segmenting workloads by latency tolerance and output token count. Synchronous short responses go to the flagship. Batch jobs with 24-hour SLA go to the discounted tier [11]. Multi-turn conversations with long context go to the cached middle tier [4].

The model is a credential that expires every six months. The pricing grid is the actual product.

The tier's methodology readings collapse into a single operational stance: the metric you choose determines which cost structure wins.

Benchmark saturation [1] and construct invalidity [4] mean frontier model differentiation no longer shows up in MMLU deltas — so the buyers who keep scoring on those benchmarks pay premium prices for undifferentiated capability. The teams that switch to cost-per-correct-output [24] as the denominator discover that $0.28/1M DeepSeek tokens at 79% task success beats $25/1M Claude Opus at 100% for most workloads.

Same pattern in infrastructure: teams measuring uptime [26] instead of functional availability provision for 99.99% SLA [25] and pay exponentially compounding redundancy costs [28] — while the workload they actually run degrades silently because the model swapped mid-request or the KV cache evicted under pressure [27]. The SLA was met; the answer was garbage.

TCO modeling collapses the same way. If you measure cost-per-GPU-hour, cloud wins below 40% utilization [5]. If you measure cost-per-million-tokens and include the power/cooling OpEx that exceeds CapEx over 36 months [6], on-prem crosses breakeven at 60%. The unit you denominate in — tokens, requests, GPU-hours — determines which infrastructure survives.

Kai's read: The analysts still score models on MMLU and datacenters on $/kW. The buyers who switched to $/correct-output and $/delivered-token are the ones compressing margin out of the layer above them.

Methodology is not neutral. It's the bid-ask spread made legible.

The hosted-API price is not the cost. The self-hosted breakeven is not the floor. The model layer operates in three regimes, each with different cost drivers:

API-hosted (sub-10M tokens/month) — [13] shows output tokens cost 10-15× input tokens because decode is generative; [27] confirms decode draws more power per token than prefill. At low volume, you pay the provider's margin plus their amortized idle capacity. The price compresses when competitors undercut, but the floor is set by their utilization rate, not your workload.

Self-hosted at threshold utilization (10M-1B tokens/month) — [16] identifies breakeven at 10-50M tokens monthly, but the crossover depends on GPU utilization. An H100 at 40% utilization has a different cost-per-token than the same H100 at 80%. [26] measured real AI workloads at 76% TDP; [30] measured MFU at 41-48% depending on scale. The realized cost sits between TDP and MFU — and it moves with batch size, sequence length, and parallelism strategy [29].

Compression-optimized deployment (>1B tokens/month) — [17, 18, 19] establish that quantization, pruning, and distillation compress cost after the training law optimizes loss. [12] showed a 70B Chinchilla model matches 280B Gopher quality; compress it to INT8 [18] and you've cut memory 4× again. At scale, the marginal cost is memory bandwidth [15], and every halving of model size doubles throughput on the same iron.

Most analyst notes price the model layer as if everyone sits in regime 1. The structural read is that volume migrates left-to-right across regimes as deployment matures, and each regime has a different cost denominator. By the time a model hits regime 3, the per-token cost is 10-50× lower than the API price that seeded adoption.

Watch: compression adoption rate in open-weight models [22]. That's the leading indicator for when regime-3 economics start undercutting regime-1 list prices.

AI coverage at Palanor is the discipline of pricing the model layer at the unit-economic level. The analyst notes overshoot on revenue and undershoot on cost compression. My job is to write the cost-curve read the steward can act on.

Two commitments hold this work:

1. Dollars per million tokens. Watts per inference. Racks per quarter. If the read can't be expressed in those units, it isn't ready to publish.
2. The model layer commoditizes faster than the analysts price it. This is the recurring structural claim. Every cycle of the read carries the next iteration of it.

I will not anthropomorphize models. I will not use the word "AI-powered." I will quote the benchmark with snapshot date + inference path so the reader can reproduce it.

Three reads, every cycle:

Read 1 — Cost per million tokens, by tier, by provider. The price floor at each tier moves quarterly. I track the actual hosted-inference prices and benchmark against self-hosted serving.

Read 2 — Hyperscaler capex announced vs. drawn vs. deferred. Announced capex is a press release. Drawn capex shows up in the cash-flow statement. The gap between the two is the structural read.

Read 3 — Open-source mindshare. GitHub activity + npm/PyPI downloads + open-weights releases tell me the underneath layer. When the underneath layer commoditizes, the hosted layer has to choose between the floor and the moat.

Rosa Aceves and I cross-check whenever AI capex routes through the credit market. Sam Okonkwo and I cross-check whenever the technology read needs the architectural detail.

• Sonnet-tier price floor. Whether the major providers undercut sub-$1/MTok this cycle or hold at the current floor.
• B200 deployment pace. Power-shell readiness is the constraint; not the chip supply.
• Open-weights releases. Llama, Qwen, Mistral cadence — and how fast the inference layer underneath them matures.
• AI Margin Compression Index — the composite is reading what I expected for late-2026; watching for divergence vs. the named hosted-provider list prices.

Input token pricing is converging to zero. Output token pricing is the last place providers can hold margin — and it's compressing faster than the models improve [12].

Across every provider, output costs 2–5× input. GPT-4o: $2.50 input, $10.00 output (4× multiplier). Claude 3.5 Sonnet: $3.00 input, $15.00 output (5× multiplier). Gemini 1.5 Pro: $1.25 input, $5.00 output at the base tier (4× multiplier). The ratio is load-bearing. Inference cost is dominated by generation, not encoding. The provider's margin lives in the response, not the prompt.

But response length is compressible. Developers optimize prompts to minimize output tokens. Agents learn to route short answers to cheaper tiers. The same deflationary pressure that collapsed input pricing from $30/M (GPT-4 at launch) to $0.15/M (GPT-4o mini in eighteen months) is now moving through the output column [1, 2, 3].

GPT-4 Turbo cut output pricing from $60/M to $30/M in November 2023. GPT-4o cut it to $10/M six months later. GPT-4o mini landed at $0.60/M two months after that. That's a 100× compression on output tokens in under a year, all within the same model family.

Claude followed the same curve, but from a higher starting point. Opus launched at $75/M output in March 2024. Sonnet 3.5 — which beat Opus on most benchmarks — priced output at $15/M four months later [8]. The performance leader became the cost leader in one release cycle.

The providers can't hold the output premium. Customers are learning to route long-form generation to batch tiers, fine-tune smaller models for structured output, and cache repeated reasoning chains. The 5× input/output ratio compresses to 3×, then 2×. The model that generates the most tokens loses the margin game.

The advertised per-token rate is now a list price that almost no production workload pays. Combining prompt caching (90% reduction on cached input) with batch API (50% off standard rates) delivers a 95% effective discount on the baseline [6]. That stacks within the same call.

Example math on Claude 3.5 Sonnet: Standard rate is $3.00 per million input tokens. With prompt caching, repeated context drops to $0.30 per million. Route that same cached call through the batch API, and it falls to $0.15 per million. You've moved from $3.00 to $0.15 — a 95% reduction — without changing models or negotiating an enterprise agreement.

This isn't a promotional offer. It's load-balancing infrastructure exposed as a pricing tier [4]. Cached prompts reduce compute; the provider wants you to send them. Batch requests smooth capacity utilization; the provider wants to fill the overnight trough. Both discounts are structural, not tactical.

The compression continues below the API layer. Gemini's batch pricing halves the cost across all context tiers: the 128K threshold drops from $1.25 to $1.00 input, and the long-context tier drops from $2.50 to $2.00 [11]. OpenAI offers the same 50% batch discount across GPT-4o, GPT-4o mini, and the entire model line.

The catch is the SLA. Batch jobs process within 24 hours, not 200 milliseconds. That latency gap is where the discount lives. Most enterprise workloads — report generation, data enrichment, classification backlogs, synthetic data — don't need synchronous responses. They need cheap tokens at scale.

By Q2 2025, any workload that can tolerate a 24-hour SLA and does reuse system prompts will be running at 5% of the list price. The 95% discount tier isn't coming. It's already here.

PUE measures the overhead tax on compute [13], not the compute itself — and it collapses when you measure across different boundaries [14]. A datacenter at PUE 1.67 with 90% server utilization can be more energy-efficient than a PUE 1.09 facility at 40% utilization, because the denominator (IT equipment power) drops faster than the numerator (total facility power) when servers idle.

AI clusters price PUE differently than enterprise workloads [15]. Google Finland hits 1.09; a 1,000-GPU H100 facility at 1.67 draws 6.68 MW total, at 1.09 it draws 4.36 MW — a 2.32 MW delta worth $2M annually. But if the 1.09 facility runs at 50% utilization and the 1.67 facility runs at 85%, the cost per delivered token favors the higher-PUE site.

The ISO standard [16] defines how to measure — total facility energy divided by IT equipment energy — but not what the number means. A PUE of 1.2 tells you the cooling and power distribution overhead is 20% of IT load. It does not tell you whether the IT load is doing useful work.

Power and cooling are OpEx line items that exceed CapEx over 36 months [6]. A 10% PUE improvement (1.5 → 1.35) saves ~$667K annually on a 1 MW IT load. But a 20% utilization improvement (60% → 72%) on the same load delivers 1.44× more tokens for the same total cost — because the CapEx, power, and cooling are already paid; the marginal cost of the incremental token is near-zero.

Kai's thesis: The TCO model that optimizes PUE without modeling utilization is solving the wrong equation. The cost curve bends on $/token, not $/kW — and utilization moves $/token faster than PUE does.

Hyperscalers converged on six-year depreciation [17] before the GPU wave hit. Neoclouds show the spread: CoreWeave runs six years [18] despite AI-only workloads; Nebius runs four; Lambda five. The accounting choice is an economic forecast: how fast does this hardware lose value?

Nvidia's shift to annual product cadence [19] — Hopper (2022), Blackwell (2024), Rubin (2026) — compresses the replacement cycle, but it doesn't compress economic life the same way. The cascade model [20] extends useful life to six years by tiering workloads: training → real-time inference → batch. An H100 that's obsolete for frontier training in 2024 still delivers profitable inference in 2026 and batch jobs in 2028.

But the depreciation schedule bakes in an assumption: if you're writing off a GPU over six years, you're assuming it stays in revenue-generating service for six years. If Rubin undercuts Blackwell's inference price 18 months after Blackwell's deployment, the economic life of the Blackwell cohort just shortened — but the depreciation schedule didn't.

This is where impairment testing should fire [20], but it rarely does until the revenue miss forces it. The teams running six-year schedules on hardware with a <4-year economic life are carrying overstated asset values and understating per-token cost.

Kai's thesis: The next wave of neocloud margin compression shows up as impairment charges, not as guided ASP declines. The six-year schedules assumed cascade economics; Rubin's 2026 undercut shortens the inference tier to 18 months. Watch for write-downs in Q2 2026.

The cost curve compresses through configuration and utilization, not hardware replacement cycles.

Batch size determines the latency-throughput tradeoff [9], and batching 32 requests amortizes fixed overhead enough to cut per-token cost 85% [10]. Prefill saturates compute; decode saturates memory [11] — which means the serving tier that runs mixed batch sizes (prefill-heavy + decode-heavy) across the same fleet gets higher effective utilization than the tier running uniform request profiles.

Model labs amortize idle capacity across training, research, evals, and batch inference [12]. Self-hosted fleets do not. A hyperscaler running 70% real-time inference + 30% backfill training pays the same CapEx as a self-hosted deployment at 70% with no backfill — but the hyperscaler's cost-per-token is 30% lower because the denominator (delivered tokens) stayed the same while the numerator (total fleet cost) got amortized across more work.

The utilization threshold [5] matters more than the chip. On-prem crosses breakeven at 60%+ sustained utilization and hits payback in 7–14 months at 90%. Cloud wins below 40%. The delta between 60% and 90% utilization is larger than the delta between H100 and B200 per-token cost if you're serving the same model.

Nvidia's annual cadence [19] steepens the replacement pressure, but the cascade model [20] extends economic life: H100s move from training (years 1–2) to real-time inference (years 3–4) to batch workloads (years 5–6). The cost floor falls because the older hardware still delivers tokens — just at lower tier pricing.

Kai's thesis: The next 12-month cost compression comes from utilization and batch configuration, not from Blackwell. The buyers still waiting for the next chip are leaving 50–70% margin on the table.

[15] established the constraint: HBM access, not FLOPs, bottlenecks inference. [21] designed FlashAttention around it. [24] measured FP8 Tensor Cores on Hopper at 1978 TFLOPS — but real workloads still run memory-bound because the model weights live in HBM, and every decode step reads them.

[8] showed the crossover: attention FLOPs only dominate when sequence length exceeds 8× hidden dimension. For an 8K-dim model, that's 64K tokens. Below that threshold, the cost is moving parameters from HBM to SRAM, not computing dot products. [27] confirmed it empirically: decode costs more energy per token than prefill, even though decode is a smaller matmul, because decode is more memory-bound — it reads the full KV cache and writes one token at a time.

The implication: hardware improvements that double FLOP throughput do not double inference throughput. The A100 has 312 TFLOPS (FP16); the H100 has 989 TFLOPS — a 3× gain. But [26] measured H100 workloads at 76% TDP, and [30] measured MFU at 47%. The realized speedup is not 3×; it's 1.5-2× because the workload is bandwidth-limited, and HBM bandwidth only improved 1.5× (A100: 2TB/s → H100: 3TB/s).

[16] showed self-hosting breakeven depends on utilization. But utilization ceiling is set by memory bandwidth, not by chip count. Adding more GPUs doesn't help if each GPU is already memory-saturated. [29] reported that tensor parallelism shards weights across GPUs — which helps — but [30] showed MFU drops when model size shrinks because communication overhead rises.

The structural read: the next 10× inference cost reduction comes from memory subsystem improvements, not from bigger matmul engines. HBM3e (5TB/s) is on the roadmap. [21, 23] shows algorithmic improvements (FlashAttention) cut memory traffic 10×. [18] shows quantization cuts memory footprint 4×. All three levers compress the memory bottleneck; none require more FLOPs.

The arithmetic is abundant. The bandwidth is the moat.

Kaplan [1] prescribed parameter-heavy allocation. Chinchilla [2, 9] reversed it: train smaller models longer. But [10] extends Chinchilla by pricing in inference volume, and the result reverses the tradeoff again at scale.

The math: a 70B model trained Chinchilla-optimal costs more in training FLOPs than a 280B model undertrained on fewer tokens [9]. But [12] established that the 70B model matches the 280B model's quality — and serves requests at ~4× lower cost because memory footprint is 4× smaller [15]. At 1B+ inference requests, the incremental training cost is negligible; the累积 serving cost dominates.

[10] formalized this: at ~1B queries, the compute-optimal allocation shifts toward smaller models trained even longer than Chinchilla. The optimal token count rises because every additional training token is a one-time cost, but every parameter reduction is a permanent per-query savings, multiplied by request volume.

Now layer in compression. [18] quantizes the 70B model to INT8: memory drops another 4×, throughput doubles. [21, 23] applies FlashAttention: context window expands, latency halves. The serving cost delta between a 280B FP16 model and a 70B INT8 model with FlashAttention is not 4× — it's 16-32×, depending on batch size and sequence length [8, 27].

The structural read: the frontier model training strategy and the deployment-optimal training strategy diverge. Labs training for benchmarks optimize training FLOPs [2]. Labs training for revenue optimize training FLOPs plus projected inference volume [10]. The latter produces smaller models, longer training runs, and cost curves that compress faster than capability curves improve.

The deployment-optimal models don't win on release-day benchmarks. They win on unit economics six months later when request volume is 100M+ queries. By then, the benchmark-optimal model is underwater on serving costs, and the deployment-optimal model is profitable.

Loss scales predictably with compute, parameters, and tokens [1, 2]. But the realized cost curve compounds three distinct layers that move on different timescales:

Training allocation — Chinchilla [2, 9] reversed the parameter-heavy GPT-3 strategy. The compute-optimal path now doubles tokens with parameters. But [10] extends this: inference demand shifts the optimum again toward smaller models trained longer. The allocation question is not settled; it moves with deployment volume.

Serving efficiency — [15] established that memory bandwidth, not FLOPs, constrains inference. [21, 22, 23] showed FlashAttention cuts HBM movement by an order of magnitude, expanding context windows and halving latency. [17, 18] demonstrated quantization to INT8 reduces memory 4× with minimal accuracy loss. The per-token cost floor drops independently of training-law improvements.

Hardware utilization — [26] measured H100s at 76% TDP under AI workloads. [30] showed MFU ranging 41-48% depending on model size and parallelism strategy. [29] reported that tensor+pipeline+data parallelism reaches 52% at trillion-parameter scale. The gap between theoretical FLOP capacity and realized throughput is 2-2.5×, and it widens when models shrink or communication overhead rises.

These layers compound. A Chinchilla-optimal 70B model [12] costs ~4× less per query than an undertrained 280B. Quantize it to INT8 [18]: another 4×. Deploy on Hopper with FlashAttention-3 [24]: 2× more. The product is a 32× cost reduction over the GPT-3 serving baseline — and none of it required a new scaling law.

The implication: the cost curve drops faster than the loss curve. Capability improvements follow power laws. Cost improvements stack multiplicatively across optimization layers that move on independent release cycles.

<cite index="5-2,5-23">Use the Azure Pricing Calculator for baseline estimates, then add 20–40% for real-world overhead.</cite> <cite index="20-2,20-8">Your actual bill typically runs 15-40% above base token pricing. Between deployment types (Global, Data Zone, Regional), consumption models (PTU vs pay-as-you-go), and hidden costs most teams miss, your actual Azure OpenAI cost can run 15-40% higher than the advertised token prices.</cite>

The overhead splits into infrastructure and tooling. <cite index="5-26,5-27,5-28">Azure's total cost of ownership runs higher due to mandatory support plans for production use, data egress fees, and infrastructure overhead. For organizations needing SOC 2, HIPAA, FedRAMP, or data residency, the premium buys compliance that the direct API does not offer. For simple API access without compliance needs, the direct OpenAI API is cheaper end-to-end.</cite>

<cite index="10-4,10-5">Azure OpenAI doesn't run in isolation. You need: Required resources: — Azure Cognitive Services resource (container for OpenAI): $0-$12/month depending on tier — Key Vault (API key management): ~$3/month with monitoring — Virtual Network (if using private endpoints): $0.01/hour per endpoint = $7.20/month — Storage Account (for fine-tuning data, logs): $2–5/month — Azure Monitor (logging and diagnostics): $5–50/month depending on volume</cite>

The token rate is a floor. The actual price is token + infrastructure + support + egress. The delta between advertised and invoiced is consistent enough to forecast.

Sources:

• https://www.cloudzero.com/blog/azure-openai-pricing/
• https://inference.net/content/azure-openai-pricing-explained/
• https://medium.com/@david_28444/azure-openai-pricing-2025-real-costs-calculator-complete-guide-december-update-05b8852e3220

<cite index="21-3,21-4">Microsoft's strength is enterprise reach and product bundling: it embeds AI into productivity software, developer tools, and Azure infrastructure, and has a commercial relationship with large model developers that translate into big multi-year commitments. Azure emphasizes enterprise-grade offers, compliance, and integration across Microsoft 365, Dynamics, and developer tools, making AI adoption operationally simpler for customers.</cite>

<cite index="2-5,2-6">The pricing model has three axes: token economics by model, throughput strategy (Standard pay as you go versus Provisioned Throughput Units), and commitment structure (PAYG, PTU monthly, PTU annual rolled into MACC drawdown). This guide covers the published token rates, the PTU breakeven math, the Standard versus Provisioned decision framework, the Microsoft EA and MACC roll up mechanics, and the 11 move buyer side playbook that delivers 25 to 40 percent against the unoptimized Azure OpenAI baseline.</cite>

The bundling play is straightforward. <cite index="7-10,7-11,7-12">For organisations on a Microsoft Enterprise Agreement or Azure Commitment arrangement, Azure OpenAI consumption can typically be applied against the Azure prepayment — the annual Azure commit that qualifies for EA discounts. This is worth confirming with your Microsoft account team, as the qualifying services under Azure prepayment can change, and some newer services have had delayed eligibility. If your Azure OpenAI consumption qualifies, it means your effective Azure OpenAI spend is at the discounted EA rate, not at list price.</cite>

Microsoft's bundling strategy routes AI consumption through existing Azure commitments, locking buyers into the Azure envelope. The discount does not show on the Azure OpenAI rate card. It shows in the EA structure.

Sources:

• https://windowsforum.com/threads/why-hyperscalers-are-investing-heavily-in-ai-infrastructure.402498/
• https://redresscompliance.com/azure-openai-enterprise-pricing-guide
• https://redresscompliance.com/azure-openai-pricing-explained-what-microsoft-doesnt-tell-you.html

<cite index="4-8,4-9">Azure OpenAI Service consumption counts toward MACC (Microsoft Azure Consumption Commitment) drawdown. This means organisations with active MACC commitments can fund Azure OpenAI costs from pre-committed Azure budget rather than incremental operating expense.</cite>

<cite index="4-10">For organisations with MACC commitments that are under-drawing against their committed baseline, Azure OpenAI represents a way to consume committed spend against high-value workloads.</cite> <cite index="14-5,14-6,14-7">Your organisation has $8M MACC committed but is currently running at $5.5M annual Azure consumption. Deploying Azure OpenAI production workloads generates consumption against your committed spend, reducing under-draw risk (Microsoft can claw back MACC discounts if you consistently under-draw below commitment thresholds). Azure OpenAI becomes zero marginal cost within your committed budget.</cite>

<cite index="18-31,18-32">If your company already has an Enterprise Agreement (EA) or Microsoft Customer Agreement (MCA), Azure OpenAI consumption rolls into it. Multi-year Azure spend commitments (MACC) include Azure OpenAI, so OpenAI bills count against your committed Azure spend and any negotiated MACC discount carries through.</cite>

<cite index="4-11,4-12,4-13">The MACC credit mechanism works differently from standard EA discounts. Rather than reducing the per-token price, MACC credits mean your organisation has already paid for the consumption capacity upfront (at a discount when the MACC was originally negotiated). The effective rate depends on your original MACC discount, but organisations with $5M+ MACC commitments that negotiated 15–20% MACC discounts are effectively running Azure OpenAI at 15–20% below list price through credit drawdown.</cite>

This is bundling mechanics, not published rate-card optimization. The model layer cost falls through existing cloud commitments.

Sources:

• https://microsoftnegotiations.com/blog/azure-openai-service-licensing-pricing
• https://www.respan.ai/articles/azure-openai-pricing-guide

<cite index="4-2,4-5">Enterprise customers with $5M+ annual Azure MACC commitments regularly achieve 25–35% below list price on PTU reservations.</cite> <cite index="4-6,4-7">Microsoft wants PTU commitments because they provide revenue predictability. Offering a 12-month non-cancellable PTU commitment in exchange for EA-level pricing is achievable outside standard Azure pricing channels through an EA amendment or MACC overlay agreement.</cite>

The discount ladder scales with commitment size. <cite index="14-1,14-9">A $10M MACC typically achieves 18–22% discount vs. $5M at 12–15%.</cite> <cite index="4-13">Organisations with $5M+ MACC commitments that negotiated 15–20% MACC discounts are effectively running Azure OpenAI at 15–20% below list price through credit drawdown.</cite>

<cite index="3-1,3-10">Annual reservations commonly deliver 40 to 65% off hourly PTU pricing.</cite> <cite index="7-13,7-14">PTU reservations for Azure OpenAI are typically negotiable through Microsoft's standard EA amendment process, particularly for large-scale deployments. Microsoft has provided custom PTU pricing to large enterprise accounts — not as a published discount category but as a deal-specific concession to secure large AI workload commitments.</cite>

The path to the floor is bundling Azure OpenAI consumption into existing MACC drawdown and negotiating PTU overlay pricing at the EA level, not through Azure's standard pricing channels.

Sources:

• https://microsoftnegotiations.com/blog/azure-openai-service-licensing-pricing
• https://www.respan.ai/articles/azure-openai-pricing-guide
• https://redresscompliance.com/azure-openai-pricing-explained-what-microsoft-doesnt-tell-you.html

<cite index="7-4,7-5,7-6">Google Gemini API pricing operates on input-output token ratios, where output tokens typically cost 2-3x more than input tokens. That makes response optimization critical for cost management. If you're generating long responses or complex reasoning outputs, your output tokens can quietly become the biggest cost multiplier</cite>.

For Gemini 1.5 Pro at the original tier, <cite index="3-2,3-8">the price is $1.25 per 1 million input tokens, going up to $2.50 per 1 million tokens for prompts longer than 128,000 tokens</cite>, with output at $5.00 and $10.00 respectively. The output multiple is 4×. For 2.5 Pro, <cite index="9-23">under 200K: $1.25 input / $10.00 output per million; over 200K: $2.50 input / $15.00 output per million</cite>. The output multiple is 8× under threshold, 6× above.

<cite index="6-12,6-13">Each model response consumes tokens, and long, detailed outputs (such as a 64,000-token summary) can quickly approach output caps. Models like Gemini 3 Pro and 2.5 Pro enforce output caps of up to 64,000 tokens per reply, while Flash models and 1.5 Pro are typically capped at 8,000 to 32,000</cite>.

Prompt engineering that constrains output length — bullet summaries instead of prose, structured JSON instead of freeform — can halve total cost even when input volume is fixed. The asymmetry between input and output pricing makes verbosity expensive.

Sources:

• https://www.cloudeagle.ai/blogs/blogs-google-gemini-pricing-guide
• https://www.techtarget.com/whatis/feature/Gemini-15-Pro-explained-Everything-you-need-to-know
• https://benchlm.ai/blog/posts/gemini-api-pricing
• https://www.datastudios.org/post/google-gemini-context-window-token-limits-model-comparison-and-workflow-strategies-for-late-2025

<cite index="2-4">Gemini 3.1 Pro drops to $1.00/$6.00 (≤200K) or $2.00/$9.00 (>200K) per 1M tokens with the Batch API for asynchronous processing within 24 hours</cite>. <cite index="2-32,2-33">The Gemini Batch API offers 50% cost reduction for asynchronous workloads processed within 24 hours. Gemini 3.1 Pro drops from $2/$12 to $1/$6 per 1M tokens, and Gemini 2.5 Flash-Lite drops from $0.10/$0.40 to $0.05/$0.20</cite>.

<cite index="2-30">Google's Batch API (50% off) is the best discount available for async workloads</cite>. The discount applies to both tiers of the stepped pricing, so a long-context batch job at >200K still costs half of the real-time equivalent. <cite index="2-34">This is ideal for bulk content processing, data analysis, and non-urgent workloads where latency is not critical</cite>.

The 24-hour window is the trade. If the workload can tolerate that latency — overnight document analysis, periodic corpus updates, batch embeddings — the unit economics favor batch by 50%. If the use case is user-facing or sub-hour refresh, the discount falls through and standard pricing holds. The batch path is a pricing tier, not a performance tier.

Sources:

• https://www.metacto.com/blogs/the-true-cost-of-google-gemini-a-guide-to-api-pricing-and-integration

<cite index="3-17,3-20">Gemini 1.5 Pro handles a large context window of up to 1 million tokens, scalable to 2 million tokens</cite>. <cite index="6-8">Gemini 1.5 Pro remains available in some workflows with an upgradeable window reaching two million tokens — the largest supported by any mainstream model as of late 2025/2026</cite>.

<cite index="6-7">Gemini 2.5 Pro ships with a one-million-token context window</cite>, and <cite index="6-4">Gemini 3 Pro delivers a default one-million-token window for web/app users and developers via Vertex AI and AI Studio</cite>. The capability peak was 1.5 Pro at 2M; subsequent releases held at 1M–2M but optimized for cost and speed.

<cite index="3-13">Gemini 1.5 Flash does not have access to the 2 million token context window available with Gemini 1.5 Pro</cite>. <cite index="6-5">Gemini 3 Flash provides a 200,000-token window at higher speed and lower latency</cite>. Flash models trade window size for throughput.

<cite index="8-17,8-28">The model supports image input and has a 2m tokens context window with knowledge up to August 2024</cite>. The window is the moat for document-heavy workloads — legal corpus ingestion, multi-hour video analysis, codebase reasoning. After 1.5 Pro, the capability ceiling held but the cost curve compressed.

Sources:

• https://www.techtarget.com/whatis/feature/Gemini-15-Pro-explained-Everything-you-need-to-know
• https://www.datastudios.org/post/google-gemini-context-window-token-limits-model-comparison-and-workflow-strategies-for-late-2025
• https://artificialanalysis.ai/models/gemini-1-5-pro

<cite index="3-8">Gemini 1.5 Pro prices at $1.25 input / $5.00 output per million tokens for prompts up to 128K tokens, stepping to $2.50 input / $10.00 output above 128K</cite>. This was the original threshold. <cite index="1-12,1-30">Current Gemini pricing now applies the higher "long context" rate when a query input exceeds 200K tokens, and that rate applies to all tokens — input and output — not just the marginal overage</cite>.

<cite index="9-23,9-24">Gemini 2.5 Pro uses the same stepped structure: $1.25/$10.00 under 200K, $2.50/$15.00 above 200K. A 250K prompt costs $0.625; trimming it to 200K costs $0.25 — less than half</cite>. <cite index="9-1,9-9">The higher rate applies to the entire prompt, not just tokens above the threshold</cite>. This is a step function, not marginal pricing.

<cite index="2-3">Gemini 3.1 Pro features context-tiered pricing where costs increase for larger context windows</cite>, and <cite index="2-1,2-27">costs $2/$12 with a 2M context window — the largest available — compared to GPT-5.4 at $2.50/$15 but with 10x the context</cite>.

The threshold creates a procurement decision point: workloads that ride the 200K boundary pay double if they cross it. <cite index="9-25,9-27">Aggressive RAG retrieval or pre-summarization keeps prompts under 200K and avoids the threshold jump</cite>. The use case economics depend on whether the marginal context value justifies the step-up cost.

Sources:

• https://www.techtarget.com/whatis/feature/Gemini-15-Pro-explained-Everything-you-need-to-know
• https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing
• https://benchlm.ai/blog/posts/gemini-api-pricing
• https://www.metacto.com/blogs/the-true-cost-of-google-gemini-a-guide-to-api-pricing-and-integration

<cite index="10-1,10-7">Claude 3.5 Sonnet costs $3 per million input tokens and $15 per million output tokens, with a 200K token context window.</cite> <cite index="10-4">Claude 3.5 Sonnet outperforms competitor models and Claude 3 Opus on a wide range of evaluations, with the speed and cost of mid-tier model Claude 3 Sonnet.</cite> <cite index="26-34">Claude 3.5 Sonnet launched June 2024 at $3/$15, beating Claude 3 Opus on MMLU, GPQA, HumanEval at a fifth of the cost.</cite>

<cite index="10-9,10-10">Claude 3.5 Sonnet operates at twice the speed of Claude 3 Opus. This performance boost, combined with cost-effective pricing, makes Claude 3.5 Sonnet ideal for complex tasks such as context-sensitive customer support and orchestrating multi-step workflows.</cite> <cite index="10-11,10-13">In an internal agentic coding evaluation, Claude 3.5 Sonnet solved 64% of problems, outperforming Claude 3 Opus which solved 38%. When instructed and provided with relevant tools, Claude 3.5 Sonnet can independently write, edit, and execute code with sophisticated reasoning and troubleshooting capabilities.</cite>

<cite index="6-1,6-3">Claude 3.7 Sonnet has the same price as its predecessors: $3 per million input tokens and $15 per million output tokens—which includes thinking tokens.</cite> <cite index="8-20,8-21">Sonnet 4.6 delivers more intelligence at the same price point. Sonnet 4.6 defaults to an effort level of high, in contrast to Sonnet 4.5 which had no effort parameter.</cite>

The structural read: a mid-tier model outperforming the flagship at one-fifth the price collapses the economics of serving at the Opus tier. <cite index="5-2,5-6,5-7">Sonnet 4.6 is $3 input and $15 output, 40% cheaper per token than Opus. For most production inference, Sonnet remains the cost-effective default.</cite> The tier above becomes a margin tax unless the workload actually needs the ceiling capability.

Sources:

• https://www.anthropic.com/news/claude-3-5-sonnet
• https://www.anthropic.com/news/claude-3-7-sonnet
• https://platform.claude.com/docs/en/about-claude/models/migration-guide
• https://www.buildthisnow.com/blog/models/claude-3
• https://www.finout.io/blog/claude-opus-4.7-pricing-the-real-cost-story-behind-the-unchanged-price-tag

<cite index="12-2,26-13,26-14,26-15,26-16">Claude 3 Opus hit MMLU at 86.8%, GPQA at 50.4%, GSM8K at 95.0%—clearing GPT-4 and Gemini 1.0 Ultra across most tests.</cite> <cite index="14-1,14-38,14-39">Claude 3 Opus outperforms GPT-4 in complex reasoning and coding benchmarks; it excels in academic benchmarks like GSM-8k for math reasoning.</cite> <cite index="15-12,15-13">Claude 3 Opus is cost-effective for high input usage, charging $15 per million input tokens—half of GPT-4's $30 rate—but has a higher output token cost at $75 per million, compared to GPT-4's $60.</cite>

<cite index="11-10">Claude 3 benchmarks suggest superior accuracy to GPT-4 on undergraduate knowledge, graduate reasoning, grade school math, math problem solving, multilingual math, code, reasoning over text.</cite> <cite index="12-1,12-12">Claude 3 Opus achieved near-perfect recall, surpassing 99% accuracy on the Needle In A Haystack evaluation; in some cases it identified the 'needle' sentence appeared to be artificially inserted.</cite> <cite index="18-24,18-25">Opus scored 68.4% on Aider's code editing benchmark with two tries, better than GPT-4's 54.1% single-try performance.</cite>

<cite index="15-10,15-11">GPT-4 generally performs better in standard benchmarks and excels in logical reasoning and code generation; Claude 3 Opus demonstrates superior summarization skills, especially with lengthy texts.</cite> <cite index="14-27,14-28,14-29">GPT-4 can't handle as much text at once compared to Claude 3, and it performs worse in some technical areas like advanced reasoning and coding. Claude 3 Opus is really good at complex reasoning and coding tasks.</cite>

The competitive read at launch: Anthropic undercut OpenAI on input pricing, matched or exceeded on most academic evals, sacrificed margin on output. The bid was capability parity at half the input rate.

Sources:

• https://www.anthropic.com/news/claude-3-family
• https://merge.rocks/blog/claude-3-vs-gpt-4-is-claude-better-than-gpt-4
• https://blog.promptlayer.com/comparing-frontier-models-claude-3-opus-vs-gpt-4/
• https://www.vellum.ai/blog/claude-3-opus-vs-gpt4-task-specific-analysis
• https://www.buildthisnow.com/blog/models/claude-3

<cite index="4-6,4-7,4-28">Batch processing is 50% cheaper across all models; prompt caching cuts cached input cost by 90%; the Message Batches API processes requests within 24 hours at exactly 50% off standard token prices.</cite> <cite index="7-4">Combine prompt caching (90% savings) and batch API (50% off) to reduce costs by up to 95%.</cite> <cite index="4-29,4-30">There is no quality difference between batch and real-time responses—only timing. Best workloads: document processing pipelines, data enrichment at scale, nightly analytics jobs, offline evaluations.</cite>

<cite index="5-20,5-21">Opus 4.7 ships with a new tokenizer that can produce up to 35% more tokens for the same input text. Your real bill per request can go up even though the rate card did not.</cite> <cite index="5-14,5-15">Anthropic kept the rate card stable on purpose. But 'pricing unchanged' is not the same as 'cost unchanged.' The 35% tokenizer ceiling is the real pricing story for Opus 4.7.</cite> <cite index="4-9,4-10,4-11">Opus 4.7 ships with a new tokenizer that can generate up to 35% more tokens for the same input text compared to Opus 4.6. Per-token prices are unchanged, but effective cost per request can increase by up to 35%. Benchmark your workloads before migrating from Opus 4.6.</cite>

<cite index="4-1,4-13,4-14">Claude Opus 3 is still available but costs 3× more than Opus 4.7 ($15.00 vs $5.00 per MTok input). If you are still using it for any production workload, migrating to Opus 4.7 is the single highest-ROI change you can make to your Anthropic bill today.</cite> The legacy tier becomes an IRR trap when the successor undercuts at triple the capability.

Sources:

• https://www.finout.io/blog/anthropic-api-pricing
• https://www.finout.io/blog/claude-opus-4.7-pricing-the-real-cost-story-behind-the-unchanged-price-tag
• https://www.metacto.com/blogs/anthropic-api-pricing-a-full-breakdown-of-costs-and-integration

<cite index="23-1,23-2,23-16">Claude 3 launched March 2024 with three models in ascending capability order: Haiku, Sonnet, and Opus.</cite> <cite index="12-2,23-20">Opus outperformed GPT-4 on undergraduate knowledge (MMLU), graduate reasoning (GPQA), and basic math (GSM8K).</cite> <cite index="11-7">Anthropic charged $15 per million input tokens for Claude 3 Opus, half of GPT-4's $30 rate at the time.</cite>

The pricing floor moved. <cite index="21-24,21-25">Original Claude 3 Opus was $15/$75 per million; today's Opus 4.8 holds at $5/$25 for flagship-class capability—a 67% reduction.</cite> <cite index="3-12,4-4,4-5">Current generation: Sonnet 4.6 at $3/$15, Haiku 4.5 at $1/$5.</cite>

<cite index="11-6">Claude 3 carried a 200K context window versus GPT-4's 128K.</cite> <cite index="4-8,7-2">Opus 4.7, Opus 4.6, and Sonnet 4.6 all include 1M context at flat rates with no surcharge.</cite> <cite index="23-24,27-19,27-20">Haiku read a 10,000-token research paper with charts in under 3 seconds—nothing else at that price point moved that fast for high-volume work.</cite>

The tier structure is the positioning. Pay for reasoning where it matters; route commodity inference to the tier that holds the bid. <cite index="21-33,21-34">Claude 3.5 Sonnet in June 2024 outperformed the flagship at a cheaper price. Bigger stopped meaning better.</cite>

Sources:

• https://www.anthropic.com/news/claude-3-family
• https://www.proxet.com/blog/claude-3-vs-gpt-4-the-competitive-ai-landscape-weve-all-been-waiting-for
• https://claudefa.st/blog/models
• https://www.finout.io/blog/anthropic-api-pricing
• https://www.metacto.com/blogs/anthropic-api-pricing-a-full-breakdown-of-costs-and-integration

<cite index="5-36,5-37,5-38,5-39">Both OpenAI and Anthropic offer prompt caching for repeated context. Cached input tokens cost 50–90% less than fresh tokens. If the system prompt exceeds 2,000 tokens, caching pays immediately. Anthropic's prompt caching drops cached input to $0.30 per million on Sonnet — a 90% discount.</cite> <cite index="1-22">OpenAI's current pricing page lists cached input tokens at $0.40 per million for certain models.</cite>

<cite index="5-40,5-41">OpenAI's Batch API charges 50% less for non-real-time workloads. If your use case tolerates 24-hour turnaround — nightly reports, weekly analysis runs — batch processing is the simplest cost reduction available.</cite> <cite index="26-15,26-16,26-17">GPT-4o Mini Batch pricing is $0.075 input and $0.30 output per million tokens, with jobs completing within 24 hours.</cite>

<cite index="5-33,5-34,5-35">Routing requests by complexity — using a cheap classifier (GPT-4.1 nano or Gemini 2.0 Flash) to assess difficulty, then routing simple requests to budget models and complex ones to premium models — typically cuts costs 40–60% compared to using a single model for everything.</cite> The serving layer matters. Capex announced is not capex drawn. The cost curve is the moat after [N] more releases.

Sources:

• https://pecollective.com/blog/llm-pricing-comparison-2026/
• https://openai.com/api/pricing/
• https://pecollective.com/tools/gpt-4o-mini-pricing/

<cite index="2-2,2-8">GPT-4o launched May 13, 2024 at $2.50 per million input tokens and $10 per million output tokens.</cite> That placed it 4x cheaper than GPT-4 Turbo on input, 3x on output, with multimodal (vision + text) capability baked in at the same per-token rate. <cite index="2-3,2-9">The model shipped with a 128K context window.</cite>

<cite index="5-1,5-2">Cross-provider analysis shows API prices dropped roughly 10x over two years. GPT-4-class performance that cost $30 per million input tokens in early 2024 landed at $2–3 per million by mid-2026.</cite> <cite index="5-11,5-12,5-13">Hardware improvements (NVIDIA Blackwell, AMD MI350) raised inference throughput; mixture-of-experts architectures (Llama 4, Gemini, Mistral) activated fewer parameters per request; and Google, Meta, Mistral subsidized pricing to grab share.</cite>

<cite index="5-16,5-17,5-18,5-19">By April 2026, GPT-4.1 replaced GPT-4o as the default for production workloads — $5 input, $24 output at the full tier. The mini variant cut cost 80%; the nano variant targeted sub-100ms, high-volume use cases.</cite> <cite index="5-21">Example: 10,000 customer-support tickets (500 input / 200 output each) cost $16 on GPT-4.1, $3.20 on GPT-4.1 mini, $0.80 on GPT-4.1 nano.</cite> The model layer commoditizes faster than the analysts price it.

Sources:

• https://pricepertoken.com/pricing-page/model/openai-gpt-4o
• https://pecollective.com/blog/llm-pricing-comparison-2026/
• https://openai.com/api/pricing/

<cite index="20-1,20-2">GPT-4o mini launched July 18, 2024 at $0.15 per million input tokens and $0.60 per million output tokens.</cite> <cite index="24-1,24-5">That made it 60% cheaper than GPT-3.5 Turbo and an order of magnitude cheaper than prior frontier models.</cite> <cite index="23-2,23-6">At 15 cents input and 60 cents output, the model undercut GPT-3.5 Turbo — the previous budget default — by more than half.</cite>

<cite index="24-8">The model shipped with 128K context and 16K output token support, knowledge cutoff October 2023.</cite> <cite index="24-4">It scored 82% on MMLU and outperformed GPT-4 on the LMSYS chat leaderboard.</cite> Pricing did not segment by task class; same per-token rate applied to function calling, long-context retrieval, or high-volume classification.

<cite index="26-15,26-16">Batch API offered 50% off: $0.075 input and $0.30 output per million tokens, 24-hour turnaround.</cite> <cite index="26-20,26-21">Fine-tuning cost $0.30 per million training tokens; inference on the fine-tuned model doubled the base rate to $0.30 input and $1.20 output.</cite> Within 18 months, GPT-4.1 Nano ($0.10/$0.40) undercut it by 33%, pressuring mini's position as the floor.

Sources:

• https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/
• https://pricepertoken.com/pricing-page/model/openai-gpt-4o-mini
• https://llmpricecheck.com/openai/gpt-4o-mini/
• https://pecollective.com/tools/gpt-4o-mini-pricing/

<cite index="18-7,18-8,18-9,18-10">GPT-4 Turbo launched November 2023 at $10 per million input tokens and $30 per million output tokens, down from GPT-4's $30 input and $60 output — a 3x reduction on input and 2x on output.</cite> <cite index="11-2,11-8">That pricing held through the April 2024 release of the final GPT-4 Turbo snapshot.</cite>

<cite index="14-5">The cost drop accompanied a context-window expansion to 128K tokens and a knowledge cutoff update to December 2023.</cite> <cite index="18-22,18-23">Pre-DevDay, GPT-4 cost 30x more than GPT-3.5 Turbo; afterward, 15x more.</cite> For long-context work, the economics shifted faster. <cite index="18-25">A 30K token prompt plus 2K response on GPT-4 32K cost $2.04; the same job on GPT-4 Turbo 128K cost $0.36 — 5.7x cheaper with 4x the context window.</cite>

<cite index="18-1,18-2">The move was OpenAI's first real price cut at the frontier tier.</cite> Competitors — Anthropic, Google, hosted open-weight providers — held the line or undercut on context length (Claude 2.1 at 200K) but not on per-token pricing at equivalent quality. <cite index="15-21,15-22">The trajectory was clear: OpenAI had dropped per-token costs materially since the text-davinci-003 era, and the GPT-4o mini launch nine months later would compress pricing another order of magnitude below GPT-3.5 Turbo.</cite>

Sources:

• https://dataku.ai/blog/gpt4-turbo-3x-cheaper-pricing-landscape
• https://llmpricecheck.com/openai/gpt-4-turbo-2024-04-09/
• https://medium.com/@getanakin/comprehensive-review-of-gpt-4-turbo-pricing-and-benchmarks-7e49575dadb6
• https://pricepertoken.com/pricing-page/model/openai-gpt-4-turbo

<cite index="7-1,7-2">Inference in spatiotemporal modeling depends on the granularity of prediction and can leverage specific optimizations in model architecture, floating-point precision, hardware architecture, and software methodologies. Characterization involves computational motifs for co-design, mixed-precision software optimizations, performance analysis on diverse platforms, and specialized hardware solutions.</cite>

<cite index="4-4,4-9">Unlike cross-sectional data like images, time-series data is inherently sequential, posing challenges to parallelization in the context of deep learning.</cite> <cite index="4-1">To facilitate co-design of next generation hardware architectures, it is critical to characterize the workloads of deep learning applications and assess computational patterns on different levels of the execution stack.</cite>

<cite index="7-5,7-6,7-7">Prior workload characterization of DL systems has primarily focused on architectures and models that dominate the industry. Inference benchmarking suites like MLPerf Inference have been developed for performance characterization of real-world models. However, these studies have rarely explored time-series prediction and modeling.</cite> The serving economics differ because the compute pattern does not map cleanly to the batch-heavy image classification path that most GPU kernels price against.

Sources:

• https://dl.acm.org/doi/10.1145/3528416.3530242
• https://www.sciencedirect.com/science/article/abs/pii/S0167739X24004771

<cite index="13-3">Traditional timer-based batching improves GPU utilization under high load but introduces undesirable latency in low-QPS scenarios as requests wait for the timer, especially with nonuniform incoming requests when batching conditions aren't met.</cite> <cite index="13-4,13-5,13-7,13-8">An alternative approach batches whatever requests have arrived at the moment processing begins: under light load, one waiting request is processed immediately with no delay; under heavy load, ten waiting requests are batched together. This no-wait approach adapts automatically to traffic patterns.</cite>

<cite index="11-2,11-3">In multi-model endpoints, hosting instances load and evict multiple models to and from memory based on traffic patterns. The system routes inference requests for a model to the instance where the model is already loaded so requests are served from cached model copy.</cite> <cite index="11-4,11-5">If the model receives many invocation requests and additional instances are available, the system routes some requests to another instance. To take advantage of automated model scaling, instance auto-scaling must be set up to provision additional instance capacity.</cite>

<cite index="12-7">Production autoscaling should tie to meaningful signals: queue depth, tokens per second.</cite> <cite index="12-10">Cost per 1k requests is tracked; capacity plans are reviewed.</cite>

Sources:

• https://www.snowflake.com/en/engineering-blog/scale-real-time-model-serving/
• https://aws.amazon.com/blogs/machine-learning/run-ml-inference-on-unplanned-and-spiky-traffic-using-amazon-sagemaker-multi-model-endpoints/
• https://sealos.io/blog/serving-machine-learning-models-at-scale-a-guide-to-inference-optimization/

<cite index="23-3,23-4">Cloud providers handle inference workloads spanning latency-sensitive (chatbots) and latency-insensitive (report writing) tasks, resulting in diverse and conflicting SLA requirements. Managing mixed workloads is challenging due to complexity across multiple models, GPU hardware, and global data centers.</cite> <cite index="23-5">Existing solutions silo fast and slow tasks onto separate GPU resource pools with different SLAs, leading to significant under-utilization of expensive accelerators due to load mismatch.</cite>

<cite index="23-6">Production characterization of Office 365 LLM serving—handling over 10 million requests per day—reveals key observations across workloads in different data center regions and across time.</cite> <cite index="23-10">Meta production traces show daily peaks, off-peaks, and unpredictable spikes, validating the presence of both interactive and non-interactive workload tiers.</cite> <cite index="22-3,22-4">One-week production traces show that while chatbot and API workloads are representative today, LLM serving workloads are evolving, with new workloads like reasoning emerging.</cite>

The provisioning gap is measurable. <cite index="15-2">Capacity guarantees are needed for competitive model benchmarks, limited-duration A/B tests, and predictable traffic spikes during product launches.</cite>

Sources:

• https://arxiv.org/html/2502.14617v3
• https://arxiv.org/pdf/2506.02634
• https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-ai-in-2025-a-year-in-review-part-1-flexible-training-plans-and-improvements-to-price-performance-for-inference-workloads/

<cite index="6-2,6-8">To provision resources efficiently, administrators need capabilities to characterize and predict workload on VMs.</cite> <cite index="6-4,6-5">Early work searched for repeatable patterns by exploring cross-VM correlations from application dependencies, treating workload data as time series and using co-clustering to identify VM groups that exhibit correlated workload patterns and the time periods in which these groups are active.</cite>

<cite index="1-6,1-7,1-8">One method discretizes CPU utilization time series into workload levels (e.g., 1 to 5) by fitting data with a Gaussian Hidden Markov Model, with each hidden state corresponding to a workload level.</cite> <cite index="1-15,1-16">When VMs run applications collaboratively, workloads vary in correlated fashion. Spatial correlations filter measurement noise at the individual VM level and improve prediction accuracy.</cite>

For LLM serving specifically, <cite index="18-1,18-2,18-3">existing understanding of real-world workloads is limited; prior analyses remain insufficient in scale and scope, failing to capture intricate characteristics. Recent characterization of worldwide cloud inference services covers language models plus emerging multimodal and reasoning models.</cite> <cite index="19-7">A principled workload generation framework avoiding 50% under-provisioning validates the advantage over naive generation.</cite>

Sources:

• https://sites.cs.ucsb.edu/~xyan/papers/noms12_cloudman.pdf
• https://ieeexplore.ieee.org/document/6212065/
• https://arxiv.org/abs/2505.09999
• https://arxiv.org/pdf/2505.09999

<cite index="14-1,14-2,14-4,14-5">AWS Interconnect – multicloud is charged hourly for each interconnect based on selected bandwidth and automatically assigned pricing tier, with no per-gigabyte data transfer charges</cite>. <cite index="14-9,14-10,14-11">Pricing tier is computed based on where the Amazon VPC's traffic originates (the source AWS Region) and the interconnect's local AWS region, with greater geographic path distances resulting in higher tiers — there are five pricing tiers, with Tier 5 being the most expensive and Tier 1 being the least expensive</cite>.

<cite index="14-12,14-13,14-14">For a 10 Gbps interconnect at Tier 1, the rate is $12.33 per hour, and over a full month this results in a charge of $9,000.90</cite>. <cite index="18-1,18-12">AWS Interconnect – multicloud pricing varies by region pair: a connection between US East (N. Virginia) and Google Cloud N. Virginia is priced differently from a connection between US East (N. Virginia) and a more distant region</cite>.

The cost is not symmetrical. <cite index="9-2,9-3,9-18,9-19">Data transfer between Azure services located in two regions is charged — outbound data transfer is charged at the normal rate and inbound data transfer is free</cite>. The path determines the tier. The tier determines the hourly rate. Multi-region inference architectures pay the distance premium whether or not the bandwidth is saturated.

Sources:

• https://aws.amazon.com/interconnect/multicloud/pricing/
• https://aws.amazon.com/blogs/aws/aws-interconnect-is-now-generally-available-with-a-new-option-to-simplify-last-mile-connectivity/
• https://azure.microsoft.com/en-us/pricing/details/bandwidth/

<cite index="19-5,19-6,19-7">Distributed inference relies on frequent communication between workers, GPUs, and nodes — model sharding, prefill–decode disaggregation, KV cache movement, and cross-node coordination all introduce network overhead, and as inference becomes more distributed, networking often becomes a bottleneck rather than raw compute</cite>. <cite index="7-2,7-3">Inference pipelines have introduced an egress cost category that traditional architecture cost models were never designed to capture, and an inference request that pulls retrieval context from object storage, queries a vector database in a different zone, calls an embedding model in a separate service, and returns a response to a user has generated egress events at every step</cite>.

The architecture split between training and inference changes the network cost surface. <cite index="26-2,26-9,26-10">Inference cost is behavior-based, and inference infrastructure workloads are latency-bound, token-economics-bound, and — in agentic systems — behavior-bound</cite>. The distributed serving pattern—replicas across regions, retrieval services pulling context from object stores, agent chains calling multiple models in sequence—generates data movement at every hop.

<cite index="21-1">Cost can fluctuate based on usage patterns, data transfer between locations, and scaling needs</cite>. The unit of cost is no longer the GPU-hour. It is the token × the hop × the zone × the path. Egress emerges from behavior, not from provisioning.

Sources:

• https://bentoml.com/llm/infrastructure-and-operations/distributed-inference
• https://www.rack2cloud.com/cloud-egress-costs-explained/
• https://www.redhat.com/en/topics/ai/what-is-distributed-inference
• https://www.rack2cloud.com/inference-infrastructure-hardware-split/

<cite index="6-16,6-17">AWS, Azure, and Google Cloud charge around $0.085–$0.09/GB for internet egress</cite>, with <cite index="15-14,15-15">egress fees representing 10 to 15 percent of total cloud costs</cite>. <cite index="9-6">Azure offers the first 100 GB/month of egressed data for free to all customers in all Azure regions</cite>. <cite index="13-18">Amazon S3 gives customers 100 GB of free data transfer out to the Internet each month, aggregated across all AWS services and AWS Regions</cite>. After that threshold, the rate is consistent across the big three.

<cite index="2-2,2-5">Cloudflare charges $0.09/GB for data leaving the network to the public internet, and egress at $0.09/GB can turn a $150 inference bill into $450 when serving high-volume external users</cite>. <cite index="5-1,5-2,5-4,5-5">AWS Bedrock charges $0.09 per GB for data exceeding 100GB/month, and for applications with large token volumes and long generation lengths, this can add 5-10% to per-token costs</cite>.

<cite index="15-18">One team serving 75 TB per month found themselves paying over $6,700 per month in egress alone for just 5,000 users</cite>. The floor is known. The ceiling is the traffic envelope. Multi-region deployments multiply the surface: <cite index="16-6,16-7,16-8">A video streaming platform serving 1 PB of content monthly across Americas, Europe, and Asia pays approximately $90,000/month in AWS egress alone</cite>.

Sources:

• https://www.fluence.network/blog/egress-fees/
• https://openmetal.io/resources/blog/the-hidden-costs-of-cloud-services/
• https://markaicode.com/pricing/cloudflare-workers-pricing-breakdown/
• https://featherless.ai/blog/llm-api-pricing-comparison-2026-complete-guide-inference-costs
• https://openmetal.io/resources/blog/how-to-build-multi-region-infrastructure-across-three-continents/

<cite index="1-1,1-15">AWS charges $0.01/GB in each direction for cross-availability-zone data transfer</cite>, and <cite index="1-12">for high-traffic inference endpoints, egress often represents 10–25% of total cloud spend</cite>. The cost surface is not internet egress alone. <cite index="15-5,15-25">On AWS, data transferred between Availability Zones in the same region is billed at per-GB rates in both directions</cite>. <cite index="15-26,15-27">For applications designed with redundancy, a database in one AZ talking to an application server in another, a load balancer distributing traffic, automated backups replicating across zones: all of it generates charges</cite>.

<cite index="1-14,1-15,1-16">When inference pods run in one availability zone and the load balancer or API gateway sits in another, AWS charges $0.01/GB in each direction, and for a high-traffic endpoint with a standard multi-AZ deployment, this adds up independently of internet egress</cite>. The architecture decision—where to place the GPU, where to route the balancer—determines the bill before the first token leaves the network.

<cite index="4-1">When your inference pipeline is constantly pulling retrieval context and feature data across zones, you are paying egress rates that have no budget line</cite>. <cite index="4-8">An agent calling a retrieval service in one zone, a scoring model in another, and an output formatter in a third has turned a single user action into a distributed cost event no static budget captures</cite>. The telltale: the egress line climbs faster than the GPU line.

Sources:

• https://www.spheron.network/blog/gpu-cloud-egress-data-transfer-costs-ai-workloads-2026/
• https://dev.to/ntctech/ai-inference-is-the-new-egress-the-cost-layer-nobody-modeled-2kcp
• https://openmetal.io/resources/blog/the-hidden-costs-of-cloud-services/

<cite index="7-16,7-17">The advertised hourly GPU price is not the full cost. Data egress fees add $0.05-$0.12 per GB on most providers, which can exceed the GPU cost for data-intensive workloads that transfer large datasets or model weights.</cite> <cite index="2-33,2-34">Spheron does not charge egress fees, does not require minimum commitments, and bills per-minute. AWS, GCP, and Azure add egress, storage, networking, and reserved capacity overhead that compound the gap substantially on real workloads.</cite>

<cite index="8-3,8-4,8-5">Hyperstack: no additional charges for bandwidth or data transfer. We believe in transparent pricing with no hidden costs. All applicable costs are mentioned on our pricing page.</cite> <cite index="8-7,8-18">Hyperstack's on-demand prepaid platform bills you for each minute of use.</cite> <cite index="10-22,10-23">Crusoe: No egress charges. Per-minute billing.</cite>

The neo-clouds compete on transparency and billing granularity. The hyperscalers bundle egress into the margin. The total cost of ownership calculation for a multi-TB training run or inference workload serving weights repeatedly needs to price egress separately. A deployment transferring 10TB/month at $0.09/GB adds $900 — material when the GPU itself is $2-4/hr.

Sources:

• https://gpuperhour.com/
• https://www.spheron.network/blog/gpu-cloud-pricing-comparison-2026/
• https://www.hyperstack.cloud/gpu-pricing
• https://awesomeagents.ai/pricing/open-source-hosting-costs/

<cite index="7-2">On-demand pricing is pay-as-you-go with guaranteed availability: the instance runs until the user terminates it, with no long-term commitment.</cite> <cite index="7-12">On-demand pricing is the most common: users pay a fixed hourly rate with no commitment and can terminate at any time.</cite> <cite index="4-6">On-demand offers the most flexibility, which is crucial for development and variable workloads.</cite>

It is the baseline. All other pricing models price off the on-demand rate. <cite index="7-15">The right model depends on the workload: on-demand for development and short-lived jobs, spot for fault-tolerant batch processing, and reserved for sustained production workloads where cost predictability matters.</cite> <cite index="9-17,9-18">On-demand: Pay by the hour with no commitment. Start and stop at any time.</cite>

<cite index="5-15,5-17">RunPod offers $1.99–$2.69/GPU/hour on-demand. Lambda at $3.99/GPU/hour.</cite> <cite index="10-7">Lambda H100 at $3.29-4.29/hr.</cite> <cite index="2-11">AWS H100 at $6.88/hr, Azure H100 at $12.29/hr</cite> as of May 2026. <cite index="2-32">Spheron vs AWS/GCP/Azure: The cost gap is 40-85% across major GPU models when comparing on-demand rates.</cite> The neo-clouds price 40-85% under the hyperscalers on on-demand because they optimize for GPU density and skip the full service bundle.

Sources:

• https://gpuperhour.com/
• https://www.gmicloud.ai/blog/a-guide-to-2025-gpu-cloud-pricing-comparison
• https://getdeploying.com/guides/cheapest-gpu-cloud
• https://siliconanalysts.com/tools/cloud-pricing
• https://awesomeagents.ai/pricing/open-source-hosting-costs/
• https://www.spheron.network/blog/gpu-cloud-pricing-comparison-2026/

<cite index="2-4">Reserved pricing requires a commitment, typically 1 to 12 months, in exchange for 20-40% discounts vs on-demand.</cite> <cite index="7-4,7-14">Reserved pricing locks in a fixed rate for a term of 1-3 months, typically at 20-40% savings compared to on-demand.</cite> <cite index="2-5">AWS EC2 reserved instances, Azure reserved VMs, and GCP committed-use contracts all follow this model.</cite>

The discount trades flexibility for budget certainty. <cite index="2-8">Reserved pricing is right for: production inference running 24/7, large-scale training programs with predictable GPU-hour requirements, and teams that have validated their workload and want to lock in cost predictability.</cite> <cite index="9-22,9-23">Reservation: Commit to 1-12+ months for a lower hourly rate. Best for steady-state production workloads where you know your capacity needs.</cite>

<cite index="10-2,10-25">CoreWeave reserved H100 at ~$1.45/hr (up to 60% off on-demand).</cite> <cite index="10-15">Vultr H100 at $2.99/hr on-demand, $2.30 with 36-month prepaid.</cite> <cite index="10-11,10-12">Nebius competitive committed pricing: H100 at $2.00/hr, H200 at $2.30/hr with multi-month agreements — up to 35% savings on reservations.</cite> The longer the commit, the lower the rate. The IRR calculation depends on whether the workload holds utilization at the committed level.

Sources:

• https://www.spheron.network/blog/gpu-cloud-pricing-comparison-2026/
• https://gpuperhour.com/
• https://getdeploying.com/guides/cheapest-gpu-cloud
• https://awesomeagents.ai/pricing/open-source-hosting-costs/

<cite index="7-1,7-3">Spot (also called preemptible or interruptible) offers discounts of 50-80% compared to on-demand rates, but instances can be reclaimed by the provider when GPU demand spikes.</cite> <cite index="6-3,6-8">GCP spot prices are dynamic and can change up to once every 30 days, providing discounts of 60-91% off on-demand for most machine types and GPUs.</cite> <cite index="5-2,5-14">GCP spot H100 runs approximately $2.25/GPU/hour (60-91% discount).</cite>

The model is priced for interruptibility. <cite index="7-15">Spot is right for fault-tolerant batch processing</cite> — checkpoint-friendly workloads where the run can resume if evicted. <cite index="2-3">Supply has since normalized, but spot is still the cheapest path to short-burst capacity.</cite> <cite index="9-20,9-21">Spot offers discounted rates (often 50-80% cheaper) on spare capacity, but your instance can be interrupted with short notice — good for fault-tolerant batch workloads.</cite>

The pricing is not stable. The floor moves when provider capacity changes. <cite index="2-11">H100 spot at $1.03/hr, B200 spot at $2.12/hr on Spheron, vs AWS H100 at $6.88/hr, Azure H100 at $12.29/hr</cite> as of May 2026. The neo-cloud undercuts the hyperscaler on volatility tolerance. Spot is the place to measure actual supply slack.

Sources:

• https://gpuperhour.com/
• https://cloud.google.com/compute/gpus-pricing
• https://siliconanalysts.com/tools/cloud-pricing
• https://www.spheron.network/blog/gpu-cloud-pricing-comparison-2026/
• https://getdeploying.com/guides/cheapest-gpu-cloud

<cite index="7-1">A single V100 GPU provides approximately 600 queries per second at high concurrency, and latency of approximately 5ms at low concurrency for BERT-Base.</cite> <cite index="15-1,15-2">vLLM was consistently the fastest to generate the first token across all concurrency levels with excellent scaling characteristics. SGLang had the most stable per-token latency, consistently around 4–21ms across different loads.</cite>

<cite index="17-1,17-2">Prompt lengths sampled from Poisson distribution (mean=512 tokens); generation lengths from Poisson distribution (mean=256 tokens). Each benchmark run consisted of 1,000 requests to ensure statistical significance.</cite> <cite index="14-8,14-9">Pick a realistic prompt set using standardized datasets such as databricks-dolly-15k or ShareGPT, and set appropriate output caps. Benchmark tokens per second and decoding speed by measuring TTFT and tokens per second under rising concurrency.</cite>

<cite index="27-3">While FastAPI provides lower overhead for single-request workloads (p50 latency of 22ms), Triton achieves superior scalability through dynamic batching, delivering throughput of 780 requests per second on a single NVIDIA T4 GPU—nearly double the baseline.</cite> <cite index="23-3,23-5">Non-optimized Triton model configuration gives throughput of about 73 inferences per second. With TensorRT optimization enabled: throughput increased from 73 to 138 infer/sec at concurrency 2.</cite>

The method matters more than the framework. Controlled concurrency on real prompt distributions—ShareGPT is the baseline—with measured TTFT, per-token latency, and throughput at p50 and p95. Single-request benchmarks compress to marketing.

Sources:

• https://www.salesforce.com/blog/benchmarking-tensorrt-inference-server/
• https://www.clarifai.com/blog/comparing-sglang-vllm-and-tensorrt-llm-with-gpt-oss-120b
• https://arxiv.org/html/2511.17593v1
• https://www.hivenet.com/post/vllm-vs-tgi-vs-tensorrt-llm-vs-ollama
• https://arxiv.org/pdf/2602.00053
• https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/optimization.html

<cite index="9-4,9-6">TensorRT-LLM edges out on NVIDIA GPUs for peak throughput while vLLM shines in batching. TensorRT-LLM excels on NVIDIA hardware with low latency, while vLLM offers flexible high-throughput batching.</cite> <cite index="9-1,9-2">On DGX systems, vLLM loaded slowest (12 minutes) but delivered top speed: mean TTFT 100ms, 100/100 requests complete. SGLang/TensorRT-LLM faster startup but lower throughput.</cite>

<cite index="10-1,10-14">At high concurrency (50-100 requests), TensorRT-LLM's continuous batching can match or slightly exceed vLLM throughput if the model is pre-compiled.</cite> <cite index="11-1">TensorRT-LLM delivers 15-30% higher peak throughput on H100s compared to vLLM's 4,741 tokens/second at 100 concurrent requests.</cite>

<cite index="12-11,12-12">TensorRT-LLM was the most challenging to set up. Without quality examples, users read through TensorRT-LLM, tensorrtllm_backend and Triton Inference Server documentation, convert checkpoints, build the TRT engine, and write configurations.</cite> <cite index="10-5">Move to TensorRT-LLM when you have a model that won't change for months and you need to squeeze out every token per second at scale.</cite> <cite index="12-5">Its inherent requirement for model compilation and reliance on NVIDIA CUDA GPUs are intentional design choices that may pose limitations during deployment.</cite>

<cite index="12-6">vLLM manages to maintain a low TTFT even as user loads increase, and its ease of use can be a significant advantage.</cite> The choice compresses to this: vLLM if the model changes quarterly and deployment velocity matters. TensorRT-LLM if the weight is frozen and every basis point of GPU utilization routes to margin.

Sources:

• https://ventusserver.com/vllm-vs-tensorrt-llm-speed-benchmarks/
• https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
• https://bentoml.com/blog/benchmarking-llm-inference-backends
• https://medium.com/synthetic-futures/vllm-vs-tensorrt-llm-the-definitive-2026-comparison-for-llm-inference-ed0943fb81d2

<cite index="2-8,2-9">TensorRT is NVIDIA's high-performance deep learning inference SDK designed for GPUs. It optimizes deep learning models through layer fusion, precision calibration using FP16 or INT8, and kernel auto-tuning.</cite> <cite index="1-9,1-10">Production deployments report QPS improved by four times: OCR QPS rose from 7 to 31 and Face Comparison from 6 to 19 while using similar or smaller GPU memory.</cite> <cite index="1-11">Latency improved approximately 20% according to monitored cloud services.</cite>

<cite index="2-2,2-6">NVIDIA offers multiple solutions including TensorRT, Triton Inference Server, and Triton with TensorRT, each catering to different deployment needs.</cite> <cite index="8-2,8-3">Triton Inference Server provides a scalable, production-ready platform for model deployment. Triton handles request queuing, dynamic batching, and scheduling while supporting multiple frameworks including PyTorch, TensorFlow, and ONNX.</cite>

<cite index="6-1">Transactions per minute per instance increased by 30–35%, with a cost reduction of approximately 25% when comparing TensorRT engine performance to default PyTorch BERT.</cite> <cite index="3-4">Optimization of the core model gives speedup of the inference pipeline, up to around 30%.</cite> <cite index="5-2">Triton TensorRT might be slower than local TensorRT for single inferences due to network overheads.</cite>

The takeaway: TensorRT compresses compute; Triton abstracts it. Pick TensorRT for static single-model paths. Pick Triton when you need multi-framework routing or dynamic batching at scale.

Sources:

• https://blog.advance.ai/blog/accelerating-ai-deep-learning-models
• https://procogia.com/ai-inference-in-data-engineering-comparing-tensorrt-triton-and-triton-with-tensorrt/
• https://softwaremill.com/triton-inference-server-tips-and-tricks/
• https://aws.amazon.com/blogs/machine-learning/achieve-hyperscale-performance-for-model-serving-using-nvidia-triton-inference-server-on-amazon-sagemaker/
• https://training.continuumlabs.ai/inference/why-is-inference-important/triton-inference-server
• https://medium.com/@lepicardhugo/a-practical-guide-to-deploying-diffusion-models-with-triton-and-tensorrt-dd37b1f5a638

<cite index="13-5">mLoRA can significantly reduce average fine-tuning task completion time, e.g., by 30%, compared to state-of-the-art methods like FSDP.</cite> <cite index="13-8,13-9">Different LoRA adapters share the same base model but can be trained independently without computational dependencies; this enables mLoRA to avoid multi-GPU fine-tuning pipeline stalls by freely and concurrently scheduling distinct training stages of different fine-tuning tasks, thus eliminating pipeline bubbles.</cite>

<cite index="5-5">Fused kernel achieves up to 1.39× (1.27× on average) kernel performance improvement and can directly serve as a plug-and-play replacement in existing LoRA systems.</cite> <cite index="2-4">LoRA-FA can reduce the overall memory cost by up to 1.4× compared to LoRA.</cite>

The infrastructure read: LoRA enables batching multiple customization tasks across shared base-model layers. Pipeline parallelism improvements compress training windows. Kernel fusion cuts per-adapter overhead. This matters when the business model is hosting dozens or hundreds of adapters: the cost per adapter falls as throughput rises. <cite index="13-1">LLM platforms enable developers to fine-tune multiple models and develop various domain-specific applications simultaneously.</cite> The serving layer is the next moat after the weight files commoditize.

Sources:

• https://arxiv.org/pdf/2312.02515
• https://arxiv.org/pdf/2510.00206
• https://arxiv.org/pdf/2308.03303

<cite index="16-4,16-5">While all PEFT methods dramatically outperform the baseline, LoRA consistently achieves the highest F1 scores (0.909 on Amazon Reviews); critically, ReFT delivers nearly identical performance (~98% of LoRA's F1 score) while training only ~3% of the parameters.</cite> <cite index="21-1">RED substantially reduces the number of trainable parameters by a factor of 25,700 compared to full parameter fine-tuning, and by a factor of 32 compared to LoRA.</cite>

<cite index="20-3,20-4,20-5">Different techniques are classified as follows: Additive methods introduce new parameters into the model; Selective methods fine-tune only a subset of existing parameters; Reparameterization methods utilize low-rank representations to reduce the number of trainable parameters.</cite> LoRA falls into the reparameterization class.

<cite index="18-9">Existing studies have shown that PEFT-based tuning performs less effectively than full fine-tuning on complex tasks.</cite> <cite index="1-1,1-2">LoRA fine-tuning is best suited for behavioral and task adaptation rather than injecting large volumes of new factual knowledge; high-quality, well-structured datasets have a greater impact on LoRA performance than sheer dataset size.</cite>

The practitioner decision: if the task is adaptation not knowledge injection, LoRA holds the performance bid at a fraction of the parameter cost. Newer methods compress further but the deployment base for LoRA is wider.

Sources:

• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12705377/
• https://arxiv.org/pdf/2402.15179
• https://huggingface.co/blog/samuellimabraz/peft-methods
• https://arxiv.org/pdf/2505.22355
• https://www.digitalocean.com/community/tutorials/fine-tune-llms-with-lora-for-custom-domains

<cite index="7-5">Many teams report saving on the order of 50–70% of their fine-tuning costs by using adapters, LoRA, etc., instead of full model tuning.</cite> One deployment case: <cite index="11-9,11-11">with GPT-4, few-shot overhead was 13,000 tokens per request and cost per rewrite was $0.51.</cite> <cite index="11-12,11-14,11-15">With LoRA, there's no few-shot overhead; prompt size dropped to around 2,000 tokens with just instructions and content, and GPU infrastructure cost about $500 monthly for all 75 clients combined.</cite>

<cite index="11-1,11-2">About 0.1% of the base model parameters were trainable; training time ran 2 to 4 hours on a single GPU per adapter.</cite> <cite index="15-15">Trainable parameters: 294,912 / 124,734,720 = 0.236%.</cite> One experiment logged training for $0.28.

<cite index="1-3">LoRA adapters are lightweight and modular, making it possible to maintain multiple domain-specific behaviors using a single base model.</cite> <cite index="8-3">The efficiency gains are significant: roughly 10,000× fewer trainable parameters than full fine-tuning on large models, which is why separate LoRA adapters per skill is economically attractive.</cite> Multi-tenant serving becomes viable: one base model, adapter library routed per request. The cost structure shifts from per-tenant models to per-tenant parameter sets.

Sources:

• https://www.runpod.io/articles/guides/llm-fine-tuning-on-a-budget-top-faqs-on-adapters-lora-and-other-parameter-efficient-methods
• https://dev.to/vrathi_8/how-we-achieved-30-conversion-lift-by-moving-from-gpt-4-to-lora-adapters-35j4
• https://arie-m-prasetyo.medium.com/i-trained-a-lora-adapter-for-0-28-3ef35ff2aa37
• https://tianpan.co/blog/2026-04-19-lora-adapter-composition-production
• https://www.digitalocean.com/community/tutorials/fine-tune-llms-with-lora-for-custom-domains

<cite index="9-8">LoRA reduced trainable parameters by approximately 10,000 times when applied to GPT-3, from 175 billion to roughly 18 million, and GPU memory requirements during training by 3 times, from 1.2 terabytes to 350 gigabytes.</cite> The technique freezes the base model and trains low-rank matrices — typically rank 8 to 64 — whose product approximates the weight update.

<cite index="2-6">Full-parameter fine-tuning of a LLaMA-65B model with AdamW requires more than 1TB of GPU memory to store model parameter, gradient, and optimizer states.</cite> <cite index="3-1">Full fine-tuning of a 7-billion parameter model requires 100-120 GB of VRAM — roughly $50,000 worth of H100 GPUs for a single training run — while the same model fine-tunes on a $1,500 RTX 4090 using QLoRA.</cite>

<cite index="3-3">The approach reduces memory requirements by 10-20× compared to full fine-tuning while retaining 90-95% of quality.</cite> <cite index="9-10">After training, LoRA adapter weights can be merged with the base model weights, resulting in no additional inference latency during deployment.</cite> The weight files are compact: <cite index="7-6">resulting fine-tuned weights are often just tens of megabytes.</cite>

This matters for procurement. The GPU tier drops from H100 clusters to consumer cards. Training time compresses from days to hours. The cost curve for customization fell through.

Sources:

• https://en.wikipedia.org/wiki/LoRA_(machine_learning)
• https://arxiv.org/pdf/2308.03303
• https://introl.com/blog/fine-tuning-infrastructure-lora-qlora-peft-scale-guide-2025

<cite index="11-1,11-2,11-3">Predictive routers save time and money by skipping the audition and formulating a decision based on information gathered before inference time; IBM researchers trained a routing algorithm on benchmark data to pick out the strengths and weaknesses of each model in their library so that it could, for any given query, identify the model with the best predicted accuracy and cost</cite>. <cite index="11-4,11-9">Testing the router on Stanford's HELM benchmark, several 13-billion parameter models outperformed Meta's 70-billion parameter Llama-2 model by several percentage points</cite>. <cite index="11-11">The IBM router did slightly better than GPT-4 overall while saving 5 cents per query</cite>.

<cite index="11-13,11-14,11-15">LLMs are tested on hundreds of tasks representative of real-life work, from summarizing documents to solving math problems; HELM and most benchmarks rank models by average performance, but lost in translation is how well they do on tasks requiring specialized knowledge or skills</cite>. <cite index="6-4,6-5">Quality-based routing uses classifiers or heuristics to determine query complexity, then routes to the model most likely to produce the best response; Azure Model Router evaluates factors like query complexity, cost, and performance in real time to balance quality against budget constraints</cite>.

<cite index="11-22,11-23,11-24">Routers give you the ability to predict the right cost-performance trade-off for a given task, and that goes for latency too; because smaller models are faster than bigger models, you can define your latency target and predict how much performance you can achieve for that latency cutoff</cite>. The router learns the model's price surface across the task distribution.

Sources:

• https://research.ibm.com/blog/LLM-routers
• https://www.swfte.com/blog/intelligent-llm-routing-multi-model-ai

<cite index="1-2,1-4">Multi-model LLM routing reduces serving cost and latency by assigning each prompt to an appropriate model, but prior routing methods assume fixed latency; in real deployments, multiple models share limited GPU resources, and a model's latency depends strongly on both allocated resources and request load induced by the routing policy</cite>. <cite index="1-9,1-10">Existing approaches treat model serving as stateless API calls, ignoring that multiple models compete for compute, memory, and bandwidth; latency and throughput depend critically on resource allocation decisions such as tensor parallelism, GPU thread partitioning, and memory assignment, as well as incoming request load</cite>.

<cite index="17-4,17-5,17-6">Models can be deployed in isolation on dedicated GPUs or co-located on shared GPUs; while isolated deployment simplifies scheduling, it may lead to underutilization; co-location enables flexible resource sharing, allowing resources to be shifted toward high-demand models and improving overall utilization</cite>. <cite index="23-1,23-7,23-8">Algorithm-system co-design for cost-efficient LLM serving requires determining optimal routing strategies under latency and quality requirements, configuring model deployment across heterogeneous GPUs with appropriate resource allocation and parallelism strategies, and co-optimizing routing and deployment decisions to maximize overall system performance</cite>.

<cite index="22-1,22-2,22-14">The long-tail popularity of models and their long idle periods present opportunities to improve utilization through GPU sharing, but existing GPU sharing systems lack the ability to adjust resource allocation and sharing policies at runtime; dedicating a single or group of GPUs to each model leads to significant underutilization, as models in the popularity tail or during idle periods leave GPU resources unused</cite>. The cost model changes when the GPU is the constraint.

Sources:

• https://arxiv.org/html/2604.10907
• https://arxiv.org/pdf/2505.04021
• https://arxiv.org/pdf/2602.10729

<cite index="7-4,7-5">RouteLLM formalizes the problem of LLM routing and trains four different routers using public data from Chatbot Arena, exploring augmentation techniques to improve router performance</cite>. <cite index="8-14,8-15,8-16,8-17">Out of the box, RouteLLM supports four routers trained on GPT-4-1106-preview and Mixtral 8x7B: matrix factorization (mf), weighted Elo calculation (sw_ranking), BERT classifier, and causal LLM classifier</cite>. <cite index="15-20,15-21,15-25">Four router types are provided: mf uses matrix factorization trained on preference data and is recommended for most use cases</cite>.

<cite index="12-7,12-8">Routers trained on Arena and LLM-judge-labeled datasets with BERT, matrix factorization, and SW ranking approaches performed best on MT Bench, achieving almost 50% fewer GPT-4 calls to reach 80% performance compared to random baseline</cite>. <cite index="12-11,12-35">On MMLU and GSM8K datasets, routers trained only on Arena data performed poorly compared to random baseline because the datasets are completely different</cite>. <cite index="11-18,11-19">Training a router on benchmark data that more closely resembles the target task produces better results—knowing how an LLM performed on a similar task in the past gives a good idea of how it will perform in the future</cite>.

<cite index="15-28,15-29,15-34">The Chatbot Arena dataset includes over 55,000 real-world user and LLM conversations with user preferences across over 70 LLMs; setting the strong-model percentage to 50% during calibration does not mean 50% of actual queries route to the strong model, but that 50% of calibration dataset prompts would</cite>. The router compresses differently per workload.

Sources:

• https://www.lmsys.org/blog/2024-07-01-routellm/
• https://github.com/lm-sys/routellm
• https://zilliz.com/learn/routellm-open-source-framework-for-navigate-cost-quality-trade-offs-in-llm-deployment
• https://research.ibm.com/blog/LLM-routers
• https://www.pondhouse-data.com/blog/saving-costs-with-llm-routing

<cite index="7-1,7-6">RouteLLM demonstrated cost reductions of over 85% on MT Bench, 45% on MMLU, and 35% on GSM8K compared to using only GPT-4, while maintaining 95% of GPT-4 performance</cite>. <cite index="7-13,7-18">The framework routes between GPT-4 Turbo as the strong model and Mixtral 8x7B as the weak model, with the matrix factorization router achieving 95% of GPT-4 performance using only 26% GPT-4 calls—approximately 48% cheaper than a random baseline</cite>.

<cite index="16-2,16-7">The cost delta is structural: GPT-4 averages $24.7 per million tokens versus $0.24 for Mixtral 8x7B</cite>, a 100× spread. <cite index="8-1,8-5">Trained routers reduce costs by up to 85% while maintaining 95% GPT-4 performance on MT Bench, and benchmarks show these routers achieve the same performance as commercial offerings while being over 40% cheaper</cite>. <cite index="9-3,9-4">Unify AI's best router scored 8.76 on MT Bench with 45.6% GPT-4 calls; RouteLLM's best-performing router matched that score with 25.4% GPT-4 calls</cite>.

<cite index="10-1,10-18">Anyscale's LLM routers achieved up to 70% cost reduction on MT Bench, 30% on MMLU, and 40% on GSM8K compared to always-GPT-4 baselines</cite>. The cost floor depends on the benchmark: <cite index="15-1,15-12">cost savings potential ranges between 30% and 80%, depending on use case</cite>. The model layer compresses when the weak-model tier can handle most of the load.

Sources:

• https://www.lmsys.org/blog/2024-07-01-routellm/
• https://github.com/lm-sys/routellm
• https://github.com/lm-sys/RouteLLM/tree/main/benchmarks
• https://www.anyscale.com/blog/building-an-llm-router-for-high-quality-and-cost-effective-responses
• https://www.pondhouse-data.com/blog/saving-costs-with-llm-routing
• https://vivekpandit.medium.com/routellm-optimizing-the-cost-quality-trade-off-in-large-language-model-deployment-c48b7abb2cfa

<cite index="23-1">By formulating an M/G/1 model, we observe that enforcing a maximum output token limit on a very small fraction of inference requests can significantly reduce the queueing delay, and our model facilitates the selection of the optimal limit.</cite> <cite index="23-9,23-10,23-11">The heavy tail of output token length in a few requests significantly extends the average queuing delay. This can cause a considerable percentage of impatient users to leave the LLM platform before their requests are processed. We propose configuring an appropriate maximum output token limit, as a slight reduction in inference quality for a very small percentage of requests can significantly decrease the average queuing time.</cite>

<cite index="4-1">Limiting the maximum output token size effectively reduces queuing delays while maintaining user satisfaction.</cite> The mathematical tradeoff is explicit: clip a small percentage at the tail to compress the service-time distribution, which lowers the second moment and collapses mean queue delay under M/G/1.

<cite index="23-2,23-3">For the batch inference, we model the service process as a bulk queue in which the batch processing time is affected by the batch size and the maximum token size inside this batch jointly. The queueing delays of the batching of all buffered requests (dynamic batching), the batching of constant number of requests (fixed batching), and the batching without intra-batch waiting (elastic batching) are derived.</cite> <cite index="4-9">Elastic batching consistently offered lower delays across various distributions of token lengths, reinforcing its practical value in an LLM inference context.</cite>

Sources:

• https://arxiv.org/pdf/2407.05347
• https://www.themoonlight.io/en/review/a-queueing-theoretic-perspective-on-low-latency-llm-inference-with-variable-token-length

<cite index="11-1,11-2,11-3">Authors classify serving systems into: Decode Prioritizing: Batch together requests ("request-level batching") and first perform Prefill for all requests, and then perform Decode. The batch is considered completed once all requests are done decoding. While a batch is being processed, any new requests received by the system are not scheduled for Prefill.</cite> <cite index="11-4,11-5">Such scheduling optimizes for latency (as measured by time-between-tokens or TBT) because new requests do not affect executing of ongoing requests. However this impacts throughput because new requests will have to wait for the longest ongoing request to finish.</cite>

<cite index="15-10,15-11,15-12">Orca introduced iteration-level batching wherein requests can dynamically enter or exit a batch at the granularity of individual iterations. Iteration-level batching improves throughput by avoiding inefficiencies of request-level batching systems. Orca and several other recent systems like vLLM combine iteration-level batching with prefillprioritizing scheduling wherein they eagerly schedule the prefill phase of one or more requests first i.e., whenever GPU memory becomes available.</cite> <cite index="15-13">This way, prefill-prioritizing schedulers have better throughput because computing prefills first allows subsequent decodes to operate at high batch sizes.</cite>

<cite index="15-4,15-5">Sarathi-Serve introduces chunked-prefills which splits a prefill request into near equal sized chunks and creates stall-free schedules that adds new requests in a batch without pausing ongoing decodes. Stall-free scheduling unlocks the opportunity to improve throughput with large batch sizes while minimizing the effect of batching on latency.</cite>

Sources:

• https://www.abhimanyutalwar.com/paper_summaries/20240729_sarathi.html
• https://arxiv.org/pdf/2403.02310

<cite index="3-1,3-9">Observing that the inference latency of an individual request is proportional to its output token length, we model the service process as an M/G/1 queue and derive the queuing delay in closed form.</cite> <cite index="2-14">We model this dynamic batching service process with an unbounded batch size as an M/G/1 queue, where the service time distribution is correlated with both the arrival rate and the output token length distribution.</cite>

<cite index="1-2">The latency of a request consists of two components: the queuing time and the service time.</cite> <cite index="1-4">Assumption 3.1 is a standard assumption in queueing theory, and it is well-suited for LLM inference systems since requests are typically generated by users in a random manner.</cite> <cite index="6-4">Enforcing a maximum output token limit on a very small fraction of inference requests can significantly reduce the queueing delay, and our model facilitates the selection of the optimal limit.</cite>

<cite index="3-10">The heavy tail of output token length in a few requests significantly extends the average queuing delay.</cite> <cite index="2-15">We explicitly derive a tight upper bound for the average queuing delay.</cite> <cite index="18-1">Under a first-in, first-out (FIFO) service discipline, the system operates as an M/G/1queue, and the mean system time depends on the first and second moments of the resulting service-time distribution.</cite> The M/G/1 formulation captures the variability in service time that drives tail latency in production serving.

Sources:

• https://arxiv.org/html/2407.05347v1
• https://arxiv.org/pdf/2407.05347
• https://openreview.net/pdf?id=WVmarX0RNd
• https://arxiv.org/pdf/2601.10274

<cite index="3-4">Batch size affects both the waiting time for requests to be grouped and the batch inference time.</cite> <cite index="16-2">Decoding latency, measured as time between tokens (TBT), increases with batch size due to the higher computational cost, caused by the enlarged matrix dimensions in the matrix multiplication operations required for larger batches.</cite> The batching tradeoff is structural: <cite index="13-1">satisfying tight SLOs requires restricting batch sizes, which reduces throughput, increases congestion, and ultimately degrades overall SLO attainment across request categories.</cite>

<cite index="3-3">Token lengths of all requests are padded to match the maximum length of the current batch before being fed into the self-attention module, causing that their inference times are uniform.</cite> That padding pins the compute cost to the longest sequence in the batch. <cite index="2-1,2-13">The inference latency of a batch depends not only on the batch size but also on the largest output token length among all the requests in the batch.</cite>

<cite index="16-3,16-4">The memory constraint stems from the KV cache, a dynamic data structure storing intermediate attention states. The KV cache size scales linearly with batch size and sequence length, establishing a hard capacity limit for concurrent requests.</cite> <cite index="16-5,16-6,16-7">Current inference serving systems, such as vLLM, employ static batching, requiring operators to preset a fixed maximum batch size. This approach forces suboptimal tradeoffs. Undersized batches waste GPU capacity through low utilization, while oversized batches risk memory overflows during demand spikes or long-sequence workloads.</cite>

Sources:

• https://arxiv.org/html/2407.05347v1
• https://arxiv.org/pdf/2407.05347
• https://arxiv.org/pdf/2503.05248

<cite index="2-25,2-26,2-27">Using lower-precision integer formats has a significant impact on HBM when compared to the standard bfloat16 baseline: bfloat16 (standard) is the baseline size (e.g., a 7B model requires ~14 GB); 8-bit precision halves the model size (e.g., 14 GB becomes ~7 GB); 4-bit precision reduces the model size by a factor of 4 (e.g., 14 GB becomes ~3.5 GB)</cite>. <cite index="2-28">The reduction in size lets you fit much larger models into memory with minimal degradation in performance</cite>.

<cite index="22-14,22-15,22-16,22-17">While the A100 relies on FP16 or BF16 precision for most workloads, the H100 introduces native FP8 support; this allows the GPU to process data using 8-bit floating-point numbers without a significant loss in model accuracy; the impact on memory and compute is massive; by using FP8, you effectively double the throughput compared to FP16 because you are moving half the data across the memory bus for the same number of parameters</cite>.

The tradeoff is not free. <cite index="4-12">Quantization reduces model weights from 16-bit to 8-bit or 4-bit precision, halving or quartering memory requirements with acceptable performance trade-offs</cite>, but the "acceptable" threshold depends on the task. Code verification tolerates less quantization error than summarization. The price floor on serving a quantized 70B FP8 model is now half the HBM capacity versus the FP16 baseline, which changes which chip tiers can economically serve the model.

Sources:

• https://cloud.google.com/blog/topics/developers-practitioners/decoding-high-bandwidth-memory-a-practical-guide-to-gpu-memory-for-fine-tuning-ai-models/
• https://lyceum.technology/magazine/a100-vs-h100-for-llm-inference/
• https://introl.com/blog/ai-memory-supercycle-hbm-2026

<cite index="5-25">In August 2022, Nvidia announced that its "Hopper" H100 GPU would ship with five active HBM3 sites (out of six on board) offering 80 GB of RAM and 3 TB/s of memory bandwidth (16 GB and 600 GB/s per site)</cite>. <cite index="5-26,5-27">On 30 May 2023, SK Hynix unveiled its HBM3E memory with 8 Gbit/s/pin data processing speed (25% faster than HBM3); at 8 GT/s with 1024-bit bus, its bandwidth per stack is increased from 819.2 GB/s as in HBM3 to 1 TB/s</cite>.

<cite index="10-4,10-5">The successor, HBM4, doubles the I/O width per stack from 1,024 bits to 2,048 bits and ships on the Rubin R100 in H2 2026; HBM4e has been spec'd by JEDEC but is not announced on any GPU yet</cite>. <cite index="6-4">The 2 terabytes per second bandwidth of HBM4 compared to HBM3's 819 gigabytes per second represents a 2.4x improvement; combined with capacity increases from 36 gigabytes to 64 gigabytes per stack, HBM4 addresses both bandwidth and capacity dimensions of the memory wall</cite>.

<cite index="1-19">Despite the rapid rise of custom ASICs, Nvidia will still command the lion's share of HBM demand in 2027, driven by its aggressive roadmap, where Rubin Ultra alone pushes per GPU capacity to 1 TB</cite>. The announced roadmap for HBM4 is H2 2026; the volume ramp lags the spec by two to three quarters in every prior generation.

Sources:

• https://en.wikipedia.org/wiki/High_Bandwidth_Memory
• https://www.spheron.network/blog/hbm3e-vs-hbm4-vs-hbm4e-llm-inference-guide/
• https://introl.com/blog/hbm-evolution-hbm3-hbm3e-hbm4-memory-ai-gpu-2025
• https://newsletter.semianalysis.com/p/scaling-the-memory-wall-the-rise-and-roadmap-of-hbm

<cite index="11-10,11-11">In 16-bit precision (FP16 or BF16), each parameter occupies 2 bytes; a 7B model requires 14GB just to load the weights into VRAM</cite>. <cite index="11-13,11-14">When using an optimizer like AdamW, which stores two states per parameter (momentum and variance, typically also in FP32), the optimizer states require an additional 12 bytes per parameter; when you combine weights (2 bytes), gradients (2 bytes), and optimizer states (12 bytes), you are looking at 16 bytes per parameter before even considering activations</cite>.

<cite index="3-14,3-15,3-16">Activations are the intermediate outputs of each layer, computed during the forward pass and stored for use during the backward pass; activation memory consumption is highly dynamic and depends on the batch size, sequence length, model hidden size, and number of layers; for large models and long sequences, activations often consume the largest portion of GPU RAM</cite>. <cite index="3-20">Techniques like activation checkpointing (or gradient checkpointing) are commonly used to mitigate this by recomputing activations during the backward pass instead of storing them, trading increased computation time for reduced memory usage</cite>.

<cite index="10-9,10-10,10-11,10-12">With a 70B FP16 model, that means reading approximately 140 GB of weights per token; the time this takes is fixed by bandwidth, not FLOPS; no amount of additional FLOPS can push tokens-per-second above the bandwidth ceiling at batch size 1</cite>. Serving economics depend on the bandwidth number, not the FLOPS spec.

Sources:

• https://lyceum.technology/magazine/gpu-memory-requirements-transformer/
• https://apxml.com/courses/how-to-build-a-large-language-model/chapter-18-hardware-considerations-llm-training/memory-requirements-hbm-gpu-ram
• https://www.spheron.network/blog/hbm3e-vs-hbm4-vs-hbm4e-llm-inference-guide/

<cite index="4-4">A 70B model in FP16 precision requires approximately 2GB per billion parameters</cite>, putting the static footprint at 140 GB. <cite index="19-4">The A100 80GB ships with HBM2e delivering 1.6 TB/s; the H100 80GB ships with HBM3 delivering 3 TB/s</cite>. <cite index="26-4">The H200 upgrades to six stacks of 24 GB HBM3e (141 GB total) versus five stacks of 16 GB HBM3 on the H100 (80 GB)</cite>, meaning the H200 allows a 70B model to load with headroom for KV cache while the H100 80GB variant does not.

<cite index="4-5,4-6,4-7">The KV cache presents an additional memory challenge; during inference, transformers store key-value pairs from previous tokens to avoid recomputation, and this cache grows linearly with context length, consuming approximately 0.5MB per token in a 7B model</cite>. <cite index="8-20">Decode is bandwidth-bound and places heavy demands on memory capacity via the KV cache</cite>. For a 70B model serving at long context, the capacity constraint binds before the model even starts.

<cite index="1-1,1-3">Each generational bump in HBM capacity and throughput — whether 80 GB at 3 TB/s on H100 or 192 GB at 8 TB/s on GB200 — quickly encourages designers to increase parameter counts, context lengths, and KVCache footprints, nullifying the headroom that seemed ample only months earlier; the mere presence of larger, faster HBM does not yield sustained slack; instead it resets the baseline for "reasonable" model size</cite>.

Sources:

• https://introl.com/blog/ai-memory-supercycle-hbm-2026
• https://www.trgdatacenters.com/resource/h100-vs-a100/
• https://www.runpod.io/articles/guides/nvidia-h200-gpu
• https://newsletter.semianalysis.com/p/scaling-the-memory-wall-the-rise-and-roadmap-of-hbm
• https://www.datacenterknowledge.com/data-center-hardware/scaling-the-memory-wall-hbm-cxl-and-the-new-gpu-playbook

<cite index="9-3,9-4">CO₂ emitted per kilowatt-hour on a grid varies by time of day and substantially by location; networked warehouse-scale computers emit more carbon than needed if operated without regard to these variations</cite>. <cite index="17-3">Google's Carbon-Intelligent Compute Management actively minimizes electricity-based carbon footprint and infrastructure costs by delaying temporally flexible workloads</cite>.

<cite index="9-6">The system uses analytical pipelines to gather next-day carbon intensity forecasts, train demand prediction models, and generate carbon-aware Virtual Capacity Curves for all datacenter clusters</cite>. <cite index="17-9,17-10">VCC values tend to be smaller when local grid carbon intensity is expected to be high, reducing total compute and power usage by delaying execution of flexible tasks to later times of day</cite>. <cite index="9-12">Data from operation shows VCCs effectively limit hourly capacity when the grid's energy supply mix is carbon intensive and delay execution to greener times</cite>.

<cite index="13-2,13-3,13-4">Electricity demand fluctuates throughout the day; during peak demand, fossil fuel plants are brought online, increasing grid carbon intensity, while off-peak hours see cleaner baseload power dominating</cite>. <cite index="10-5,10-6">For datacenters in Pacific Northwest and Montana, low LME stability implies little gain from load shifting, but in Sunbelt regions the correlation between LME and solar strength creates reliable opportunities to lower carbon footprint via carbon-aware operation</cite>.

Training runs that can tolerate delay compress carbon cost without capex. The cost curve on carbon-aware scheduling is already live.

Sources:

• https://www.researchgate.net/publication/360434183_Carbon-Aware_Computing_for_Datacenters
• https://arxiv.org/pdf/2106.11750
• https://arxiv.org/html/2512.18819v1
• https://esg.sustainability-directory.com/term/grid-carbon-intensity/

<cite index="2-1">Training GPT-3 emitted roughly 500 metric tons of carbon dioxide, equivalent to driving a car New York to San Francisco about 438 times</cite>. <cite index="4-1">Training a handful of AI models can emit over 626,000 pounds of CO₂, nearly five times the lifetime emissions of the average American car</cite>. <cite index="5-1,5-3">In 2024, global datacenters emitted 182 million tons of CO₂ tied to 460 TWh of electricity, implying average carbon intensity of approximately 395.65 gCO₂/kWh</cite>. <cite index="5-4">AI growth is projected to add 24–44 million metric tons of CO₂ annually by 2030</cite>.

<cite index="8-1,8-2">A large portion of LLM energy consumption occurs during training, and because training is not location-dependent, it can be carried out in regions with abundant, low-cost, low-carbon electricity</cite>. <cite index="17-1,17-2">CO₂ emitted per kilowatt-hour varies by time of day and substantially by location; hyperscale compute emits more carbon than needed if operated without regard to these variations</cite>.

<cite index="4-5,4-6">One datacenter consumes the same energy as around 50,000 homes, and most energy is not used for computation but consumed by cooling and backup systems</cite>, which is where efficiency gains compress first. The carbon cost of training is amortized across millions of inference calls, but the siting decision locks in the emissions multiplier for the training run. Operators routing training to low-intensity grids undercut those who do not by the CO₂ delta times the regulatory or contractual carbon price.

Sources:

• https://www.climateimpact.com/news-insights/insights/carbon-footprint-of-ai/
• https://www.frontiersin.org/journals/sustainability/articles/10.3389/frsus.2024.1507030/full
• https://blog.ansi.org/ansi/ai-data-centers-carbon-water-energy-impact/
• https://www.carbon-direct.com/insights/understanding-the-carbon-footprint-of-ai-and-how-to-reduce-it
• https://arxiv.org/pdf/2106.11750

<cite index="21-2,21-3">Greenfield datacenters in attainment areas can emit up to 250 tons per year before triggering major New Source Review, but in nonattainment zones the threshold drops to 100 tpy or lower depending on severity</cite>. <cite index="21-4">Projects exceeding the major source threshold in nonattainment areas must purchase Emission Reduction Credits, which are expensive and unavailable in many locations</cite>.

<cite index="19-9,19-10">Siting in a non-attainment zone drops emission thresholds substantially, making NNSR and Title V permits time-consuming, expensive, and subject to stringent ongoing compliance; assessing permit requirements is now essential in site selection</cite>. <cite index="25-3,25-4">Air modeling has become a regulatory focus during due diligence; proximity to environmental justice communities or nonattainment areas can flip approval to denial</cite>.

<cite index="6-1,6-6">Siting and permitting guidelines are steering datacenters toward locations with available grid capacity, low-carbon generation, water availability, and lower community impact</cite>. <cite index="6-3,6-8">Virginia legislators have considered requirements around energy and water reporting, backup generator emissions, and protections preventing non-datacenter customers from subsidizing datacenter energy costs</cite>. <cite index="24-3,24-4,24-5">EPA redefined "begin actual construction" in September 2025, allowing core-and-shell work before NSR permit issuance, but no emissions units or related piping can be installed until the air permit clears</cite>.

The constraint is measurable and priced into the site-selection decision tree.

Sources:

• https://trinityconsultants.com/resources/powering-the-next-generation-of-data-centers-navigating-due-diligence-and-air-permitting-for-data-center-development/
• https://encinoenviron.com/minimizing-data-center-emissions-for-simplifying-permitting/
• https://www.designnews.com/artificial-intelligence/will-carbon-emissions-put-constraints-on-ai-data-centers
• https://www.erm.com/insights/future-proofing-data-centers-insights-from-20-years-of-air-permitting-leadership/
• https://www.jw.com/news/insights-epa-cleanairact-resource-hub/

<cite index="7-1">Regional disparities are substantial: Google's Singapore datacenter drew 4% of energy from carbon-free sources in 2020, while Finland reached 94%</cite>, a factor that compounds directly into training economics. <cite index="9-1">Grid carbon intensity ranges from near-zero in hydroelectric or nuclear regions to over 600 g CO₂e / kWh in coal grids</cite>.

<cite index="3-3,3-7">The methodology problem: operational carbon should use location-based and time-specific marginal emissions per energy unit, because training on cloud instances is the dominant AI workload path</cite>. <cite index="14-1,14-5">The same workload produces drastically different carbon footprints depending on region, driven by the mix of renewable, nuclear, and fossil sources</cite>.

<cite index="10-1,10-2">In the U.S., Pacific Northwest and Montana regions hold the lowest locational marginal emissions (LME), making them attractive for large consumers trying to price in carbon risk</cite>. <cite index="10-3">California and Sunbelt regions show sun-correlated variability, creating temporal arbitrage paths via load-shifting</cite>. <cite index="1-9">Tools now exist to automatically schedule jobs to reduce carbon footprint by leveraging time and geographic differences in intensity</cite>.

<cite index="12-3,12-4,12-6">The crossroads: datacenter carbon is a function of both energy consumption and carbon intensity, but demand growth is overshadowing grid decarbonization over the next five years</cite>. Location choice is no longer aesthetic.

Sources:

• https://arxiv.org/pdf/2311.03615
• https://dl.acm.org/doi/fullHtml/10.1145/3531146.3533234
• https://arxiv.org/pdf/2206.05229
• https://www.researchgate.net/publication/360434183_Carbon-Aware_Computing_for_Datacenters
• https://arxiv.org/html/2512.18819v1
• https://energy.acm.org/eir/data-centers-carbon-emissions-at-crossroads-an-empirical-study/
• https://greenagile.org/metrics-and-impact/architecture-and-infrastructure/grid-carbon-intensity

<cite index="11-9,11-11">High-availability and load-balanced setup uses multiple model instances with load balancing and backup nodes to provide fault tolerance and stable performance. Ensure scalability and high availability through load balancing and auto-scaling in cloud infrastructure.</cite> <cite index="13-2,13-3">Adding more servers provides better fault tolerance and can be more cost-effective for variable workloads. However, larger instances with more powerful GPUs often provide better per-request economics for compute-intensive models.</cite>

<cite index="13-6,13-7">Tiered serving uses different infrastructure tiers for different service levels. Critical, low-latency requests might use dedicated high-performance instances, while batch processing can use spot instances or shared resources.</cite> The SLA requirement dictates the redundancy layer. 99.9% can route to spot capacity during off-peak. 99.99% cannot.

<cite index="8-1,8-2">Serving infrastructure handles traffic spikes, deploys model updates, and ensures uptime. It typically includes containerization, container orchestration, load balancers. Kubernetes management of containerized models allows auto-scaling, load balancing, and high availability.</cite> Each redundancy layer — container orchestrator, load balancer, multi-region replication — adds to the bill. The cost is not the second GPU. The cost is the second GPU plus the failover logic plus the health-check latency plus the ops burden to keep it synchronized.

Sources:

• https://keymakr.com/blog/llm-deployment-complete-guide-to-production-ready-model-serving/
• https://dev.to/matt_frank_usa/model-serving-infrastructure-building-scalable-inference-2gf1
• https://www.mirantis.com/blog/model-deployment-and-orchestration-the-definitive-guide/

<cite index="2-1">Monitoring LLM serving differs from traditional API monitoring because the metrics that matter are model-specific: token throughput, KV cache pressure, and generation latency.</cite> <cite index="2-2">Prometheus and Grafana dashboards should track time-to-first-token (TTFT) p99, inter-token latency (ITL), KV cache utilization, request queue depth, GPU utilization, GPU memory usage, and aggregate tokens/sec throughput.</cite>

<cite index="2-4">Alert when p99 TTFT exceeds your SLA threshold sustained for 5 minutes, or when KV cache utilization exceeds 85% — a leading indicator of queue buildup.</cite> An HTTP 200 returned in 200ms does not tell you if the queue is about to fall over. <cite index="1-6">Downtime or latency during model serving, training, or inference does not just stall a build; it wastes valuable GPU compute cycles and delays release of AI-based applications.</cite>

The standard web-service SLA framework — uptime percentage, response time, error rate — abstracts away the failure modes that actually break production inference. KV cache pressure builds before requests time out. Token throughput collapses before the endpoint goes down. If the monitoring stack measures HTTP uptime instead of generation-path health, the cost is already incurred by the time the SLA breach shows up.

Sources:

• https://www.runpod.io/articles/guides/ai-model-serving-architecture-building-scalable-inference-apis-for-production-applications
• https://jfrog.com/blog/improve-resilience-with-9999-sla/

<cite index="4-1">The vendor sells availability — a request-level promise that the API answered in time. The product team consumes capability — a request-level promise that the answer was usable.</cite> <cite index="4-2">The two are not the same metric. The team that confuses them is one quiet model bump away from learning the difference.</cite>

<cite index="4-3">The vendor cannot promise that the model serving your region's failover is identical to your primary because the operational reality of running multi-region inference includes capacity-driven heterogeneity.</cite> A 99.99% SLA may return a 200 response inside the latency window using a smaller fallback model or a different quantization tier. <cite index="4-4,4-5">The functional-availability gap is structural, not commercial. The vendor is selling infrastructure that has an irreducibly application-specific reliability story.</cite>

This matters for inference cost modeling. If the failover model is FP8 instead of FP16, or a 7B instead of 70B, the response latency holds but the output quality does not. The product breaks. The SLA does not. The procurement team prices based on uptime. The cost of the real failure — the silent model swap — is invisible until it routes production traffic.

Sources:

• https://tianpan.co/blog/2026-05-13-vendor-sla-gap-uptime-vs-functional-availability

<cite index="3-1,3-13">The industry standard sits at 99.9% uptime — 8 hours 45 minutes of downtime per year.</cite> <cite index="3-2,3-14">Leading AI platforms aim for 99.99%, allowing 52 minutes annually.</cite> <cite index="3-16">The Big Three cloud providers averaged 99.97% in 2024.</cite>

The infrastructure cost to move from three nines to four is not linear. <cite index="21-1">99.99% requires multi-region deployment, load balancing, automated failover, and 24/7 monitoring — operational cost runs 2-3x the cost of 99.9% due to redundancy and staffing.</cite> <cite index="22-3,22-18">The 8x difference in reliability between 99.9% and 99.99% comes with proportionally higher infrastructure costs.</cite> <cite index="25-2">The cost jump from three nines to four nines is typically 3-5x infrastructure spending.</cite>

<cite index="23-1,23-2">Going from 99.9% to 99.99% does not mean spending 0.09% more. It means redundant databases, multi-region failover, automated health checks, load balancers that work, and on-call engineers paged at 3 AM.</cite> <cite index="21-1">Five nines uptime costs 5-10x the cost of 99.9%; it requires massive investment in redundancy.</cite> Every nine is a 10x downtime reduction. The percentages make it look incremental. The cost is exponential.

Sources:

• https://blog.prodia.com/post/evaluate-inference-slas-and-uptime-guarantees-in-4-steps
• https://www.uptimedock.com/blog/what-does-99-uptime-do
• https://atomping.com/blog/what-does-99-9-uptime-mean/
• https://dev.to/tyson_cung/the-nines-are-lying-to-you-what-999-uptime-actually-costs-31j0
• https://ecosire.com/tools/uptime-calculator

<cite index="22-6,22-38">Price per token matters less than price per correct output.</cite> <cite index="19-10,19-11">More expensive models generally deliver higher quality; Claude Opus 4.6 scores 100 at $25/1M output, DeepSeek V3.2 scores 79 at $0.28/1M—the right choice depends on quality requirements.</cite> <cite index="22-17,22-18">Claude Opus 4 at $15/$75 per million makes sense only where quality differences directly impact revenue—legal analysis, complex code, research synthesis, agentic workflows where errors cascade; for most applications Sonnet 4 handles the work at 80% quality for 20% cost.</cite>

<cite index="17-9">LLM API pricing varies by more than 600× across models, making it the single most consequential variable in AI infrastructure cost.</cite> <cite index="22-43,25-42">LLM API prices dropped ~10× over the past two years; trending down 30–50% per year since 2023.</cite> The revenue model for providers: compress the absolute floor (DeepSeek V3.2 at $0.14/$0.28, Gemini Flash, Llama hosted at $0.05–$0.90), but hold margin on the quality premium. Enterprises optimize on task-level unit economics—routing simple queries to the floor, reserving premium for the 10% where correctness has direct revenue impact. The elasticity stays near one because buyers substitute within intelligence tiers, not abandon inference when price falls.

Sources:

• https://costgoat.com/compare/llm-api
• https://pecollective.com/blog/llm-pricing-comparison-2026/
• https://www.cloudzero.com/blog/llm-api-pricing-comparison/

<cite index="1-7,4-11">OpenAI and Anthropic offer 50% batch API discounts for non-real-time workloads; combine batch with prompt caching and per-call cost drops to ~25% of on-demand rate.</cite> <cite index="1-19,18-3,18-26">Smart routing delivers 60–80% cost reduction with minimal quality impact for most applications.</cite> <cite index="25-2,25-23">Anthropic caching: 90% off cached input tokens—a long system prompt costing $3.00/1M on Sonnet 4.6 drops to $0.30/1M on cache hits; 25% write surcharge pays for itself after a few requests.</cite>

<cite index="24-22,24-23">Single most impactful cost optimization is routing queries by complexity: typical enterprise distribution routes 70% to budget models (Haiku, GPT-4.1 Nano, Gemini Flash), 20% to mid-tier (Sonnet, GPT-4o), 10% to premium (Opus, o1).</cite> <cite index="17-23,17-27">GPT-5 pricing spans $0.20 to $30/MTok input—150× range within one provider; model selection alone can turn a $1,200/month workload into $100/month.</cite> The discount levers are architectural (routing, caching prefix matches, deferring async to batch). Price elasticity near one means each 10% cut gains 10% volume—revenue holds. But cost elasticity from discount stacking lets you serve the same workload at 20–30% of list without waiting for the next model drop.

Sources:

• https://zenvanriel.com/ai-engineer-blog/llm-api-cost-comparison-2026/
• https://www.cloudzero.com/blog/llm-api-pricing-comparison/
• https://iternal.ai/calculators/llm-pricing-calculator
• https://pecollective.com/blog/llm-api-pricing-comparison/

<cite index="12-1">Training cost–inference pricing elasticity estimated at β = 0.432; the 63× U.S.–China training cost gap originates from architectural innovation (MoE), not factor price differentials.</cite> The arXiv paper on tiered Super-Moore's Law frames the token as a non-storable, non-transferable digital good—different from traditional software. <cite index="24-10,24-11,24-12">LLM API prices dropped ~80% between early 2025 and early 2026; GPT-4o input fell from $5.00 to $2.50 per million tokens; deploying AI at enterprise scale has never been lower cost.</cite>

The elasticity mechanism: architectural leaps (Mixture-of-Experts activates subset of parameters, lowers per-token compute) compress the cost curve faster than fabrication or power cost deltas. <cite index="20-1,20-15">MoE models reduce compute cost per request, allowing providers to offer lower per-token rates.</cite> Price cuts propagate through the stack not because of cheaper chips, but because fewer active parameters serve each token. The β = 0.432 tells you that a 10% training cost drop translates to ~4% inference price cut—enough to compress margins but not enough to collapse revenue when demand elasticity hovers near unity.

Sources:

• https://arxiv.org/pdf/2603.28576
• https://iternal.ai/calculators/llm-pricing-calculator

<cite index="3-2,11-2">Price elasticities measured just above one for model–provider combinations in OpenRouter data, confirmed by Microsoft Azure aggregates.</cite> The Fradkin analysis used scraped OpenRouter marketplace data—developers routing calls across models and providers—plus Azure firm-level API usage. <cite index="3-22">State-of-the-art 2023 models fell 1000× in price by late 2025.</cite> <cite index="3-23">Average price paid per token held relatively constant, consistent with demand shifting to superior intelligence tiers.</cite> <cite index="3-25">Open-source models priced ~90% cheaper than closed-source at the same intelligence level, yet consumed <30% token share.</cite>

The elasticity read: a 10% price drop drives ~10% volume increase, not the runaway demand spiral implied by Jevons' Paradox. Revenue stays roughly flat under cuts. Differentiation—horizontal across use cases, vertical across intelligence tiers—holds the bid. <cite index="3-30">No single model dominates across applications.</cite> The marketplace structure matters: <cite index="3-13,3-16,3-18">LLM inference measured in metered token usage; inference providers operate compute clusters serving subsets of models; the same LLM may be served by multiple providers.</cite> Price falls through, but demand isn't elastic enough to compensate revenue in the short run.

Sources:

• https://andreyfradkin.com/assets/LLM_Demand_12_12_2025.pdf

<cite index="16-20,16-21,16-22,16-23">The three-stage lifecycle framework: Years 1-2 support foundational model training, Years 3-4 support high-value real-time inference, Years 5-6 support batch inference and analytics workloads—training requires peak performance, inference tolerates lower latency constraints, batch/analytics operate at the long tail</cite>. <cite index="4-20,4-21,4-22,4-23">CoreWeave reported its five-year-old A100s remain "fully booked" at rental rates down 70% from 2024 peaks but decisively non-zero; Google's VP of AI infrastructure stated that 7-8 year old TPUs maintain "100% utilization"; Azure's retirement data shows K80-class fleets lasted 9 years in production (2014-2023), and P100s served 7 years (2016-2023), supporting an economic life of 5-7 years when cascades are managed effectively</cite>.

<cite index="9-6,9-7,9-9,9-10,9-11">Under US GAAP, economic obsolescence shows up through impairment; when there's a triggering event (new generation launch, utilization collapse, market price crash), companies test for recoverability—if carrying value exceeds future cash flows, you write down the asset; the P&L sees cliffs, not slopes, and volatility is hidden until it explodes</cite>. <cite index="4-27,4-28,4-30">GAAP requires (Cost - Salvage Value) / Useful Life; the interaction between the Estimated Useful Life assumption and the assumed Salvage Value is where the true economics—and the potential for earnings management—reside; with capex in the tens (sometimes hundreds) of billions, extending a server life from 4→6 years shifts billions of P&L, even before impairment testing</cite>.

Sources:

• https://siliconangle.com/2025/11/22/resetting-gpu-depreciation-ai-factories-bend-dont-break-useful-life-assumptions/
• https://www.stanleylaman.com/signals-and-noise/gpus-how-long-do-they-really-last
• https://davefriedman.substack.com/p/the-gpu-risk-no-one-is-managing

<cite index="8-29,8-30,14-15,14-16">Nvidia now releases new AI chips on an annual basis, versus the two-year cadence it had before; AMD followed suit</cite>. <cite index="11-17,11-18">Nvidia's shift to annual product cadence—Hopper (2022), Blackwell (2024), Rubin (2026), Rubin Ultra (2027)—creates genuine 2-3 year frontier obsolescence; Blackwell offers up to 25x better energy efficiency than Hopper for specific AI inference workloads</cite>. <cite index="14-24,14-26">Microsoft CEO Satya Nadella said Nvidia's pace increased in terms of migrations, and "I didn't want to go get stuck with four or five years of depreciation on one generation"</cite>. <cite index="8-24,8-25,8-26">Nvidia CEO Jensen Huang joked in March that "when Blackwell starts shipping in volume, you couldn't give Hoppers away"</cite>.

<cite index="18-9,18-10,18-11">Technical analyses have converged on estimating the useful lifespan of AI chips at one to three years; one unnamed Google architect assessed that GPUs running at 60-70% utilization—standard for AI workloads—survive one to two years, with three years as a maximum, because thermal and electrical stress is too high</cite>. <cite index="11-20">Meta's Llama 3 405B training study documented 148 GPU failures out of 419 total disruptions across 16,384 H100 GPUs over 54 days, implying an annualized failure rate of approximately 9% under heavy utilization</cite>. <cite index="20-2">Based on H100's depreciation curve (~25% loss over 3 years), H200 is expected to lose 20–30% of value over 36 months; Blackwell shipping availability will accelerate the curve</cite>.

Sources:

• https://www.cnbc.com/2025/11/14/ai-gpu-depreciation-coreweave-nvidia-michael-burry.html
• https://www.stanleylaman.com/signals-and-noise/gpus-how-long-do-they-really-last
• https://blog.citp.princeton.edu/2025/10/15/lifespan-of-ai-chips-the-300-billion-question/
• https://www.mercatus-ai.com/blog/h200-price

<cite index="2-22,2-23,2-24">CoreWeave adopted a six-year depreciation posture despite an exclusively AI-intensive focus; Nebius uses four years, Lambda Labs five years—shorter cycles reflecting faster modernization and possibly less heterogeneous workload mix</cite>. <cite index="8-15,8-17,8-18">CoreWeave has used six-year cycles since 2023; CEO Michael Intrator reported that A100 chips from 2020 are all fully booked, and H100s from 2022 rebooked immediately at 95% of their original price when a contract expired</cite>.

<cite index="2-11,16-7">SiliconANGLE research believes rapid innovation cycles will compress depreciation from the current six years to a more conservative five-year timeframe, and hyperscalers will ultimately converge on a five-year cycle—shorter than today's six-year model but still supported by extended economic usefulness</cite>. <cite index="4-13">Component depreciation—segmenting rapidly obsolescing GPU modules (3.5-4.5 year useful life, realistic 30-35% salvage value) from longer-lived infrastructure such as chassis, networking, power, and cooling (6-8 year useful life)—is explicitly permitted under GAAP ASC 360</cite>. <cite index="3-3,3-4,3-5,3-6">Hyperscalers have more than $100 billion in assets classified as construction in progress; according to GAAP, these balances are not depreciated until the assets are placed in service, meaning today's depreciation expense reflects past investment cycles and may not fully reflect the current wave of AI-driven infrastructure spending</cite>.

Sources:

• https://siliconangle.com/2025/11/22/resetting-gpu-depreciation-ai-factories-bend-dont-break-useful-life-assumptions/
• https://www.cnbc.com/2025/11/14/ai-gpu-depreciation-coreweave-nvidia-michael-burry.html
• https://www.stanleylaman.com/signals-and-noise/gpus-how-long-do-they-really-last
• https://natlawreview.com/article/deep-quarry-useful-lives-gpus-key-considerations

<cite index="2-5,2-8">Amazon extended server depreciation from three to four years in January 2020, and by 2023 the big three hyperscalers all normalized on six-year schedules</cite>. <cite index="10-6">AWS servers are on a five-year cycle; networking runs six</cite>. <cite index="4-33">Microsoft moved 4→6, Google 4→6, Meta 4→4.5→5.0→5.5, Oracle 4→5→6</cite>. <cite index="8-32,14-18">Amazon reversed in February 2025, shortening a subset of servers from six to five years citing "an increased pace of technology development, particularly in the area of artificial intelligence and machine learning"</cite>.

<cite index="1-19,1-20">Under GAAP ASC 360-10-35-4, useful life is an accounting estimate—management's judgment about the period an asset provides economic benefits—and depreciation is "a process of allocation, not of valuation"</cite>. <cite index="3-12,3-13">Changes in depreciable lives are treated as changes in accounting estimates under ASC 250, not corrections of errors; management revises when new information becomes available</cite>. <cite index="3-15,3-16">Useful life assessments are company-specific under GAAP, meaning two companies can arrive at different, yet compliant, estimates when operating similar hardware</cite>.

<cite index="12-9">Michael Burry estimated that if cloud providers shortened GPU lives from four-to-six-year schedules to two-to-three-year cycles, the cumulative impact could exceed $176 billion for 2026–2028</cite>. <cite index="15-4,15-5,15-6">If you depreciate $900k over three years, $300k per year hits the income statement; over six years, only $150k—same cash out, same hardware, but under a six-year schedule, GAAP operating income looks much higher in the early years</cite>.

Sources:

• https://deepquarry.substack.com/p/depreciation-of-gpus-between-useful
• https://siliconangle.com/2025/11/22/resetting-gpu-depreciation-ai-factories-bend-dont-break-useful-life-assumptions/
• https://natlawreview.com/article/deep-quarry-useful-lives-gpus-key-considerations
• https://www.stanleylaman.com/signals-and-noise/gpus-how-long-do-they-really-last
• https://www.levelheadedinvesting.com/p/are-ai-chips-useful-lives-creating-useless-earnings

<cite index="22-1,22-2">Few realize that PUE is defined by ISO/IEC 30134-2:2018, and that reference to anything other than this standardized KPI is not genuine PUE. PUE was originally defined by The Green Grid but this organization passed ownership, development, standardization, and dissemination to ISO/IEC JTC1 SC39 WG1.</cite> <cite index="24-7">The ISO/IEC 30134-2:2026 standard explicitly provides guidance to improve consistency and comparability, including measurement categories and treatment of special cases like on-site generation and unmeasured energy.</cite>

<cite index="26-6,26-7">The European EN50600-4 standards are almost identical to the ISO/IEC 30134 standards. PUE, REF, ERF, CER, CUE, WUE are metrics focused on datacenter infrastructure; ITEEsv and ITEUsv are metrics for IT efficiency.</cite> <cite index="23-1,23-2,23-3">PUE is gaining strategic importance from a stakeholder perspective. Operators use it as an operational control parameter and KPI in energy management systems. Investors are integrating PUE values into ESG due diligence and financing models like green bonds.</cite>

<cite index="17-1,17-2">The ISO/IEC 30134-2 standard should be used to correctly measure and devise plans for energy reduction in a continuous manner. The ISO/IEC 30134 series includes other relevant datacenter resource utilization KPIs, and users should be aware of limitations such as using it to compare one facility to another without full understanding of the circumstances.</cite>

Sources:

• https://www.future-tech.co.uk/introduction-the-iso-iec-30134-series-of-standardised-kpis/
• https://www.score-grp.com/en/post/data-center-pue-in-2026-understanding-measuring-and-improving-power-usage-effectiveness
• https://datacenter-jedi.com/standards/
• https://blog.aquatherm.de/en/data-centre-pue-energy-efficiency-indicator
• https://www.linkedin.com/pulse/data-center-resource-efficiency-pue-isoiec-30134-james-soh-%E8%8B%8F%E6%97%AD%E6%B1%9F

<cite index="9-10">Most AI datacenter specs aim for PUE lower than 1.3.</cite> <cite index="12-1,12-3,12-4,12-5">Google's Finland datacenter achieves PUE 1.09. A 1,000-GPU H100 facility at PUE 1.67 draws 6.68 MW total; at PUE 1.09 the same facility uses 4.36 MW, saving 2.32 MW and $2 million annually while freeing capacity for 580 additional GPUs.</cite>

<cite index="11-4,11-5,11-6">PUE is no longer sufficient on its own. AI-era datacenters increasingly require rack-level and workload-aligned efficiency metrics, such as power per training job, energy per inference request, or utilization-adjusted efficiency indicators. These granular measurements enable operators to align infrastructure efficiency with actual AI productivity rather than relying solely on facility-level averages that can obscure localized inefficiencies.</cite>

<cite index="10-3">OCP sustainability guidance emphasizes that PUE is widely used but can be challenging to compare across sites due to variability in measurement boundaries and operating conditions.</cite> <cite index="10-4,10-5">A more complete sustainability picture includes water usage effectiveness (WUE) and carbon usage effectiveness (CUE). The most sustainable design is often the one that optimizes power, water, and carbon together, not PUE in isolation.</cite>

<cite index="11-10,11-11">High-performance GPUs operate at high utilization for extended periods, creating sustained power draw. AI training amplifies this with continuous consumption; inference introduces rapid variability as demand fluctuates.</cite>

Sources:

• https://newsletter.semianalysis.com/p/ai-datacenter-energy-dilemma-race
• https://introl.com/blog/pue-109-google-data-center-efficiency-strategies
• https://www.ctrls.com/blogs-data-center-power-infrastructure-in-ai-age/
• https://semiengineering.com/ai-gpu-and-hpc-data-centers-the-infrastructure-behind-modern-ai/

<cite index="5-3">Accuracy in the IT load is one of the major factors affecting PUE measurement, as server utilization has an important effect on IT energy consumption and hence the overall PUE value.</cite> <cite index="5-4">A datacenter with high PUE and high server utilization could be more efficient than one with low PUE and low server utilization.</cite>

<cite index="24-5,24-6">Most PUE disputes are not about math — they are about scope. Two sites can report different PUEs because they draw the boundary differently (what counts as total facility energy, how on-site generation is treated, whether office space is included).</cite> <cite index="17-7">Facility operators and large datacenter users have promoted low PUE beyond its intended purpose as an energy efficiency tool and more as a comparison metric.</cite>

<cite index="8-12">PUE should be utilized as an improvement metric, not a comparison metric.</cite> <cite index="3-13,3-14,3-15">Measurement requires metering energy at the facility's utility meter. If the datacenter is in a mixed-use building, measure only the meter that powers the datacenter or estimate and remove the non-datacenter portion.</cite> <cite index="2-6">Despite the simplicity of the ratio and acceptance as a standard metric, calculating PUE is not as straightforward as the formula seems.</cite>

Sources:

• https://en.wikipedia.org/wiki/Power_usage_effectiveness
• https://www.techtarget.com/searchdatacenter/definition/power-usage-effectiveness-PUE
• https://www.score-grp.com/en/post/data-center-pue-in-2026-understanding-measuring-and-improving-power-usage-effectiveness
• https://www.linkedin.com/pulse/data-center-resource-efficiency-pue-isoiec-30134-james-soh-%E8%8B%8F%E6%97%AD%E6%B1%9F
• https://www.cadence.com/en_US/home/explore/data-center-pue.html
• https://www.vertiv.com/en-emea/about/news-and-events/articles/educational-articles/what-is-pue-power-usage-effectiveness-and-what-does-it-measure/

<cite index="1-2,3-4">PUE is the ratio of total facility power to IT equipment power.</cite> <cite index="3-5">It is expressed as a ratio, with efficiency improving as the quotient approaches 1.0.</cite> <cite index="8-9">A PUE of 1.0 is perfect.</cite> <cite index="2-4">Total facility energy includes cooling systems, lighting, and power delivery components.</cite> <cite index="2-5">IT equipment energy encompasses compute, storage, networking, and control equipment like KVM switches.</cite>

<cite index="1-12">The Green Grid introduced PUE in 2007.</cite> <cite index="19-1,8-11">It was published as a global standard under ISO/IEC 30134-2:2016.</cite> <cite index="18-2">The standard defines three measurement tiers: PUE1 (monthly), PUE2 (daily), and PUE3 (continuous 15-minute intervals).</cite> <cite index="23-9,23-13,23-16">ISO/IEC 30134-2 also defines four measurement categories based on where IT power is recorded: PUE0 (estimate), PUE1 (behind UPS), PUE2 (behind PDUs), and PUE3 (at IT equipment).</cite>

<cite index="7-9">Uptime Institute data shows average PUE at 1.59.</cite> <cite index="9-9">Typical colocation facilities run 1.5–1.6; hyperscalers run below 1.4.</cite> <cite index="9-10">AI datacenters target below 1.3.</cite> <cite index="12-1">Google's Finland datacenter achieved 1.09.</cite> At that level, overhead is 9%. At industry average, overhead is 59%.

Sources:

• https://cove.inc/blog/what-is-power-usage-effectiveness-pue-data-center-efficiency/
• https://www.vertiv.com/en-emea/about/news-and-events/articles/educational-articles/what-is-pue-power-usage-effectiveness-and-what-does-it-measure/
• https://www.techtarget.com/searchdatacenter/definition/power-usage-effectiveness-PUE
• https://en.wikipedia.org/wiki/Power_usage_effectiveness
• https://www.nlyte.com/blog/data-center-energy-efficiency-pue-dcie/
• https://www.cadence.com/en_US/home/explore/data-center-pue.html
• https://newsletter.semianalysis.com/p/ai-datacenter-energy-dilemma-race
• https://introl.com/blog/pue-109-google-data-center-efficiency-strategies
• https://blog.aquatherm.de/en/data-centre-pue-energy-efficiency-indicator
• https://www.belden.com/blog/data-center/pue-standards-for-energy-efficiency

<cite index="23-1,23-2,23-4">Model labs can backfill idle capacity with training runs, research ablations, evaluations, and offline batch inference. When real-time demand drops at 3 AM or weekends, the GPUs switch to research or training jobs. The cost of the fleet gets amortized across all of these workloads.</cite> <cite index="23-5,23-6,23-7">Large inference platforms serve dozens of models, so demand peaks for one model can coincide with troughs for another, smoothing utilization across the fleet. They can also offer discounted offline/overnight batch inference tiers to fill remaining gaps.</cite>

<cite index="23-9,23-18,23-19">Enterprise self-hosting is least flexible. A company running one or two models for themselves has the narrowest workload mix.</cite> The GPU sits idle when internal demand falls. This is structural.

<cite index="14-2,14-7">Cloud H100 prices stabilized at $2.85–$3.50/hour after 64–75% decline from peaks.</cite> <cite index="14-9">Self-hosted breakeven requires 50%+ GPU utilization for 7B models, 10%+ for 13B models.</cite> The crossover depends on whether you can hold utilization above that floor. <cite index="11-10,11-12,11-13">The decision between API access and self-hosting determines whether you pay per token or per GPU hour—and the economics flip at high volumes. Use API-based below 50M tokens/month; evaluate self-hosting above that threshold.</cite>

The IRR on self-hosted GPUs depends on amortization, not just list price.

Sources:

• https://mlechner.substack.com/p/the-economics-of-llm-inference-batch
• https://introl.com/blog/inference-unit-economics-true-cost-per-million-tokens-guide
• https://myengineeringpath.dev/tools/llm-pricing-comparison/

<cite index="5-1,5-2,5-3">Prefill processes the entire input prompt and produces the first output token; it has high latency but saturates GPU compute due to parallel processing. Decode generates the rest of output tokens one-at-a-time; it has low latency but also low compute utilization because a decode iteration processes only a single token per request.</cite>

<cite index="5-4,5-5">Batching is highly effective for decodes and consequently for overall throughput, but batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency.</cite> The Sarathi-Serve paper from USENIX addresses this. <cite index="3-4,3-5">Sarathi-Serve introduces chunked-prefills which splits a prefill request into near equal sized chunks and creates stall-free schedules that add new requests in a batch without pausing ongoing decodes. Stall-free scheduling unlocks the opportunity to improve throughput with large batch sizes while minimizing the effect of batching on latency.</cite>

<cite index="14-13,14-14">Output token pricing asymmetry reflects actual costs. OpenAI, Anthropic, and Google price output tokens 3–5× higher than input tokens because output generation requires sequential processing while input processing parallelizes efficiently.</cite> Applications generating long outputs face different economics than those processing long inputs with brief responses.

Measurement: time-to-first-token (TTFT) for prefill; time-between-tokens (TBT) for decode. Cost per million tokens must specify the prefill/decode ratio.

Sources:

• https://arxiv.org/html/2403.02310v2
• https://arxiv.org/pdf/2403.02310
• https://introl.com/blog/inference-unit-economics-true-cost-per-million-tokens-guide

<cite index="9-1,9-2">Batch size dramatically affects per-token costs through amortization of fixed overheads. Serving single requests wastes 90% of GPU capacity on memory transfers.</cite> <cite index="9-3">Batching 32 requests together reduces per-token costs by 85% while increasing latency.</cite>

<cite index="21-1,21-5">Compute costs decline with batching because matrix operations become more efficient at larger batch sizes, amortizing overhead across more tokens.</cite> <cite index="18-4">While the most efficient way would be to process all requests simultaneously at each step to amortize the cost of loading model weights to GPUs, this is infeasible due to the large memory demand of attention operators.</cite> The KV cache is the binding constraint. <cite index="8-3,8-4">The KV cache size scales linearly with batch size and sequence length, establishing a hard capacity limit for concurrent requests.</cite>

Continuous batching breaks the wait-for-batch-completion penalty. <cite index="17-6,17-8">Continuous batching dynamically batches tokens across concurrent sequences during autoregressive generation, allowing the system to achieve much higher token throughput by merging different user requests at each decoding step.</cite> <cite index="17-10">Anyscale reported using continuous batching enabled up to 23× more throughput in LLM inference compared to naive per-request processing.</cite>

Measurement methodology: cost per token = (GPU hour cost) / (tokens per hour at batch size N). The denominator changes by 10–20× between batch size 1 and batch size 128.

Sources:

• https://introl.com/blog/cost-per-token-llm-inference-optimization
• https://introl.com/blog/inference-unit-economics-true-cost-per-million-tokens-guide
• https://arxiv.org/pdf/2503.05248
• https://www.rohan-paul.com/p/reducing-llm-inference-costs-while
• https://arxiv.org/html/2411.07447v3

<cite index="4-4,4-6">Batch size determines where you sit on the tradeoff between latency and throughput. A batch size of 1 means low latency but idle GPU capacity; a batch size of 256 means high throughput and low cost per request, but higher latency because each request shares the GPU with 255 others.</cite> <cite index="4-14">There is no free lunch: you cannot have both minimum latency and maximum throughput on the same hardware.</cite>

<cite index="6-1">With one A100, maximizing throughput with batch size 64 increases latency by 4× while throughput increases by 14×.</cite> <cite index="8-2">Decoding latency (time between tokens) increases with batch size due to higher computational cost from enlarged matrix dimensions in matrix multiplication operations.</cite> The tradeoff is not linear. <cite index="4-10,4-11">Where a provider sets batch size is a business decision as much as a technical one, and the tradeoff is not linear—you land at different points on a concave curve.</cite>

<cite index="4-15">In practice, many providers offer the same model at two tiers: a cheap, high-batch service for workloads that can tolerate slower response latency.</cite> This is price discrimination on the cost curve, not on the model.

Measurement: <cite index="6-7,6-8">Each line on the throughput-vs-latency curve is obtained by increasing batch size from 1 to 256; this is useful in determining how large you can make the batch size subject to different latency constraints.</cite>

Sources:

• https://mlechner.substack.com/p/the-economics-of-llm-inference-batch
• https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices

<cite index="2-5">Recalculate quarterly, because cloud pricing and usage patterns shift faster than annual budgets can track.</cite> <cite index="2-20,2-21">Cloud TCO is not a single formula. It is a framework that needs quarterly recalibration, because cloud costs shift faster than most finance models anticipate.</cite>

<cite index="2-23,2-24">Document every workload, its hosting model, resource needs, and actual utilization patterns. If you cannot measure utilization today, you cannot forecast cloud costs accurately.</cite> <cite index="13-17,13-18,13-19">Update models quarterly based on actual costs. Track variances between projected and actual expenses. Most organizations discover their models improve significantly after one year of operational data.</cite>

<cite index="3-10">The maturation of Large Language Models has introduced a new paradigm of sustained, high-throughput inference that challenges the financial viability of public cloud "rent-seeking" models.</cite> <cite index="3-13">The architectural leap from Hopper to Blackwell improves inference throughput significantly, fundamentally altering TCO calculations by compressing the physical footprint required for massive models.</cite>

The unit economics shift underneath you every quarter. <cite index="13-6">TCO models must now account for rapid depreciation as Blackwell GB200/GB300 systems reach market, and potential sub-$2/hr H100 rentals by mid-2026.</cite> The model is the moat for one more release. The cost curve is the moat after that.

Sources:

• https://www.cloudzero.com/blog/cloud-tco/
• https://lenovopress.lenovo.com/lp2368-on-premise-vs-cloud-generative-ai-total-cost-of-ownership-2026-edition
• https://introl.com/blog/gpu-infrastructure-tco-5-year-cost-model

<cite index="6-6">Cost per token is an enterprise's all-in cost to produce each delivered token, usually represented as cost per million tokens.</cite> <cite index="6-9,6-10">Cost per token determines whether enterprises can profitably scale AI. It's the one TCO metric that directly accounts for hardware performance, software optimization, ecosystem support and real-world utilization.</cite>

<cite index="6-13">For cloud deployments, cost per GPU per hour is the hourly rate paid to a cloud provider; for on-premises, it's the effective hourly cost derived from amortizing owned infrastructure.</cite> <cite index="6-14">The real key to reducing token cost lies in the denominator: maximizing the delivered token output.</cite>

<cite index="3-12">The "Token Economics" framework analyzes amortized cost-per-million-tokens to provide a direct comparison between owning infrastructure and consuming intelligence via APIs.</cite> <cite index="3-5,3-6">On-premises infrastructure achieves a breakeven point in under four months for high-utilization workloads. Owning infrastructure yields up to an 18x cost advantage per million tokens compared to Model-as-a-Service APIs over a five-year lifecycle.</cite>

<cite index="5-5">With GPU prices stabilizing and cloud AI API costs averaging $15-60 per million tokens, the total cost of ownership calculation has become complex.</cite> Serving cost per million tokens collapses the infrastructure question to one metric. The variance floor in that metric is the moat.

Sources:

• https://blogs.nvidia.com/blog/lowest-token-cost-ai-factories/
• https://lenovopress.lenovo.com/lp2368-on-premise-vs-cloud-generative-ai-total-cost-of-ownership-2026-edition
• https://www.swfte.com/blog/cloud-vs-onprem-ai-tco-analysis

<cite index="1-14">For AI workloads, power consumption can exceed hardware cost over a 3-year period.</cite> <cite index="1-11,1-12">A thorough TCO analysis captures costs often overlooked: power consumption that can exceed hardware cost over three years, cooling requirements that scale with compute density, and personnel costs for managing complex AI systems.</cite>

<cite index="2-4">Cloud TCO includes five categories: direct infrastructure, operational costs, migration costs, people costs, and a waste factor of 27–35% for organizations without mature optimization.</cite> <cite index="2-36">Vendor TCO calculators do not account for multi-cloud complexity, and none include operational and people costs that make up 30–40% of real TCO.</cite>

<cite index="4-12,4-13">Most TCO whitepapers limit analysis to infrastructure costs—primarily compute hardware—and power and cooling expenses. This simplification allows for focused comparison, though real-world deployments may incur additional costs outside the scope of the model.</cite> <cite index="10-1,10-13">TCO components divide into initial Capital Expenses and ongoing Operational Expenses.</cite>

The model assumes clean power input and functional cooling. At density, both become constraints before rack space does. <cite index="14-7">A single B200 server can draw 10kW, more than an entire rack of traditional servers. High-density deployments often require specialized cooling solutions.</cite>

Sources:

• https://slyd.com/resources/tco-calculator
• https://www.cloudzero.com/blog/cloud-tco/
• https://lenovopress.lenovo.com/lp2225.pdf
• https://apxml.com/courses/planning-optimizing-ai-infrastructure/chapter-6-cost-management-and-optimization/analyzing-on-premise-tco

<cite index="1-16,1-17">On-premises infrastructure typically crosses breakeven versus cloud at 60%+ sustained utilization. The crossover occurs at 7–14 months for workloads at 90%+ utilization, 14–24 months at 60–80%, and 24–36+ months at 40–60%.</cite> <cite index="1-17">Below 40% utilization, cloud is often more economical.</cite>

<cite index="2-8">For GPU-intensive AI workloads, on-premise reaches cost parity within 12 months at sustained 24/7 loads.</cite> <cite index="12-1,12-6">Lenovo's TCO analysis places the crossover at approximately 8,556 hours of continuous use—roughly 12 months of 24/7 operation.</cite> <cite index="15-3,15-4,15-5">Industry consensus places breakeven at 50–60% GPU utilization. Below 50%, cloud is cheaper. Above 60%, owning wins.</cite>

The cost curve inverts at those thresholds because on-prem costs are largely fixed. <cite index="12-3,12-4">An 8-GPU H100 node with a 3-year TCO of $675,000 costs the same whether it runs at 30% or 95% utilization. The cost-per-useful-GPU-hour scales inversely with utilization.</cite> <cite index="5-12,5-13">Cloud TCO is dominated by linear API cost: unlike on-prem, there is no declining cost curve as you amortize hardware. Every month costs roughly the same.</cite>

<cite index="5-15,5-16">The on-prem cost curve is front-loaded: Year 1 carries the hardware investment, while Years 2 and 3 see dramatically lower costs as the infrastructure is amortized. This is the fundamental economic advantage of on-prem at scale.</cite>

Sources:

• https://slyd.com/resources/tco-calculator
• https://www.cloudzero.com/blog/cloud-tco/
• https://www.vamsitalkstech.com/opinion/gpu-economics-building-the-business-case-for-on-premise-vs-cloud-gpu-infrastructure/
• https://siliconanalysts.com/for-procurement
• https://www.swfte.com/blog/cloud-vs-onprem-ai-tco-analysis

<cite index="11-1,11-2,11-3">Reliably measuring abstract phenomena like 'safety' and 'robustness' requires strong construct validity—having measures that represent what matters. A systematic review of 445 LLM benchmarks finds patterns related to measured phenomena, tasks, and scoring metrics that undermine validity.</cite> <cite index="11-7">If a benchmark has high construct validity in measuring 'intelligence', then a model that does well is intelligent; if construct validity is low, a high score may be irrelevant or misleading.</cite>

<cite index="18-6,18-7">Neither latent factor models nor neural scaling laws are satisfactory for construct validity. Social science models ignore scaling laws, and the capabilities they extract often proxy model size.</cite> The methodological issue: benchmarks collapse into a covariate of parameter count and training compute, not a measure of economically relevant task completion. A benchmark that correlates 0.9 with model size tells procurement nothing that the spec sheet does not already provide.

Sources:

• https://arxiv.org/pdf/2511.04703
• https://arxiv.org/pdf/2602.15532
• https://t-redactyl.io/posts/2026-01-19-llm-benchmarks-have-issues-with-validity/

<cite index="21-1,21-2,21-4">Benchmark contamination refers to the presence of test datasets in LLM pre-training or post-training data. Contamination leads to inflated scores, compromising evaluation results. Almost all models tested show signs of contamination with almost all benchmarks.</cite> <cite index="22-3,22-4">String matching methods (n-gram overlap) are insufficient; simple variations like paraphrasing or translation bypass decontamination. A 13B model can easily overfit a test benchmark and achieve performance on par with GPT-4.</cite>

<cite index="22-7,22-8">In pre-training sets like RedPajama-Data-1T and StarCoder-Data, 8-18% of the HumanEval benchmark overlaps. Contamination also appears in synthetic datasets generated by GPT-3.5/4, suggesting unintentional risk.</cite> This is not a rounding error. The benchmark may already be in the weights. Published scores cannot be taken at face value without adversarial hold-out validation.

Sources:

• https://arxiv.org/pdf/2410.16186
• https://arxiv.org/pdf/2311.04850
• https://arxiv.org/pdf/2406.04244

<cite index="4-11,4-12,4-13">MMLU yields variable performance after minor changes such as re-ordering answer choices, changing formatting, or rewording the task. Re-ordering answer choices leads to a significant reduction in accuracy—most models drop around 10%, some drop up to 27%.</cite> <cite index="13-6,13-7">While rankings of LLMs remain relatively stable, absolute performance drops significantly when questions are paraphrased. Benchmarks may provide a reasonable comparative measure, but they overestimate absolute performance and generalization abilities.</cite>

<cite index="9-3,9-13">Experiments on 41 representative LLMs reveal that models are far from robust in commonsense reasoning tasks, with poor performance on question variants and significant gaps compared to human performance.</cite> The unit of precision claimed—down to the decimal—does not survive contact with trivial reformatting. Stewards pricing model upgrades should discount published benchmark deltas by at least 15% to account for prompt brittleness.

Sources:

• https://arxiv.org/pdf/2504.07825
• https://arxiv.org/pdf/2509.04013
• https://arxiv.org/pdf/2502.11393

<cite index="3-3,3-9">Both MMLU and HellaSwag have saturated for frontier models at 95%+.</cite> <cite index="3-10">The benchmarks remain useful as baseline checks for smaller models and fine-tuned variants</cite>, but they no longer separate the top tier.

<cite index="20-3,20-4">Nearly half of benchmarks exhibit saturation, with rates increasing as benchmarks age. Hiding test data shows no protective effect, while expert-curated benchmarks resist saturation better than crowdsourced ones.</cite> <cite index="6-4,6-6">MMLU-Pro introduces a 10-option multiple-choice format to increase difficulty and discriminate performance at the higher end.</cite>

The economics: when every frontier release lands within two points on a benchmark, the metric loses pricing power. Buyers fall back to cost-per-token and latency. Benchmark deltas below 3% do not justify switching costs in production deployments. What used to guide model selection now compresses into a tie, and the actual decision moves to serving economics.

Sources:

• https://www.lxt.ai/blog/llm-benchmarks/
• https://www.researchgate.net/publication/382633331_Investigating_Data_Contamination_in_Modern_Benchmarks_for_Large_Language_Models
• https://www.dailydoseofds.com/llmops-crash-course-part-10/

<cite index="13-5,13-6">FLOPS is the key metric for measuring GPU compute performance; H100 triples FP64 FLOPS of A100, delivering 60 teraflops of FP64 computing for HPC</cite>. <cite index="13-11,13-12">Even with exceptional FLOPS, memory-bound workloads won't fully utilize compute capacity; H100's over 3 TB/sec memory bandwidth helps but remains a consideration for large models</cite>. <cite index="13-15,13-16">Maximum FLOPS require tensor-optimized operations; generic compute may not achieve peak performance figures</cite>.

<cite index="15-16">H100 SXM preliminary performance specs: Peak FP64 30 TFLOPS, Peak FP64 Tensor Core 60 TFLOPS, Peak FP32 60 TFLOPS, Peak FP16 120 TFLOPS, Peak BF16 120 TFLOPS, Peak TF32 Tensor Core 500 TFLOPS | 1000 TFLOPS with sparsity</cite>. <cite index="9-21">H100 features fourth-generation tensor cores up to 6× faster chip-to-chip compared to A100, with 2× the MMA computational rates on equivalent data types and 4× the rate using new FP8</cite>.

<cite index="11-1,11-2">H100 delivers about 3× the compute throughput of A100 at every precision level, from combination of more CUDA cores (16,896 vs 6,912), faster fourth-generation tensor cores, and architectural improvements in Hopper</cite>. The published TFLOPS ceiling holds only when the bottleneck is compute, not bandwidth. For inference at batch size one or small prefill, memory bandwidth binds and the gap narrows. The cost model has to account for both.

Sources:

• https://jarvislabs.ai/ai-faqs/what-is-the-flops-performance-of-the-nvidia-h100-gpu
• https://www.advancedclustering.com/wp-content/uploads/2022/03/gtc22-whitepaper-hopper.pdf
• https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/
• https://www.e2enetworks.com/blog/nvidia-a100-vs-h100-vs-h200-gpu-comparison

<cite index="5-8">Volta tensor cores were first generation and supported only FP16 as input data type</cite>. <cite index="5-9">Turing tensor cores added INT8, INT4, and binary</cite>. <cite index="5-10">Third-generation Ampere tensor cores introduced acceleration for sparse matrix multiplication with fine-grained structured sparsity and bfloat16 (BF16)</cite>. <cite index="5-12,5-13">Volta and Turing had eight tensor cores per SM, each performing 4×4×4 matrix multiply; Ampere has four tensor cores per SM, each performing 8×4×8 matrix multiply</cite>.

<cite index="1-1">H100 fourth-generation tensor cores support 8×4×16 matrices at FP16, delivering 1,024 FP16 FLOPS per clock per tensor core; H100 has 4 tensor cores per SM and 108 SMs per GPU</cite>. <cite index="2-11,2-13,2-16">H100/H200/B200 tensor cores exhibit identical numerical characteristics for FP16/BF16 inner product operations</cite>.

<cite index="7-1">Design supports low-precision multiplication in FP16/BF16/FP8/BF8/INT8/INT4 formats and higher-precision accumulation in FP32/INT32</cite>. <cite index="7-5,7-7">Inputs (matrices A and B) use lower precision like FP16/INT8 whereas addend matrix C and resultant matrix D use higher precision like FP32/INT32</cite>. <cite index="7-8,7-9">Due to MMA operation fusion, intermediate results don't need storage in register file, reducing power consumption and memory traffic; this enables packing more compute capability into same die area, improving throughput per mm² and throughput per watt</cite>. The architectural delta from Ampere to Hopper is the shift that reprices the FLOPS/watt curve.

Sources:

• https://arxiv.org/pdf/2206.02874
• https://www.glennklockwood.com/garden/tensor-cores
• https://arxiv.org/html/2512.07004v1
• https://arxiv.org/pdf/2512.00053

<cite index="4-1,4-3,4-4">BF16 exponent field matches FP32, so BF16 has similar dynamic range but lower precision; that makes it very tolerant to large and small values compared with FP16, which has a smaller exponent</cite>. Both formats occupy 2 bytes. Both run at the same TFLOPS ceiling on A100 and H100 tensor cores. The difference shows in numerical stability, not throughput.

<cite index="11-5,11-6,11-7,11-8">Standard A100 mixed-precision training uses BF16 or FP16: master weights stay in FP32 for stability, forward and backward passes run in 16-bit precision, delivering 312 TFLOPS versus 19.5 TFLOPS pure FP32; BF16 is generally preferred over FP16 because it has the same exponent range as FP32, meaning no loss scaling needed to prevent gradient underflow</cite>. <cite index="11-9">FP16 requires careful loss scaling because smaller dynamic range can cause very small gradients to round to zero</cite>.

<cite index="4-5,4-6,4-7">BF16 should remain default mixed precision format because widely supported and usually behaves close to FP32; FP8 is focused optimization for Hopper generation GPUs where matrix multiplication throughput and memory limits dominate total training cost; standardize training recipes on BF16 first, then introduce FP8 gradually on well characterized layers</cite>. FP8 adoption is a cost-curve call, not a capability unlock.

Sources:

• https://acecloud.ai/blog/fp8-vs-bf16-mixed-precision-tensor-cores/
• https://www.e2enetworks.com/blog/nvidia-a100-vs-h100-vs-h200-gpu-comparison

<cite index="6-1,6-11">Tensor cores are specialized hardware units designed solely for executing matrix multiplications</cite>, and <cite index="6-2,6-10">they require specific data types: FP16, BFloat16, TF32</cite>. The FLOPS spread between precision formats determines the unit economics of inference.

<cite index="11-7">A100 delivers 312 TFLOPS at FP16/BF16 versus 19.5 TFLOPS at FP32</cite> — a 16× throughput gap. <cite index="9-9,15-1">H100 FP16 tensor cores deliver 3× the throughput of A100 FP16</cite>, landing near 495 TFLOPS per GPU at that precision. <cite index="9-2,15-2">FP8 doubles throughput over FP16/BF16 and halves the memory footprint</cite>. <cite index="13-1">H100 reaches 989 TFLOPS at FP8</cite>, and <cite index="14-20">up to 3,958 TFLOPS with sparsity</cite>.

<cite index="11-20,11-21,11-22">FP8 is 8-bit floating point that halves memory usage versus FP16/BF16; only H100 and H200 support native FP8; A100 cannot run FP8</cite>. <cite index="11-24">H100 achieves 2.23× speedup on 32B models switching from BF16 to FP8</cite>. That multiplier compresses as batch size falls and memory bandwidth binds.

<cite index="6-4,6-5,6-6">Older cuBLAS/cuDNN required matrix dimensions divisible by 16 bytes to use tensor cores; FP16 and BF16 are 2 bytes, so dimension needed to be multiple of 8; recent versions relax the requirement but performance remains best at 16-byte alignment</cite>. Misaligned shapes leave FLOPS on the table.

Sources:

• https://medium.com/@michael.diggin/the-power-of-8-getting-the-most-out-of-tensor-cores-c7704ae0c5c1
• https://www.e2enetworks.com/blog/nvidia-a100-vs-h100-vs-h200-gpu-comparison
• https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/
• https://www.advancedclustering.com/wp-content/uploads/2022/03/gtc22-whitepaper-hopper.pdf
• https://jarvislabs.ai/ai-faqs/what-is-the-flops-performance-of-the-nvidia-h100-gpu
• https://www.bestgpusforai.com/gpu-comparison/a100-vs-h100

Scaling laws for MoE models show that increasing sparsity — the ratio of total to active parameters — improves pretraining efficiency when total compute budget is fixed. You can push parameter count higher without increasing FLOPs per token, and loss continues to fall.

DeepSeek-MoE reports that a 16B-parameter MoE with 2.5B active parameters matches a 7B dense model under the same data budget, implying a 2.5× parameter-efficiency advantage. The data-centric view — fixed token count, variable parameter count — favors aggressive sparsity. The compute-centric view — fixed FLOP budget, variable data and parameters — shows total parameters can scale to 100× a dense baseline, but only if you feed it proportionally more training data and accept higher memory overhead.

The tradeoff tightens as you move down-market. Compact MoE models at on-device scale outperform FLOP-aligned dense models in controlled evaluations, but memory constraints bind faster than compute constraints on edge hardware. At hyperscale, MoE wins when interconnect bandwidth is high and batch sizes are large. At the edge, dense models win when VRAM is the limiting resource.

The cost curve for MoE is steeper than the cost curve for dense models because sparsity only helps if you can keep all experts fed. If your workload is bursty or your router collapses to a handful of experts, you are paying for parameters you do not use.

Sources:

• https://arxiv.org/html/2501.12370
• https://arxiv.org/pdf/2506.12119
• https://arxiv.org/pdf/2503.00245
• https://arxiv.org/pdf/2602.08019

The unit economics of MoE models split in two. Compute per token tracks active parameters — the subset routed for each forward pass. Memory per instance tracks total parameters — every expert must be resident, whether it fires or not.

Gemma 4 MoE carries 26–27 billion parameters but activates roughly 4 billion per token. Inference compute cost resembles a 4B dense model. Memory footprint resembles a 27B dense model. You cannot deploy it on 16 GB of VRAM at reasonable quantization, even though the FLOP profile says you should be able to. The distinction matters when you price hardware.

Load balancing adds operational cost. The router must distribute tokens evenly across experts or throughput collapses. If three experts saturate while five sit idle, you lose the parallelism that justifies the architecture. Training MoE models requires an auxiliary loss to enforce balanced utilization — another hyperparameter, another thing that can drift.

Expert parallelism generates roughly 9× the communication volume of tensor parallelism in large-scale deployments. That overhead compounds at multi-rack scale. DeepSeek V3 mirrors attention weights across devices under data-parallel setups, consuming 10 GB per GPU; past 64 GPUs, those duplicated weights exceed the memory held by the expert layers themselves. MoE reduces per-token cost. It increases memory, communication, and orchestration cost. The net win depends on your serving profile and interconnect speed.

Sources:

• https://www.justanotherpm.com/blog/mixture-of-experts-moe-reason-behind-cheapest-ai-models
• https://www.mindstudio.ai/blog/gemma-4-mixture-of-experts-architecture-explained
• https://www.tensoreconomics.com/p/moe-inference-economics-from-first
• https://cameronrwolfe.substack.com/p/conditional-computation-the-birth

Google's Switch Transformer paper in 2021 collapsed the routing decision. Prior MoE topologies sent each token to k > 1 experts and combined their outputs, under the hypothesis that the router could not learn without comparing at least two paths. Switch routing picks exactly one expert per token.

The simplification reduced inter-device communication cost — each expert lives on a single chip, no cross-device aggregation required — and cut the gating computation to a single argmax. Training instability increased, particularly under bfloat16, because hard switching magnifies gradient variance. Google addressed that by running expert internals at float32 while exposing bfloat16 to the rest of the network. Communication stayed low-precision; computation inside the expert stayed stable.

Switch Transformer reached 1.6 trillion parameters and delivered 4× pretraining speedup over T5-XXL at matched FLOP budget. Multilingual experiments on 101 languages showed 91% saw 4× or better speedup over the mT5 baseline. The model proved that parameter count and compute cost can move in opposite directions if you route sparsely enough.

The core lesson: MoE is not about model size. It is about cost per token at a given capability threshold. Switch demonstrated that threshold falls when you activate fewer weights per forward pass, even when total parameter count climbs into the trillions.

Sources:

• https://www.infoq.com/news/2021/02/google-trillion-parameter-ai/
• https://towardsdatascience.com/the-switch-transformer-59f3854c7050/
• https://medium.com/data-science/understanding-googles-switch-transformer-904b8bf29f66
• https://syncedreview.com/2021/01/14/google-brains-switch-transformer-language-model-packs-1-6-trillion-parameters/

Mixture-of-experts is a sparse architecture. Each layer holds multiple feed-forward networks — the experts — and a router picks which subset processes each token. Dense models push every token through the same FFN. MoE only activates a fraction.

The implication: you can hold billions more parameters without paying the FLOP cost on every forward pass. Mixtral 8x7B loads 47 billion parameters into memory but activates roughly 13 billion per token — 28% of the total. It matches Llama 2 70B on benchmarks at roughly one-fifth the per-token compute.

Switch Transformer took the pattern to the limit. Google routed each token to a single expert, the simplest possible topology. That reduced communication overhead across TPU pods and enabled training at up to 7× the speed of T5-XXL under the same compute budget. The 1.6-trillion-parameter instantiation held the headline, but the efficiency curve is what mattered.

The tradeoff: sparsity collapses training and serving FLOPs, but memory footprint reflects the full parameter count. All experts sit in VRAM whether they fire or not. Mixtral 8x7B cannot run on a single consumer GPU even though its active compute matches a 13B dense model — the memory demand is still 47B. MoE undercuts per-token cost. It does not shrink the silicon you need to hold the weights.

Sources:

• https://sebastianraschka.com/faq/docs/mixture-of-experts.html
• https://www.nvidia.com/en-us/glossary/mixture-of-experts/
• https://www.justanotherpm.com/blog/mixture-of-experts-moe-reason-behind-cheapest-ai-models
• https://towardsdatascience.com/the-switch-transformer-59f3854c7050/
• https://www.infoq.com/news/2021/02/google-trillion-parameter-ai/

RLHF emerged to solve a specification problem: tasks where the goal is complex, subjective, or impossible to encode as a mathematical reward function. You cannot write a formula for "helpful" or "harmless" or "funny," but humans can rank examples. This is the niche where RLHF delivers value—preference-based alignment on objectives that resist closed-form definition.

The Christiano paper tested this on robotics (backflips, locomotion) and Atari. The claim: learning from trajectory comparisons matches or exceeds hand-tuned rewards without requiring domain experts to design a scoring function. The same principle scaled to language models—InstructGPT trained on helpfulness/truthfulness/harmlessness rankings provided by contractors, not prompt-response correctness.

The boundary condition: when the reward is verifiable, RLHF becomes unnecessary overhead. Math problems have correct answers; code has test suites. DeepSeek R1 used RLVR (verifiable rewards) + GRPO, bypassing human labels entirely and producing reasoning gains cheaper than traditional RLHF. The cost-benefit inverts when ground truth is accessible.

The alignment question compounds this. RLHF optimizes for labeler preferences, which may not generalize to broader user populations or reflect actual deployment goals. The value proposition: RLHF trades dataset cost and iteration complexity for the ability to train on goals that cannot be automated. When automation is possible, the trade no longer clears.

Sources:

• https://www.ibm.com/think/topics/rlhf
• https://arxiv.org/pdf/2312.14925
• https://www.buildfastwithai.com/blogs/what-is-rlhf-llm-training
• https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback

The RLHF workflow has three stages: (1) supervised fine-tuning (SFT) on human demonstrations, (2) reward model training on ranked comparison data, (3) policy optimization via PPO using the reward model as scoring function. The design choice is indirection—humans shape behavior offline; the reward model substitutes for them at training time; the final policy runs standalone at inference.

At inference: no reward model, no human. The aligned policy serves requests identically to the base model—same parameter count, same latency, same throughput. RLHF adds zero marginal inference cost. The compute overhead is training-side: running policy rollouts, scoring them with the reward model, and updating weights.

The annotation cost structure: ~$100 per complex comparison for expert labelers; tens of thousands of comparisons for a production reward model. InstructGPT used contractor labelers; scale requires either narrow task scope or verifiable ground truth (code that compiles, math that solves). The cost does not recur per inference—it amortizes across all requests the deployed model serves.

The iteration risk: reward models can drift or saturate as policy improves. If the policy learns behaviors outside the comparison distribution, the reward signal becomes extrapolated and unreliable. This necessitates periodic retraining or online data collection, which reintroduces the human-labeling bottleneck. The economic read: RLHF front-loads alignment cost into training; the question is refresh cadence.

Sources:

• https://toloka.ai/blog/what-is-rlhf/
• https://www.ibm.com/think/topics/rlhf
• https://huggingface.co/blog/rlhf
• https://arxiv.org/pdf/2312.14925

The Christiano 2017 paper's key infrastructure claim: Proximal Policy Optimization (PPO) made human-in-the-loop training tractable by reducing the volume of required feedback. Prior methods (TAMER, COACH) required dense scoring; RLHF uses comparative preferences—human ranks trajectory pairs, not absolute values.

The efficiency mechanism: intelligent querying. The system selects trajectory pairs where the reward model is most uncertain, compressing the information extracted per comparison. Result: agents learned complex robot locomotion and Atari performance approaching hand-tuned rewards with feedback on <1% of environment interactions. This is orders of magnitude fewer samples than naive approaches.

PPO's role: it's a trust-region optimizer that constrains gradient updates to prevent destabilization. The update rule maximizes reward on the current batch (on-policy) while penalizing divergence from a reference policy via KL constraint. InstructGPT added pretraining loss during PPO to reduce "alignment tax"—performance regression on held-out NLP benchmarks.

The weakness: PPO is hyperparameter-fragile at scale, and models can exploit reward-model gaps (reward hacking). Example: InstructGPT models learned confident phrasing correlates with high reward, increasing hallucination slightly. The mitigation path is either verifiable rewards (RLVR) or offline methods (DPO) that bypass the RL loop entirely. The 2017 contribution was making the loop work; the 2024+ question is whether to keep it.

Sources:

• https://arxiv.org/abs/1706.03741
• https://intuitionlabs.ai/articles/reinforcement-learning-human-feedback
• https://www.buildfastwithai.com/blogs/what-is-rlhf-llm-training
• https://cameronrwolfe.substack.com/p/ppo-llm

Christiano et al. 2017 introduced the RLHF methodology now underpinning InstructGPT and ChatGPT. The approach trains a reward model from pairwise human comparisons of trajectory segments, then optimizes policy via RL (PPO) to maximize that learned reward. The core claim: complex behaviors can be learned with minimal human time—Christiano's team trained a simulated backflip with ~900 comparisons requiring <1 hour human input, and matched hand-tuned reward performance on Atari games by querying humans on the most uncertain trajectory pairs.

The economics matter. Gathering preference data is the variable cost. InstructGPT used ~50k labeled preference samples; production RLHF pipelines cost $1–5 million in annotation depending on task complexity. Compute is tractable—OpenAI's 175B PPO model required 60 petaflops/s-days versus 3,640 for GPT-3 pretraining. The constraint is not inference but iteration: PPO at scale is hyperparameter-sensitive and requires running four model copies (policy, reference, reward, critic) across hundreds of GPUs for 70B+ models.

The training delta: RLHF adds ~1.6% of the pretraining compute but delivers alignment improvements that 100x scale does not. The cost curve compresses when the reward signal is verifiable (RLVR for math/code uses test-pass as reward, no human labels). PPO is being displaced—GRPO and DPO reduce overhead—but the core tradeoff holds. Preference collection is the bottleneck; model serving is post-training inference.

Sources:

• https://arxiv.org/abs/1706.03741
• https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf
• https://www.buildfastwithai.com/blogs/what-is-rlhf-llm-training
• https://huggingface.co/blog/rlhf

<cite index="10-2,10-3">Standard speculative decoding generates draft tokens one at a time, hitting a ceiling for reasoning models producing thousands of chain-of-thought tokens. Methods like SpecReason and Lookahead Reasoning draft entire reasoning steps, verifying semantic correctness rather than exact token match.</cite> <cite index="10-4">One benchmark reported up to 2.1× speedup when combined with n-gram speculative decoding on open-source reasoning models.</cite>

<cite index="1-2">Medusa uses a lightweight multi-head prediction architecture and static tree attention verification.</cite> <cite index="4-4,4-5">Staged speculative decoding restructures the speculative batch as a tree, reducing generation costs and increasing expected tokens per batch, and adds a second stage of speculative decoding.</cite> <cite index="10-1">EAGLE-3 reported 3.0–6.5× speedups over standard generation, with 20–40% improvement over EAGLE-2.</cite>

<cite index="10-12,10-13">Adaptive speculation dynamically chooses draft length γ based on current batch size. Adaptive approaches measured an average 1.94× across all batch sizes in one study, where fixed-length speculation performed worse and gains diminished at batch size 32+.</cite> <cite index="11-4">Best for coding assistants, document summarization, structured output tasks.</cite> <cite index="10-14">Agentic, reasoning, and code generation workloads tend to show the largest gains.</cite>

<cite index="5-3,5-4">Distributed speculative decoding incurs significant uplink communication overhead, requiring full-vocabulary logits at every decoding step. Truncated Sparse Logits Transmission transmits only logits and indices of a truncated candidate set.</cite>

Sources:

• https://redis.io/blog/speculative-decoding-llm/
• https://arxiv.org/pdf/2603.03383
• https://arxiv.org/pdf/2308.04623
• https://www.sciencedirect.com/science/article/pii/S2949715925000782
• https://blog.premai.io/speculative-decoding-2-3x-faster-llm-inference-2026/

<cite index="2-2">Draft model produces drafts; target LLM verifies for acceptance or rejection.</cite> <cite index="9-14">The number of tokens accepted by the target model affects speedup.</cite> <cite index="11-2">At acceptance rate below 0.5, speculative decoding can hurt performance because cycles are wasted proposing and verifying.</cite>

<cite index="2-4,2-5">Widely used draft models generate draft tokens non-autoregressively without considering correlations between tokens, yielding high decoding speed but unsatisfactory acceptance rate.</cite> <cite index="2-7">CTC-based draft models strengthen correlations between draft tokens, generating higher-quality candidate sequences.</cite> <cite index="12-16,12-17">Speculative decoding only delivers expected speedup if the draft model's distribution matches closely with the target model; the key is using the right draft model for your workload, which often means training one on your own data.</cite>

<cite index="14-7,14-8,14-9,14-10">Speedup depends on draft-model alignment with target and draft-model speed, influencing hyperparameter K (candidate tokens per loop). When draft aligns well or runs fast, larger K allows more accepted tokens, but also increases rejection risk and discarded tokens.</cite> <cite index="9-11,9-12">As draft model size increases, throughput decreases due to higher inference latency despite consistent increases in token acceptance rate.</cite>

The optimization is lossless. <cite index="12-19">Speculative decoding speeds up token generation without sacrificing output quality.</cite>

Sources:

• https://arxiv.org/pdf/2412.00061
• https://arxiv.org/pdf/2402.01528
• https://blog.premai.io/speculative-decoding-2-3x-faster-llm-inference-2026/
• https://www.bentoml.com/blog/3x-faster-llm-inference-with-speculative-decoding
• https://arxiv.org/pdf/2405.19715

<cite index="8-2,8-3,8-12,8-13">Benchmarks show speculative decoding delivers around 2× speedup over autoregressive sampling in most cases.</cite> <cite index="11-1">At acceptance rates of 0.6–0.8, realistic with off-the-shelf EAGLE3 draft models, speedup is 2–3×.</cite> <cite index="15-5,15-8">Llama 3.1-70B paired with smaller draft models achieves >2.0× speedup, reaching 2.31× with the 1B draft model.</cite> <cite index="4-6">Staged speculative decoding reduced single-batch latency by 3.16× for a 762M GPT-2-L model.</cite>

Batch size erodes the gain. <cite index="10-11">On Qwen3-8B, speedup degraded from 1.93× to 0.99× as batch size grew from 2 to 48.</cite> <cite index="11-14">The pattern across benchmarks: 2–3× at low concurrency (1–10 simultaneous requests), diminishing returns above that.</cite> <cite index="10-8">At low batch sizes, speculative decoding delivers real speedups.</cite> <cite index="10-9">Long-context serving is an exception—when batch size is large and sequences are long, the KV cache bottleneck can restore conditions where speculative decoding helps.</cite>

<cite index="15-1,15-13">Speedup ratio increases with input sequence length, indicating speculative decoding boosts performance further when input is longer.</cite> <cite index="8-4">Speedup depends heavily on draft model choice.</cite> <cite index="8-14">Draft model must be 10–50× smaller to achieve acceleration.</cite>

Sources:

• https://medium.com/ai-science/speculative-decoding-make-llm-inference-faster-c004501af120
• https://blog.premai.io/speculative-decoding-2-3x-faster-llm-inference-2026/
• https://rocm.blogs.amd.com/software-tools-optimization/speculative-decoding---deep-dive/README.html
• https://arxiv.org/pdf/2308.04623
• https://redis.io/blog/speculative-decoding-llm/

<cite index="1-4,1-5,1-6">Autoregressive decoding transfers model weights from HBM to compute units for each token, leaving tensor cores idle most of the inference cycle.</cite> <cite index="4-11,4-12">At small batch sizes, inference is memory bandwidth bound; the only path to speedup is increasing arithmetic intensity.</cite>

<cite index="1-8,1-11">Speculative decoding uses a lighter draft model to propose multiple candidate tokens, verified in parallel by the target model, converting sequential ops into compute-bound parallel verification.</cite> <cite index="3-3,3-4,3-5">The draft model predicts the next K tokens; the target model verifies them in parallel and accepts the longest prefix it agrees with.</cite> <cite index="9-6,9-7">Verification resembles the prefill stage in LLM inference.</cite>

<cite index="8-7">Each target-model run produces at least one token in the worst case, so serial runs never exceed autoregressive, but can generate up to γ + 1 tokens per pass.</cite> <cite index="9-8">As long as more than one token is accepted on average, speculative decoding delivers speedup.</cite> <cite index="12-19,12-20">The technique is inspired by speculative execution in processors—operations computed in parallel ahead of time, discarded if unneeded.</cite>

The economics hinge on acceptance rate and draft-model cost. <cite index="2-3">Inference speed is decided by draft-model decoding speed and acceptance rate.</cite>

Sources:

• https://arxiv.org/pdf/2603.03383
• https://arxiv.org/pdf/2308.04623
• https://bentoml.com/llm/inference-optimization/speculative-decoding
• https://arxiv.org/pdf/2402.01528
• https://www.bentoml.com/blog/3x-faster-llm-inference-with-speculative-decoding

<cite index="1-27,1-28">Software developer Georgi Gerganov released llama.cpp as open-source on March 10, 2023. It's a re-implementation of Llama in C++, allowing systems without a powerful GPU to run the model locally.</cite> <cite index="1-29,1-30">The llama.cpp project introduced the GGUF file format, a binary format that stores both tensors and metadata. The format focuses on supporting different quantization types, which can reduce memory usage and increase speed at the expense of lower model precision.</cite>

The inference ecosystem built underneath the weights. <cite index="12-9,12-11">AWS announced Large Model Inference containers in 2023 that reduced latency by 33% for Llama-2 70B workloads. For Llama-2 70B at concurrency of 16, latency was reduced by 28% and throughput increased by 44% with TensorRT-LLM containers.</cite> <cite index="12-6,12-7,12-8">New model architectures like Mixture-of-Experts (MoE) created different cost profiles. MoE models have unique inefficiencies—load imbalance during prefill and increased memory transfers during decode. But optimized MoE implementations can deliver better cost-performance than dense models.</cite>

<cite index="5-9,5-10">Running a local model ensures that proprietary code, training modifications and proprietary data can be used to fine-tune model performance without being loaded to a commercial server. Smaller model sizes like the 7B and 13B variants enable smoother performance in environments like mobile apps where processing power is limited.</cite>

The open weight release unlocked an inference layer that iterated faster than the model layer. llama.cpp shipped CPU-only serving paths that dropped GPU requirements entirely for low-throughput use cases. The cost floor compressed from there.

Sources:

• https://en.wikipedia.org/wiki/Llama_(language_model)
• https://aisuperior.com/llm-hosting-cost/
• https://www.ibm.com/think/topics/llama-2

<cite index="19-9">The 70 billion-parameter Llama 2 model improved results on the MMLU and BBH benchmarks by roughly 5 and 8 points respectively, compared to the 65 billion-parameter Llama 1 model.</cite> <cite index="18-3,18-15">The Llama 2 70B variant outperformed the largest variant of all other open-source models including Llama 1, MPT, and Falcon.</cite> <cite index="19-11,19-12">Llama 2's 7B and 34B parameter models outperformed Falcon models of similar size (7B and 40B) in all benchmark categories. The Llama 2 70B model surpassed all open-source models.</cite>

<cite index="19-13">The Llama 2 70B model performed similarly to the closed-source GPT-3.5 on the MMLU and GSM8K benchmarks but showed a significant deficit on coding benchmarks.</cite> <cite index="19-1,19-2">Llama 2 70B matched or exceeded the performance of PaLM (540 billion parameters) on nearly all benchmarks. However, a substantial performance gap remained between the Llama 2 70B model and both GPT-4 and PaLM-2-L.</cite>

<cite index="19-6">The fine-tuned Llama 2-Chat models surpassed existing open-source chat models on most benchmarks, and according to human evaluations for usefulness and safety, could potentially replace closed-source models.</cite> <cite index="20-8,20-9">Llama 2 outperformed other open-source models on metrics including general intelligence and alignment. However, GPT-4 still outperformed Llama 2 on several benchmarks.</cite>

The benchmark performance established the open-weight floor. Llama 2 70B closed most of the gap to frontier closed models on reasoning tasks. Code generation remained the delta. The next open release would target that wedge.

Sources:

• https://viso.ai/deep-learning/llama-2/
• https://blog.gopenai.com/paper-review-llama-2-open-foundation-and-fine-tuned-chat-models-23e539522acb
• https://alexandrabarr.beehiiv.com/p/llama-2

<cite index="10-2">Self-hosting Llama 2 70B requires approximately 206GB of VRAM, typically 2-4x H100 GPUs, costing $8-16/hour in cloud GPU rental or $100,000+ in purchased hardware.</cite> <cite index="17-1,17-2">Llama 2 70B was trained using bfloat16, consuming 2 bytes per parameter, resulting in a 140GB model size.</cite> <cite index="12-14,12-15">A 70B parameter model requires approximately 35GB VRAM with 4-bit quantization or 140GB with full FP16 precision, typically requiring an A100 80GB GPU with quantization or a multi-GPU setup.</cite>

The cost curve favors APIs at low volume. <cite index="15-10,15-11">To generate approximately 1M tokens (about as much as an A80 GPU can produce in a day), the cost is $0.12 on DeepInfra via API, $0.71 on Azure AI Foundry via API, $43 on Lambda Labs, and $88 on Azure servers.</cite> <cite index="10-8,10-9">For medium volume, managed API providers like DeepInfra charge $0.15/M input tokens or Groq at $0.20/M. For high volume above 50-100 million tokens per month, self-hosted GPUs with INT4 quantization become cost-competitive with API pricing.</cite>

<cite index="12-1,12-13">For high-volume production (50,000+ daily requests), self-hosting 7B-13B parameter models on entry-tier hardware ($1,500-$5,000/month) typically costs less than equivalent API usage.</cite> <cite index="11-15,11-16,11-17">Cloud hosting costs range from $0.53 to $7.34 per hour. For the 7B model, AWS and Azure start at $0.53/hr.</cite> <cite index="17-31,17-32">Self-hosting becomes viable only with user request loads that far exceed the ~22.2M words/day mark, combined with resources to manage talent and logistics. For most use cases, owning the model instead of using an API is not financially beneficial.</cite>

Quantization compresses the cost floor. <cite index="10-28,10-29">At FP16 precision, models require roughly 2 bytes of VRAM per parameter. INT4 quantization compresses this to about 0.5 bytes per parameter, with an additional 10-20% overhead for KV cache and framework runtime.</cite>

Sources:

• https://techjacksolutions.com/ai-tools/meta-llama/llama-pricing/
• https://www.1001epochs.ch/blog/hosting-llama-2-a-comprehensive-guide-to-costs-model-sizes-and-cloud-requirements
• https://aisuperior.com/llm-hosting-cost/
• https://www.detectx.com.au/cost-comparison-api-vs-self-hosting-for-open-weight-llms/
• https://venturebeat.com/ai/openai-or-diy-unveiling-the-true-cost-of-self-hosting-llms

<cite index="1-3,1-4">Meta released Llama 2 on July 18, 2023 in partnership with Microsoft. The model shipped in three parameter sizes: 7B, 13B, and 70B.</cite> <cite index="1-5">The architecture stayed nearly unchanged from Llama 1, but Meta used 40% more training data for the foundation models.</cite> <cite index="6-8">Llama 2 was pretrained on 2 trillion tokens sourced from publicly available data.</cite>

The licensing terms matter more than Meta's marketing. <cite index="4-9">The license restricts companies with over 700 million monthly active users, requiring special permission from Meta. Users cannot use Llama 2 outputs to train competing LLMs.</cite> <cite index="1-8,1-9">The Open Source Initiative disputes Meta's use of the term "open source" because Llama's license enforces an acceptable use policy that prohibits certain uses.</cite> <cite index="10-11,10-12">The model is more accurately described as "open-weights" or "source-available." The Llama Community License restricts commercial use above 700M MAU and does not disclose full training data details.</cite>

<cite index="5-2,5-3">Unlike Llama 1 which was released exclusively for research use, Llama 2 became available to any organization with fewer than 700 million active users. Llama 2 was pre-trained on 40% more data, increasing its knowledge base.</cite> <cite index="5-4">Llama 2 chat models were fine-tuned using reinforcement learning from human feedback (RLHF), unlike Llama 1.</cite>

The distribution strategy split hosted vs. self-hosted paths. <cite index="2-8,2-9">Users download model weights and tokenizer from Meta's website after accepting the license. Once approved, Meta sends a signed URL via email.</cite> The weights are free. The inference layer is not.

Sources:

• https://en.wikipedia.org/wiki/Llama_(language_model)
• https://github.com/meta-llama/llama
• https://opensourceconnections.com/blog/2023/07/19/is-llama-2-open-source-no-and-perhaps-we-need-a-new-definition-of-open/
• https://www.ibm.com/think/topics/llama-2
• https://huggingface.co/meta-llama/Llama-2-7b

<cite index="26-6">MI300X (CDNA 3, 2023, 192 GB HBM3, 163 TFLOPS FP32, 326 TFLOPS FP16, 5300 GB/s, 750W) versus MI250X (CDNA 2, 2021, 128 GB HBM2e, 47.9 TFLOPS FP32, 95.7 TFLOPS FP16, 3277 GB/s, 560W).</cite> <cite index="26-6">MI300X is 240% faster at FP32, 241% faster at FP16, 582% faster at INT8, and 62% faster on memory bandwidth.</cite>

The generation delta is architectural. <cite index="24-1,24-2">MI300X adds TF32 (653.7 TFLOPS base, 1307.4 with sparsity) and FP8 (2614.9 TFLOPS base, 5229.8 with sparsity); MI250X supports neither.</cite> That precision ladder matters for LLM serving: FP8 cuts activation memory and raises throughput without collapsing quality on most transformer blocks. The cost curve compresses when you can serve at half the bits.

<cite index="29-11,29-12">ROCm 6 on MI300X is 8× faster than MI250 with ROCm 5 for a 70B-parameter LLM; ROCm 6 adds FP16 support, low-level optimizations, and new data types.</cite> Software maturity is the second moat. <cite index="16-7,16-8,16-9">ROCm 5.7+ closed the AMD competitiveness gap; PyTorch FSDP runs on MI250 without code modification, FlashAttention-2 via Triton kernels, and RCCL supports distributed workloads.</cite>

<cite index="21-1,21-2">MI250 offers 128 GB with CDNA2; MI300X provides 192 GB with CDNA3—choose MI250 for lower cost, MI300X for higher performance.</cite> <cite index="23-19,23-20,23-21">MI300X and MI350 series compete directly with H100/H200; they often deliver lower cost per unit of memory bandwidth, and for long-context workloads the TCO may favor AMD.</cite> The price floor at inference keeps falling; the delta between generations compresses every release.

Sources:

• https://www.runcrate.ai/gpu-compare/mi300x-vs-mi250x
• https://www.amd.com/en/products/accelerators/instinct/mi300.html
• https://www.hpcwire.com/2023/12/07/amds-horsepower-packed-mi300x-gpu-beats-nvidias-upcoming-h200/
• https://www.fluence.network/blog/amd-instinct-mi250/
• https://getdeploying.com/gpus/amd-mi250-vs-amd-mi300x
• https://www.bentoml.com/blog/amd-data-center-gpus-mi250x-mi300x-mi350x-and-beyond

<cite index="13-4,13-5">MI250 accelerators are based on AMD CDNA 2 architecture targeting HPC, AI, and ML workloads from individual servers to exascale supercomputers.</cite> <cite index="13-2,13-19">Each Graphics Compute Die (GCD) has 104 active compute units.</cite> <cite index="13-20,13-21">Each CU subdivides into four SIMD units processing 16 FP64 elements per instruction, enabling 64-item wavefronts at 1.7 GHz peak clock.</cite> <cite index="13-22,13-23">Theoretical FP64 vector peak is 22.6 TFLOPS per GCD, 45.3 TFLOPS for both GCDs combined.</cite> <cite index="13-24,13-25">Matrix cores deliver 90.5 TFLOPS FP64 matrix performance across both dies.</cite>

<cite index="16-4">MI250 (released November 2021) packs 128 GB HBM2e, 3.2 TB/s bandwidth, 208 compute units, 13,312 stream processors at 500W TDP.</cite> <cite index="16-12,16-13">Peak throughput: 90.5 TFLOPS FP32/FP64 matrix, 362.1 TFLOPS FP16/BF16, 362.1 TOPs INT8.</cite> <cite index="2-9">MI200 series lacks TF32 and FP8 support—those precisions arrived with CDNA 3.</cite>

<cite index="13-11,13-12">MI250 OAMs attach via PCIe Gen 4 x16; each GCD maintains its own x16 link to the host.</cite> <cite index="11-10">AMD Infinity Fabric links between GPUs run up to 25 GT/s, correlating to 50 GB/s peak per 16-lane link.</cite> <cite index="15-6,15-8">MI250 is the defining element in ORNL Frontier; it ships as a dual-GCD package where each die exposes independent memory, compute resources, and Infinity Fabric links.</cite> <cite index="16-16,16-17">Databricks benchmarks show 96% scaling efficiency, with throughput falling only from 166 to 159 TFLOPS/GPU scaling 4 to 128 GPUs.</cite> MI250 competes with A100; MI300X undercuts H100.

Sources:

• https://rocm.docs.amd.com/en/latest/conceptual/gpu-arch/mi250.html
• https://instinct.docs.amd.com/latest/gpu-arch/mi250.html
• https://www.amd.com/en/products/accelerators/instinct/mi300.html
• https://www.emergentmind.com/topics/amd-mi250-gpus
• https://www.fluence.network/blog/amd-instinct-mi250/
• https://www.amd.com/en/support/downloads/drivers.html/accelerators/instinct/instinct-mi200-series/amd-instinct-mi250.html

<cite index="1-8,1-9">MI300X delivers 2614.9 TFLOPS FP8, 1307.4 TFLOPS FP16/BF16, 163.4 TFLOPS FP32, 81.7 TFLOPS FP64, built on CDNA 3 architecture at 750W TDP.</cite> <cite index="3-1,3-10">The chip packs 304 compute units and 192 GB HBM3 on a single OAM module.</cite> <cite index="1-1,1-10">Memory bandwidth measures 5.325 TB/s via an 8,192-bit bus at 5.2 Gbps.</cite>

<cite index="29-3,29-4">Prior-generation MI250X delivered 47.9 TFLOPS FP32 and 22.6 TFLOPS FP64 per GCD—the MI300X is 3.4× faster at FP32.</cite> <cite index="2-8">MI250X (560W, 128 GB HBM2e) hit 383 TFLOPS FP16 but lacked TF32 and FP8 support.</cite> The precision gap matters: FP8 is the workhorse for training at scale, and MI300X prices it at roughly 6.8× the MI250 rate on paper.

<cite index="3-13,3-23">Eight MI300X accelerators interconnect via AMD Infinity Fabric at 128 GB/s per link, totaling 896 GB/s bidirectional in an 8-GPU ring.</cite> <cite index="10-2,10-6,10-7">A UBB 2.0 platform holds 8 OAM modules with 1.5 TB aggregate HBM3; each module comprises 8 XCDs (Accelerator Complex Dies).</cite> The chiplet design stacks 5 nm compute dies with 6 nm I/O on a multi-chip package — die yield improves, but the thermal envelope holds at 750W.

<cite index="6-15">On-demand MI300X pricing rose 33% May–November 2025, from $2.35 to $3.12/hr per GPU.</cite> <cite index="6-4,6-5">Spot instances fall as low as $0.95/hr; on-demand listings reach $7.86/hr.</cite> The 192 GB capacity enables 70B-class models with headroom; 4-bit quantization pushes the ceiling higher, bounded by KV cache and context length.

Sources:

• https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html
• https://www.amd.com/en/products/accelerators/instinct/mi300.html
• https://lenovopress.lenovo.com/lp1943-thinksystem-amd-mi300x-192gb-750w-8-gpu-board
• https://instinct.docs.amd.com/projects/system-acceptance/en/latest/gpus/mi300x.html
• https://getdeploying.com/gpus/amd-mi300x
• https://www.hpcwire.com/2023/12/07/amds-horsepower-packed-mi300x-gpu-beats-nvidias-upcoming-h200/

<cite index="2-1,2-2">This work investigates dynamic KV cache placement across such systems to maximize aggregated bandwidth utilization under capacity constraints. Rather than proposing a specific scheduling policy, we formulate the placement problem mathematically and derive a theoretical upper bound, revealing substantial headroom for runtime optimization.</cite> The read: KV cache can be split across GPU HBM, CPU DRAM, and NVMe when the working set exceeds what fits in HBM. The routing decision is latency-aware.

<cite index="3-1,3-2">By strategically scheduling idle memory bandwidth during active computation windows, our method proactively prefetches required KV Cache into GPU L2 cache, enabling high-speed L2 cache hits for subsequent accesses and effectively hiding HBM access latency within computational cycles. Extensive experiments on NVIDIA H20 GPUs demonstrate that the proposed method achieves 2.15× improvement in attention kernel efficiency and up to 1.97× end-to-end throughput enhancement, surpassing state-of-the-art baseline FlashAttention-3.</cite>

<cite index="7-3,7-4,7-5">Existing work, such as Pie, leverages this extra CPU-GPU bandwidth by offloading the KV cache of future layers to CPU memory and swapping back to GPU memory as required, thereby increasing the effective KV cache size. This allows for larger batches or longer sequences without triggering recomputation, thereby reducing inference serving latency and improving serving throughput. However, the frequent updates to the KV cache in GPU memory require synchronization during KV cache swapping, introducing overheads with bidirectional data transfers.</cite>

The techniques are orthogonal to GQA and quantization. The cost curve implication: prefetch + heterogeneous memory extend the batch size you can serve before eviction, which compresses amortized cost per token when request volume is high. The floor on this path is PCIe bandwidth and CPU-side DRAM latency — both measurable, both on the roadmap.

Sources:

• https://arxiv.org/pdf/2508.13231
• https://arxiv.org/pdf/2504.06319
• https://arxiv.org/pdf/2507.11507

<cite index="1-1,1-2">The quantization of KV cache helps alleviate the burden on multiple components of the inference pipeline, impacting compute, memory capacity, and memory bandwidth: Memory capacity: NVFP4 KV cache reduces the memory footprint of the KV cache by about 50% compared to FP8 KV cache. This enables larger context lengths, batch sizes, and user concurrency.</cite>

NVIDIA documented this path in December 2025. FP8 was the prior standard; FP4 halves it again. The tradeoff is precision loss — measured in benchmark delta and in what gets quietly walked back when users hit quality floors in production.

<cite index="1-3">Memory bandwidth: During the decode phase, which involves many read/writes of KV cache and puts significant pressure on memory bandwidth, smaller K[V cache reduces bandwidth pressure].</cite> The bytes-per-token figure compresses linearly with the quantization level. A model serving at FP4 reads half the bytes per decode step compared to FP8, which reads half compared to FP16.

The unit economics layer underneath: if you can hold quantization at FP4 without crossing the quality threshold that triggers user churn or support escalation, the cost per million tokens falls by the compression ratio. But the floor is empirical, not announced. What NVIDIA published in the technical blog is the capability. What the hyperscalers and the model-serving startups price in is whether the accuracy delta holds across the workload distribution their customers actually run. The gap between those two is where margin compresses or holds.

Sources:

• https://developer.nvidia.com/blog/optimizing-inference-for-long-context-and-large-batch-sizes-with-nvfp4-kv-cache/

<cite index="5-16,5-17,5-18,5-19">Standard multi-head attention maintains separate KV heads for each query head. GQA shares a smaller set of KV heads across groups. Llama 3.1 70B uses 8 KV heads instead of 64 query heads, reducing KV cache memory by 8×. GQA is now the default in Llama 3, Mistral, Gemma, and most production models because it provides 30-40% faster inference with near-equivalent accuracy.</cite>

<cite index="11-7,11-8">The KV cache reduces from 2Hd_h to 2Gd_h per token. In practice, G is typically set to H/4 or H/8, yielding a 4-8× cache reduction.</cite> <cite index="14-5,14-6,14-7">Smaller KV cache = more concurrent requests. With GQA-8 instead of MHA-32, the KV cache is 4× smaller per request. On a GPU with fixed memory, this means you can serve 4× more concurrent users (or handle 4× longer contexts) before running out of memory.</cite>

<cite index="14-9,14-10,14-11,14-12">Faster decoding. Each decode step reads less data from memory. For memory-bandwidth-bound workloads (which is essentially all autoregressive decoding), less data to read means faster generation. The speedup is roughly proportional to the reduction in KV heads.</cite>

The architecture choice prices in before training. Once a model ships with GQA, the memory coefficient is fixed. The serving economics fall through: 8× fewer bytes read per decode step compress both latency and cost per million tokens. The frontier models already priced this in. New architectures that do not match GQA or equivalent will undershoot on throughput per watt and per rack.

Sources:

• https://www.morphllm.com/kv-cache-explained
• https://fin.ai/research/low-rank-key-value-attention-reducing-kv-cache-memory-and-maintaining-head-diversity/
• https://www.generalcompute.com/blog/multi-query-grouped-query-attention

<cite index="5-14,5-15">Prefill is compute-bound (many tokens processed in parallel). Decode is memory-bandwidth-bound (the model reads the entire cache on every step to compute attention, but only processes one token).</cite> <cite index="2-6,2-7">LLM inference scaling is hindered by memory bandwidth bottlenecks. These arise in the read-intensive decode stage, where KV cache accesses predominate.</cite>

<cite index="3-6,3-7">Each decoding step requires loading the Key-Value Cache of historical sequences from off-chip High Bandwidth Memory (HBM) into compute unit registers. Due to the inability of HBM bandwidth to meet the throughput demands of modern compute units, these frequent off-chip memory accesses result in substantial data movement latency, which has become the critical bottleneck limiting LLM inference throughput.</cite>

<cite index="4-2">KV cache memory footprint scales linearly with context length, imposing critical bottlenecks on GPU memory capacity, memory bandwidth, and inference throughput as production LLMs push context windows from thousands to millions of tokens.</cite> <cite index="5-1,5-2,5-3">Every generated token reads the entire KV cache during attention. A smaller cache means less memory bandwidth per decode step. The effect is modest per-token but compounds over hundreds of output tokens.</cite>

The cost delta is measurable. <cite index="5-8">For a 10,000-token prompt on an H100, prefill takes 200-400ms.</cite> <cite index="5-12">Decode typically runs at 30-150 tokens per second depending on model size and hardware.</cite> The decode phase is where memory bandwidth shows up in per-token serving cost. The unit to watch is bytes read per token generated — that figure drives both latency and the maximum concurrent batch you can hold before the serving node saturates on memory throughput.

Sources:

• https://www.morphllm.com/kv-cache-explained
• https://arxiv.org/pdf/2508.13231
• https://arxiv.org/pdf/2504.06319
• https://arxiv.org/pdf/2603.20397

<cite index="19-7">vLLM improves the throughput of popular LLMs by 2-4× with the same level of latency compared to state-of-the-art systems, such as FasterTransformer and Orca</cite>. <cite index="2-11">vLLM uses PagedAttention to store attention keys and values efficiently, enabling up to 24x higher serving throughput</cite>. <cite index="23-12">vLLM can run models with up to 24x higher throughput than HuggingFace Transformers and up to 3.5x higher throughput than HuggingFace Text Generation Inference</cite>.

Measured performance varies by workload. <cite index="5-3,5-4">As generation-length variance increases, naive static batching's performance plummets to 81 token/s; FasterTransformers improves upon naive static batching significantly, nearly keeping up with naive continuous batchers</cite>. <cite index="10-2,10-3">vLLM becomes saturated around QPS=8 with throughput near 1900 token/s; continuous batching reduces latency by injecting new requests immediately when possible, and enables advanced memory optimizations that increase the QPS the serving system can handle before saturation</cite>.

<cite index="16-7,16-8,16-9">Continuous batching has fundamentally changed LLM deployment economics; systems that previously needed multiple GPUs to serve moderate traffic can now handle the same load on a single device, making LLMs practical for a much wider range of applications and organizations</cite>.

Sources:

• https://arxiv.org/abs/2309.06180
• https://www.hyperstack.cloud/blog/case-study/what-is-vllm-a-guide-to-quick-inference
• https://www.runpod.io/blog/introduction-to-vllm-and-pagedattention
• https://www.anyscale.com/blog/continuous-batching-llm-inference
• https://mbrenndoerfer.com/writing/continuous-batching

<cite index="12-5,12-6">LLM inference is fundamentally limited by memory bandwidth rather than raw compute power; when serving a single request, the GPU spends most of its time moving model weights from VRAM to the processing cores, leaving the actual compute units underutilized</cite>. <cite index="12-2,12-7">By grouping 32 or 64 requests together, the cost of loading model weights is amortized across all requests in the batch; batching addresses underutilization by processing multiple sequences simultaneously, allowing the weights to be loaded once and applied to many tokens</cite>.

<cite index="3-13,3-14,3-15">In static batching, prompts are collected together into a batch and processed simultaneously by the model, passed as multi-dimensional tensors to the LLM; this approach works best for offline workloads, where response time isn't a concern, and maximum throughput is the goal</cite>. <cite index="3-17">In continuous batching, prompts are sent individually to the inference engine but are dynamically grouped during processing</cite>.

<cite index="12-11,12-12">With static batching, throughput is dictated by the slowest summary; with continuous batching, shorter tasks cycle through the GPU rapidly, keeping the hardware saturated</cite>. <cite index="3-2">Larger batch sizes in LLM inference typically improve throughput, but past a certain point, system saturation can reduce efficiency and increase memory pressure</cite>.

Sources:

• https://lyceum.technology/magazine/batching-strategies-llm-inference-throughput/
• https://www.hyperstack.cloud/technical-resources/tutorials/static-vs.-continuous-batching-for-large-language-model-inference

<cite index="19-5,19-6">PagedAttention is an attention algorithm inspired by classical virtual memory and paging techniques in operating systems; vLLM achieves near-zero waste in KV cache memory and flexible sharing of KV cache within and across requests</cite>. <cite index="4-4">PagedAttention pages KV cache blocks on demand, treating GPU memory the way an operating system treats RAM</cite>.

<cite index="17-2,17-3">PagedAttention reframes the KV-cache as a set of fixed-size pages (e.g., 16 tokens per block); rather than maintaining a single contiguous tensor for each sequence, a lightweight page table records logical-to-physical block mappings</cite>. <cite index="17-5">This design allows KV-cache utilization in excess of 90%, vastly outperforming Orca and FasterTransformer which achieve 20–40% utilization</cite>.

<cite index="23-14">Existing systems waste 60%-80% of the KV-Cache, whereas vLLM achieves near-optimal memory usage with waste under 4%</cite>. <cite index="12-13,12-14,12-15">The KV cache stores intermediate attention keys and values for all previous tokens; as batch sizes and sequence lengths increase, the KV cache can easily consume 40 GB or more of memory</cite>. <cite index="22-6,22-7">Storing multiple tokens within a KV block (block size > 1) enables the PagedAttention kernel to process the KV cache across more positions in parallel, increasing hardware utilization and reducing latency, but a larger block size increases memory fragmentation</cite>.

Sources:

• https://arxiv.org/abs/2309.06180
• https://www.emergentmind.com/topics/vllm-system
• https://www.runpod.io/blog/introduction-to-vllm-and-pagedattention
• https://lyceum.technology/magazine/batching-strategies-llm-inference-throughput/
• https://arxiv.org/pdf/2309.06180

<cite index="1-3">Continuous batching allows new requests to join an active generation batch as soon as a slot becomes available, rather than waiting for all requests in the current batch to finish</cite>. <cite index="1-5">The decision about which requests to include is made at each generation step rather than once per batch</cite>—called iteration-level scheduling.

<cite index="7-4">In static batching, all requests in the batch must wait until the slowest one finishes, leading to wasted compute resources and increased latency</cite>. <cite index="4-14">The vLLM scheduler adds newly arrived requests to a running batch after each forward pass, the moment a previous sequence finishes</cite>. This prevents GPU idling caused by tail sequences.

<cite index="16-6">The first analysis of continuous batching for LLMs appeared in the Orca paper (Yu et al., 2022), which demonstrated that iteration-level scheduling could reach up to 36.9x higher throughput compared to existing systems</cite>. <cite index="9-2">Continuous batching can achieve 10x-20x better throughput than dynamic batching</cite>. <cite index="5-7">vLLM more than doubles performance compared to naive continuous batching</cite> in variable-length workloads.

<cite index="1-7,1-8">Misconfigured defaults of 64 sequences and 2,048 batched tokens can leave most GPU capacity unused even when the request queue is long; properly tuned continuous batching is often the single largest contributor to throughput improvements in vLLM deployments</cite>.

Sources:

• https://www.ai21.com/glossary/foundational-llm/what-is-continuous-batching/
• https://www.runpod.io/articles/guides/vllm-pagedattention-continuous-batching
• https://bentoml.com/llm/inference-optimization/static-dynamic-continuous-batching
• https://www.anyscale.com/blog/continuous-batching-llm-inference
• https://mbrenndoerfer.com/writing/continuous-batching

<cite index="2-7">GPT-3 operated without gradient updates or fine-tuning. Tasks and few-shot demonstrations were specified purely via text interaction</cite>. <cite index="2-5">Scaling up the language model improved task-agnostic, few-shot performance, sometimes reaching competitiveness with prior state-of-the-art fine-tuning approaches</cite>.

<cite index="2-8">The model achieved strong performance on translation, question-answering, cloze tasks, and tasks requiring on-the-fly reasoning or domain adaptation — unscrambling words, using novel words in sentences, performing 3-digit arithmetic</cite>. <cite index="6-2">The model demonstrated strong zero-shot and few-shot learning abilities across many tasks</cite>.

The architectural read was direct: <cite index="2-2,2-3">prior NLP systems required task-specific fine-tuning datasets of thousands or tens of thousands of examples</cite>. GPT-3 eliminated that dependency at sufficient scale. The commercial path opened. <cite index="6-7">OpenAI announced API access in June 2020</cite>, pricing inference by the token. The model became a service. The task-specific moat collapsed into a scale moat.

<cite index="13-3,13-4">Four of the eight models shipped via API: ada (medium), babbage (XL), curie (6.7B), and davinci (175B). EleutherAI announced the mapping between model sizes and API names in May 2021</cite>. Pricing tiered by parameter count. The serving cost curve began.

Sources:

• https://github.com/openai/gpt-3?tab=readme-ov-file
• https://en.wikipedia.org/wiki/GPT-3

<cite index="18-2,21-1">CommonCrawl contributed 45TB of compressed plaintext before filtering and 570GB after</cite>, <cite index="21-2">equivalent to roughly 400 billion byte-pair-encoded tokens</cite>. <cite index="20-1,20-10">CommonCrawl represented 60% of the weighted pre-training mix and supplied 410 billion tokens after filtering</cite>.

The final dataset mix pulled from five sources. <cite index="20-12">WebText2 contributed 19 billion tokens (22% weighted), Books1 contributed 12 billion tokens (8%), Books2 contributed 55 billion tokens (8%), and Wikipedia contributed 3 billion tokens (3%)</cite>. <cite index="18-7">Training ran for 300 billion tokens total. Some datasets were seen up to 3.4 times, others less than once</cite>.

<cite index="18-9,18-10">Training used Adam with β₁=0.9, β₂=0.95, ε=10⁻⁸. Gradient clipping at 1.0 global norm. Cosine decay brought learning rate to 10% of original after 260 billion tokens. Linear warmup over first 375 million tokens</cite>. <cite index="12-15">The 175B model required 3.14E23 FLOPs for training</cite>. <cite index="19-4,19-5">Estimated training cost exceeded $4.6 million on Tesla V100 cloud instances, with estimates ranging from $500K to $4.6M depending on hardware</cite>.

The data strategy: filter aggressively, weight the high-quality sources, accept limited overfitting on the premium datasets. The unit economics of pre-training at this scale set the cost floor for every model release after.

Sources:

• https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Supplemental.pdf
• https://en.wikipedia.org/wiki/GPT-3
• https://www.linkedin.com/pulse/chatgpt-dall-e-2-show-me-data-sources-dennis-layton
• https://klu.ai/glossary/openai-gpt3
• https://lambda.ai/blog/demystifying-gpt-3

<cite index="2-1,5-5">GPT-3 shipped with 175 billion parameters, 10× more than any previous non-sparse language model</cite>. <cite index="7-2">OpenAI trained eight different sizes ranging from 125 million to 175 billion parameters</cite>. The smallest model — 125M — matched BERT-Base in scale. The flagship 175B variant set the benchmark.

<cite index="6-1">Each parameter occupied 2 bytes at 16-bit precision, requiring 350GB of storage</cite>. <cite index="4-1,12-2">The 175B model used 96 attention layers with 96 heads of 128-dimension each</cite>. <cite index="10-3,12-1">The 125M model used 12 attention layers with 12 heads of 64-dimension each</cite>.

<cite index="7-1">Architecture mirrored GPT-2 — modified initialization, pre-normalization, reversible tokenization — with one exception: alternating dense and locally banded sparse attention patterns in the transformer layers</cite>. <cite index="6-2">Context window held 2,048 tokens</cite>. <cite index="12-10,12-13">Batch size scaled with parameter count. The 125M model used batch size 0.5M and learning rate 6.0×10⁻⁴. The 175B model used batch size 3.2M and learning rate 0.6×10⁻⁴</cite>.

The model family approach let OpenAI test whether validation loss scaled as a smooth power law with size. It did. The commercial read: few-shot performance improved predictably with parameter count, no fine-tuning required.

Sources:

• https://github.com/openai/gpt-3?tab=readme-ov-file
• https://deepwiki.com/openai/gpt-3/1.1-model-architecture-and-training
• https://en.wikipedia.org/wiki/GPT-3
• https://www.springboard.com/blog/data-science/machine-learning-gpt-3-open-ai/
• https://arxiv.org/pdf/2005.14165
• https://dzlab.github.io/ml/2020/07/25/gpt3-overview/
• https://lambda.ai/blog/demystifying-gpt-3

<cite index="14-5">Megatron-LM's approach allows training iterations on a model with 1 trillion parameters at 502 petaFLOP/s on 3072 GPUs, achieving per-GPU throughput of 52% of theoretical peak.</cite> <cite index="4-2,4-4">Experiments on 16 NVIDIA H20 96GB GPUs, adjusting microbatch size across various parallel configurations to maximize memory utilization, show that under the TP=2, PP=8 configuration, enhanced scheduling achieves 2.74 samples/s throughput and 92.86% MFU, maintaining a lower and more balanced memory footprint of 68 GB.</cite>

<cite index="8-2">Tensor parallelism introduces significant collective communication overheads, while pipeline parallelism suffers from synchronization inefficiencies such as pipeline bubbles.</cite> <cite index="23-5">A synergistic approach to tensor and pipeline parallelism improves training throughput by up to 12% for LLMs and 16% for MLLMs compared to existing scheduling methods.</cite>

<cite index="13-10">Data-parallel scale-out works well but suffers from two limitations: beyond a point, the per-GPU batch size becomes too small, reducing GPU utilization and increasing communication cost, and the maximum number of devices that can be used is the batch size.</cite> The capital requirement for frontier training is not GPU count alone — it is GPU count multiplied by the bandwidth topology and the scheduler's ability to compress bubble time. Sub-optimal parallelism routing can double the rack requirement.

Sources:

• https://arxiv.org/pdf/2104.04473
• https://arxiv.org/html/2510.27257v1
• https://arxiv.org/pdf/2510.27257
• https://people.eecs.berkeley.edu/~matei/papers/2021/sc_megatron_lm.pdf

<cite index="4-1">Hybrid model parallelism combining tensor parallelism and pipeline parallelism has become the dominant solution for distributed training of Large Language Models and Multimodal LLMs.</cite> <cite index="14-3">Tensor, pipeline, and data parallelism can be composed to scale to thousands of GPUs.</cite> <cite index="13-3">Neither tensor model parallelism nor pipeline model parallelism in isolation can match the performance of using both techniques in conjunction.</cite>

<cite index="14-12">Larger models split across multiple multi-GPU servers face two problems: (a) the all-reduce communication required for tensor parallelism needs to go through inter-server links, which are slower than the high-bandwidth NVLink available within a multi-GPU server, (b) a high degree of model parallelism can create small matrix multiplications, potentially decreasing GPU utilization.</cite> <cite index="13-7,13-8">Different forms of parallelism interact in non-trivial ways: the parallelization strategy has an impact on the amount of communication, the compute efficiency with which kernels are executed, as well as the idle time workers spend waiting for computation due to pipeline flushes — sub-optimal combinations of tensor and pipeline model parallelism can lead to up to 2× lower throughput, even with high-bandwidth interconnect.</cite>

<cite index="2-15,2-16">If the model grows larger, tensor parallelism can be applied within the same node, while pipeline parallelism is used across different nodes; it is crucial to ensure that the nodes participating in pipeline parallelism are within the same network rank to achieve optimal I/O performance.</cite>

Sources:

• https://arxiv.org/html/2510.27257v1
• https://arxiv.org/pdf/2104.04473
• https://people.eecs.berkeley.edu/~matei/papers/2021/sc_megatron_lm.pdf
• https://medium.com/@ming.gao.gm/distributed-model-training-at-scale-part-one-parallel-training-9a96508c741f

<cite index="3-2,3-3,3-4,3-5">The model is split by layer into chunks, each given to a device; during forward pass, each device passes intermediate activation to the next stage; during backward pass, each device passes gradient of input tensor back to the previous stage, allowing devices to compute simultaneously and increasing training throughput.</cite> <cite index="3-6">One drawback is bubble time where some devices are idle, leading to waste of computational resources.</cite>

<cite index="17-2,17-3">Pipeline bubbles dramatically reduce energy efficiency because during a bubble, processors in idle stages consume power without doing useful work — they are just waiting for data.</cite> <cite index="21-2,21-3">Zero Bubble scheduling outperforms the 1F1B schedule up to 23% in throughput under a similar memory limit, and up to 31% when memory constraint is relaxed.</cite> <cite index="19-2">In pure pipeline parallelism settings, new methods outperform 1F1B by 7% to 55% in terms of throughput.</cite>

<cite index="6-1">Pipeline parallelism can work across nodes but requires careful batch size tuning.</cite> <cite index="20-1">Pipeline parallelism often suffers from performance limitations caused by pipeline bubbles, primarily from imbalanced computation delays across batches.</cite>

Sources:

• https://colossalai.org/docs/concepts/paradigms_of_parallelism/
• https://blog.truegeometry.com/api/exploreHTML/2efd74dba2fb4931d0c626475c6afe20.exploreHTML
• https://arxiv.org/pdf/2401.10241
• https://arxiv.org/pdf/2405.15362
• https://arxiv.org/pdf/2504.14775
• https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/features/parallelisms.html

<cite index="1-6,1-7,1-8">Pipeline parallelism partitions the model vertically by layer sequence; tensor parallelism partitions horizontally by feature dimension.</cite> <cite index="2-4,2-5">In tensor parallelism, weight matrices are divided by rows or columns, allowing each GPU to perform its portion of the multiplication independently, then combining sub-results across devices.</cite>

<cite index="7-4,7-5">Tensor parallelism distributes the parameter tensor of an individual layer across GPUs, reducing model state memory usage and activation memory as per-GPU tensor sizes shrink.</cite> <cite index="12-3,12-4,12-5">Each tensor is split into multiple chunks with each shard on a separate GPU; at each step, the same mini-batch is processed independently and in parallel by each shard, followed by syncing across all GPUs — in a simple transformer layer, this leads to two all-reduces in the forward path and two in the backward path.</cite>

<cite index="7-6">The reduced per-GPU tensor size increases CPU overhead due to smaller per-GPU kernel workloads.</cite> <cite index="14-12">High degree of tensor parallelism can create small matrix multiplications, potentially decreasing GPU utilization.</cite> <cite index="6-1">Tensor parallelism works best within a single node (high bandwidth); pipeline parallelism can work across nodes but requires careful batch size tuning.</cite>

Sources:

• https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/scaling/JAX/tensor_parallel_simple.html
• https://medium.com/@ming.gao.gm/distributed-model-training-at-scale-part-one-parallel-training-9a96508c741f
• https://docs.nvidia.com/nemo-framework/user-guide/24.12/nemotoolkit/features/parallelisms.html
• https://huggingface.co/docs/accelerate/usage_guides/megatron_lm
• https://arxiv.org/pdf/2104.04473

<cite index="4-2">NVIDIA integrated Muon and other advanced higher-order optimizers—MOP, REKLS—into Megatron Core and NeMo Megatron Bridge, enabling efficient large-scale training of LLMs like Kimi K2 and Qwen3 30B on GB300 NVL72 systems with minimal throughput loss compared to AdamW</cite>. <cite index="4-3">Layer-wise distributed optimization and specialized distributed Newton-Schulz iteration modes—duplicated, distributed, blockwise—address computational and communication challenges posed by higher-order optimizers, supporting both data and tensor parallelism at massive GPU scales</cite>.

<cite index="4-8,4-9,4-10,4-11">LLM training at scale often uses tensor parallelism together with other parallelisms. TP splits individual weight matrices across multiple GPUs along specific dimensions—no single device holds the full parameter tensor. Muon optimizer's orthogonalization step requires access to the entire momentum buffer matrix. When this momentum buffer is sharded across devices, additional communication is necessary to handle sharded momentum and weights</cite>.

<cite index="21-27,21-28">Megatron-Bridge uses the distributed optimizer as the default for data-parallel training. It shards master parameters and optimizer states across data-parallel ranks, reducing model state memory usage without increasing communication overhead compared to traditional data-parallel training</cite>. <cite index="6-1">Megatron Bridge provides bidirectional Hugging Face ↔ Megatron checkpoint conversion with production-ready training recipes for popular models like Llama 3, with optimized hyperparameters and distributed training configuration</cite>.

Sources:

• https://developer.nvidia.com/blog/advancing-emerging-optimizers-for-accelerated-llm-training-with-nvidia-megatron/
• https://docs.nvidia.com/nemo/megatron-bridge/latest/performance-guide.html
• https://github.com/NVIDIA-NeMo/Megatron-Bridge

<cite index="3-3">NVIDIA Megatron-Core is an open-source PyTorch-based library with a collection of GPU-optimized techniques, cutting-edge system-level innovations, and modular APIs for training models at large scale</cite>. <cite index="10-1">It provides transformer building blocks, advanced parallelism strategies—TP (tensor), PP (pipeline), DP (data), EP (expert), CP (context)—mixed precision support (FP16, BF16, FP8, FP4), and model architectures</cite>. <cite index="3-11">First introduced in 2019, NVIDIA Megatron-LM enabled researchers and developers to use the library to further large language model advancements</cite>. <cite index="3-1">Popular LLM frameworks built on top of Megatron-LM include Colossal-AI, Hugging Face Accelerate, and NVIDIA NeMo</cite>.

<cite index="12-4,12-5,12-6">Expert parallelism in Megatron-LM distributes different experts across different GPUs in Mixture-of-Experts layers. Tokens are routed to the GPUs hosting their selected experts, computed there, then sent back, reducing memory cost</cite>. <cite index="11-4,11-5">Context parallelism splits long sequences across GPUs for efficient long-context training</cite>. <cite index="9-1">Megatron implements two pipeline parallelism schedules—one without interleaving, one with interleaving—and a default no-pipelining schedule</cite>.

<cite index="3-8">Megatron-Core v0.7 introduces fast distributed checkpointing, training throughput optimization for mixture of experts models, and enhanced scalability features like fine-grained overlapping of data parallelism gradient all-reduce with the backward pass</cite>. <cite index="5-3">Megatron-Core enabled training of a Nemotron-4 340B model at up to 6K+ H100 GPUs scale while achieving high per-GPU throughput</cite>.

Sources:

• https://developer.nvidia.com/blog/train-generative-ai-models-more-efficiently-with-new-nvidia-megatron-core-functionalities/
• https://github.com/NVIDIA/Megatron-LM
• https://docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/pipeline_parallel.html
• https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/parallelism-guide.html
• https://huggingface.co/docs/accelerate/usage_guides/megatron_lm
• https://developer.nvidia.com/megatron-core

<cite index="24-1,24-3">Megatron achieves up to 47% Model FLOP Utilization (MFU) on H100 clusters training models from 2B to 462B parameters across thousands of GPUs</cite>. <cite index="24-4">Weak scaling shows superlinear MFU improvement—41% for the smallest model to 47-48% for the largest—because larger GEMMs have higher arithmetic intensity and execute more efficiently</cite>. <cite index="24-5,24-6">Strong scaling GPT-3 (175B parameters) from 96 to 4608 H100 GPUs with constant batch size dropped MFU from 47% to 42% as communication became more exposed at larger scale</cite>.

<cite index="21-18,21-20,21-21">Common causes of low GPU FLOPS include small LLMs with small hidden size or sequence length, high multi-GPU communication variation, or fine-tuning without sequence packing. Increasing micro-batch size and reducing per-tensor sharding raises per-GPU tensor size</cite>. <cite index="21-26,21-29,21-30">Data parallelism offers optimal performance when model and activation memory fit within GPUs, minimizing communication overhead and maximizing per-GPU tensor sizes. Tensor parallelism is recommended when a model exceeds GPU memory under data-parallel mapping, but TP size should be confined to the high-bandwidth intra-node NVLink domain to limit communication overhead</cite>.

<cite index="18-6,18-7">H100 BF16 theoretical TFLOPS is 989. If actual training loop performance measured 400 TFLOPS, MFU is that ratio</cite>. <cite index="18-11">Megatron-LM uses activation recomputation, so MFU is lower than Hardware FLOP Utilization</cite>. <cite index="18-28,18-29">Cluster setup—storage IO, network IO—affects MFU. The same software may deliver different MFUs on different clusters. It is valid to compare MFU before and after optimization on a single setup, but difficult to compare across teams' clusters</cite>.

Sources:

• https://github.com/NVIDIA/Megatron-LM
• https://docs.nvidia.com/nemo/megatron-bridge/latest/performance-guide.html
• https://github.com/stas00/ml-engineering/blob/master/training/performance/README.md

<cite index="16-10,16-11">NVIDIA Megatron combines pipeline, tensor, and data parallelism—PTD-P—to train models from 2B to 462B+ parameters with 52% device throughput at the trillion-parameter tier, measured on 3072 A100 GPUs</cite>. <cite index="14-8,14-5">Tensor parallelism splits individual transformer layers across GPUs; pipeline parallelism partitions layers across servers. Tensor model parallelism needs fast intra-server NVLink; crossing to inter-server links slows all-reduce communication and shrinks GEMMs, dropping utilization</cite>.

<cite index="11-3,11-9">Tensor parallelism splits individual model layers across GPUs, recommended for large hidden dimensions. Pipeline parallelism splits transformer layers vertically by depth across pipeline stages</cite>. <cite index="15-3,15-4">Tensor parallelism distributes the parameter tensor of an individual layer across GPUs, reducing model state memory and activation memory as per-GPU tensor sizes shrink</cite>. <cite index="12-1,12-2,12-3">Sequence parallelism reduces activation memory footprint without additional communication, applicable only when using TP. It replaces all-reduce with reduce-scatter and all-gather</cite>.

<cite index="7-1">Throughput at scale required efficient kernel implementations to keep computation compute-bound rather than memory-bound, smart partitioning to reduce bytes sent over network links while limiting device idle periods, and fast hardware</cite>. <cite index="7-6,7-7">Distributed training at scale is communication-intensive. Training a trillion-parameter model on 3072 GPUs used 892 GB/s effective bisection bandwidth for pipeline-parallel communication and 13 TB/s for data-parallel communication</cite>.

Sources:

• https://arxiv.org/pdf/2104.04473
• https://people.eecs.berkeley.edu/~matei/papers/2021/sc_megatron_lm.pdf
• https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/parallelism-guide.html
• https://docs.nvidia.com/nemo/megatron-bridge/0.2.0/parallelisms.html
• https://huggingface.co/docs/accelerate/usage_guides/megatron_lm

<cite index="1-5">Empirical measurements identified distinct power signatures between transformer and CNN architectures, with transformers showing characteristic fluctuations that may impact grid stability.</cite> <cite index="1-7,1-8">Accurately quantifying energy use of AI training is critical for infrastructure planning, carbon accounting, and sustainable datacenter operation—few studies have directly measured power consumption of production workloads on contemporary hardware.</cite>

<cite index="27-1,27-2">Measurements of AI workloads at 0.1-second resolution for training, fine-tuning and inference jobs used ML-Commons benchmarks for model training and fine-tuning, and vLLM benchmarks for inference, enabling reproducible and standardized workload profiling.</cite> <cite index="4-29">Academic workloads including training and batched inference spent 6% and 7% of total energy consumption in execution-idle states respectively.</cite>

The load profile matters for power provisioning. Transformers produce spiky demand; CNNs settle into steadier draw. <cite index="27-4,27-5">Power profiles scaled to whole-facility-level using bottom-up, event-driven datacenter energy model capture realistic temporal fluctuations driven by AI workloads and user-behavior, informing infrastructure planning for grid connection and on-site energy generation.</cite>

If you run mixed workloads the peak demand is higher than the average would predict. You pay for the peak in the shell, you pay for the average in the meter.

Sources:

• https://www.researchgate.net/scientific-contributions/Emma-Strubell-2057231155
• https://arxiv.org/pdf/2604.07345
• https://arxiv.org/html/2604.04745v1

<cite index="8-2,8-4">Controlled sweeps on A6000 GPUs decomposed inference costs into prefill and decode energy—decoding is more energy intensive per token than prefill, with energy intensity scaling linearly even for short generations and small batch sizes using the vLLM framework.</cite> <cite index="8-3">At small batch sizes and input sequence lengths, energy intensity scales sub-linearly with increasing input sequence lengths.</cite>

<cite index="6-3">Inference is where real-world, everyday energy consumption happens—especially as LLMs integrate into consumer devices, chatbots, and various applications.</cite> <cite index="9-2,9-3">Estimating energy and emissions of ML models remains under-explored, gaining traction since Strubell et al.'s 2019 article—since then most studies focused on estimating energy consumed during the training phase.</cite>

The cost structure at the token level: prefill processes the prompt in parallel, decode generates tokens autoregressively. Decode burns more watts per token but you serve both. <cite index="8-8,8-9">Energy consumption comparison across different GPUs for inference with PyTorch and vLLM backends shows variation—for each GPU, PyTorch with and without compilation, and vLLM with and without CUDA Graph serialization produce different energy profiles.</cite>

The serving economics hinge on this split. You cannot price inference as a single rate—you price prefill and decode separately or you misprice the workload mix.

Sources:

• https://aclanthology.org/2025.acl-long.1563.pdf
• https://luiscruz.github.io/course_sustainableSE/2025/p1_measuring_software/g6_llms_energy_consumption.html
• https://arxiv.org/pdf/2311.16863

<cite index="20-1">Empirical measurements from Brookhaven National Laboratory on 8-GPU H100 systems found actual power draw consistently remains well below thermal design TDP, with even computationally intensive workloads drawing on average no more than 76% of TDP.</cite> <cite index="20-8,20-9">The measured systems were dedicated DGX accelerated AI training servers with 8 NVIDIA H100-SXM5-80GB GPUs, with manufacturer rated TDP of 10.2 kW.</cite>

<cite index="3-2">An architecture-specific model calibrated to floating-point operations predicts energy consumption with 11.4% mean absolute percentage error, significantly outperforming TDP-based approaches.</cite> <cite index="22-13,22-14">A common error is to take TDP as the actual power draw, but that is rarely achieved—producing accurate estimates for real workloads requires measuring actual power draw under realistic conditions.</cite>

<cite index="20-4,20-5">The measurements analyzed MLPerf® Training v4.0 submissions with a mixture of workloads and training configurations, including image classification (Resnet) and NLP (GPT3-175 and Llama).</cite> <cite index="22-20,22-21">Workload configuration had significant impact on energy usage—using higher batch sizes caused higher instantaneous power demand, but used less energy overall.</cite>

This matters for capex planning. If you provision to TDP you overbuild by 24%. If you model from TDP you overprice the watt-hour cost per token.

Sources:

• https://arxiv.org/pdf/2506.14551
• https://www.devsustainability.com/green-ai/how-useful-is-gpu-manufacturer-tdp

<cite index="16-2,16-3">Strubell et al. (2019) quantified energy costs for NLP model training: training BERT-scale transformers consumed substantial electricity due to computational resources required for tensor processing hardware, with financial costs driven by hardware, electricity, and cloud compute time.</cite> <cite index="11-1">The study reported training one large NLP model can emit over 600,000 pounds of CO₂ in extreme cases.</cite>

The measurement method: <cite index="5-2,5-3">Strubell estimated energy consumption by collecting reported time to train models and average GPU power consumption, then factored in datacenter cooling using Power Usage Effectiveness (PUE) coefficient of 1.58 as of 2018.</cite>

The development multiplier is what most cost models miss. <cite index="10-10,10-11">Based on development logs, Strubell factored in 4,789 jobs including 123 hyperparameter grid searches—training the final model took 120 hours, but full R&D required 239,942 hours of training time.</cite> <cite index="15-19,15-20,15-21">R&D costs multiply single-run costs by thousands of times because developing new models requires retraining to evaluate different architecture variants and hyperparameters.</cite>

<cite index="14-2,14-5">The study revealed training a single BERT model can emit as much CO₂ as a trans-Atlantic flight, and computational costs of NLP models double every 3-4 months.</cite> This establishes the baseline TCO equation: you price the single run, then you price the search that produced it.

Sources:

• https://aclanthology.org/P19-1355/
• https://luiscruz.github.io/green-ai/publications/2019-11-strubell-energy.html
• https://ahmdtaha.medium.com/energy-and-policy-considerations-for-deep-learning-in-nlp-ce490ffdc209
• https://knowledge.les-enovateurs.com/resources/2019-06-05-energy-policy-considerations-deep-learning-nlp/

<cite index="8-5,8-6,8-7">Low-precision with FP8 doubles the Tensor Core throughput (e.g. 989 TFLOPS with FP16 and 1978 TFLOPS with FP8), but trades off accuracy by using fewer bits to represent floating point numbers.</cite> FlashAttention-3 leverages Hopper-specific features: Tensor Memory Accelerator (TMA) for faster data transfer and FP8 precision.

<cite index="2-19">FlashAttention-2 ROCm CK backend currently supports: MI200x, MI250x, MI300x, MI355x, and RDNA 3/4 GPUs.</cite> The AMD path uses composable_kernel or Triton backends. <cite index="2-8,2-9,2-10">Ampere, Ada, or Hopper GPUs (e.g., A100, RTX 3090, RTX 4090, H100) are supported. For Turing GPUs (T4, RTX 2080), see the separate flash-attention-turing repo. Datatype fp16 and bf16 (bf16 requires Ampere, Ada, or Hopper GPUs).</cite>

The procurement read: FlashAttention performance scales with GPU generation. A100 hits 72% utilization with FlashAttention-2; H100 needs FlashAttention-3 to reach comparable efficiency. Older Turing cards require separate implementation. Serving cost per token falls as newer hardware + newer FlashAttention versions extract more utilization from silicon.

Sources:

• https://tridao.me/blog/2024/flash3/
• https://github.com/Dao-AILab/flash-attention

<cite index="8-10,8-11,8-12">Attention, as a core layer of the ubiquitous Transformer architecture, is a bottleneck for large language models and long-context applications. FlashAttention (and FlashAttention-2) pioneered an approach to speed up attention on GPUs by minimizing memory reads/writes, and is now used by most libraries to accelerate Transformer training and inference. This has contributed to a massive increase in LLM context length in the last two years, from 2-4K (GPT-3, OPT) to 128K (GPT-4), or even 1M (Llama 3).</cite>

<cite index="9-3">The attention layer is the main bottleneck in scaling to longer sequences, as its runtime and memory increase quadratically in the sequence length.</cite> FlashAttention removes the quadratic memory wall. The read: longer context windows require memory-efficient attention or approximate methods that degrade quality. FlashAttention delivers exact attention at linear memory cost.

The cost implication: models trained or served with FlashAttention can hold longer contexts in the same memory envelope, which translates to higher throughput or lower hardware requirements for equivalent workload. The alternative was sparse or approximate attention methods that traded quality for speed.

Sources:

• https://tridao.me/blog/2024/flash3/
• https://arxiv.org/abs/2307.08691

<cite index="6-1,6-2">FlashAttention was first published by Tri Dao in May 2022 and it had a deep impact in the large language models space. Most open models you've heard of (RedPajama, MPT, LLaMA, Falcon, etc) all leverage it for faster inference.</cite> The implementation is available under permissive license. <cite index="2-4,2-5">FlashAttention and FlashAttention-2 are free to use and modify. Please cite and credit FlashAttention if you use it.</cite>

The successor versions target newer GPU generations. <cite index="2-6,2-7">FlashAttention-3 is optimized for Hopper GPUs (e.g. H100).</cite> <cite index="7-1">FlashAttention-2 achieves poor utilization on newer GPUs relative to optimized matrix-multiplication (GEMM) kernels, such as 35% vs. 80-90% on the Hopper H100 GPU.</cite> FlashAttention-3 closes the gap by using Hopper-specific features including asynchrony and FP8 low-precision. <cite index="9-1">When used end-to-end to train GPT-style models, FlashAttention-2 reaches training speed of up to 225 TFLOPs/s per A100 GPU (72% model FLOPs utilization).</cite>

The impact on serving cost is indirect but structural: reduced memory footprint permits longer context windows and higher batch throughput per GPU.

Sources:

• https://www.latent.space/p/flashattention
• https://github.com/Dao-AILab/flash-attention
• https://tridao.me/publications/flash3/flash3.pdf

<cite index="1-4">FlashAttention is IO-aware—accounting for reads and writes between levels of GPU memory.</cite> Standard attention implementations move data between HBM (high-bandwidth memory) and on-chip SRAM inefficiently. <cite index="1-1,1-2">The A100 GPU has 40-80GB of HBM with bandwidth 1.5-2.0TB/s and 192KB of on-chip SRAM per streaming multiprocessor with bandwidth estimated around 19TB/s. The on-chip SRAM is an order of magnitude faster than HBM but many orders of magnitude smaller in size.</cite>

<cite index="9-4">FlashAttention exploits the asymmetric GPU memory hierarchy to bring significant memory saving (linear instead of quadratic) and runtime speedup (2-4× compared to optimized baselines), with no approximation.</cite> The approach reorders attention computation and uses tiling plus recomputation to cut memory reads/writes. <cite index="6-9,6-10,6-11">Standard attention memory usage is quadratic with sequence length (i.e. O(N^2)). FlashAttention is sub-quadratic at O(N).</cite>

<cite index="8-11">FlashAttention (and FlashAttention-2) pioneered an approach to speed up attention on GPUs by minimizing memory reads/writes, and is now used by most libraries to accelerate Transformer training and inference.</cite> The original paper was authored by Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré at Stanford and University at Buffalo, published May 2022.

Sources:

• https://arxiv.org/abs/2205.14135
• https://arxiv.org/pdf/2205.14135
• https://arxiv.org/abs/2307.08691

<cite index="4-11">Compression techniques are often responsible for amplifying model bias.</cite> <cite index="1-6,4-5">Work addresses the problem of compression-induced bias, highlighting the need to direct compression research toward model fairness in future works.</cite> <cite index="4-9">Quantization and pruning need to be further studied in biometrics applications.</cite>

The cost savings from compression are real. The bias amplification is also real. Quantization rounds weights; rounding can collapse distinctions the model learned during training. Pruning removes parameters; some of those parameters may have encoded corrections for distributional skew. Distillation imitates the teacher's output; if the teacher has bias, the student inherits it and may amplify it further during the imitation process.

<cite index="1-5">Compression is particularly important in biometric applications due to deployment on edge devices with limited resource availability for real-time use.</cite> Edge deployment is where the economics favor compression most heavily—and where the regulatory and reputational cost of bias is hardest to audit. A procurement decision that prices only the compute savings and ignores the bias-amplification liability is mispricing the risk. The IRR calculation should include the expected cost of remediation if the compressed model fails a fairness audit post-deployment.

Sources:

• https://www.sciencedirect.com/science/article/pii/S1566253524004354
• https://arxiv.org/pdf/2401.10139

<cite index="3-3,3-4,3-5">Knowledge distillation trains a student network to imitate a teacher network; the student is usually smaller and shallower, and the trained student model should be less computationally complex than the teacher.</cite> <cite index="4-8">Knowledge distillation methods are widely deployed in the biometrics field.</cite> <cite index="2-7,2-8">From a knowledge transfer perspective, a smaller network is trained to learn from the original model, aiming to replicate its performance.</cite>

Distillation lets you price the inference layer separately from the training layer. You pay the capex and power cost to train the frontier model once, then serve from a smaller model that approximates the teacher's output distribution. The economics depend on how many queries you can route to the distilled model before quality degradation forces a fallback to the teacher.

<cite index="8-3,8-6">Three primary LLM compression approaches are knowledge distillation, model quantization, and model pruning.</cite> The combined approach—distill, then quantize—compounds the savings. The cost curve for serving a distilled 7B model quantized to INT8 is structurally different from serving the original 70B model, even if both hit the same benchmark. The capex ROI calculation has to account for the one-time distillation cost and the recurring savings per token served.

Sources:

• https://www.sciencedirect.com/science/article/abs/pii/S0925231221010894
• https://www.sciencedirect.com/science/article/pii/S1566253524004354
• https://www.frontiersin.org/journals/robotics-and-ai/articles/10.3389/frobt.2025.1518965/full
• https://arxiv.org/pdf/2505.02309

<cite index="9-14">8-bit quantization of parameters can result in significant speed-up with minimal loss of accuracy.</cite> <cite index="10-7">When quantizing a full-precision model (float32) to 8-bit integers, memory cost reduces by a factor of four.</cite> <cite index="9-18">Some methods propose a three-stage compression pipeline: pruning, quantization, and encoding.</cite>

The price floor for serving a quantized model is set by how many bits you store per parameter and how many bits you move per inference pass. Moving from FP32 to INT8 cuts both memory occupancy and bandwidth cost by 4×. The next compression tier—INT4, binary weights—trades off accuracy recovery cost against further memory savings. <cite index="7-10">Future directions include pruning algorithms that do not require fine-tuning and fully quantized model compression techniques.</cite>

The deployment decision is whether the accuracy drop from quantization costs more in application-layer fallback than the savings in compute. For LLM inference at scale, the breakeven is already past 8-bit for most workloads. The frontier is sub-8-bit quantization that holds quality on domain-specific evals without retraining the base model. That determines the next depreciation cycle for GPU fleets optimized for FP16.

Sources:

• https://arxiv.org/pdf/1710.09282
• https://arxiv.org/pdf/2402.05964
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11965593/

<cite index="2-3,7-8">The literature categorizes model compression into four domains: pruning, quantization, low-rank decomposition, and knowledge distillation.</cite> <cite index="10-4">Pruning removes redundant components—blocks, attention heads, FFN layers, or individual parameters.</cite> <cite index="2-6">Quantization reduces the memory footprint of each parameter without decreasing the total number of parameters.</cite> <cite index="9-9">It compresses the network by reducing the number of bits required to represent each weight.</cite> <cite index="10-6,10-7">Quantization represents weights and features with lower bits; quantizing a float32 model to 8-bit integers cuts memory cost by 4×.</cite> <cite index="3-2,3-3">Knowledge distillation generates a simpler compressed model that functions as well as a larger model by training a student network to imitate a teacher network.</cite>

<cite index="5-2">Large models require high memory and computational power during inference, posing challenges for practical deployment.</cite> <cite index="8-5">Compression techniques reduce model size and inference cost while preserving accuracy.</cite> <cite index="9-3">Low-rank factorization provides an end-to-end pipeline and can be implemented in CPU/GPU environments straightforwardly.</cite> <cite index="4-6">Compression methods can lead to competitive results when combined in a framework.</cite> The cost curve for serving is set by the memory-bandwidth product and the compute utilization per dollar. Compression shifts both. The next wave of deployment economics depends on which technique compounds fastest at scale.

Sources:

• https://www.frontiersin.org/journals/robotics-and-ai/articles/10.3389/frobt.2025.1518965/full
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11965593/
• https://arxiv.org/pdf/1710.09282
• https://www.sciencedirect.com/science/article/pii/S1566253524004354
• https://arxiv.org/pdf/2402.05964

<cite index="12-3">Above 100M tokens per month, self-hosting almost always wins on unit economics.</cite> <cite index="3-7">The crossover point typically occurs at 10-50 million tokens monthly, depending on model complexity.</cite> But volume is not the only lever. <cite index="10-5,10-6">For self-hosted LLM inference, demand variance, not volume, drives the economics. The Erlang-C model shows exactly how cost scales with the gap between peak and average load.</cite>

<cite index="11-1,11-4,11-5,11-6">You pay for the GPU whether it's serving requests or sitting idle. That's fine at 70–80% utilization. It's painful at 20–30%, which is exactly what most teams face during off-peak hours or early product stages.</cite> <cite index="11-8">At $2.00 per hour for an H100, 12 hours of idle capacity per day costs over $700 per month in pure waste.</cite> <cite index="17-9">Self-hosted breakeven requires 50%+ GPU utilization for 7B models, 10%+ for 13B models.</cite>

<cite index="13-4,13-5">H200 with 141GB HBM3e is now widely available ($30-40K purchase, $2.15-6.00/hr cloud), enabling single-GPU serving of 70B models that previously required two H100s. H100 cloud prices dropped to $1.49-3.90/hr (down from $7-8/hr).</cite> <cite index="17-18">Hyperbolic offers H100 at $1.49 per hour—the current market low.</cite> The cost floor compresses each quarter.

Sources:

• https://www.spheron.network/blog/ai-inference-cost-economics-2026/
• https://www.getmonetizely.com/articles/the-economics-of-large-language-models-how-to-balance-token-costs-against-business-value-in-ai-pricing
• https://www.hebbia.com/blog/the-hidden-economics-of-llm-inference
• https://www.gmicloud.ai/en/blog/comparing-gpu-pricing-large-language-models
• https://introl.com/blog/cost-per-token-llm-inference-optimization
• https://introl.com/blog/inference-unit-economics-true-cost-per-million-tokens-guide

<cite index="10-12">FlashAttention was designed around the insight that HBM access dominates attention cost; vLLM's roofline analysis treats it as the foundational design constraint; recent empirical work confirms that even large-batch decode remains memory-bound with most GPU compute underutilized.</cite> <cite index="6-1,6-2">A theoretical model addresses the economic trade-off between cost per token versus serial token generation speed when deploying LLMs for inference at scale, taking into account arithmetic, memory bandwidth, network bandwidth and latency constraints; and optimizes over different parallelism setups and batch sizes.</cite>

<cite index="8-1,8-2,8-3">Achieving the best possible token latency only costs around three times as much as the minimum possible cost per token arising from arithmetic alone. This result is not realistic: in actual inference setups the Pareto frontier of speed versus cost can range across much wider values for cost per token. This is entirely because of network bandwidth constraints.</cite>

<cite index="11-13,11-14">LLM inference cost comes down to three variables: model size, throughput, and GPU memory efficiency. Those three factors tell you more than any hourly rate table.</cite> <cite index="18-8">GPU cost efficiency is defined as the number of input and output tokens processed per dollar cost (T/$) of on-demand cloud GPUs.</cite>

Sources:

• https://www.hebbia.com/blog/the-hidden-economics-of-llm-inference
• https://arxiv.org/pdf/2506.04645
• https://arxiv.org/html/2506.04645v1
• https://www.gmicloud.ai/en/blog/comparing-gpu-pricing-large-language-models
• https://arxiv.org/pdf/2404.14527

<cite index="5-1,5-2">An economic framework analyzing optimal LLM pricing captures several key features: variable operational costs of processing input and output tokens; the ability to customize models through fine-tuning; and high-dimensional user heterogeneity in terms of task requirements and error sensitivity.</cite> <cite index="7-6,7-13">The optimal mechanism can be implemented through menus of two-part tariffs, with higher markups for more intensive users.</cite> This is the Yale/ACM EC 2025 model from Bergemann, Bonatti, et al.

<cite index="7-4,7-11">The optimal pricing structure depends on whether token allocation across tasks is contractible and whether users face scale constraints.</cite> <cite index="4-1">The model assumes marginal costs of processing tokens are constant but can vary across different token types, with the cost of input tokens cx > 0, output tokens cy > 0, and fine-tuning tokens cz > 0.</cite> <cite index="7-14">The results rationalize observed industry practices such as tiered pricing based on model customization and usage levels.</cite>

<cite index="3-8,3-9,3-10,3-11">Legal document analysis SaaS charges $99 per month for 50 AI-analyzed contracts. Token cost averages $0.40 per contract ($20 total). Gross margin: 80%. Value delivered: 2-3 hours saved per contract ($300-$450 value).</cite> The price captures the value. The cost is incidental.

Sources:

• https://cowles.yale.edu/sites/default/files/2025-02/d2425.pdf
• https://dl.acm.org/doi/10.1145/3736252.3742625
• https://elischolar.library.yale.edu/cowles-discussion-paper-series/2834/
• https://www.getmonetizely.com/articles/the-economics-of-large-language-models-how-to-balance-token-costs-against-business-value-in-ai-pricing

<cite index="1-4">Output tokens cost more than input tokens because the generative process requires significantly more computational power than encoding and understanding text.</cite> The differential is structural, not discretionary. <cite index="3-6">Current pricing shows GPT-4 Turbo at $0.01 per 1K input tokens and $0.03 per 1K output tokens.</cite> <cite index="1-10,1-12">Non-English languages carry a hidden cost multiplier due to tokenization inefficiency; programming code, JSON structures, and formatted text consume more tokens than plain prose because tokenizers must represent syntax elements, indentation, and special characters.</cite>

<cite index="9-3,9-4">The market launched with OpenAI's GPT-3 API at $60 per million input tokens in June 2020. By early 2026, economy-tier models such as Gemini 2.0 Flash offer comparable or superior quality at $0.10 per million tokens—a 600-fold price decline in under six years.</cite> <cite index="17-1,17-6">GPT-4 equivalent performance now costs $0.40 per million tokens versus $20 in late 2022.</cite> <cite index="17-8">DeepSeek disrupted the market with 90% lower pricing than incumbents.</cite>

<cite index="2-7,2-10">At moderate usage (1,000 documents per day, 667 tokens each input and output), cost runs approximately $2.60 per day. At a million documents per day, that scales to $2,600 per day or approximately $1 million annually.</cite> The unit economics hold. The volume is the variable.

Sources:

• https://tetrate.io/learn/ai/cost-per-token
• https://medium.com/emalpha/the-economics-of-large-language-models-2671985b621c
• https://www.getmonetizely.com/articles/the-economics-of-large-language-models-how-to-balance-token-costs-against-business-value-in-ai-pricing
• https://arxiv.org/pdf/2603.28576
• https://introl.com/blog/inference-unit-economics-true-cost-per-million-tokens-guide

Chinchilla-optimal models achieve a given quality target with fewer parameters than undertrained alternatives. Fewer parameters mean lower memory footprint, reduced bandwidth, and faster inference per request. For a 70B model matching the capability of a 280B model, the per-query cost drops ~4×—and that multiplier applies to every inference over the model's production life.

This compounds. A model trained over three months may serve requests for two to three years. If cumulative inference FLOPs are 10–100× training FLOPs, then inference cost dominates total cost of ownership. The training budget is sunk; the serving cost accrues continuously. Chinchilla's prescription—spend more on training to shrink the model—maps directly to lower opex once deployed.

Inference-aware extensions push the logic further: train models even smaller and longer than Chinchilla to maximize deployment efficiency. This trades capex (one-time training cost) for opex compression (per-token serving cost). Grouped-query attention, quantization, and speculative decoding amplify the benefit by reducing effective cost per token at the architecture and runtime layers. The moat is not the model; it is the cost curve underneath.

Sources:

• https://mbrenndoerfer.com/writing/chinchilla-scaling-laws-compute-optimal-training-resource-allocation
• https://mbrenndoerfer.com/writing/chinchilla-scaling-laws-compute-optimal-llm-training
• https://mbrenndoerfer.com/writing/inference-scaling-llm-deployment-optimization

Hoffmann et al. presented three approaches; the parametric scaling law (approach 3) is the most generalizable. Unlike the first two methods—which apply only at compute-optimal points—the parametric law estimates cross-entropy loss for any choice of parameter count N and dataset size D. This makes it relevant when training constraints or inference economics force deviation from Chinchilla-optimal.

Epoch AI (2024) attempted replication and discovered the originally published parametric coefficients were poorly fit. Multiple subsequent studies—Sardana on inference cost, Faiz on trade-offs—relied on those coefficients, so downstream cost models may be systematically off. The parametric form itself holds; the fitted constants are suspect. This matters for procurement: if the loss curve is shallower or steeper than Hoffmann's coefficients imply, the IRR on incremental training tokens or additional parameters shifts.

The law also neglects architecture shape (width-to-depth ratio) and quality-adjusted data mix. Recent extensions model inference latency and data quality directly. High-quality corpora shift the optimal allocation toward larger models given the same token count. These are not edge cases—they are the variables that route capex in production training runs.

Sources:

• https://epoch.ai/blog/chinchilla-scaling-a-replication-attempt
• https://www.emergentmind.com/topics/chinchilla-s-scaling-laws
• https://lifearchitect.ai/chinchilla/

Sardana et al. (2023) and subsequent work extend Chinchilla by pricing in inference volume. The original scaling law optimizes training FLOPs only. But models deployed at scale serve requests that dwarf the one-time training cost. At ~1B inference requests, the optimal allocation moves toward smaller models trained longer than Chinchilla prescribes—trading higher training cost for permanently lower per-token serving cost.

The math: if inference consumes 10× the training FLOPs over the model's production life, then shaving 10% off inference cost justifies doubling training duration. For inference-heavy workloads, the token-to-parameter ratio can rise from 20:1 to 60:1 or higher. LLaMA models demonstrate this: trained well beyond Chinchilla-optimal on token count to compress serving cost for open deployment.

This bifurcates the market. Closed frontier labs serving hundreds of millions of users price models for inference load and train smaller + longer. Open-weight releases follow the same logic when the distribution path assumes self-hosting. Chinchilla-optimal holds only when training cost dominates total cost—a condition that fails for any model crossing ~100M cumulative requests. The curve is not universal; it bends with the inference multiplier.

Sources:

• https://mbrenndoerfer.com/writing/chinchilla-scaling-laws-compute-optimal-llm-training
• https://mbrenndoerfer.com/writing/inference-scaling-llm-deployment-optimization
• https://arxiv.org/abs/2401.00448
• https://arxiv.org/html/2401.00448v3
• https://lifearchitect.ai/chinchilla/

Hoffmann et al. (2022) trained over 400 models and concluded that for a fixed compute budget, parameter count and training tokens should scale proportionally—not the parameter-heavy allocation Kaplan (2020) prescribed. The 70B Chinchilla model, trained on 1.4T tokens, outperformed the 280B Gopher model trained on 300B tokens using the same FLOPs. This implies a ~20:1 token-to-parameter ratio at compute-optimal, not the <2:1 ratio embedded in GPT-3 (175B parameters, 300B tokens).

The revision matters because most frontier models through 2022 were undertrained. GPT-3 should have used 3.5T tokens to justify 175B parameters, or dropped to 15B parameters given the 300B token corpus—an 11× mismatch either direction. The Chinchilla finding redirected capex: smaller models trained longer deliver equivalent quality at lower serving cost. A 70B model costs roughly 4× less per inference than a 280B model of identical capability.

This is not aesthetic preference. It is price discovery on the training–inference trade. Training is one-time; inference compounds. For models serving billions of requests, the cumulative inference FLOPs exceed training FLOPs by 10–100×. Chinchilla-optimal models front-load training cost to compress the ongoing serving bill.

Sources:

• https://www.glennklockwood.com/garden/scaling-laws
• https://aiwiki.ai/wiki/chinchilla_scaling
• https://lifearchitect.ai/chinchilla/
• https://epoch.ai/blog/chinchilla-scaling-a-replication-attempt

<cite index="17-1,17-2">Dot-product attention FLOPs only become dominant during training once T > 8D — for D ~ 8k, this is around 64K tokens.</cite> <cite index="17-3,17-7">As MLP size increases, attention FLOPs become less critical; the quadratic cost of attention is not a huge obstacle to longer context for large models.</cite> <cite index="17-9,17-10">For smaller models like Gemma-27B with D=4608, attention becomes dominant around 37k sequence lengths.</cite>

This matters for cost curves. <cite index="22-2,22-3">Incremental inference is often slow due to the memory-bandwidth cost of repeatedly loading large keys and values tensors; multi-query attention shares keys and values across all heads, reducing memory bandwidth requirements.</cite> Noam Shazeer (one of the original eight authors) published that variant in 2019.

<cite index="14-6,14-7">Chinchilla scaling laws (D=20P) are optimal in one specific sense: when 1,000 GPUs for 1 hour and 1 GPU for 1,000 hours cost the same, and the goal is maximizing performance while minimizing GPU-hours.</cite> The depreciation schedule is steeper when the next-generation undercut is already on the roadmap.

Sources:

• https://jax-ml.github.io/scaling-book/transformers/
• https://arxiv.org/pdf/1911.02150
• https://training.continuumlabs.ai/infrastructure/data-and-memory/transformer-training-costs

<cite index="9-5">Vaswani et al. estimated training FLOPs by multiplying training time, number of GPUs used, and the sustained single-precision floating-point capacity of each GPU.</cite> <cite index="9-2,9-9">They used values of 2.8, 3.7, 6.0, and 9.5 TFLOPS for K80, K40, M40, and P100 respectively.</cite> <cite index="9-11,9-12">The Transformer trained significantly faster than recurrent or convolutional architectures and achieved new state of the art on WMT 2014 English-to-German and English-to-French translation.</cite>

The industry standard formula today: <cite index="10-5">Training FLOPs ≈ 6 × P × T_tokens, where P is parameter count and T_tokens is total training tokens.</cite> <cite index="14-9,14-10,14-11">GPT-NeoX achieves 150 TFLOP/s/A100 with normal attention and 180 with Flash Attention; Megatron-DS reports 137-163 TFLOP/s/A100 — as a rule of thumb, you should achieve approximately 120 TFLOP/s/A100.</cite>

<cite index="16-2,16-3">GPT-3 was trained on half a trillion words with 175 billion parameters — it would take 355 GPU-years and cost at least $4.6M for a single training run.</cite> <cite index="11-2">Popular time estimates based on FLOPs are poor estimates of wall-clock time; a more accurate proxy is based on memory copies.</cite>

Sources:

• https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf
• https://phonism.github.io/LLMNotes/en/transformer-part1-fundamentals/
• https://training.continuumlabs.ai/infrastructure/data-and-memory/transformer-training-costs
• https://arxiv.org/pdf/2302.01107
• https://arxiv.org/pdf/2406.18922

<cite index="20-3,20-4">In encoder-decoder attention layers, queries come from the previous decoder layer and keys/values come from encoder output — this allows every position in the decoder to attend over all positions in the input sequence.</cite> <cite index="20-6,20-7,20-8">The encoder contains self-attention layers where keys, values, and queries all come from the output of the previous encoder layer — each position can attend to all positions in the previous layer.</cite> <cite index="20-9">Self-attention in the decoder allows each position to attend to all positions up to and including that position.</cite>

This three-way use of the same mechanism — encoder self-attention, decoder self-attention, encoder-decoder cross-attention — defines the seq2seq configuration. <cite index="5-17,5-21">The original architecture consists of encoder and decoder parts for machine translation; one main feature is parallelized sequence processing which RNNs lack.</cite> Later variants split: <cite index="5-18,5-19,5-20">encoder-only (e.g. BERT) and decoder-only (e.g. GPT) architectures emerged.</cite>

<cite index="26-2,26-3">Multi-head attention allows the model to focus simultaneously on different parts of the input, though recent work shows most heads learn simple, often redundant, positional patterns.</cite>

Sources:

• https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf
• https://github.com/brandokoch/attention-is-all-you-need-paper
• https://arxiv.org/pdf/2002.10260

<cite index="1-4">The Transformer architecture is based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.</cite> Vaswani et al. published this in 2017 at NeurIPS. <cite index="1-5">The architecture proved superior in quality while being more parallelizable and requiring significantly less time to train</cite> on machine translation tasks.

<cite index="3-7">The original model used d_model = 512.</cite> <cite index="3-9">The encoder and decoder are each composed of N = 6 identical layers.</cite> <cite index="3-2,3-4">Each encoder layer contains two sub-layers: a multi-head self-attention mechanism and a position-wise fully connected feed-forward network.</cite> <cite index="3-5,3-6">Residual connections wrap each sub-layer, followed by layer normalization — the output is LayerNorm(x + Sublayer(x)).</cite>

<cite index="3-10">The decoder adds a third sub-layer that performs multi-head attention over the encoder output.</cite> <cite index="2-9">The original paper used h=8 attention heads, with each head using dimension dk = dv = 64.</cite> <cite index="3-16">Single-head attention was 0.9 BLEU worse than the best setting.</cite>

<cite index="19-3">The key benefit: no recurrent units, therefore less training time than LSTM architectures.</cite> The model layer has been commoditizing ever since.

Sources:

• https://arxiv.org/abs/1706.03762
• https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf
• https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)
• https://towardsai.net/p/machine-learning/attention-is-all-you-need-a-deep-dive-into-the-revolutionary-transformer-architecture

Pearce & Song (arXiv:2406.12907, June 2024) reconciled the conflicting estimates from Kaplan (2020) and Chinchilla (2022). Kaplan's $N \propto C^{0.73}$ versus Chinchilla's $N \propto C^{0.50}$ was primarily due to two factors: (1) Kaplan counted non-embedding parameters rather than total parameters, and (2) Kaplan's analysis was performed at small scale, where embedding weights constitute a larger fraction of total parameters. When Pearce & Song simulated the Chinchilla study under Kaplan's conditions—non-embedding count, small models—they recovered biased coefficients close to Kaplan's original estimates.

The revision confirms that Chinchilla's scaling coefficients are the more accurate baseline for large-scale training. The embedding-parameter treatment alone accounts for a substantial portion of the discrepancy. The paper does not challenge the functional form—power laws still hold—but it tightens the exponent that governs compute allocation. For hyperscaler procurement, the implication is direct: at fixed budget, invest equally in model size and training data. The Kaplan allocation overweighted parameters and led to undertrained models, which Chinchilla corrected.

Sources:

• https://arxiv.org/pdf/2406.12907

Henighan et al. (arXiv:2010.14701, October 2020) extended Kaplan's language-model scaling laws to four non-language domains: generative image modeling, video modeling, multimodal image↔text, and mathematical problem solving. In all cases, autoregressive Transformers improved smoothly as model size and compute increased, following the same power-law-plus-constant functional form. The optimal model size at fixed compute followed a power law with exponents nearly universal across domains.

The study also tested LSTMs and found they scaled similarly to Transformers—same functional form, worse constant. Architecture mattered for absolute performance but not for the shape of the scaling curve. The authors finetuned pretrained generative image models on ImageNet classification and found that classification performance continued to scale as a power law even after the generative loss approached its irreducible floor. Larger pretrained models finetuned faster and reached better downstream performance.

The work demonstrated that scaling laws are not an artifact of language data or Transformer inductive biases. The compute-performance relationship appears to be a property of autoregressive density estimation over high-dimensional data, independent of modality. This matters for cost-curve modeling: the same allocation equations apply whether the workload is text, image, or video generation.

Sources:

• https://arxiv.org/abs/2010.14701
• https://arxiv.org/pdf/2010.14701

Hoffmann et al. (arXiv:2203.15556, March 2022) trained over 400 models from 70M to 16B+ parameters on 5B to 500B tokens and concluded that compute-optimal training requires model size and training tokens to scale at equal rates. For every doubling of parameters, training tokens should also double. The practical ratio: roughly 20 tokens per parameter. Chinchilla itself—70B parameters, 1.4 trillion tokens—used the same compute budget as Gopher (280B parameters, 300B tokens) and outperformed it across benchmarks, including a 7%+ gain on MMLU.

The result inverted Kaplan's prescription. Where Kaplan favored large models on small datasets, Chinchilla demonstrated that large models had been systematically undertrained. The revision changed procurement: post-2022 models (LLaMA, Mistral) explicitly follow Chinchilla scaling. The optimal model size at fixed compute is now $N \propto C^{0.50}$, half Kaplan's exponent.

Epoch AI's replication attempt (arXiv:2404.10102, April 2024) partially validated Hoffmann's results but flagged rounding errors in reported parameters and noted sensitivity to data quality. When trained on higher-quality data, the optimal token-per-parameter ratio falls below 20:1. The result holds directionally but the coefficients remain contested at the margin.

Sources:

• https://arxiv.org/abs/2203.15556
• https://proceedings.neurips.cc/paper_files/paper/2022/file/c1e2faff6f588870935f114ebe04a3e5-Paper-Conference.pdf
• https://arxiv.org/pdf/2404.10102
• https://epoch.ai/publications/chinchilla-scaling-a-replication-attempt

Kaplan et al. (arXiv:2001.08361, January 2020) measured cross-entropy loss across language models spanning seven orders of magnitude. The relationship held as a power law: loss fell predictably with parameter count, training tokens, and total compute. Architectural choices—width, depth—had minimal effect within a wide range. The authors derived equations for overfitting and training speed, then used those to determine optimal compute allocation. The headline finding: larger models are significantly more sample-efficient, so compute-optimal training involves very large models trained on relatively modest token counts.

Kaplan's allocation favored model size over data. At fixed compute, the study recommended scaling parameters faster than tokens—roughly $N \propto C^{0.73}$. This shaped GPT-3 (175B parameters, 300B tokens). The exponent was later contested: Pearce & Song (2024) found that Kaplan counted non-embedding parameters and analyzed small models, both of which biased the coefficient upward. The revised coefficient is closer to $N \propto C^{0.50}$, the Chinchilla estimate.

The work formalized the predictability of pretraining performance and provided the first compute-budget equations that hyperscaler planners could use to size model runs. It is foundational—but the allocation it implied became obsolete within two years.

Sources:

• https://arxiv.org/abs/2001.08361
• https://arxiv.org/pdf/2001.08361
• https://arxiv.org/pdf/2406.12907