Research No. 05
Interactive- Efficiency
- Optimization
- AI systems
- Industrial engineering
The constraint always moves
The 140-year evolution of efficiency and optimization — from factories, Lean manufacturing and statistical quality control to AI models, chatbots, GPUs, chips, data centres and biotech. What each generation was actually up against, what they built to fix it, and which of it still works.
Applied research · synthesis across industrial engineering, operations research and AI systems · Updated through September 8, 2026 · 67 sources
Free handout2 pages · PDF
The whole study, in plain words, on one page
Written for a room aged 16 to 70 with no background assumed. Page one is the story, the five numbers and a six-month route into this work — share it on screen or print it. Page two is a 90-second script you can read out loud.
Open the PDF →What this study is
How companies learned to make systems work better — from Taylor’s stopwatch and Ford’s assembly line to AI models, GPUs, chips, data centres and biotech, current through September 2026.
- 140 years
- of efficiency and optimization
- 16 milestones, 1881 to 2026
- 11 companies
- in one comparable shape
- What they optimise, how, and who does it
- 21 techniques
- for making AI cheaper
- Each with the tradeoff it creates
- Sep 2026
- current through
- 67 sources, each read directly
And the one thing it found
Relieving a bottleneck never removes it. It hands it to someone else.
That is why the techniques look nothing alike across a hundred and forty years while the method for choosing between them has barely changed. Find the one thing setting the rate, check that your instrument can see it, relieve it — then go and find out where it went.
Eight constraints, each relieved by the methods of its era and each handing the problem to the next. Power is where it sits today, not what the study is about. Dates mark when a constraint became the binding one rather than when its technique was invented; several overlapped, and the ordering is an argument, not a measurement.
What do you want to look at?
Jump straight in
The study runs in the order the constraint actually moved, which is the right shape for reading it and the wrong shape for finding one thing in it. This is the other view.
How it started
4 topicsThe methods that still run every factory, plant and serving fleet on this page — and the people who worked them out.
How big tech optimizes today
11 topicsEleven companies, in one consistent shape: what they optimise, what they use, why it pays, who does the work, and what changed in 2026.
How chatbots get cheaper and faster
4 topicsChatGPT, Claude, Gemini and Grok, using only what each company has published about its own serving.
AI model optimization
10 topicsTwenty-one techniques, filterable by the bottleneck each one attacks. Every entry names its tradeoff.
GPUs, chips and memory
5 topicsThe silicon layer, and why the thing that binds it is usually not arithmetic.
Data centres and power
5 topicsWhere the binding constraint sits in September 2026, and the economics that make a hundredth of a point worth a building programme.
Biotech, medicine and cancer research
6 topicsWhere optimization is not mainly about cost — because when nineteen of twenty programmes fail, the valuable thing is an earlier no.
Manufacturing and other industries
6 topicsSeven methods chosen because each names a mechanism that transfers, not because a large company saved money.
The people who do this work
6 topicsTwelve job families doing the same work in different units, and the seven skill tiers underneath them.
The whole subject on one page
Business → power grid
Every layer of a modern system, and the number the people who own that layer are judged on. Read top to bottom, it is the claim this study spends seventeen sections defending: these are all the same activity, measured in different units.
Sixteen milestones, six eras, one moving constraint
1881 → 2026
Each milestone names the problem someone was actually facing, what they built, what it optimised, what it was worth, and what survives today. The eras are labelled with the constraint that was binding during them.
1880 – 1930
Labour & workflow
↓ Human motion
1924 – 1960
Quality & method
↓ Variation
1950 – 1990
Flow & inventory
↓ Work in progress
1984 – 2016
Systems & supply
↓ The whole chain
2004 – 2020
Software & compute
↓ Cycles, then cores
2020 – 2026
Models & infrastructure
↓ Memory, then power
Select a milestone · 16 in total
1881–1911
Labour & workflow
The stopwatch turns work into data
Frederick Winslow Taylor, Midvale and Bethlehem Steel
- The problem
- Nobody could say how long a job should take. Pay, staffing and scheduling all rested on foremen's opinion, and everybody's opinion was self-serving.
- The innovation
- Break a task into its component motions, time each one, and rebuild the task from the fastest safe version of each. Taylor published the method in 1911 as scientific management.
- What was optimised
- Labour time per unit of output
- Business impact
- It created industrial engineering as an occupation and made the standard time the unit of factory planning. It also made the work miserable enough to fuel a generation of labour conflict, and the pig-iron figures Taylor published have since been shown to be dramatised.
- What we still use
- Standard work, takt time, time-and-motion study, the whole idea that a process has a measurable baseline before you argue about improving it.
01Where efficiency started — and why nobody invented it
Taylor's stopwatch, Shewhart's memo, Blackett's scientists and Toyota's loom: four traditions, four countries, four unrelated problems, one shared stance.
It is tempting to hand this story to one person. Taylor is the usual candidate, sometimes Ford, occasionally Ohno. The evidence does not support any of them, and the tidy version does real damage, because it hides the fact that the important ideas arrived independently, from people who were not talking to each other, working on problems that looked nothing alike.
Frederick Taylor introduced time study at Midvale Steel in 1881 and published The Principles of Scientific Management in 1911.1 Walter Shewhart wrote a one-page memorandum on 16 May 1924, in the Western Electric engineering department that became Bell Telephone Laboratories the following year, containing the first control chart — a completely different idea, about variation rather than motion, from a completely different problem: keeping the American telephone network reliable.3 British operational research began in 1939 as an attempt to fit radar into air-defence tactics, and INFORMS is explicit that the early work was mostly empirical data-gathering; the mathematics came later, in the civilian profession.5 Toyota's system grew from a loom, a cash shortage and a small domestic market.6
What they share is not a technique. It is a stance: that an operation is a legitimate object of measurement, that the measurement should come before the argument, and that the thing limiting output is usually not the thing everyone is looking at.
02Ford moved the work, not the worker
12½ hours to 93 minutes — and the first time relieving one constraint visibly created the next one, in the supply chain.
Taylor optimised the task. Ford optimised the connection between tasks, which turned out to be worth far more. Under stationary assembly, The Henry Ford records 12½ hours per car; on the moving line, a finished Model T every 93 minutes. Between 1908 and 1927 Ford built over 15 million of them, and the price fell from $850 to as little as $260.2
The mechanism is worth being precise about, because it recurs everywhere below. The line did not make anyone work faster. It removed the waiting, the walking and the searching between operations, and it made the rate a property of the system rather than a negotiation with a foreman. That is the same argument as continuous batching in a GPU server: the arithmetic did not get faster, the gaps got smaller.
And the constraint moved, immediately and expensively. A plant that could finish a car every ninety-three minutes needed parts at that rate, which is how Ford ended up owning iron mines, rubber plantations and a railway. Vertical integration was not a strategy in the modern sense. It was a bottleneck relief programme.
03Quality became a measurement, not a verdict
Shewhart's control chart answered a question every operator faces: this number is worse than the last one — do I act? Software keeps rediscovering it badly.
Shewhart's control chart answers a question every operator faces and almost nobody had a rule for: this measurement is worse than the last one — do I act? His answer was that variation comes in two kinds. Common-cause variation is what the process always does, and reacting to it makes the process worse. Assignable-cause variation is a signal, and it is worth chasing. The chart draws the line between them.3
That is one of the most transferable ideas on this page, and it is routinely rediscovered badly. Every alerting system that pages an engineer for ordinary noise is a control chart with the limits set wrong. Every A/B test declared a winner on a difference smaller than the week-to-week spread is the same mistake in a different costume.
Deming's contribution was to insist on where the responsibility for that variation lies. From summer 1950 he taught Japanese managers and engineers that most quality problems are properties of the system, and that only management can change a system. American industry noticed thirty years later, when NBC broadcast If Japan Can, Why Can't We? in June 1980.4 The lag is instructive: the technique was public the whole time. What was missing was a reason to believe it mattered.
04A war turned operations into a science
Radar worked and the tactics did not. The physicists and biologists sent to fix that invented operations research.
Operational research began because Britain had installed a new technology into an operation designed without it. INFORMS dates the term to that work — scientific research aimed at integrating radar into Royal Air Force tactics — and notes that by 1941 it had widened to cover tactics and planning generally, whether or not technology was involved.5
What made it work was the composition of the teams. Physicists, statisticians and biologists were pointed at convoy sizes, depth-charge settings and search patterns — problems where nobody present was the incumbent expert, and so nobody was defending an existing answer. That is a structural advantage, not a personal one, and it is the closest thing this study has to a hiring recommendation.
The mathematics arrived afterwards. Linear programming, queueing theory and simulation were built when operational research became a civilian profession, and they are the direct ancestors of the routing, scheduling and capacity models running inside every large logistics and cloud operation today.
05Toyota and Lean — flow, waste and the right to stop the line
Just-in-Time and jidoka both deliberately reduce local efficiency, which is why competitors could not copy the system one piece at a time.
Toyota could not copy Ford. It had little capital, a small home market and no room to float production on inventory. Toyota's own account names two pillars: Just-in-Time — making only what is needed, when it is needed, in the amount needed — and jidoka, automation with a human touch, in which a machine detects an abnormality and stops itself rather than producing a hundred more defects.6
Both are constraints imposed deliberately. Just-in-Time removes the inventory that would otherwise hide every scheduling problem, which forces those problems into view where they can be fixed. Jidoka stops the line — deliberately reducing throughput in the moment — because a defect that travels downstream costs far more than the stoppage. Neither would survive a spreadsheet that optimised local utilisation.
That is the deepest thing Toyota contributed, and it is why the system was so hard for competitors to copy piecemeal. Kanban cards are easy to install. The willingness to let an operator halt a production line is not, because it requires management to have already decided that finding problems is worth more than not stopping.
The MIT International Motor Vehicle Program's five-year, fourteen-country plant study named this lean production in 1990 and moved it out of automotive.7 Within a decade the vocabulary — value streams, waste, flow, pull, kaizen — was in hospitals, banks and software teams, where roughly half of it worked and the other half was cargo-culted.
06One thing sets the rate — and the twelve things companies optimise
Goldratt's Theory of Constraints, Six Sigma arriving from the other direction, and the quantities every industry on this page is actually trying to move.
In 1984 Eliyahu Goldratt published a business novel, The Goal, containing an argument that sounds obvious and almost nobody acts on:47 in any system, one thing sets the rate. Improving anything else produces no additional output. Worse, improving anything upstream of the constraint actively harms the system, because it piles inventory in front of a station that cannot use it.
The Theory of Constraints turns that into a procedure. Identify the constraint. Exploit it — get everything you can from it before spending money. Subordinate everything else to it. Elevate it, which is where capital finally enters. Then return to the first step, because the constraint has moved.
Six Sigma arrived from a different direction three years later. ASQ dates the concepts to Bill Smith and Bob Galvin at Motorola in the 1980s: a numerical target of 3.4 defects per million opportunities, and a five-phase project method — define, measure, analyse, improve, control.8 Lean and Six Sigma are often presented as rivals. They answer different questions. Lean asks where the time goes; Six Sigma asks where the variation comes from. A system with a flow problem and a system with a variation problem need different tools, and the expensive mistake is reaching for the one you know.
Six Sigma also carries the discipline this site has written about before: before comparing two processes, qualify the instrument that measures them. Measurement-system analysis is a required step in DMAIC, and it is the single most transferable piece of manufacturing practice into software and AI, where it is almost never done.
What companies actually optimise
Twelve quantities, in every industry here. Most disagreements about optimization are really disagreements about which of these is the objective and which are constraints — an argument worth having explicitly, before anyone touches the system.
Throughput
Units of value the system produces per unit of time.
cars/hour, tokens/second, patients/day
Ford's line; a GPU serving fleet
Cycle time
How long one unit takes to get from start to finish.
minutes, days
Lean; time-to-first-token
Yield
The share of started work that comes out usable.
%, g/L
Wafer fab; a bioreactor; a clinical portfolio
Utilisation
Share of a resource's capacity actually doing useful work.
%
Machine hours; MFU on a GPU
Latency
Delay between a request and its answer.
ms
Inference SLOs; trading; page loads
Cost per unit
All-in cost to produce one unit of output.
$/unit, $/1M tokens
Everything, eventually
Quality
Conformance to what the customer actually needs.
defects per million
Six Sigma; eval pass rates
Reliability
Probability the system does what it promised, when asked.
%, MTBF
Uptime; batch success
Energy
Work delivered per unit of power consumed.
tokens/MWh, PUE
Data centres; Apple silicon
Capital efficiency
Value produced per dollar of asset tied up.
$ output / $ invested
Fab economics; AI factory ROI
Inventory
Value sitting still between two useful steps.
$, units, days
JIT; a KV cache is inventory too
Variability
Spread of outcomes around the average.
σ, p99
SPC; tail latency
072005: the constraint moved into the silicon
Dennard scaling ended, the free lunch stopped, and every optimization problem in software changed shape — which is why GPUs, not faster CPUs, became the substrate of AI.
For roughly thirty years, each process generation delivered more transistors at the same power, and clock speeds rose with them. Software got faster by waiting. Then leakage current stopped voltage from scaling and the chips could no longer be cooled. Hennessy and Patterson's Turing lecture describes the consequence directly: general-purpose performance growth is exhausted, and what remains is specialising the machine to the workload.11
This is the pivot the rest of this study runs on. Every accelerator below — GPU, TPU, Trainium, MTIA, Maia — exists because the general lever broke and specialisation was the only one left. It also re-pointed the entire software industry at concurrency, which is why a discipline that had spent forty years optimising instruction counts had to learn to optimise data movement instead.
Cloud computing did something adjacent and less discussed. It converted computing capacity from a capital purchase into an operations problem, which meant utilisation — the oldest metric in this study — suddenly applied to servers. It also made the waste visible in a way owned hardware never had, and the waste turned out to be enormous. One vendor measuring production Kubernetes clusters across the three major clouds before optimisation reports 8% CPU, 20% memory and 5% GPU utilisation.40 That is a vendor selling the fix, measuring customers who came to it with a problem, and the absolute level should not be read as a population estimate — but the direction is not seriously disputed by anyone who runs a fleet.
08AI model optimization — making every GPU do more useful work
Twenty-one techniques, filterable by the bottleneck each attacks: quantization, caching, batching, speculative decoding, routing, data efficiency, synthetic data, retrieval. Every one names its tradeoff.
For a few years the interesting question in AI was capability: can the model do the thing. That question has not gone away, but it has stopped being the one that decides whether a system ships. The operative question in 2026 is how much useful output you can produce per dollar, per watt, per accelerator and per second — and, increasingly, per unit of a customer's patience.
The economics have already moved a very long way. Epoch AI measured the price of reaching a fixed capability and found declines between 9× and 900× per year depending on the benchmark, with GPT-4-level performance on PhD-level science questions falling about 40× per year — while noting that the fastest of those declines are the most recent and may not persist.39 That is not one improvement. It is a stack of them compounding, and the sections below take them apart.
Two structural changes did most of the heavy lifting. Sparsity: DeepSeek-V3 has 671B parameters and activates 37B per token, breaking the link between the quality that scales with total parameters and the cost that scales with activated ones. Its full training run took 2.788M H800 GPU-hours.14 And precision: NVIDIA pretrained a 12B model on 10 trillion tokens entirely in 4-bit NVFP4 and reported loss and downstream accuracy comparable to an FP8 baseline — a vendor result, on one model, with a recipe that needed Hadamard transforms, two-dimensional scaling, stochastic rounding and some layers kept high.15
The rest is serving, and serving is where the old operations vocabulary transfers almost without translation. A KV cache is inventory. Continuous batching is line balancing. Prefix caching is setup reduction. Speculative decoding is a two-machine cell where a cheap machine does the first pass and an expensive one inspects. Disaggregating prefill from decode is splitting a mixed-model line into two dedicated ones because the changeover was killing you.
Underneath all of that sits a quieter question: what the model is learning from, and whether more of it helps. Epoch AI put the effective stock of quality-adjusted public human text at around 300 trillion tokens and projected it fully used somewhere between 2026 and 2032 — a modelled interval, not a date, and one their own paper lists four ways around.53 The response has been to get more capability out of each token rather than to find more tokens. Hugging Face ablated every filtering choice in a 15-trillion-token corpus and found that an educational-quality filter discarding roughly 92% of it produced better models on knowledge benchmarks than the whole thing.51 Deduplication tells the same story from the other side: one 61-word sentence appears over 60,000 times in a widely used corpus, and removing the duplicates reaches the same accuracy in fewer steps.52
The other response is to make the data. NVIDIA reports that 98% of the alignment data for one of its open models was synthetically generated, using about 1% of the human annotation that trained a comparable model.54 That is a constraint being traded rather than removed: scarce human labelling becomes generation plus verification, which scales with compute instead of headcount — and which only works where an answer can be checked. The failure mode is documented and severe. Training indiscriminately on recursively generated data causes irreversible collapse, with the tails of the distribution going first; mixing real data back in slows it.55 Synthetic data is not better data. It is data whose cost curve you control, provided you own a verifier.
And there is a prior question that decides how large the model needs to be at all: should knowledge live in the weights, or be fetched when a question needs it? DeepMind's RETRO matched GPT-3 on the Pile with 25× fewer parameters by retrieving from a two-trillion-token database.57 But the honest comparison is less flattering to retrieval than its advocates suggest — Google DeepMind found long context wins on average when you can afford it, and, more usefully, that for 63% of queries the two approaches produce exactly identical predictions. Routing between them rather than choosing cut cost by 65% on one model and 39% on another at comparable quality.56 This is the thesis in miniature: relieve model capacity with retrieval and the constraint moves to retrieval latency, then to context length, then to the cache, then to the orchestration deciding which path a request takes.
Filter by the bottleneck a technique attacks. The useful question is never what a technique is — it is which constraint it relieves, and what it costs you in exchange.
- Why it works
- Weights and activations are the largest objects in the system, and neural networks tolerate a surprising amount of numerical noise. Halving the bits halves the bytes moved and, on hardware with native support, roughly doubles the arithmetic rate.
- What it improves
- Memory footprint, memory bandwidth, arithmetic throughput, energy per token.
- The tradeoff
- Accuracy loss that is small on average and not uniform: it concentrates in outlier channels and in the hardest examples. Below 8 bits it takes real engineering — NVIDIA's 4-bit pretraining recipe needed Hadamard transforms, two-dimensional scaling, stochastic rounding and some layers kept in high precision.
- Who uses it
- Universal. NVIDIA pretrained a 12B model on 10T tokens in NVFP4 and matched FP8. Every serving stack ships INT8/FP8 paths.
- When to reach for it
- Almost always for inference. For training, when you have the hardware support and can afford to validate the recipe properly.
- Why it works
- Model quality scales with total parameters; cost scales with activated parameters. MoE breaks the link between the two.
- What it improves
- FLOPs per token at a given quality — often by an order of magnitude.
- The tradeoff
- Memory does not shrink: every expert must be resident somewhere. Routing creates load imbalance across devices, and expert parallelism turns a compute problem into a network problem.
- Who uses it
- DeepSeek-V3 activates 37B of 671B parameters per token. Most frontier-scale open models now use it.
- When to reach for it
- When you are compute-bound at scale and have the interconnect to pay for the routing.
- Why it works
- Most production traffic is far narrower than the capability frontier. A student that only has to cover your traffic can be a fraction of the size.
- What it improves
- Cost and latency per request, often by 5–20× on the covered distribution.
- The tradeoff
- It transfers the teacher's behaviour, including its mistakes, and it degrades ungracefully off-distribution. The student's competence boundary is invisible until you cross it.
- Who uses it
- Standard practice across every major lab's small-model line.
- When to reach for it
- When traffic is stable, well-characterised, and large enough that the training cost amortises.
- Why it works
- Training compute buys you passes over data, and a pass over a duplicate teaches nothing. Hugging Face ablated every filtering and deduplication choice in FineWeb and found that an educational-quality filter discarding roughly 92% of a 15-trillion-token corpus produced better models on knowledge benchmarks than the full set — not merely cheaper ones. Deduplication is the same story from the other direction: one 61-word sentence appeared over 60,000 times in C4, and models trained on deduplicated data reach the same accuracy in fewer steps while emitting memorised text about ten times less often.
- What it improves
- Capability per training token, and therefore per dollar and per megawatt of training compute. It also improves what you can say about the model, because a curated corpus is one you can characterise.
- The tradeoff
- Filtering is a bet about what matters, made before you know what the model will be asked to do — an aggressive quality filter is also a narrowing filter, and the benchmarks that reward it may not be the ones you care about. Building the classifier, running the ablations and storing the variants is real work that competes with training. And discarding 92% of a corpus is only clever if the remaining 8% covers your domain.
- Who uses it
- Hugging Face published the corpus, the classifier and more than seventy ablation models, which is what makes this checkable rather than folklore. Every frontier lab does some version of it; almost none publish the recipe.
- When to reach for it
- Before buying more compute, and before buying more data. It is the cheapest lever in this library and the one most often skipped, because curating a corpus is less interesting than training on one.
- Why it works
- The constraint moved. Epoch AI puts the effective stock of quality-adjusted public human text at around 300 trillion tokens and projects it fully utilised somewhere between 2026 and 2032. Human annotation is scarcer still, and it is the slow step in post-training. Generation is not: NVIDIA reports that 98% of the alignment data for Nemotron-4-340B-Instruct was synthetic, produced with about 1% of the human annotation used for a comparable model, and matching or exceeding it on standard benchmarks. The move is to trade a data problem for a generation-and-verification problem, which scales with compute rather than with headcount.
- What it improves
- Coverage of cases nobody collected, and the cost and turnaround of post-training. It works best where an answer can be checked automatically — maths, code, formal reasoning, simulated physics — because then the verifier, not the generator, is what makes the data good.
- The tradeoff
- This is the one technique here with a documented catastrophic failure mode. Shumailov et al. showed in Nature that indiscriminate training on recursively generated data causes irreversible collapse, with the tails of the distribution disappearing first — and that mixing real data back in slows it. Beyond collapse, synthetic data inherits and can amplify the generator's biases and blind spots, and it is thin exactly where verification is hard, which is most of what people actually want models to do. Epoch's own caveat is that synthetic data has reliably helped only in narrow, checkable domains.
- Who uses it
- NVIDIA open-sourced a full generation pipeline with base, instruct and reward models. It is standard in reasoning post-training, and in robotics, where simulation generates what the physical world cannot supply at volume — though the sim-to-real gap remains the unsolved part and no vendor publishes a real-versus-synthetic ratio.
- When to reach for it
- Where you have a verifier. Without one you are not generating data, you are generating the model's existing opinions and training on them.
- Why it works
- Trained networks are over-parameterised for inference. Unstructured pruning saves memory on paper; structured pruning saves it in practice, because the kernels can skip the zeros.
- What it improves
- Memory footprint and arithmetic, if the sparsity pattern matches what the hardware supports.
- The tradeoff
- Unstructured sparsity rarely produces real speedups. Structured sparsity costs more accuracy for the same reduction, and usually needs fine-tuning to recover.
- Who uses it
- Supported in NVIDIA tensor cores and the major compilers; used far less than quantization.
- When to reach for it
- After quantization, when you still need footprint and can afford a recovery fine-tune.
- Why it works
- Requests finish at wildly different times. Static batching makes every request in a batch wait for the slowest, leaving the GPU partly idle for most of the batch's life.
- What it improves
- GPU utilisation and throughput, with no change to the model.
- The tradeoff
- Tail latency becomes sensitive to admission policy, and a request's neighbours now affect its speed. Scheduling gets much harder to reason about.
- Who uses it
- Default in vLLM, SGLang, TensorRT-LLM.
- When to reach for it
- Any multi-tenant serving system. There is essentially no case against it.
- Why it works
- Decode is memory-bandwidth-bound: reading the weights costs the same whether you produce one token or several. Verification amortises that read across multiple tokens, and the output distribution is preserved exactly.
- What it improves
- Latency per token at low to medium concurrency — up to 2.5× measured on Llama models.
- The tradeoff
- It buys bandwidth with arithmetic. At high batch sizes the system is already compute-bound and speculation makes it slower. Red Hat measured the optimum draft length at 1 — or 0 — on translation workloads.
- Who uses it
- EAGLE-family drafts in vLLM, SGLang and TensorRT-LLM. OpenAI attributes over 15% better token-generation efficiency in 2026 to a redesigned draft model.
- When to reach for it
- Latency-sensitive traffic at low concurrency, on workloads where the draft model is accurate. Measure the acceptance rate before committing.
- Why it works
- The most expensive property of a request is usually its deadline, not its size. Providers price this explicitly: Anthropic's Message Batches API halves both input and output prices, and the caching multipliers stack on top.
- What it improves
- Cost, by roughly half, for the large share of enterprise work that was never actually interactive.
- The tradeoff
- Latency, obviously — and the organisational work of finding out which jobs genuinely have a deadline, which is harder than it sounds.
- Who uses it
- Every major provider offers a batch tier.
- When to reach for it
- Evaluation runs, backfills, enrichment, classification, report generation. The default should be batch, with interactive as the exception.
- Why it works
- The cache is the dominant consumer of GPU memory in serving, and memory caps batch size, and batch size sets cost per token. vLLM measured 60–80% of that memory lost to fragmentation and over-reservation in earlier systems.
- What it improves
- Effective batch size, and therefore throughput and cost per token — 2–4× against 2023-era baselines.
- The tradeoff
- An extra indirection on the hottest path in the system, and a scheduler that is now managing memory as well as work.
- Who uses it
- vLLM, SGLang, TensorRT-LLM — effectively every serving stack.
- When to reach for it
- Table stakes. If your stack does not do this, nothing else you do to it will matter much.
- Why it works
- Agent and RAG traffic re-sends enormous, nearly identical prefixes. Recomputing them is pure waste, and the providers price it accordingly: Anthropic charges 0.1× base input for a cache read.
- What it improves
- Time to first token and input cost — up to a 90% discount on the reused portion.
- The tradeoff
- The cache matches prefixes, not content. Anything dynamic near the front of the prompt invalidates everything after it, and the standard levers for controlling context growth — compaction, clearing tool results — are edits to the middle. Cache lifetime is a real cost: Anthropic's default is five minutes.
- Who uses it
- Anthropic, OpenAI and Google all sell it; vLLM and SGLang implement it structurally.
- When to reach for it
- Whenever the same prefix is seen more than once. Design the prompt so the stable part comes first.
- Why it works
- Attention was assumed to be arithmetic-limited. FlashAttention showed the cost was data movement — and that counting bytes moved, not operations performed, is the right way to reason about GPU performance.
- What it improves
- Wall-clock time and memory footprint, with mathematically identical output — 3× end-to-end on GPT-2 at 1K context.
- The tradeoff
- Kernels are hardware-specific and have to be rewritten for each new architecture. This is where a great deal of unglamorous, highly paid engineering now lives.
- Who uses it
- The default attention implementation everywhere.
- When to reach for it
- Already on, in every framework you are likely to use.
- Why it works
- A chain of small operations spends most of its time launching kernels and round-tripping tensors through HBM. Fusing them removes both costs.
- What it improves
- Throughput on everything that is not a large matrix multiply, which is more of the model than people expect.
- The tradeoff
- Compilation time, debuggability, and shape specialisation — a fused graph that is fast at one batch size can recompile or fall back at another.
- Who uses it
- OpenAI reports 20% lower end-to-end serving cost in 2026 from production kernel work in Triton and Gluon — much of it written by GPT-5.6 Sol, driven through Codex, with a purpose-built floating-point sanitiser to check the results.
- When to reach for it
- Once the model is stable and the shapes are known. It is the last 20–30%, not the first.
- Why it works
- Backpropagation needs the activations of every layer, and storing them is what actually fills the memory on a training run — not the weights. Recomputing a layer is cheap; holding it for the whole step is not. Chen et al. showed an n-layer network can be trained in O(sqrt(n)) memory for roughly one extra forward pass, cutting a 1,000-layer residual network from 48GB to 7GB.
- What it improves
- Memory headroom, which converts directly into larger batches, longer sequences or bigger models on the hardware you already have.
- The tradeoff
- It buys memory with arithmetic — about 30% more running time in the original demonstration. That is a good trade when memory is binding and a bad one when it is not, which makes it the clearest example in this library of why you profile before you optimise: the same change is a win or a 30% regression depending on which resource is short.
- Who uses it
- Built into PyTorch, JAX and every large-scale training framework, and on by default in most frontier-scale recipes.
- When to reach for it
- When a training run will not fit, or fits only at a batch size too small to keep the accelerators busy. Never as a reflex.
- Why it works
- Frontier models do not fit on one accelerator. Which split you choose determines what you spend on communication and how much of the fleet sits waiting.
- What it improves
- The ability to run the model at all, and then the efficiency with which it runs.
- The tradeoff
- Every split buys memory with communication. Tensor parallelism needs very fast links and stops scaling past a node; pipeline parallelism creates bubbles; expert parallelism creates load imbalance. This is why rack-scale interconnect became a product category.
- Who uses it
- Every lab. It is the reason NVLink domains, and their size, are a competitive variable.
- When to reach for it
- As soon as the model exceeds one device. Choose the split from the interconnect you have, not the other way round.
- Why it works
- The two phases stress different resources — prefill is compute-bound, decode is bandwidth-bound — so a single pool sized for both is wrong for each. Splitting them lets each scale on its own demand.
- What it improves
- Utilisation of both resources, and independent control of time-to-first-token and inter-token latency.
- The tradeoff
- You now have to move gigabytes of KV state between machines, which needs NVLink, RDMA or a shared cache tier — and a whole new class of failure. Meta found it only pays above a certain scale and concurrency.
- Who uses it
- Meta in production with vLLM; NVIDIA built Rubin CPX as a distinct GPU class for the prefill half.
- When to reach for it
- Large deployments with long prompts, short answers, and a real inter-token latency SLO.
- Why it works
- Traffic is not uniformly hard. Paying frontier prices for trivially easy requests is the single largest avoidable cost in most production AI systems.
- What it improves
- Blended cost per useful result, often dramatically.
- The tradeoff
- The router is a model too, and it can be wrong. A misroute is a quality failure that looks like a model regression, and it makes evaluation harder: you are now measuring a system, not a model.
- Who uses it
- Now a product feature at every major provider as well as a common in-house pattern.
- When to reach for it
- Whenever traffic has a wide difficulty distribution and you can measure per-class quality separately.
- Why it works
- Parameters are a very expensive place to keep facts: you pay for all of them on every token, they are fixed at training time, and updating one means training again. DeepMind's RETRO matched GPT-3 on the Pile with 25x fewer parameters by conditioning on chunks retrieved from a two-trillion-token database — the clearest published evidence that capability and stored knowledge can be separated. The economics are the argument: a retrieval store is updated in seconds, audited, and cited back to the user.
- What it improves
- Cost and latency per request for knowledge-heavy work, freshness, groundedness, and the ability to show a source. It also makes a smaller model viable, which compounds through every other technique in this library.
- The tradeoff
- Retrieval is not free and it is not always better. Google DeepMind's comparison found long context outperforms retrieval on average when you can afford it — and, more usefully, that for 63% of queries the two produce identical predictions, token for token, which is why routing between them beats choosing. Their Self-Route hybrid cut cost by 65% on Gemini-1.5-Pro and 39% on GPT-4o at comparable quality. The failure modes are real: a bad retrieval is a confidently wrong answer, the index becomes a system to run and reindex, and at scale retrieval latency lands directly in your time-to-first-token budget.
- Who uses it
- Effectively every enterprise deployment, and increasingly agentic search rather than a single retrieval pass. The published comparisons are on question-answering benchmarks, so treat the ratios as directional for other workloads.
- When to reach for it
- When knowledge changes faster than you retrain, when provenance matters, or when a smaller model plus a store is cheaper than the larger model. Route rather than choose: send what retrieval handles to retrieval, and escalate the rest.
- Why it works
- Agent cost is dominated by input tokens, and most of those tokens carry no new information. Anthropic's worked example of moving a transcript between two systems falls from 150,000 tokens to 2,000 when the agent writes code against the tools instead of loading all of them.
- What it improves
- Input tokens, latency, and — because attention is finite — often accuracy as well.
- The tradeoff
- Running agent-written code needs a sandbox, resource limits and monitoring that direct tool calls do not. And most compaction strategies edit the middle of the context, which is exactly what invalidates the prefix cache. The two biggest levers on agent cost pull against each other.
- Who uses it
- Anthropic has published the pattern and the caveat. OpenAI's agentic harness makes the same two moves — deferred tool discovery, and tool output capped at 10,000 tokens by default — while explicitly working to preserve exact prefixes for the cache.
- When to reach for it
- Any long-running agent. It is usually the highest-leverage change available and the one least likely to be attempted.
- Why it works
- Static allocation guarantees idle silicon, because every team sizes for its own peak and the peaks do not coincide. This is the oldest result in operations research applied to the newest asset, and the numbers are startling: one vendor measuring production Kubernetes clusters before optimisation reports 5% GPU utilisation.
- What it improves
- Utilisation of hardware already bought and already drawing power — the cheapest capacity there is.
- The tradeoff
- Teams lose the certainty of their own reservation, and pre-emption makes long training runs riskier. It is mostly an organisational problem wearing a technical costume, which is why it is so often left undone.
- Who uses it
- Standard inside hyperscalers and frontier labs; rare and highly valuable everywhere else.
- When to reach for it
- Before buying more hardware. Almost always before buying more hardware.
- Why it works
- Air stops working somewhere around 30–50kW per rack, and modern AI racks are past that. Warm-water cooling removes the chiller from the energy budget entirely, and closed loops remove the water consumption.
- What it improves
- PUE, rack density, and — because you can fit more racks in the same power envelope — capital productivity. NVIDIA claims 45°C warm-water operation allows up to 10% more racks in the same power budget.
- The tradeoff
- Capital cost, plumbing in facilities never designed for it, and a new set of failure modes in a room full of electronics. Uptime Institute finds industry PUE improving only marginally, because most of the fleet is legacy.
- Who uses it
- Google's Ironwood pods are liquid-cooled at 9,216 chips; Microsoft's newest sites use second-generation closed-loop heat exchangers; NVIDIA's rack-scale systems assume it.
- When to reach for it
- Any new build at AI densities. Retrofits are a much harder economic case.
09How chatbots get cheaper and faster — ChatGPT, Claude, Gemini, Grok
The same five questions of each: what is the bottleneck, what do they do about it, what did they publish, what does it cost them, and why it reaches you.
Everything below comes from what each company has published about its own serving. That constraint is why these entries are different lengths: OpenAI and Anthropic publish engineering detail, Google publishes the mechanism without the discount, and xAI publishes a cache hit rate and a price while describing its serving techniques only as “several innovative techniques”. Where a company has not said how something works, this page does not guess.
ChatGPT
OpenAI- The bottleneck
- Sequential decoding and kernel efficiency. Generating a token means reading the weights whether you produce one token or several, and an agent turn can take thirty model calls.
- What they do about it
- Production GPU kernels rewritten in Triton and Gluon; an improved speculative-decoding draft model; KV-cache and scheduling tuned per workload; tool output capped at 10,000 tokens by default and integrations surfaced only when needed.
- The published result
- OpenAI reports 20% lower end-to-end serving cost from the kernel work and over 15% better token-generation efficiency from the draft model. Prices fell 80% for GPT-5.6 Luna and 20% for Terra on 30 July 2026.
- The tradeoff
- Much of this was done by GPT-5.6 Sol itself inside a human-run process, which makes it fast and hard to audit — OpenAI built a floating-point sanitiser specifically to check the model's kernels. Everything here is self-reported and unreplicated.
- Why it reaches you
- Serving cost is what sets the price of the tier you can afford and how much reasoning you get before it becomes uneconomic.
Claude
Anthropic- The bottleneck
- Input tokens. Agent workloads re-send an enormous, nearly identical prefix on every step, and input volume — not output — dominates the bill.
- What they do about it
- Prompt caching, with prefixes built tools → system → messages; the Message Batches API for work with no deadline; and code execution with MCP so an agent calls tools as code instead of loading every definition into context.
- The published result
- A cache read costs 0.1× base input. Batch processing halves both input and output prices, and the multipliers stack. One worked task — moving a transcript from Google Drive to Salesforce — falls from 150,000 tokens to 2,000.
- The tradeoff
- The cache matches prefixes, not content, and its default lifetime is five minutes. Compaction and tool-result clearing shrink context but edit the middle of it, which invalidates everything after. Agent-written code needs a sandbox, resource limits and monitoring.
- Why it reaches you
- It is the difference between an assistant that can afford to keep reading your codebase and one that cannot.
Gemini
Google- The bottleneck
- Repeated prefixes and paying frontier prices for work a smaller model handles. Attention over hundreds of thousands of tokens is expensive on every request that carries them.
- What they do about it
- Implicit context caching, on by default for Gemini 2.5 and newer, with a minimum of 4,096 tokens on current Flash models; a Flash tier for high-frequency work; and TPU silicon designed for inference rather than adapted to it.
- The published result
- Google states cost savings are passed on automatically when a request hits cache, and reports customers finding Flash close to Pro in quality at materially lower latency and cost. Ironwood is quoted at 2× the performance per watt of the previous TPU generation.
- The tradeoff
- Google does not publish the size of the caching discount, only that it exists. The customer quality figures are partner statements in a launch post, not independent benchmarks. Caching still rewards putting stable content first — the same constraint everyone else has.
- Why it reaches you
- Implicit caching means the saving arrives without you designing for it, which is unusual and worth knowing.
Grok
xAI- The bottleneck
- Latency inside the agentic coding loop, where the model is called dozens of times per turn and each round trip is felt.
- What they do about it
- A model architecture built from scratch for agentic coding rather than adapted from a general model, plus heavy investment in prompt caching. The serving stack is Kubernetes, Rust and JAX.
- The published result
- xAI reports prompt-cache hit rates regularly above 90% with launch partners, and prices cached input at $0.02 per million against $0.20 for fresh input — a tenth, the same ratio Anthropic charges.
- The tradeoff
- xAI credits its inference and supercomputing teams with 'several innovative techniques' to accelerate serving and does not say what they are. There is no published engineering account to check, and its data-centre capacity claims have been disputed in reporting.
- Why it reaches you
- It is the clearest published example of building a separate small model for the agent loop rather than routing to one.
10From model to power grid — where AI bottlenecks actually move
Thirteen layers, re-shaded for four different workloads. The most expensive mistake in performance work is optimising the layer you happen to own.
The most expensive mistake in AI performance work is optimising the layer you happen to own. A kernel engineer will find kernel wins, a model team will find model wins, and both can be right and irrelevant, because the thing setting the rate was three layers away and belonged to someone else.
Two published results make the point better than an argument would. FlashAttention did not reduce the arithmetic in attention at all — it produces mathematically identical output — and delivered 3× end-to-end on GPT-2 at 1K context by keeping intermediates in on-chip SRAM instead of writing them back to high-bandwidth memory. The constraint was never the maths.12 And PagedAttention borrowed virtual-memory paging from operating systems to fix a KV cache in which 60–80% of the memory was being lost to fragmentation and over-reservation, producing 2–4× throughput against contemporaneous systems.13 When a field hits a wall, the answer is often forty years old and filed under a different discipline.
The layer model below is a working tool rather than a taxonomy. Read it with the workload selected, because the binding layer genuinely differs: training binds on interconnect and power, prefill on arithmetic, decode on memory bandwidth, and agentic loops on context and cache — which is why a serving stack tuned for chat is wrong for agents in four separate places at once.
Writing the answer, one token at a time. Bandwidth-bound and hard to keep busy.
The largest wins on this page are usually here and are usually declined, because they are not technically interesting. A request never made costs nothing and never fails.
Usually owned by
Product
Choosing a smaller model that is good enough beats every kernel optimisation below it, and takes less engineering.
Usually owned by
Research / applied ML
This is where exact-output optimisations live — the ones you can adopt without re-validating quality.
Usually owned by
Research engineering
vLLM, SGLang, TensorRT-LLM, PyTorch. Most teams' biggest single jump comes from configuring this layer properly, not from replacing it.
Usually owned by
Inference / platform
Increasingly written with model assistance. OpenAI attributes a 20% serving-cost reduction in 2026 partly to kernels its own model wrote.
Usually owned by
Performance engineering
Deep, narrow, and extremely well paid. It is also the layer most sensitive to hardware changing underneath it.
Usually owned by
GPU engineering
For decode this is usually the true constraint. Microsoft gave Maia 200 216GB of HBM3e at 7 TB/s and 272MB of on-chip SRAM for exactly this reason.
Usually owned by
Systems / hardware
GPU, TPU, Trainium, MTIA, Maia. Hennessy and Patterson predicted this: when general-purpose scaling stops, specialisation is what is left.
Usually owned by
Silicon
Every parallelism strategy is a bet on this layer. It is why the size of a coherent NVLink domain is a competitive variable and not a spec-sheet detail.
Usually owned by
Network engineering
Meta's disaggregated serving needed multiple NICs bound to the nearest GPUs before the KV transfer stopped being the bottleneck.
Usually owned by
Hardware / infrastructure
The rack, not the chip, is now the unit of purchase. Uptime Institute reports a growing share of operators at peak densities of 30kW and above.
Usually owned by
Data centre engineering
Google runs a fleet PUE of 1.09 against an industry average of 1.54. At hyperscale that gap is worth more than most product features.
Usually owned by
Facilities / capacity planning
As of September 2026 this is where the binding constraint sits. A median wait of over five years to interconnect is not an engineering problem you can optimise your way out of from layer 8.
Usually owned by
Energy / corporate development
Shading marks how often each layer is the binding constraint for the selected workload — rarely, sometimes, usually. It is a judgement argued from the sources on this page, not a measurement, and it will be wrong for some specific deployment. The point it is making is structural: the layer that binds moves, and it is frequently not the one your team owns.
11How big tech optimizes today — eleven companies, one shape
OpenAI, Anthropic, Google, xAI, NVIDIA, AMD, Apple, Microsoft, AWS, Meta and Tesla: what each optimises, what they use, why it pays, who does the work, and what changed in 2026.
Almost every performance figure below is a company measuring its own product. That does not make it false; it makes it a design target rather than an independent result. Each profile carries the claim, then a reading of what the claim can actually support.
| Company | What they optimise | What they use | Why it matters |
|---|---|---|---|
| OpenAIChatGPT, Codex | Cost per served token, Kernel efficiency, Draft-model acceptance | Triton and Gluon kernels, Speculative decoding, Prompt caching | Serving cost sets the price, and the price sets which applications are viable at all. |
| AnthropicClaude, Claude Code | Input token cost, Cache hit rate, Context window usage | Prompt caching, Message Batches API, Code execution with MCP | Agent cost is dominated by input tokens, not output — the opposite of the chat era. |
| GoogleGemini, TPUs | Performance per watt, Overhead energy (PUE), Cached input tokens | TPU v7 Ironwood, Implicit context caching, Gemini Flash tier | At Google's base, a hundredth of a PUE point is worth a building programme. |
| xAIGrok | Time to first token in coding agents, Cached input cost, Tool-call latency | Prompt caching, A model architecture built from scratch for agentic coding, Kubernetes, Rust and JAX serving stack | In an agent loop the model is called dozens of times per turn, so per-call latency compounds. |
| NVIDIAGPUs, CUDA | Useful FLOPs per watt, Memory bandwidth, Token cost | CUDA, TensorRT-LLM, NVLink | Cost per token is the metric its customers are judged on, so it is the metric NVIDIA sells against. |
| AMDInstinct, ROCm | HBM capacity and bandwidth, Rack-scale integration, Software portability | CDNA architecture, Chiplets and Infinity Fabric, HBM4 | Leading with memory rather than FLOPS tells you which resource AMD expects to bind. |
| AppleApple silicon, Macs and iPhones | Performance per watt, Data movement between CPU and GPU, Sustained rather than peak performance | Unified memory, Neural Accelerators in each GPU core, Dedicated media engines | On a battery, work per watt is the only figure that reaches the customer. |
| MicrosoftAzure, Copilot | Performance per dollar, Memory bandwidth for decode, Water consumption | Maia 200, 216GB HBM3e and 272MB SRAM, All-Ethernet networking | Performance per dollar is the number a cloud actually sells. |
| AWSTrainium, Graviton | Tokens per megawatt, Training cost per model, Utilisation of reserved capacity | Trainium3, Inferentia, Graviton | A cloud's margin is the gap between what capacity costs it and what it sells for. |
| MetaLlama, PyTorch, MTIA | Cost of ranking and recommendation inference, Prefill and decode utilisation, Prefix cache hit rate | MTIA accelerators, PyTorch and vLLM, Prefill/decode disaggregation | Ranking inference is enormous, narrow and stable — the exact case where a custom chip beats a general one. |
| TeslaModel Y, Gigafactories | Part count, Robot count on the body line, Floor space per vehicle | High-pressure aluminium die casting, Single-piece structural castings, Vertical integration | Removing a joint removes the fixture, the robot, the inspection and a failure mode with it. |
OpenAI
ChatGPT, CodexServing more tokens on the same hardware, increasingly by letting the model optimise itself
What they optimise
- Cost per served token
- Kernel efficiency
- Draft-model acceptance
- Agent-loop overhead
- Load balancing
What they use
- Triton and Gluon kernels
- Speculative decoding
- Prompt caching
- KV-cache tuning
- Request routing
- FpSan verification
Why it matters commercially
- Serving cost sets the price, and the price sets which applications are viable at all.
- A 20% cut on a fleet this size is a capital decision, not an engineering detail.
- Compute is the binding constraint, so efficiency is capacity you did not have to buy.
Latest · through Sep 2026
29 July 2026: GPT-5.6 Sol, driven through Codex, autonomously rewrote production GPU kernels for a 20% reduction in end-to-end serving cost, and redesigned its own speculative-decoding draft model across hundreds of experiments for over 15% better token generation. Prices fell 80% for Luna and 20% for Terra the following day.
How to read it
The most interesting item on this page and the hardest to check. The system being optimised is doing the optimising, which is genuinely new — and it makes independent verification harder, not easier. OpenAI's answer to that objection is a floating-point sanitiser built specifically to verify kernels the model wrote. Self-reported and unreplicated.
Who does this work
Anthropic
Claude, Claude CodeMaking context cheaper — both the price of re-reading it and the need to send it at all
What they optimise
- Input token cost
- Cache hit rate
- Context window usage
- Throughput on non-urgent work
What they use
- Prompt caching
- Message Batches API
- Code execution with MCP
- Context compaction
- Trainium capacity
Why it matters commercially
- Agent cost is dominated by input tokens, not output — the opposite of the chat era.
- A cache read costs a tenth of a fresh read, so cache design is pricing design.
- Halving the tokens an agent needs is worth more than any kernel on the same workload.
Latest · through Sep 2026
April 2026: over one million Trainium2 chips already training and serving Claude, with up to 5GW of capacity committed and nearly 1GW of Trainium2 and Trainium3 due by the end of 2026.
How to read it
The published caveat is the useful part. Anthropic states plainly that running agent-written code needs sandboxing, resource limits and monitoring that direct tool calls do not — and its own cache contract means the compaction techniques that shrink context are exactly the edits that invalidate the cache. Two of the biggest levers pull against each other.
Who does this work
Compounding efficiency across seventeen years of data centres and seven generations of silicon
What they optimise
- Performance per watt
- Overhead energy (PUE)
- Cached input tokens
- Model tier per request
What they use
- TPU v7 Ironwood
- Implicit context caching
- Gemini Flash tier
- Liquid cooling
- JAX and XLA
Why it matters commercially
- At Google's base, a hundredth of a PUE point is worth a building programme.
- Owning the silicon means the model and the chip can be designed against each other.
- A Flash tier that is close enough on most traffic is a router in product form.
Latest · through Sep 2026
Ironwood, the seventh-generation TPU, is quoted at twice the performance per watt of Trillium, with 192GB of HBM and 7.37 TB/s per chip in liquid-cooled pods of 9,216. Fleet PUE was 1.09 for 2025 against an industry survey average of 1.54.
How to read it
The boring number is the impressive one: overhead fell from 0.20 to 0.09 watts per watt of computing across seventeen years with no single breakthrough in the curve. Google does not publish the discount its implicit caching applies, only that savings are passed on.
Who does this work
xAI
GrokA purpose-built fast model for the agentic loop, and aggressive prompt caching
What they optimise
- Time to first token in coding agents
- Cached input cost
- Tool-call latency
What they use
- Prompt caching
- A model architecture built from scratch for agentic coding
- Kubernetes, Rust and JAX serving stack
Why it matters commercially
- In an agent loop the model is called dozens of times per turn, so per-call latency compounds.
- Cached input priced at a tenth of fresh input makes the repeated prefix affordable.
Latest · through Sep 2026
grok-code-fast-1 is priced at $0.20 per million input tokens, $1.50 per million output and $0.02 per million cached input — cached reads at a tenth of fresh input, the same ratio Anthropic charges. xAI reports cache hit rates regularly above 90% with its launch partners.
How to read it
Included because there is real first-party evidence, and bounded because there is not much of it. xAI credits its inference and supercomputing teams with 'several innovative techniques' to accelerate serving and does not say what they are. Its data-centre capacity claims are contested in reporting. Nothing about xAI's serving internals is asserted here beyond what it has published itself.
Who does this work
NVIDIA
GPUs, CUDAMoving the unit of optimisation from a kernel, to a chip, to a rack, to a building
What they optimise
- Useful FLOPs per watt
- Memory bandwidth
- Token cost
- Rack-level power smoothing
What they use
- CUDA
- TensorRT-LLM
- NVLink
- NVFP4 low precision
- Warm-water cooling
- Rack-scale energy storage
Why it matters commercially
- Cost per token is the metric its customers are judged on, so it is the metric NVIDIA sells against.
- When the chip stops being the constraint, whoever ships the rack owns the optimisation.
- Performance per watt decides how much compute fits inside a fixed power contract.
Latest · through Sep 2026
Vera Rubin, announced 16 March 2026: up to 4× training and up to 10× inference performance per watt against Blackwell, one-tenth the token cost, 45°C warm-water cooling and 6× more rack-level energy storage. Data Center revenue was $89.0B in the quarter ended 26 July 2026, up 117% year on year.
How to read it
The performance figures are vendor claims for hardware nobody outside NVIDIA has measured — read them as design targets. What is not a claim is the change in the unit of design: a rack, a cooling regime and a power-smoothing scheme now ship as one product, because the constraint stopped being the chip.
Who does this work
AMD
Instinct, ROCmThe same constraint, answered with memory capacity and open standards
What they optimise
- HBM capacity and bandwidth
- Rack-scale integration
- Software portability
What they use
- CDNA architecture
- Chiplets and Infinity Fabric
- HBM4
- ROCm
- Helios rack design
Why it matters commercially
- Leading with memory rather than FLOPS tells you which resource AMD expects to bind.
- An open-standards stack is a bet that vendor lock-in becomes a constraint of its own.
Latest · through Sep 2026
The Helios rack-scale design pairs 72 Instinct MI455X GPUs with EPYC CPUs and Pensando networking on open industry standards: up to 2.9 exaflops of OCP MXFP4 compute and 31TB of HBM4 at up to 1.7 PB/s peak theoretical bandwidth.
How to read it
Read the order of the numbers. Memory capacity and bandwidth come before FLOPS — the same conclusion Microsoft reached independently with Maia 200. 'Peak theoretical' is doing work in that bandwidth figure, and the platform is in early deployment with no independent measurement.
Who does this work
Apple
Apple silicon, Macs and iPhonesTreating the power budget as the product, and designing chip and OS against each other
What they optimise
- Performance per watt
- Data movement between CPU and GPU
- Sustained rather than peak performance
What they use
- Unified memory
- Neural Accelerators in each GPU core
- Dedicated media engines
- Per-block power gating
Why it matters commercially
- On a battery, work per watt is the only figure that reaches the customer.
- Unified memory removes a copy — the same data-movement argument as FlashAttention, at a different scale.
- Running useful models on the device removes a server request entirely.
Latest · through Sep 2026
August 2026: M6 quotes 170GB/s of unified memory bandwidth, 2.5× M1, with Neural Accelerators in each of twelve GPU cores. M5 Ultra quotes 1.2TB/s and up to 4.5× the peak GPU compute for AI of M3 Ultra.
How to read it
Every figure is Apple comparing Apple to Apple across its own generations. The architectural argument stands on its own regardless: a battery and a substation are both power budgets, and the same design instinct applies to each.
Who does this work
Microsoft
Azure, CopilotInference-shaped silicon and closed-loop heat, measured against its own fleet
What they optimise
- Performance per dollar
- Memory bandwidth for decode
- Water consumption
- Cluster scheduling
What they use
- Maia 200
- 216GB HBM3e and 272MB SRAM
- All-Ethernet networking
- Second-generation closed-loop cooling
Why it matters commercially
- Performance per dollar is the number a cloud actually sells.
- Closed-loop cooling removes water as a siting constraint entirely.
- Decode is bandwidth-bound, so a decode machine is a memory machine.
Latest · through Sep 2026
Maia 200, announced 26 January 2026: over 10 petaFLOPS FP4, 216GB of HBM3e at 7 TB/s, 272MB of on-chip SRAM on TSMC 3nm, and a claimed 30% better performance per dollar than the newest hardware already in Microsoft's fleet.
How to read it
The memory numbers say what the chip is for: 272MB of SRAM and 7 TB/s of HBM is built to keep weights moving, not to set a FLOPS record. The performance-per-dollar claim is measured against Microsoft's own fleet, not against competitors.
Who does this work
AWS
Trainium, GravitonOwning the silicon to change the cost curve, and quoting throughput per megawatt
What they optimise
- Tokens per megawatt
- Training cost per model
- Utilisation of reserved capacity
What they use
- Trainium3
- Inferentia
- Graviton
- Nitro
- UltraServers and UltraClusters
Why it matters commercially
- A cloud's margin is the gap between what capacity costs it and what it sells for.
- Quoting tokens per megawatt tells you which resource the seller expects to run out of.
Latest · through Sep 2026
Trainium3 UltraServers reached general availability with claims of up to 4.4× the compute and 4× the energy efficiency of Trainium2, and over 5× the output tokens per megawatt at similar per-user latency. Anthropic runs over a million Trainium2 chips on it.
How to read it
Vendor benchmarks on vendor silicon. The giveaway is the unit: when a cloud provider starts quoting throughput per unit of power rather than per unit of hardware, it is telling you what it expects to be short of.
Who does this work
Meta
Llama, PyTorch, MTIACustom silicon for a workload it knows exactly, and splitting inference into two machines
What they optimise
- Cost of ranking and recommendation inference
- Prefill and decode utilisation
- Prefix cache hit rate
What they use
- MTIA accelerators
- PyTorch and vLLM
- Prefill/decode disaggregation
- Sticky routing
- Multi-NIC KV transfer
Why it matters commercially
- Ranking inference is enormous, narrow and stable — the exact case where a custom chip beats a general one.
- Prefill is compute-bound and decode is bandwidth-bound; one pool sized for both is wrong for each.
- Upstreaming the work means the improvements outlive the internal deployment.
Latest · through Sep 2026
March 2026: hundreds of thousands of MTIA chips in production, with generations 300 through 500 deployed or scheduled through 2027 — a 4.5× increase in HBM bandwidth and 25× in compute FLOPS from MTIA 300 to 500.
How to read it
Meta publishes no GPU comparison in that post and none should be inferred. Its disaggregated serving results come with the benchmark shape attached — Llama 4 Maverick on 8×H100 at 2,000 tokens in and 150 out — and Meta is explicit that the pattern only pays above a certain scale and concurrency.
Who does this work
Tesla
Model Y, GigafactoriesOptimising the factory by redesigning the product, and paying for it in repairability
What they optimise
- Part count
- Robot count on the body line
- Floor space per vehicle
What they use
- High-pressure aluminium die casting
- Single-piece structural castings
- Vertical integration
Why it matters commercially
- Removing a joint removes the fixture, the robot, the inspection and a failure mode with it.
- Part count is the process — the same argument as kernel fusion, in metal.
Latest · through Sep 2026
Tesla's own collision repair manual records that a Model Y may have a one-piece or a two-piece cast rear under body 'depending on the build date and location of manufacture' — the tradeoff, documented by the company that made the change.
How to read it
The weakest-sourced company on this page, and marked as such. The seventy-parts-to-one and roughly-300-robots figures are company statements relayed by trade press; Tesla has published no engineering account of them. What Tesla does publish is the repair procedure, which is where the cost of fusing a system shows up.
Who does this work
12Data centres, power and the grid — where the constraint sits now
Why a hundredth of a PUE point is worth a capital programme, and why a five-year interconnection queue is something no kernel can optimise around.
At hyperscale, arithmetic that looks trivial becomes decisive. Google's fleet has run at a trailing PUE of 1.09 for 2025 against an industry survey average of 1.54, from a published series that starts at 1.20 in 2008.24 Overhead fell from 0.20 to 0.09 watts per watt of computing over those seventeen years, with no single breakthrough visible in the curve. On a base that large, a hundredth of a point is worth a building programme — which is the whole argument for why infrastructure optimisation attracts the engineering that it does.
The vocabulary of every vendor shifted at the same time, and it is worth reading as evidence. AWS quotes Trainium3 at over 5× the output tokens per megawatt of the previous generation.25 NVIDIA quotes up to 10× inference performance per watt for Vera Rubin against Blackwell, one-tenth the token cost, 45°C warm-water cooling, and rack-level energy storage to smooth load.21 Google quotes Ironwood at 2× the performance per watt of Trillium.23 These are vendor claims about vendor products and none of them has been independently measured. What they collectively reveal is not a performance number. It is which resource everyone selling infrastructure expects to run out of.
The IEA put data centres at about 415 TWh in 2024 — roughly 1.5% of world electricity — with a base case near 945 TWh by 2030 and accelerated servers growing around 30% a year against 9% for conventional ones.30 Berkeley Lab's US reference case is 649 TWh in 2030, about 11.8% of national electricity, in a scenario range from 9.5% to 15.3%.31 The width of that range is the interesting part: it is driven by assumptions about AI server idle power and utilisation. Whether US data centres consume 9.5% or 15.3% of the country's electricity in 2030 is not a hardware question. It is an operations question, and it is the same one Toyota was asking about a machine sitting idle.
And then the constraint left the building. At the end of 2025, roughly 8,200 projects representing 1,312 GW of generation and 749 GW of storage were sitting in US interconnection queues. The median time from interconnection request to commercial operation was over five years for projects built in 2025, and of the capacity that requested interconnection between 2000 and 2020, only 13% had reached commercial operation by the end of 2025 — 75% was withdrawn.32 Meanwhile the IEA expects data centres to account for about half of US electricity demand growth to 2030.46
13Biotech, medicine and cancer research — where cost is not the objective
When nineteen of every twenty programmes fail, the valuable optimization is not a cheaper trial. It is an earlier no.
Everything above has an objective function that reduces, eventually, to cost or speed. Life sciences does not, and reading it with a manufacturing cost mindset produces confident, wrong answers.
Start with the portfolio arithmetic. Across 20,398 clinical development programmes from 2001 to 2023, the most recent window shows about 50% of programmes clearing Phase 1, about 29% clearing Phase 2, about 54% of those reaching approval, and roughly 5% overall.34 In a business where nineteen of every twenty programmes fail, making each trial 10% cheaper is worth far less than learning six months earlier which trial to stop. The optimisation target is not cost per experiment. It is information per unit of time and risk.
That said, the manufacturing curve in biologics is one of the most impressive in this entire study and almost nobody outside the field knows about it. Antibody titre rose from roughly 50 mg/L in 1986 to 3–10 g/L by the 2010s — between sixty- and two-hundred-fold depending where you draw the line — from cell-line engineering, media and feed development, and process control applied patiently over thirty years.41 It is the same shape as Google's PUE curve: no dramatic step, an enormous total, and only visible if someone was keeping the series.
Three more patterns transfer directly. Capacity can be the constraint, on a decade-long lead time — Lilly announced a $6 billion API site in Alabama in December 2025 with construction starting in 2026 and completion expected in 2032, the third of four new US sites in the same programme.48 Cycle time can be the whole game — Moderna delivered the first clinical batch of mRNA-1273 to the NIH 42 days after the viral sequence was published, because design, ordering and a highly automated plant were one connected process rather than a series of handoffs.45 And the binding constraint is frequently a document: ICH Q13 made continuous manufacturing a normal regulatory filing rather than an exception, which is what allowed integrated processes with fewer steps and real-time quality monitoring to reach commercial scale.42 A regulatory constraint is still a constraint. It just has a five-year lead time, like a substation.
More product from the same bioreactor
Yield: a hundred-fold, quietly, over thirty years
Antibody titre rose from roughly 50 mg/L in 1986 to 3–10 g/L by the 2010s — not from one breakthrough but from cell-line engineering, media and feed development, and process control applied over three decades. It is the same compounding curve as Google's PUE, in a different medium: no single step was dramatic and the total is two orders of magnitude.
Fewer expensive failures
The real objective is failing earlier, not cheaper
Across 20,398 clinical development programmes from 2001 to 2023, the 2015–2023 window shows roughly 50% of programmes clearing Phase 1, about 29% clearing Phase 2, about 54% of those reaching approval — and about 5% overall. Phase 2 is consistently the narrowest gate. In a portfolio like that, the highest-value optimisation is not making trials cheaper. It is learning that a molecule will fail before Phase 3 rather than during it.
A better shortlist, before anything is spent
The one input that measurably moves the odds
If Phase 2 is where programmes die, the highest-leverage decision is which target to take there. Human genetic evidence is the one selection input with a measured effect on approval. The number has moved, and the movement is the interesting part: the original finding was that genetically supported targets are twice as likely to be approved, and a 2019 revision in PLOS Genetics found that Mendelian and coding-variant evidence holds up or exceeds that, while GWAS evidence is consistently weaker and, in independent replication, not clearly different from zero. A 2026 preprint across 26,278 target-disease pairs replicates the enrichment — odds ratio 3.25, falling to 2.79 once genes shared across pairs are accounted for — and then says the thing that matters most: the predictive value is poor and the model is badly calibrated. A real effect that shifts a portfolio's odds is not the same as a rule that picks the next drug, and Regeneron has built a database of over three million sequenced exomes on the strength of the first without anyone claiming the second.
Cycle time and data throughput
The same two levers, in a regulated plant
Pfizer's own account of manufacturing and running PAXLOVID reads like a Lean case study with different nouns: analysing supply-chain data to find and monitor problems, and cutting the cycle time of one critical step by 67% — which produced 20,000 extra doses per batch, because a step that ran faster released capacity everywhere downstream of it. On the clinical side it reports checking and analysing trial data 50% faster, with AI or ML now used in more than half its trials. It is first-party and specific, with no methodology published and one named step rather than a plant-wide number — but it is the clearest published example of throughput and cycle time behaving in pharma exactly as they behave on a line.
Fewer wet experiments per answer
Moving the experiment before the bench
Structure prediction changed the shape of the work rather than the speed of any single step: simulate the interface, identify the contacts that matter, and take only the reduced subset to the bench. The constraint moves from 'can we determine this structure' to 'can we design the right experiment' — which is exactly the pattern this study keeps finding. Genentech has a name for the resulting shape and describes it in almost the words section 15 uses: experimental data feeds computational models, the models make testable predictions, scientists assess them in the lab, and the results retrain the models. They call it the lab in a loop. No efficiency figures are published for it, and none are claimed here.
Days from decision to dose
When cycle time is the whole game
Moderna delivered the first clinical batch of mRNA-1273 to the NIH 42 days after the viral sequence was published — possible because sequence design, ordering and a highly automated plant were one connected digital process rather than a series of handoffs. The figure comes from a cloud vendor's case study of its own customer, and is corroborated by the trial record.
Doses that physically exist
When the constraint is a building, the lead time is a decade
Lilly announced a $6 billion active-pharmaceutical-ingredient site in Huntsville, Alabama in December 2025 — construction from 2026, completion expected in 2032, around 450 roles. It is the third of four new US sites in the same programme, and Lilly’s ninth announced since 2020. It is the pharmaceutical version of the interconnection queue: when demand outruns installed capacity, no amount of process optimisation upstream changes the answer, and the relief has a six-year lead time. Planning under that constraint is a different discipline from improving throughput inside it.
What the filing will allow
The constraint that is written down
ICH Q13 made continuous manufacturing a normal regulatory pathway rather than an exception, which is what let integrated processes with fewer steps, shorter processing times and real-time quality monitoring reach commercial scale. In regulated industries the binding constraint is often a document, and relieving it takes years rather than quarters — but it is still a constraint, and it still responds to the loop.
On cancer specifically
Nothing here claims that operational optimization improves a cancer treatment. It does not. What optimization reaches is the system around the treatment — how quickly a failing target is dropped, how reliably a batch is released, how long a patient waits between a decision and a dose. An earlier study on this site takes that last one apart, and finds that most of a patient’s wait sits in the part nobody was measuring.
Read the cancer-care study →14Seven methods worth stealing from other industries
Airlines, parcels, fulfilment, semiconductors, consumer hardware, pharma and automotive — chosen because each names a mechanism that transfers.
The examples below were chosen because each illustrates a method rather than a success. A case study that only demonstrates that a large company saved money is not useful; a case study that names a transferable mechanism is.
The pattern worth noticing across all of them: in six of the seven, the winning move was to change what was being optimised, not to get better at optimising the existing thing. American stopped optimising load factor and started optimising revenue per seat mile. UPS stopped planning routes and started re-solving them. Amazon stopped making individual robots faster and started coordinating the fleet. The semiconductor industry stopped selling frequency. Apple stopped optimising peak performance and optimised work per watt. Tesla changed the part count instead of the assembly sequence. Redefining the objective is the highest-leverage and least technical move available, and it almost never appears in an engineering roadmap.
- AirlinesAmerican Airlines
Revenue management
Treat a perishable seat as an inventory problem: forecast demand by fare class, overbook against the no-show distribution, and price the last seats against the opportunity cost of the next booking. American estimated $1.4B of incremental revenue over three years.
Where it transfers
Any capacity that expires — cloud instances, ad impressions, freight, clinic slots, and now GPU-hours on a reserved cluster.
- Parcel logisticsUPS
Continuous re-optimisation
Solve every driver's route every day against street-level data, then keep solving as the day changes. Over ten years to build; INFORMS records more than $320M saved by December 2015 against a $250M deployment.
Where it transfers
The pattern matters more than the routing: re-solve continuously instead of planning once, and treat the plan as a control variable.
- FulfilmentAmazon
Fleet-level coordination
Rather than making each robot faster, coordinate the whole fleet's movement. Amazon reports its DeepFleet model improved robot travel time by 10% across more than 300 facilities.
Where it transfers
Congestion is a system property. The same argument applies to GPU schedulers, warehouse aisles and city traffic: local speed is not fleet speed.
Sources36 Amazon → - SemiconductorsThe industry, after ~2005
Specialisation over scaling
When Dennard scaling ended, the industry stopped selling frequency and started selling parallelism and then domain-specific accelerators — the shift Hennessy and Patterson describe as the reason a new golden age of architecture began.
Where it transfers
When the general-purpose lever breaks, the answer is to give up generality where you can afford to. Every AI accelerator in this study is an instance.
- Automotive manufacturingTesla
Optimising the factory by redesigning the product
Replace an assembly of about seventy stamped and welded underbody parts with a single aluminium casting. Tesla reported reaching volume production of the Model Y single-piece rear underbody in early 2021 and said the change removed roughly 300 robots from the body line — figures that remain company statements relayed by trade press, because Tesla has published no engineering account of them. What Tesla does publish is the consequence: its own collision repair manual sets out which damage to the casting can be welded or cold-straightened and which forces replacement of the entire under body, and records that a Model Y may have a one-piece or a two-piece rear under body 'depending on the build date and location of manufacture'. The tradeoff is documented by the company; the benefit is asserted by it.
Where it transfers
The part count is the process. Removing an interface removes the fixture, the robot, the inspection, the fastener inventory and a failure mode along with it — which is the same argument as kernel fusion in section 08, and the same one as unified memory in Apple silicon. The two-piece variant is the rest of the lesson: fusing a system raises the cost of every repair to it, and that cost lands on somebody who was not in the room when the decision was made.
- Consumer hardwareApple
Performance per watt as the design objective
Design silicon, operating system and product together and optimise for work per watt rather than peak clock: unified memory so data is not copied between CPU and GPU, dedicated media and neural blocks, and accelerators inside each GPU core. Apple's M6 quotes 170GB/s unified bandwidth, 2.5× M1; M5 Ultra quotes 1.2TB/s.
Where it transfers
It is the same objective function as a data centre, at a different scale. A battery and a substation are both power budgets.
Sources28 Apple → - Pharmaceutical manufacturingICH / global regulators
Changing what the rules allow
ICH Q13 made continuous manufacturing a normal regulatory filing rather than an exception, which is what let integrated processes with fewer steps and real-time quality monitoring reach commercial scale.
Where it transfers
Sometimes the binding constraint is not physics or capital but what a regulator, an auditor or a standard will accept. That constraint is also relievable, just on a longer timescale.
15Roles that optimize AI, factories, chips and data centres
Twelve job families doing the same work in different units — with the metric each is judged on, the tools they use, and a real problem from the desk.
This work is done under at least thirty different job titles across a dozen industries, and the titles obscure how similar the work is. An industrial engineer balancing a line and an inference engineer tuning a serving fleet are asking the same question in different units: what is setting the rate, and what does it cost to relieve it.
The market for the traditional end of this is not shrinking, which surprises people. The US Bureau of Labor Statistics reports 365,100 industrial engineers at a median of $102,440, growing 12% between 2025 and 2035, and 113,100 operations research analysts at a median of $88,940, also growing 12%.37 38 Both are described as much faster than the average for all occupations. The specialised AI-systems end of the ladder pays far more and is far smaller; the useful observation is that the two ends are converging, and almost nobody can read both.
That gap is the actual opportunity. Constraint analysis, measurement-system discipline and flow thinking are a hundred years old and are not standard equipment in AI infrastructure teams. Kernel-level and fleet-level knowledge is not standard equipment in operations. The scarce person is the one who can hold both, because the constraint is usually at the seam where nobody owns it.
Operations
Industrial Engineer
Optimises throughput, cycle time and labour content in a physical process.
Judged on
- Units per hour
- Takt vs cycle time
- OEE
- Cost per unit
Core skills
- Process mapping
- Line balancing
- Work measurement
- Capacity modelling
- Statistics
Common tools
- Value-stream maps
- Simulation (Arena, FlexSim)
- SQL
- Python
- MES data
Where the work is
Manufacturing, logistics, healthcare operations, fulfilment, aerospace
A problem on the desk
A line runs at 82% of its rated rate and nobody agrees why. Instrument it, find the true constraint, and prove the fix in a pilot cell before capital is spent.
16The skills behind all of it, arranged as a stack
Seven tiers, from framing the problem to negotiating for megawatts, in the order the dependencies actually run.
The map below runs from the most general skill to the most specific, and it is deliberately shaped like the technical stack in section 09, because the two mirror each other. Systems thinking sits at the top for the same reason the application layer does: a mistake there cannot be recovered by excellence anywhere below it.
Read it downward to see what each tier depends on. Read it upward to see the more common career path — someone who is excellent at tier six or seven and has never been given the vocabulary for tiers one to four, which is why so much brilliant performance engineering lands on a system nobody needed to be faster.
- 01
Systems thinking
Seeing the whole before touching a part. Without this layer the rest optimises the wrong thing very efficiently.
- Constraint identification
- Tradeoff analysis
- Root cause analysis
- Second-order effects
- Problem framing
- Decision quality
Everything below it is a means to an end this layer defines.
- 02
Operations
The century-old toolkit for making flow visible and rate-limited by something you chose.
- Lean
- Value-stream mapping
- Six Sigma / DMAIC
- Bottleneck analysis
- Queueing
- Capacity analysis
- Scheduling
- Standard work
Turns a systems view into a measurable process.
- 03
Analytics & OR
Making the improvement provable and the decision optimal rather than plausible.
- Statistics
- Design of experiments
- Measurement-system analysis
- Forecasting
- Optimisation modelling
- Simulation
- Causal inference
Supplies the evidence operations claims and product spends against.
- 04
Product & business
Connecting a technical gain to money, and choosing which gain to pursue at all.
- Unit economics
- KPI design
- Prioritisation
- Cost–benefit analysis
- Customer value
- Roadmapping
- Written argument
Decides which constraint is worth relieving this quarter.
- 05
AI systems
The 2020s layer: what makes a model cheap, fast and good enough for the job in front of it.
- Evaluation design
- Inference economics
- Quantization
- Batching
- Caching
- Model routing
- Context engineering
- Agent design
Where product intent meets the cost of a token.
- 06
Compute
Where the bytes actually move. Narrow, deep, and the difference between a 40% gain and a 4× one.
- CUDA / Triton
- Kernel optimisation
- Profiling
- Memory hierarchy
- Numerics
- Distributed training
- Parallelism strategy
Executes what the AI layer asked for, at the rate the hardware allows.
- 07
Infrastructure & energy
The layer that turns out to bind everything above it, and the one fewest software people can read.
- GPU cluster operations
- Networking and topology
- Resource scheduling
- Capacity planning
- Cooling and power
- Grid and siting
Sets the ceiling every other layer negotiates under.
17The playbook — eight steps that keep working
Applied to a Toyota line, a software service, a GPU fleet, a data centre and a biologics plant, without rewording them. Two steps get dropped every time.
What follows is the method this whole study is evidence for. It is not new — steps four through six are Goldratt's, step two is borrowed from metrology, and steps three and five are Lean. What is worth arguing about is which steps get dropped, because the dropped ones are consistent across every industry represented on this page.
Step two goes first. Qualifying the measurement before arguing about results is required practice in manufacturing and is almost never done in software or AI. A team that cannot say whether its instrument can resolve the difference it is about to act on is going to spend a quarter chasing noise, and will not find out.
Step eight goes second. Relieving a constraint does not remove it, and the organisation that treats improvement as a project rather than a loop will declare victory precisely at the moment the problem relocates — usually onto a team that was not consulted. The entire timeline in section 01 is one long sequence of step eight, and the fact that it now terminates at a substation rather than a stopwatch is the strongest evidence the loop is real.
The loop is also the part of this that does not date. The techniques on this page will look quaint within a decade — most of the 2026 serving optimisations will be defaults or obsolete, and the power constraint will be relieved and replaced by something not yet visible. The method for finding out what to fix next has survived four industrial revolutions without material change, which is the strongest claim this study makes and the one most open to being wrong.
- 01
Name the outcome
What are we actually trying to produce more of, and for whom?
Almost every failed improvement programme optimised a proxy. If you cannot say what the system is for in one sentence, everything downstream will be locally correct and globally useless.
An AI serving fleet
Useful completed tasks per dollar — not tokens per second.
- 02Usually dropped
Qualify the measurement
Can our instrument resolve the difference we are about to act on?
This is the step everyone skips. Manufacturing has refused to argue about parts before qualifying the gauge for sixty years; most software and AI organisations have never done it once.
An AI serving fleet
Two teams, same weights, different serving stack — do they get the same score?
- 03
Map the flow
Where does a unit of work actually spend its time?
Value-stream mapping under a different name. The answer is almost never where the team's attention is, because attention follows what is interesting rather than what is slow.
An AI serving fleet
Split the step: prefill, decode, tool calls, and the human pause.
- 04
Find the constraint
What one thing sets the rate for everything else?
Goldratt's insight, and the reason most improvement is wasted: an hour saved anywhere except the constraint is a mirage. It also means most of the system should be deliberately idle.
An AI serving fleet
For decode it is memory bandwidth; for agents it is context.
- 05
Remove the waste around it
What can we stop doing before we spend anything?
Exploit before you elevate. The cheapest capacity is the capacity you were already paying for and throwing away — and this step frequently makes the expensive step unnecessary.
An AI serving fleet
Cache the prefix, batch the non-urgent, route the easy work down.
- 06
Elevate the constraint
Now, what do we buy or rebuild?
This is the step organisations start with, which is why so much capital gets spent relieving a constraint that was not binding.
An AI serving fleet
Disaggregate prefill from decode; add HBM; change silicon.
- 07
Test it honestly
Did the system get better, or did the number?
A local improvement that moves inventory or latency somewhere else is not an improvement. Check the outcome from step 1, and check it against the instrument from step 2.
An AI serving fleet
Did cost per completed task fall, or only cost per token?
- 08Usually dropped
Find where it moved
What is the constraint now?
The step that makes this a loop rather than a project. Relieving a bottleneck does not remove it; it relocates it, usually to a team that was not in the room. A century of this page is one long instance of step 8.
An AI serving fleet
Fix memory and the interconnect binds; fix that and it is power.
And then it starts again, because the constraint has moved. That is the difference between an improvement programme and an improvement loop.
What is mine and what is not
Almost everything on this page was invented by somebody else. The contribution is narrower than the length might imply, and worth stating plainly.
Not mine
- Every technique on this page. Taylor, Ford, Shewhart, Deming, Ohno, Goldratt, Smith, Dao, Kwon and the engineering teams cited invented all of it.
- Every number. Each one belongs to the organisation that measured or published it, with its own setup and its own limits.
- The Theory of Constraints, which supplies the spine of the argument and is Goldratt's, from 1984.
Mine
- The framing: that the history of business optimization is best read as one relay of a moving constraint, not as a sequence of unrelated management fashions.
- The eight-step loop, and specifically the two steps most versions omit — qualify the measurement before arguing about results, and expect the constraint to move once you relieve it.
- The layer-by-layer stack model and the judgement about where each workload actually binds. That judgement is argued from the sources, not measured.
- The selection: which milestones and which 2026 techniques are load-bearing, and which are interesting but not.
Explicitly not claimed
- That any of this is new research. It is a synthesis of published work, assembled and argued.
- That any vendor performance figure here has been independently verified. Where a company measured its own product, the citation says so.
- That the ordering of the timeline implies causation. Ideas here overlapped, competed and were rediscovered; several were developed independently in more than one place.
- That optimization is always good. Taylor's methods raised output and degraded work; efficiency gains are routinely captured by someone other than the people who produced them. This page is about how systems are made to work better, not about who benefits when they are.
What would falsify this
If nothing would change the argument, it is not an argument. These would.
- 01A durable counter-example: a large system where relieving the binding constraint did not relocate it, but simply left the system unconstrained for years. The loop's eighth step predicts this should be rare.
- 02Evidence that the historical methods do not transfer — for instance, that teams applying constraint analysis to AI serving do no better than teams optimising whatever is nearest to hand.
- 03A measured demonstration that power, not memory or interconnect, is not the binding constraint on frontier AI capacity by 2027 — which would mean this study named the wrong bottleneck at the moment it mattered most.
- 04Independent measurement showing that the vendor performance-per-watt claims quoted here are off by enough to change the argument about where optimisation effort belongs.
Method and disclosure
What kind of study this is, what was and was not done, and where the weak points are.
This is a literature-and-documentation study. No benchmark was run and no experiment was performed. Every measured figure is attributed to the organisation that produced it, with its setup and its limits recorded in the citation note rather than rounded away.
Historical claims are sourced to the institutions holding the records — The Henry Ford, the Deming Institute, INFORMS, Toyota, ASQ — or to the primary text itself, as with Taylor. Where a historical account is contested, the text says so; Taylor’s published pig-iron figures are the clearest case.
Current technical claims prioritise papers and the engineering publications of the teams that built the thing. Where a company published its own performance figures, the citation is labelled Vendor and the reading beside it says what the number can and cannot support. Nothing here is an independently verified benchmark of unreleased hardware, and it does not pretend otherwise.
The judgements — the ordering of the constraint relay, which layer binds for which workload, which milestones are load-bearing — are arguments from the sources, not measurements, and they are labelled as arguments wherever they appear. Everything was checked against its source on 8 September 2026.
Sources
67 sources, each with what it does and does not support. Every one was read directly rather than from a search result.
- 01The Principles of Scientific Management
Frederick Winslow Taylor · Harper & Brothers (Project Gutenberg full text) · 1911 · Book
The primary text, read directly. Taylor's own account of the Bethlehem pig-iron and shovelling studies. Historians working from Taylor's papers have shown the published figures were selected and dramatised; the book is cited here as evidence of what was claimed and taught, not as a measurement.
- 02Assembly line — K-12 research guide
The Henry Ford · accessed September 2026 · Documentation
The museum holding Ford's own records. Gives 12½ hours per car under stationary assembly, a finished Model T every 93 minutes after the moving line, 15 million cars 1908–1927, and a price fall from $850 to as little as $260.
- 03Walter A. Shewhart — ASQ Honorary Member
American Society for Quality · accessed September 8, 2026 · Documentation
ASQ states it directly: “One of these was the historic memorandum of May 16, 1924, in which he proposed the control chart to his superiors.” Also dates Shewhart at Western Electric 1918–1924 and Bell Telephone Laboratories from 1925, and Economic Control of Quality of Manufactured Product to 1931. The text of the memorandum itself is reproduced by the W. Edwards Deming Institute.
- 04Deming, the man
The W. Edwards Deming Institute · accessed September 2026 · Documentation
Dates Deming's teaching in Japan from summer 1950 and the NBC documentary 'If Japan Can, Why Can't We?' to June 1980. The Institute is Deming's own foundation; treat its framing of his influence as an interested account.
- 05The Origins of O.R.
INFORMS · accessed September 2026 · Documentation
The professional society's own history. Dates the term 'operational research' to wartime Britain and radar integration, and notes that the mathematical machinery arrived later, when OR became a civilian profession.
- 06Toyota Production System
Toyota Motor Corporation · accessed September 2026 · Vendor
Toyota's own account of its two pillars, jidoka and Just-in-Time, and of Sakichi Toyoda, Kiichiro Toyoda, Eiji Toyoda and Taiichi Ohno. A company describing its own system.
- 07The Machine That Changed the World: The Story of Lean Production
James P. Womack, Daniel T. Jones, Daniel Roos · MIT International Motor Vehicle Program · 1990 · Book
The five-year, fourteen-country plant study that named 'lean production' and moved Toyota's methods out of automotive. Cited for the framing and the fact of the study, not for any individual plant figure.
- 08What is Six Sigma?
American Society for Quality · accessed September 2026 · Documentation
ASQ dates the concepts to work by Bob Galvin and Bill Smith at Motorola in the 1980s and gives the 3.4 defects per million opportunities target and the DMAIC phases.
- 09Yield Management at American Airlines
Barry C. Smith, John F. Leimkuhler, Ross M. Darrow · Interfaces 22(1), INFORMS · January–February 1992 · Paper
The paper that put a number on revenue management: American estimated $1.4 billion of incremental revenue over three years from DINAMO. A company's own estimate, published under peer review.
- 10UPS On-Road Integrated Optimization and Navigation (ORION)
INFORMS — Franz Edelman Award, 2016 · 2016 · Documentation
More than ten years to build, $250M to deploy; UPS reported over $320M saved by December 2015 and expected $300–400M annually at full deployment, 10 million gallons of fuel and 100,000 metric tons of CO₂ a year.
- 11A New Golden Age for Computer Architecture
John L. Hennessy, David A. Patterson · Communications of the ACM 62(2) · February 2019 · Paper
The Turing-lecture paper that dates the end of Dennard scaling and argues that with general-purpose performance growth exhausted, the remaining gains come from domain-specific architectures.
- 12FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré · NeurIPS 2022 · May 2022 · Paper
Reports 3× end-to-end speedup on GPT-2 at sequence length 1K and 2.4× on long-range arena, with mathematically identical attention. The gain is in memory traffic, not arithmetic.
- 13Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng et al. · SOSP 2023 · September 2023 · Paper
Measures 60–80% of KV-cache memory lost to fragmentation and over-reservation in the systems of the day, and reports 2–4× throughput at matched latency for vLLM against them. The comparison baselines are 2023 systems.
- 14DeepSeek-V3 Technical Report
DeepSeek-AI · December 2024 · Paper
671B total parameters with 37B activated per token; 2.788M H800 GPU-hours for the full training run. The oft-quoted $5.576M is the authors' own illustration at an assumed $2/GPU-hour rental and explicitly excludes prior research, ablations and data work.
- 15Pretraining Large Language Models with NVFP4
NVIDIA · September 2025, revised March 2026 · Paper
A 12B hybrid Mamba-Transformer trained on 10T tokens in 4-bit, reaching loss and downstream accuracy comparable to an FP8 baseline. NVIDIA publishing on NVIDIA's own numeric format; one model, one recipe.
- 16Prompt caching
Anthropic — Claude platform documentation · accessed September 8, 2026 · Documentation
Cache reads are priced at 0.1× base input tokens; 5-minute writes at 1.25×, 1-hour writes at 2×. Prefixes are built tools → system → messages, and a change at any level invalidates that level and everything after it.
- 17Code execution with MCP: building more efficient AI agents
Anthropic Engineering · November 2025 · Engineering
One worked example — a transcript moved from Google Drive to Salesforce — falls from 150,000 tokens to 2,000. Anthropic states the cost plainly: agent-generated code needs a sandbox, resource limits and monitoring that direct tool calls do not.
- 18Anthropic and Amazon expand collaboration for up to 5 gigawatts of new compute
Anthropic · April 20, 2026 · Vendor
States over one million Trainium2 chips already serving and training Claude, nearly 1GW of Trainium2 and Trainium3 by end of 2026, and run-rate revenue past $30 billion. A company describing its own commitments.
- 19Disaggregated Inference at Scale with PyTorch & vLLM
PyTorch Foundation / Meta engineering · September 2025 · Engineering
Meta's production account of splitting prefill from decode: why prefill is compute-bound and decode memory-bound, and how sticky routing plus load balancing held a 40–50% prefix-cache hit rate at 90% HBM utilisation. Benchmarks are Llama 4 Maverick on 8×H100 at 2,000 in / 150 out — one shape of traffic.
- 20Fly Eagle(3) fly: Faster inference with vLLM and speculative decoding
Alexandre Marques, Megan Flynn, Mark Kurtz · Red Hat Developer · July 2025 · Engineering
Up to 2.5× on latency at low request rates, and the honest converse: at high batch sizes the system is compute-bound and speculative decoding can make things worse. On translation the measured optimum was a draft length of 1, or 0.
- 21NVIDIA Vera Rubin POD: Seven Chips, Five Rack-Scale Systems, One AI Supercomputer
NVIDIA · March 16, 2026 · Vendor
NVIDIA claims up to 4× training performance and up to 10× inference performance per watt against Blackwell, and one-tenth the token cost, with 45°C warm-water cooling. Vendor figures for an unreleased platform; no independent measurement exists.
- 22NVIDIA Announces Financial Results for Second Quarter Fiscal 2027
NVIDIA · August 26, 2026 · Vendor
Quarter ended July 26, 2026: revenue $96.2B, Data Center $89.0B, up 117% year on year, gross margin 75.0%. Audited financial reporting, not a performance claim.
- 23Ironwood: The first Google TPU for the age of inference
Google · April 2025 · Vendor
2× performance per watt against Trillium, 192GB HBM and 7.37 TB/s per chip, 9,216 liquid-cooled chips per pod. Google's own figures for Google's own silicon.
- 24Power usage effectiveness — Google Data Centers
Google · accessed September 2026 · Vendor
Fleet-wide trailing-twelve-month PUE of 1.09 for 2025, against 1.54 for the Uptime Institute's 2025 survey average, and a published series back to 1.20 in 2008. Self-reported, but with the calculation shown.
- 25Trainium3 UltraServers now available
Amazon Web Services · December 2025 · Vendor
AWS claims up to 4.4× compute and 4× energy efficiency against Trn2, and over 5× the output tokens per megawatt at similar per-user latency. Vendor benchmarks on vendor silicon.
- 26Maia 200: The AI accelerator built for inference
Microsoft · January 26, 2026 · Vendor
30% better performance per dollar than the newest hardware in Microsoft's own fleet; 216GB HBM3e at 7 TB/s, 272MB on-chip SRAM, TSMC 3nm, second-generation closed-loop liquid cooling. A fleet-relative claim, not a cross-vendor benchmark.
- 27Four MTIA Chips in Two Years: Scaling AI Experiences for Billions
Meta · March 11, 2026 · Vendor
Hundreds of thousands of MTIA chips in production, with generations 300 through 500 deployed or scheduled through 2027; 4.5× HBM bandwidth and 25× compute FLOPS from MTIA 300 to 500. No GPU comparison is offered in the post, and none is inferred here.
- 28Apple introduces M6 and M5 Ultra for a big leap in performance and AI compute
Apple · August 2026 · Vendor
M6: 170GB/s unified memory bandwidth, 2.5× M1; Neural Accelerators in each GPU core. M5 Ultra: 1.2TB/s unified memory, up to 4.5× M3 Ultra peak GPU compute for AI. Marketing comparisons against Apple's own prior parts.
- 29Advancing the price-performance frontier with GPT-5.6
OpenAI · July 30, 2026 · Vendor
The commercial half of the same announcement: from 30 July 2026, GPT-5.6 Luna costs 80% less and GPT-5.6 Terra 20% less. OpenAI frames it as passing on the efficiency gains described the previous day in source 49. A vendor announcing its own prices.
- 30Energy demand from AI — Energy and AI
International Energy Agency · April 2025 · Documentation
Data centres consumed about 415 TWh in 2024, roughly 1.5% of world electricity, projected to about 945 TWh by 2030 in the base case. Accelerated servers grow ~30%/yr against ~9% for conventional. A scenario, not a forecast.
- 31United States Data Center Energy Usage Report: 2025 Update
Arman Shehabi et al. · Lawrence Berkeley National Laboratory · 2026 · Paper
Reference case of 649 TWh in 2030 — 11.8% of US electricity — with a scenario range of 9.5% to 15.3%. The high end is driven by assumptions about AI server idle power and utilisation, which is the point: the spread is an operations question, not a hardware one.
- 32Queued Up: 2026 Edition — Characteristics of Power Plants Seeking Transmission Interconnection
Joseph Rand, Anna Cheyette, Chris Talley, Steven Zhang, Will Gorman, Ryan Wiser, Joachim Seel, Seongeun Jeong, Fredrich Kahrl · Lawrence Berkeley National Laboratory · June 2026 · Paper
End of 2025: ~8,200 active projects, 1,312 GW of generation and 749 GW of storage waiting. Median time from interconnection request to commercial operation was over five years for projects built in 2025, and only 13% of capacity requested between 2000 and 2020 had reached operation by end-2025.
- 3316th Annual Global Data Center Survey: high-density racks rising fast
Uptime Institute · July 28, 2026 · Documentation
Over 800 operators surveyed. Growing numbers report peak rack densities of 30kW and above; PUE improved only marginally again; more than half report difficulty hiring. A self-selected industry survey, not a census.
- 34Dynamic clinical trial success rates for drugs in the 21st century
Ying Zhou et al. · Nature Communications · October 29, 2025 · Paper
20,398 clinical development programmes, 9,682 molecules, 2001–2023. In the 2015–2023 window: Phase 1→2 about 50%, Phase 2→3 about 29%, Phase 3→approval about 54%, and roughly 5% overall. Phase 2 is consistently the narrowest gate.
- 35The transformative impact of AI-enabled AlphaFold 3: evolution, current status and future prospects in structural biology
Frontiers in Artificial Intelligence · 2026 · Paper
A review, not new measurement. Useful for what changed in practice: designing the experiment in silico and validating only the reduced subset at the bench.
- 36Amazon deploys over 1 million robots and launches new AI foundation model
Amazon · July 2025 · Vendor
DeepFleet improves robot fleet travel time by 10% across more than 300 facilities. Amazon's own figure, with no methodology published.
- 37Occupational Outlook Handbook: Industrial Engineers
US Bureau of Labor Statistics · last modified August 27, 2026 · Documentation
Median pay $102,440 (May 2025), 365,100 jobs, projected 12% growth 2025–35.
- 38Occupational Outlook Handbook: Operations Research Analysts
US Bureau of Labor Statistics · last modified August 27, 2026 · Documentation
Median pay $88,940 (May 2025), 113,100 jobs, projected 12% growth 2025–35.
- 39LLM inference prices have fallen rapidly but unequally across tasks
Epoch AI · March 2025 · Documentation
Rates of decline for a fixed capability run from 9× to 900× per year depending on the benchmark; GPT-4-level performance on PhD-level science questions fell about 40× per year. Epoch flags that the fastest declines are the most recent and may not persist.
- 402026 Kubernetes Resource Optimization Report
Cast AI · 2026 · Vendor
Reports 8% CPU, 20% memory and 5% GPU utilisation measured across production clusters on AWS, Azure and GCP before optimisation. A vendor whose product sells the fix, measuring customers who came to it with a problem — the direction is credible, the level is not a population estimate.
- 41Population balance modelling captures host cell protein dynamics in CHO cell cultures
PLOS ONE · March 2022 · Paper
Cited for its summary of the field's yield history: antibody titre rising from about 50 mg/L in 1986 to 3–10 g/L in the 2010s through cell-line engineering and media and feed development.
- 42ICH guideline Q13 on continuous manufacturing of drug substances and drug products
European Medicines Agency / ICH · 2023 · Documentation
The regulatory text that made continuous manufacturing a normal filing rather than an exception. Cited for what regulators will accept, not for any yield claim.
- 43In Model Y, Tesla replaces 70 underbody parts with one casting
Charged EVs · February 2021 · Press
Trade press reporting Tesla's own video and Elon Musk's statements, including the claim that the change removed around 300 robots from the body line. Tesla has published no engineering paper on this; treat the figures as company statements relayed by reporters, not as verified measurement.
- 44NVIDIA Unveils Rubin CPX: A New Class of GPU Designed for Massive-Context Inference
NVIDIA · September 2025 · Vendor
A separate GPU class for the context-processing half of inference — the clearest sign that prefill and decode are now treated as different machines. The '$5 billion in token revenue for every $100 million invested' line is a vendor illustration with no published model behind it.
- 45Moderna — AWS customer case study
Amazon Web Services · 2020 · Vendor
Records 42 days from viral sequence to the first clinical batch of mRNA-1273 delivered to NIH. A cloud vendor's case study of its own customer; the 42-day figure is corroborated by Moderna's own public statements and by the trial record.
- 46Electricity 2026 — Demand
International Energy Agency · 2026 · Documentation
Data centres are expected to account for about half of US electricity demand growth to 2030, against total US growth of more than 420 TWh over the five years.
- 47The Goal: A Process of Ongoing Improvement
Eliyahu M. Goldratt, Jeff Cox · North River Press · 1984 · Book
The business novel that introduced the Theory of Constraints and the five focusing steps. A novel is an odd citation for a method, and that is part of the history: the argument for deliberately idling most of a plant needed a story to travel.
- 48Lilly to build $6 billion facility to manufacture active pharmaceutical ingredients in Alabama
Eli Lilly and Company · December 9, 2025 · Vendor
The third of four new US sites in the same programme, and Lilly's ninth announced since 2020. Construction from 2026, about 450 permanent roles on site. Cited for the timescale, which is the point being made, not as an endorsement of the investment.
- 49How GPT-5.6 fuses frontier intelligence with frontier efficiency
OpenAI · July 29, 2026 · Engineering
The engineering account behind the price cuts. GPT-5.6 Sol, driven through Codex, autonomously rewrote production kernels in Triton and Gluon — “these efforts, combined with broader kernel advancements from GPT-5.6 Sol, reduced end-to-end serving costs by 20%” — and redesigned the speculative-decoding draft model across hundreds of experiments, which “increased token-generation efficiency by more than 15%”. Also documents capping tool output at 10,000 tokens by default and preserving exact prefixes for prompt caching. Self-reported, unreplicated, and OpenAI notes it built a floating-point sanitiser to check the model's own kernels.
- 50AMD Instinct MI400 Series GPUs
AMD · accessed September 8, 2026 · Vendor
AMD leads with memory: the Helios rack-scale design is quoted at 72 MI455X GPUs, up to 2.9 exaFLOPS of OCP MXFP4 compute, and 31 TB of HBM4 at up to 1.7 PB/s peak theoretical bandwidth. Vendor specification for a platform in early deployment; no independent measurement exists, and “peak theoretical” is doing work in that sentence.
- 51The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf · Hugging Face · June 2024, revised October 2024 · Paper
15T tokens from 96 Common Crawl snapshots, with every filtering and deduplication choice ablated rather than asserted. The educational filter keeps 1.3T of that 15T — it discards roughly 92% — and the resulting models beat the full set on knowledge benchmarks. The published ablation models are what make this evidence rather than a claim.
- 52Deduplicating Training Data Makes Language Models Better
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini · ACL 2022 · July 2021, published ACL 2022 · Paper
Found a single 61-word English sentence repeated over 60,000 times in C4. Deduplicated models emit memorised text about ten times less often and need fewer training steps for the same or better accuracy. The efficiency result and the privacy result turn out to be the same result.
- 53Will we run out of data? Limits of LLM scaling based on human-generated data
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Marius Hobbhahn · Epoch AI · June 2024 · Paper
Effective stock of quality- and repetition-adjusted public human text estimated at about 300 trillion tokens, 90% CI 100T to 1000T, fully utilised somewhere between 2026 and 2032 at 80% confidence. A modelled interval, not a date. The authors list overtraining, synthetic data, non-public data and other modalities as the ways round it, and note that synthetic data has been shown to work reliably only in narrow domains like maths and code.
- 54Leverage the Latest Open Models for Synthetic Data Generation with NVIDIA Nemotron-4 340B
NVIDIA · June 2024 · Vendor
98% of the alignment data for Nemotron-4-340B-Instruct was synthetically generated, using about 1% of the human annotation that trained a comparable model and matching or exceeding it on standard benchmarks. NVIDIA is explicit that generation alone is not the method: the pipeline includes a reward model and verification, because accuracy depends on data quality rather than volume. A vendor reporting on its own models against its own benchmark selection.
- 55AI models collapse when trained on recursively generated data
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, Yarin Gal · Nature 631 · July 2024 · Paper
Indiscriminate training on model-generated content causes irreversible defects, and the tails of the original distribution disappear first. The mitigation is in the paper too: mixing real data back in slows collapse. Cited for the failure mode, and the failure mode is specifically indiscriminate recursive use, not synthetic data as such.
- 56Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky · Google DeepMind and University of Michigan, EMNLP 2024 industry track · July 2024, revised October 2024 · Paper
Long context wins on average when you can afford it; retrieval is far cheaper. The load-bearing finding is that for 63% of queries the two produce exactly identical predictions, which is what makes routing between them work: Self-Route cuts cost by 65% on Gemini-1.5-Pro and 39% on GPT-4o at comparable quality. The benchmarks are question-answering shaped, so the ratio will not transfer unchanged to other workloads.
- 57Improving language models by retrieving from trillions of tokens
Sebastian Borgeaud et al. · DeepMind, ICML 2022 · December 2021 · Paper
RETRO matched GPT-3 and Jurassic-1 on the Pile with 25x fewer parameters by conditioning on chunks retrieved from a two-trillion-token database. The clearest published demonstration that knowledge can live outside the weights. It is a 2021 result against 2021 baselines, and it did not go on to become the dominant architecture, which is part of the story rather than an omission from it.
- 58Are drug targets with genetic support twice as likely to be approved? Revised estimates of the impact of genetic support for drug mechanisms on the probability of drug approval
Emily A. King, J. Wade Davis, Jacob F. Degner · PLOS Genetics 15(12) · December 2019 · Paper
A revision downward of the famous “twice as likely” claim, and the more useful paper for it. Mendelian and coding-variant evidence holds up and can exceed two-fold; GWAS evidence is consistently weaker and, in their independent replication sets, not clearly different from zero. What kind of genetic evidence you have matters more than whether you have some.
- 59Human genetic evidence is associated with drug approval across therapeutic areas: an observational analysis of 26,278 target-disease pairs with temporal validation and feature ablation
Victoria Paterson · arXiv preprint · June 2026 · Paper
A single-author preprint, not peer reviewed — read it as replication rather than as a result. Odds ratio 3.25 at pair level, falling to 2.79 once genes shared across pairs are accounted for; post-2015 approvals replicate at 3.51. The author’s own conclusion is the important one: the association is strong and the predictive value is poor, with bad calibration. A real effect that does not by itself pick a portfolio.
- 60Regeneron Genetics Center
Regeneron Pharmaceuticals · accessed September 8, 2026 · Vendor
More than three million sequenced exomes linked to de-identified health records, across 150+ collaborations. Cited as evidence of the scale of the industrial bet on target-selection evidence, not as evidence that the bet has paid off.
- 61Data and AI are Helping to Get Medicines to Patients Faster — 2022 Annual Review
Pfizer · 2022 · Vendor
Pfizer reports cutting the cycle time of one critical supply-chain step by 67%, which produced 20,000 extra doses per batch; checking and analysing trial data 50% faster; and AI or ML in more than half of its clinical trials. First-party and specific, with no methodology published, and one named step rather than a plant-wide result.
- 62Genentech and NVIDIA Enter Into Strategic AI Research Collaboration to Accelerate Drug Discovery and Development
Genentech (Roche Group) · November 21, 2023 · Vendor
Cited for one sentence, in Genentech’s own words: experimental data feeds computational models, the models make testable predictions, scientists assess them in the lab, and the results retrain the models. That is the loop in section 15, running in a wet lab, named by the company running it. No efficiency figures are published and none are implied here.
- 63Cast Rear Under Body (1-Piece) Repair Guidelines — Model Y Collision Repair Procedures
Tesla · accessed September 8, 2026 · Documentation
Tesla’s own repair manual, and the best first-party source found for the casting. It confirms the one-piece cast rear under body as a production part, sets out which damage can be welded or cold-straightened and which forces replacement of the entire casting, and records that a Model Y may have a one-piece or a two-piece rear under body “depending on the build date and location of manufacture”. The part-count and robot-count figures remain company statements relayed by press, source 43; this is the tradeoff, documented by Tesla.
- 64Training Deep Nets with Sublinear Memory Cost
Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin · arXiv · April 2016 · Paper
Gradient checkpointing: train an n-layer network in O(sqrt(n)) memory for the cost of roughly one extra forward pass. Demonstrated on a 1,000-layer residual network, cutting memory from 48GB to 7GB for about 30% more running time. A 2016 result on convolutional and recurrent nets, not on transformers, but the tradeoff it names is the one every large training run still makes.
- 65Grok Code Fast 1
xAI · August 28, 2025 · Vendor
The only first-party xAI publication found with concrete serving detail: a model architecture built from scratch for agentic coding, prompt-cache hit rates “regularly achieving… above 90%” with launch partners, and pricing of $0.20 per million input tokens against $0.02 per million cached input — cached reads at a tenth of fresh, the same ratio Anthropic charges. xAI credits “several innovative techniques” for its serving speed and does not say what they are, so nothing is inferred about them here.
- 66Context caching — Gemini API documentation
Google · accessed September 8, 2026 · Documentation
Implicit caching is on by default for Gemini 2.5 and newer, with a 4,096-token minimum on current Flash models and 2,048 on 2.5. Google states cost savings are passed on automatically when a request hits cache but does not publish the size of the discount — unlike Anthropic and xAI, which both price cache reads explicitly. The advice to place large common content at the start of the prompt is the same prefix constraint every provider has.
- 67Gemini 3 Flash for enterprises
Google Cloud · 2026 · Vendor
A launch post. The quality figures in it — Box at 15% relative accuracy improvement, Harvey over 7% on BigLaw Bench, Geotab 10%, Warp 8% — are partner statements, not independent benchmarks, and the latency and cost claims are qualitative. Cited for the existence and positioning of a cheaper tier close to the flagship, which is model routing as a product decision, not for any specific speedup.
Research and analysis by Sina Nezhadian
Industrial engineering and Lean, then large transportation programmes, now AI product work. The through-line is the one this study is about: the expensive part of a system is usually the part nobody measures, and the constraint is usually at a seam no single team owns.