Download the Book (PDF): Introduction: The Model Has to Fit the Machine A trained neural network is, at the moment training ends, a large file of floating-point numbers and a description of how to combine them. On a workstation with a data-centre GPU, that file is a minor detail. It loads in a second, it sits comfortably in tens of gigabytes of memory, and the arithmetic that consumes it is fed by a power supply that thinks nothing of drawing several hundred watts. Move the same network to a phone, a doorbell camera, a hearing aid, a vibration sensor bolted to a factory pump, or a battery-powered tag on a shipping pallet, and every one of those comfortable assumptions fails at once. The microcontroller in the sensor may have 256 kilobytes of SRAM and two megabytes of flash. It may run at a few hundred megahertz with no operating system, no file system, and no floating-point unit worth the name. It may have to last three years on a coin cell. The phone is far richer, but it shares its memory bandwidth with the display, the camera pipeline, and every other application, and it throttles itself the moment the chassis warms. Neither device will tolerate a model that was designed with no thought for where it would run. This booklet is about closing that gap: taking neural networks that work and making them run well on devices that are small, cheap, power-limited, and frequently disconnected. The techniques have names that every practitioner eventually meets — quantization, pruning, knowledge distillation, hardware acceleration on neural processing units — and there is no shortage of papers describing each. What is in short supply is a clear account of how they fit together, why some of them pay off handsomely on one device and not at all on another, and how a working developer should decide what to do first. Why run the model on the device at all It is worth being plain about the reasons, because they shape every later decision. The first is latency. A round trip to a cloud endpoint costs tens to hundreds of milliseconds before the model even starts, and the variance is worse than the mean. A wake-word detector, a gesture recogniser, a safety interlock on a machine tool, or a camera that must decide whether to record cannot wait on a cellular link. On-device inference turns an unpredictable network delay into a predictable compute delay that the engineer controls. The second is availability. Devices go into basements, fields, mines, aircraft holds, and moving vehicles. A product whose core function disappears when the connection does is a fragile product. Offline inference makes intelligence a property of the device rather than a property of the network. The third is privacy and data minimisation. Audio that never leaves the microphone cannot be intercepted, subpoenaed, or mishandled on a server. For health monitoring, home devices, and anything that listens or watches, processing locally and transmitting only a result — "a person is present", "the bearing is degrading" — is often the only design regulators and customers will accept. The fourth is energy and cost at scale. Radios are expensive in energy. On a typical low-power sensor node, transmitting raw data costs far more than computing on it locally, so a model that reduces a stream of samples to an occasional event can extend battery life dramatically. At fleet scale, the same logic applies to money: millions of devices each sending a few seconds of audio an hour to a server is a substantial inference bill that on-device processing simply removes. None of these benefits arrive automatically. They arrive only if the model actually fits, runs fast enough, and keeps enough of its accuracy after being made to fit. That is the engineering problem this booklet addresses. The controlling idea The single idea that organises everything that follows is this: on edge devices, the scarce resource is memory — how much of it a model occupies and how many bytes it moves — far more than arithmetic, and every optimization technique should be judged by what it does to memory on the specific hardware that will run it, measured on that hardware. That claim sounds narrow, but it reorders the whole subject. It explains why converting a network from 32-bit floating point to 8-bit integers is nearly always the first and most valuable step: it cuts storage and memory traffic by a factor of four before it does anything to arithmetic. It explains why removing ninety per cent of a network's weights can produce a model that is no faster at all, because scattered zeros in a dense layout still have to be fetched or skipped with bookkeeping that costs more than it saves. It explains why a small student network distilled from a large teacher is often a better bet than an aggressively compressed version of the teacher itself. And it explains why a neural processing unit with impressive headline throughput can deliver disappointing results on a model whose activations do not fit in its local SRAM, or whose operators it cannot execute, forcing the work back onto the CPU. The second half of the idea — measured on the target — matters as much as the first. Edge optimization is full of proxies that mislead: parameter counts that ignore activation memory, operation counts that ignore memory traffic, desktop benchmarks that ignore the target's cache sizes and instruction set, and accuracy figures computed in floating point that silently differ from what the integer kernels on the device actually compute. The developer who measures on the device, early and repeatedly, will out-ship the developer who optimizes proxies. What this booklet covers, and what it leaves out The chapters run in the order a practitioner meets the problems. The first sets out the budget: how memory, energy, and latency interact on real edge hardware, and why moving data dominates computing on it. The second argues that the most powerful optimization happens before any compression technique is applied, in the choice and sizing of the architecture itself. The next four chapters take the main compression techniques in turn. Two are devoted to quantization, because it is the technique with the largest and most reliable payoff and the one where subtle mistakes cost the most accuracy: the first explains the integer arithmetic precisely, the second deals with the cases where simple post-training quantization breaks and what to do about them. A chapter on pruning separates the kinds of sparsity that hardware can exploit from the kinds that only look good in a paper. A chapter on knowledge distillation explains how to train small networks that are better than their size suggests. The final three chapters move from the model to the machine. One examines the specialised accelerators — micro-NPUs in microcontrollers and the larger neural engines in phones — and what they demand of a model. One covers the toolchains that turn a trained network into something a device can execute, and how to profile the result honestly. The last works through a complete pipeline, from requirements to a deployed and monitored model, showing the order in which decisions should be made and where they typically go wrong. Some things are deliberately left out. Training on the device, federated learning, and on-device personalisation are real and growing fields, but they are a different problem from making inference fit. Analogue and in-memory computing are promising but not yet where most developers will ship. Large language models appear where they illuminate a technique — weight-only quantization to four bits was developed largely for them, and phones now run models with a few billion parameters — but this is not a guide to deploying chat assistants. The focus is on the broad middle of edge AI: vision, audio, sensor, and small language models running on phones, embedded Linux boards, and microcontrollers. How to read it The book assumes that you can train a neural network in a mainstream framework and that you know what a convolution, a matrix multiplication, and a learning rate are. It does not assume embedded-systems experience; where hardware detail matters, it is explained. Code appears only where a short fragment makes a mechanism concrete. Where the text cites specific numbers, they come from named sources listed in the notes, and where a figure varies too much across devices to be stated honestly, the text says so rather than inventing one. Read straight through if the subject is new. If you already ship models to devices and want a specific answer, the chapters stand reasonably well alone, but the first chapter's account of the memory budget is the lens through which the rest is written, and it repays a few minutes even for experienced readers. The field moves quickly. Toolchains have been renamed within the last two years — TensorFlow Lite is now LiteRT, and PyTorch's ExecuTorch reached its 1.0 release in October 2025 — and new accelerators appear every few months. The principles in this booklet change much more slowly. A model that respects the memory budget, uses arithmetic the hardware supports natively, and is validated on the target will run well on next year's chips as well as this year's. CHAPTER 1 The Edge Budget: Memory, Energy, and Time Every edge deployment begins, whether the developer acknowledges it or not, with a budget. The device has a fixed amount of storage, a fixed amount of working memory, a fixed rate at which it can move data, a ceiling on the power it can draw and the heat it can shed, and a deadline by which an answer is needed. The model must fit inside all of these at once. Exceeding any one of them is fatal: a model that is accurate and fast but needs twice the available RAM simply does not run. The common mistake is to think of this budget mainly in terms of arithmetic — how many multiply-accumulate operations the network needs per inference, and how many the processor can execute per second. Arithmetic matters, but on edge hardware it is rarely the binding constraint. The binding constraints are almost always about memory: how many bytes the model occupies, how many bytes are alive at the same moment during inference, and how many bytes must be moved between levels of the memory hierarchy to get the answer. This chapter sets out that budget concretely, because every later technique is best understood as a way of spending it more wisely. Three classes of device "Edge" covers an enormous range of hardware, and the budget looks very different across it. Three classes are worth distinguishing. Phones and tablets carry system-on-chip processors with several CPU cores, a capable GPU, and, on nearly every recent flagship and many mid-range parts, a dedicated neural processing unit. They have gigabytes of DRAM, tens of gigabytes per second of memory bandwidth, and abundant storage. Their constraints are subtler than size. Memory bandwidth is shared with everything else on the device. Sustained workloads cause thermal throttling within minutes, so a model that benchmarks well for ten seconds may run markedly slower in a twenty-minute video call. Battery drain is visible to users and punished in app-store reviews. And the hardware is fragmented: an Android application may meet hundreds of different chipsets, each with its own accelerator and driver quirks. Embedded application processors — the class that runs embedded Linux in smart cameras, industrial gateways, robots, and appliances — sit in the middle. They typically have one to four Arm Cortex-A cores, hundreds of megabytes to a few gigabytes of DRAM, and increasingly a modest NPU rated at somewhere between half a TOPS and a few TOPS (trillions of operations per second, almost always quoted for 8-bit integer arithmetic). They run an operating system with virtual memory and a file system, which makes deployment familiar, but they are cost-sensitive parts where every dollar of bill-of-materials matters across a production run. Microcontrollers are the extreme. A typical modern part built around an Arm Cortex-M core has between 128 kilobytes and a few megabytes of on-chip SRAM, between 512 kilobytes and a few megabytes of on-chip flash, a clock somewhere between 50 and 800 megahertz, and no operating system beyond, perhaps, a small real-time kernel. Power consumption in active mode is measured in milliwatts, and in sleep in microwatts. These are the devices that end up in sensors, wearables, and battery-powered equipment by the billions, and they are where the budget is tightest and the techniques in this booklet matter most. The machine-learning community that targets them calls itself TinyML, after the book of that name by Pete Warden and Daniel Situnayake.1 The same principles apply across all three classes, but the numbers shift by three or four orders of magnitude, and so does the relative importance of each constraint. On a phone, the question is usually whether a model runs fast enough without draining the battery or overheating. On a microcontroller, the first question is whether it fits at all. Storage and working memory are different budgets A model consumes two kinds of memory, and confusing them is the most common source of early failure. The first is storage for the weights: the learned parameters, plus the description of the graph. On a phone or embedded Linux board, weights live in a file and are loaded or memory-mapped into DRAM. On a microcontroller, they usually live in flash and are read directly from it during inference, without ever being copied into RAM — a technique called execute-in-place for code and, by analogy, read-in-place for weights. This is why a microcontroller with 256 kilobytes of SRAM can run a model with 400 kilobytes of weights: the weights never need to be in SRAM at all, as long as flash read speed is adequate. The second is working memory for activations: the intermediate tensors produced by each layer and consumed by the next. These must be writable, so on a microcontroller they live in SRAM. Activations are transient — a tensor can be discarded once every layer that reads it has run — so the memory needed is not the sum of all activation tensors but the maximum, over the course of inference, of the tensors that are alive at the same moment. That figure is called peak activation memory, and it is frequently the binding constraint on small devices. Peak activation memory behaves very differently from parameter count. Convolutional networks for vision typically have their largest activations in the early layers, where spatial resolution is high, and their largest weights in the late layers, where channel counts are high. A network can therefore have very few parameters and still need a great deal of RAM. MobileNetV2, one of the most widely used mobile architectures, has roughly 3.4 million parameters and around 300 million multiply-accumulates at its standard 224-by-224 input; its early layers produce tensors of 112 by 112 spatial positions with 32 or more channels, so even in 8-bit form a single such tensor occupies around 400 kilobytes, and a layer needs its input and output alive simultaneously. That network, as designed, will not run in the SRAM of most microcontrollers regardless of how its weights are compressed. The researchers behind MCUNet, discussed in the next chapter, made exactly this observation: the activation footprint, not the weight footprint, was the barrier to running ImageNet-scale vision on microcontrollers.2 Runtimes for microcontrollers make this budget explicit. LiteRT for Microcontrollers — the runtime formerly called TensorFlow Lite Micro — requires the developer to provide a single statically allocated block of memory, the tensor arena, from which all activations and scratch buffers are carved at initialisation time.3 No dynamic allocation happens during inference. The runtime plans the arena by examining the lifetime of every tensor and packing tensors whose lifetimes do not overlap into the same addresses. If the plan does not fit, initialisation fails, and the developer learns precisely how many bytes are missing. That explicitness is a virtue: it forces the activation budget into view at the start of a project rather than at the end. On phones and Linux boards the same two budgets exist, but they are softer. Exceeding available memory does not fail cleanly; it causes paging, cache thrashing, or the operating system killing the application. The discipline of knowing both numbers — weight bytes and peak activation bytes — is just as valuable there. Moving data costs more than computing on it The deeper reason memory dominates is energy. In digital hardware, the energy cost of an arithmetic operation is small compared with the energy cost of fetching its operands from anywhere other than a register, and the cost rises steeply the further away the data lives. The most widely cited figures come from Mark Horowitz's analysis of a 45-nanometre process, which Song Han and colleagues reproduced when making the case for network compression.4 The absolute values belong to an older process node and have all fallen since, but the ratios between them — which are what matter — have remained broadly similar, because they are governed by the physics of driving wires and memory arrays rather than by transistor switching. Table 1 sets them out. Table 1. Approximate energy per operation in a 45 nm process. Operation Energy (pJ) Relative to int add 32-bit integer add 0.1 1 32-bit float add 0.9 9 32-bit integer multiply 3.1 31 32-bit float multiply 3.7 37 32-bit read, small SRAM 5 50 32-bit read, off-chip DRAM 640 6,400 Source: M. Horowitz, 45 nm energy table, as reproduced in Han et al., "Learning both Weights and Connections for Efficient Neural Networks" (NeurIPS 2015). Two lessons follow. First, a read from off-chip DRAM costs over a hundred times more than a read from a small on-chip SRAM, and thousands of times more than an addition. A network whose weights stream from DRAM on every inference spends most of its energy on that streaming, not on the arithmetic. Second, narrower data is cheaper twice over: an 8-bit integer operation costs a small fraction of a 32-bit floating-point one — Horowitz's ISSCC 2014 keynote put an 8-bit integer addition at around 0.03 picojoules and an 8-bit multiplication at around 0.2 — and, more importantly, four times as many 8-bit values fit through the same memory port, into the same cache line, and into the same SRAM. This is the single most important fact in edge AI, and it is why the techniques in this booklet are ranked as they are. Quantization reduces bytes moved and stored. Architectural choices that keep activations small enough for on-chip SRAM avoid the most expensive tier of the hierarchy entirely. Pruning helps only when the hardware can avoid fetching the pruned weights. Accelerators win mainly because they place large local SRAM next to the arithmetic units and schedule data movement carefully, not merely because they have more multipliers. Arithmetic intensity and the memory wall A useful way to make this precise is arithmetic intensity: the number of arithmetic operations performed per byte moved from memory. Every processor has a ratio between its peak compute rate and its memory bandwidth. If a workload's arithmetic intensity is below that ratio, the processor spends its time waiting for data and the workload is memory-bound; above it, the arithmetic units are the bottleneck and the workload is compute-bound. This is the reasoning behind the roofline model, a standard performance-analysis tool in high-performance computing that applies just as well to a phone NPU. Different layer types sit in very different places. A standard 3-by-3 convolution over a large feature map reuses each weight at every spatial position and each input value across many output channels; its arithmetic intensity is high and it tends to be compute-bound. A fully connected layer evaluated on a single input — the normal case at the edge, where batch size is one — uses each weight exactly once. Its arithmetic intensity is roughly one multiply-accumulate per weight fetched, which places it firmly in memory-bound territory on almost any hardware. Depthwise convolutions, which the next chapter discusses, have low arithmetic intensity as well, because each filter touches only one channel. The consequence for language models on phones is stark and easy to compute. Generating each token of output from a transformer requires reading essentially all of the model's weights once. If a three-billion-parameter model is stored at four bits per weight, that is about 1.5 gigabytes per token. A phone with 60 gigabytes per second of usable memory bandwidth can therefore generate at most around forty tokens per second from that model, however many TOPS its NPU advertises — and in practice somewhat fewer, because bandwidth is shared and never fully utilised. Halve the bytes per weight and the ceiling doubles. No amount of additional arithmetic capability changes this bound; only moving fewer bytes does. The same reasoning applies, at smaller scale, to a microcontroller reading weights from flash. Flash is slower to read than SRAM, and many microcontrollers place a small cache or prefetch buffer in front of it. A model whose weights are read once per inference in a predictable order streams well; one whose access pattern jumps around defeats the cache and runs far slower than its operation count predicts. Latency is more than inference time The third element of the budget is time. Developers usually measure the time from handing the runtime an input tensor to receiving an output tensor, and that number matters. But the end-to-end latency a user or a control loop experiences includes several other components that are easy to overlook. Pre-processing converts raw sensor data into the tensor the model expects. For audio this typically means computing a spectrogram or mel-frequency features from a window of samples, which on a microcontroller can cost as much as a small neural network. For vision it means resizing, cropping, colour conversion, and normalisation. On a phone, the camera frame may need to be copied from a hardware buffer to a CPU-accessible one, and copies of multi-megabyte images are not free. Accelerator dispatch adds overhead whenever work moves between processors. Handing a graph or a sub-graph to an NPU or GPU involves driver calls, synchronisation, and sometimes copying tensors between memory regions. For a large model this overhead is amortised; for a small one it can dominate. A model that runs in two milliseconds on the CPU may take four on an accelerator if dispatch costs three. Post-processing turns raw outputs into decisions: non-maximum suppression for object detectors, smoothing and thresholding for keyword detectors, decoding for sequence models. These steps often run on the CPU and are rarely optimised with the same care as the network. Cold start is the time to load the model, initialise the runtime, and, on some accelerators, compile the graph for the specific hardware. On phones, first-run accelerator compilation can take seconds. Applications that load a model on demand will feel sluggish unless this cost is hidden or cached. Finally, latency must be considered under realistic conditions. A phone benchmark run immediately after boot, with the device cool and nothing else running, measures the best case. The relevant figure is the sustained latency after several minutes of continuous operation, with the application's own user interface and the camera pipeline competing for the same resources. Energy per inference and duty cycling For battery-powered devices, the meaningful metric is not power but energy per inference, measured in millijoules or microjoules, combined with how often inference runs. A model that draws 50 milliwatts for 20 milliseconds uses one millijoule per inference. Run once per second, that is an average of one milliwatt — manageable for a device with a reasonable battery. Run continuously, it is 50 milliwatts, which drains a coin cell in hours. This is why so many edge systems are structured as cascades. A tiny, always-on model — sometimes running on a dedicated low-power core or even on dedicated hardware within the sensor — watches for a cheap signal: sound above a threshold, motion in the frame, a vibration pattern outside the normal envelope. Only when it fires does the device wake a larger model on a more capable processor. Smart speakers use exactly this structure, with a minimal wake-word detector gating a far larger speech pipeline. The design principle is that the expensive model should run as rarely as possible, and the cheap model should be cheap enough to run always. Energy per inference also depends heavily on how quickly the processor can return to sleep. A faster inference, even at higher instantaneous power, often uses less total energy, because the static power of the rest of the system — memory, clocks, voltage regulators — is paid for every millisecond the device is awake. This "race to sleep" logic is one of the strongest arguments for hardware acceleration on microcontrollers: a micro-NPU that finishes an inference ten times sooner lets the whole chip return to its microwatt sleep state ten times sooner. Writing the budget down The practical outcome of this chapter is a habit: before choosing an architecture or applying any compression technique, write the budget down as numbers. For a given target, that means at least the following. • Maximum weight storage, in bytes, after accounting for application code, the runtime, and any over-the-air update scheme that needs a second copy of the firmware. • Maximum working memory available for activations and scratch buffers, after accounting for the stack, the heap, sensor buffers, and communication buffers. • Latency deadline, end to end, including pre- and post-processing, at sustained rather than peak performance. • Energy per inference and inference rate, if the device is battery-powered, and the resulting average power. • The accuracy floor below which the product does not work, measured on data that represents the deployed environment rather than the benchmark. Each number should have a margin. Firmware grows, sensor buffers get larger, and a model that fits with three kilobytes to spare in the first prototype will not fit in the shipping product. With these numbers in hand, every later decision becomes a question of trade-offs within a known envelope rather than a search for "the best model". A network that exceeds the activation budget must change its architecture or input resolution; no amount of weight compression will help. A network that fits but misses its latency deadline needs faster arithmetic — integer kernels, an accelerator, or fewer operations. A network that fits and is fast enough but drains the battery needs fewer bytes moved per inference or fewer inferences. The techniques in the rest of this booklet are tools for each of those situations, and knowing which situation you are in is most of the battle. CHAPTER 2 Architecture First: Designing Within the Budget Most discussions of edge AI start with compression: take a large model and shrink it. That framing is natural, because compression techniques are well defined and make for satisfying before-and-after comparisons. But it gets the order of operations wrong. The most powerful decision a developer makes is the choice of architecture and its size, and that decision is made before any compression technique is applied. A network designed for the budget from the outset, then quantized, will almost always beat a network designed for a server and then compressed until it fits. The reason is simple. Compression techniques remove redundancy. They can take a model that is twice as large as it needs to be and bring it close to its necessary size, and with care they can go somewhat further. What they cannot do is change the shape of the computation: the spatial resolution at which early layers run, the way channels interact, the operators used, the depth of the network. Those choices determine the peak activation memory, the arithmetic intensity, and the operator compatibility that the previous chapter identified as the real constraints. Get them wrong, and compression is fighting the architecture. The factorised convolution The single architectural idea that made mobile vision practical is the depthwise separable convolution, popularised by Google's MobileNets in 2017.1 It is worth understanding precisely, because its strengths and weaknesses explain much of what follows. A standard convolution with a k-by-k kernel, C_in input channels, and C_out output channels does two things at once. It mixes information spatially, combining each pixel with its neighbours, and it mixes information across channels, combining every input channel into every output channel. For a feature map of H by W positions, it costs k² × C_in × C_out × H × W multiply-accumulates. A depthwise separable convolution splits those two jobs. First, a depthwise convolution applies one k-by-k filter to each input channel separately, mixing spatially but not across channels; it costs k² × C_in × H × W. Then a pointwise convolution — a 1-by-1 convolution — mixes across channels at each position; it costs C_in × C_out × H × W. The ratio of the factorised cost to the standard cost works out to 1/C_out + 1/k². With 3-by-3 kernels and a reasonably large number of output channels, that is a little over one-ninth: roughly an eight- to nine-fold reduction in arithmetic and a similar reduction in parameters, for a modest loss in accuracy. MobileNetV1 built a whole network from these blocks and added two knobs that remain the simplest and most effective way to fit a network to a budget. The width multiplier scales the number of channels in every layer by a factor, typically between 0.25 and 1.0; because both parameters and computation scale roughly with the square of channel count in pointwise layers, halving the width cuts cost by close to four. The resolution multiplier scales the input image size; because computation and activation memory scale with the number of spatial positions, halving resolution cuts both by four. A designer with a fixed budget can trade between them, and the MobileNet paper's experiments showed that accuracy falls off smoothly rather than abruptly as either knob is turned down. MobileNetV2, published the following year, introduced the inverted residual block that dominates efficient architectures to this day.2 Each block takes a thin input, expands it with a 1-by-1 convolution to a wider intermediate representation (typically six times wider), applies a depthwise convolution in that wide space, then projects back down to a thin output with another 1-by-1 convolution — with no activation function after the projection, a choice the authors called a linear bottleneck, because applying a non-linearity in a low-dimensional space destroys information. A residual connection joins the thin input to the thin output. The design keeps the tensors that pass between blocks small, which is good for memory, while doing the expressive work in a wider space inside the block. MobileNetV3 refined this with neural architecture search and with small, hardware-motivated choices.3 Its designers replaced the swish activation with "hard swish", a piecewise-linear approximation that is cheap to compute and, importantly, behaves well under integer quantization, where smooth functions involving exponentials are awkward. That is a recurring lesson: an operation that is cheap in floating point on a GPU may be expensive or numerically fragile in 8-bit integers on a microcontroller, and architectures intended for the edge should be chosen with the integer version in mind. When efficiency on paper is not efficiency on silicon The depthwise separable convolution reduces operation counts dramatically, and on CPUs, where performance tracks operation counts reasonably well, it delivers. But it has a weakness that matters on accelerators. Depthwise convolutions have very low arithmetic intensity: each filter touches only one channel, so there is little reuse of data, and the operation tends to be limited by memory movement. Accelerators built around large arrays of multiply-accumulate units — systolic arrays and their relatives — achieve their headline throughput only when the work can be laid out to keep the whole array busy. A depthwise layer often cannot, and utilisation drops sharply. Google's own accelerator designers confronted this directly. EfficientNet-EdgeTPU, a family tuned for the Coral Edge TPU, replaced the expand-then-depthwise pattern in its early layers with a single ordinary 3-by-3 convolution — a "fused" block — that performs more arithmetic but maps much better to the accelerator's matrix units, and runs faster in practice. EfficientNet-Lite, a variant intended for mobile CPUs and accelerators, removed the squeeze-and-excitation modules and replaced swish with ReLU6, again because those components were awkward to quantize or poorly supported. MobileNetV4, published in 2024, made the cross-hardware problem its explicit goal: its "universal inverted bottleneck" block can take several forms, including the fused and plain variants, and the authors searched for configurations that perform well simultaneously on mobile CPUs, DSPs, GPUs, and dedicated accelerators such as the Apple Neural Engine and the Edge TPU.4 The general point is that operation counts are a proxy, and on accelerators a misleading one. The architecture that is best for a phone CPU may not be best for the same phone's NPU, and the one that is best for a server GPU is almost never best for either. The only reliable metric is latency measured on the target hardware, which is why the most effective architecture search methods for the edge build measured latency directly into their objective. MnasNet, a precursor to MobileNetV3, did exactly this, running candidate models on physical phones during the search and rewarding accuracy and real latency together rather than accuracy and operation count. Designing for peak memory: the MCUNet lesson On microcontrollers, the binding constraint is usually not latency but the activation budget, and here the most instructive work is MCUNet, from Ji Lin, Song Han, and colleagues, presented at NeurIPS in 2020.5 Their goal was to run ImageNet-scale image classification on off-the-shelf microcontrollers with a few hundred kilobytes of SRAM and one or two megabytes of flash — something that had widely been considered out of reach. MCUNet had two parts designed together. TinyNAS performed architecture search, but with an important first stage: before searching for the best architecture, it searched for the best search space. It considered many combinations of input resolution and width multiplier, discarded those that could not satisfy the memory constraints of the target, and among the remainder preferred the space whose randomly sampled models that fit the constraint had higher computational capacity — on the reasoning that a space in which feasible models can afford more computation is likely to contain more accurate ones. Only then did it search for a specific architecture. TinyEngine was an inference library co-designed with the search, which generated specialised code for each model at compile time, avoided the overhead of a general interpreter, and used memory-scheduling techniques to reduce peak activation memory. The result was a model reaching about 70.7 per cent top-1 accuracy on ImageNet on a commercial microcontroller — a figure comparable to the original MobileNetV1 on a phone, achieved within a few hundred kilobytes of RAM. The specific number matters less than the method. The search was constrained by peak memory from the start, the inference engine was designed jointly with the model, and both were evaluated on the real device. A follow-up, MCUNetV2, attacked the activation peak more directly with patch-based inference. Because the largest activations in a vision network occur in the first few layers, MCUNetV2 runs those layers on small spatial patches of the input one at a time, holding only a patch's worth of activations in memory, and assembles the results only once the feature map has shrunk enough to fit. Overlapping receptive fields between patches cause some computation to be repeated, which the authors mitigated by redistributing the network's receptive field, but the trade of a little extra arithmetic for a large reduction in peak memory is exactly right on a device where memory, not arithmetic, is scarce. The two knobs that matter most: resolution and width Architecture search is powerful but requires infrastructure that many teams do not have. For the common case — choosing a well-known efficient family and fitting it to a device — two knobs do most of the work, and they should be turned deliberately. Input resolution affects activation memory and computation quadratically, and it is frequently set far higher than the task requires. A person-detection model for a doorbell that must decide whether anyone is in the frame does not need a 224-by-224 input; the Visual Wake Words benchmark, built specifically for microcontroller-scale person detection, is commonly run at 96 by 96, an input with less than a fifth of the pixels.6 Detecting a face at a distance needs more resolution; detecting a raised hand at arm's length needs less. The right resolution is the smallest at which the task's accuracy on representative data stays above the floor, and it is found empirically by training at several resolutions and comparing. This experiment is cheap and almost always worth running before anything else. Width affects weight storage roughly quadratically in pointwise layers and activation memory linearly. Reducing width is the most natural way to bring a model's weights inside a flash budget. It also interacts with hardware in a way that is easy to miss: many accelerators and optimised CPU kernels process channels in groups — multiples of 8, 16, or 32 are common — so a layer with 36 channels may run no faster than one with 48, and one with 33 may be dramatically slower than one with 32. Choosing channel counts that align with the target's natural vector width is a free improvement. Depth is a less effective knob for edge budgets. Removing layers reduces weight storage and computation but does little to the activation peak, which is set by the widest high-resolution layer, and it tends to cost more accuracy per unit of savings than reducing width or resolution. Choosing operators the device can run An architecture is not only a shape; it is a set of operators, and every operator must exist, efficiently, on the target. This is a hard constraint that architecture papers rarely mention. Microcontroller runtimes support a curated set of operators, and an application includes only those it registers, to save flash. Common operators — convolution, depthwise convolution, fully connected, pooling, ReLU and ReLU6, add, reshape, softmax — have optimised integer kernels, such as Arm's CMSIS-NN library for Cortex-M processors.7 Less common operators may have only a slow reference implementation, or none. A network that uses a novel activation function, an unusual normalisation, or a dynamic shape may convert cleanly on a desktop and then fail, or run ten times slower than expected, on the device. Accelerators are stricter still. A micro-NPU supports a fixed list of operators with specific constraints on kernel size, stride, channel count, and data type. Any operator outside that list is executed on the host CPU, which means splitting the graph, copying tensors, and losing much of the accelerator's benefit. Chapter 7 returns to this in detail, but the architectural lesson belongs here: check the operator support list of every runtime and accelerator you may target before committing to an architecture, and prefer operators that appear on all of them. Some specific choices recur often enough to state plainly. Batch normalisation is free at inference, because it can be folded into the weights and bias of the preceding convolution; layer normalisation and group normalisation cannot be folded and require computing statistics at run time, which is costly and awkward in integer arithmetic. ReLU and ReLU6 are trivial in integers and, with ReLU6, conveniently bound the activation range, which helps quantization; smooth activations involving exponentials are more expensive and are best replaced by piecewise-linear approximations. Fixed input shapes allow memory planning at compile time; dynamic shapes, beloved of research code, frustrate it. Audio and sensor models are a different shape Much of the literature on efficient architectures concerns images, but a large share of real edge workloads are audio and sensor streams: keyword spotting, sound-event detection, vibration monitoring, gesture recognition from accelerometers, and anomaly detection on industrial signals. These have their own architectural conventions, and the budget arguments apply to them in a somewhat different way. For audio, the network rarely consumes raw waveform samples. The usual input is a compact time-frequency representation — a log-mel spectrogram or mel-frequency cepstral coefficients — computed over a window of one second or so, giving a small two-dimensional array of perhaps forty or fifty time steps by ten to forty frequency bands. Yundong Zhang and colleagues at Arm, in a 2017 study titled "Hello Edge: Keyword Spotting on Microcontrollers", compared several network families on this kind of input under microcontroller memory and compute limits, and found that a depthwise separable convolutional network — the DS-CNN — gave the best accuracy within the budget. The keyword-spotting reference model in the MLPerf Tiny benchmark, discussed in Chapter 8, is a network of this type with under forty thousand parameters. The lesson is that the feature extraction does a great deal of the work, and the network itself can be tiny. That shifts the budget. On such a device, computing the spectrogram may cost as much time and energy as running the network, and the feature extractor's exact parameters become part of the model's definition: change the window length or the number of mel bands and the trained network no longer applies. Treating the feature extraction as part of the architecture, designed and versioned with it, avoids a whole class of deployment bugs. Streaming inputs also allow an optimisation that images do not. A keyword spotter typically re-runs every few tens of milliseconds on a window that overlaps almost entirely with the previous one. A naive implementation recomputes every layer for the whole window each time. A streaming implementation keeps the intermediate results for the parts of the window it has already seen, in small state buffers, and computes only the new time steps — turning each convolution over time into an incremental update. For networks built from one-dimensional or causal convolutions over time, this can reduce the work per step by an order of magnitude at the cost of a few kilobytes of state. It requires the architecture to be designed for it, with convolutions that look only backwards in time, which is another reason to settle the deployment pattern before the architecture. For low-rate sensor data such as accelerometer or vibration readings, the networks are smaller still, and sometimes a neural network is not the right tool at all. Hand-designed features — spectral peaks, root-mean-square levels, statistical moments over a window — fed to a small classifier or a gradient-boosted tree can match a neural network on simple tasks at a fraction of the cost. The neural approach earns its place when the patterns are too varied to engineer features for, and the budget arguments of this booklet then apply in full. Transformers at the edge Transformers deserve a word, because they increasingly appear in edge workloads — in on-device language models, in speech recognition, and in vision models that combine convolutional and attention layers. Their characteristic operation, self-attention, has computation and memory that grow with the square of sequence length, and it relies on softmax and layer normalisation, both of which are awkward in low-precision integer arithmetic. Two adaptations have made them more practical. Hybrid architectures use convolutions in early, high-resolution stages, where attention would be prohibitively expensive, and introduce attention only on smaller feature maps. And hardware has begun to catch up: the third generation of Arm's micro-NPU, the Ethos-U85 announced in April 2024, added native support for operators such as matrix multiplication, transpose, and gather that transformer networks require, precisely because customers wanted to run them on embedded devices.8 Even so, for a microcontroller-class target with a tight memory budget, a well-designed convolutional or recurrent network is usually the safer choice unless the task genuinely needs attention. Small models trained well A final point about architecture is really a point about training. A small network trained with the same recipe as a large one is often under-trained relative to its potential. Small networks benefit disproportionately from longer training schedules, careful data augmentation, label smoothing, and — above all — from learning to imitate a larger, more accurate model, a technique Chapter 6 covers at length. The gap between a small architecture trained naively and the same architecture trained well can be several percentage points of accuracy, larger than the loss from quantizing it. That makes the order of work clear. First, choose an efficient architecture family whose operators the target supports. Second, find the smallest resolution and width that can meet the accuracy floor, measuring peak activation memory and on-device latency as you go. Third, train that network as well as you can, including distillation from a stronger teacher. Only then apply the compression techniques of the following chapters, starting with quantization — which, for a well-designed network, is usually all the compression it needs. CHAPTER 3 Quantization: The Arithmetic of Integers If a developer could apply only one optimization to a model bound for an edge device, it should be quantization: representing weights and activations with fewer bits than the 32-bit floating point used in training. Converting to 8-bit integers cuts weight storage by a factor of four, cuts memory traffic by the same factor, lets the processor use integer instructions that are cheaper and often wider in parallel, and — crucially — makes the model eligible to run on neural processing units, most of which execute only integer arithmetic. For a well-designed convolutional network, the accuracy cost is usually under a percentage point and frequently negligible. Those benefits are reliable enough that quantization is often treated as a button to press: run the converter, ship the result. Usually that works. When it does not, the failure is usually silent — the model runs, produces plausible outputs, and is quietly a few points less accurate than it should be, or catastrophically wrong on inputs outside the range seen during conversion. Understanding the arithmetic precisely is what lets a developer tell the difference, diagnose it, and fix it. This chapter sets out the arithmetic; the next deals with the cases where the simple approach breaks. Mapping real numbers to integers The scheme used by almost every edge runtime is uniform affine quantization, formalised for integer-only inference by Benoit Jacob and colleagues at Google in a paper presented at CVPR in 2018, which became the basis of TensorFlow Lite's integer path.1 It maps a real value r to an integer q through two parameters: a positive real scale S and an integer zero point Z. The mapping from integer back to real is r = S × (q − Z). Going the other way, a real value is quantized by dividing by the scale, rounding to the nearest integer, adding the zero point, and clamping to the representable range — for signed 8-bit integers, −128 to 127. The scale sets the step size: two adjacent integers represent real values S apart. The zero point sets which integer represents the real value zero. Jacob and colleagues insisted that real zero be exactly representable, for a practical reason: neural networks pad tensors with zeros, and ReLU produces exact zeros in abundance, so if zero were represented with even a small error, that error would be added systematically across every padded border and every inactive unit. To choose S and Z for a tensor, one needs the range of real values it will hold, from some minimum to some maximum. The scale is that range divided by the number of integer steps available — 255 for 8 bits — and the zero point is chosen so that the minimum maps to −128 and zero maps exactly to an integer. Every real value inside the range is then represented with an error of at most half a step. Values outside the range are clamped to its edges, which can produce much larger errors. A small worked example makes this concrete. Suppose calibration shows that an activation tensor ranges from −1.0 to 3.0. The range spans 4.0, divided across 255 steps, so the scale is about 0.0157. To place real zero exactly on an integer, compute where −1.0 would fall relative to zero — about 64 steps below it — and set the zero point so that −1.0 maps to −128; that makes the zero point −64. Real zero is now represented exactly by the integer −64. A real value of 1.0 divides by the scale to about 63.75, rounds to 64, and adds the zero point to give the integer 0; dequantizing gives 0.0157 × (0 − (−64)), or about 1.004 — an error of under half a step, as promised. A real value of 5.0, outside the calibrated range, would compute to an integer well above 127 and be clamped to 127, which dequantizes to about 3.0: an error of 2.0, more than a hundred times larger than the rounding error. Every value the calibration data did not anticipate is treated this way. This immediately exposes the central trade-off of quantization. A wide range accommodates rare large values without clamping, but makes the step size large, so ordinary values are represented coarsely. A narrow range represents ordinary values finely, but clamps the outliers. The first kind of error is called rounding error; the second, clipping error. Choosing the range is choosing the balance between them, and much of the craft of quantization is in that choice. Symmetric, asymmetric, per-tensor, per-channel Two further choices shape how a scheme behaves. Symmetric quantization fixes the zero point at zero and uses a range symmetric around it. It simplifies the arithmetic, because the zero-point terms disappear from many calculations, but wastes half the integer range on a tensor whose values are all positive. Asymmetric quantization uses the zero point to shift the range so it fits the data. The common convention, and the one used in the LiteRT integer specification, is to quantize weights symmetrically, since trained weights are roughly centred on zero, and activations asymmetrically, since activations after ReLU are non-negative and would waste half their range under a symmetric scheme.2 Granularity determines how many tensors share a scale. Per-tensor quantization uses one scale for an entire weight tensor; per-channel quantization uses a separate scale for each output channel of a convolution or each output row of a fully connected layer. The difference is large in practice. After batch normalisation is folded into a convolution's weights, different output channels can have weight ranges that differ by an order of magnitude or more. A single per-tensor scale must accommodate the widest channel, leaving the narrow channels represented by only a handful of distinct integer values. Early experience with quantizing MobileNet architectures, whose depthwise layers are particularly prone to this, found that per-tensor weight quantization could cause severe accuracy losses that per-channel quantization largely eliminated.3 Per-channel weight quantization is now the default in mainstream toolchains, and it costs almost nothing at inference time: the per-channel scales are folded into the requantization step described below. Activations, by contrast, are almost always quantized per-tensor, because per-channel activation scales would complicate the arithmetic of every subsequent layer. How an integer layer actually computes The heart of integer-only inference is that a quantized layer can compute its output using only integer multiplication, integer addition, and a single fixed-point rescale at the end. It is worth tracing the arithmetic once, because it explains several otherwise puzzling details of quantized models. Consider one output of a fully connected layer: a sum over products of weights and inputs, plus a bias. Substitute the quantized representation of each — weight as S_w × (q_w − Z_w), input as S_x × (q_x − Z_x), output as S_y × (q_y − Z_y) — and rearrange. The output integer q_y turns out to equal the output zero point, plus a multiplier M times a sum of products of zero-point-adjusted integers, plus the bias, where M is the product of the weight and input scales divided by the output scale. Everything inside the sum is integer arithmetic. Each product of two 8-bit values fits in 16 bits, and the sum of many such products is accumulated in a 32-bit integer accumulator, which has ample headroom for the hundreds or thousands of terms in a typical layer. The terms involving zero points can be precomputed or rearranged so that the inner loop is a plain integer dot product. The bias is stored as a 32-bit integer with a scale equal to the product of the weight and input scales and a zero point of zero, so that it can be added directly to the accumulator. This is why quantized models keep biases at 32 bits even when everything else is 8 bits: the bias is added at the accumulator's precision, and quantizing it coarsely would introduce a systematic offset into every output. The only non-integer quantity is the multiplier M, a real number typically between zero and one. Jacob and colleagues showed that it can be represented as a 32-bit fixed-point integer multiplier and a right shift, so that the final rescaling is an integer multiplication followed by a rounding shift. This requantization step converts the 32-bit accumulator back to an 8-bit output in the output tensor's scale. With per-channel weights, each output channel has its own multiplier and shift. Finally, a fused activation function costs nothing. ReLU becomes clamping the output integer to be no less than the integer representing zero; ReLU6 additionally clamps it to be no greater than the integer representing six. These clamps are folded into the saturation that requantization must perform anyway. Three practical consequences follow. First, the output scale of each layer is fixed at conversion time from the calibration data, and values that would exceed it at run time are silently saturated. Second, an operator for which no integer kernel exists breaks the chain, forcing a dequantize-compute-requantize sequence in floating point that is slow and, on accelerators, may move the computation to another processor. Third, because the requantization arithmetic is specified exactly, a correct integer kernel on a microcontroller and a correct integer kernel on a desktop should produce bit-identical outputs — which makes the desktop interpreter, running the converted model, the right reference against which to test the device. Formats beyond int8 Eight-bit integer is the workhorse, but not the only option. The main alternatives are set out in Table 2. Table 2. Common numeric formats for edge inference. Format Bits Size vs FP32 Typical edge use Hardware support FP32 32 1× Training; reference All CPUs; slow on MCUs without FPU FP16 16 0.5× Phone GPUs and NPUs Most phone SoCs BF16 16 0.5× Training; some newer NPUs Limited at the edge INT8 8 0.25× Default for CNNs, MCUs, NPUs Nearly universal INT4 4 0.125× LLM weights; some NPUs Growing; often unpacked to INT8 Float16 halves storage with essentially no accuracy loss for most models and runs natively on phone GPUs. It is a reasonable choice when the target is a GPU and the model is sensitive to integer quantization, but it does not unlock integer-only NPUs and it offers nothing on a microcontroller without half-precision hardware. Dynamic range quantization — a LiteRT conversion mode — stores weights as 8-bit integers but keeps activations in floating point, computing activation scales on the fly. It requires no calibration data and delivers the fourfold storage saving, but the arithmetic remains largely floating point, so it is useful mainly for CPU deployment of models, such as recurrent networks, where full integer conversion is troublesome. 16-bit activations with 8-bit weights, sometimes written as int16x8, are supported by several runtimes and by Arm's Ethos-U NPUs. They give activations 256 times as many levels while keeping weight storage at 8 bits, and are a useful middle path for audio and sensor models whose activations have high dynamic range. Four bits and below are covered in the next chapter. They matter mainly for weights, especially in language models, where memory bandwidth dominates. Post-training quantization: calibration Post-training quantization, usually abbreviated PTQ, converts a trained floating-point model without further training. Weight ranges can be read directly from the weights. Activation ranges cannot — they depend on the input — so PTQ requires a calibration step: run a representative sample of inputs through the floating-point model, record the values each activation tensor takes, and choose a range for each. The calibration set does not need to be large — a few hundred samples is a common starting point — but it must be representative of what the device will see. This is where many silent failures originate. A model for an outdoor camera calibrated only on daytime images will set activation ranges that clip badly at night. A keyword spotter calibrated on clean studio recordings will saturate on loud, noisy audio. Calibration data should be drawn from the same distribution as deployment data, including its extremes, and ideally from the validation set rather than the training set. The simplest range estimator takes the observed minimum and maximum. It never clips calibration data but is easily dominated by rare outliers, producing a large scale and coarse resolution for typical values. Better estimators deliberately clip. A percentile estimator ignores the most extreme fraction of observed values. A mean-squared-error estimator searches for the range that minimises the total quantization error on the calibration data. An entropy-based estimator, used in NVIDIA's TensorRT, chooses the range that minimises the information lost, measured as the divergence between the original and quantized distributions. For activations with long tails, clipping estimators are usually substantially better than min-max. In LiteRT, full-integer post-training quantization takes a few lines of configuration. The essential parts are a function that yields representative inputs and a declaration that the model must use integer operators throughout: converter = tf.lite.TFLiteConverter.from_keras_model(model) converter.optimizations = [tf.lite.Optimize.DEFAULT] converter.representative_dataset = representative_samples converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8] converter.inference_input_type = tf.int8 converter.inference_output_type = tf.int8 tflite_model = converter.convert() Requiring integer-only operators is important for microcontroller and NPU targets: without it, the converter may silently leave unsupported operators in floating point, producing a model that runs on a desktop but not on the device. Setting the input and output types to integers removes floating-point conversions at the model's boundaries, which matters on processors without a floating-point unit; the application then quantizes its inputs and dequantizes outputs itself, using the scales and zero points stored in the model. PyTorch's current path, export-based quantization, follows the same logic: capture the model graph with torch.export, insert observers according to a quantizer configuration supplied for the target backend, run calibration data through, and convert. The backend-specific quantizer matters, because it encodes which operators the target can fuse and which quantization schemes its kernels expect. Repairs that need no training Before resorting to retraining, several data-free or calibration-only techniques can rescue a model that quantizes badly. Two from Markus Nagel and colleagues at Qualcomm are widely implemented.4 Cross-layer equalization addresses the problem of channels with wildly different ranges. Because ReLU commutes with positive scaling, the weights of one channel in a layer can be multiplied by a factor and the corresponding input weights of the next layer divided by the same factor without changing the network's function. Choosing these factors to equalise channel ranges makes per-tensor quantization far more accurate, and helps even per-channel schemes by balancing ranges across adjacent layers. Bias correction addresses the observation that quantization error in weights is often biased — on average it shifts a layer's outputs in one direction. That expected shift can be estimated from the batch-normalisation statistics or from calibration data and subtracted from the layer's bias, removing a systematic error that would otherwise propagate through the network. A third technique, AdaRound, also from Nagel's group, questions the assumption that each weight should be rounded to its nearest integer.5 Rounding each weight independently minimises each weight's own error but not the error in the layer's output. AdaRound learns, using a small amount of unlabelled calibration data and a layer-by-layer optimisation, whether to round each weight up or down so as to minimise output error. It requires no labels and no end-to-end training, yet frequently recovers a large share of the accuracy lost to naive rounding, especially at four bits. Evaluate the model you will ship The last rule of this chapter is procedural and important. The accuracy of a quantized model must be measured on the quantized model, running in an interpreter that uses the same integer kernels as the device, over the full validation set. Not on the floating-point model. Not on a "fake-quantized" simulation inside the training framework, whose rounding and saturation may differ in small but consequential ways. Not on a few dozen examples. The comparison worth making is three-way: the floating-point model, the quantized model in the desktop interpreter, and the same quantized model on the device, run on identical inputs. The first gap measures quantization loss and tells you whether the techniques of the next chapter are needed. The second should be zero, or nearly so; if it is not, something is wrong in the deployment — a pre-processing mismatch, an input scaling error, an operator implemented differently on the device — and finding it now is far cheaper than finding it in the field. CHAPTER 4 When Quantization Breaks: Diagnosis, Training, and Low Bit-Widths Post-training quantization to 8 bits works for most convolutional networks most of the time. This chapter is about the rest of the time: the models that lose five or ten points of accuracy on conversion, the models whose outputs collapse on certain inputs, and the push below 8 bits, where post-training methods begin to fail as a matter of course and where much of the recent research effort, driven by language models, has been concentrated. The approach is deliberately diagnostic. When a quantized model underperforms, the instinct is to reach for the heaviest tool available — quantization-aware training — and retrain. Sometimes that is right. More often, the loss is concentrated in a small number of layers for an identifiable reason, and a targeted fix is faster, cheaper, and more robust than retraining the whole network. Finding where the error comes from Quantization error is introduced at every layer and accumulates through the network, but it is rarely spread evenly. A typical poorly quantizing model has one or two layers responsible for most of the loss. The first task is to find them. The most direct method is layer-wise comparison. Run the same input through the floating-point model and the quantized model, capture every intermediate tensor, dequantize the quantized ones, and compute a measure of agreement for each layer — commonly the signal-to-quantization-noise ratio, the ratio of the floating-point tensor's power to the power of the difference between the two. A healthy 8-bit model typically shows a ratio that declines gradually through the network. A sudden drop at one layer marks where the trouble is introduced. Most toolchains provide a debugger for exactly this; LiteRT's quantization debugger and the numeric-comparison utilities in PyTorch's quantization tooling both report per-layer statistics. A complementary method is sensitivity analysis: quantize the network one layer at a time, leaving all others in floating point, and measure end-to-end accuracy for each. Layers whose quantization alone causes a large drop are sensitive; the rest can safely be quantized. Running the analysis in reverse — quantize everything, then restore one layer at a time to floating point — answers the more practical question of which single layer, if kept at higher precision, recovers the most accuracy. The causes that these analyses reveal fall into a handful of recurring patterns. Outlier activations. A small number of values in an activation tensor are far larger than the rest, forcing a wide range and a coarse step size. This is common after layers without a bounding activation and is endemic in transformer models, as discussed below. The fixes are a clipping range estimator, replacing an unbounded activation with a bounded one such as ReLU6 and retraining, or keeping that tensor at 16 bits. Per-tensor weight scales on uneven channels. Visible as poor agreement in particular convolution layers, especially depthwise ones. The fix is per-channel quantization or, where the target supports only per-tensor scales, cross-layer equalization. Mismatched scales at joins. Residual additions and concatenations combine tensors that were quantized with different scales. The runtime must requantize them to a common scale before combining, losing precision in whichever input has the finer scale. If one branch has a much wider range than the other, the narrow branch is effectively crushed. Inspecting the scales on both inputs of every addition and concatenation is a fast check. Sensitive first and last layers. The first layer consumes raw input, whose precision may matter more than intermediate features; the last layer produces logits or regression outputs whose small differences decide the answer. Keeping these at higher precision is a long-standing practice and costs little, since they are usually a small share of total computation — though on an accelerator, any layer left at higher precision may fall back to the CPU, a cost that must be measured. Nonlinear functions. Sigmoid, tanh, softmax, exponentials, and division are implemented in integer runtimes with lookup tables or polynomial approximations. Their outputs have fixed quantization parameters, and inputs outside the expected range saturate. Attention layers and gated recurrent units concentrate these functions and are often where quantization of sequence models goes wrong. Mixed precision Once sensitive layers are identified, the cheapest remedy is often mixed precision: keep those layers at 16-bit activations, or in floating point, and quantize the rest to 8 bits. Because a handful of layers usually accounts for most of the error and a small share of the computation, the cost in speed and size is modest. The constraint is hardware. On a CPU, mixing precisions costs a few conversion operations. On an NPU that executes only 8-bit operations, a floating-point layer splits the graph and sends part of the work back to the host processor, with tensor copies and synchronisation on each side. Arm's Ethos-U NPUs support 16-bit activations natively, which makes int16 activations on sensitive layers an attractive middle ground on those parts. The right choice depends on what the target executes natively, which is another reason to settle the target early. More systematic methods exist for choosing a precision per layer. Hessian-aware approaches such as HAWQ, from Zhen Dong, Amir Gholami, and colleagues, use second-order information about the loss to estimate each layer's sensitivity and allocate bits accordingly; search-based methods treat the bit-width of each layer as a hyperparameter and optimise it against measured latency. For most deployments, simple sensitivity analysis followed by keeping one to three layers at higher precision captures most of the available benefit. Quantization-aware training When targeted fixes are not enough — or when the target bit-width is below 8 — the model must learn to tolerate quantization. Quantization-aware training, or QAT, does this by simulating quantization during training, so that the optimiser sees, and compensates for, the error it introduces. The mechanism is the insertion of fake quantization operations into the training graph. Each fake-quantize operation takes a floating-point tensor, quantizes it to the target grid, and immediately dequantizes it, so that the tensor remains floating point but takes only the values the integer model will be able to represent. The forward pass therefore computes what the quantized model will compute, including rounding and clipping errors, and the loss reflects them. The difficulty is the backward pass. Rounding is a step function whose derivative is zero almost everywhere, which would stop gradients dead. The standard remedy is the straight-through estimator, an idea analysed by Yoshua Bengio and colleagues in 2013: in the backward pass, treat the rounding operation as if it were the identity, passing the gradient through unchanged, while zeroing it for values that were clipped.1 The estimator is mathematically crude but works remarkably well in practice, because the weights being trained are still floating-point values that accumulate small updates until they cross a rounding boundary. A refinement called learned step size quantization, introduced by Steven Esser and colleagues at IBM in 2020, makes the quantization scale itself a trainable parameter, with a carefully derived gradient.2 Rather than fixing ranges from calibration statistics, the network learns the scales that minimise the training loss, trading rounding error against clipping error layer by layer. LSQ and its successors are the basis of most high-quality low-bit QAT. Several practical points determine whether QAT succeeds. • Start from a well-trained floating-point model and fine-tune, rather than training with fake quantization from scratch. A short fine-tuning run — a few percent of the original training schedule — at a reduced learning rate is usually enough at 8 bits; more is needed at 4 bits. • Handle batch normalisation correctly. At inference, batch normalisation is folded into the preceding convolution's weights, so the weights that are quantized are the folded ones. QAT must simulate this folding, or it will train the network to tolerate the wrong quantization error. Toolchains handle this, but custom training loops often do not. A common practice is to freeze batch-normalisation statistics partway through QAT so that the folded weights stop shifting. • Match the target's scheme exactly. Symmetric or asymmetric, per-tensor or per-channel, the rounding mode, and which operators are fused must all match what the target runtime will do. A model trained for one scheme and deployed with another loses much of the benefit. • Evaluate the converted model, not the QAT model. The fake-quantized training graph approximates the integer model; the converted model is the integer model. Small discrepancies in rounding or fusion can make the converted model worse than the training graph suggests. At 8 bits, QAT typically closes most of the gap between a poorly quantizing model and its floating-point original. Its costs are engineering ones: a training pipeline that must be maintained alongside the floating-point one, additional hyperparameters, and longer iteration cycles. It is worth paying when PTQ with targeted fixes still falls short of the accuracy floor, and it is essentially mandatory below 8 bits for convolutional networks. Recurrent and stateful models Recurrent networks — LSTMs and GRUs in particular — remain common at the edge for audio, sensor, and time-series tasks, and they quantize less gracefully than convolutional networks. The reason is state. A recurrent layer carries a hidden state, and in the LSTM's case a separate cell state, from one time step to the next. Quantization error introduced at one step is fed back at the next and can accumulate over long sequences, so that a model that agrees closely with its floating-point original for the first few steps drifts away from it later. The gates make matters worse. Each LSTM gate passes a linear combination of inputs and state through a sigmoid or tanh, and the cell state is updated by multiplying gate outputs together and adding the results. Those elementwise products and additions combine quantities with very different ranges, and the cell state can grow over time in ways that a range fixed at calibration does not anticipate. For this reason, fully integer LSTM implementations, including the one in LiteRT, typically keep the cell state at 16 bits even when weights and most activations are at 8, and use carefully specified fixed-point approximations for the gate non-linearities. Several practices help. Calibrate on sequences at least as long as those seen in deployment, so that the recorded ranges include the state's growth over time. Evaluate on long sequences, comparing floating-point and quantized outputs step by step to see whether and where drift sets in. Where the architecture can be chosen, prefer designs whose state is bounded, or replace recurrence with causal convolutions over time, which quantize like any other convolution and have no feedback path for errors to circulate in. And if a recurrent model must be quantized aggressively, quantization-aware training on full-length sequences is usually needed, because only training can teach the network to keep its state within a range the integer arithmetic represents well. Below eight bits Fewer bits per weight means less storage and less memory traffic, which, given the arguments of Chapter 1, is attractive. But accuracy falls faster as bit-width shrinks, and hardware support thins out. At 4 bits, each weight has only sixteen possible values. Convolutional networks can be brought to 4-bit weights with 8-bit activations with modest accuracy loss using QAT and techniques such as LSQ; 4-bit activations are considerably harder. The hardware question is whether the target can compute directly in 4 bits. Some NPUs can; many cannot, and instead unpack 4-bit weights to 8 bits in their weight decoder or kernel. In that case the arithmetic is no faster, but storage and memory traffic are halved, which is precisely the benefit that matters most on memory-bound workloads. At 2 bits and 1 bit, networks become ternary or binary, and multiplication can be replaced by sign flips, bit operations, and population counts. Binary networks such as XNOR-Net drew considerable research interest because of the dramatic theoretical savings, but they have struggled to match the accuracy of 8-bit models on hard tasks and have seen limited production use, partly because mainstream processors and NPUs are not built to exploit them. They remain worth knowing about for extremely constrained targets and for simple tasks where the accuracy gap is tolerable. Language models: where low-bit quantization grew up Most of the recent progress in low-bit quantization has come from large language models, for a reason Chapter 1 made explicit: generating each token requires reading every weight, so token rate is bounded by memory bandwidth divided by model size in bytes. Every bit removed from each weight raises that ceiling proportionally. That economic pressure has produced a body of techniques that edge developers can now use directly, as phones and embedded boards increasingly run language models with a few billion parameters. The first obstacle was outliers. Tim Dettmers and colleagues, in work published in 2022 as LLM.int8(), observed that transformer language models above a certain size develop a small number of activation features with magnitudes far larger than the rest, concentrated in a few channels and appearing consistently across layers.3 Per-tensor 8-bit quantization of those activations destroyed accuracy. Their remedy was to split the matrix multiplication, computing the few outlier dimensions in 16-bit floating point and the rest in 8-bit integers. SmoothQuant, from Guangxuan Xiao, Ji Lin, and colleagues, took a different route: because the difficulty lies in activations while weights are easy to quantize, it mathematically migrates part of the activation range into the weights, scaling activation channels down and the corresponding weight rows up by the same factors, after which both quantize well to 8 bits.4 This is the same function-preserving rescaling idea as cross-layer equalization, applied to a new problem. The second development was weight-only quantization at 3 and 4 bits, which keeps activations in 16-bit floating point and quantizes only the weights. Because token generation is memory-bound, this delivers most of the speed benefit while sidestepping the activation-outlier problem. Two methods dominate. GPTQ, from Elias Frantar and colleagues and published at ICLR in 2023, quantizes weights one column at a time within each layer and, after quantizing each column, updates the remaining unquantized weights to compensate for the error just introduced, using second-order information estimated from a small calibration set.5 It brought models with over a hundred billion parameters to 3 and 4 bits in a few GPU-hours with small accuracy loss. AWQ, activation-aware weight quantization, from Ji Lin and colleagues and recognised with a best-paper award at MLSys 2024, starts from the observation that a small fraction of weight channels — those multiplied by large activations — matter disproportionately to output quality.6 Rather than keeping them in higher precision, which is awkward for hardware, AWQ scales those channels up before quantization so they are represented more finely, with the inverse scaling absorbed elsewhere. The method needs no backpropagation and generalises well across domains. Both methods are usually combined with group-wise quantization: instead of one scale per output channel, weights are divided into groups of, typically, 32 to 128 consecutive values, each with its own scale and sometimes its own zero point. Smaller groups track local variation more closely at the cost of storing more scales; a group of 128 with a 16-bit scale adds about an eighth of a bit per weight. Widely used local inference tools, such as llama.cpp and its GGUF file format, offer a family of such block-quantized formats at various bit-widths and group arrangements. For the most aggressive settings, QAT has returned. Model developers increasingly release checkpoints trained with quantization in the loop so that their 4-bit versions lose less quality than post-training methods allow; Google, for instance, published quantization-aware-trained 4-bit versions of its Gemma 3 open models in 2025. For an edge developer, the practical guidance is simple: when a vendor provides a QAT-trained low-bit checkpoint for the model you need, prefer it to quantizing the full-precision model yourself. A decision order Pulling the chapter together, a sensible sequence for quantizing a model for the edge looks like this. Start with full-integer post-training quantization to 8 bits with per-channel weights and a clipping range estimator, calibrated on representative data. Evaluate the converted model on the full validation set. If the loss exceeds the budget, run per-layer comparison and sensitivity analysis to locate the problem, and apply targeted fixes: better ranges, equalization, bias correction, AdaRound, or higher precision on a few layers, checking each time that the target hardware supports the resulting mixture without costly fallbacks. Only if those fail, or if the target requires fewer than 8 bits, move to quantization-aware training, matched exactly to the target's scheme. For language models, start from weight-only 4-bit quantization with GPTQ or AWQ and group-wise scales, or from a vendor's QAT checkpoint. At every step, the criterion is the same: accuracy of the converted model, measured on the target's arithmetic, against the floor set in the budget. CHAPTER 5 Pruning: Sparsity the Hardware Can Use Neural networks are over-parameterised. A trained network typically contains a large fraction of weights that contribute almost nothing to its output, and those weights can be removed — set to zero and left there — with little or no loss of accuracy, especially if the network is fine-tuned afterwards. This observation is as old as modern neural networks themselves: Yann LeCun, John Denker, and Sara Solla described "Optimal Brain Damage" in 1990, using second-derivative information to decide which weights to remove, and Babak Hassibi and David Stork refined it as "Optimal Brain Surgeon" a few years later. The modern revival came from Song Han and colleagues, whose 2015 paper showed that simply removing the weights with the smallest magnitudes, then retraining, could reduce the parameter count of AlexNet ninefold, from 61 million to 6.7 million, and of VGG-16 thirteenfold, from 138 million to 10.3 million, with no loss in accuracy.1 Their follow-up, Deep Compression, combined pruning with weight sharing and Huffman coding to shrink AlexNet's storage from 240 megabytes to 6.9 — a factor of 35 — and VGG-16's from 552 megabytes to 11.3, a factor of 49.2 Those numbers launched a large research field. They also created a lasting misunderstanding. A network with 90 per cent of its weights set to zero is not, in general, ten times faster or ten times smaller in memory, and on many devices it is neither faster nor smaller at all. Whether pruning pays off depends almost entirely on the pattern of the zeros and on whether the target hardware and software can exploit that pattern. This chapter is about the difference. Why scattered zeros are not free Consider a weight matrix in which 50 per cent of the entries, chosen individually by magnitude, have been set to zero. This is unstructured sparsity: the zeros fall wherever the small weights happened to be, with no regular arrangement. Stored densely, the matrix is exactly as large as before; the zeros occupy space like any other value. To save storage, it must be stored in a sparse format that records only the non-zero values and their positions. A common format, compressed sparse row, stores each non-zero value and its column index, plus a pointer to the start of each row. If values are 8-bit integers and column indices are 16-bit integers, each non-zero entry costs three bytes, compared with one byte per entry stored densely. At 50 per cent sparsity, the sparse representation is larger than the dense one. Cleverer encodings — relative indices in fewer bits, bitmaps marking non-zero positions — reduce the overhead, but the break-even point for unstructured sparsity on 8-bit weights is commonly well above 50 per cent, and the savings remain modest until sparsity is very high. Computation is worse. Dense matrix and convolution kernels are fast because they process contiguous blocks of data with vector instructions: a single instruction multiplies and accumulates sixteen or thirty-two values at once, and the memory system streams data predictably. Skipping individual zeros requires reading indices, gathering scattered input values, and breaking the regular pattern that vector units and caches depend on. Below high sparsity levels, a sparse kernel processing only the non-zeros is slower than a dense kernel processing everything, zeros included. This combines with the quantization of the previous chapters. A dense 8-bit weight is already one byte; there is little left for unstructured sparsity to save once indices are paid for. Han's original results were reported on 32-bit floating-point weights, where each removed weight saved four bytes and sparse bookkeeping was comparatively cheap. Granularity: the decisive choice The remedy is to prune in patterns the hardware can exploit. Pruning methods are best classified by granularity — the size and shape of the units removed — because granularity determines what hardware benefit is possible. Table 3 summarises the main options. Table 3. Pruning granularity and the benefit hardware can realise. Pattern Removes Accuracy cost Benefit on generic hardware Needs special support Weight Single weights (unstructured) Lowest Little, except compression Sparse kernels at high sparsity Block Small tiles of weights Low to moderate Some with block kernels Block-sparse kernels N:M N of every M weights Low None Yes, e.g. 2:4 units Channel Whole output channels or filters Higher Full: smaller dense model No Layer Whole layers Highest Full No At one end, unstructured pruning removes individual weights. It achieves the highest sparsity for a given accuracy, because it has the most freedom to choose what to remove, but delivers the least on ordinary hardware. At the other end, structured pruning removes entire output channels of a convolution — equivalently, whole filters — or entire neurons of a fully connected layer. The result is not a sparse network at all but a smaller dense one: a convolution with 64 output channels pruned to 40 is simply a convolution with 40 output channels, and the next layer's input channels shrink correspondingly. Every runtime, every kernel library, and every accelerator benefits immediately, because the network is just narrower. The cost is accuracy: removing whole channels is a coarser operation than removing individual weights, so less can be removed before accuracy suffers. Between them lie two intermediate forms. Block sparsity removes small rectangular tiles of weights, matching the tile size that a kernel or accelerator processes at once; when a whole tile is zero, it can be skipped without per-element bookkeeping. N:M or semi-structured sparsity requires that in every group of M consecutive weights, at most N are non-zero. The best-known case is 2:4 sparsity, introduced in hardware with NVIDIA's Ampere generation of GPUs: in every group of four weights, two must be zero, and the hardware stores only the two survivors plus a two-bit index for each, and skips the zero multiplications, doubling peak math throughput for eligible layers.3 The same pattern has reached the embedded world: Arm lists native hardware support for 2:4 sparsity among the features of its Ethos-U85 micro-NPU.4 N:M sparsity sits close to unstructured pruning in accuracy, because within each small group the pruner still chooses freely, while giving the hardware a perfectly regular pattern to exploit. The storage arithmetic for N:M sparsity shows why it is attractive. Under the 2:4 pattern, each group of four 8-bit weights is stored as two 8-bit values and two 2-bit position indices: twenty bits instead of thirty-two, or about 62 per cent of the dense size. That is a smaller saving than the nominal 50 per cent sparsity suggests, because of the indices, but it is achieved without any irregularity in the data layout, and on hardware that skips the zero multiplications the compute saving is the full half. The hidden difficulty of structured pruning Structured pruning sounds simple — remove channels, get a narrower network — but in modern architectures the channels of different layers are tied together, and removing one requires removing others consistently. In a plain chain of convolutions, pruning an output channel of one layer means removing the matching input channel of the next layer and nothing else. Residual networks are not plain chains. Wherever two tensors are added, as in every residual connection, they must have the same number of channels in the same order, so a channel can be removed from one only if the corresponding channel is removed from the other and from every other layer feeding or reading that same addition. In a MobileNetV2-style network, all the blocks within a stage share their residual channels, so a single pruning decision on the residual path touches many layers at once. Concatenations, depthwise convolutions (whose input and output channels correspond one to one), grouped convolutions, and attention heads impose further couplings. Getting these dependencies wrong produces a model that fails to run or, worse, runs with silently misaligned channels. Early structured-pruning code typically handled them by hand for each architecture. More recent tools build a dependency graph of the network automatically and prune coupled channels as groups; the DepGraph method of Gongfan Fang and colleagues, presented at CVPR in 2023 and implemented in the open-source Torch-Pruning library, is one widely used example. A practical consequence is that the residual channels of efficient architectures are often left unpruned, with pruning concentrated on the wide expansion layers inside each block, which are not coupled to anything outside it and which are, conveniently, where much of the computation lies. There is also a hardware subtlety. A structurally pruned network can end up with channel counts such as 37 or 53 — whatever the pruning criterion happened to leave. As Chapter 2 noted, many kernels and accelerators process channels in groups of 8, 16, or 32, so such counts waste part of every vector operation. Rounding pruned channel counts to the target's natural multiple usually costs a negligible amount of accuracy and recovers that waste. The practical rule is to choose granularity by asking what the target can exploit. On a microcontroller CPU with no sparse support, unstructured sparsity helps only storage compression, and only if the toolchain compresses weights; channel pruning helps everything. On an NPU with N:M support, N:M sparsity is the obvious choice. Pruning in a pattern the hardware ignores produces a model that is smaller in a paper and identical on the device. Where unstructured sparsity still pays Unstructured pruning is not useless at the edge, but its benefits arrive by two specific routes. The first is compression for storage and transfer. A weight tensor with many zeros compresses well with general-purpose or specialised entropy coding, even when stored densely. This matters for over-the-air model updates, where bytes transmitted cost energy and money, and for accelerators that decompress weights on the fly. Arm's Ethos-U NPUs, for instance, store weights in a compressed format produced by the Vela compiler and decompress them in hardware as they are fetched, so that zeros and repeated values reduce both the flash footprint and the bandwidth needed to stream weights to the accelerator. On such a target, unstructured sparsity and clustering translate into real savings in the resource that matters most. The second is specialised sparse kernels at high sparsity. Erich Elsen and colleagues at Google showed, in work presented at CVPR in 2020, that sparse versions of MobileNet-style networks could run substantially faster than their dense counterparts on mobile Arm CPUs, using kernels designed for the sparse 1-by-1 convolutions that dominate those networks — but at sparsity levels around 70 to 90 per cent, not 50.5 That work found its way into the XNNPACK library used by mobile runtimes. The lesson is that unstructured sparsity can pay on CPUs, but only with purpose-built kernels and only at sparsity levels that require careful training to reach without unacceptable accuracy loss. Weight clustering, a close relative of pruning, deserves mention here. Instead of setting weights to zero, it groups each layer's weights into a small number of clusters — sixteen, say — and replaces each weight with its cluster's centroid, so that the layer contains only sixteen distinct values. Deep Compression used this as its weight-sharing stage. Like unstructured sparsity, clustering does not speed up ordinary arithmetic, but it makes weights far more compressible, and the TensorFlow Model Optimization Toolkit offers it alongside pruning and quantization for exactly that purpose. How to prune: criteria and schedules The choice of which units to remove has generated an enormous literature, but the simple criteria remain strong. For unstructured pruning, weight magnitude is the standard: small weights contribute less. For channel pruning, common criteria include the magnitude of each filter's weights, such as its L1 norm, used by Hao Li and colleagues in an influential 2017 paper on pruning filters,6 or the scaling factor of the batch-normalisation layer that follows each channel — the basis of "network slimming", which adds a sparsity-inducing penalty on those factors during training so that unimportant channels are driven towards zero and can be removed cleanly. More elaborate criteria use gradients or second-order information to estimate each unit's effect on the loss. They can help, especially at high sparsity, but the gains are less consistent than many papers suggest. A 2020 survey by Davis Blalock and colleagues, "What Is the State of Neural Network Pruning?", examined some eighty pruning papers and found that inconsistent baselines, architectures, and metrics made most results impossible to compare, and that simple magnitude pruning often performed about as well as more complicated methods when compared fairly.7 For a practitioner, that is liberating: start with magnitude-based criteria and spend effort on the schedule and fine-tuning instead. The schedule matters more than the criterion. Removing a large fraction of weights at once damages a network badly; removing them gradually, with training in between, lets the remaining weights adapt. Michael Zhu and Suyog Gupta's 2017 study "To prune, or not to prune" proposed a widely adopted gradual pruning schedule in which sparsity rises from zero to its target over the course of training, quickly at first and more slowly as it approaches the target, following a cubic curve.8 They also reported a result relevant to Chapter 2: for a fixed memory budget, large networks pruned to sparsity often outperformed small dense networks of the same non-zero parameter count. That finding is real, but it compares non-zero parameters, not memory or latency on hardware that cannot exploit unstructured sparsity — exactly the distinction this chapter has been drawing. The Lottery Ticket Hypothesis, published by Jonathan Frankle and Michael Carbin at ICLR in 2019, offered a striking explanation of why pruning works.9 They found that dense networks contain sparse subnetworks which, if reset to their original initial weights and trained in isolation, can match the accuracy of the full network — winning tickets in an initialisation lottery. The result deepened understanding of over-parameterisation and spawned a large follow-on literature, but it is not itself a deployment technique: finding the ticket requires training the full network, often repeatedly. Its practical message is mostly reassurance that heavily pruned networks can be genuinely capable, not degraded approximations. For language models, one-shot methods have been developed because retraining is so expensive. SparseGPT, from Elias Frantar and Dan Alistarh, prunes large models to 50 per cent or more unstructured sparsity in one pass using the same second-order, layer-wise reconstruction ideas as GPTQ, and Wanda, from Mingjie Sun and colleagues, achieves competitive results with a criterion as simple as weight magnitude multiplied by input activation norm. Both support N:M patterns, which is where their speed benefit on hardware comes from. Pruning with quantization Pruning and quantization combine, but order and coordination matter. The usual sequence is to prune first, fine-tune to recover accuracy, then quantize — with quantization-aware training if needed. Naive QAT on a pruned model may, however, nudge zeroed weights away from zero, destroying the sparsity pattern. Toolchains therefore offer sparsity-preserving variants of QAT that hold pruned weights at zero throughout; the TensorFlow Model Optimization Toolkit calls its version "collaborative optimization" and supports chains such as pruning followed by clustering followed by quantization, each stage preserving what the previous one produced. There is also a question of whether both are worth doing. After 8-bit quantization of a well-designed efficient architecture, the remaining room for pruning to help — on hardware without sparse support — is mostly via structured pruning, which is to say via making the network narrower. If the architecture was already sized to the budget as Chapter 2 recommends, there may be little left to take. When to prune The honest summary is that pruning is a powerful technique whose value at the edge is highly conditional. It is clearly worth applying when: • The target hardware supports a sparsity pattern natively — N:M sparsity on a capable NPU or GPU, or weight compression in an accelerator's fetch path — and the pruning matches that pattern. • The network is over-sized for the task and structured channel pruning can produce a narrower dense network, particularly when retraining a smaller architecture from scratch is impractical. • Model download size matters and unstructured sparsity or clustering will make the weights compress better for transfer. • Very high unstructured sparsity is achievable and the runtime has sparse kernels for the dominant layer types. It is usually not worth the effort when the target is a CPU or NPU without sparse support, the network is already a compact architecture sized for the budget, and the only form of sparsity on offer is unstructured. In that situation, the bytes and time are better recovered through quantization and a smaller architecture — and, as the next chapter argues, by training that smaller architecture to punch above its weight. CHAPTER 6 Knowledge Distillation: Teaching Small Networks Quantization and pruning start from a trained network and remove precision or parameters from it. Knowledge distillation works differently. It leaves the large network intact and uses it as a teacher, training a separate, smaller network — the student — to reproduce its behaviour. The student can have whatever architecture the device demands: a compact convolutional network sized to a microcontroller's activation budget, a shallow transformer that fits in a phone's memory, a design whose every operator the target NPU supports. Distillation does not compress the teacher; it transfers what the teacher learned into a body built for the edge. This makes distillation the natural partner of the architecture-first approach in Chapter 2. Choose an efficient architecture sized for the budget, then train it with the help of the best model you can obtain, and it will usually be noticeably more accurate than the same architecture trained on labels alone. Then quantize it. In many edge projects, that sequence — design small, distil, quantize — beats any attempt to compress a large model directly. Soft targets and temperature The idea that a small model can learn from a large one predates deep learning's dominance. Cristian Bucilă, Rich Caruana, and Alexandru Niculescu-Mizil showed in 2006 that a large ensemble of models could be compressed into a single small network by training the small network to match the ensemble's outputs on a large set of unlabelled and synthetic examples. The form most practitioners use today was set out by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean in their 2015 paper "Distilling the Knowledge in a Neural Network".1 Their central observation concerns what a trained classifier's output contains. A network that classifies an image of a husky outputs a probability distribution over all classes. The highest probability goes to "husky", but the rest is not uniform noise: "wolf" and "malamute" receive far more probability than "teapot". Those relative probabilities on the wrong classes encode what the network has learned about similarity between classes — information that a one-hot label, which says only "husky" and nothing else, does not contain. Hinton and colleagues called this the teacher's dark knowledge. The difficulty is that a confident teacher assigns tiny probabilities to the wrong classes, so their differences have almost no effect on a training loss. The remedy is temperature. A classifier's probabilities come from applying the softmax function to its logits, the raw pre-softmax scores. Dividing the logits by a temperature T greater than one before the softmax softens the distribution, raising small probabilities and lowering the largest, so that the relative structure among the wrong classes becomes visible. The student is trained to match the teacher's softened distribution at the same temperature, measured by Kullback-Leibler divergence. In the standard formulation, the student's loss is a weighted sum of two terms: the ordinary cross-entropy against the true labels, computed at temperature one, and the divergence from the teacher's softened distribution, computed at temperature T and multiplied by T squared. The T-squared factor is easy to overlook and important: softening the distributions shrinks the gradients of the distillation term by roughly that factor, and multiplying back keeps the two terms in balance as temperature changes. At inference, the student runs at temperature one like any classifier. Two hyperparameters therefore govern the basic method: the temperature, and the weight between the label loss and the distillation loss. Neither has a universal best value; practitioners typically try a handful of temperatures in the low single digits up to around ten, and weightings that put most of the emphasis on the distillation term when the teacher is strong. Both are cheap to tune relative to the gains available. Why it works The reasons distillation helps are worth understanding, because they guide its use. The first is a richer signal per example. Each training example, labelled one-hot, provides the student with a single bit of class information per class. The teacher's distribution provides a graded similarity structure over all classes. The student learns from more information per example, which matters most when it has limited capacity and cannot afford to waste it on distinctions the data does not support. The second is regularisation. Training a network to match a soft distribution rather than a hard label discourages it from becoming over-confident, much as label smoothing does, and tends to improve generalisation and calibration. The third, and in practice often the largest, is the use of unlabelled data. The distillation term needs only the teacher's outputs, not true labels. Any in-domain data the teacher can process — and edge applications often have abundant unlabelled data from the sensors themselves — becomes training data for the student. Hinton and colleagues called the data used for distillation the transfer set; making it large and representative is one of the most effective levers available. That last point connects distillation to a widely used edge workflow that rarely calls itself distillation: pseudo-labelling with a large model. A team building a tiny detector for, say, delivery packages on doorsteps may have millions of unlabelled frames from cameras in the field and no budget to label them. A large, accurate model — perhaps a general-purpose vision model too big for any edge device — labels the frames, and the tiny model trains on those labels. This is distillation through hard labels rather than soft distributions, and it loses some of the dark knowledge, but it lets the student learn from a vast in-domain dataset that would otherwise be unusable. Where the teacher's soft outputs can be stored, they should be; where only its decisions are available, pseudo-labels still help substantially. Distilling features as well as outputs Matching only the final output leaves much of the teacher's knowledge unused. A family of methods also asks the student to match the teacher's intermediate representations. FitNets, from Adriana Romero and colleagues at ICLR in 2015, introduced "hints": an intermediate layer of the teacher serves as a target for an intermediate layer of the student, with a small learned regressor mapping the student's features to the teacher's dimensions.2 This allowed thin, deep students to be trained that would otherwise have been hard to optimise. Attention transfer, from Sergey Zagoruyko and Nikos Komodakis, asks the student to match spatial maps derived from the teacher's activations, indicating where in the input the teacher's features respond, which avoids having to match full feature tensors of different widths. Other methods match the relationships between examples — for instance, the similarity between the representations of pairs of inputs — rather than the representations themselves. Feature-based distillation is most useful when the student and teacher share a broad structure, so that their intermediate layers correspond naturally, and for tasks beyond classification. Object detection and segmentation outputs are harder to distil purely at the output level, because the outputs are structured and the background dominates; matching intermediate feature maps, particularly around objects, has proved more effective there. Language models offer the clearest demonstrations of feature distillation's value. DistilBERT, from Victor Sanh and colleagues at Hugging Face in 2019, halved the number of layers in BERT and trained the student with a combination of the teacher's soft output distribution, the ordinary language-modelling loss, and a loss aligning the directions of the student's and teacher's hidden states; the result was 40 per cent smaller and 60 per cent faster while retaining 97 per cent of BERT's language-understanding performance on the GLUE benchmark.3 TinyBERT, from Xiaoqi Jiao and colleagues, went further by distilling attention matrices and hidden states layer by layer, both during general pre-training and again during task-specific fine-tuning. The capacity gap and the patient teacher An intuitive assumption is that a better teacher always produces a better student. It does not. Jang Hyun Cho and Bharath Hariharan, in a 2019 study of distillation's efficacy, found that very large, highly accurate teachers often produced worse students than smaller teachers, because the student lacked the capacity to mimic them, and that a teacher stopped early in its training sometimes worked better than a fully trained one.4 Seyed Iman Mirzadeh and colleagues proposed bridging large gaps with a teacher assistant: distil from the large teacher into a medium-sized network, then from that into the student. A complementary and more optimistic result came from Lucas Beyer and colleagues at Google, in a CVPR 2022 paper titled "Knowledge distillation: A good teacher is patient and consistent".5 They argued that many disappointing distillation results stemmed from how the training was run rather than from a fundamental capacity limit. Two practices mattered most. Consistency: the teacher and student must see exactly the same input, including the same random augmentation — which rules out the common shortcut of precomputing the teacher's outputs on unaugmented images. Patience: distillation benefits from far longer training than ordinary supervised learning, because the student is fitting a rich function rather than a set of labels, and gains continued over schedules of hundreds or thousands of epochs. Following these practices, they distilled a large teacher into a standard ResNet-50 that set a new accuracy level for that architecture on ImageNet. These findings sit together comfortably. A student does have a capacity limit, and a teacher far beyond it may not help. But most students are well short of that limit when trained briefly and inconsistently, and the practical first step is to train longer, with consistent augmentation, before concluding that a smaller teacher is needed. There is a subtler caution. Samuel Stanton and colleagues, in "Does Knowledge Distillation Really Work?" at NeurIPS 2021, showed that even when distillation improves a student's accuracy, the student often agrees with its teacher's predictions much less closely than one would expect — a gap they traced to the difficulty of optimisation rather than to a lack of capacity.6 The practical implication is that a distilled student should be evaluated as a model in its own right, on accuracy against ground truth and on the failure cases that matter, rather than assumed to inherit the teacher's behaviour. Distillation for language models on devices Distillation has become central to how small language models are built. Rather than training a small model from scratch on next-token prediction, developers increasingly train it to match a larger model's full predicted distribution over the vocabulary at each position — a much richer signal than the single correct next token. Google's technical report for its Gemma 2 open models, for example, describes training the smaller models in the family with knowledge distillation from a larger model rather than with standard next-token prediction alone, and attributes much of their quality to it. Other model families have followed similar approaches. A second form, sequence-level distillation, described by Yoon Kim and Alexander Rush in 2016 for translation, trains the student on complete outputs generated by the teacher rather than on token-level distributions. Its modern descendant is the widespread practice of generating synthetic training data with a large model to fine-tune a small one for a narrow task. For an edge developer who needs a small on-device model to perform one job well — extracting fields from receipts, classifying support messages, summarising notifications — fine-tuning a compact model on a large set of teacher-generated examples is often the most effective route to acceptable quality within a phone's budget. Distillation and quantization together Distillation and quantization interact well. The cleanest combination is to distil a floating-point student first, then quantize it with the methods of Chapters 3 and 4. When quantization-aware training is needed, the teacher can remain in the loop: the floating-point teacher supervises the fake-quantized student, so that the quantized student is trained to match a full-precision target rather than hard labels. This is sometimes called quantization-aware distillation, and it frequently recovers more accuracy than QAT on labels alone, especially at low bit-widths. A special case uses the floating-point version of the same model as the teacher for its own quantized version; the soft targets guide the quantized network back towards the behaviour it had before quantization. Mistakes that quietly waste distillation Distillation is simple enough to implement in a few lines, which makes it easy to implement slightly wrongly. Several errors recur, and each costs accuracy without producing an obvious failure. The first is running the teacher in training mode. If the teacher contains dropout or batch normalisation, it must be in evaluation mode while producing targets; otherwise dropout randomly corrupts its outputs and batch normalisation uses the statistics of the student's current mini-batch rather than its own learned statistics. The student then imitates a noisier, worse teacher than the one that was evaluated. The second is caching teacher outputs computed on unaugmented inputs and then training the student on augmented ones. This is tempting because it saves the cost of running the teacher during student training, but it means the student is asked to produce, for a cropped and colour-shifted image, the teacher's output for the original — a mismatch that Beyer and colleagues identified as a principal reason distillation underperforms. If teacher outputs must be cached for cost reasons, cache them for a fixed set of augmented views and train the student on exactly those views. The third is mishandling temperature. The student's softened outputs must be computed at the same temperature as the teacher's for the distillation term, while the label term and inference use temperature one; forgetting the T-squared factor silently shifts the balance between the terms as temperature changes, so that tuning temperature also tunes, unintentionally, the weighting. The fourth is distilling on the wrong data. A transfer set drawn only from the labelled training set wastes the method's greatest strength. A transfer set drawn from a different distribution — public web images for a model that will see infrared night frames — transfers the teacher's knowledge about the wrong world. The most valuable transfer data is unlabelled data from the deployment environment. The fifth is evaluating against the teacher instead of the task. A student that agrees with its teacher on 95 per cent of inputs has learned the teacher's mistakes along with its successes. The measures that matter are accuracy against ground truth and behaviour on the failure cases the product cares about. Distillation also extends beyond classification in ways that matter for edge work. For regression outputs — a predicted distance, a remaining-useful-life estimate for a machine component — the student can be trained to match the teacher's predictions directly, which smooths noisy labels in much the way soft targets do. For object detectors, matching the teacher's class scores and box predictions on the anchors or regions where the teacher found objects, plus intermediate features, works better than matching outputs everywhere, where background dominates. And self-distillation, in which a network is trained to match a previous generation of itself with the same architecture, often yields a modest but reliable improvement at no cost in size — a useful last step when no larger teacher is available. A practical recipe For a developer adding distillation to an edge project, the following sequence captures most of the benefit with modest effort. 1. Obtain the strongest teacher you can run during training. It need not fit on any device, and it can be an ensemble or a large pretrained model fine-tuned for the task. 2. Choose the student architecture for the device, as in Chapter 2, and verify that it fits the budget before training. 3. Assemble the largest representative transfer set you can, including unlabelled in-domain data from the deployment environment. 4. Train with the combined loss — labels plus softened teacher outputs — computing the teacher's outputs online on the same augmented inputs the student sees. 5. Train for longer than you would without distillation, and watch validation accuracy rather than training loss to decide when to stop. 6. If the student underperforms and the teacher is very much larger, try a smaller or earlier-stopped teacher, or an intermediate teacher assistant. 7. Quantize the student, using the teacher to supervise quantization-aware training if post-training quantization falls short. Distillation is the one technique in this booklet that typically improves a model's accuracy at a fixed size, rather than trading accuracy for size. That makes it the best investment available once the architecture has been sized to the device. The next chapter turns from the model to that device — and in particular to the specialised hardware that increasingly determines what a well-designed model can achieve on it. CHAPTER 7 Micro-NPUs and Neural Engines: What Accelerators Demand For most of the history of TinyML, running a neural network on a microcontroller meant running it on the CPU. Optimised kernel libraries squeezed remarkable performance from Cortex-M cores — Arm's CMSIS-NN, first described in 2018, reported several-fold improvements in speed and energy over naive implementations by exploiting the cores' single-instruction, multiple-data instructions1 — but a general-purpose core running at a few hundred megahertz can only do so much. A modest vision model might take hundreds of milliseconds, too slow for interactive use and too energy-hungry for continuous operation. That has changed. Neural processing units, long standard in phones, have moved down into microcontrollers and low-cost application processors. Arm announced its first micro-NPU, the Ethos-U55, in February 2020, alongside the Cortex-M55 processor, claiming that the pair could deliver up to 480 times the machine-learning performance of previous Cortex-M systems.2 Silicon vendors have since shipped parts built on that design and its successors, and several have designed NPUs of their own. A developer choosing hardware for an edge AI product today can buy a microcontroller with a dedicated accelerator for a few dollars. Accelerators change the economics of edge AI dramatically, but only for models that suit them. This chapter explains what an NPU actually is, what it demands of a model, and how to tell whether a given model will run well on one. What an NPU is An NPU is not a faster CPU. It is a fixed-function or narrowly programmable engine built around three components. The first is an array of multiply-accumulate units, typically 8-bit integer multipliers feeding wider accumulators. A micro-NPU might have between 32 and a few thousand such units; a phone NPU has many more. Each clock cycle, the array performs one multiply-accumulate per unit, so a 256-unit array at 500 megahertz performs 128 billion multiply-accumulates per second — conventionally quoted as 256 billion operations per second, or 0.256 TOPS, because each multiply-accumulate counts as two operations. The second is local SRAM close to the array, together with direct memory access engines that move data between that SRAM and the system's larger, slower memory. The array is only useful if it is fed, and feeding it is the main design problem. A good NPU tiles each layer's computation so that blocks of weights and activations are loaded into local memory, reused as many times as possible, and written back only when finished. The third is a command stream: a precompiled sequence of instructions describing which operations to perform, on which tiles, in which order, with which data movements. The NPU does not interpret a neural network graph at run time. An offline compiler translates the model into this command stream ahead of time, deciding the tiling, scheduling, and memory placement. The quality of that compiler matters as much as the hardware. Many NPUs add specialised units: a weight decoder that decompresses weights on the fly as they are fetched; elementwise units for additions and activations; lookup tables for non-linear functions; and pooling units. Each covers a specific set of operations, and the combination of all of them defines exactly what the NPU can do. The Ethos-U family as a worked example Arm's Ethos-U line is the most widely licensed micro-NPU design and illustrates the general pattern well, because its toolchain is open and its constraints are documented. The Ethos-U55 is designed to sit beside a Cortex-M processor in a microcontroller. It can be configured with 32, 64, 128, or 256 multiply-accumulate units per cycle, supports 8-bit weights with 8-bit or 16-bit activations, and decompresses weights in hardware. The Ethos-U65 is a larger relative intended for systems with more memory bandwidth, including those with an application processor; NXP's i.MX 93, for example, pairs Cortex-A55 cores with an Ethos-U65. The Ethos-U85, announced in April 2024 as the third generation, scales from 128 to 2,048 units per cycle, which Arm quotes as 256 billion to 4 trillion operations per second at a one-gigahertz clock, and added hardware support for transformer operations such as matrix multiplication, transpose, and gather, as well as native 2:4 sparsity.3 The deployment flow is the same across the family. The developer trains a model, quantizes it to 8-bit integers (or 16-bit activations with 8-bit weights), and exports it as a LiteRT flatbuffer. Arm's open-source Vela compiler then reads that file, identifies the operators the NPU can execute, compiles them into a command stream, compresses their weights, and writes out a new model file in which the supported portions are replaced by a single custom operator containing the command stream. At run time, LiteRT for Microcontrollers executes the model on the Cortex-M; when it reaches the custom operator, it hands the command stream to the Ethos-U driver, and the NPU runs it. Any operators Vela could not place on the NPU remain in the model as ordinary operators and run on the Cortex-M, using CMSIS-NN kernels where available. Vela produces a report, and reading it is the single most useful thing a developer can do when targeting an Ethos-U part. It lists, operator by operator, whether each was placed on the NPU or left for the CPU, and why; it estimates cycle counts and memory traffic for the NPU portions; and it reports peak SRAM usage. A model that Vela maps entirely to the NPU will typically run many times faster than on the CPU alone. A model in which a few operators in the middle of the network fall back to the CPU may run disappointingly, because each fallback splits the NPU's work into separate command streams, forces intermediate tensors through memory, and puts the comparatively slow CPU on the critical path. Operator coverage is the first constraint Every NPU supports a defined list of operators, each with its own conditions. A convolution may be supported only for certain kernel sizes, strides, and dilation factors; a pooling operation only up to a maximum window; an elementwise operation only when its inputs' shapes broadcast in particular ways; a reshape only in some positions. Activation functions beyond the common ones may be handled through lookup tables with specific constraints on quantization parameters. The lists are published in each vendor's documentation and enforced by their compilers. For model design, this means the operator constraints of the target accelerator should be checked before an architecture is committed to — the same advice Chapter 2 gave for runtimes, with higher stakes. The commonest causes of fallback are familiar: an unusual activation function, a normalisation layer that cannot be folded, an operator introduced by a framework's export path that the developer never wrote explicitly (a transpose inserted to change memory layout, a cast between data types, a dynamic reshape), and post-processing operations such as non-maximum suppression appended to the end of a detection model. Some of these can be fixed by changing the export settings; others require changing the model; post-processing usually belongs on the CPU anyway and should be separated from the network. A useful discipline is to run the target compiler on a candidate architecture in its first, untrained form — random weights quantized with dummy calibration data — before investing in training. The compiler's report on operator placement and memory use depends on the architecture, not on the trained weights, and an incompatibility discovered in the first hour costs nothing. The cost of a single fallback is worth reasoning through, because it is routinely underestimated. Imagine a network of twenty layers in which the NPU can run every layer except the tenth. Without that fallback, the whole network is one command stream: the NPU runs from input to output, keeping intermediate tensors in its working memory and overlapping data movement with computation. With it, the work splits into three pieces. The NPU runs layers one to nine and writes the ninth layer's output to SRAM; the CPU reads it, runs layer ten with its own kernels, and writes the result back; the NPU starts again with layers eleven to twenty. The CPU portion may be only one layer out of twenty, but the CPU might be ten or twenty times slower than the NPU at that operation, and the hand-offs add synchronisation, cache maintenance, and lost overlap. It is common for one or two fallbacks to account for a large share of total inference time, which is why a placement report showing "nearly all" operators on the NPU is not the good news it sounds like. Memory is still the second constraint The arguments of Chapter 1 apply to accelerators with extra force, because an NPU's arithmetic is so fast that almost any layer can become memory-bound. On a microcontroller with an Ethos-U55, the NPU usually has no large memory of its own; it reads weights from flash or external memory and reads and writes activations in the microcontroller's SRAM. If weights must be fetched from slow external flash, the NPU waits on them; Vela's compression helps, and so does placing weights in faster internal memory when they fit. If activations do not fit in on-chip SRAM, they must spill to external RAM, and throughput falls sharply. Vela offers several memory modes reflecting different system designs — everything in SRAM, weights in flash with activations in SRAM, or a dedicated SRAM used as a cache in front of slower memory on the larger Ethos-U parts — and the right choice depends on the board, not the model. This is why advertised throughput is a ceiling rather than a prediction. The TOPS figure quoted for an NPU assumes every multiply-accumulate unit is busy every cycle. Real models achieve some fraction of that, depending on how well each layer's shape fits the array, how much each layer reuses its data, and whether memory can keep up. Arm, for instance, describes the Ethos-U85 as reaching up to 85 per cent utilisation on popular networks, which is a good result and still implies a gap between the headline and practice;3 depthwise convolutions and small layers typically fall well short of whatever figure a vendor quotes. Comparisons between accelerators on TOPS alone are close to meaningless, all the more so because vendors do not always quote the same data type — some figures assume 4-bit arithmetic or count skipped operations in sparse layers. A landscape of edge accelerators The range of available accelerators is wide and changes quickly, but a few representative examples, set out in Table 4, illustrate the classes a developer is likely to meet. Two cautions apply to reading it. The Ethos-U entries are licensed designs rather than chips, so their real throughput depends on the configuration and clock speed each silicon vendor chooses; the other entries are specific products. And the figures are peak ratings quoted by their makers, on the terms each maker chose, so they indicate scale rather than support a direct comparison. Table 4. Representative edge neural accelerators. Part Host Peak int8 Notes Ethos-U55 Cortex-M 32–256 MACs/cycle Licensed IP; many MCU vendors Ethos-U65 Cortex-M or A Up to 512 MACs/cycle Higher-bandwidth systems Ethos-U85 Cortex-M or A 256 GOPS–4 TOPS at 1 GHz Transformer ops; 2:4 sparsity Neural-ART STM32N6 (M55) 600 GOPS ST's in-house NPU, 2024 Edge TPU USB/PCIe module 4 TOPS About 2 TOPS per watt Source: vendor announcements from Arm (2020, 2024), STMicroelectronics (December 2024), and Google Coral documentation. Beyond the Ethos-U designs, which appear in microcontrollers from vendors such as Alif Semiconductor, Infineon, and Himax as well as in NXP's application processors, several silicon makers now design their own. STMicroelectronics' STM32N6, launched in December 2024, pairs an 800-megahertz Cortex-M55 with an in-house accelerator it calls the Neural-ART Accelerator, rated at 600 billion operations per second, together with a hardware image-processing pipeline for camera input. NXP's MCX N microcontrollers carry its own eIQ Neutron NPU. Google's Coral Edge TPU, available as a USB accelerator, a PCIe card, or a module, is rated at 4 TOPS at about two TOPS per watt and runs models compiled with its own Edge TPU compiler. It is worth noting a less dramatic but important development alongside the NPUs: vector extensions to the microcontroller CPU itself. The Cortex-M55 and Cortex-M85 implement Helium, Arm's M-profile vector extension, which lets the CPU process several 8-bit values per instruction. Arm's own figure for the Cortex-M55 alone was up to fifteen times the machine-learning performance of earlier Cortex-M cores.2 Helium matters even in systems with an NPU, because it accelerates the operators that fall back to the CPU, as well as pre-processing such as audio feature extraction. Phone NPUs and their gateways Phone NPUs are far more powerful than micro-NPUs, but developers rarely program them directly. They are reached through vendor or platform software layers, and the choice of layer often matters more than the hardware. On Apple devices, the Neural Engine is reached through Core ML. The developer converts a model to Core ML's format, and the framework decides at run time how to divide the work among the CPU, GPU, and Neural Engine. The developer can express a preference but not force placement, and operators or configurations the Neural Engine does not support run elsewhere. Apple's tools report which compute unit each layer ran on, and checking that report is the equivalent of reading Vela's. On Android, the landscape is more fragmented. Google's Neural Networks API, intended as a common interface to vendor accelerators, was deprecated as of Android 15, and the recommended path is now to use a runtime — LiteRT or ExecuTorch — with delegates or backends for specific accelerators: a GPU delegate, and vendor-specific delegates for Qualcomm's Hexagon NPU, MediaTek's APU, and others. Each delegate takes the portion of the graph it supports and leaves the rest on the CPU, exactly as Vela does on a microcontroller. ExecuTorch's 1.0 release in October 2025 listed production-ready backends for Qualcomm's AI Engine, Apple's Core ML, Arm's Ethos-U, Vulkan GPUs, and XNNPACK on CPUs, with further vendor backends at earlier stages.4 The practical difficulty on Android is that a model tuned for one vendor's NPU may run poorly or fall back to the CPU on another's. Applications that must run across many devices commonly ship a model that runs well on CPU and GPU everywhere, with accelerator delegates enabled where they have been validated, and fall back gracefully elsewhere. Choosing hardware with the model in mind For teams choosing hardware rather than inheriting it, the questions follow from everything above. What does the model actually need? Compute the multiply-accumulates per inference and the inference rate. A keyword spotter running a few million multiply-accumulates ten times a second does not need an NPU; a capable Cortex-M with Helium handles it with ample margin. A camera-based detector running hundreds of millions of multiply-accumulates at several frames per second does. Does the accelerator support the model's operators? Run the vendor's compiler on the candidate model and read the placement report. Partial support can be worse than none. Does the memory system support the accelerator? Check how much on-chip SRAM is available for activations and how fast weights can be fetched. An NPU starved by slow external flash will not reach its rated performance. Is the toolchain mature? An accelerator is only as good as its compiler, runtime, and documentation. Open, well-documented flows with active communities save weeks of debugging. What is the energy per inference, measured? Vendor figures are useful starting points, but energy per inference on the developer's own model, including the CPU, memory, and idle power consumed while waiting, is what determines battery life. These questions point to a single conclusion. An accelerator is not a feature that a model benefits from automatically; it is a target that the model must be built for. The compression techniques of the preceding chapters produce models that are small and integer-valued; the accelerator rewards models that are also made of operators it supports, arranged in shapes it handles efficiently, with activations that fit in its memory. The next chapter turns to the software that connects the two — the toolchains that turn a trained model into something a device can execute, and the profiling that shows whether it worked. CHAPTER 8 From Checkpoint to Device: Toolchains, Runtimes, and Honest Measurement A trained model is not a deployable artefact. Between the checkpoint a training script saves and the binary that runs on a phone or microcontroller lies a chain of tools — graph capture, conversion, quantization, backend compilation, runtime integration — each of which can change the model's behaviour in small ways, and any of which can silently break it. Most of the time lost in edge AI projects is lost here, not in model design. A model that scored well in a notebook and then misbehaves on the device has usually been altered somewhere along this chain, or is being fed different input than it was trained on. This chapter describes the chain as it stands in 2026, the runtimes that sit at its end, and the measurement practices that make problems visible early. The tool names will change; the shape of the chain and the discipline of measuring along it will not. The shape of the chain Every deployment path, whatever its branding, performs the same five steps. Capture. The model's computation is extracted from the training framework as a static graph: a fixed list of operators, their connections, and their weights, free of the Python control flow that training code relies on. In TensorFlow, this is a SavedModel or a traced Keras function. In PyTorch, the current mechanism is torch.export, which traces the model into a graph of standard operators with explicit shapes. Conversion. The captured graph is translated into the representation the target runtime understands: a LiteRT flatbuffer, an ExecuTorch program, an ONNX file, a Core ML package. Conversion maps framework operators onto runtime operators, folds constants, fuses patterns such as convolution followed by batch normalisation and activation into single operators, and may rewrite parts of the graph to suit the runtime. Quantization. Weights and activations are converted to integers, using the calibration and training techniques of Chapters 3 and 4. Depending on the path, quantization happens before conversion (quantization-aware training in the framework), during it (LiteRT's converter calibrates as it converts), or as a separate stage applied to the captured graph (PyTorch's export-based quantization). Backend compilation. Where an accelerator is involved, a vendor compiler or delegate partitions the graph, compiling the supported portions into the accelerator's own format — an Ethos-U command stream from Vela, a Core ML program for the Neural Engine, a Qualcomm context binary for Hexagon — and leaving the rest for the CPU. Integration. The result is embedded in the application: loaded from a file on a phone, compiled into the firmware image as a constant array on a microcontroller. The application provides inputs, invokes the runtime, and consumes outputs. At each step, the model can change. Conversion can substitute an operator with a numerically different implementation. Fusion changes the order of floating-point operations. Quantization introduces error by design. Backend compilation may use different rounding or saturation from the reference kernels. And integration can feed the model input that differs from training input in ways that no tool can detect. Knowing where each change can enter is the first defence against it. The runtimes Three runtime families cover most edge deployments today. LiteRT, formerly TensorFlow Lite and renamed by Google in September 2024, remains the most widely deployed. Models are stored as flatbuffers — a serialisation format that can be read in place without parsing or copying, which suits memory-mapped loading on phones. On Android and embedded Linux, an interpreter executes the model with optimised CPU kernels, chiefly from the XNNPACK library, and hands sub-graphs to delegates for GPUs and NPUs. Despite its origins, LiteRT now accepts models authored in PyTorch and JAX as well as TensorFlow through Google's AI Edge conversion tools. LiteRT for Microcontrollers, formerly TensorFlow Lite Micro, is a separate, stripped-down runtime for devices without an operating system. Its design, described by Robert David and colleagues at MLSys in 2021, follows from the constraints of Chapter 1.1 It performs no dynamic memory allocation during inference; the application supplies the tensor arena, and the runtime plans all activations within it at initialisation. It includes only the operators the application registers through an operator resolver, so that unused kernels do not consume flash. It runs the same flatbuffer format as the larger runtime, and it accepts optimised kernel implementations — CMSIS-NN for Cortex-M, vendor libraries for DSPs — in place of its portable reference kernels. ExecuTorch is PyTorch's on-device runtime, which reached its 1.0 release in October 2025.2 Its flow begins with torch.export, lowers the exported program to a restricted operator set suitable for edge devices, and then hands partitions of the graph to backends — XNNPACK for CPUs, Core ML, Qualcomm's AI Engine, Arm's Ethos-U, Vulkan, and several others — before serialising the result as a program file the runtime executes. A selective build mechanism links only the operators a given model uses. Its main attraction for many teams is that it removes a framework conversion step: a model developed in PyTorch stays in PyTorch's operator semantics until it reaches the backend, reducing one common class of conversion mismatches. Meta reports that it runs inside its own applications on billions of devices. Alongside these sit ONNX Runtime, which offers a mobile build and execution providers for various accelerators and is common where models arrive in ONNX format from mixed toolchains; vendor toolchains such as ST's tools for STM32 parts and NXP's eIQ, which wrap the general runtimes or generate their own code for each vendor's chips; and code-generating compilers, which skip the interpreter entirely and emit C code specialised to one model, as MCUNet's TinyEngine did. Code generation can save memory and time by eliminating interpretive overhead and planning memory more aggressively, at the cost of a less general deployment flow. The choice among them is usually dictated by the target: the vendor's supported path for a given accelerator, the platform's preferred runtime for a phone, and the team's training framework. Where there is freedom, the deciding factors should be operator coverage on the target, maturity of the backend for the specific accelerator, and quality of the profiling tools — the last being consistently underrated. The pre-processing trap The most common cause of a model that works in development and fails on the device is not quantization or conversion. It is a mismatch between how inputs were prepared during training and how they are prepared on the device — what machine-learning engineers call training-serving skew. Training pipelines typically prepare inputs with high-level library calls: decode an image, resize it with a particular interpolation method, convert its colour space, normalise it with particular means and standard deviations. Audio pipelines compute spectrograms with particular window sizes, hop lengths, and mel filterbanks, apply a logarithm, and perhaps normalise. On the device, those steps are reimplemented, often in C, often by a different engineer, often with a different library — and small differences abound. An image arrives in BGR rather than RGB order. A camera delivers YUV data and the conversion uses a slightly different matrix. Resizing uses nearest-neighbour rather than bilinear interpolation. An audio feature extractor uses a different window function, a different number of mel bands, or a natural logarithm where training used base ten. Each difference shifts the input distribution, and the model, which has never seen such inputs, degrades. Quantization adds a subtler variant. A fully integer model expects its inputs already quantized with the input tensor's scale and zero point, which are stored in the model. If the application quantizes inputs with different parameters — or, very commonly, feeds unsigned 8-bit pixel values to a model that expects signed 8-bit inputs — the model sees systematically shifted data. Three practices guard against this. Use the same pre-processing code, or a bit-exact port of it, in training and on the device; where possible, generate the device's feature extraction code from the same source, or move pre-processing into the model graph itself so that it is converted with everything else. Record golden test vectors — raw sensor inputs, the pre-processed tensors computed from them by the training pipeline, and the expected outputs — and check them on the device, comparing each stage separately. And include the device's own sensor in data collection as early as possible, so that the model is trained on data captured through the same microphone, lens, and analogue front end it will see in the field. Testing for equivalence Chapter 3 recommended a three-way comparison: floating-point model, quantized model in the desktop interpreter, quantized model on the device. The full chain suggests a more complete ladder of checks, each comparing adjacent stages on identical inputs. 1. Training framework, floating point, versus converted model, floating point. Differences should be at the level of floating-point rounding; anything larger indicates a conversion problem. 2. Converted floating point versus converted integer, on the desktop interpreter. This measures quantization loss, and should be checked on the full validation set. 3. Desktop integer versus device integer, CPU only. For a correct integer implementation this should be bit-exact or nearly so; differences indicate kernel bugs, different library versions, or integration errors. 4. Device CPU versus device with accelerator. Small differences are possible where the accelerator uses different rounding; large ones indicate a compiler or driver issue. 5. Golden sensor input versus device pre-processing. Run the device's feature extraction on recorded raw input and compare with the training pipeline's output. Automating these comparisons, so that each new model version runs through them before release, is one of the highest-value investments an edge AI team can make. The checks are cheap; the failures they catch are expensive. Profiling where the time goes A single end-to-end latency figure tells the developer whether the model meets its deadline, but not why it does or does not. For that, per-operator profiling on the target is essential. On microcontrollers, LiteRT for Microcontrollers includes a profiler interface that records the time spent in each operator invocation, and Cortex-M processors provide a hardware cycle counter that can time any section of code to the cycle. On Android and embedded Linux, LiteRT's benchmark tool runs a model on the device for a fixed number of iterations, with warm-up runs, and reports average latency and per-operator breakdowns with or without delegates. ExecuTorch provides developer tools that record execution traces on the device and map them back to the original PyTorch operators. Apple's Xcode generates performance reports for Core ML models showing which compute unit each layer ran on and how long it took. Per-operator profiles routinely reveal surprises. A handful of operators dominate; often one or two layers account for half the latency. An operator that was expected to run on the accelerator is on the CPU. A reshape or transpose inserted by the converter is costing more than the convolution beside it. Pre-processing, running outside the model, takes longer than the model. Each of these suggests a specific fix, and none is visible from end-to-end timing alone. Measuring energy For battery-powered devices, energy must be measured, not estimated. The standard method is to power the device through an instrument that records current at high time resolution — a dedicated power analyser or one of the inexpensive current-profiling tools common in embedded development — and to integrate current times voltage over an inference to obtain energy. The device should toggle a pin at the start and end of inference so that the trace can be aligned with the computation. Measurements should cover the whole cycle: waking from sleep, capturing sensor data, pre-processing, inference, post-processing, any radio activity, and returning to sleep. Inference alone is often not the largest consumer. A model that halves inference energy but requires the processor to stay awake longer for pre-processing may save nothing. Benchmarks and what they are for Public benchmarks offer a common reference point for comparing hardware and software stacks. The most relevant for microcontroller-class devices is MLPerf Tiny, developed under MLCommons by a consortium led by Colby Banbury, Vijay Janapa Reddi, and colleagues from more than fifty organisations, and first described in 2021.3 It defines four tasks with fixed datasets, reference models, and accuracy thresholds, and measures latency and, optionally, energy on submitted systems. Table 5 lists its tasks. Table 5. The MLPerf Tiny benchmark tasks. Task Dataset Reference model Size Quality target Keywords Speech Commands DS-CNN 52.5 KB 90% top-1 Person Visual Wake Words, 96×96 MobileNetV1 325 KB 80% top-1 Images CIFAR-10 ResNet 96 KB 85% top-1 Anomaly ToyADMOS FC autoencoder 270 KB 0.85 AUC Source: Banbury et al., "MLPerf Tiny Benchmark" (NeurIPS Datasets and Benchmarks, 2021), Table 1. The reference models are instructive in themselves. They show the scale of model that serves real tasks on microcontrollers: tens to a few hundred kilobytes of weights, input resolutions far below what server models use, and accuracy targets set at what the application needs rather than what a leaderboard rewards. Published results from the benchmark's rounds are a useful starting point for comparing hardware platforms on comparable workloads. They are not a substitute for measuring the developer's own model. Benchmark submissions are heavily tuned; the reference models use operators that every platform supports well; and a system that leads on keyword spotting may be ordinary on a model with different operators or memory behaviour. Benchmarks narrow the candidate list. Measurements of the actual model on the actual hardware make the decision. Fitting it into firmware and apps A few integration details recur often enough to note. On microcontrollers, the model is usually compiled into the firmware as a constant byte array, placed in flash, and aligned as the runtime requires; misalignment produces crashes that look like model bugs. The tensor arena must be sized by measurement — the runtime can report how much of the arena was actually used after a test inference — with a margin for future model updates. Over-the-air update schemes that keep two firmware images double the flash required for the model, and some products store the model in a separate flash partition so that it can be updated independently of the application. On phones, models are often too large to ship inside the application package and are downloaded on first use, which raises questions of versioning, integrity checking, and compatibility between model versions and application versions. Accelerator compilation caches should be kept between launches so that cold-start costs are paid once. Pin the toolchain, version the model Every tool in the chain changes from release to release, and those changes can alter the deployed model. A new converter version may fuse operators differently, choose a different implementation of an activation function, change the default quantization scheme for some operator, or produce a model that an older on-device runtime cannot read. A new accelerator compiler may tile layers differently, changing both latency and, occasionally, numerical results. A team that regenerates its model with whatever tool versions happen to be installed will eventually ship a model that differs from the one it tested, without anyone having changed the model. The defence is the same as for any other build artefact: make it reproducible. Pin the exact versions of the training framework, converter, quantization tools, and backend compiler used to produce each released model, ideally in a container image that can be rebuilt later. Record the calibration data set, by content hash rather than by file path. Embed metadata in the model file itself — a version string, the hash of the source checkpoint, the toolchain versions, and the input pre-processing parameters — so that a device in the field can report exactly which model it is running and a developer can reconstruct how it was produced. LiteRT's flatbuffer format supports arbitrary metadata for this purpose, and the other formats have equivalents. Runtime and model must also stay compatible. On a microcontroller, the runtime is compiled into the firmware, so a new model that uses an operator the deployed firmware did not register, or relies on a newer version of an operator's kernel, will fail to initialise on devices in the field. Firmware should check a model's required operator set and schema version before switching to it, and keep the previous model available if the check fails. On phones, where the runtime is part of the application or the operating system, the same check belongs in the code that downloads models. None of this is glamorous, and none of it appears in papers. It is, nonetheless, where edge AI projects succeed or fail. The final chapter assembles everything — budget, architecture, compression, hardware, toolchain, and measurement — into a single pipeline, and walks it end to end. CHAPTER 9 A Complete Pipeline: From Requirement to Fielded Model The preceding chapters examined techniques one at a time. Real projects do not encounter them that way. They encounter a product requirement, a device, a deadline, and a stream of decisions whose order matters as much as their content. This chapter walks through two projects end to end — one on a microcontroller, one on a phone — to show how the pieces fit together and where projects typically go wrong. The projects are composites, built from the patterns that recur across many real deployments; the numbers in their budgets are design choices of the kind a team would make, not measurements of any particular product. Project one: a battery-powered person detector The requirement: a small, battery-powered sensor mounted in a corridor or doorway must report whether a person is present, for building occupancy analytics. It must run for at least two years on its battery, must never transmit images, and must report presence within about one second of a person entering its view. The device is a microcontroller with a Cortex-M55 core, an Ethos-U55 micro-NPU configured with 128 multiply-accumulate units, around 1.5 megabytes of SRAM, several megabytes of flash, a low-power monochrome image sensor, and a low-energy radio. Step one: write the budget Following Chapter 1, the team writes the budget before touching a model. For memory, the firmware — real-time kernel, radio stack, sensor driver, update logic, and the inference runtime — is estimated at a few hundred kilobytes of flash, and the update scheme keeps two images, so the model's weights are allowed half a megabyte with margin. For SRAM, the frame buffer for a low-resolution greyscale image, the radio's buffers, and the stack consume a known share; the tensor arena is allowed 400 kilobytes. For time, one second end to end is generous for inference, but energy turns it into a tighter constraint. The battery's capacity and the two-year target give an average power budget, measured in tens of microwatts. Sleep current consumes much of it. Each wake-capture-infer-report cycle must therefore cost little enough that the expected number of cycles per day fits within what remains. That arithmetic immediately shapes the design. Running the neural network continuously at several frames per second would blow the energy budget many times over. The device needs a cascade: a cheap trigger that runs always, and the neural network only when the trigger fires. The team chooses a hardware motion-detection mode in the image sensor, which compares successive low-resolution frames internally and raises an interrupt when enough pixels change, with the microcontroller asleep in between. When woken, the device captures frames and runs the network on a few of them before returning to sleep. The neural network's job is now to distinguish people from the other things that cause motion — doors swinging, light changes, a cleaning robot, a passing reflection. Step two: collect data through the device The team's first data collection uses the actual sensor, lens, and mounting height, not a public dataset. Public person-detection datasets such as Visual Wake Words are useful for pre-training, but they consist mostly of colour photographs taken at eye level in good light, while the device sees greyscale frames from a ceiling-height angle, often in dim corridors with fluorescent flicker. A few dozen prototype units installed in representative buildings for several weeks capture frames whenever the motion trigger fires. This produces exactly the distribution the network must handle: what the trigger sends it. Labelling is the bottleneck. Following Chapter 6, the team uses a large, accurate person detector, far too big for any edge device, to label the captured frames automatically, and has people review a sample — particularly the frames where the large model was uncertain — to correct errors and estimate label quality. Privacy rules for the collection are agreed with the building owners in advance; the prototype units are configured for data collection only, with signage and retention limits, and the production devices will never store or transmit images. The team also sets aside slices of the evaluation data: night-time frames, frames with partial occlusion, frames where a person is far from the sensor, frames with flicker. Overall accuracy can hide failures concentrated in one slice, and the product's credibility depends on all of them. Step three: choose and size the architecture The team selects a MobileNet-style architecture, as recommended in Chapter 2, and checks two constraints before training anything. First, operator support. A candidate network is instantiated with random weights, converted to an 8-bit LiteRT model with a few dummy calibration samples, and passed through Vela. The report shows every operator mapped to the NPU except the final softmax, which is moved out of the model and computed trivially in application code. It also shows that one candidate using a hard-swish activation in an early layer is placed on the NPU through a lookup table, while another variant's unusual pooling configuration falls back to the CPU; the latter is changed. Second, size. The team trains small proxy models at three input resolutions — 64, 96, and 128 pixels square — and three width multipliers, measuring both accuracy on the evaluation slices and, through Vela's estimates and early on-device runs, peak SRAM and latency. The highest resolution helps markedly on the "person far from sensor" slice, where a distant figure occupies only a few pixels at lower resolutions, and costs too much memory at the largest width. The chosen design runs at 128 by 128 with a moderate width, at roughly 300 kilobytes of int8 weights and a peak activation footprint comfortably inside the arena. That choice — driven by a slice the team would not have examined without collecting its own data — is the most consequential of the project. Step four: train well, then quantize The chosen architecture is pre-trained on a public person dataset converted to greyscale, then fine-tuned on the team's device-captured data. Fine-tuning uses distillation from the large detector's soft per-frame person score, combined with the reviewed labels, with consistent augmentation for teacher and student — random crops, brightness and contrast changes, simulated sensor noise, and synthetic flicker — and a long schedule, as Chapter 6 recommends. Distillation improves accuracy on the night-time slice most, where the teacher's graded confidence carries information that hard labels lose. Quantization follows Chapter 3: full-integer post-training quantization with per-channel weights, calibrated on several hundred frames drawn from every slice, including the darkest. The first attempt, calibrated on a random sample dominated by daytime frames, loses noticeably more accuracy at night than during the day; recalibrating with a deliberately balanced set fixes most of it. The remaining loss is small enough that quantization-aware training is not needed. The team checks the converted model on the full evaluation set, slice by slice, in the desktop interpreter. Step five: validate on the device The validation ladder from Chapter 8 runs next. Golden test vectors — raw sensor frames, the training pipeline's pre-processed tensors, and the expected integer outputs — are loaded into a test build of the firmware. The ladder catches two problems. The first is in pre-processing: the sensor delivers frames in a slightly different crop from the one the data collection firmware used, because a register setting differs between builds, so the production model sees a marginally zoomed view. The difference is small, but it shifts accuracy on the distant-person slice, and it would have been nearly impossible to diagnose from field reports. The second is in input quantization: the application converts pixels from unsigned to signed integers by subtracting 128, as the model expects, but one code path skips the subtraction after a sensor reset. Both bugs are fixed before any production unit ships. With the ladder clean, the team profiles. The NPU portion of the network takes a few tens of milliseconds; the per-operator profile shows that the image-scaling routine, running on the CPU before inference, takes almost as long. Rewriting it to use the Cortex-M55's Helium vector instructions halves it. The team then measures energy for the whole wake cycle with a current-profiling instrument, toggling a pin at each stage. Inference is not the largest consumer; waking and stabilising the image sensor is. Capturing two frames instead of four per wake, with temporal agreement required between them, saves more energy than any further model optimization could. Step six: decide, smooth, and ship A network outputs a score; the product needs a decision. The team chooses a threshold from the evaluation data by deciding which error is worse — missing a person or reporting a phantom one — and setting the threshold to meet the product's tolerance for each. They add simple temporal logic: presence is reported when two consecutive frames exceed the threshold, and absence only after a period without detections, suppressing flicker in the reported state. This logic is tested on recorded sequences, not just individual frames. Step seven: monitor and update Shipping is not the end. Models degrade in the field when the world they see differs from the world they were trained on — a new building with unusual lighting, a change of carpet, a seasonal shift in daylight. Since the devices must never transmit images, the team designs privacy-preserving telemetry: histograms of the network's output scores, counts of motion triggers and of positive detections per hour, and battery voltage. A shift in the score distribution, or a rise in the ratio of triggers to detections, flags a site for investigation, where — with the building owner's agreement — a separate data-collection unit can be installed temporarily. Model updates travel over the air in the dedicated model partition, signed, versioned, and checked against the firmware's runtime version. They are rolled out to a small fraction of devices first, with telemetry compared against the previous model before wider release, and the previous model retained for rollback. The flash budget set in step one, with its allowance for two images, is what makes this possible. Project two: an on-device language model in a phone app The second project is a phone application that summarises and categorises incoming messages locally, so that their content never leaves the device. The requirement is a short summary within a couple of seconds and a category label, on recent mid-range and flagship phones, without noticeable battery drain or heat. The budget is bandwidth and memory Chapter 1's arithmetic dominates. Generating each output token requires reading all of the model's weights, so the generation rate is bounded by memory bandwidth divided by weight bytes. The application can also claim only a modest share of the phone's memory before the operating system starts closing other applications or, worse, the app itself. Both point to the same conclusion: the fewer bytes the weights occupy, the better on every axis. The team considers models of around one, two, and four billion parameters. At four bits per weight, these occupy roughly half a gigabyte, one gigabyte, and two gigabytes. The largest leaves too little headroom on mid-range phones and generates too slowly to meet the latency target on them. The team chooses a model in the one-to-two-billion range for which the developer has published a quantization-aware-trained 4-bit checkpoint, following Chapter 4's advice to prefer such checkpoints to post-training quantization when they exist. Memory also goes to the key-value cache, which stores intermediate attention results for every token processed so far. Its size grows with the number of layers, the attention dimensions, and the length of the context, and for long inputs it can rival the weights. The team caps input length — long message threads are truncated to their most recent portion — and stores the cache at reduced precision, a supported option in their chosen runtime. Make the small model good at one job A general small model summarises adequately but categorises inconsistently, and its summaries are longer than the interface can show. Following Chapter 6, the team generates a large set of synthetic training examples with a much larger model — realistic messages, with high-quality summaries of the required length and correct category labels — and fine-tunes the small model on them, using parameter-efficient fine-tuning so that the adaptation adds only a small set of extra weights. The fine-tuned model is then re-quantized to 4 bits, and evaluated on held-out real messages contributed with consent by internal testers, slice by slice — short messages, long threads, multiple languages, messages containing mostly links or numbers. Run it where it runs best, and measure it hot The runtime choice follows from the platforms. On iOS, the team uses Core ML and checks Xcode's performance report to confirm which parts run on the Neural Engine. On Android, it uses a runtime with a GPU backend and vendor NPU backends where they have been validated, and a well-optimised CPU path everywhere else, because the device population includes chipsets from several vendors. Measurement is done the way Chapter 1 warned it must be: not on a cool phone straight after boot, but after several minutes of sustained use, with the application's interface active. The first measurements show that the NPU path on one chipset is fast for the first few summaries and then slows markedly as the device warms and throttles. The team schedules summarisation for messages as they arrive in small batches while the phone is idle and charging where possible, and limits the number of interactive summaries generated back to back. Cold-start compilation for the accelerator, several seconds on first launch, is moved to a background task after installation. Fail gracefully Not every phone can run the model well. The application measures a short benchmark on first launch and, on phones below a threshold, falls back to the category label alone, generated by a small classifier distilled from the language model — a model of a few megabytes that runs in milliseconds on any device. The classifier is the fruit of the same distillation pipeline and costs almost nothing to maintain alongside the main model. What the two projects share The two projects differ by several orders of magnitude in model size, yet the sequence of decisions is the same. The budget comes first, and it is written in bytes, milliseconds, and joules, measured on the real device in realistic conditions. Data comes from the deployment environment, through the deployment sensor, and is evaluated in slices that reflect where the product could fail. The architecture is chosen and sized against that budget and checked against the target's operator support before any expensive training. Distillation from a larger model makes the small model better than its size suggests. Quantization is applied with representative calibration and evaluated on the converted model, and heavier techniques are reserved for when lighter ones fail. Deployment is validated stage by stage against golden data, profiled per operator, and measured for energy across the whole cycle, where the largest costs often turn out to lie outside the network. And the model is monitored and updated in the field, with the budget having reserved room for that from the beginning. The specific techniques that mattered most were not the most sophisticated ones. In the first project, the decisive choices were the input resolution, driven by a slice of the data, and the number of frames captured per wake, driven by an energy measurement. In the second, they were the bytes per weight and the decision to specialise a small model for one task. Pruning played no role in either, because neither target could exploit it profitably, and quantization-aware training was needed only where a vendor had already done it. That pattern is typical, and it is the practical meaning of this booklet's controlling idea: the work that pays is the work that respects the memory budget of the actual device and checks itself against that device at every step. Conclusion: Engineering for the Device in Front of You The techniques in this booklet are often presented as a menu: quantize, prune, distil, accelerate, choose any combination. Having worked through each of them against the budget of real devices, a different picture emerges. They are not equals, and they are not interchangeable. They form a hierarchy, ordered by how reliably each one reduces the resource that edge hardware lacks most — bytes stored and bytes moved — on the hardware a developer actually has. At the top of that hierarchy sit two decisions that are not compression techniques at all. The first is the choice of architecture and its size, above all the input resolution and the width of the network, which together set the activation peak that decides whether a model fits a microcontroller and the memory traffic that decides how fast it runs on anything. The second is how well the chosen network is trained, which is where knowledge distillation earns its place: it is the one technique that reliably makes a small model better rather than trading its accuracy for size. Then comes 8-bit integer quantization, which cuts storage and traffic fourfold, unlocks integer accelerators, and costs little accuracy for a well-designed network. Below that sit the conditional techniques: low-bit quantization where memory bandwidth dominates, structured pruning where a network is over-sized, unstructured and semi-structured sparsity where the hardware can exploit them. And underneath all of it sits the work that is never published — matching pre-processing, validating the chain stage by stage, profiling per operator, measuring energy across the whole wake cycle — without which none of the rest can be trusted. What follows for practice Several practical consequences follow from that hierarchy, and they are worth stating as plainly as possible because they run against common habits. Start on the device, not in the notebook. The single most effective change most teams can make is to get a trivial model running on the target hardware in the first week — before collecting serious data or training anything good. A random-weight network passed through the real toolchain, deployed in the real firmware or application, and measured with the real profiler exposes operator gaps, memory limits, pre-processing pipelines, and integration problems while they are still cheap to fix. Projects that reach the device last discover their constraints last. Write the budget in the units that bind. Parameter counts and operation counts are convenient, but they are proxies. The budget that governs a deployment is written in bytes of flash and SRAM, milliseconds of sustained latency, and joules per inference. Every architectural and compression decision should be justified in those units, measured on the target. Spend effort in order. Before reaching for quantization-aware training, sparsity, or custom kernels, ask whether the input resolution is larger than the task needs, whether the network is wider than the budget allows, whether the model has been trained with a strong teacher, and whether post-training quantization has been given representative calibration data. Most of the available improvement usually lies in those questions, and they are the cheapest to answer. Treat the accelerator as a target, not a feature. An NPU accelerates the models built for it and quietly fails to accelerate the rest. Check its operator coverage and memory behaviour before the architecture is fixed, read its compiler's placement report for every model version, and judge it by measured latency and energy on your own network, never by its headline throughput. Evaluate the model you ship, where you ship it. Accuracy is a property of the converted integer model running on the device's kernels, fed by the device's sensor and pre-processing, measured across the slices of data where the product could fail. Any accuracy figure measured elsewhere is an estimate of that one. What remains unsettled The principles above are stable; several parts of the landscape are not. The numeric formats of edge inference are in flux. Eight-bit integers will remain the default for convolutional models on microcontrollers for years, but below that the industry has not settled. Integer formats at four bits and below, block-scaled formats in which groups of values share an exponent — such as the microscaling formats standardised through the Open Compute Project — and low-precision floating-point variants are all competing, with different hardware supporting different subsets. A developer targeting several devices today may need several quantized versions of the same model, and the toolchains for producing and validating them are less mature than those for 8-bit integers. Transformers on microcontroller-class hardware are at an early stage. Accelerators such as the Ethos-U85 have added the operators they require, but the memory demands of attention, the awkwardness of softmax and layer normalisation in integer arithmetic, and the scarcity of well-tested small transformer architectures for sensor tasks mean that convolutional and recurrent networks remain the safer choice for the smallest devices. That balance is likely to shift, but it has not shifted yet. Toolchain fragmentation remains the largest practical cost in the field. Each accelerator has its compiler, each phone vendor its delegate, each runtime its operator set, and a model that runs well through one path may fail or crawl through another. Recent consolidation — LiteRT's acceptance of models from several frameworks, ExecuTorch's growing set of production backends — helps, but developers targeting diverse hardware should still expect to spend real time on per-platform validation. Finally, the line between inference and learning on the device is beginning to blur. Personalising a model to a single user's voice or a single machine's vibration signature, and adapting to drift without sending data off the device, are increasingly practical. They raise their own questions of memory, energy, and validation, and they are beyond the scope of this booklet, but they will draw on the same discipline: the device's budget comes first. The device in front of you Edge AI is sometimes described as a matter of shrinking models until they fit. That description misses what makes the work succeed. The developers who ship good edge models are not, in the main, those with the most elaborate compression pipelines. They are those who know their device's budget precisely, design within it from the start, train small models as carefully as large ones, apply the simple and reliable techniques before the clever ones, and measure everything on the hardware that will run it. Those habits make intelligence a property of the device itself — available without a network, answering within a deadline, keeping private data private, and running for years on a battery. That is the promise of running AI at the edge, and it is kept or broken in the bytes. Notes Chapter 1 1. Pete Warden and Daniel Situnayake, TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers (O'Reilly Media, 2019). 2. Ji Lin, Wei-Ming Chen, Yujun Lin, John Cohn, Chuang Gan, and Song Han, "MCUNet: Tiny Deep Learning on IoT Devices," Advances in Neural Information Processing Systems 33 (NeurIPS 2020). arXiv:2007.10319. 3. Google AI Edge, "TensorFlow Lite is now LiteRT," Google Developers Blog, 4 September 2024; and LiteRT for Microcontrollers documentation, ai.google.dev/edge/litert/microcontrollers. 4. Song Han, Jeff Pool, John Tran, and William J. Dally, "Learning both Weights and Connections for Efficient Neural Networks," Advances in Neural Information Processing Systems 28 (NeurIPS 2015), arXiv:1506.02626, reproducing Mark Horowitz's 45 nm energy table; see also Mark Horowitz, "Computing's Energy Problem (and What We Can Do About It)," IEEE International Solid-State Circuits Conference (ISSCC), 2014. Chapter 2 1. Andrew G. Howard et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications," arXiv:1704.04861, 2017. 2. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 3. Andrew Howard et al., "Searching for MobileNetV3," IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 4. Danfeng Qin et al., "MobileNetV4: Universal Models for the Mobile Ecosystem," European Conference on Computer Vision (ECCV), 2024. arXiv:2404.10518. 5. Lin et al., "MCUNet" (as Chapter 1, note 2); and Ji Lin, Wei-Ming Chen, Han Cai, Chuang Gan, and Song Han, "MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning," NeurIPS 2021. 6. Aakanksha Chowdhery, Pete Warden, Jonathon Shlens, Andrew Howard, and Rocky Rhodes, "Visual Wake Words Dataset," arXiv:1906.05721, 2019. 7. Liangzhen Lai, Naveen Suda, and Vikas Chandra, "CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs," arXiv:1801.06601, 2018. 8. Arm, "Arm Ethos-U85: Addressing the High Performance Demands of IoT in the Age of AI," Arm Newsroom, 9 April 2024. Chapter 3 1. Benoit Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference," CVPR 2018. arXiv:1712.05877. 2. LiteRT 8-bit quantization specification, ai.google.dev/edge/litert (formerly tensorflow.org/lite/performance/quantization_spec). 3. Raghuraman Krishnamoorthi, "Quantizing Deep Convolutional Networks for Efficient Inference: A Whitepaper," arXiv:1806.08342, 2018. 4. Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling, "Data-Free Quantization Through Weight Equalization and Bias Correction," ICCV 2019. 5. Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort, "Up or Down? Adaptive Rounding for Post-Training Quantization," International Conference on Machine Learning (ICML), 2020. Chapter 4 1. Yoshua Bengio, Nicholas Léonard, and Aaron Courville, "Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation," arXiv:1308.3432, 2013. 2. Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha, "Learned Step Size Quantization," International Conference on Learning Representations (ICLR), 2020. 3. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer, "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale," NeurIPS 2022. 4. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han, "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models," ICML 2023. 5. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers," ICLR 2023. 6. Ji Lin et al., "AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration," Proceedings of Machine Learning and Systems (MLSys), 2024. Chapter 5 1. Han et al., "Learning both Weights and Connections for Efficient Neural Networks" (as Chapter 1, note 4). 2. Song Han, Huizi Mao, and William J. Dally, "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding," ICLR 2016. arXiv:1510.00149. 3. Asit Mishra et al., "Accelerating Sparse Deep Neural Networks," arXiv:2104.08378, 2021. 4. Arm, "Arm Ethos-U85" (as Chapter 2, note 8). 5. Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan, "Fast Sparse ConvNets," CVPR 2020. 6. Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf, "Pruning Filters for Efficient ConvNets," ICLR 2017. 7. Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag, "What Is the State of Neural Network Pruning?" MLSys 2020. 8. Michael Zhu and Suyog Gupta, "To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression," arXiv:1710.01878, 2017. 9. Jonathan Frankle and Michael Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks," ICLR 2019. Chapter 6 1. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, "Distilling the Knowledge in a Neural Network," arXiv:1503.02531, 2015. 2. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio, "FitNets: Hints for Thin Deep Nets," ICLR 2015. 3. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf, "DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter," arXiv:1910.01108, 2019. 4. Jang Hyun Cho and Bharath Hariharan, "On the Efficacy of Knowledge Distillation," ICCV 2019. 5. Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov, "Knowledge Distillation: A Good Teacher Is Patient and Consistent," CVPR 2022. 6. Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew Gordon Wilson, "Does Knowledge Distillation Really Work?" NeurIPS 2021. Chapter 7 1. Lai, Suda, and Chandra, "CMSIS-NN" (as Chapter 2, note 7). 2. Arm announcement of the Cortex-M55 and Ethos-U55, February 2020, as reported in "Cortex-M Gets Heavy-Duty Machine Learning," Electronic Design, 11 February 2020. 3. Arm, "Arm Ethos-U85" (as Chapter 2, note 8). 4. PyTorch, "Introducing ExecuTorch 1.0," PyTorch Blog, 22 October 2025. Chapter 8 1. Robert David et al., "TensorFlow Lite Micro: Embedded Machine Learning for TinyML Systems," Proceedings of Machine Learning and Systems (MLSys), 2021. 2. PyTorch, "Introducing ExecuTorch 1.0" (as Chapter 7, note 4). 3. Colby Banbury et al., "MLPerf Tiny Benchmark," NeurIPS Datasets and Benchmarks Track, 2021. arXiv:2106.07597. Further Reading Pete Warden and Daniel Situnayake, TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers (O'Reilly Media, 2019).** The book that named the field. Its project-by-project approach to keyword spotting, person detection, and gesture recognition on microcontrollers remains an excellent hands-on introduction, even though some tool names have since changed. Daniel Situnayake and Jenny Plunkett, AI at the Edge: Solving Real-World Problems with Embedded Machine Learning (O'Reilly Media, 2023).** A broader, workflow-oriented companion that covers problem framing, data collection, evaluation, and responsible deployment for edge AI products rather than individual techniques. Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer, Efficient Processing of Deep Neural Networks (Morgan & Claypool, 2020).** The standard reference on how neural network accelerators are designed, with a rigorous treatment of dataflow, data reuse, and why memory access dominates energy. Essential for anyone who wants to understand what an NPU does with a model. Markus Nagel et al., "A White Paper on Neural Network Quantization" (arXiv:2106.08295, 2021). A clear, practical account of post-training quantization and quantization-aware training from the Qualcomm research group behind several of the techniques in this booklet, with recommended pipelines and debugging advice. Amir Gholami et al., "A Survey of Quantization Methods for Efficient Neural Network Inference" (arXiv:2103.13630, 2021). A thorough survey of quantization schemes, granularities, and training methods, useful for placing any given technique in context. Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste, "Sparsity in Deep Learning: Pruning and Growth for Efficient Inference and Training in Neural Networks," Journal of Machine Learning Research 22 (2021).** A long and careful survey of pruning methods, sparsity patterns, and the hardware conditions under which sparsity yields real speedups. Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao, "Knowledge Distillation: A Survey," International Journal of Computer Vision 129 (2021).** A structured overview of response-based, feature-based, and relation-based distillation and the training schemes that combine them. Vijay Janapa Reddi et al., Machine Learning Systems (open online textbook, mlsysbook.ai).** A freely available textbook from Harvard that treats machine learning as a systems discipline, with substantial material on efficient models, optimization, hardware acceleration, and on-device deployment. Hashtags: #EdgeAI #OnDeviceAI #EmbeddedAI #MobileAI #InternetOfThings #IoT #TinyML #NeuralNetworkOptimization #ModelCompression #DeepLearning #EfficientNeuralNetworks #ModelQuantization #PostTrainingQuantization #QuantizationAwareTraining #INT8Inference #INT4Quantization #NeuralNetworkPruning #StructuredPruning #KnowledgeDistillation #HardwareAwareOptimization #NeuralProcessingUnits #MobileNet #MCUNet #TensorFlowLite #LiteRT #ExecuTorch #MicrocontrollerAI #LowPowerAI #EdgeInference #AIModelDeployment

Download the Book (PDF): Introduction Pick up a small quadcopter, power it on, and take your hands off the sticks without arming the flight controller's stabilization, and it will not fly. It will not even fail gracefully. Within a fraction of a second of leaving the ground, one side will produce slightly more thrust than the other, the airframe will tilt, the tilted thrust will push it sideways, and nothing in the physics of the machine will act to bring it back. A paper aeroplane has more natural stability than a quadcopter. The machine that inspects wind turbines, maps quarries, films weddings and carries blood samples between hospitals is, in its raw mechanical form, an object that wants to fall over. Everything that makes a multirotor useful is therefore software. The four motors and four propellers are only actuators; the aircraft is the control system. A few hundred times a second, a small processor reads a gyroscope, decides how wrong the airframe's rotation is, and changes the speed of each motor by a few percent to correct it. Above that loop sits another one that cares about angle rather than rotation rate, above that one that cares about velocity, above that one that cares about position, and above that the mission logic that decides where the aircraft ought to be at all. Each layer trusts the one beneath it to do its job, and each layer is only as good as that trust is justified. That layered structure is the controlling idea of this booklet. A quadcopter flight controller is a stack of nested feedback loops, each running faster than the one above it, each hiding the messy dynamics of the layer below behind a simpler promise. The inner rate loop promises: "tell me how fast to rotate and I will rotate that fast." The attitude loop promises: "tell me what angle to hold and I will hold it." The position controller promises: "tell me where to be and I will get there smoothly." Waypoint navigation, terrain following, obstacle avoidance and the rest of autonomy are all built by issuing requests to these promises. When a drone oscillates, drifts, overshoots a waypoint, or flies confidently into a tree, the cause can almost always be traced to a specific layer whose promise was broken, and the fix lives in that layer, not the one where the symptom showed up. Understanding the stack this way turns a bewildering list of parameters into a map. ArduPilot, the open-source autopilot this booklet uses as its main example, exposes well over a thousand configurable parameters. Newcomers meet them as a wall of cryptic names: ATC_RAT_RLL_P, INS_HNTCH_FREQ, PSC_VELXY_P, WPNAV_SPEED, EK3_SRC1_POSXY. Read as a stack, they sort themselves. The ATC_RAT family belongs to the innermost loop. The INS family shapes the sensor data before any loop sees it. PSC is the position controller. WPNAV shapes the paths that the position controller is asked to follow. EK3 is the estimator that tells all of them where the aircraft believes it is. Knowing which layer a parameter lives in tells you what it can affect, what it cannot, and which other parameters it depends on. What this booklet covers The book moves up the stack from the bottom. It begins with the physics, because a control system cannot be understood apart from the thing it controls: how four fixed-pitch propellers produce lift, roll, pitch and yaw; why the reaction torque of spinning propellers is the only handle on yaw; and why the system is unstable in the first place. It then turns to sensing. The "gyroscopic stabilization" of a modern drone involves no spinning gyroscope at all, but a vibrating silicon structure a few millimetres across, and the difference matters, because it determines what the sensor gets wrong and how badly vibration corrupts it. From raw sensor data the controller must estimate attitude, and the chapter on sensing explains the filters that do this, from the simple complementary filter to the extended Kalman filters that modern autopilots run. Two chapters are given to the PID controller, because it is the heart of the inner loops and because tuning it is the single most common practical problem pilots and builders face. The first explains what proportional, integral and derivative terms actually do, why the textbook form of PID needs several modifications before it can fly an aircraft, and how ArduPilot arranges its controllers in a cascade. The second is a practical guide to tuning: filtering vibration before touching gains, using logs rather than impressions, recognising the signatures of too much and too little of each term, and understanding what automated tuning can and cannot do for you. The middle of the book climbs to navigation. Knowing where the aircraft is involves satellite navigation, barometers, magnetometers and optical sensors, all fused by an estimator that must constantly decide which of them to believe. Getting from one place to another involves turning a list of waypoints into a smooth trajectory that respects the aircraft's limits on speed, acceleration and jerk. Avoiding obstacles along the way involves sensors that see the world, algorithms that decide what to do about what they see, and a hard-earned understanding of where those algorithms fail. The last chapters deal with the software itself and with safety. ArduPilot is a large, mature codebase with a hardware abstraction layer, a cooperative scheduler, a simulator that lets you fly the real flight code on a laptop, an embedded scripting engine, and a standard protocol, MAVLink, for talking to ground stations and companion computers. Understanding how these fit together is what lets a developer extend the system without breaking the loops that keep it in the air. Finally, a chapter on failsafes and regulation treats safety as an engineering discipline: what the aircraft should do when a sensor lies, a battery sags, or a radio link drops, and what the rules in the United States and Europe expect of the person who sets it flying. Scope and assumptions The focus is the multirotor, and specifically the quadcopter, because it is the dominant civilian platform and the clearest example of a machine that exists only through its control system. Hexacopters and octocopters follow the same principles with more motors to mix. Fixed-wing aircraft and VTOL hybrids appear only in passing; their control problems are different enough to deserve their own treatment. ArduPilot is the main reference platform because it is open, extensively documented, widely used in survey, inspection, agriculture and research, and because its parameter names give concrete handles to abstract ideas. PX4, the other major open-source autopilot, appears where comparison sheds light, and Betaflight, the firmware of choice for acrobatic first-person-view racing, appears where its very different priorities illuminate a tuning question. The principles in this book apply to all of them; the parameter names do not. Where a parameter is named, it is an ArduPilot Copter parameter unless otherwise stated, and readers should check the documentation for their firmware version, since names and defaults change between releases. The applications in view are civilian: hobby flying, aerial photography, mapping and surveying, infrastructure inspection, agricultural spraying, search and rescue support, delivery trials, and academic research. The booklet assumes a reader comfortable with high-school physics and willing to read a few lines of code, but it does not assume a background in control theory. Every concept from that field is introduced when it is first needed, in words rather than equations wherever words will do. One practical warning belongs at the start. Everything described here can be tried on a simulator before it is tried on hardware, and should be. ArduPilot's software-in-the-loop simulator runs the same flight code that runs on the aircraft, and a mistake made in simulation costs nothing. A mistake made with real propellers spinning at tens of thousands of revolutions per minute can cost a finger. The habit of removing propellers before working on a powered aircraft on the bench, and of testing every change in the simulator first, is the most valuable thing any reader can take from this book, and it is worth adopting before reading any further. Chapter 1: Four Propellers and an Unstable Machine A helicopter controls itself by changing the pitch of its rotor blades as they spin. A swashplate tilts the whole rotor disc to steer, collective pitch raises and lowers the total lift, and a tail rotor cancels the torque of the main rotor. It is a mechanically intricate solution, full of linkages, bearings and hinges, and it took decades of engineering to make reliable. The quadcopter replaces all of it with arithmetic. Four fixed-pitch propellers, bolted directly to four electric motors, are the only moving parts. Every manoeuvre the aircraft makes, from a gentle hover to an aggressive flip, is produced by changing the speeds of those four motors relative to one another. This simplicity is why multirotors became cheap and ubiquitous in the 2010s, once three technologies matured at the same time: small brushless motors with fast electronic speed controllers, lithium-polymer batteries with enough energy density to fly for useful periods, and microelectromechanical (MEMS) inertial sensors cheap enough to put in phones. None of these was new, but together they made it possible to build a machine whose mechanical design was trivial and whose control was entirely electronic. The price of that trade is that the quadcopter has no mechanical stability whatsoever. To understand why the flight controller must work so hard, and what it is working against, it helps to look closely at how four spinning propellers produce motion. Thrust, torque and the four controls A propeller is a rotating wing. As it spins, each blade meets the air at an angle and accelerates a column of air downward; by Newton's third law, the air pushes the propeller up. For a fixed-pitch propeller in still air, the thrust it produces is, to a good approximation, proportional to the square of its rotational speed. Double the speed and thrust roughly quadruples. The constant of proportionality depends on the propeller's diameter, pitch, blade shape and the density of the air, which is why the same aircraft needs more throttle to hover on a hot day at a high-altitude site than on a cold morning at sea level. The same blades that push air down also drag on the air as they turn, and that drag produces a torque on the motor. The air resists the propeller's rotation, and by reaction the motor and the frame it is attached to are pushed in the opposite direction. This reaction torque is also roughly proportional to the square of the speed. On a single-rotor helicopter, it would spin the fuselage, which is why the tail rotor exists. On a quadcopter, the solution is to spin two propellers clockwise and two counter-clockwise. In the common X configuration, motors on one diagonal turn one way and motors on the other diagonal turn the other way. When all four run at the same speed, their reaction torques cancel and the frame does not spin. From these two facts, thrust and reaction torque, come the four basic controls. Total thrust, usually called throttle, is changed by speeding up or slowing down all four motors together. If thrust exceeds the aircraft's weight, it climbs; if thrust falls short, it descends. Roll is produced by speeding up the motors on one side and slowing those on the other. More thrust on the left than on the right creates a torque about the aircraft's front-to-back axis, and it rolls to the right. Because one clockwise and one counter-clockwise motor sit on each side, the reaction torques remain balanced and the aircraft does not yaw while it rolls. Pitch works the same way about the side-to-side axis: speed up the rear motors and slow the front ones, and the nose drops. Yaw is the subtle one. To rotate the aircraft about its vertical axis, the controller speeds up the two motors that spin in one direction and slows the two that spin in the other. Total thrust stays the same, so the aircraft does not climb or descend, and the thrust remains symmetric across the frame, so it does not roll or pitch. But the reaction torques no longer cancel, and the net torque turns the frame. A quadcopter yaws by exploiting the very effect its counter-rotating layout was designed to cancel. The yaw handle is weak. Reaction torque is a small fraction of the thrust-induced torques that produce roll and pitch, so a quadcopter yaws far more sluggishly than it rolls. This is why yaw authority is the first thing to run out when an aircraft is overloaded or flying at its limits, and why flight controllers typically limit how much of the motors' range the yaw controller is allowed to use. If roll and pitch need the motors' full range to stay level, yaw must give way. A drone that spins uncontrollably when pushed hard has usually run out of yaw authority, not suffered a software failure. The mixer The flight controller does not think in terms of individual motors. Its attitude and rate controllers produce four abstract demands: a roll torque, a pitch torque, a yaw torque and a total thrust. A component called the mixer translates these into four motor outputs. For an X quadcopter, the arithmetic is simple. Each motor's output is the throttle demand plus or minus the roll demand, plus or minus the pitch demand, plus or minus the yaw demand, with the signs determined by where the motor sits and which way it spins. The front-right motor, for instance, gets less output for a right roll (it is on the side that must drop), less output for a nose-down pitch (the front must drop), and more or less for yaw depending on its spin direction. ArduPilot encodes these factors for each supported frame type in a table inside its motors library, and selecting a frame class and type in the configuration chooses the right table. The same abstraction lets the same controller fly a hexacopter, an octocopter, a Y6 with coaxial motor pairs, or a dozen other layouts. The controller asks for torques; the mixer works out which motors to change. Real mixers must deal with saturation, and this is where the arithmetic stops being trivial. Each motor can only run between a minimum and a maximum. If the throttle is high and the roll controller asks for a large correction, the motors on the high side may hit their ceiling while the others still have room. A naive mixer would clip the saturated motors and produce less roll torque than requested, with a side effect of altering yaw and thrust. ArduPilot's mixer instead prioritises: it preserves roll and pitch first, sacrificing yaw, and if necessary it reduces total thrust to make room for attitude corrections. The logic is that an aircraft which briefly loses some altitude while keeping itself level is recoverable, whereas one that holds altitude while tumbling is not. ArduPilot logs a flag when the motors are saturated in this way, and a log showing frequent saturation is a sign that the aircraft is underpowered for what it is being asked to do. The relationship between motor command and thrust is itself not linear. Electronic speed controllers translate a command into a voltage applied across the motor windings, the motor's speed depends on that voltage, its load and the battery's state of charge, and thrust depends on the square of speed. ArduPilot exposes a parameter called MOT_THST_EXPO that approximates this curve so the mixer can linearise it, along with MOT_SPIN_MIN and MOT_SPIN_MAX for the usable range and battery voltage compensation that scales outputs as the pack sags during a flight. Getting these right matters more than most builders expect. A controller tuned with a linearised thrust curve behaves consistently from hover to full throttle; one that is not will feel crisp at one throttle and sloppy or twitchy at another, and no amount of PID adjustment will fix it everywhere at once. Thrust margin How much spare thrust an aircraft has is the single physical number that most constrains its control. Builders usually express it as a thrust-to-weight ratio: the total thrust all motors can produce at full power, divided by the aircraft's weight. A ratio of two means the aircraft hovers at roughly half its maximum thrust. Because thrust rises with the square of motor speed, half thrust corresponds to roughly seventy percent of maximum speed, and since the relationship between throttle command and speed is not linear either, the hover throttle a log reports will not simply equal the fraction of thrust in use. ArduPilot learns the actual value in flight and stores it as MOT_THST_HOVER, which the altitude controller then uses as its baseline. The margin above hover is what the controller spends on everything else. To tilt and accelerate sideways, the aircraft must increase total thrust so that the vertical component still balances weight; at a 30-degree lean, it needs about 15 percent more thrust than in hover, and at 45 degrees about 41 percent more. To correct a roll, it must raise the motors on one side, and if those motors are already near their limit, the correction is lost. To climb, it needs thrust beyond weight. And to fight a gust, all of these may be needed at once. An aircraft with a thrust-to-weight ratio near two has generous room for all of them. One loaded close to a ratio of 1.5, as heavy survey and spraying aircraft often are at takeoff, hovers at about two-thirds of its maximum thrust and has much less room, so its controller saturates more readily, its maximum safe lean angle is lower, and its tune must be gentler. Racing quadcopters, at the other extreme, may exceed a ratio of eight, hovering near the bottom of their throttle range where small changes in command produce large changes in thrust, which brings its own tuning difficulties. Thrust margin also falls during a flight. As the battery discharges, its voltage drops and the maximum speed each motor can reach falls with it, so the same aircraft that had a ratio of two on a fresh pack may have noticeably less near the end. Voltage compensation keeps the controller's response consistent while there is margin to compensate with, but it cannot create thrust the motors cannot produce. This is one reason a heavy aircraft that handles well at the start of a flight can become sluggish and prone to saturation in wind toward the end of it, and why battery failsafe thresholds, discussed in the last chapter, should be set with control margin in mind as well as flight time. Why it is unstable A conventional aeroplane is designed to be stable. Its centre of gravity sits ahead of the wing's aerodynamic centre, its tailplane produces a restoring moment when the nose pitches up or down, and its wings are often set at a slight dihedral angle so that a sideslip rolls it back towards level. Disturb a well-designed aeroplane and, left to itself, it will drift back toward its trimmed condition. Pilots can fly them hands-off for a while. A hovering quadcopter has none of this. Suppose it hovers perfectly level and a gust tilts it five degrees to the right. Its thrust vector is now tilted too, so part of the thrust pushes it sideways. Nothing in the airframe resists the tilt: there is no tailplane, no dihedral, no pendulum effect worth mentioning, because the thrust acts at roughly the same height as the centre of mass. The aircraft stays tilted, accelerates sideways, and begins to lose altitude because less of its thrust now points up. Any further disturbance adds to the tilt rather than being cancelled. In the language of control theory, the attitude dynamics of a hovering multirotor behave like a double integrator: torque produces angular acceleration, angular acceleration integrates into angular rate, and rate integrates into angle. With no natural restoring force, small errors accumulate. The translational motion is equally unforgiving. Tilt produces horizontal acceleration, acceleration integrates into velocity, and velocity into position. So the chain from a motor speed change to a change in position runs through four integrations: torque to rotation rate, rate to angle, angle (through tilted thrust) to acceleration, and acceleration to velocity and position. Each integration is a delay and an accumulation, and that chain is exactly why the control system is built as a stack of nested loops. Rather than trying to command position directly through four layers of integration, the flight controller closes a fast loop around rotation rate, a slower one around angle, and slower ones still around velocity and position. Each loop sees a plant that is close to a single integrator, which is far easier to control than four in a row. A small amount of aerodynamic damping does exist. When a quadcopter moves through the air, the advancing side of each propeller meets faster air than the retreating side, which produces a phenomenon called blade flapping that tilts the effective thrust vector slightly back against the motion. Propellers also experience a form of drag that rises with forward speed, sometimes called rotor drag or induced drag, which acts a little like friction. These effects matter to high-performance estimators and to research on aggressive flight, but they are far too weak to stabilise the aircraft. For practical purposes, the quadcopter flies only because its controller makes it. Gyroscopic effects and what "stabilization" means The phrase "gyroscopic stabilization" is often used loosely about drones, and it is worth being precise about it. A spinning propeller is a small gyroscope, and spinning gyroscopes resist changes in the direction of their spin axis. When the aircraft pitches, each propeller experiences a gyroscopic torque at right angles to the pitch, a precession effect that tries to roll the aircraft slightly. On a quadcopter with two propellers spinning each way at similar speeds, these gyroscopic torques largely cancel. On aircraft with large, heavy propellers, or during violent manoeuvres where the motor speeds differ greatly, they become noticeable couplings between axes that a good controller must reject. But they do not stabilise the aircraft. A quadcopter's propellers are far too light, and their axes are aligned with the very motion the aircraft must make, for gyroscopic stiffness to do anything useful. The stabilization in a drone comes from a different device that happens to share the name: the rate gyroscope, a sensor that measures how fast the airframe is rotating. It has no spinning parts, and the next chapter explains how it works. The point here is that the stabilization is active, not passive. It consists of measuring rotation and responding to it with motor commands, hundreds or thousands of times a second. Remove the sensor, or slow the loop down, and the aircraft becomes unflyable within moments. How fast must the loop run? The answer depends on how quickly the aircraft can rotate and how quickly disturbances arrive. A small, light racing quadcopter with powerful motors can change its rotation rate in a few milliseconds; a heavy survey hexacopter with large propellers responds more slowly, because large propellers have more rotational inertia and take longer to change speed. The motors themselves are a limit: a brushless motor and propeller cannot change speed instantaneously, and the time constant of that response, typically tens of milliseconds for a small aircraft and longer for a large one, bounds how fast the whole system can react. ArduPilot's main loop runs at 400 hertz by default on Copter, and the fastest inner rate loop can be configured to run faster still on capable hardware. Betaflight, aimed at small agile aircraft, commonly runs its gyro sampling and PID loop in the multi-kilohertz range. Both choices reflect the same principle: the control loop must be several times faster than the fastest dynamics it needs to correct, or it will always be reacting to a situation that has already changed. What the physics asks of the software The physics of a quadcopter thus sets the agenda for everything that follows. The aircraft is unstable, so it needs feedback. The chain from motor to position runs through several integrations, so the feedback should be arranged in nested loops. Yaw authority is weak and motors saturate, so the mixer must prioritise. Thrust is non-linear and varies with battery voltage and air density, so the outputs must be linearised and compensated. Motors and propellers take time to respond, so the loops must be fast and the controller must anticipate rather than merely react. And the aircraft responds to disturbances so quickly that the sensor feeding the inner loop must be fast, clean and trustworthy. That last requirement turns out to be the hardest to meet. A quadcopter is a vibrating machine. Four motors spinning at thousands of revolutions per minute, each turning a propeller that is never perfectly balanced, shake the frame at frequencies that sit uncomfortably close to the bandwidth the control loop needs. The sensor that tells the controller how the aircraft is rotating is also, unavoidably, a vibration sensor. Separating the motion the controller needs to see from the vibration it must ignore is the subject of the next chapter, and it is the foundation on which every later layer rests. Chapter 2: Sensing Rotation and Estimating Attitude The innermost loop of a flight controller needs one number above all others: how fast the aircraft is rotating about each of its three axes, right now. The attitude loop above it needs a second: which way the aircraft is pointing, relative to the ground and to gravity. The first comes almost directly from a sensor. The second has to be estimated, because no single sensor on a small drone measures attitude directly and reliably. This chapter covers both, starting with the device at the centre of it all. The MEMS gyroscope The gyroscope in a modern flight controller is a microelectromechanical system, a structure of silicon etched at the scale of micrometres and packaged in a chip a few millimetres across. Common parts in flight controllers come from manufacturers such as TDK InvenSense, Bosch and STMicroelectronics, and a single package usually holds a three-axis gyroscope and a three-axis accelerometer together, an arrangement called an inertial measurement unit, or IMU. The gyroscope has no spinning rotor. Instead, a tiny proof mass is driven to vibrate back and forth along one axis at a fixed frequency, typically in the tens of kilohertz. When the chip rotates, the Coriolis effect acts on the moving mass and pushes it sideways, perpendicular to both its vibration and the axis of rotation. The sideways displacement is proportional to the rotation rate, and it is detected as a change in capacitance between the mass and fixed electrodes nearby. Electronics on the chip demodulate this signal, filter it, and present the rotation rate as a digital number, typically in degrees per second, over a serial bus to the flight controller's processor. Three properties of this sensor shape everything the flight controller does with it. The first is that it measures rate, not angle. To know the aircraft's attitude from the gyroscope alone, the controller must integrate the rate over time, summing up small rotations to track the total. This works well over short periods but drifts over long ones, because any small constant error in the measured rate, called bias, accumulates into a growing angle error. A bias of a fraction of a degree per second, quite normal for a consumer MEMS part, becomes tens of degrees of error in a minute. Bias also changes with temperature, which is why some flight controllers heat their IMUs to a fixed temperature and why ArduPilot can perform temperature calibration of its inertial sensors. The second is that the sensor is fast and low-noise in the band the controller cares about. Over a few milliseconds, the gyroscope is by far the most trustworthy source of information about rotation on the aircraft. This is why the innermost loop runs directly on gyroscope data with minimal processing. The third, and the one that causes the most trouble, is that the gyroscope responds to vibration. A MEMS gyroscope is itself a vibrating mechanical structure, and it has limited ability to distinguish the rotation it is meant to measure from mechanical shaking of the board it is soldered to. More importantly, the frame genuinely does rotate slightly with each vibration cycle, and the gyroscope faithfully reports that rotation. From the controller's point of view, the airframe is rocking back and forth hundreds of times a second at the motor rotation frequency and its harmonics. That motion is real, but it is not motion the controller can or should correct, and if the controller tries, it will drive the motors back and forth in step with the noise, heating them and amplifying the very vibration it is trying to cancel. There is an extra trap. A digital sensor samples its signal at a fixed rate. Any vibration at a frequency higher than half the sampling rate does not simply disappear; it appears in the samples as a false, lower-frequency signal, an effect called aliasing. A vibration at 1,050 hertz sampled at 1,000 hertz looks exactly like a 50 hertz oscillation, which is squarely in the band the controller acts on. Once aliased, the false signal cannot be removed by any later filter, because it is indistinguishable from real motion. This is why modern IMUs sample internally at several kilohertz and apply anti-aliasing filters before the data leave the chip, why flight controller designers care about the IMU's internal configuration, and why physical vibration isolation of the flight controller and good propeller balance remain important even with sophisticated digital filtering. The rule is that vibration should be prevented mechanically where possible, measured honestly, and filtered digitally only after it has been sampled fast enough to be seen for what it is. The accelerometer and the magnetometer The gyroscope tells the controller how fast it is turning. To know which way is down, the controller needs a reference that does not drift, and on a small aircraft the obvious candidate is gravity. A MEMS accelerometer is a proof mass on a spring, again etched in silicon, whose displacement under acceleration is measured capacitively. Three of them at right angles measure the acceleration of the chip along each axis. At rest, an accelerometer reads one gravity pointing up, because the sensor is being held up against gravity, and the direction of that reading tells the controller which way is down. If the aircraft is tilted, the gravity vector appears tilted in the sensor's frame, and the tilt angle can be computed from simple trigonometry. The catch is that an accelerometer cannot distinguish gravity from any other acceleration. During a hover it reads gravity. During a turn, a climb, a braking manoeuvre or a gust, it reads gravity plus the aircraft's own acceleration, and the computed "down" direction swings away from the true vertical. Worse, a multirotor in steady flight is an object whose thrust always points along its own vertical axis. In a coordinated, unaccelerated tilt the accelerometer mostly sees thrust along the body axis and drag, and the information it carries about tilt is weaker than intuition suggests. Accelerometers are also even more sensitive to vibration than gyroscopes, since vibration is acceleration. They are a reliable reference for "down" only when averaged over seconds, not milliseconds. For heading, the controller needs a reference for which way is north. A three-axis magnetometer measures the direction of the Earth's magnetic field, and once the aircraft's tilt is known, the horizontal component of that field gives magnetic heading. Magnetometers are the most temperamental sensors on a drone. The Earth's field is weak, and the currents flowing to the motors, which can be tens of amps, create magnetic fields of their own that change with throttle. Steel structures, reinforced concrete, power lines and buried pipes distort the local field. For these reasons the compass is usually mounted on a mast with the GPS receiver, away from the power wiring, and ArduPilot provides both a calibration routine that corrects for fixed distortions caused by the airframe itself and a separate compensation for motor current interference. Even so, compass errors remain one of the commonest causes of poor navigation, and later chapters return to them. Fusing the sensors The three sensors have complementary strengths. The gyroscope is excellent over short timescales and drifts over long ones. The accelerometer is noisy and corrupted by manoeuvres over short timescales but gives a stable reference for down over long ones. The magnetometer does the same for heading, with extra vulnerabilities. Combining them so that each covers the others' weaknesses is the problem of attitude estimation. The simplest workable answer is the complementary filter. It integrates the gyroscope to track attitude, and at every step nudges the result a small amount toward the attitude implied by the accelerometer (and magnetometer, for heading). The size of the nudge sets a crossover time. Faster than that, the estimate follows the gyroscope; slower than that, it follows gravity. A crossover of a second or two lets the estimate ride smoothly through short manoeuvres while still slowly correcting gyroscope drift. Mathematically, it is a high-pass filter on the gyroscope's integrated angle added to a low-pass filter on the accelerometer's angle, with the two filters chosen so their responses sum to one at every frequency, which is where the name comes from. The complementary filter is cheap, intuitive and good enough for a surprising range of aircraft. Early hobby flight controllers used versions of it, and refinements remain in wide use. One influential form, the nonlinear complementary filter on the rotation group published by Robert Mahony, Tarek Hamel and Jean-Michel Pflimlin in 2008, adds an integral term that estimates and removes the gyroscope bias rather than merely nudging the angle, and it works directly with rotations rather than Euler angles to avoid their singularities. Another widely implemented approach, described by Sebastian Madgwick in 2010, uses a gradient-descent step to align the measured gravity and magnetic field vectors with their expected directions. ArduPilot's original attitude estimator, known as the DCM algorithm after the direction cosine matrix it uses to represent attitude, belongs to this family, and it still runs in the background today as a fallback. The complementary filter's limitation is that its trust in each sensor is fixed. It cannot recognise that the accelerometer is unreliable during a hard turn and lean on the gyroscope harder, or that a GPS fix has become available and could help separate gravity from the aircraft's own acceleration. For that, the controller needs a filter that maintains not only an estimate but a measure of how uncertain the estimate is. The extended Kalman filter A Kalman filter keeps track of a set of quantities, called the state, together with a matrix describing how uncertain each quantity is and how the uncertainties are correlated. At each step it does two things. It predicts: using a model of how the system evolves, it moves the state forward in time and grows the uncertainty to reflect what the model does not know. Then it corrects: when a measurement arrives, it compares what the sensor reported with what the state predicted it would report, and blends the difference into the state in proportion to how uncertain the prediction was relative to how noisy the sensor is. A confident prediction and a noisy sensor yield a small correction; an uncertain prediction and a precise sensor yield a large one. The weighting is recomputed at every step, so the filter's trust in each sensor changes as conditions change. The classic Kalman filter assumes the system is linear. Attitude and navigation are not, because rotations combine non-linearly and the relationship between position, velocity and sensor readings depends on the attitude. The extended Kalman filter, or EKF, handles this by linearising the model around the current estimate at each step. It is an approximation, and it can misbehave if the estimate strays far from the truth, but for the gentle non-linearities of a drone's motion it works well and has been the workhorse of aircraft navigation for decades. ArduPilot's current estimator, EKF3, is an extended Kalman filter with a state of around two dozen quantities. It tracks attitude, represented as a quaternion to avoid the singularities that Euler angles suffer at ninety degrees of pitch; velocity and position in a local north-east-down frame; the biases of the gyroscopes and accelerometers; the Earth's magnetic field and the aircraft's own magnetic offsets; and wind velocity, which it can estimate when an airspeed sensor or, on multirotors, a configured model of the airframe's drag gives it something to work with. The gyroscope and accelerometer drive the prediction step at high rate. GPS position and velocity, barometric altitude, magnetometer readings, and optionally optical flow, rangefinders, visual odometry and wheel encoders are fused as corrections when they arrive. Because the filter carries the accelerometer bias and the aircraft's velocity in its state, it can separate gravity from manoeuvring acceleration far better than a complementary filter can, provided it has an external velocity reference such as GPS. Two engineering details make EKF3 robust in practice. First, sensors report with delay; a GPS fix may describe where the aircraft was a hundred or more milliseconds ago. Fusing it as if it were current would corrupt the estimate. EKF3 therefore runs on a delayed time horizon, fusing each measurement at the moment it actually describes, and uses a separate fast complementary filter to project the delayed estimate forward to the present for the controllers. Second, every measurement is checked before it is fused. The filter computes how surprising each reading is, given its own uncertainty, and rejects readings whose surprise exceeds a threshold. This innovation gating, as it is called, is what lets the estimator survive a brief GPS glitch or a magnetic disturbance without being dragged off course, and the size of the recent innovations is what ArduPilot reports as the EKF's health. When innovations grow persistently large, the EKF failsafe triggers, a mechanism discussed in the last chapter. ArduPilot can run several instances of the EKF at once, typically one per IMU on boards that carry two or three, and switch between them if one diverges or its sensor fails. This redundancy is one reason serious survey and inspection aircraft fly controllers with multiple IMUs, often of different models and sometimes on separately isolated mountings, so that a vibration or a failure that afflicts one is unlikely to afflict all. What the controllers actually receive It is worth being clear about what each loop sees, because confusion here is a common source of tuning mistakes. The innermost rate loop sees gyroscope data, filtered by a low-pass filter and by notch filters aimed at motor noise, at the full loop rate. It does not use the EKF's output for its feedback, because the EKF is too slow and too smoothed for a loop that must respond in milliseconds. It does use the EKF's estimate of gyroscope bias, which is subtracted from the raw readings. The attitude loop sees the attitude estimate, which comes from the EKF's quaternion propagated to the present. Its job is to compare that estimate with the desired attitude and produce a desired rotation rate for the rate loop. The position and velocity loops see the EKF's position and velocity estimates. They are slow enough that the EKF's latency is acceptable, and they depend completely on its accuracy. A position controller with perfect gains will still wander if the position estimate is wrong. This separation explains why some problems look like tuning problems but are not. An aircraft whose rate loop oscillates has a gain or filtering problem, almost always in the rate controller or in the vibration that feeds it. An aircraft that holds attitude well but drifts slowly in position or "toilet bowls", circling in a growing spiral around the point it is meant to hold, usually has an estimation problem, very often a compass heading error that makes the position controller push in slightly the wrong direction. No change to the rate PID gains will cure toilet-bowling, and no amount of compass calibration will cure a twitchy roll. Knowing which layer owns the symptom is the first step to fixing it, and the next two chapters turn to the layer that most often needs attention: the PID controllers at the bottom of the stack. Chapter 3: The PID Controller and the Cascade The proportional-integral-derivative controller is the most widely used feedback algorithm in engineering. It regulates the temperature of chemical reactors, the speed of conveyor belts, the pressure in gas pipelines and the cruise speed of cars. Its theory was worked out in the early twentieth century, notably by Nicolas Minorsky, who analysed automatic ship steering in a paper published in 1922, and its practical tuning was codified by John Ziegler and Nathaniel Nichols in 1942. It is used so widely not because it is optimal for anything in particular but because it is simple, robust and adequate for an enormous range of problems, and because its three terms correspond to three intuitive ideas about how to correct an error. A quadcopter's inner loops are PID controllers, or close relatives of them. The textbook PID, however, would make a poor flight controller. This chapter explains what each term does, why each needs modification before it can fly, and how ArduPilot arranges its controllers into a cascade so that each one faces a problem it can actually solve. Three ways to correct an error Every feedback controller starts with an error: the difference between what is wanted, called the setpoint or target, and what is measured. For the roll rate controller, the target might be "rotate right at 30 degrees per second" and the measurement "currently rotating right at 20 degrees per second", for an error of 10 degrees per second. The controller's job is to produce an output, here a roll torque demand to the mixer, that reduces the error. The proportional term is the obvious response: push in proportion to how wrong things are. Output equals a gain, P, times the error. A larger error produces a larger correction. Proportional control alone is often surprisingly good, but it has two weaknesses. If the gain is too low, the response is sluggish and the system lags behind its target. If the gain is too high, the correction overshoots: by the time the aircraft has responded, the controller has pushed too hard, the error has reversed, and the controller pushes back too hard the other way. With enough gain, and with the delays inherent in motors, propellers and filters, this becomes a sustained oscillation. There is always a ceiling on P, and the ceiling is set by how much delay the loop contains. The integral term addresses a different problem: persistent error. Suppose the aircraft's centre of gravity is slightly off to one side, so holding a steady roll rate of zero requires the motors on one side to run slightly faster than those on the other. A pure proportional controller can only produce that offset if there is an error to multiply, so the aircraft must always be slightly wrong to generate the correction it needs. The integral term accumulates the error over time and adds a correction proportional to the running total. As long as any error persists, the integral keeps growing, until it supplies exactly the offset needed and the error falls to zero. Integral action is what makes a controller precise in the long run. Its cost is that it responds slowly, and because it remembers past errors, it can keep pushing after the error has reversed, contributing its own overshoot and oscillation if its gain is too high. The derivative term looks at how fast the error is changing and pushes against that change. If the error is shrinking rapidly, the derivative term reduces the output in anticipation of arriving, which damps overshoot. It acts like a shock absorber: it resists motion, not position. Derivative action allows a higher proportional gain to be used without oscillation, which is why it is valuable in flight control. Its weakness is noise. Differentiating a signal amplifies its high-frequency content, so any vibration in the measurement becomes a much larger fluctuation in the derivative term, which then drives the motors. Most of what goes wrong with derivative gain in practice is noise amplification, not control theory. These intuitions can be put compactly. P responds to the present error, I to the accumulated past error, and D to the predicted future error. Table 1 summarises the typical effect of increasing each gain, the trade-off it introduces, and the failure that appears when it is pushed too far. Table 1. Typical effects of raising each PID gain in a multirotor rate loop. Gain Main benefit Main cost Sign of excess P Faster, tighter response Less stability margin Fast oscillation after stick input I Removes steady offsets, holds attitude against disturbances Slower to settle, can wind up Slow wobble, overshoot that drifts back D Damps overshoot, allows more P Amplifies noise Hot motors, high-pitched buzz, jitter FF Crisp response to commands without extra feedback Does nothing for disturbances Overshoot on stick input only What the textbook form gets wrong A PID controller written straight from its definition would cause several problems on an aircraft, and each has a standard fix. The first is derivative kick. If the derivative is taken of the error, then any sudden change in the target, such as a pilot snapping the stick from centre to full deflection, produces an instantaneous spike in the derivative and a violent jolt in the output. One common remedy is to take the derivative of the measurement alone rather than the error, so that the derivative term reacts to how the aircraft is actually moving and ignores sudden changes in what is wanted. Another, which ArduPilot uses, is to keep the derivative on the error but pass both the target and the error through low-pass filters, with separate filter frequencies for target, error and derivative (ATC_RAT_RLL_FLTT, FLTE and FLTD for roll), and to shape the target upstream so it never steps instantaneously. The derivative filter frequency is one of the most important parameters in the whole tuning process, because it sets the balance between derivative usefulness and noise amplification. The second is integrator windup. If the motors are saturated, the controller cannot reduce the error no matter what it demands, but the integral term keeps accumulating. When the saturation finally ends, the integrator holds an enormous stored correction that drives a large overshoot. Practical controllers limit how large the integral can grow, stop integrating while outputs are saturated, and often reset or relax the integrator while the aircraft is on the ground, since it is meaningless to accumulate attitude corrections against the reaction of the landing gear. ArduPilot provides an integrator limit for each axis (ATC_RAT_RLL_IMAX and its equivalents) and suppresses integration when the mixer reports saturation. It also deliberately relaxes the integrators before takeoff, because a drone sitting on uneven ground can otherwise accumulate a correction that tips it over the moment it leaves. The third is that pure feedback is always late. A feedback controller only acts once an error exists, so when the pilot or the navigation code requests a new rotation rate, the controller does nothing until the error appears, and then it must build its output from the error. Feedforward fixes this by adding a term proportional to the target itself, not the error. If the target is "rotate at 100 degrees per second", feedforward immediately produces approximately the output known to give that rate, and the feedback terms only need to correct what feedforward gets wrong. ArduPilot exposes this as ATC_RAT_RLL_FF and its counterparts. Feedforward does nothing for disturbances like gusts, which arrive without a change in target, so it complements feedback rather than replacing it. Betaflight's feedforward works on the rate of change of the pilot's stick, which suits its racing audience's demand for instant response to rapid stick movement. The fourth is that the plant's gain varies. The torque a given output produces depends on battery voltage, air density, and where on its thrust curve each motor is operating. ArduPilot compensates for voltage sag and for the thrust curve in the mixer, as described in Chapter 1, so that the PID gains see a roughly constant plant across the flight. Without this, a tune that was perfect on a fully charged battery would grow sluggish as the pack discharged. Betaflight offers a similar compensation and, in its own way, a mechanism that varies gains with throttle. The general principle is gain scheduling: when the plant changes predictably, change the controller with it. The cascade ArduPilot's attitude control is not one PID loop but a cascade of them, each wrapped around the one below. Understanding the arrangement is essential to tuning it. At the bottom are three rate controllers, one each for roll, pitch and yaw. Each is a full PID with feedforward, operating on gyroscope rate at the main loop rate. Its target is a desired rotation rate in degrees per second; its output is a torque demand to the mixer, normalised to the range of the motors. These are the controllers whose gains appear as ATC_RAT_RLL_P, ATC_RAT_RLL_I, ATC_RAT_RLL_D and so on for pitch (PIT) and yaw (YAW). Above them sit three angle controllers, one per axis. These are much simpler: each is a proportional controller only, with gain ATC_ANG_RLL_P for roll. The attitude error, the difference between the desired and estimated attitude, is multiplied by this gain to produce a desired rotation rate, which becomes the target for the rate controller below. Why no integral or derivative term? Because the rate controller below already handles disturbances and steady offsets, and because the integration from rate to angle means a proportional angle controller wrapped around a well-tuned rate loop already behaves like a well-damped system. Adding I or D at this level would mostly add complexity and new ways to oscillate. The attitude error itself is computed using quaternions, not by subtracting Euler angles, so the controller takes the shortest rotational path from the current attitude to the desired one even when the aircraft is far from level. Above the angle controller, but still within the attitude control library, is input shaping. When a pilot moves a stick or the navigation code requests a new attitude, the request is not passed straight to the angle controller as a step. Instead it is shaped into a smooth trajectory, limited by a maximum angular acceleration (ATC_ACCEL_R_MAX for roll, with equivalents for pitch and yaw) and smoothed by an input time constant (ATC_INPUT_TC). The angle controller then tracks a target that moves in a way the aircraft can physically follow, and the shaped rate of that target is fed forward directly to the rate controller. This arrangement is the core of ArduPilot's crisp but smooth feel: the commanded motion is planned to be achievable, the feedforward path delivers most of it immediately, and the feedback loops only clean up the residue. Above the attitude controller sits the position controller, AC_PosControl, with its own cascade: a proportional position controller produces a desired velocity, a PID velocity controller produces a desired acceleration, and the desired acceleration is converted into a desired lean angle and a thrust demand, which become the targets for the attitude controller. Its parameters are prefixed PSC_: PSC_POSXY_P for horizontal position, PSC_VELXY_P, I and D for horizontal velocity, and PSC_POSZ_P, PSC_VELZ_P and PSC_ACCZ_P, I and D for the vertical axis, which has an extra acceleration loop because vertical acceleration is controlled directly through thrust rather than through tilt. Seen whole, the stack looks like this. A navigation layer decides where the aircraft should be. The position controller turns that into a desired acceleration. The conversion from acceleration to lean angle and thrust turns that into a desired attitude. Input shaping turns the desired attitude into a smooth attitude trajectory with a matching rate feedforward. The angle controller turns the attitude error into a desired rate. The rate controller turns the rate error into a torque demand. The mixer turns torque demands into motor outputs. Each layer runs at a rate appropriate to its dynamics, and each relies on the layers below it. Bandwidth and separation The cascade works only if each inner loop is substantially faster than the loop wrapped around it. This principle, sometimes called time-scale separation, is the reason tuning must proceed from the inside out. An outer loop treats its inner loop as if it were perfect: when the angle controller asks for 50 degrees per second, it assumes it gets 50 degrees per second, immediately. That assumption is only approximately true, and it holds well enough only if the rate loop settles much faster than the angle loop needs. If the rate loop is sluggish, the angle loop's requests are fulfilled late, and the angle loop, seeing its corrections lag, behaves as if it had too much gain. A common rule of thumb in cascaded control is that each inner loop should be several times faster than the outer loop around it. This has direct practical consequences. If the rate loop is badly tuned, raising or lowering the angle P will not fix the aircraft's behaviour; it will only shift the symptoms. If the position loop oscillates, the cause may be a position gain set too high, but it may equally be an attitude loop too slow for the position gain to be safe. And if the estimator's position and velocity are noisy or delayed, the position loop's useful bandwidth is limited no matter how well the attitude loops perform. The position controller can never be faster than its information. It also explains why default gains for outer loops tend to transfer well between aircraft while rate gains do not. The rate loop deals directly with the physics of the particular airframe: its inertia, its motor response time, its propeller size, its vibration. The angle and position loops deal with an idealised plant made uniform by the loops beneath them. A well-tuned rate loop on a small inspection quadcopter and a well-tuned rate loop on a large spraying octocopter present roughly similar behaviour to the angle loop above, even though their rate gains may differ by a large factor. This is the payoff of the nested structure: once the bottom layer is right, the layers above are much more forgiving. Beyond PID Researchers have flown multirotors with a long list of more sophisticated controllers: linear quadratic regulators that compute optimal gains from a model, sliding-mode controllers that are robust to model error, geometric controllers that operate directly on the rotation group and remain well-behaved through extreme attitudes, incremental nonlinear dynamic inversion that uses measured angular acceleration to cancel unmodelled dynamics, and model predictive controllers that optimise a trajectory over a short future horizon at every step. Some of these have reached open-source autopilots in limited form, and they perform impressively in aggressive flight, in heavy wind and on unusual airframes. PID remains dominant in the inner loops of production civilian autopilots for reasons that are worth stating plainly. It needs no detailed model of the airframe, which matters when users build thousands of different aircraft. Its parameters have intuitive meanings that pilots can adjust by observation. Its failure modes are well understood, so a bad tune produces recognisable symptoms rather than mysterious behaviour. And wrapped in the modifications this chapter has described, filtered derivatives, anti-windup, feedforward, input shaping, thrust linearisation and voltage compensation, it performs well enough that the remaining gains from more advanced methods rarely justify their cost in complexity. The modified PID inside a modern autopilot is not the textbook controller; it is a carefully engineered descendant of it. Tuning that controller well is the subject of the next chapter. Chapter 4: Tuning in Practice Ask experienced builders how they tune a new aircraft and most will describe the same order of work, even if they express it differently: make the airframe mechanically sound, measure and filter its vibration, set sensible starting gains from its size, fly it gently and read the logs, then tune the rate loops, automatically or by hand, before touching anything above them. The order matters more than any individual technique. Most bad tunes come not from a wrong gain but from doing a later step before an earlier one, usually by raising and lowering PID gains to fight a problem that is actually vibration, or a thrust curve that was never set. This chapter follows that order. The examples use ArduPilot parameter names and tools, but the sequence applies to any autopilot. Before the first flight The mechanical state of the aircraft sets the ceiling on what any tune can achieve. Propellers should be balanced, undamaged and correctly seated; a chipped propeller can double the vibration level of an otherwise clean airframe. Motor bearings should turn smoothly. Arms should be rigid, because a flexible arm combined with a fast rate loop forms a resonant system that the controller can excite. The flight controller should be mounted so that its IMU is isolated from high-frequency vibration without being so soft that it wobbles relative to the frame at low frequencies. Many modern controllers include internal isolation for their IMUs, and adding a second soft mount on top can do more harm than good. Wires should be secured so they cannot slap against the frame or the controller. The frame must also be configured correctly: frame class and type, motor order and direction, and propeller direction on each motor. ArduPilot's motor test function in Mission Planner and other ground stations spins each motor individually, with propellers removed, to confirm that the motor that spins is the one the mixer expects and that it turns the right way. An incorrect motor order is one of the classic causes of a quadcopter flipping violently on its first attempt to take off, and the motor test exists so that no one has to discover it with propellers fitted. ArduPilot then needs a few physical facts about the aircraft before any loop is tuned. The battery cell count and voltage range set the voltage compensation. The thrust curve exponent, MOT_THST_EXPO, depends mainly on propeller size; smaller propellers generally use a lower value. The minimum spin and arming spin values, MOT_SPIN_MIN and MOT_SPIN_ARM, should be set so that all motors start reliably and never stall in flight. Mission Planner provides an initial parameter setup helper that takes propeller diameter, battery cell count and a few other values and computes reasonable starting points for the thrust curve, the gyro filter, the rate gains, and the angular acceleration limits. These starting values are not a tune, but they are close enough to fly safely on most aircraft, and they put the filters in roughly the right range so that the first flight produces usable data. The first flight should be short, gentle and close to the ground, in a stabilised mode such as Stabilize or AltHold, in a clear area with no one nearby. Its purpose is not to judge how the aircraft feels but to collect logs: a period of steady hover, a few gentle inputs on each axis, and a landing. Those logs drive every decision that follows. Filtering vibration first Vibration that reaches the rate controller is the most common obstacle to a good tune, and the most frequently misdiagnosed. Its signature is that the aircraft cannot tolerate as much P or D as it should, that motors run hot after short flights, and that the logged motor outputs look noisy even in a calm hover. The temptation is to reduce gains until the noise goes away. This works, after a fashion, but it leaves the aircraft sloppy in wind and slow to recover from disturbances. The better approach is to remove the noise from the gyroscope signal so that the gains can be set where the dynamics, rather than the vibration, require. Multirotor vibration is dominated by the motors. Each motor spins at a frequency set by its speed, and each propeller blade passing a given point adds energy at multiples of that frequency. A quadcopter hovering at a motor speed of, say, 6,000 revolutions per minute has a fundamental vibration at 100 hertz, with harmonics at 200, 300 and higher. Larger aircraft with larger propellers spin slower and vibrate at lower frequencies, which brings the noise uncomfortably close to the frequencies the rate controller must respond to. A general low-pass filter strong enough to remove 100 hertz noise from a large aircraft's gyro signal would also add enough delay to compromise the rate loop. The solution is a notch filter: a filter that removes a narrow band of frequencies around a centre while leaving everything else largely intact. Because the motor frequency changes as throttle changes, a fixed notch would be in the right place only at one throttle setting. ArduPilot's harmonic notch filter solves this by moving its centre frequency in flight, tracking the motor frequency and placing additional notches at its harmonics. The filter is enabled with INS_HNTCH_ENABLE, and INS_HNTCH_MODE selects how the frequency is tracked. In throttle-based mode, the filter estimates motor frequency from throttle output, using a reference frequency measured at hover (INS_HNTCH_FREQ) and the hover throttle (INS_HNTCH_REF). In ESC telemetry mode, it reads actual motor speeds reported by electronic speed controllers that support bidirectional DShot or a serial telemetry protocol, and can even run a separate notch for each motor. In FFT mode, the flight controller performs a running spectral analysis of the gyro data in flight and places the notch on whatever peak it finds. ESC telemetry is the most accurate where available, because it follows each motor's real speed rather than an estimate. To set the notch, the builder first flies with the filter disabled and logging enabled for raw gyro data. ArduPilot's batch sampler, controlled by INS_LOG_BAT_MASK and related parameters, records short bursts of high-rate IMU data that can be turned into a frequency spectrum. Tools such as the ArduPilot filter review tool on the project's website, and the FFT views in the UAV Log Viewer and Mission Planner, display the spectrum and show the peaks clearly: usually a dominant peak at the motor frequency and smaller ones at its harmonics, with a broad floor of noise below. The builder sets the notch frequency and bandwidth to cover the dominant peak, enables the filter, flies again, and checks that the peaks have been flattened in the post-filter data. An aircraft that has been properly notched will usually tolerate substantially higher rate gains than it did before, and its motors will run noticeably cooler. After the notch, the static low-pass filter on the gyro (INS_GYRO_FILTER) and the derivative filter in the rate controller (ATC_RAT_RLL_FLTD and its equivalents) can usually be set higher than would otherwise be safe, which reduces the delay in the loop and allows more aggressive control. This is the main reason vibration work comes first: it changes how much performance the loop can achieve before a single PID gain is touched. Reading the logs The single most useful habit in tuning is to make decisions from logs rather than impressions. An aircraft that "feels a bit twitchy" might have too much P, too much D, too little D, a noisy gyro, a loose propeller, or an overly aggressive input shaping, and the feel alone cannot tell them apart. The logs can. ArduPilot records a large amount of data to its onboard storage at each flight. The most important messages for tuning include the rate controller's targets and achieved rates, the PID terms themselves for each axis, the attitude targets and achieved attitudes, the motor outputs, the vibration metrics, and the IMU data. Plotting the desired roll rate against the actual roll rate during a series of stick inputs shows how faithfully the rate loop follows its target. A well-tuned loop tracks the target closely, with a slight lag and a small, well-damped overshoot at most. A loop with too little gain lags visibly and rounds off sharp inputs. A loop with too much gain overshoots and rings, oscillating briefly at a high frequency after each input. A loop with too much I overshoots and drifts back slowly. Vibration metrics are worth checking on every flight. ArduPilot reports vibration levels on each accelerometer axis and counts clipping events, which occur when an acceleration exceeds the sensor's measurement range. Clipping is serious because a clipped accelerometer reports a false average, and the EKF can be misled into a wrong estimate of vertical velocity. The ArduPilot documentation gives guidance on acceptable vibration levels, and a sudden rise in vibration between flights is a reliable sign that something mechanical has changed, such as a damaged propeller or a loose motor. The motor outputs tell their own story. In a calm hover, all four outputs should sit close together. A persistent offset between diagonal pairs indicates a yaw imbalance, often a motor mounted slightly twisted. A persistent offset between sides indicates an off-centre centre of gravity or a weak motor. Outputs that frequently touch their minimum or maximum indicate saturation, which may mean the aircraft is underpowered or that its tune is too aggressive for the available authority. Table 2 gathers the commonest symptoms seen in flight and in the logs, with the layer and the most likely cause for each. It is a starting point for diagnosis, not a substitute for looking at the data. Table 2. Common tuning symptoms and their usual causes. Symptom Where it shows Usual cause First thing to check Fast oscillation after inputs Rate target versus actual Rate P or D too high Reduce rate P and D together Hot motors, buzzing, noisy outputs Motor outputs, gyro spectrum Unfiltered motor noise Harmonic notch setup Slow wobble, overshoot that creeps back Attitude target versus actual Rate I too high or too low Rate I relative to P Sluggish response, rounded inputs Rate target versus actual Rate gains or feedforward too low Rate P, FF, acceleration limits Circling drift around a hold point Position track in Loiter Compass heading error Compass calibration and interference Bouncing in altitude hold Vertical velocity and throttle Vertical gains or hover throttle wrong Learned hover throttle, PSC_ACCZ gains Spinning under hard manoeuvres Yaw target versus actual Yaw authority exhausted Motor saturation, weight, motor alignment Automatic tuning Tuning rate loops by hand is slow and requires experience. ArduPilot provides AutoTune, a flight mode that does most of the work for the rate and angle loops. With the aircraft hovering in a large open area in calm air, the pilot switches to AutoTune, and the aircraft begins performing small, rapid twitches on one axis at a time, typically roll, then pitch, then yaw, as selected by AUTOTUNE_AXES. Each twitch is a step request to the rate or angle controller. AutoTune measures the response, adjusts the D gain upward until it sees a small bounce-back, then reduces it to a safe fraction; it does the same for P; and it then measures the maximum angular acceleration the aircraft can achieve, which it uses to set the acceleration limits and the angle P. The process takes several minutes and a substantial portion of a battery. The pilot can take control at any time by moving the sticks, during which AutoTune pauses and resumes when the sticks are released. When it finishes, the new gains are tested in flight and saved only if the pilot lands and disarms while still in AutoTune mode. AUTOTUNE_AGGR sets how aggressive the resulting tune will be, in effect how much bounce-back AutoTune will accept before backing off. Lower values give a crisper tune with less margin; higher values give a softer one. The default suits most aircraft. AutoTune works best on an aircraft whose vibration has already been filtered, whose thrust curve and hover throttle are correctly set, and which is flown in calm conditions. Wind confuses its measurements, because a gust looks to the algorithm like an unexpected response to its own input. AutoTune has limitations worth knowing. It tunes the rate and angle loops, not the position controller or the navigation layer. It can produce a tune that is too aggressive for very large or very flexible aircraft, where its step inputs excite structural modes. It can struggle with aircraft that have very little yaw authority. And it requires the pilot to keep the aircraft in a safe area while it twitches and drifts, which demands both space and attention. For these reasons, some builders of large aircraft prefer to tune manually or to start from AutoTune's result and refine it by hand. ArduPilot also offers QuickTune, a Lua script that tunes the rate loops by raising each gain until it detects the onset of oscillation and then backing it off by a set margin. It is faster than AutoTune and uses smaller motions, which suits aircraft where AutoTune's twitches are inconvenient. Like any automatic method, it assumes the earlier steps have been done: the frame is sound, the vibration filtered, the thrust curve set. Tuning by hand Manual tuning follows the same logic that AutoTune encodes. Starting from conservative gains, the builder raises the rate D gain on one axis in small steps until a fast oscillation appears after sharp inputs, then reduces it to perhaps half of that value. Then the rate P is raised in the same way, typically kept at or slightly above the value of I. The rate I is then set, in ArduPilot conventionally equal to P for roll and pitch as a starting point, and adjusted if the aircraft does not hold its attitude firmly in wind or drifts back slowly after inputs. Feedforward can be added for a crisper stick response. Only once the rate loops are satisfactory are the angle P values adjusted, and only once the attitude loop is satisfactory are the position controller and navigation speeds considered. Some pilots prefer to tune by injecting test signals rather than by stick inputs. ArduPilot supports a system identification mode that injects a frequency sweep, called a chirp, into a chosen axis and logs the response. From this, the frequency response of the aircraft can be computed offline, showing directly how much gain and phase margin the loop has at each frequency. This is the method control engineers use on full-size aircraft, and it gives a far richer picture of the loop than step responses alone, at the cost of more analysis. It is particularly useful for large, heavy or unusual airframes where the safety margin matters more than the speed of tuning. Whatever the method, a tune is complete only when it has been tested across the aircraft's envelope: with a full and a depleted battery, with and without payload, in calm and windy conditions, at hover and in fast forward flight. A survey aircraft that will carry a heavy camera should be tuned with that camera fitted. An aircraft whose weight changes during flight, such as an agricultural sprayer that empties its tank, needs a tune that is stable at both ends of the range, which may mean accepting less performance at one end. The best tune is the one that is stable everywhere the aircraft will actually fly, not the one that feels sharpest on a single calm test flight. Racing and cinematic aircraft Firmware such as Betaflight, built for small first-person-view aircraft flown in acrobatic and racing styles, approaches tuning with different priorities. Its pilots want the aircraft to follow stick inputs instantly, often at rotation rates exceeding a thousand degrees per second, and they fly in "rate" or "acro" mode, where the sticks command rotation rate directly and the angle loop is not used. Betaflight therefore emphasises the rate loop above all, with sophisticated filtering, RPM-based notch filtering driven by bidirectional DShot motor telemetry, feedforward tied to stick movement, and tuning sliders that adjust groups of gains together. Its blackbox logs are analysed with tools such as the community-developed PID Toolbox and the Betaflight Blackbox Explorer. ArduPilot's defaults favour a different kind of aircraft, one that must hold position, navigate autonomously, and fly stably for long periods with payloads. But the physics is the same, and the lessons transfer in both directions. Betaflight's community demonstrated, for example, how much rate loop performance could be gained by RPM-based notch filtering, an idea that ArduPilot's harmonic notch in ESC telemetry mode embodies. Whichever firmware is used, the order remains: mechanics, then vibration, then rate loops, then everything above them. With the inner loops tuned, the aircraft holds its attitude reliably. The next question is where it is. Chapter 5: Knowing Where You Are A quadcopter with well-tuned attitude loops will hold itself level, but it will not stay in one place. Hovering level in still air, the slightest residual tilt, a few tenths of a degree, produces a sideways acceleration that becomes a drift of metres within seconds. In wind, the drift is immediate. To hold a position, let alone fly to one, the aircraft needs to know where it is, how fast it is moving, and which way it is facing, and it needs to know these things continuously, reliably and with enough accuracy for the task at hand. A hobby aircraft circling a field needs a few metres of accuracy. A survey aircraft producing a map of a construction site needs centimetres. An inspection aircraft flying close to a bridge pier where satellite signals are blocked needs a way to navigate without them at all. This chapter covers the sources of position information available to a civilian drone and how the estimator decides which to trust. It is the layer between the attitude controllers of the previous chapters and the navigation and avoidance algorithms of the next ones, and it is the layer most often responsible for navigation failures that look, at first glance, like something else. Satellite navigation The Global Positioning System and its counterparts, Russia's GLONASS, the European Union's Galileo and China's BeiDou, collectively known as global navigation satellite systems, or GNSS, are the primary source of position for almost every outdoor drone. Each satellite broadcasts a signal containing its precise orbital position and a timestamp derived from an onboard atomic clock. A receiver measures the time of arrival of signals from several satellites and, knowing the speed of light, computes its distance from each, a quantity called a pseudorange because it includes an unknown offset from the receiver's own imprecise clock. With signals from at least four satellites, the receiver can solve for its three position coordinates and its clock offset simultaneously. Modern receivers on drones, such as the u-blox modules that dominate the hobby and professional market, track several constellations at once, which improves both accuracy and availability, since more satellites in view means better geometry and more resilience to signals being blocked. Standard GNSS positioning on a drone typically achieves horizontal accuracy of the order of a couple of metres under an open sky, and vertical accuracy somewhat worse, because all the satellites are above the receiver and their geometry constrains height less well. Several error sources contribute. The signals are delayed passing through the ionosphere and troposphere by amounts that vary with time and location. Satellite clock and orbit predictions contain small errors. Signals reflected off buildings, terrain or the aircraft itself arrive slightly late and corrupt the measurement, an effect called multipath that is especially troublesome near structures, which is precisely where inspection drones work. And the geometry of the satellites in view affects how much these errors translate into position error, a quantity summarised by the dilution of precision. For many purposes, a couple of metres is enough. For precision work, two enhancements are common. Satellite-based augmentation systems such as WAAS in North America and EGNOS in Europe broadcast corrections for the large-scale atmospheric and satellite errors over a region, and most modern receivers apply them automatically when available. Real-time kinematic positioning, or RTK, goes much further. Instead of using only the coded timing signal, an RTK receiver measures the phase of the carrier wave itself, whose wavelength is around nineteen centimetres for the main GPS frequency. By comparing its carrier phase measurements with those of a base station at a precisely known location nearby, and resolving the whole number of wavelengths between receiver and satellite, an RTK system can achieve centimetre-level relative accuracy. Corrections reach the aircraft over a telemetry radio link from a local base station or over the internet from a network of reference stations, using a standard format called RTCM. ArduPilot can inject these corrections into the aircraft's receiver through its ground station link. Two receivers can also give heading. Mounted a known distance apart on the airframe, a pair of RTK-capable receivers can measure the vector between their antennas with centimetre accuracy, and from it the aircraft's yaw angle. ArduPilot supports this "moving baseline" configuration, and it is increasingly used on aircraft where a magnetometer would be unreliable, for instance those flying close to steel structures or carrying high-current payloads. For survey work, the distinction between relative and absolute accuracy matters. RTK gives centimetre accuracy relative to the base station. If the base station's own position is only known to a metre, the whole survey is shifted by that metre. Professional mapping workflows therefore either place the base over a surveyed point, use a correction network tied to a national reference frame, or tie the final map to ground control points measured independently. An alternative to RTK is post-processed kinematic positioning, in which the raw satellite data are recorded on the aircraft and processed after the flight against base station data, giving similar accuracy without a live link. Many mapping drones use this approach to geotag images, while flying with ordinary GNSS for navigation. Altitude and heading GNSS altitude is less accurate than horizontal position and is vulnerable to the same disturbances. Almost every flight controller therefore also carries a barometer, a pressure sensor whose reading falls as the aircraft climbs. Barometers resolve changes of well under a metre and respond quickly, which makes them good for holding altitude over short periods. Their weakness is that air pressure changes with the weather, so absolute barometric altitude drifts over the course of a flight, and that the rotor downwash and airflow over the airframe create local pressure changes. At high forward speed, or when the aircraft descends into its own downwash near the ground, the barometer can report altitude errors of a metre or more. Good flight controller designs enclose the barometer in foam to shield it from airflow, and ArduPilot can compensate for some speed-related pressure errors. The EKF fuses the barometer and GNSS altitude together, using the barometer's short-term stability to smooth GNSS noise and GNSS to correct the barometer's long-term drift. Above terrain, absolute altitude is not always the useful quantity. An aircraft spraying a crop or inspecting a wind turbine blade needs to know its height above the surface beneath or beside it. Downward-facing rangefinders, using lidar, radar or ultrasound, provide this directly. ArduPilot uses rangefinder data both for terrain following and for precision landing, and the EKF can use it as a height source when flying low over flat ground. Terrain databases provide a second option: ArduPilot can download elevation data for the area around a mission from the ground station and use it to fly at a constant height above the ground, without a rangefinder, though only as accurately as the database itself. Heading comes from the magnetometer, the dual GNSS moving baseline described above, or from GNSS velocity during sustained forward flight, since the direction of travel constrains the heading once the aircraft moves fast enough. ArduPilot's EKF3 can also run without a magnetometer at all, using a separate estimator, sometimes called the Gaussian sum filter, that runs a bank of small filters with different heading hypotheses and selects the one most consistent with GNSS velocity as the aircraft manoeuvres. This allows recovery from a compass failure in flight, and it has become an important safety feature, because compass problems are one of the commonest causes of navigation trouble. Table 3 compares the principal sources of position, height and heading information available to a civilian multirotor. Table 3. Principal navigation sources on a civilian multirotor. Source Measures Typical strength Typical weakness Standard GNSS Position, velocity Global, absolute Metre-level error, multipath near structures RTK GNSS Position, heading (dual) Centimetre relative accuracy Needs corrections link, base accuracy Barometer Altitude change Fast, precise short term Weather drift, downwash and airspeed errors Magnetometer Heading Works at rest, cheap Motor currents, steel, local anomalies Rangefinder Height above surface Direct terrain clearance Limited range, surface-dependent Optical flow Velocity over ground Works without GNSS Needs texture, light and known height Visual odometry Position, velocity, attitude Indoor and GNSS-denied use Computation, lighting, drift over distance Navigating without satellites There are many places where satellite signals are weak or absent: inside buildings, under bridges, inside tunnels and mines, beneath dense forest canopy, and close to tall structures where multipath is severe. Inspection is often done exactly there. Several alternatives allow a drone to estimate its motion without GNSS. Optical flow sensors use a small downward-facing camera to track the apparent motion of the ground texture between frames. The flow, combined with the aircraft's height from a rangefinder and its rotation rate from the gyroscope, gives velocity over the ground. ArduPilot can fuse optical flow into EKF3 to allow position hold indoors or in GNSS-denied areas, and the technique is the basis of the stable indoor hover that small consumer drones achieve. It needs a textured surface, adequate light, and a reliable height measurement; over calm water, a uniform floor or long grass moving in the wind, it struggles. Visual odometry and visual-inertial odometry go further, using one or more cameras together with the IMU to track features in the environment and estimate the aircraft's full position and orientation over time. Stereo cameras and depth sensors provide scale directly; monocular systems recover it by combining camera and IMU motion. These systems typically run on a companion computer, a separate small computer carried alongside the flight controller, and send their estimates to ArduPilot over MAVLink, where EKF3 fuses them as an external navigation source. Odometry drifts over distance, since small errors accumulate, so for long flights it is often combined with a map of the environment, an approach called simultaneous localisation and mapping, or SLAM. Motion capture systems, using arrays of infrared cameras tracking reflective markers on the aircraft, give millimetre-accurate position indoors and are standard in university research laboratories. Ultra-wideband radio beacons placed around a site can provide decimetre-level positioning indoors, and ArduPilot supports several such systems. Each of these feeds the same estimator through the same interfaces, which is the value of a general fusion architecture: the controllers above need not know or care whether the position estimate comes from satellites, cameras or beacons. ArduPilot's EKF3 allows several sets of navigation sources to be defined, using parameters of the form EK3_SRC1_POSXY, EK3_SRC1_VELXY, EK3_SRC1_POSZ and EK3_SRC1_YAW for the first set and equivalents for the second and third, and to switch between them in flight, either by a transmitter switch, by a Lua script, or by a command from a ground station or companion computer. An aircraft can thus fly outdoors on GNSS, switch to visual odometry as it enters a building or passes under a bridge, and switch back as it emerges. Managing these transitions so that the estimate does not jump when the source changes is one of the harder practical problems in GNSS-denied navigation, and it is an area of active development. When the sensors lie Every navigation sensor fails in characteristic ways, and the estimator must handle them without letting a bad reading drive the aircraft off course. GNSS glitches are the most familiar. A sudden jump in reported position, caused by multipath, a brief loss of satellites or receiver error, would, if fused naively, tell the position controller that the aircraft had suddenly moved, and the controller would fly it in the opposite direction to correct. The EKF's innovation gating, described in Chapter 2, rejects measurements that disagree too strongly with the prediction, and if the disagreement persists, the filter gradually reconciles with the new data rather than jumping. ArduPilot's pre-arm checks also refuse to arm in modes that need position until the GNSS fix is of adequate quality, with enough satellites and a low enough reported error. GNSS interference is a growing concern. Intentional jamming and spoofing, along with unintentional interference from badly designed electronics, have become more common in several parts of the world, and civil aviation authorities have issued warnings about their effects on both crewed and uncrewed aircraft. A jammed receiver loses its fix entirely, which an aircraft can detect and respond to by switching to a non-GNSS mode or triggering a failsafe. A spoofed receiver is more dangerous, because it reports a plausible but false position. Defences include cross-checking GNSS against inertial and other sensors, monitoring signal strength and consistency, using multi-constellation and multi-frequency receivers, and fitting antennas designed to reject signals from below the horizon. None of these is perfect, and operators flying beyond line of sight in areas with known interference should plan accordingly. Compass errors are subtler and, in practice, more common. A magnetometer affected by motor currents or nearby steel reports a heading offset by some angle. The position controller, believing the aircraft faces a slightly different direction, applies its corrections in the wrong direction, and the aircraft drifts in a curve rather than holding position. The result is the circling toilet-bowl pattern described earlier, which grows as the controller chases an error it is itself creating. Preventing it involves careful compass placement, calibration in the configuration in which the aircraft will fly, motor interference compensation, and refusal to take off near large steel objects or reinforced concrete. Increasingly, ArduPilot's ability to estimate heading from GNSS velocity alone allows the EKF to detect and recover from a bad compass in flight. Barometric errors show up as altitude bounce in fast flight or near the ground, and vibration-induced accelerometer clipping, mentioned in the previous chapter, can corrupt the vertical velocity estimate badly enough that the aircraft climbs or descends unexpectedly. This is one reason vibration matters beyond the rate loop: it reaches the estimator too. The general lesson is that the navigation layer is not a single sensor but a judgement, made continuously by the estimator, about which of several imperfect sources to trust and by how much. When the judgement is right, the controllers above have an accurate, smooth estimate to work with and the aircraft flies precisely. When it is wrong, the controllers will faithfully fly the aircraft to the wrong place. With a trustworthy estimate in hand, the aircraft is ready for the next layer: deciding where to go. Chapter 6: From Waypoints to Trajectories The attitude loops make the aircraft stable. The estimator tells it where it is. The position controller can now hold it at a point or move it toward one. What remains is to decide where the aircraft should go and how it should get there: the navigation layer. For a pilot flying by hand, this layer is the pilot, translating intentions into stick movements. For an autonomous aircraft, it is software that turns a mission, usually expressed as a list of waypoints, into a continuous trajectory that the position controller can follow without demanding more than the aircraft can deliver. The challenge is not geometric. Drawing a path between points on a map is trivial. The challenge is dynamic: producing a sequence of positions, velocities and accelerations that respects the aircraft's limits on speed, acceleration and jerk, that passes through or near the right places, and that leaves enough margin for the position controller to correct disturbances along the way. A path that the aircraft cannot follow is worse than useless, because the position controller will try to follow it anyway and end up saturating its lean angle, overshooting corners, and arriving late and in the wrong place. Flight modes as layers ArduPilot's flight modes are a direct expression of the control stack, and understanding them as such clarifies what each one does and does not do. In Stabilize mode, the pilot's sticks command attitude directly. The roll and pitch sticks set lean angles, the yaw stick sets yaw rate, and the throttle stick sets motor output. Only the attitude and rate loops are active; there is no altitude hold or position control. Acro mode goes one layer lower, with all three sticks commanding rotation rates, so the aircraft holds whatever attitude it was left at. AltHold adds the vertical position controller: the throttle stick now commands climb rate, and centring it holds altitude. Horizontal motion is still controlled through lean angle. Loiter and PosHold add the horizontal position controller, so the sticks now command horizontal velocity or, when centred, hold position. Loiter is designed for smooth position control with braking when the sticks are released; PosHold behaves more like AltHold under stick input and only locks position when the sticks are centred. Guided mode hands the navigation layer to an external source. A ground station or companion computer sends position, velocity or acceleration targets over MAVLink, and the aircraft's position controller follows them. Guided mode is the basis of most custom autonomy, including the obstacle avoidance and companion computer behaviours discussed later. Auto mode runs a stored mission, a list of commands uploaded from a ground station, and it is the mode most associated with autonomous flight. RTL, short for return to launch, climbs to a set altitude and flies home; SmartRTL retraces the path the aircraft actually flew, which is useful when the direct route home is obstructed. Land descends and lands at the current position. Brake stops the aircraft as quickly as possible and holds. Each mode, then, defines which layers of the stack are under software control and which are under the pilot's. Switching modes in flight swaps the source of commands at a given layer without disturbing the layers below. This is why a pilot can switch from Auto to Loiter at any moment and immediately have a stable, position-holding aircraft: the position controller, attitude loops and estimator are already running, and only the source of targets has changed. The mission An ArduPilot mission is a numbered list of items, each a MAVLink mission command with up to seven parameters. Most are navigation commands that tell the aircraft to go somewhere or do something that involves moving; others are actions that happen immediately or conditions that delay the next item. The list is uploaded from a ground station such as Mission Planner or QGroundControl using the MAVLink mission protocol and stored in the flight controller's memory, where it survives a reboot. A handful of commands do most of the work in civilian missions. Table 4 lists the commonest, with their MAVLink command identifiers, which are fixed by the MAVLink standard and shared by all autopilots that implement it. Table 4. Common MAVLink mission commands used in civilian multirotor missions. Command ID What it does Typical use NAV_TAKEOFF 22 Climb vertically to a set altitude First item of most missions NAV_WAYPOINT 16 Fly to a position, optionally pausing Path points, survey lines NAV_SPLINE_WAYPOINT 82 Fly to a position along a smooth curve Cinematic and inspection paths NAV_LOITER_TIME 19 Hold position for a set time Observation, sensor dwell DO_CHANGE_SPEED 178 Set new horizontal speed Slow sections near structures DO_SET_ROI 201 Point the aircraft or gimbal at a location Orbit and facade inspection DO_SET_CAM_TRIGG_DIST 206 Trigger camera every set distance Photogrammetry surveys NAV_RETURN_TO_LAUNCH 20 Fly home and land Last item of most missions Altitudes in a mission can be relative to the home position, relative to sea level, or relative to terrain, and choosing the wrong frame is a classic source of trouble. A mission planned with altitudes relative to home over a hillside site will fly at a constant height above the takeoff point, which may be dangerously low over rising ground or uselessly high over falling ground. Terrain-relative altitudes avoid this, using either a rangefinder or a terrain database. Shaping the path Given a sequence of waypoints, the navigation library must produce a trajectory. The simplest approach is to fly straight from each waypoint to the next at constant speed, stopping at each one. This works but wastes time and energy, and the abrupt changes of direction are hard on both the aircraft and any camera it carries. Every modern autopilot therefore shapes the path to account for the aircraft's dynamics. ArduPilot's waypoint navigation, in the AC_WPNav library, uses the parameters WPNAV_SPEED for horizontal speed, WPNAV_SPEED_UP and WPNAV_SPEED_DN for vertical speeds, WPNAV_ACCEL for horizontal acceleration, and WPNAV_RADIUS for how close the aircraft must come to a waypoint before it is considered reached. In recent versions, ArduPilot generates trajectories using S-curves: velocity profiles in which not only speed and acceleration but also jerk, the rate of change of acceleration, is limited. A trapezoidal profile, with constant acceleration up to cruise speed and constant deceleration down to rest, has instantaneous changes in acceleration at the corners, which demand instantaneous changes in lean angle. The attitude controller cannot deliver those, so the aircraft lags its trajectory at every transition. An S-curve rounds off the corners of the acceleration profile, so lean angle changes smoothly, and the aircraft can follow the trajectory closely. The jerk limit is linked to how quickly the attitude controller can change lean angle, which is one more instance of the outer loop being shaped to respect the inner loop's capabilities. At intermediate waypoints, ArduPilot does not stop unless told to. As the aircraft approaches a waypoint, the navigation library begins the turn toward the next one early enough to blend the two legs into a smooth curve, limited by the acceleration available. The aircraft therefore cuts the corner slightly, passing near rather than exactly through the waypoint. For many tasks this is ideal. For tasks where the aircraft must pass exactly over a point, such as triggering a camera above a ground control marker, a delay parameter on the waypoint forces a stop. Spline waypoints produce smoother curves still. Rather than straight legs joined by short turns, a spline waypoint defines a curve whose shape is determined by the positions of the waypoints before and after it, so the aircraft flies a continuous arc through the whole sequence. Spline paths are favoured for cinematography, where smooth camera motion matters, and for some inspection tasks, where they allow the aircraft to follow a curved structure without stopping. The research literature has gone further. A widely cited approach, published by Daniel Mellinger and Vijay Kumar in 2011, generates trajectories that minimise snap, the fourth derivative of position, through a sequence of waypoints. Because a quadcopter's motor commands relate to the fourth derivative of its position through the chain of integrations described in Chapter 1, a minimum-snap trajectory is in a precise sense the smoothest the motors can follow. Minimum-snap and related polynomial trajectories are widely used in research and in some commercial systems, usually generated on a companion computer and fed to the autopilot as a stream of setpoints in Guided mode. Surveys, inspections and deliveries Civilian missions fall into a few recognisable patterns, each with its own requirements of the navigation layer. Mapping surveys use a lawnmower pattern: parallel lines flown back and forth over an area, with a camera triggered at regular distances so that consecutive images overlap along each line and adjacent lines overlap across. Photogrammetry software then matches features between the overlapping images to reconstruct a three-dimensional model and an orthomosaic, a map corrected for perspective. Typical overlaps are high, often seventy to eighty percent, because the reconstruction depends on seeing each ground point from several angles. Ground sample distance, the size of one image pixel on the ground, is set by the camera's sensor, its lens and the flight height: flying twice as high doubles the ground sample distance, covering more area with less detail. Mission planning tools such as the survey grid function in Mission Planner compute the line spacing, trigger distance and speed from the camera specifications, the desired overlap and the ground sample distance, and generate the waypoints automatically. The navigation requirement is accurate, straight lines at constant speed and height, which means good GNSS, well-tuned position control, and terrain following over uneven ground. A worked example shows how these quantities interact. Suppose the camera has a sensor 13.2 millimetres wide behind an 8.8 millimetre lens, producing images 5,472 pixels wide and 3,648 pixels tall, a common format for one-inch-sensor survey cameras. Ground sample distance is the sensor width multiplied by the flight height, divided by the product of focal length and image width in pixels. At 100 metres, that gives 13.2 times 100 divided by 8.8 times 5,472, or about 2.7 centimetres per pixel. Each image then covers about 150 metres across the flight line and about 100 metres along it, if the camera is mounted with its long side across track. For 75 percent forward overlap, consecutive photographs must be taken every quarter of the along-track footprint, so every 25 metres. For 70 percent side overlap, adjacent lines must be 30 percent of the across-track footprint apart, about 45 metres. These numbers then constrain the flight. If the camera can reliably capture and store an image every two seconds, the aircraft cannot fly faster than 25 metres every two seconds, or 12.5 metres per second, without missing triggers. Motion blur sets a second limit: at 10 metres per second with a shutter speed of one thousandth of a second, the aircraft moves one centimetre during the exposure, comfortably less than one pixel, but at a slower shutter in poor light the blur would approach the ground sample distance and degrade the reconstruction. Wind sets a third: flying lines into and with a strong wind gives very different ground speeds on alternate passes, and the trigger distance command, which fires on distance travelled rather than time, keeps the overlap constant where a fixed time interval would not. Finally, turning at the end of each line costs time and battery, which is why survey planners orient lines along the long axis of the area and, on aircraft that allow it, fly the turns at a lower acceleration limit than the lines. A planner that gets all of this right produces a mission in which the navigation layer's only job is to fly straight, level and steady, which is exactly what a well-tuned position controller does best. Inspection missions are more varied. A facade inspection might fly vertical columns up and down a building at a constant standoff distance, with the camera pointing at the wall. A tower inspection might orbit at several heights. A bridge inspection might fly beneath the deck, without GNSS, using visual odometry or optical flow. These missions often use region-of-interest commands to keep the camera pointed at the structure, and they demand precise positioning close to obstacles, which makes them both the most demanding and the riskiest navigation tasks most civilian drones perform. They are also where obstacle sensing and avoidance, discussed in the next chapter, matter most. Delivery flights combine long, fairly simple cruise segments with demanding endpoints: a precise descent to a drop point or a landing pad, sometimes using a downward camera and a visual marker for precision landing, sometimes lowering the package on a tether while the aircraft hovers. ArduPilot supports precision landing using infrared beacons or visual markers detected by a companion computer, and it supports winch control for tethered delivery. Delivery operations at scale also involve the regulatory and safety requirements of flying beyond visual line of sight, discussed in the final chapter. Guided mode and external planners Missions stored on the aircraft are fixed once uploaded. Many applications need the path to change in flight in response to what the aircraft sees: following a moving subject, tracking a crop row, revising a route around a newly detected obstacle, or searching an area adaptively. For these, the navigation layer moves off the autopilot to a companion computer or ground station that sends targets in Guided mode. ArduPilot's Guided mode accepts targets at several levels. A position target makes the aircraft fly to a point, using the same S-curve shaping as Auto mode. A velocity target makes it fly at a given velocity until told otherwise, with a timeout that brings it to a stop if updates cease, which is a vital safety feature when the external planner might crash. An acceleration target, or a combined position, velocity and acceleration target, lets the external planner specify a full trajectory that the position controller then tracks. Attitude targets bypass the position controller entirely and command the attitude loops directly, for applications that need to control the aircraft's motion more aggressively than the position controller allows. This flexibility is what makes ArduPilot a platform rather than a closed product. The stack is open at every layer: an external program can replace the mission planner, the trajectory generator or even the position controller, while the autopilot continues to provide stabilisation, estimation and failsafes beneath it. The design choice that makes this possible, a clean separation between layers with well-defined interfaces between them, is the same one that makes the stack tunable in the first place. It is also what makes it possible to add the capability that turns a navigating aircraft into one that can safely share space with the world: the ability to sense and avoid obstacles. Chapter 7: Seeing and Avoiding A drone flying a pre-planned mission in an open field needs no knowledge of its surroundings. The planner has already decided where it can safely go, and the aircraft simply follows. As soon as drones work near structures, trees, people and other aircraft, that assumption fails. The world changes, maps are wrong, and the precise positioning that makes close inspection possible also makes small errors consequential. A survey aircraft whose mission was planned from a map made last year may meet a crane that was not there. An inspection aircraft holding position near a tower in gusty wind may be pushed toward it. A delivery aircraft descending to a garden may find a washing line across its path. Collision avoidance is the layer that handles these surprises. It sits between the navigation layer and the position controller, modifying the targets that navigation requests so that the aircraft does not fly into anything it can detect. Like every other layer, it depends on the ones below it: avoidance is only as good as the sensing that informs it and the position control that executes it, and the commonest failure of avoidance systems is not a bad algorithm but a sensor that did not see the obstacle. Sensing the world Several kinds of sensors give a drone information about its surroundings, each with characteristic strengths and blind spots. Single-beam rangefinders, using a laser, ultrasound or radar, measure the distance to whatever lies in one direction. They are cheap, light and simple, and several pointing in different directions give coarse coverage. Ultrasound works only at short range and is confused by soft surfaces and by the noise of propellers. Laser rangefinders give longer range and better precision but can miss thin objects, transparent surfaces and dark materials. Radar works in dust, fog and darkness, and sees through foliage to some extent, at the cost of lower angular resolution. Scanning lidars sweep a laser beam around the aircraft, producing a ring of distance measurements in a horizontal plane or, in more expensive units, a three-dimensional point cloud. A 360-degree scanning lidar mounted on top of an aircraft gives a detailed picture of obstacles at its own height, which suits horizontal avoidance in inspection and indoor flight. ArduPilot supports several such sensors through its proximity library, configured with parameters prefixed PRX. Cameras provide the richest information, from which depth can be recovered by stereo matching between two cameras, by structured light, by time-of-flight measurement, or by algorithms that infer depth from a single image. Depth cameras produce dense three-dimensional data over a cone in front of the aircraft. Their range is limited, their field of view is narrow compared with a lidar, they need light, and they require a companion computer to process their data. But they see thin wires and branches that a scanning lidar's sparse beams may miss, and their output can feed object recognition as well as geometric avoidance. For other aircraft, the main civilian tool is cooperative surveillance. Many crewed aircraft broadcast their position, altitude and velocity using ADS-B, Automatic Dependent Surveillance-Broadcast, and small ADS-B receivers can be carried on drones. ArduPilot's avoidance library, enabled with AVD_ENABLE, can monitor these broadcasts and take action, such as climbing, descending or moving away, when a crewed aircraft is predicted to come too close. ADS-B only detects aircraft that carry a transmitter, which many light aircraft, gliders and helicopters at low altitude do not. Remote ID broadcasts, discussed in the final chapter, provide similar information about other drones but are not designed as a collision avoidance system. Every sensor has blind spots, and the gaps between them define the limits of any avoidance system. A forward-facing depth camera does not see what is beside or behind the aircraft. A horizontal lidar does not see power lines above or below its scanning plane. No common onboard sensor reliably detects every thin wire at useful range. Operators who rely on avoidance should know exactly what their sensors can and cannot see, and plan missions accordingly. Simple avoidance: stop and slide The most basic and robust form of avoidance is to prevent the aircraft from flying too close to any detected obstacle. ArduPilot's simple avoidance, enabled through AVOID_ENABLE, works in pilot-controlled modes such as Loiter and AltHold, as well as in some autonomous modes. It takes the pilot's velocity request and, for each direction in which an obstacle is detected, limits the velocity component toward the obstacle so that the aircraft can stop at a set margin, AVOID_MARGIN, given its maximum deceleration. If the pilot pushes toward a wall, the aircraft slows and stops at the margin. If the pilot pushes diagonally toward the wall, the component toward the wall is removed and the aircraft slides along it. A related behaviour backs the aircraft away if it finds itself closer than the margin, for instance because it was pushed by a gust. The same mechanism enforces geofences, the virtual boundaries discussed in the final chapter: a fence boundary is treated as an obstacle, and the aircraft stops before crossing it. Simple avoidance is a velocity filter: it does not plan a route around an obstacle, only prevents the aircraft from moving into one. This makes it easy to reason about and hard to defeat. Its weakness is that an autonomous mission that encounters an obstacle across its path will simply stop, which is safe but does not complete the mission. Path planning around obstacles To get past an obstacle rather than merely stop in front of it, the aircraft must plan a new path. ArduPilot provides two object avoidance algorithms for this, selected with OA_TYPE, and the wider research literature offers many more. They fall into two broad families, local reactive methods and global planners, which make different trade-offs. Table 5 compares representative examples of each. Table 5. Representative obstacle avoidance approaches for small multirotors. Approach Family Needs a map Strength Weakness Stop and slide Velocity filter No Simple, predictable Does not route around obstacles BendyRuler Local reactive No Handles unexpected obstacles in flight Can be trapped in concave shapes Vector field histogram Local reactive Short-term grid Smooth steering through clutter Local minima, tuning sensitive Velocity obstacles Local reactive No Handles moving obstacles Requires obstacle velocity estimates Dijkstra or A* on a graph Global planner Yes Finds a route if one exists Map must be accurate and current Sampling planners (RRT family) Global planner Yes Copes with complex 3D spaces Computation, path smoothing needed BendyRuler is ArduPilot's local method. At each update, it probes a set of candidate directions around the line toward the destination, extending a short "ruler" in each direction and checking whether the path along it would stay clear of detected obstacles and fences. It picks the clear direction closest to the destination, with a preference for continuing in the direction it chose last time to avoid dithering, and flies that way. It works with live sensor data and needs no map, which makes it suited to obstacles that were unknown when the mission was planned. Like all local methods, it can be trapped: an aircraft approaching the inside of a U-shaped obstacle will find every direction toward the destination blocked and may oscillate or stop. Its parameters allow the probe distance and the preference for continuing in the same direction to be tuned. The vector field histogram, introduced by Johann Borenstein and Yoram Koren in 1991 for ground robots, builds a short-term grid of obstacle densities around the robot, reduces it to a polar histogram of how blocked each direction is, and steers toward the least obstructed direction near the goal. Velocity obstacle methods, developed in the late 1990s by Paolo Fiorini and Zvi Shiller, and their reciprocal variants, compute the set of velocities that would lead to a collision with each moving obstacle within a time horizon, and choose a velocity outside all of them. These methods handle moving obstacles explicitly, which makes them a natural fit for multi-drone coordination, where each aircraft knows the others' velocities. ArduPilot's second algorithm is a global planner based on Dijkstra's shortest path algorithm. It builds a graph whose nodes are points around the edges of known exclusion zones and fences, connects pairs of nodes that can see each other without crossing a boundary, and finds the shortest route from the aircraft to the destination through the graph. Because it needs the boundaries in advance, it works with polygon fences and exclusion zones defined at mission planning time rather than with obstacles seen in flight. It always finds a route if one exists and never gets trapped, but it can only avoid what it knows about. ArduPilot allows the two to be combined, using Dijkstra's for known boundaries and BendyRuler for unexpected obstacles. More capable global planning, used in research and in some commercial systems, works in three dimensions on a map built from sensor data in flight. A common approach represents the environment as a voxel grid or an octree, a hierarchical structure such as the widely used OctoMap library, in which each cell records the probability that it is occupied. A planner such as A* searches a graph of cells for a collision-free path, or a sampling-based planner such as a rapidly exploring random tree grows a set of random collision-free paths until one reaches the goal. These planners run on a companion computer, typically within the Robot Operating System, and send their paths to the autopilot as a stream of targets in Guided mode. Where avoidance fails Avoidance systems fail in recognisable ways, and the patterns are worth knowing because they recur across products and platforms. The commonest failure is a sensor that does not see. Thin wires, bare branches, glass, water, dark matte surfaces and objects outside the sensor's field of view are all frequently missed. An aircraft with forward-facing sensors that is flown sideways or backwards may have no protection at all in the direction of travel. Some commercial consumer drones disable avoidance in their fastest modes, and some disable it in low light. The first rule of avoidance is that its absence must be assumed until its presence has been confirmed for the specific sensor, direction, speed, lighting and obstacle type in question. The second is stopping distance. Avoidance can only work if the aircraft detects an obstacle far enough away to stop or turn before reaching it. A sensor with ten metres of range on an aircraft flying at fifteen metres per second gives less than a second of warning, which may be insufficient once the sensor's own processing delay and the aircraft's deceleration limit are taken into account. Avoidance systems therefore usually limit speed in proportion to sensor range, and operators who raise speed limits without understanding this remove the margin avoidance depends on. It is worth doing the arithmetic once, because it makes the constraint concrete. An aircraft flying at speed v and able to decelerate at a rate a needs a distance of v squared divided by twice a to stop. At 10 metres per second with a deceleration of 5 metres per second squared, a reasonable figure for a survey quadcopter that limits its lean angle, the braking distance is 10 metres. Before braking can begin, the sensor must take its measurement, the processing chain must recognise an obstacle, and the controller must begin to tilt back; if that chain takes 0.2 seconds, the aircraft covers another 2 metres. Add a 2-metre margin so that the aircraft stops short of the obstacle rather than at it, and the sensor must detect obstacles at least 14 metres away. If the sensor reliably detects the obstacle in question only at 8 metres, perhaps because it is a dark, thin structure, the safe speed falls to a little over 6 metres per second. Halving the deceleration, as happens on a heavily loaded aircraft or one that limits its lean angle to protect a gimbal, doubles the braking distance and lowers the safe speed further. Avoidance specifications quoted as a single range figure hide this dependence; the useful question is always at what range a given sensor sees a given obstacle, and how fast the aircraft can be flying for that range to be enough. The same reasoning applies sideways. An aircraft holding a two-metre standoff from a wall in gusty wind relies on its position controller to reject a gust before it closes the gap. If a gust can push the aircraft a metre before the controller responds, the standoff margin has already lost half its value before avoidance has anything to do. Facade and tower inspection procedures commonly specify standoff distances and maximum wind speeds together for this reason, and an operator who reduces one without reconsidering the other has removed a margin the system was relying on. The third is local traps and oscillation, already described. A reactive algorithm in a cluttered environment may wander, dither between two directions, or stop. This is safe in the sense that the aircraft does not collide, but it may then drain its battery hovering, which turns an avoidance problem into a power problem. Mission design and failsafe configuration must anticipate it. The fourth is the interaction between avoidance and position control. If the position estimate is wrong, a map-based planner will plan around obstacles that are not where it thinks they are. If the aircraft is buffeted by wind close to a structure, the position controller may not be able to hold the margin avoidance requests. Avoidance cannot be better than the estimation and control layers beneath it, and testing it in calm conditions does not prove it will work in gusty ones. The last is moving obstacles, especially other aircraft. Most onboard avoidance assumes a static world. ADS-B gives warning of cooperative crewed aircraft, but non-cooperative traffic, birds and other drones are hard to detect at useful range with the sensors a small drone can carry. Detecting and avoiding non-cooperative aircraft, a capability called detect and avoid, is the central technical requirement for routine beyond-line-of-sight operation, and it is the subject of considerable industry and regulatory work, including a consensus standard for small unmanned aircraft, ASTM F3442. For now, most beyond-line-of-sight operations rely on a combination of airspace restrictions, ground-based surveillance, cooperative signals and operational procedures rather than on the aircraft's own sensors alone. Avoidance as a layer The lesson running through this chapter is the same as in the earlier ones. Avoidance is a layer that intercepts the targets flowing from navigation to position control and modifies them. It makes promises to the layer above: "give me a destination and I will try to reach it without hitting anything I can see." It relies on promises from the layers below: that the estimator knows where the aircraft is and the position controller can put it where it is told. When it fails, the cause can usually be found in a specific broken promise: the sensor did not see, the estimator was wrong, the controller could not hold the margin, or the algorithm was trapped. Diagnosing which one is far more productive than treating avoidance as a single feature that either works or does not. Building, configuring and extending this layer, and every other layer of the stack, ultimately means writing and running software. The next chapter turns to the software itself: how ArduPilot is organised, how to run it without an aircraft, and how to add new behaviour without disturbing the loops that keep the aircraft in the air. Chapter 8: Programming the Flight Controller Everything described so far, from the mixer to the obstacle avoidance planner, is software, and in an open-source autopilot all of it can be read, changed and extended. That openness is what has made ArduPilot and PX4 the default platforms for research, for small manufacturers of survey and inspection aircraft, and for a large community of builders who want to understand their machines rather than merely operate them. It also carries responsibility. The same code that can be modified to add a new sensor or a new mission behaviour can be modified to break the loop that keeps the aircraft in the air. Programming a flight controller well means understanding how the codebase is organised, which parts are safety-critical, and which interfaces exist precisely so that new behaviour can be added without touching them. This chapter introduces ArduPilot's architecture, the simulator that makes development safe, the scripting engine that allows behaviour to be added without recompiling, and the MAVLink protocol through which external programs talk to the aircraft. It ends with a comparison with PX4 and a note on how the two projects differ in philosophy. How ArduPilot is organised ArduPilot began in 2009 as a hobby project built around Arduino-based hardware, growing out of the DIY Drones community. It has since become a mature, professionally maintained open-source project, licensed under the GNU General Public License version 3, with a core development team, regular releases, and use in commercial aircraft as well as hobby and research platforms. The single source repository contains firmware for several vehicle types: Copter for multirotors and traditional helicopters, Plane for fixed-wing and VTOL aircraft, Rover for ground and surface vehicles, Sub for underwater vehicles, and a few others. Each vehicle has its own directory containing its flight modes and top-level logic, and all of them share a large collection of libraries. The libraries are where most of the code discussed in this book lives. AP_InertialSensor reads and filters the IMUs, including the harmonic notch. AP_NavEKF3 is the estimator. AC_AttitudeControl contains the attitude and rate controllers, and AC_PosControl the position controller. AP_Motors contains the mixers for every frame type. AC_WPNav handles waypoint navigation and trajectory shaping. AP_Mission stores and executes missions. AC_Avoidance contains simple avoidance and the object avoidance algorithms. GCS_MAVLink implements the communication protocol. AP_Param provides the parameter system that makes all of these configurable without recompiling. Reading the source of a parameter's library, and finding where it is used, is often the quickest way to understand exactly what it does. Beneath the libraries is the hardware abstraction layer, AP_HAL. The flight code never talks to hardware directly; it calls HAL interfaces for timers, serial ports, SPI and I2C buses, PWM outputs, storage and threads. Each supported platform implements these interfaces. The main implementation for flight hardware runs on top of ChibiOS, a small real-time operating system, on STM32 microcontrollers from the F4, F7 and H7 families that dominate current flight controllers. Others run on Linux boards and, crucially, on a desktop computer for simulation. A board's specific wiring, which pins connect to which sensors and outputs, is described in a text file called a hardware definition, from which the build system generates the configuration. Adding support for a new flight controller board is usually a matter of writing that file rather than writing code, which is why ArduPilot supports hundreds of boards. The firmware is built with the waf build system. Configuring for a board and building the Copter firmware looks like this: ./waf configure --board MatekH743 ./waf copter At run time, the vehicle code is driven by a scheduler. The fastest, most critical work, reading the IMU, running the rate and attitude controllers, and writing to the motors, runs in a fast loop at the main loop rate. Everything else is listed in a scheduler table, each task with a target rate and a maximum expected run time in microseconds: the EKF update, GPS reading, navigation, logging, MAVLink communication, battery monitoring, and dozens of others. At each loop, after the fast work is done, the scheduler runs as many due tasks as fit in the remaining time. This cooperative design is simple and predictable, but it means any task that takes longer than expected steals time from others. A new piece of code that blocks, waiting on a slow sensor or doing a long calculation in one go, can delay the loop and degrade control. ArduPilot logs loop timing and reports when the scheduler is overrunning, and developers adding code to the main thread are expected to keep it short and non-blocking, moving anything slow to a separate thread. The simulator The single most important tool for developing on ArduPilot is SITL, software in the loop. SITL compiles the real flight code, the same libraries and vehicle code that run on the aircraft, for a desktop computer, and connects it to a simulated physical model of the vehicle and its sensors. The flight code sees simulated IMU, GPS, barometer and compass readings, computes motor outputs, and the simulator turns those outputs into simulated motion. From the flight code's point of view, nothing is different from real flight. Starting a simulated Copter with a console and a map is one command, run from the vehicle's directory in a development environment: sim_vehicle.py -v ArduCopter --console --map The simulator speaks MAVLink, so a ground station such as Mission Planner or QGroundControl can connect to it exactly as it would to a real aircraft. A developer can plan and fly missions, change parameters, inject sensor failures, simulate wind, and watch how the controllers and estimator respond. SITL can also connect to external physics simulators such as Gazebo and the commercial RealFlight simulator, which provide more realistic dynamics, three-dimensional environments and simulated cameras and lidars for testing perception and avoidance. The ArduPilot project's automated test suite is built on SITL. Every proposed change to the code is run through hundreds of simulated flights that check that the vehicle still takes off, holds position, flies missions, handles failsafes and lands. This is one of the main reasons ArduPilot is trustworthy despite its size and the number of people who contribute to it, and it is a model for anyone developing their own extensions: test in simulation first, automatically if possible, and only then on hardware. Simulation has limits. The simulated vehicle's dynamics are simplified, simulated vibration is not real vibration, and simulated GPS errors are not real multipath. A tune that is perfect in simulation will not be perfect on a real aircraft. But logic errors, wrong signs, unit mistakes, mode transition bugs and failsafe misconfigurations all show up in simulation, and catching them there is far cheaper than catching them in a field. Scripting with Lua Modifying and recompiling the firmware is powerful but heavy: it requires a development environment, careful testing, and maintenance of a private branch that must be updated as ArduPilot evolves. For many applications, ArduPilot's embedded Lua scripting engine is a better choice. Scripts are small text files placed on the flight controller's SD card, in a scripts directory, and enabled with SCR_ENABLE. They run in a sandbox with their own memory allowance and time limits, so a script that errors or runs away is stopped without crashing the flight code. A script typically defines a function that does some work and then returns itself with a delay, so the scripting engine calls it again after that interval. A minimal script that reports altitude to the ground station every two seconds looks like this: function update() local loc = ahrs:get_location() if loc then gcs:send_text(6, string.format("Alt: %.1f m", loc:alt() * 0.01)) end return update, 2000 end return update() Scripts have access to a wide set of bindings: reading the estimator's position and attitude, reading and setting parameters, reading sensors, sending and receiving MAVLink messages, changing flight modes, setting Guided mode targets, controlling servos and relays, and interacting with serial devices. The ArduPilot repository includes a library of example and applet scripts for tasks such as switching EKF sources between GNSS and non-GNSS navigation, running the QuickTune procedure, adding custom flight modes for particular inspection patterns, controlling winches and parachutes, and driving custom sensors. Scripting is well suited to mission logic, payload control, custom failsafe behaviour and sensor integration. It is not suited to anything that must run in the fast loop, such as a new rate controller, because scripts run at a lower priority and much more slowly than compiled code. The division mirrors the stack: the inner loops stay in compiled, tested core code, and new behaviour at the upper layers can be added in scripts that cannot destabilise them. A worked extension It helps to follow one small extension from idea to field, because the sequence matters more than the code. Suppose an inspection operator wants the aircraft, during an Auto mission, to pause and hold position for ten seconds whenever a payload sensor on a serial port reports a reading above a threshold, logging the event and notifying the ground station, and then to resume the mission. This could be implemented by modifying the mission code in C++, but nothing about it touches the inner loops, so it belongs in a script. The script's update function reads the sensor through the serial binding, checks the reading, and when the threshold is exceeded switches the vehicle to a holding mode such as Loiter, records the time, sends a status message, and writes a custom log entry. On later calls it checks whether ten seconds have passed and, if so, switches back to Auto, which resumes the mission from the current item. Several details need care, and each corresponds to a general lesson. The script must do nothing if the pilot has switched to a manual mode in the meantime, since overriding a pilot's deliberate mode change is dangerous; so it checks the current mode before acting and only intervenes while the vehicle is in Auto. It must handle the sensor going silent, rather than waiting indefinitely, so it treats a missing reading as no event rather than blocking. It must not trigger repeatedly while the reading stays high, so it records that an event has been handled until the reading falls. And it must leave the failsafes untouched: if the battery failsafe triggers RTL during the pause, the script must recognise that the vehicle is no longer in its holding mode for its own reasons and stand aside. The script is then tested in SITL, with the serial sensor replaced by a simulated input, often a second script or a parameter the developer toggles. The tests cover the normal case, the case where the pilot takes control during the pause, the case where a failsafe triggers during the pause, the case where the sensor stops reporting, and the case where the reading stays high. Only when each behaves as intended does the script go onto real hardware, first with the aircraft flown in a large open area with an experienced pilot ready to take over, then on a representative mission. At each stage the logs are checked to confirm that the script's mode changes happened when and only when expected. None of these steps is exotic, and together they take far longer than writing the script itself. That ratio is normal and correct. MAVLink and companion computers MAVLink is the message protocol through which an autopilot communicates with ground stations, companion computers, cameras, gimbals and other devices. It defines a set of messages, each a compact binary structure with a numeric identifier: HEARTBEAT, sent once a second by every component to announce its presence and state; ATTITUDE and GLOBAL_POSITION_INT for telemetry; COMMAND_LONG for commands such as arming, takeoff or mode changes; SET_POSITION_TARGET_LOCAL_NED and its global equivalent for Guided mode targets; and many more. Messages carry the system and component identifiers of their sender and target, so several aircraft and devices can share a link. MAVLink 2, the current version, adds message signing to authenticate commands, which matters for any link where an unauthorised party might try to send commands. The protocol is lightweight enough to run over low-bandwidth telemetry radios, and both ArduPilot and PX4 implement it, along with ground stations, camera manufacturers and many other vendors. Several libraries make MAVLink easy to use from a program. pymavlink, the Python library maintained alongside ArduPilot, gives direct access to every message and is widely used for scripting tests and simple applications. The following lines connect to a simulated aircraft, switch it to Guided mode, arm it and command a takeoff to ten metres: from pymavlink import mavutil m = mavutil.mavlink_connection('udp:127.0.0.1:14550') m.wait_heartbeat() m.set_mode('GUIDED') m.arducopter_arm() m.motors_armed_wait() m.mav.command_long_send(m.target_system, m.target_component, mavutil.mavlink.MAV_CMD_NAV_TAKEOFF, 0, 0, 0, 0, 0, 0, 0, 10) MAVSDK, a higher-level library developed in the Dronecode ecosystem with bindings for C++, Python and other languages, offers task-level functions such as arm, take off, upload a mission and start offboard control, and works with both PX4 and, to a large extent, ArduPilot. DroneKit, an older high-level Python library once widely used with ArduPilot, still appears in many tutorials, but its development has largely stalled, and new projects are generally better served by pymavlink or MAVSDK. A companion computer is a separate, more powerful computer carried on the aircraft, typically a single-board computer such as a Raspberry Pi or an NVIDIA Jetson module, connected to the flight controller by a serial link or Ethernet. It runs the tasks the flight controller cannot: image processing, depth perception, mapping, path planning, machine learning and high-bandwidth communication. It talks to the autopilot over MAVLink, reading telemetry and sending Guided mode targets, external navigation estimates, obstacle distances and commands. The flight controller remains responsible for stabilisation, estimation and failsafes, so if the companion computer crashes or its software misbehaves, the aircraft can still stabilise itself and execute a failsafe. Many companion computer applications are built with the Robot Operating System, ROS, a middleware framework widely used in robotics research. For years, the standard bridge between ROS and MAVLink autopilots was MAVROS. More recently, ArduPilot has added native support for ROS 2 through the DDS communication standard, allowing the autopilot to publish and subscribe to ROS 2 topics directly through a lightweight agent on the companion computer. This makes it easier to integrate ArduPilot into the large ecosystem of ROS 2 perception and planning software. PX4 and the choice of platform PX4 is the other major open-source autopilot. It grew out of a research project at ETH Zurich around 2009 led by Lorenz Meier, and it is now hosted under the Dronecode Foundation, part of the Linux Foundation. Its architecture differs from ArduPilot's in several ways. PX4 is structured as a set of independent modules, each a separate task, that communicate through an internal publish-subscribe message bus called uORB. This makes it natural to replace or add modules and to run modules in separate threads. Its estimator, EKF2, is a descendant of the same lineage of work as ArduPilot's EKF3 and shares many ideas. Its default operating system on flight hardware is NuttX, a small POSIX-like real-time operating system. And it is licensed under the permissive BSD three-clause licence, which allows companies to modify it and ship the result without publishing their changes, whereas ArduPilot's GPL requires that modified firmware distributed to others be made available in source form. The licence difference shapes the ecosystems. PX4 is popular with companies building proprietary products on an open base, and with some research groups for its modular architecture. ArduPilot is popular with builders, survey and agricultural operators, and companies that value its breadth of supported hardware and vehicles, its mature failsafes and its community. Both are capable of professional work, both implement MAVLink, both have SITL simulators and integrate with ROS 2, and the control concepts in this book apply to both. The parameter names, mode names and some behaviours differ, and anyone moving between them should read the documentation for the platform at hand rather than assume equivalence. Whichever platform is chosen, the discipline is the same. Change one thing at a time. Test in simulation first. Keep slow or uncertain code out of the fast loop. Keep new behaviour at the highest layer that can do the job. And never forget that the code under development is the only thing keeping a fast-spinning, unstable machine in the air, which is why the last layer of the stack, the one that decides what to do when everything else goes wrong, deserves a chapter of its own. Chapter 9: Failsafes, Rules and the Engineering of Safety Every layer of the control stack makes promises to the layer above it, and every promise can be broken. A propeller cracks, and the rate loop can no longer deliver the torque the attitude loop expects. A battery sags, and the motors can no longer deliver the thrust the position controller assumes. The GNSS receiver loses its fix, and the estimator can no longer tell the position controller where the aircraft is. The radio link drops, and the pilot can no longer tell the aircraft what to do. A well-engineered aircraft does not assume these things will not happen. It decides in advance what it will do when they do. That decision is the failsafe system, and it is the top of the control stack in a different sense from navigation. Navigation decides where to go when everything is working. Failsafes decide what to do when something is not. This chapter describes the failsafes in ArduPilot, the checks that prevent an aircraft from taking off in an unsafe state, the geofences that keep it within bounds, and the regulatory framework that governs civilian drone flight in the United States and Europe. It treats safety as engineering, which means thinking about failure modes, their likelihood, their consequences, and the actions that reduce both. Checks before flight The cheapest failure to handle is one that is caught before takeoff. ArduPilot's pre-arm checks run continuously while the aircraft is disarmed and refuse to allow arming if any of a long list of conditions is not met. The checks include: the IMUs are calibrated, consistent with each other and not reporting excessive bias; the compass is calibrated, its readings are consistent with the expected field strength and with each other if there are several; the GNSS has an adequate fix in modes that require one; the EKF is healthy, its innovations small and its estimates consistent; the battery voltage is above its failsafe threshold; the radio is calibrated; the board voltage is within limits; the logging system is working; required parameters are set to sane values; and the geofence, if enabled, is loaded and the aircraft is inside it. A failed check is reported to the ground station with a message naming the cause. The checks can be disabled individually through ARMING_CHECK, and the temptation to do so when a check is blocking a flight is strong. It should almost always be resisted. A check that fails is almost always reporting a real problem, and the problems pre-arm checks catch, such as a compass that is badly affected by a nearby steel object or an EKF that has not converged, are exactly the problems that cause crashes a few seconds after takeoff. Experienced operators treat a pre-arm failure as information to be understood, not an obstacle to be removed. Beyond automated checks, a physical preflight inspection matters as much: propellers for chips and cracks, motors for free rotation, frame for loose fasteners, battery for swelling and correct charge, payload mountings, and antennas. Professional operations use written checklists for this, partly because they are required to and partly because checklists work. Failsafes in flight ArduPilot provides a set of failsafes, each triggered by a particular kind of failure and each configurable to take one of several actions. The actions form a hierarchy of severity, from continuing the mission, through holding position or returning home, to landing where the aircraft is, to cutting the motors in the extreme case of an aircraft that has left its permitted area and must be stopped regardless of what happens on the ground. The radio failsafe triggers when the aircraft stops receiving valid signals from the pilot's transmitter, configured with FS_THR_ENABLE. The ground station failsafe, FS_GCS_ENABLE, does the same for loss of the telemetry link to the ground station, which matters for aircraft flown entirely from a ground station or companion computer. The default response to either is usually RTL, return to launch: the aircraft climbs to RTL_ALT if it is below that height, flies straight home, descends and lands. A variant, SmartRTL, retraces the path the aircraft flew to reach its current position, which is safer where the direct path home is obstructed. For an aircraft already executing an autonomous mission, the operator may choose to let the mission continue rather than abort, if the mission is designed to finish safely without pilot input. The battery failsafe monitors voltage and, if a current sensor is fitted, consumed capacity. Two thresholds are configurable: a low threshold, BATT_LOW_VOLT and its capacity equivalent, with an action set by BATT_FS_LOW_ACT, typically RTL; and a critical threshold with its own action, typically Land. The thresholds should be set so that the low action leaves enough energy to get home from the farthest point the aircraft will fly, in the strongest headwind it will face, with a margin. Lithium batteries sag under load and recover at rest, so voltage thresholds should be set with the load in mind; capacity-based thresholds, using a current sensor, are more reliable. A worked example shows how the low threshold should be derived rather than guessed. Imagine a survey hexacopter whose current sensor shows it draws an average of 30 amps in cruise, carrying a 16,000 milliamp-hour pack, flying a mission whose farthest point is 1,500 metres from home, in a forecast wind of 6 metres per second. If it returns at an airspeed of 12 metres per second directly into that wind, its ground speed is 6 metres per second and the return leg takes 250 seconds. Add 60 seconds for the climb to return altitude, the descent and the landing, and the return needs a little over 300 seconds of flight, which at 30 amps is about 2,600 milliamp-hours. Lithium-polymer packs should not be routinely discharged close to empty, both for their health and because voltage collapses quickly near the end, so a reserve of perhaps 20 to 25 percent of capacity, here around 3,500 milliamp-hours, should remain on landing. The low capacity threshold therefore needs to be set at roughly 6,000 milliamp-hours remaining, not at the round number of 20 or 30 percent that a default might suggest, and the critical threshold somewhat below it. If the wind is stronger than forecast, or the aircraft is heavier, the numbers shift, which is why operators of long-range aircraft increasingly use failsafe logic that estimates the energy needed to return from the aircraft's current position rather than relying on a fixed threshold. The EKF failsafe triggers when the estimator's health degrades, when its innovations, the disagreement between what it predicted and what sensors report, exceed a threshold set by FS_EKF_THRESH for a sustained period. This usually indicates a compass or GNSS problem. The default action is to switch to a mode that does not require position, such as AltHold, or to land, because an aircraft whose estimator no longer knows where it is cannot safely fly home. An aircraft that triggers the EKF failsafe will drift with the wind unless the pilot takes manual control, which is a strong argument for every operator being capable of flying in a non-position mode. Other failsafes cover vibration high enough to corrupt the estimator, in which case ArduPilot switches to a more vibration-resistant method of estimating vertical velocity; loss of a companion computer's heartbeat in Guided mode; the detection of a crash, when the aircraft finds itself at an extreme attitude and not moving and disarms to protect the motors and people nearby; and the loss of a motor on aircraft with more than four, where ArduPilot can detect the failure and redistribute thrust across the remaining motors. Hexacopters and octocopters are often chosen for professional work precisely because they can survive the loss of a motor, which a quadcopter cannot. Parachute recovery systems, triggered automatically by the autopilot or manually by the pilot, are increasingly used on aircraft that fly over people or carry heavy payloads, and some regulatory frameworks treat them as a key mitigation for operations over populated areas. ArduPilot supports parachute release, with checks that the aircraft is high enough for the parachute to deploy and that the motors are stopped before release. Geofences A geofence is a virtual boundary the aircraft will not cross. ArduPilot supports several types, enabled with FENCE_ENABLE and selected with FENCE_TYPE: a maximum altitude, FENCE_ALT_MAX; a cylinder of radius FENCE_RADIUS around the home position; and arbitrary inclusion and exclusion polygons and circles uploaded from a ground station. If the aircraft breaches a fence, it takes a configured action, typically RTL or Land. In pilot-controlled modes with simple avoidance enabled, the aircraft stops at the fence boundary rather than crossing it, as described in Chapter 7. Fences enforce operational limits in a way that does not depend on the pilot's attention. An operator flying a survey near an airport's controlled airspace can set a fence along the boundary; an operator flying near a road can exclude it; an operator flying a demonstration can keep the aircraft away from the audience. Fences are also a natural way to encode the operational volume and buffers that regulators increasingly expect for complex operations. A fence does not make an aircraft safe on its own, since an aircraft with a failed estimator may not know where the fence is, but as one layer among several it catches a large class of errors. The regulatory frame Civilian drone flight is regulated in almost every country, and an engineer building or programming drones needs to understand the rules that apply, because they shape what the aircraft must be able to do. In the United States, the Federal Aviation Administration regulates small unmanned aircraft under 25 kilograms through Part 107 of Title 14 of the Code of Federal Regulations for commercial operations, with a separate framework for recreational flyers, who must pass a short knowledge test and follow community safety guidelines. Part 107 requires a remote pilot certificate obtained by passing a knowledge test, limits flights to below 400 feet above ground level in most circumstances, requires the aircraft to remain within the pilot's visual line of sight, and restricts flight over people and at night unless specific conditions are met. Rules for flight over people and at night were expanded in 2021, with categories of aircraft defined by the injury they could cause. Operations outside these rules, such as beyond visual line of sight, have required waivers granted case by case. In August 2025, the FAA published a proposed rule, Part 108, that would establish a performance-based framework for routine beyond-visual-line-of-sight operations, with requirements on the aircraft, the operator and the management of air traffic risk, to replace much of the waiver process for activities such as delivery, agriculture and infrastructure inspection. The FAA's Remote ID rule, Part 89, requires most drones to broadcast their identity, position, altitude, velocity and the location of their control station during flight, either through built-in broadcast or through an add-on module. Compliance was required from September 2023, with the FAA giving operators until March 2024 before enforcement. ArduPilot supports Remote ID through an open-source module design and through commercial modules, and Remote ID compliance has become a standard consideration in the design of any aircraft sold or flown in the United States. In the European Union, drone operations are governed by common rules adopted in 2019, which have applied across member states since the end of 2020. They divide operations into three categories by risk. The open category covers low-risk operations within visual line of sight, below 120 metres, with aircraft classified by class marks C0 to C4 according to their mass and safety features, and with subcategories that define how close to people the aircraft may fly. The specific category covers higher-risk operations, including most beyond-line-of-sight flights, and requires an operational authorisation based on a risk assessment, usually carried out using the Specific Operations Risk Assessment methodology, known as SORA, or by following a published standard scenario. The certified category covers the highest-risk operations, such as carrying people or flying large aircraft over crowds, and requires certification of the aircraft, operator and pilot on terms closer to those of crewed aviation. Network and direct Remote ID requirements also apply in Europe for many aircraft. Other countries have their own regimes, broadly similar in principle: registration, pilot competence, altitude limits, line of sight requirements in the basic category, and a risk-based path to more complex operations. Anyone flying professionally needs to know the specific rules where they fly, which change frequently and are best checked directly with the national aviation authority. Safety as a system The engineering lesson of this chapter, and in a sense of this book, is that safety is not a feature but a property of the whole system. A drone does not become safe because it has a return-to-launch button, an obstacle sensor or a parachute. It becomes safe through a chain: a sound airframe, clean sensor data, well-tuned control loops, a trustworthy estimator, sensible navigation, honest avoidance, careful failsafe configuration, a competent operator and a mission planned within the aircraft's capabilities and the rules that apply. Each link depends on the ones before it. A failsafe that triggers RTL is useless if the estimator does not know where home is. A battery failsafe set to a voltage that leaves too little energy to get home will trigger and still lose the aircraft. An avoidance system that cannot see wires gives false confidence near power lines. The risk-based approach that regulators have adopted, epitomised by SORA, asks operators to think in exactly this way: identify the ways an operation can go wrong, estimate the consequences, and apply mitigations until the residual risk is acceptable. For the engineer, that means configuring each failsafe deliberately, testing it in simulation and then on the aircraft, understanding what the aircraft will do in each failure and whether that is what the operator wants, and documenting it. The flight control stack provides the tools. Using them well is a matter of discipline, and of understanding, at every layer, what promise is being made and what happens when it is broken. Conclusion A quadcopter is a machine that cannot fly without software, and the software that flies it has a shape. At the bottom, a rate loop running hundreds of times a second turns gyroscope readings into motor commands. Above it, an angle loop turns attitude errors into rate requests. Above that, a position controller turns position and velocity errors into attitude requests, a navigation layer turns missions into smooth trajectories, an avoidance layer bends those trajectories around what the sensors can see, and a failsafe layer decides what to do when any of this breaks. Feeding all of it, an estimator weighs the testimony of a handful of imperfect sensors and decides, continuously, where the aircraft is and which way it is facing. The value of seeing the system this way is practical rather than theoretical. It tells the builder what to do first and what to leave until later. It tells the pilot where to look when the aircraft misbehaves. It tells the developer where new code can safely go. Three consequences in particular follow from the argument of the preceding chapters, and they are worth stating as rules. Work from the bottom up The first rule is that problems should be fixed in the layer that owns them, and that the layers should be made sound in order from the bottom. A drone whose position hold wanders cannot be cured by retuning its rate loop, and a drone whose rate loop oscillates cannot be cured by slowing its waypoint speed. But the lower layers constrain the upper ones absolutely. If vibration corrupts the gyroscope signal, the rate loop cannot be tuned properly; if the rate loop is sluggish, the angle loop will overshoot; if the compass is wrong, the position controller will circle; if the position estimate is noisy, the trajectory generator's smooth path will be followed roughly. The order of work in Chapter 4, mechanics before vibration, vibration before gains, rate loops before angle loops, angle loops before position, is not a convention. It follows from the way each layer depends on the ones beneath it. This rule applies equally to diagnosis. When something goes wrong in flight, the most productive question is not "what parameter should I change?" but "which layer's promise was broken?" The logs can usually answer it. Rate targets and achieved rates show whether the rate loop delivered. Attitude targets and estimates show whether the angle loop delivered. EKF innovations show whether the estimator trusted its sensors. Position targets and estimates show whether the position controller delivered. Following the chain from the symptom down to the first broken promise finds the cause far faster than adjusting parameters by intuition. Trust the estimate only as far as its inputs The second rule is that every autonomous behaviour is, in the end, a bet on the estimator. Waypoint navigation, terrain following, geofencing, return to launch, precision landing and map-based avoidance all assume that the aircraft knows where it is. When it does, these features are remarkably capable. When it does not, they become dangerous in proportion to their capability, because they will confidently fly the aircraft to the wrong place. A failsafe that returns home is only safe if home is where the estimator thinks it is. An avoidance system that plans around mapped obstacles is only safe if the map and the aircraft agree about where the aircraft is within it. The practical upshot is that the navigation sensors and their configuration deserve as much attention as the control gains, and usually get less. Compass placement and calibration, GNSS antenna placement and quality, barometer shielding, vibration isolation for the accelerometers, and careful configuration of alternative sources for GNSS-denied flight are where many navigation failures begin. Operators who fly in difficult places, near steel structures, under bridges, in areas of known GNSS interference, should treat the estimator's health as the first thing to monitor in flight and should know how to fly the aircraft by hand in a mode that does not depend on it. Keep new behaviour high in the stack The third rule concerns change. Open-source autopilots invite modification, and the layered architecture makes modification safe if it respects the layers. New mission behaviour belongs in mission commands, scripts or a companion computer sending Guided mode targets, not in the attitude controller. New sensors belong behind the estimator's interfaces, not wired directly into the position controller. New avoidance logic belongs in the layer between navigation and position control, where it can modify targets without bypassing the controllers that execute them. The lower the layer, the higher the bar for changing it, and the inner loops should be changed only by people who understand them, after extensive simulation, and with a clear plan for what happens if the change misbehaves in flight. This rule is also why the simulator matters so much. Running the real flight code against a simulated aircraft costs nothing and catches the logic errors, sign errors and mode transition bugs that cause most crashes after software changes. Every change should fly in simulation before it flies for real, and every failsafe should be triggered deliberately in simulation so that its behaviour is known before it is needed. What remains unsettled Some parts of the stack are mature. Rate and attitude control of multirotors is, for practical civilian purposes, a solved problem; well-configured autopilots fly stably and precisely on an enormous variety of airframes, and automatic tuning works well for most of them. GNSS-based navigation in open sky is reliable and, with RTK, precise enough for survey-grade work. Other parts are still moving. Navigation without satellites is improving quickly but remains fragile, especially at transitions between sources and over long distances. Onboard obstacle sensing still struggles with thin wires, transparent surfaces and non-cooperative aircraft, and this limits what can be done beyond visual line of sight without heavy reliance on airspace management and procedure. The regulatory frameworks for routine beyond-line-of-sight operations in the United States and Europe are taking shape around risk-based principles, but the details, and how they will interact with the technical capabilities of detect-and-avoid systems, are still being worked out. Machine learning is entering perception and, in research, control, and how to verify that a learned component behaves safely across the full range of conditions an aircraft meets is an open question. What will not change is the underlying structure. However capable the upper layers become, they will still rest on an estimator that must decide which sensors to believe, and on control loops that must turn requests into motor commands faster than an unstable machine can fall. The reader who understands that structure, who can find the layer that owns a problem, read the logs that show which promise was broken, and add new behaviour where it will not disturb the loops beneath, is equipped not just to tune today's aircraft but to understand whatever autonomy is built on top of them next. Notes Chapter 2 Robert Mahony, Tarek Hamel and Jean-Michel Pflimlin, "Nonlinear Complementary Filters on the Special Orthogonal Group," IEEE Transactions on Automatic Control 53, no. 5 (2008): 1203–1218. Sebastian O. H. Madgwick, "An Efficient Orientation Filter for Inertial and Inertial/Magnetic Sensor Arrays," technical report, University of Bristol, 2010. Chapter 3 Nicolas Minorsky, "Directional Stability of Automatically Steered Bodies," Journal of the American Society of Naval Engineers 34, no. 2 (1922): 280–309. J. G. Ziegler and N. B. Nichols, "Optimum Settings for Automatic Controllers," Transactions of the ASME 64 (1942): 759–768. Chapter 6 Daniel Mellinger and Vijay Kumar, "Minimum Snap Trajectory Generation and Control for Quadrotors," Proceedings of the IEEE International Conference on Robotics and Automation (2011): 2520–2525. Chapter 7 Johann Borenstein and Yoram Koren, "The Vector Field Histogram: Fast Obstacle Avoidance for Mobile Robots," IEEE Transactions on Robotics and Automation 7, no. 3 (1991): 278–288. Paolo Fiorini and Zvi Shiller, "Motion Planning in Dynamic Environments Using Velocity Obstacles," International Journal of Robotics Research 17, no. 7 (1998): 760–772. Armin Hornung, Kai M. Wurm, Maren Bennewitz, Cyrill Stachniss and Wolfram Burgard, "OctoMap: An Efficient Probabilistic 3D Mapping Framework Based on Octrees," Autonomous Robots 34, no. 3 (2013): 189–206. ASTM International, ASTM F3442/F3442M, Standard Specification for Detect and Avoid System Performance Requirements. Chapter 8 Lorenz Meier, Dominik Honegger and Marc Pollefeys, "PX4: A Node-Based Multithreaded Open Source Robotics Framework for Deeply Embedded Platforms," Proceedings of the IEEE International Conference on Robotics and Automation (2015): 6235–6240. Chapter 9 Federal Aviation Administration, 14 CFR Part 107, Small Unmanned Aircraft Systems; 14 CFR Part 89, Remote Identification of Unmanned Aircraft. Federal Aviation Administration, "Normalizing Unmanned Aircraft Systems Beyond Visual Line of Sight Operations," notice of proposed rulemaking, published in the Federal Register, August 2025. Commission Implementing Regulation (EU) 2019/947 on the rules and procedures for the operation of unmanned aircraft; Commission Delegated Regulation (EU) 2019/945 on unmanned aircraft systems and on third-country operators of unmanned aircraft systems. Joint Authorities for Rulemaking on Unmanned Systems (JARUS), Specific Operations Risk Assessment (SORA). Further Reading ArduPilot documentation (ardupilot.org). The Copter wiki is the authoritative reference for every parameter, flight mode and failsafe named in this book, and its tuning process guide, vibration and harmonic notch pages, and EKF source selection pages are the best practical companions to Chapters 4 and 5. The developer section covers building the firmware, SITL, Lua scripting and the codebase's libraries. PX4 User Guide and Developer Guide (docs.px4.io). The equivalent references for PX4, with clear explanations of its controller architecture, uORB messaging and offboard control. MAVLink Developer Guide (mavlink.io). The definitive description of the protocol, its messages, its mission and command microservices, and message signing. Karl Johan Åström and Richard M. Murray, Feedback Systems: An Introduction for Scientists and Engineers** (Princeton University Press, 2008; second edition 2021). A lucid, freely available introduction to feedback control, covering stability, frequency response and PID control with physical intuition rather than heavy formalism. Karl Johan Åström and Tore Hägglund, Advanced PID Control** (ISA, 2006). The standard practical reference on PID controllers, including derivative filtering, anti-windup, feedforward and tuning methods. Randal W. Beard and Timothy W. McLain, Small Unmanned Aircraft: Theory and Practice** (Princeton University Press, 2012). A textbook treatment of modelling, sensing, state estimation, control and path planning for small aircraft, written for engineering students. It focuses on fixed-wing aircraft, but its chapters on sensors, estimation and guidance apply directly to multirotors. Robert Mahony, Vijay Kumar and Peter Corke, "Multirotor Aerial Vehicles: Modeling, Estimation, and Control of Quadrotor," IEEE Robotics and Automation Magazine 19, no. 3 (2012). A concise, authoritative tutorial on quadrotor dynamics, including blade flapping and rotor drag, and on attitude estimation and control. Paul D. Groves, Principles of GNSS, Inertial, and Multisensor Integrated Navigation Systems** (Artech House, second edition 2013). The comprehensive reference on satellite navigation, inertial sensors and the Kalman filter integration that underlies modern autopilot estimators. Sebastian Thrun, Wolfram Burgard and Dieter Fox, Probabilistic Robotics** (MIT Press, 2005). The classic text on probabilistic state estimation, mapping and localisation, including Kalman filters, occupancy grids and SLAM, essential background for GNSS-denied navigation and map-based avoidance. Steven M. LaValle, Planning Algorithms** (Cambridge University Press, 2006). A thorough treatment of motion planning, including graph search and sampling-based planners such as rapidly exploring random trees, and freely available online from the author. Federal Aviation Administration and European Union Aviation Safety Agency websites (faa.gov/uas and easa.europa.eu). The primary sources for current rules on drone operation, Remote ID, beyond-visual-line-of-sight operations and the specific category, which change frequently enough that the regulators' own guidance should be checked before any professional operation. Hashtags: #DroneFlightControl #FlightControlSystems #PIDControl #PIDTuning #AutonomousNavigation #DroneTechnology #QuadcopterControl #UAVEngineering #DroneAutopilot #ArduPilot #PX4Autopilot #Betaflight #ControlSystemsEngineering #FeedbackControl #CascadedPIDControl #AttitudeControl #FlightStabilization #SensorFusion #ExtendedKalmanFilter #InertialNavigation #GNSSNavigation #RTKPositioning #WaypointNavigation #TrajectoryPlanning #ObstacleAvoidance #DroneSensors #HarmonicNotchFiltering #FlightControllerTuning #MAVLink #DroneSafety

Download the Book (PDF): Introduction On a clear night in the right city, you can watch several thousand drones assemble a rotating globe in the sky, dissolve it, and redraw it as a dragon. The images that circulate afterwards are usually captioned with the word "swarm." It is the wrong word. Every one of those aircraft is flying a trajectory computed in advance on a ground computer, timed to a shared clock, and steered by satellite positioning corrected to within a few centimetres. If the positioning fails, the show stops. If a drone loses its link, it lands. Nothing in the sky is deciding anything. A drone light show is a remarkable piece of choreography, and it is a useful place to begin precisely because it is the opposite of what this book is about. A swarm, in the sense used by roboticists, is a group of machines whose collective behaviour emerges from local decisions. Each member senses a small part of the world, talks to a few neighbours, and acts on what it knows. There is no master copy of the plan. There may be a human supervisor, and there may be goals handed down from above, but the moment-to-moment work of staying together, avoiding one another, dividing up a search area and replacing a member that has dropped out is done by the members themselves. The interest of such systems is not aesthetic. It is that they can keep working when the things a light show depends on, the satellite fix, the ground station, the uninterrupted radio link, are gone. That is the situation in most of the places where a team of drones would do real good. Inside a collapsed parking structure after an earthquake, satellite signals do not reach. In a steep forested valley where a hiker has been missing for two days, the canopy blocks both satellite positioning and line-of-sight radio. In a mine, a tunnel, a wildfire's smoke column, or a flooded town whose cellular towers have lost power, the infrastructure that ordinary drone operations take for granted is partial or absent. The engineering question is not how to make many drones fly together under perfect conditions. It is how to make a group of drones remain useful when each of them knows only a little, can tell its neighbours only some of what it knows, and cannot count on that conversation lasting. The argument of this book The single idea that runs through every chapter is this: a drone swarm's intelligence is bounded by its network. Decentralized control is not a style borrowed from birds because birds are elegant. It is the engineering consequence of operating where communication, positioning and information are partial. Once you accept that the network is the binding constraint, the three technical subjects named in this book's subtitle stop looking like separate disciplines. Flocking algorithms are rules for acting well on local, delayed, incomplete information about neighbours. Mesh networking is the machinery that determines which neighbours, how delayed and how incomplete. Distributed task allocation is the problem of reaching a good enough division of labour when no one can see the whole board and the board keeps changing. Each of these is a different face of the same problem, and a swarm that is designed well treats them together, so that when the network degrades, the behaviour, the communication and the division of work degrade gracefully together rather than failing separately. This framing has practical consequences, and the book tries to draw them out. It means that a flocking controller is only as good as its tolerance for the communication delays the real radio will impose, which is one reason the first outdoor flocks of the 2010s needed careful parameter tuning before they flew well. It means that the cheapest way to improve a swarm's search performance is sometimes to change how it routes messages, or what it chooses not to send, rather than to improve its cameras. It means that a task allocation algorithm proved optimal under full communication may be worse in the field than a crude one that degrades predictably. And it means that the people supervising a swarm need to understand its network state as much as its positions, because what the swarm can do next depends on who can still hear whom. What the chapters do The first chapter establishes what a swarm is and is not, why one would decentralize at all, and what decentralization costs. It argues that a swarm is justified only by specific properties, robustness to loss, scalability, and coverage, and that many problems are better solved by one capable aircraft or by a centrally planned fleet. The second chapter turns to the living collectives that inspired the field: starling murmurations, fish schools, foraging ants and house-hunting honeybees. It concentrates on what biologists have actually measured about how animals coordinate, including the finding that starlings track a fixed number of neighbours rather than all neighbours within a given distance, and on which biological lessons survive translation into engineering. The third chapter follows flocking from Craig Reynolds' 1987 computer-graphics model through control theory to drones flying in forests. It explains the mathematics of consensus in plain terms, the reasons outdoor flocking proved harder than simulation suggested, and how collision avoidance is made safe when every agent is moving. The fourth chapter deals with positioning when satellite navigation is unavailable or untrustworthy: visual-inertial odometry, ultra-wideband ranging between drones, and the collaborative mapping methods proved under pressure in the DARPA Subterranean Challenge. Its theme is that in a swarm, knowing where you are is a shared and negotiated fact, and the negotiation runs over the network. The fifth chapter is about that network. It describes flying ad hoc networks, the radio physics that make them difficult, the main families of routing protocol, and a result from information theory that sets a ceiling on how much any individual node in a large wireless network can say. The sixth chapter accepts that links will fail and examines how swarms are designed to tolerate it: maintaining connectivity deliberately, dropping relay nodes, carrying data physically when it cannot be transmitted, and budgeting the scarce bandwidth that remains. The seventh chapter addresses distributed task allocation: how a swarm decides which drone searches which sector, investigates which signal, or returns to recharge, using methods ranging from insect-inspired response thresholds to market auctions and consensus-based bundle algorithms. The eighth chapter brings everything together in search and rescue, the application that most clearly rewards what swarms are good at, and then looks more briefly at mapping, agriculture and environmental research. The ninth chapter deals with trust: how swarms fail, how they can be verified, how they can be attacked, how they are regulated, and how public policy is treating the fact that the same technologies have military uses. Scope and stance This book focuses on civilian work: search and rescue, mapping and surveying, agriculture, infrastructure inspection and scientific research. Swarm technologies are undeniably dual-use, and military investment has shaped the field. Where that matters, it is discussed at the level of public policy and international law. The book does not discuss the design of weapons or tactics, and nothing in it is intended to help anyone build them. The reader the book has in mind is an intelligent non-specialist with some appetite for technical ideas: an engineer from an adjacent field, an emergency-services manager evaluating what drones can do, a policy adviser, a student deciding where to specialize, or a researcher in one corner of the subject who wants the whole picture. No mathematics beyond the ability to follow an argument about averages and graphs is assumed. Where the text names a protocol, an algorithm or a study, it does so because the specifics matter and because the reader may want to look them up. The field moves quickly, and several of its regulatory questions are unsettled as this book is written in late 2026. The United States rule intended to normalize routine flight beyond the operator's visual line of sight, proposed in August 2025, was still under final review at the White House in the autumn of 2026. International talks on autonomy in weapons systems have reached a point of decision without yet producing a treaty. Research systems that were laboratory curiosities a decade ago now fly through forests unaided. What does not move as quickly are the underlying constraints: radio waves still attenuate through concrete, batteries still hold limited energy, and a group of machines still cannot agree on anything faster than they can exchange messages. Those constraints are the foundation of everything that follows. CHAPTER 1 What Makes a Swarm The word "swarm" is used so loosely that it has almost stopped meaning anything. Marketing material applies it to any group of more than two drones. News reports apply it to formations that are flown by a single pilot through a scripted sequence. Researchers themselves disagree at the edges. Before anything useful can be said about decentralized control, the term has to be pinned down, and the reasons for wanting a swarm, as opposed to a fleet or a single machine, have to be stated honestly, including the reasons not to want one. Three ways to fly many drones There are three broad architectures for operating many drones at once, and the differences between them explain most of what follows in this book. The first is centralized choreography. A ground computer knows where every drone is and where it should be, computes every trajectory, and either uploads those trajectories in advance or streams commands in real time. Drone light shows are the most visible example. Intel's display at the opening of the 2018 Winter Olympics in Pyeongchang used 1,218 drones, a record at the time; by 2025, shows in China and Vietnam had pushed past ten thousand aircraft. Each drone in such a show is essentially a pixel. It follows a timeline, holds its position with real-time kinematic satellite corrections, and has no knowledge of its neighbours beyond what the timeline implies. The architecture is powerful because a central planner can optimize globally: it can guarantee that trajectories never intersect and that every drone arrives exactly on cue. Its weakness is that every drone's competence depends on the ground station, the positioning system and the link between them. The second is the coordinated fleet. Here each drone has substantial onboard autonomy, it can fly a route, avoid an obstacle, land itself, but tasks are assigned by a central mission planner and the drones report back to it. Most commercial multi-drone operations today work this way. A surveying company may launch four mapping drones, each covering a quarter of a site, with the ground software dividing the area and stitching the results. A fleet tolerates the loss of an individual aircraft well, because the planner simply reassigns its area. It tolerates the loss of the planner, or of communication with the planner, poorly. The third is the swarm proper. Each member runs the same, or nearly the same, control software. Each decides its own actions from what it senses directly and what it hears from neighbours within radio range. There is no single node whose loss halts the group. Behaviour at the level of the whole, a coherent formation, an even spread over a search area, a division of labour, emerges from the local rules. Marco Dorigo, one of the founders of swarm robotics, and his colleagues have described the defining features in roughly these terms: a relatively large number of robots, which are largely homogeneous, which act on local sensing and local communication, and which together produce collective behaviour that no individual is programmed to produce directly. In practice, real systems mix these architectures. A search-and-rescue team may launch a swarm whose members coordinate locally but which also streams imagery to an incident commander whenever a link is available, and which accepts high-level instructions such as "prioritize the river corridor." That hybrid is not a dilution of the swarm idea; it is how swarm ideas are most likely to be used. The useful question is not whether a system is a "true" swarm but which decisions are made locally, which are made centrally, and what happens to each kind of decision when the network fails. Why decentralize at all Decentralized control is harder to design, harder to predict and harder to certify than central control. It needs a justification. Swarm robotics has traditionally offered three: robustness, scalability and flexibility. Each is real, and each has limits that are often glossed over. Robustness means that the system keeps working when parts of it fail. If no member is special, the loss of any member is a degradation rather than a collapse. A search swarm of twelve drones that loses two to battery faults is a search swarm of ten. This is the property that matters most in the environments this book is concerned with, because those environments are hard on equipment. Drones crash into branches, lose motors to debris, and drift out of radio contact behind walls. A design in which the loss of the lead aircraft or the ground station stops the mission is fragile in exactly the conditions where reliability is most needed. The limit of this argument is that robustness to member loss is not the same as robustness in general. A swarm whose members all run identical software can fail all at once from a single software defect, a single bad parameter, or a single environmental condition that confuses every member's sensors in the same way. Homogeneity buys tolerance of random, independent faults and exposes the group to common-mode faults. Scalability means that the system works with ten members and with a thousand without being redesigned. The classic demonstration is the Kilobot project at Harvard. In 2014, Michael Rubenstein, Alejandro Cornejo and Radhika Nagpal reported in Science that 1,024 small, cheap robots, each able to communicate only with neighbours within about three body lengths, could assemble themselves into shapes such as a star and the letter K without any central controller directing individual robots. The process was slow, taking hours, and it was error-prone, but it scaled: the same algorithm ran on every robot regardless of how many there were. Scalability is attractive for drones because the costs of central coordination, computing trajectories for every pair of aircraft and maintaining a link to every one of them, grow faster than the number of aircraft. A local rule that consults only nearby neighbours has a cost per drone that stays roughly constant as the swarm grows. The limit is that scalability of computation does not imply scalability of everything else. As the fifth chapter shows, the capacity of a shared wireless channel does not grow with the number of transmitters, and a swarm that relies on sharing data can saturate its own spectrum long before its algorithms run out of steam. Flexibility means that the system adapts to changing conditions and tasks without being reprogrammed. A swarm that is spreading out to search can contract to investigate a detection, then spread out again. A swarm in which a member's battery runs low can redistribute that member's search area without being told to. This flexibility is a natural consequence of agents repeatedly reacting to their local situation rather than executing a fixed plan. The limit is predictability. A system that adapts on its own can surprise its operators, and surprise is precisely what aviation regulators, incident commanders and insurers dislike. There is also a fourth argument, which the founders of swarm robotics made less often but which is now among the most persuasive: the decentralized design matches the communication environment. In a tunnel, a forest or a damaged building, any architecture that relies on a continuous link to a central node will spend much of its time broken. A design that assumes local, intermittent communication from the start is not taking a philosophical position. It is matching the system to the radio physics. The economics of many small drones Even granting those arguments, why many drones rather than one better drone? The answer has several strands, and they do not all point the same way. The first strand is coverage. The time to search an area with downward-facing sensors is roughly the area divided by the product of sensor swath width, speed and the number of searchers. Doubling the number of aircraft halves the search time, provided they do not duplicate one another's work. In search and rescue, where survival probabilities fall steeply with time, this is the central attraction. A single large drone with a long endurance can fly for hours, but it can be in only one place at a time. The second strand is perspective. Many problems are easier with simultaneous views from several positions. Three-dimensional reconstruction of a structure benefits from images taken at the same moment from different angles. Locating a radio beacon, such as an avalanche transceiver or a mobile phone, is far faster when several receivers measure signal strength at once from different positions than when a single receiver must move around to triangulate. The third strand is size. Small drones can go places large ones cannot: through windows, into culverts, between trees. Small drones are also less dangerous. The kinetic energy of a 250-gram aircraft falling onto a person is very different from that of a 25-kilogram aircraft, and regulators in both the United States and Europe draw their lightest categories of operation around that difference. If a task can be decomposed into pieces small enough for small drones, a swarm may be able to do in confined or populated spaces what no single large aircraft could be permitted to do. The fourth strand is redundancy as a cost strategy. A single drone that must not fail needs redundant motors, redundant flight controllers and certified components. A swarm can accept that individual members fail occasionally, which allows each member to be cheaper and simpler. This is the logic of the Kilobot, which cost about fourteen dollars in parts per robot. Against these strands sits a set of costs. Small drones carry small batteries; endurance for small multirotors is often measured in tens of minutes. Small drones carry small sensors, and the best thermal cameras, laser scanners and radio direction finders are heavy. Small drones are blown about by wind that a larger aircraft would shrug off. Every drone in a swarm must be charged, maintained, transported and, depending on jurisdiction, registered. And every additional drone adds radio traffic to a shared channel. The right size and number of drones for a task is an engineering trade-off, not a matter of "more is better," and some of the most honest work in the field consists of showing where a swarm loses to a single capable aircraft. When a swarm is the wrong answer It is worth being explicit about the situations where swarms are not justified, because enthusiasm for the idea has led to proposals where the decentralized architecture adds complexity without adding value. If the environment provides reliable positioning and reliable communication, and the task can be planned in advance, central planning will usually do better. A field survey over open farmland with good satellite coverage and a clear radio path to a ground station is a planning problem, not a swarm problem. A central planner can divide the field into strips, compute efficient paths, and monitor progress. Nothing is gained by asking the drones to negotiate among themselves what a laptop can compute in a second. If the task depends on a single expensive sensor, a swarm of small drones may be unable to carry it at all. A crewed helicopter with a high-end thermal imaging system and a trained observer remains, in many conditions, a formidable search platform, and the question of whether a swarm of small drones can match it is empirical, not ideological. If the operation must be predictable in detail, for example because it takes place in shared airspace near people, the emergent character of swarm behaviour can be a liability. Regulators need to know what an aircraft will do, and "it will do something sensible based on what its neighbours are doing" is not an answer an aviation authority will accept without extensive evidence. This does not rule swarms out, but it raises the bar for evidence, a subject returned to in the ninth chapter. Finally, if the people who will operate the system cannot understand its state, the system will not be trusted, and an untrusted system is not used. Emergency services already work under heavy cognitive load. A tool that adds a new class of uncertainty, "why are the drones doing that?", may not survive contact with an incident command post, however good its algorithms. The cases where a swarm is the right answer are therefore those where at least some of the following hold: the area to be covered is large relative to what one aircraft can cover in the time available; simultaneous observation from multiple points matters; the environment degrades positioning or communication enough that central control would be unreliable; the platforms can be small enough to be cheap and relatively safe; and the task can be broken into pieces that do not each require a large sensor. Search and rescue after disasters, especially in built-up or forested terrain, meets many of these conditions, which is why it serves as this book's main application. Emergence, and why it is not magic Discussions of swarms often reach for the word "emergence" as though it explained something. It does not, by itself. It names the observation that a group can do things none of its members is programmed to do. A flock turns as a body although no bird decides that the flock should turn. A colony of ants finds the shortest of several paths to food although no ant compares the paths. For an engineer, emergence is a design problem with a specific shape. You want a particular collective outcome. You can program only individual behaviour. You must therefore find individual rules whose combined effect, across the range of conditions the system will meet, is the outcome you want, and you must be able to show that it is. There are two general strategies. The first is to work from insight or inspiration, often biological, to propose rules, and then test them in simulation and on hardware. Most of the classic work in the field followed this path. The second is to specify the collective outcome mathematically and derive individual rules that provably achieve it, which is the program of distributed control theory. Increasingly, a third strategy is used alongside both: search automatically over a space of possible rules, by evolutionary optimization or machine learning, for those that perform well in simulation, then transfer them to hardware. The flocking drones of Gábor Vásárhelyi and colleagues, described in the third chapter, were tuned this way. All three strategies share a difficulty that is the deepest problem in swarm engineering: the gap between the micro level, where you write the code, and the macro level, where you judge the result. Small changes to individual rules can produce large changes in collective behaviour, and the relationship is often non-linear. A flocking rule that produces a smooth, cohesive group at one communication delay may produce oscillation, fragmentation or collisions at a slightly longer delay. Heiko Hamann's textbook on swarm robotics treats this micro-macro problem as the central theoretical challenge of the field, and it recurs in every chapter of this book. The network is where the micro-macro gap is widest. The rules running on each drone can be specified exactly. The pattern of who hears whom, with what delay and what loss, depends on the positions of all the drones, the terrain, the weather, the radio hardware and the traffic on the channel. That pattern changes from second to second, and the collective behaviour depends on it. A swarm designer who treats communication as an idealized given, every drone instantly hearing every neighbour within some fixed radius, will produce a design that works in simulation and surprises everyone in the field. The remaining chapters are, in large part, an account of how the field learned this and what it has done about it. A working definition For the rest of this book, a drone swarm means a group of aerial robots in which the decisions that keep the group functioning, staying safely separated, maintaining useful spatial arrangement, sharing information, and dividing work, are made by the members on the basis of local sensing and local communication, such that the group continues to function, in degraded form, when any individual member, link, or external service is lost. That definition admits human supervision and centrally specified goals. It excludes light shows and centrally planned fleets, not because those are unimportant but because they solve a different problem. The definition also implies a standard for judging any swarm system: what does it do when things go wrong? A swarm that performs beautifully with perfect links and perfect positioning, and fails abruptly without them, has gained nothing from decentralization except complexity. A swarm whose performance falls smoothly and predictably as links drop, positioning degrades and members are lost has earned the name. The chapters that follow are organized around that standard. The next one begins where the field itself began, by looking closely at animals that solved these problems long before engineers attempted them. CHAPTER 2 Lessons from Living Collectives Swarm robotics borrowed its name, much of its vocabulary and many of its first algorithms from biology. That borrowing has been productive and also misleading. Productive, because animal groups demonstrate that coordination among many agents with limited perception and no leader is possible, and they suggest mechanisms. Misleading, because a casual reading of biology encourages the idea that simple rules are enough, when the animals that use those rules bring sensing, redundancy and tolerance of loss that no drone engineer can assume. This chapter looks at what biologists have actually measured about how animals coordinate, and at which of those findings survive translation into machines that must communicate by radio. Starlings and the number seven Murmurations of European starlings, tens of thousands of birds wheeling over a roost at dusk, are the canonical image of collective motion. For a long time, the question of how each bird decides what to do was answered by plausible models rather than measurement, because tracking individual birds in three dimensions inside a dense flock was beyond available technology. That changed with the STARFLAG project, a European collaboration led by physicists in Rome, which used synchronized high-speed cameras on a building overlooking a starling roost to reconstruct the three-dimensional positions of every bird in flocks numbering up to a few thousand. In 2008, Michele Ballerini and colleagues published the central finding in the Proceedings of the National Academy of Sciences. They asked whether a starling's movement was influenced by all neighbours within a fixed distance, which is how most models had assumed interaction worked, or by a fixed number of nearest neighbours regardless of distance. The data favoured the second. Each bird's interactions were best explained by its six or seven nearest neighbours, whether those neighbours were close together in a dense part of the flock or spread out in a sparse part. The authors called this topological interaction, in contrast with the metric interaction of the earlier models. The finding has a clear functional interpretation, and the authors tested it. Starling flocks change density constantly, compressing and expanding as they turn and as predators such as peregrine falcons attack. If each bird responded to all neighbours within a fixed radius, a bird at the edge of a suddenly expanded region might find no neighbours at all and lose contact with the group. If each bird tracks a fixed number of neighbours, it always has someone to follow, and the flock remains cohesive across a wide range of densities. Simulations showed that topological interaction made groups markedly more robust to fragmentation under perturbation. A second finding from the same group, published by Andrea Cavagna and colleagues in 2010, was that correlations in velocity fluctuations in starling flocks are scale-free: a change in direction by one bird is correlated with changes in birds across the whole flock, however large the flock, rather than only with near neighbours. This means that information about a turn, for example in response to a predator, propagates across the group with little attenuation. The flock behaves as though it is poised near a critical point, where the system is maximally responsive. For a drone engineer, the topological finding is immediately suggestive, because communication in a swarm has a natural analogue. A drone with limited bandwidth cannot listen to and process messages from every other drone within radio range. It must choose. Choosing a fixed number of neighbours, rather than everyone within a distance, bounds the communication and computation load per drone and keeps it constant as the swarm densifies. It also, as with the starlings, keeps every drone connected to the group when the swarm stretches out. The radio does not do this automatically; radio range is metric, defined by signal strength falling with distance. The topological rule must be imposed in software on top of a metric medium, which is a design decision, not a gift of nature. From particles to schools: models of collective motion The starling measurements arrived after two decades of theoretical models, and those models remain the conceptual backbone of flocking algorithms. In 1995, Tamás Vicsek and colleagues published in Physical Review Letters a model of what they called self-driven particles. Each particle moves at constant speed. At each time step, it adopts the average direction of motion of all particles within a fixed radius, plus a random perturbation. That is the whole model. Its importance lies in what it does: as the noise is reduced or the density increased, the system undergoes a sharp transition from disordered motion, particles heading in all directions, to ordered motion, in which nearly all particles head the same way. Alignment with neighbours alone, with no leader and no global information, is enough to produce global order, provided noise is low enough and neighbours are close enough. The model also shows the converse, which matters for engineers: when noise rises past a threshold, order collapses abruptly, not gradually. In a drone swarm, "noise" includes sensor error, communication delay and message loss, and the abruptness of the transition is a warning. Iain Couzin and colleagues, in a 2002 paper in the Journal of Theoretical Biology, extended this kind of model with three concentric zones around each individual. In the innermost zone of repulsion, an individual moves away from neighbours to avoid collision. In a middle zone of orientation, it aligns with neighbours. In an outer zone of attraction, it moves toward neighbours to avoid isolation. Varying the widths of these zones produced four distinct collective states: a loose, disordered swarm; a rotating torus or mill, as seen in some fish schools; a dynamic parallel group; and a highly aligned parallel group. More strikingly, the transitions between states showed hysteresis. Starting from a swarm and widening the orientation zone produced one sequence of states; starting from a parallel group and narrowing it produced another. The group's current shape depended on its history, a property the authors called collective memory. For drones, the lesson is that the same parameters can sustain different collective behaviours depending on initial conditions, which complicates testing: a flock that behaves well when launched one way may behave differently when launched another. These zone models are recognizably the same as the three rules Craig Reynolds had proposed for computer animation in 1987, separation, alignment and cohesion, and the convergence between graphics and theoretical biology is not coincidence. Both communities were looking for the minimal set of local rules that would produce realistic group motion. Later experimental work refined the picture. Studies of golden shiner fish by Yael Katz, Iain Couzin and colleagues in 2011, which tracked every individual in small schools, found that much of the interaction was mediated by changes in speed rather than heading, and that explicit alignment was weaker than the models assumed; attraction and repulsion, acting through acceleration and deceleration, did much of the work. The precise rules differ between species, and even between contexts within a species. What generalizes is the architecture: short-range repulsion, longer-range attraction, and some coupling of motion between neighbours. Leadership without leaders A flock that only aligns with itself goes nowhere in particular. Real animal groups travel to destinations: food, roosts, breeding grounds. How do groups reach goals when most members do not know where the goal is? Iain Couzin, Jens Krause, Nigel Franks and Simon Levin addressed this in a 2005 paper in Nature. They modelled groups in which a small fraction of individuals had a preferred direction of travel and the rest simply followed the local rules. The informed individuals did not signal that they were informed, and no one knew who they were. Yet a small informed minority could steer the whole group with high accuracy, and the proportion of informed individuals needed for a given accuracy fell as group size increased. In large groups, a few per cent could suffice. When two informed subgroups preferred different directions, the group either compromised or, if the directions differed enough, chose the direction of the larger subgroup, a form of decision-making that emerges from motion alone. This result has a direct engineering reading. An operator guiding a drone swarm need not command every drone. Giving a goal to a few members, which then follow it while still obeying the flocking rules, can steer the entire group, and if those members are lost, others can be given the goal. The informed minority is a natural interface between human intention and swarm behaviour, and several research systems have used exactly this approach. Leadership in real animal groups is sometimes more structured than the 2005 model. Máté Nagy, Zsuzsa Ákos, Dora Biro and Tamás Vicsek, in a 2010 paper in Nature, fitted homing pigeons with lightweight GPS loggers and analyzed who followed whom during flocking flights. They found a well-defined hierarchy: some birds' changes in direction were consistently followed, with a short delay, by others, and the pattern was largely stable across flights. But the hierarchy was not a chain of command. Many birds influenced and were influenced by several others, and the structure was a weighted network rather than a tree. For engineers, this suggests that swarms need not choose between pure equality and rigid hierarchy; they can weight neighbours differently according to role, reliability or information quality, and the weights can change. Honeybees and the problem of choosing Flocking is about moving together. Many of the most important swarm tasks are about choosing together: which of several sites to search first, which detection to investigate, whether a signal is real. Here the most instructive animal is the honeybee, as described in Thomas Seeley's research and in his 2010 book Honeybee Democracy. When a honeybee colony outgrows its hive, roughly half the workers leave with the old queen and cluster on a branch. A few hundred of them, experienced foragers, act as scouts. They fly out, find candidate nest cavities, and return to advertise them with the waggle dance. The vigour and duration of a scout's dance depend on her assessment of the site's quality. Other scouts, recruited by dances, visit the advertised sites, assess them independently, and dance in turn if they approve. Over time, dances for poor sites fade because scouts dance less and for shorter periods for them, while dances for good sites recruit more and more scouts. The decision is triggered not by a census of dances but by a quorum at the site itself: when scouts at a candidate site sense that enough other scouts are present, on the order of a few dozen, they return to the cluster and produce signals that prepare the swarm to fly. Seeley and colleagues reported in Science in 2012 a further element: stop signals. Scouts committed to one site deliver short vibrational signals, often accompanied by head-butting, to dancers advertising competing sites. These signals inhibit the competing dancers. The effect is cross-inhibition between options, which, as modelling showed, helps the swarm break deadlocks between sites of similar quality and reach a decision faster. Each element of this process maps onto a problem in distributed decision-making for machines. Independent assessment, where each scout visits the site rather than trusting the dance, guards against error cascades. Positive feedback through recruitment amplifies good options. Decay of advocacy prevents the group from locking onto an early, mediocre choice. A quorum threshold, rather than unanimity, allows a decision to be reached without everyone agreeing, which matters when some members are out of contact. Cross-inhibition breaks ties. Collective decision-making research in swarm robotics, much of it by Dorigo's group in Brussels and by Andreagiovanni Reina and colleagues, has implemented versions of these mechanisms in robot swarms and analyzed their speed-accuracy trade-offs mathematically. Ants and the environment as memory Ants offer a different lesson: coordination through the environment rather than through direct communication. In the 1950s the French biologist Pierre-Paul Grassé coined the term stigmergy to describe how termites coordinate nest building: a deposit of material by one termite stimulates further deposits nearby by others, so that structures grow without any termite holding a plan. Ant foraging works similarly through pheromone trails. An ant that finds food lays a trail on its way back; other ants follow trails in proportion to their strength; trails that are used are reinforced and those that are not evaporate. Experiments by Jean-Louis Deneubourg, Simon Goss and colleagues around 1989 and 1990 showed the consequences elegantly. Argentine ants given two bridges of different lengths to a food source came, over time, to use predominantly the shorter one, because ants on the shorter bridge returned sooner and reinforced their trail faster. No ant measured anything. The environment integrated the experience of many ants and presented the result as a gradient. Marco Dorigo's ant colony optimization algorithms, beginning with his doctoral work in 1992, turned this mechanism into a general method for solving routing and scheduling problems, and versions of it have been used for routing in ad hoc networks, a connection that returns in the fifth chapter. Drones cannot lay chemical trails in the air, but they can lay virtual ones. A shared map in which drones mark areas as searched, with marks that decay over time so that areas are eventually revisited, is a direct analogue of pheromone. The key property is that the information lives in a shared medium rather than in messages addressed to particular recipients. A drone arriving in an area reads the state of the area, not the history of who said what. This decouples the drones from one another in time: a drone does not need to be in contact with the one that searched an area earlier in order to benefit from its work. In a network that is frequently partitioned, that decoupling is valuable. The cost is that the shared map must itself be maintained and synchronized across drones, which brings the problem back to the network. Deborah Gordon's long-term studies of harvester ants in Arizona add one more mechanism. Gordon found that colonies regulate how many foragers go out not through any signal about food but through the rate at which returning foragers encounter ants waiting at the nest entrance. A high encounter rate, meaning foragers are coming back quickly with food, stimulates more ants to leave. The rate of interaction itself carries information. For a swarm, this suggests that the frequency of contact between members, which is observable without exchanging any data, can serve as a signal of local conditions: of density, of the rate at which tasks are being completed, of whether a region is busy or empty. What transfers and what does not The biological record establishes a few things firmly. Coordinated motion can arise from local interaction with no leader. Interacting with a fixed number of neighbours rather than everyone within range makes groups more robust to changes in density. A small informed minority can guide a group. Good collective choices can be made through independent assessment, positive feedback, decay, quorum thresholds and cross-inhibition. Shared environmental marks can coordinate agents who never meet. Each of these principles has been implemented in robots and each has proved useful. Several things do not transfer, and ignoring them has produced disappointment. First, animals sense their neighbours directly and continuously, mostly by vision, with high bandwidth and almost no delay. Drones mostly learn about neighbours through radio messages, which are delayed, lossy, and contend with one another for the channel. Vision-based relative sensing between drones exists and is improving, but it is limited by camera field of view, lighting and onboard computation. Second, animal groups are often very large, and large numbers average away individual error. A swarm of twelve drones does not have the statistical cushion of a murmuration of twenty thousand. Third, animals are expendable from the perspective of the group in ways that drones flying over people are not; evolution tolerates collisions and losses that regulators will not. Fourth, animal behaviour has been tuned by selection across countless generations in the environments where it is used. Drone behaviour is tuned in simulation and in a few field trials, and the gap between those and reality is where most failures occur. The deepest lesson of the biology is therefore not any particular rule. It is that collective behaviour depends on the interaction structure: who senses whom, how quickly, how reliably. Starlings achieve what they achieve because their interaction structure is fast, rich and adaptive. A drone swarm's interaction structure is its communication network and its relative sensing. Get that structure right, and simple rules can do a great deal. Get it wrong, and no rule will save the group. The next chapter shows what happened when engineers took the flocking models off the screen and put them into aircraft. CHAPTER 3 Flocking in Metal and Air In 1987 Craig Reynolds, then at Symbolics, presented a paper at the SIGGRAPH computer graphics conference titled "Flocks, Herds, and Schools: A Distributed Behavioral Model." He wanted animated birds that moved like real birds without an animator scripting every wingbeat, and he got them with three rules applied by each simulated bird, which he called a boid, to the neighbours it could perceive: steer to avoid crowding them, steer toward their average heading, and steer toward their average position. Separation, alignment, cohesion. The results were convincing enough that boids were used for the bat swarms and penguin flocks in the 1992 film Batman Returns, and the paper became one of the most cited in the history of computer graphics. It took roughly a quarter of a century to get those three rules to work reliably in real aircraft outdoors. The reasons it took so long, and the ways engineers eventually succeeded, are the subject of this chapter. The short version is that Reynolds' boids lived in a world with perfect, instantaneous perception of neighbours, no inertia worth mentioning and no wind. Real drones have none of these luxuries, and the gap is filled almost entirely by the network. From rules to guarantees The first step from animation to engineering was mathematical. Reynolds' rules were heuristics; they looked right, but nothing guaranteed that a flock running them would stay together, avoid collisions, or head anywhere in particular. Control theorists in the early 2000s set out to provide guarantees. The core concept they used is consensus. Imagine each drone holds a number, say its heading, and repeatedly replaces it with a weighted average of its own value and its neighbours' values. Under what conditions do all the numbers converge to a common value? The answer depends on the communication graph, the network of who hears whom. In 2003, Ali Jadbabaie, Jie Lin and Stephen Morse showed that for the alignment rule in Vicsek's model, headings converge to a common value provided the communication graph is connected often enough. It does not need to be connected at every instant; it is enough that over every interval of some bounded length, the union of the links that existed during the interval connects everyone. This was an important result because real swarms never have a permanently connected graph. Links appear and disappear as drones move. The theorem says that intermittent connectivity is sufficient for agreement, provided the gaps are bounded. The speed of convergence also depends on the graph, through a quantity called algebraic connectivity, introduced by the Czech mathematician Miroslav Fiedler in 1973. Informally, algebraic connectivity measures how hard it is to cut the network into two pieces. A long chain of drones, each hearing only its immediate neighbours, has low algebraic connectivity; information takes many steps to travel from one end to the other and consensus is slow. A densely interconnected cluster has high algebraic connectivity and reaches agreement quickly. A swarm's ability to coordinate is therefore a property of the shape of its network, not just of its rules. Reza Olfati-Saber's 2006 paper in the IEEE Transactions on Automatic Control was the most influential attempt to put flocking as a whole on a rigorous footing. He showed that Reynolds' three rules, implemented as gradients of a pairwise potential function plus velocity alignment, do not in general guarantee cohesion: a flock can fragment into subgroups that drift apart permanently. To prevent this, he added a navigational feedback term, steering each agent toward a common group objective, and introduced virtual agents representing obstacles, so that obstacle avoidance could be treated within the same framework. With these additions, he proved that the flock forms a cohesive lattice-like configuration, avoids collisions and moves toward the goal. The paper's lesson for engineers was that pure emergence is not enough; some shared reference, however weak, is needed to hold a group together reliably. One more result from this period matters for everything that follows. In 2004, Olfati-Saber and Richard Murray analyzed consensus with communication delays. They showed that for the simplest case, uniform delay on every link, consensus remains stable only if the delay is below a threshold that shrinks as the maximum number of neighbours grows. In plain terms, the more neighbours each agent listens to, the less delay the system can tolerate before it starts to oscillate. This is counterintuitive: listening to more neighbours seems like it should make the group more stable. But each neighbour's information is stale, and combining many stale inputs with high gain amplifies the staleness. The practical consequence is that delay and neighbourhood size must be designed together. The first outdoor flocks The first serious attempts to fly flocking algorithms outdoors ran directly into these issues. In 2011, Sabine Hauert and colleagues at the École Polytechnique Fédérale de Lausanne reported flying Reynolds-style flocking with ten fixed-wing drones. Fixed-wing aircraft cannot hover or stop, which makes separation hard to maintain, and the aircraft relied on radio to share positions obtained from satellite navigation. The team found that communication range and the rate at which drones exchanged positions strongly affected flock behaviour, and they had to adapt the rules to the aircraft's turning constraints. The flock worked, but it was clear that the communication channel was a dominant factor in performance. Gábor Vásárhelyi and colleagues in Tamás Vicsek's group at Eötvös Loránd University in Budapest reached the same conclusion with quadcopters. In 2014 they reported outdoor flocking and formation flight with ten quadcopters exchanging satellite-derived positions by radio. Their key insight was that delay, both in communication and in the drones' own response, destabilizes naive alignment rules, and they introduced a term that acts like viscous friction, damping the velocity differences between neighbours rather than simply matching velocities. Without such damping, flocks oscillated. Their 2018 paper in Science Robotics, "Optimized flocking of autonomous drones in confined environments," took the work much further. They flew thirty drones as a coherent flock outdoors, inside a bounded arena, avoiding the arena walls and one another, without central control of the motion. The algorithm had a set of terms, repulsion, velocity alignment with friction-like damping, wall avoidance and obstacle avoidance, each with tunable parameters, about a dozen in all. Rather than tune these parameters by hand, the team built a simulator that included realistic communication delay, sensor noise, inertia and limited acceleration, and used an evolutionary algorithm to search for parameter sets that maximized a fitness function combining speed, order, safety and avoidance of walls. The parameters that emerged from simulation transferred to the real drones and produced stable flocking at speeds that earlier hand-tuned systems could not sustain. Two features of this work deserve emphasis. The first is that the simulator's realism about delays and noise was what made the transfer to hardware succeed. A simulator with idealized communication would have produced parameters that failed in the field. The second is that the drones in that study still used satellite navigation to know their own positions and radio to broadcast them. The flocking was decentralized, but it rested on two external services, GNSS and a shared radio channel, that are exactly what this book's target environments take away. Flocking without satellites The next generation of research removed the dependence on satellite navigation, and in doing so turned flocking from a coordination problem into a combined perception, planning and communication problem. In 2022, Xin Zhou, Fei Gao and colleagues at Zhejiang University published "Swarm of micro flying robots in the wild" in Science Robotics. A swarm of ten palm-sized quadrotors flew through a dense bamboo forest, keeping formation and avoiding trees and one another, with no satellite positioning and no motion-capture system. Each drone carried its own cameras and inertial sensors, estimated its own motion, perceived obstacles with a depth camera, and planned its own trajectory on board. Drones shared their planned trajectories with one another over a wireless link, and each drone's planner treated the other drones' trajectories as moving obstacles to avoid while also optimizing for formation, smoothness and progress toward the goal. The swarm also tracked a moving person through the forest. The architecture here differs in kind from Reynolds-style rules. Instead of reacting to neighbours' current positions, each drone reasons about where its neighbours intend to be over the next few seconds and plans around those intentions. This is more robust to delay, because a trajectory remains informative even when it arrives late: the recipient can evaluate it at the current time. It is also more demanding of communication, because a trajectory is a larger message than a position, and it introduces a new failure mode. If a drone does not receive a neighbour's updated trajectory, it plans against an out-of-date intention. The planner must therefore keep safety margins that account for the possibility that its information is stale, and the margins must grow with the age of the information. The precursor system from the same group, EGO-Swarm, published at the 2021 IEEE International Conference on Robotics and Automation, was released as open-source software and has been widely reused. It made explicit a design pattern that now dominates high-performance aerial swarm research: decentralized trajectory optimization with asynchronous broadcasting of plans, each drone replanning frequently and treating others' published plans as constraints. At the other extreme of scale, researchers at Delft University of Technology have shown how much can be done with very little. Shushuai Li, Guido de Croon and colleagues developed relative localization for Crazyflie drones weighing about 33 grams, with processors running at 168 megahertz and 192 kilobytes of memory. Each drone measured its distance to neighbours using ultra-wideband radio and combined those ranges with neighbours' shared velocity and heading estimates. They reported position errors below 0.2 metres at 16 updates per second for swarms of up to thirteen drones, enough for formation flight without any external positioning. Their work also found that some quantities that appear unobservable from range measurements alone become observable once the drones are moving, because motion provides the geometric variety that a single range cannot. This is a recurring theme: in a swarm, the act of flying is part of the act of sensing. Table 1 summarizes the main families of decentralized flocking and formation control as they stand. Table 1. Families of decentralized flocking and formation control for aerial swarms. Approach What each drone needs Main strength Main weakness Representative work Rule-based flocking Neighbour positions and velocities Simple, cheap, scales well Fragile under delay without damping Reynolds 1987; Hauert et al. 2011 Consensus with navigation feedback Neighbour states plus a shared goal Provable cohesion and convergence Assumes bounded delay and connectivity Olfati-Saber 2006 Simulation-optimized flocking Neighbour states; tuned parameters Robust to modelled delay and noise Only as good as the simulator Vásárhelyi et al. 2018 Decentralized trajectory planning Own map plus neighbours' broadcast plans Handles clutter and delay well Heavier compute and messages Zhou et al. 2022 Learned policies Local observations; trained network Can capture complex trade-offs Hard to verify; sim-to-real gap Tolstaya et al. 2019; Batra et al. 2021 The last row reflects a growing body of work in which flocking controllers are learned rather than designed. Ekaterina Tolstaya and colleagues at the University of Pennsylvania showed in 2019 that graph neural networks, which process information passed along the edges of the communication graph, can learn decentralized flocking controllers that imitate a centralized expert while using only local communication. Sumeet Batra and colleagues at the University of Southern California trained quadrotor swarm controllers end to end with reinforcement learning in simulation and transferred them to real Crazyflies. These approaches are promising, but they inherit the old problem in sharper form: the behaviour is only as good as the training distribution, and proving properties of a neural controller is much harder than proving properties of a consensus law. Not colliding: safety when everyone moves Flocking rules include a repulsion term, but repulsion alone does not guarantee safety. Two drones approaching head-on may each turn toward the same side; three drones may be pushed by repulsion from each other into a fourth; a drone surrounded by obstacles may find the forces balancing at a point that is not where it wants to be, a local minimum that has troubled potential-field methods since at least the early 1990s, when Yoram Koren and Johann Borenstein catalogued their limitations. A swarm that collides even occasionally is unusable, so collision avoidance has been the subject of its own body of work. One influential line is velocity obstacles, and specifically optimal reciprocal collision avoidance, developed by Jur van den Berg, Stephen Guy, Ming Lin and Dinesh Manocha around 2011. The idea is that each agent computes, for each neighbour, the set of velocities that would lead to a collision within some time horizon, assumes the neighbour will take half the responsibility for avoiding the collision, and chooses the velocity closest to its preferred one that lies outside all of these sets. Because each agent assumes reciprocity, the method avoids the oscillation that occurs when both agents try to dodge fully. It requires only neighbours' positions and velocities, which can in principle be sensed rather than communicated. A second line uses control barrier functions, a technique from control theory that Li Wang, Aaron Ames and Magnus Egerstedt applied to multi-robot systems in 2017. A barrier function defines a safe set, for example all configurations in which every pair of drones is more than a minimum distance apart, and constrains the control input so that the system cannot leave that set. In practice, each drone takes the command produced by its flocking or mission controller and minimally modifies it, by solving a small optimization problem, so that safety constraints are satisfied. The appeal is modularity: the mission controller can be anything, including a learned policy, and the barrier layer acts as a safety filter. The limitation is that guarantees depend on accurate knowledge of neighbours' states and on the drone being physically able to execute the filtered command. A third approach, buffered Voronoi cells, introduced by Dingjiang Zhou, Mac Schwager and colleagues in 2017, divides space among drones so that each drone has a cell, shrunk by a safety buffer, in which it alone may move during the next interval. Each drone can compute its own cell from neighbours' positions, without communicating plans, and as long as each stays within its cell no collision can occur. This method is attractive precisely because it needs no communication beyond position sensing, which makes it a useful fallback when the network is degraded. Physical realities add further constraints. A multirotor pushes air downward, and a drone flying directly below another experiences its downwash, which can destabilize it. Wolfgang Hönig, James Preiss and colleagues at the University of Southern California, flying forty-nine Crazyflies in their Crazyswarm system in 2017, modelled each drone's collision volume as an ellipsoid elongated vertically, so that drones keep more distance above and below than side to side. Wind outdoors pushes drones at different rates depending on their exposure, and gusts arrive at different members of a spread-out swarm at different times, which looks to a flocking controller like a disturbance propagating through the group. Battery depletion changes each drone's agility over the course of a flight. Each of these factors widens the safety margins needed, and wider margins reduce how tightly a swarm can pack, which in turn affects how well it can fly through clutter. What the network does to flocking Every part of this chapter has come back to communication. Flocking rules depend on neighbours' states, and those states arrive by radio unless they can be sensed directly. Consensus depends on the connectivity of the communication graph and degrades with delay. Trajectory-sharing planners depend on timely receipt of plans. Even collision avoidance, which can in principle rely on direct sensing, works best when drones can share intentions. The engineering response has several elements, which the next chapters develop. First, sense what can be sensed. Relative positions measured by ultra-wideband ranging or cameras reduce dependence on messages. Second, send what is robust to delay. A planned trajectory is more useful when late than a current position. Third, design for stale information explicitly, with safety margins that grow with information age. Fourth, choose neighbourhoods carefully. Listening to a bounded number of neighbours, as starlings do, keeps both bandwidth and delay sensitivity under control. Fifth, have a fallback mode that needs no communication at all, such as buffered Voronoi cells or simply hovering in place, and switch to it when the network fails. The progression from Reynolds' boids to Zhou's forest swarm can be read as a progression in how seriously the network is taken. The boids assumed it away. The first outdoor flocks discovered it. The optimized flocks modelled it. The current systems design around it. That progression continues into the next problem, which is that a drone cannot flock with its neighbours, or do anything else useful, unless it knows where it is. CHAPTER 4 Knowing Where You Are When the Sky Goes Quiet Almost every consumer and commercial drone in the world depends on satellite navigation. It anchors the autopilot's estimate of position, holds the aircraft steady in wind, defines geofences, records where images were taken and brings the drone home when the link fails. For a single drone flying over open ground, this dependence is rarely a problem. For a swarm working inside a building, under a forest canopy, beneath a bridge or in a mine, it is the first thing that breaks, and when it breaks every other capability in this book is affected. Flocking requires knowing where neighbours are relative to you. Search requires knowing which ground you have covered. Task allocation requires knowing which drone is closest to which task. All of these depend on position, and in the environments that matter most, position must be built from the drones' own sensors and from what they tell one another. Why satellites fail and what replaces them Satellite navigation signals are extraordinarily weak by the time they reach the ground; they are transmitted from roughly twenty thousand kilometres away and arrive well below the background noise, recovered only by correlating against known codes over time. Anything that blocks or reflects them degrades the fix. Concrete and earth block them entirely, which rules out tunnels, mines and building interiors. Dense foliage attenuates them, particularly when wet. Tall buildings reflect them, so that a receiver in a street canyon may compute its position from signals that have bounced off a wall and travelled extra distance, producing errors of tens of metres. And because they are weak, they are easy to drown out. Since 2022, aviation authorities including the European Union Aviation Safety Agency have issued repeated warnings about widespread interference with satellite navigation, both jamming and spoofing, affecting civil aircraft in regions near armed conflicts, including the Baltic, the eastern Mediterranean and the Black Sea. Civilian drones operating anywhere near such regions cannot assume that the position their receiver reports is true. The oldest alternative is dead reckoning: integrate acceleration and rotation rate from an inertial measurement unit to track motion from a known starting point. The problem is drift. The tiny micro-electromechanical sensors in small drones have biases and noise that, when integrated twice to get position, produce errors that grow rapidly with time. Inertial sensing alone is useful for seconds, not minutes. The workhorse alternative on modern drones is visual-inertial odometry, which fuses a camera with the inertial unit. The camera tracks distinctive points in the scene from frame to frame; the motion of those points constrains how the camera moved; the inertial unit fills in between frames and resolves the scale ambiguity that a single camera cannot. Mature open-source implementations, such as VINS-Mono from Tong Qin, Peiliang Li and Shaojie Shen at the Hong Kong University of Science and Technology, published in 2018, and ORB-SLAM3 from Carlos Campos and colleagues at the University of Zaragoza, published in 2021, achieve drift on the order of a small fraction of the distance travelled in favourable conditions. Visual-inertial odometry runs on processors small enough for palm-sized drones and underpins the forest swarm described in the previous chapter. It also has well-known failure conditions, and many of them coincide with disaster environments. Cameras need light, so darkness defeats them unless the drone carries illumination. They need texture, so blank walls, snowfields and still water give them nothing to track. Smoke and dust obscure the scene. Fast motion blurs images. Repetitive structures, such as rows of identical windows or the regular supports of a tunnel, can cause false matches. For these conditions, drones carry other sensors. Laser scanners measure geometry directly and work in darkness; odometry methods such as FAST-LIO2, published in 2022 by Wei Xu and colleagues at the University of Hong Kong, fuse laser scans with inertial data robustly. But laser scanners are heavier and more power-hungry than cameras, and small swarms often cannot carry them. Thermal cameras see through light smoke and in darkness. Small radars penetrate dust and smoke. Optical flow sensors pointed at the ground give velocity at low altitude cheaply. Each sensor has its own failure conditions, and the robust approach is to carry more than one kind and fuse them, at the cost of weight and power. Whatever the sensor suite, all of these methods share a structural limitation: they estimate motion relative to where the drone started, and their errors accumulate. A drone that has flown a kilometre through a forest using visual-inertial odometry knows roughly where it is relative to its launch point, with an error that has been growing the whole way. For one drone, that may be acceptable. For a swarm, it creates a problem that has no counterpart in single-drone operation: each drone's estimate drifts differently, so the drones gradually disagree about where everything is. Relative positioning: measuring each other In a swarm, what often matters most is not where each drone is in absolute terms but where drones are relative to one another. Flocking and collision avoidance need relative positions. So does stitching together what different drones have observed. Relative positions can be measured directly, without reference to any external frame, and this is one of the main advantages a swarm has over a lone drone. Ultra-wideband radio is the most widely used technology for this. An ultra-wideband transceiver sends extremely short pulses spread over a wide band of frequencies, and by timing how long a pulse takes to travel to another transceiver and back, it can estimate the distance between them, typically to within around ten centimetres in clear conditions. The chips are small, cheap and low-power; the Decawave, now Qorvo, family of chips is used in research platforms including the Crazyflie's positioning deck. Ranging between drones gives each pair a distance measurement. A distance alone does not give direction, but combining ranges to several neighbours, or ranges over time as drones move, does. The Delft work on 33-gram drones described in the previous chapter used exactly this combination, and it is notable that the observability of relative position improved when drones moved in varied ways, so that a swarm that holds perfectly still learns less about its own shape than one that moves. Ultra-wideband ranging has limits. Signals passing through bodies, walls or the drone's own frame are delayed, which biases the range upward. Each ranging exchange takes time on a shared channel, so ranging between every pair of drones in a large swarm costs airtime that grows with the square of the swarm size unless ranging is scheduled cleverly. And ranges establish the swarm's shape only up to rotation and reflection; to know which way is north, or which way the swarm is heading, some additional information is needed, such as each drone's compass or its odometry. Vision is the other direct option. Each drone can carry markers that others detect with cameras. The Multi-Robot Systems group at the Czech Technical University in Prague, led by Martin Saska, developed a system called UVDAR in which drones carry blinking ultraviolet light-emitting diodes and others observe them with cameras filtered to see only ultraviolet, which suppresses most background clutter. The blinking patterns identify each drone. This gives bearing and approximate distance to neighbours without any radio exchange, which makes it valuable when the radio channel is congested or jammed. Its limitation is line of sight: a neighbour hidden behind a tree is invisible to the camera, though it may still be audible to the radio. Collaborative mapping: agreeing on the world Relative positioning tells drones where they are relative to one another at a moment. Over a mission, they need more: a shared understanding of the environment, so that a corridor explored by one drone is not explored again by another, so that a victim seen by one drone can be found by a second, and so that the human supervisors receive a single coherent map rather than a dozen inconsistent fragments. Building that shared understanding is the problem of collaborative simultaneous localization and mapping, usually abbreviated as collaborative SLAM. The basic mechanism is the loop closure. As a single drone explores, it builds a graph whose nodes are its estimated poses at successive moments and whose edges are the relative motions between them measured by odometry. When the drone returns to a place it has seen before and recognizes it, it adds an edge connecting the current pose to the earlier one. That edge contradicts the accumulated drift, and solving the graph to satisfy all edges as well as possible corrects the drift along the whole loop. In the multi-robot case, loop closures can also occur between drones: when drone A recognizes a place that drone B observed earlier, it creates an edge between A's graph and B's. Those inter-robot edges are what tie separate maps into a single map in a common frame. There are two broad architectures. In the centralized one, each drone sends its data, or a compressed summary, to a server, which detects inter-robot loop closures and solves the combined graph. This is simpler and allows the server to use heavy computation, but it requires a link to the server. In the decentralized architecture, drones exchange information directly with whichever neighbours they can reach, detect loop closures among themselves, and solve the combined graph cooperatively, each drone holding and optimizing its own part. Pierre-Yves Lajoie, Giovanni Beltrame and colleagues at Polytechnique Montréal developed DOOR-SLAM, published in 2020, and Swarm-SLAM, published in IEEE Robotics and Automation Letters in 2024, as decentralized systems designed explicitly for intermittent communication. Yulun Tian, Jonathan How, Luca Carlone and colleagues at the Massachusetts Institute of Technology developed Kimera-Multi, published in 2022 in the IEEE Transactions on Robotics, which builds dense three-dimensional meshes labelled with object categories and optimizes them in a distributed way. The network shapes every design choice in these systems. Raw images are far too large to share freely over a mesh network, so drones exchange compact descriptors, short numerical fingerprints of places, first. Only when two descriptors match closely enough to suggest a loop closure do the drones exchange the more detailed information needed to verify it and compute the relative pose. Swarm-SLAM goes further, selecting which candidate loop closures to verify according to how much they would improve the map, so that scarce bandwidth is spent where it matters most. The optimization itself is distributed: each drone solves for its own poses given its neighbours' current estimates, exchanges the results, and repeats, converging toward the solution that a central server would have found, but without any drone needing all the data. When drones lie to each other by mistake Collaborative mapping has a failure mode that is specific to swarms and that illustrates a broader point about decentralized information. When two places look alike, a drone may declare a loop closure between them that is false. An office building with identical corridors on every floor, a forest of similar trees, or a tunnel with regular supports is full of such traps; roboticists call the problem perceptual aliasing. A single false loop closure, if accepted, can distort a map badly, folding one corridor onto another. In a multi-robot system, the damage spreads: a false inter-robot closure distorts the relationship between two drones' maps, and since those maps are then used by other drones, the error propagates through the team. The standard defence is to check loop closures for consistency with one another. Joshua Mangelson and colleagues at the University of Michigan proposed pairwise consistency maximization in 2018: among a set of candidate inter-robot loop closures, find the largest subset that are mutually consistent, on the reasoning that true closures agree with each other while false ones agree with nothing in particular. Kimera-Multi uses a related robust optimization method that progressively down-weights measurements that do not fit. These methods work well when there are enough true closures to outvote the false ones, and less well when a drone has seen only a small part of the environment. There is a subtler problem in decentralized estimation, even without false measurements. Suppose drone A shares its position estimate with drone B, which fuses it with its own information and shares the result with drone C, which fuses it and shares the result back with A. A now receives information that partly originated with itself and may treat it as independent confirmation, becoming more confident than its evidence justifies. In a network with loops, which every mesh network has, this double counting can make the whole swarm overconfident. Combining estimates correctly requires knowing how they are correlated, and in a decentralized network that knowledge is generally unavailable. Simon Julier and Jeffrey Uhlmann proposed covariance intersection in 1997 as a conservative fusion rule that guarantees consistency without knowing the correlations, at the cost of some lost precision. More sophisticated methods track information provenance explicitly. The underlying lesson is that in a swarm, information that has travelled around the network is not the same as fresh evidence, and the network topology determines how much of what a drone hears is genuinely new. The Subterranean Challenge The most demanding public test of these ideas was the DARPA Subterranean Challenge, which ran from 2018 to 2021. Teams of robots, ground vehicles, legged robots and drones, had to explore underground environments, tunnels, urban underground structures and natural caves, find specified artifacts such as mannequins representing survivors, backpacks, mobile phones and gas sources, and report their locations accurately to a base station at the entrance. There was no satellite navigation underground, communication through rock and concrete was severely limited, and the courses were unknown in advance. Though the challenge was sponsored by a defence research agency, its tasks were explicitly modelled on underground rescue and exploration, and the results bear directly on civilian search and rescue. The final event took place in September 2021 in the Louisville Mega Cavern in Kentucky, on a course combining tunnel, urban and cave sections. Team CERBERUS, a collaboration whose members included the Norwegian University of Science and Technology, ETH Zurich, the University of Nevada, Reno, the University of California, Berkeley, and the University of Oxford, won the systems competition, tied on artifacts found with Team CSIRO Data61 from Australia and placed first on the tiebreaker. Team MARBLE, led by the University of Colorado Boulder, took third. Several lessons from the challenge, documented in a 2023 survey in the IEEE Transactions on Robotics by Kamak Ebadi and co-authors from many of the competing teams, bear directly on swarms. First, every successful team relied on multi-sensor fusion, typically combining laser scanning, cameras and inertial sensing, because no single sensor was reliable across all conditions. Second, maps had to be merged across robots despite the robots having explored largely separate parts of the course, which made inter-robot loop closures scarce and the consistency checking described above essential. Third, and most important for this book, communication shaped strategy at every level. Teams deployed radio nodes as the robots advanced, dropping them like breadcrumbs to extend the network into the cave, and designed their robots to explore autonomously beyond the network's reach and return within range, or into range of a dropped node, to deliver their findings. Exploration decisions were made with an eye on where communication would be available. A robot that found an artifact but could not report it before time ran out earned nothing. Drones were part of most top teams' systems, but in a supporting role. They could reach places ground robots could not, such as vertical shafts and areas beyond rubble, but their short endurance and limited payload made them less capable mappers than the larger ground robots. That result is a fair snapshot of the state of the art for aerial swarms in severely degraded environments as of the early 2020s: valuable, but not yet self-sufficient, and most valuable when integrated with other systems. Position as a negotiated fact The thread running through this chapter is that, in a swarm operating without satellites, position is not a measurement that each drone makes independently. It is a fact the swarm negotiates. Each drone has its own drifting estimate. Ranging and mutual observation constrain the relationships between those estimates. Loop closures, within and between drones, correct drift and tie frames together. All of this is carried out over a network that is bandwidth-limited and intermittently partitioned, which determines how quickly corrections spread and how consistent different drones' pictures of the world are at any moment. A swarm's spatial picture is therefore always somewhat out of date and somewhat inconsistent, and the inconsistency is largest across network partitions. Two groups of drones that have been out of contact for ten minutes may disagree about the position of a corridor by metres. When they reconnect, they must reconcile. Designs that acknowledge this, carrying uncertainty explicitly, guarding against double counting, checking loop closures for consistency and prioritizing the exchange of information that most improves the shared map, work in the field. Designs that assume every drone knows exactly where it is fail in ways that are hard to diagnose. Having established that position itself depends on the network, the next chapter turns to the network directly. CHAPTER 5 The Mesh Every drone that leaves the ground carries at least one radio, and most carry several: a control link to the pilot, a telemetry link to the ground station, a video downlink, perhaps a satellite navigation receiver and a cellular modem. In a conventional operation, all of these links run between the drone and the ground. A swarm adds something different in kind: links between the drones themselves, forming a network whose shape changes as the drones move. That network is how drones learn their neighbours' states, share maps, divide tasks and relay findings to the people who need them. When it works well, it is invisible. When it does not, every capability described in this book degrades. This chapter describes how such networks are built, why they are difficult, and what fundamental limits constrain them. Flying ad hoc networks Networks that form among moving nodes without fixed infrastructure are called mobile ad hoc networks, and research on them goes back to the packet radio programmes of the 1970s. Each node acts both as an endpoint and as a router, forwarding traffic for others, so that two nodes out of direct radio range can still communicate through intermediaries. The network is called a mesh when nodes have multiple paths to one another, which gives it resilience: if one path breaks, traffic can take another. Drone networks are a special case, which the research literature has called flying ad hoc networks, or FANETs, since an influential 2013 survey by İlker Bekmezci, Ozgur Koray Sahingoz and Şamil Temel in the journal Ad Hoc Networks. They differ from ground-based ad hoc networks in several ways. Nodes move fast, often tens of metres per second, so links form and break quickly. They move in three dimensions. They are often sparse, spread over a large area with only a few neighbours each. Their energy is precious, because every watt spent on radio is a watt not spent on flight. And the radio environment is unusual: links between drones in the air often have clear line of sight and propagate well, while links between drones and the ground, or between drones close to the ground, suffer from obstruction and reflections. A 2016 survey by Samira Hayat, Evşen Yanmaz and Raheeda Muzaffar in IEEE Communications Surveys and Tutorials examined these properties from the perspective of civilian applications and concluded that requirements vary enormously by application: a swarm sharing positions for flocking needs small messages delivered with very low delay, while a swarm streaming imagery needs high throughput and can tolerate more delay. The physics that sets the rules Most of what makes drone networking hard comes from radio propagation, and a few principles explain most of it. Signal strength falls with distance. In free space, received power falls with the square of distance, so doubling the distance between two drones reduces received power to a quarter. Near the ground, reflections and obstructions make it fall faster, often with the third or fourth power of distance. Every reduction in signal strength reduces the data rate the link can support, because the receiver needs a certain signal-to-noise ratio to decode each modulation scheme; as the signal weakens, radios step down to more robust, slower schemes until eventually the link fails. Frequency matters. Most small drones use the unlicensed bands around 2.4 and 5.8 gigahertz, the same bands used by Wi-Fi. Higher frequencies offer more bandwidth and thus higher data rates, but they are absorbed more by walls, foliage and bodies, and they diffract less around obstacles. Lower frequencies, such as the bands around 900 megahertz in the Americas and 868 megahertz in Europe, penetrate and bend better but carry less data. A swarm working inside a building or under forest canopy may find that 5.8 gigahertz links die within a few rooms while sub-gigahertz links persist, at a fraction of the throughput. Some systems carry radios in two bands for this reason, using the low band for small, critical messages and the high band for bulk data when it is available. Antennas are not uniform. A drone's antenna radiates more strongly in some directions than others, and as the drone pitches and banks, the pattern moves. The drone's own body, its battery, motors and carbon-fibre frame, shadows the antenna in some directions. Two drones that are close together can have a poor link if each is in the other's shadow, while two drones far apart can have a good one. This makes link quality depend not just on position but on attitude, which in turn depends on wind and manoeuvring. Indoors, reflections from walls create multipath interference, in which copies of a signal arriving by different paths cancel at some points and reinforce at others; moving a drone by a few centimetres can change link quality dramatically. The channel is shared. In the unlicensed bands, a swarm competes not only with itself but with every Wi-Fi network, wireless camera and other radio nearby. Within the swarm, radios that transmit on the same channel at the same time interfere. The standard Wi-Fi approach, carrier sense multiple access with collision avoidance, has each radio listen before transmitting and back off if it hears activity. This works tolerably in an office where all devices can hear each other. In a spread-out swarm, it suffers from the hidden terminal problem: drones A and C may both be in range of B but out of range of each other, so neither hears the other, both transmit to B at once, and B receives neither. Alternatives include time-division schemes, in which each drone is assigned slots, which avoid collisions but require time synchronization and waste capacity when slots go unused. Routing: finding paths through a moving network When drone A wants to send a message to drone D, which is out of direct range, the network must find a path through intermediate drones. Doing this in a network whose links change every few seconds is the routing problem, and decades of research on ad hoc networks have produced several families of solutions, each with a characteristic trade-off. Table 2 compares the main ones as they apply to drone swarms. Table 2. Main families of ad hoc routing and their fit to drone swarms. Family Examples How routes are found Overhead pattern Best suited to Proactive link-state OLSR, OLSRv2 Nodes flood link updates continuously Constant, grows with network size Moderate swarms needing low delay Proactive distance-vector B.A.T.M.A.N., Babel Nodes track best next hop from periodic messages Constant, modest Dense, slowly changing swarms Reactive on-demand AODV, DSR Route discovered by flooding when needed Bursty, grows with route breaks Sparse traffic, stable links Geographic GPSR Forward toward destination's position Low; needs position knowledge Fast-moving swarms with good positioning Store-carry-forward Bundle Protocol Hold data until a contact appears Low airtime; high delay Partitioned networks Proactive protocols maintain routes to every destination continuously. The Optimized Link State Routing protocol, standardized by the Internet Engineering Task Force as RFC 3626 in 2003 and revised as OLSRv2 in RFC 7181 in 2014, has each node periodically advertise its links, with a clever reduction in flooding: only a subset of neighbours, called multipoint relays, rebroadcast each advertisement. Every node thus knows the network topology and can compute shortest paths instantly. The cost is constant background traffic that grows with network size and with the rate of link change. The B.A.T.M.A.N. protocol, developed by the Berlin community wireless network Freifunk and implemented in the Linux kernel as batman-adv, takes a lighter approach: each node only learns which neighbour is the best next hop toward each destination, inferred from how many of each destination's periodic messages arrive through each neighbour. The Babel protocol, standardized as RFC 8966 in 2021, is another distance-vector design with good loop avoidance. The IEEE 802.11s amendment, published in 2011, builds mesh routing directly into Wi-Fi at the link layer. Reactive protocols find routes only when needed. Ad hoc On-Demand Distance Vector routing, standardized as RFC 3561 in 2003, floods a route request through the network when a node needs to send to a destination it has no route for; the destination, or a node with a fresh route, replies, and the route is used until it breaks, at which point discovery starts again. Reactive protocols generate no traffic when nothing is being sent, which suits networks with sparse traffic. In a fast-moving swarm, however, routes break often, and each break triggers another flood, which can generate more overhead than a proactive protocol would have. Geographic protocols avoid route discovery entirely. Greedy Perimeter Stateless Routing, proposed by Brad Karp and H. T. Kung in 2000, forwards each packet to whichever neighbour is closest to the destination's position, with a fallback procedure for routing around voids where no neighbour is closer. This needs no routing tables beyond a list of neighbours and their positions, and it copes well with fast movement. Its requirement is that every node knows the destination's position, which is easy with satellite navigation and harder without it. In a GPS-denied swarm, geographic routing must use the drones' own position estimates, which, as the previous chapter showed, may be inconsistent across the swarm. Routing then depends on localization quality, an example of how the subsystems of a swarm are coupled. There is also a family of biologically inspired routing protocols that use the ant foraging mechanism described in the second chapter. AntHocNet, developed by Gianni Di Caro, Frederick Ducatelle and Luca Maria Gambardella around 2005, sends small probe packets that lay virtual pheromone on the paths they take, reinforcing good paths and letting stale ones evaporate. These protocols adapt well to changing conditions in simulation, though they have seen less use in deployed systems than the standardized protocols. Field experience with drone meshes has tended to favour the proactive protocols, particularly B.A.T.M.A.N. and OLSR, for small to moderate swarms, mainly because they provide routes immediately when needed, which matters for time-critical coordination messages. But every comparative study finds the same thing: no protocol is best across all conditions, and performance depends heavily on how parameters such as advertisement intervals are tuned to the swarm's speed and density. A protocol tuned for a slowly expanding search pattern may fail when the swarm suddenly converges on a detection. The capacity ceiling There is a fundamental limit on how much a large wireless network can carry, and it has direct consequences for swarm design. In 2000, Piyush Gupta and P. R. Kumar published "The Capacity of Wireless Networks" in the IEEE Transactions on Information Theory. They considered many nodes scattered over an area, sharing a single wireless channel, each wanting to send data to some other randomly chosen node through multi-hop relaying. They showed that as the number of nodes grows, the throughput each node can achieve falls roughly in proportion to one over the square root of the number of nodes, ignoring a slowly varying logarithmic factor. The total capacity of the network grows, but more slowly than the number of nodes, so each node's share shrinks. The intuition is that in a larger network, a typical message must travel through more hops, and every hop consumes channel capacity in its neighbourhood. Each relay must both receive and retransmit, and its transmissions prevent nearby nodes from using the channel at the same time. As the network grows, an increasing fraction of every node's capacity is spent forwarding other nodes' traffic. For swarm designers, the Gupta-Kumar result is a warning against architectures in which every drone needs to talk to every other drone, or in which all drones stream data to a single distant ground station through the mesh. Both patterns create traffic that must cross many hops, and both will saturate as the swarm grows. The architectures that scale are those in which most communication is local, between neighbours, as in flocking and in the stigmergic coordination of the second chapter. This is a mathematical reason, not merely a biological analogy, for keeping swarm communication local. It is also why the Kilobot-style principle that each agent talks only to near neighbours is not just convenient but essential at scale. A second result, from Matthias Grossglauser and David Tse in 2002, offers a partial way out. They showed that if nodes are mobile, and if the application can tolerate delay, throughput per node can be held constant as the network grows: a node hands its data to a neighbour, which physically carries it closer to the destination before handing it on. Movement does the work that multi-hop relaying would otherwise do. The price is delay, potentially very large. For a swarm, this means that bulk data, such as high-resolution images or detailed map segments, can be delivered by drones physically flying it toward the ground station rather than by relaying it hop by hop, provided the mission can wait. That observation underlies the delay-tolerant strategies in the next chapter. Flooding, broadcast and the swarm's chatter Much swarm communication is not point to point at all. A drone's position and velocity are of interest to all its neighbours; a newly detected obstacle or victim is of interest to everyone. The natural mechanism is broadcast: transmit once, and every drone in range receives it. Broadcast is efficient for one-hop communication. For network-wide dissemination, the naive approach is flooding, where every drone rebroadcasts every new message it receives. Sze-Yao Ni and colleagues described the consequence in a 1999 paper titled "The Broadcast Storm Problem in a Mobile Ad Hoc Network": in dense networks, flooding produces massive redundancy, contention and collisions, so that the channel is swamped and many messages are lost. Their remedies, rebroadcasting only with some probability, only if few copies have been heard, or only if the drone is far from the sender, remain the basis of efficient dissemination today. The software stacks used for robots can make this worse if used carelessly. The Robot Operating System, version 2, the most widely used robot software framework, uses by default a communication middleware called DDS, which performs automatic discovery of participants by multicasting announcements. On a wired network, this is harmless. Over a multi-hop wireless mesh, the discovery traffic for a few dozen robots, each with many software components, can consume a large fraction of available capacity. The ROS community has worked around this by tuning DDS settings, by bridging only selected topics between robots, and increasingly by using the Zenoh protocol, which was designed for constrained and intermittently connected networks and became an officially supported ROS 2 middleware in 2025. For swarm designers, the lesson is that the software framework is part of the network design, and its default behaviour should never be assumed suitable. Swarms also benefit from classifying messages by urgency, size and scope. Safety-critical state, such as positions and intended trajectories, is small, needed frequently and needed mostly by near neighbours; it should be broadcast locally at high priority and never forwarded far. Coordination messages, such as task bids and claims, are small and needed by a wider but still limited group. Map updates are larger, valuable to everyone eventually, and tolerant of delay. Imagery and video are very large and usually needed only by human operators. A swarm that sends everything on a single undifferentiated channel will find its safety messages delayed by its video. Separating these classes, by priority queues, by separate radios or bands, or by scheduling, is one of the most effective improvements available. Hardware and the edge of the network The radios used in swarm research range from cheap Wi-Fi modules and low-rate long-range radios to purpose-built mesh radios used by public safety agencies. Low-cost systems are attractive for small drones, and research continues on making them more capable: in 2026, Germany's Fraunhofer Institute for Integrated Circuits described work on a low-cost, low-energy mesh based on short-range wireless technology for coordinating small civilian drones on tasks such as post-disaster search, explicitly contrasting it with expensive, bulky mesh systems designed for heavy aircraft. At the other end, commercial mesh radios offer higher power, multiple antennas and sophisticated scheduling, at a weight and price that only larger drones can carry. Most real operations also have an edge where the swarm's mesh meets other networks. When cellular coverage exists, a drone at the swarm's periphery may act as a gateway to it; the 3GPP standards body has specified features for aerial cellular users since Release 15. Satellite internet terminals at a command post can backhaul data from the swarm to distant experts. Rescue teams may carry their own portable mesh nodes or deploy tethered drones as elevated relays. In such hybrid networks, the swarm need not carry all traffic itself; it needs only to get data to the nearest gateway. Where gateways exist is therefore a planning input, and when they disappear, as they do when a swarm enters a building or a valley, the swarm's design must cope. The mesh, then, is not a passive pipe. Its capacity depends on the swarm's geometry, its routes depend on the swarm's motion, its usefulness depends on the swarm's choice of what to send, and its limits, set by radio physics and information theory, constrain which coordination strategies can work at scale. A swarm that moves without regard to its network will break it. The next chapter is about swarms that move with regard to their network: that maintain connectivity deliberately when they can, and that carry on sensibly when they cannot. CHAPTER 6 Designing for Broken Links The previous chapter described the mesh as it behaves when it works. This chapter is about what to do when it does not, which in the environments that matter is much of the time. A swarm entering a collapsed building will lose links behind every concrete wall. A swarm spreading out across a forested valley will stretch until its outermost members fall out of range of everyone. A swarm working near a river gorge will find that the drones on one side cannot hear the drones on the other. None of this is a malfunction. It is the normal condition of radio in cluttered terrain, and a swarm that treats it as an exception will spend its missions in exception handling. There are three broad strategies, and good systems use all three. The first is to keep the network connected deliberately, by constraining how drones move or by adding relays. The second is to accept disconnection but arrange for it to be temporary and predictable, so that drones reconnect at known times and places. The third is to design the swarm's information and behaviour so that it functions sensibly while disconnected and reconciles cleanly when it reconnects. Each strategy has a cost, and the art lies in choosing among them as the mission and the terrain change. Keeping the network whole The most direct approach is to make connectivity a constraint on motion. If every drone stays within reliable range of at least one other drone, and the pattern of links connects everyone, the network stays whole. The simplest way to enforce this is to preserve existing links: a drone may move freely as long as it does not move out of range of the neighbours it is currently linked to. This is easy to implement locally and guarantees connectivity, but it is very conservative. It prevents the swarm from ever rearranging its links, even when a rearrangement would keep the network connected and let the swarm spread out further. A more flexible approach maintains a global measure of connectivity rather than specific links. Recall from the third chapter that algebraic connectivity measures how well connected a network is; it is zero exactly when the network has fallen into separate pieces. Michael Zavlanos and George Pappas, in a 2008 paper in the IEEE Transactions on Robotics, and Lorenzo Sabattini, Cristian Secchi, Nikhil Chopra and Cesare Fantuzzi, in a 2013 paper in the International Journal of Robotics Research, developed controllers in which each robot adds to its motion a term that pushes algebraic connectivity away from zero. Individual links can break and form as the swarm moves, provided the network as a whole stays connected. The difficulty is that algebraic connectivity is a global property, and no drone can compute it from local information alone. These controllers therefore run a distributed estimation procedure alongside the motion control, in which drones exchange messages to estimate the connectivity of the whole network, which itself requires the network to be working. Peng Yang, Randy Freeman and colleagues showed in 2010 how this estimation can be done with only neighbour-to-neighbour communication, at the cost of some convergence time. Both approaches have the same fundamental limitation: maintaining connectivity limits coverage. A swarm that must stay connected can spread only as far as its radio range times the number of drones, arranged in the most stretched configuration that keeps links alive. In open terrain with good air-to-air links, that may be several kilometres. Inside a building, where links fail at every turn of a corridor, it may be a few rooms. For a search task, where the goal is precisely to cover ground, continuous connectivity can be a severe handicap. Relays, tethers and breadcrumbs The second approach to keeping the network whole is to add nodes whose job is purely communication. Some members of the swarm can be designated relays: instead of searching, they position themselves to bridge gaps between the searching drones and the ground station. Placing relays well is an optimization problem. In open terrain, relays along the straight line between endpoints, spaced at a fraction of radio range, are near optimal. In cluttered terrain, they must be placed at corners and openings where they can see around obstacles, which requires knowledge of the environment that the swarm may be in the process of acquiring. Relays do not need to fly. In the DARPA Subterranean Challenge, described in the fourth chapter, the leading teams carried radio nodes on their ground robots and dropped them as they advanced into the course, extending the network behind them like a trail of breadcrumbs. Each robot decided when to drop a node by monitoring signal strength back to the previous node and dropping a new one before the link degraded too far. The approach is effective because a node on the ground uses no flight energy and can operate for hours on a small battery, whereas a hovering relay drone exhausts its battery in minutes. For civilian rescue teams, the analogue is portable mesh nodes carried in by people or dropped by drones at building entrances, stairwells and junctions. Tethered drones are a third option. A drone connected to the ground by a cable that carries power and data can hover for many hours at a height of tens of metres, providing an elevated relay with line of sight over terrain that blocks ground-level links. Public safety agencies already use tethered drones for persistent overhead observation; as relays, they extend the network's reach over a valley or across a disaster area, with the limitation that they are fixed in place. The cost of every relay strategy is that resources spent on communication are not spent on the mission. A swarm of twelve drones that uses four as relays has eight searchers. Whether that is worthwhile depends on the value of continuous communication relative to the value of additional search coverage, which depends in turn on the mission. If the purpose is to find a survivor and report the location as soon as possible, a relay chain that delivers the report immediately may be worth the reduced coverage. If the purpose is to map an area for later analysis, relays may be wasted effort. Planned disconnection Rather than prevent disconnection, a swarm can allow it but control it. The research literature on exploration with limited communication, surveyed by Francesco Amigoni, Jacopo Banfi and Nicola Basilico in 2017 in IEEE Intelligent Systems, distinguishes between systems that require continuous connectivity, systems that require connectivity only at certain events or intervals, and systems that require no connectivity at all. The middle category, often called recurrent or periodic connectivity, has proved especially practical. The idea is that drones may leave the network to explore, provided they return to it, or to a meeting point with other drones, at agreed times or when they have something important to report. Geoffrey Hollinger and Sanjiv Singh, in a 2012 paper in the IEEE Transactions on Robotics, formalized multi-robot coordination with periodic connectivity and showed that requiring the team to be connected only at intervals, rather than continuously, could greatly improve search performance while still allowing the team to share information regularly. Their experiments used ground robots searching a building, but the principle transfers to drones. Role-based exploration makes the planned disconnection explicit. Julian de Hoog, Stephen Cameron and Arnoud Visser proposed in 2009 a scheme in which some robots act as explorers, venturing beyond communication range, and others as relays, which shuttle between explorers and the base, carrying information. Explorers and relays agree on rendezvous points and times. When an explorer reaches its rendezvous, it hands its accumulated data to the relay, receives updated information and instructions, and continues. Relays thus act as messengers physically carrying data, the same idea that Grossglauser and Tse showed could restore network capacity in the previous chapter. The design questions in planned disconnection are when and where to reconnect. Fixed schedules are simple and predictable; a drone knows it must be back at the stairwell landing at minute fifteen. Event-based reconnection is more efficient: a drone returns when it has found something worth reporting, when its battery reaches a threshold, or when it has explored a certain amount of new territory. Rendezvous points must be places where communication is reliably available, which the swarm learns as it explores. A drone that discovers a good place for a relay, such as a window with line of sight to the outside, has discovered something as valuable as a new room. Carrying data when it cannot be sent When a network is partitioned, conventional internet protocols, which assume an end-to-end path exists at the moment of sending, fail. Delay-tolerant networking was developed for exactly this situation. Its origins lie in work on an interplanetary internet led by Vint Cerf and colleagues at NASA's Jet Propulsion Laboratory around the turn of the century, where communication with spacecraft is periodically interrupted by planetary motion. The architecture was published by the Internet Engineering Task Force as RFC 4838 in 2007, and the current Bundle Protocol, version 7, as RFC 9171 in 2022. The central mechanism is store-carry-forward. Data is packaged into bundles, each with a destination and a lifetime. A node that receives a bundle stores it until a contact, a link to another node, appears, and then forwards it if that node is a better carrier toward the destination. The data may sit in a drone's memory for minutes, travel physically as the drone flies, and be handed over when the drone comes within range of another drone or a ground station. Earlier research named such carriers data mules, after a 2003 paper by Rahul Shah and colleagues, and message ferries, after work by Wenrui Zhao, Mostafa Ammar and Ellen Zegura in 2004, who showed that nodes could deliberately adjust their movement to carry messages between disconnected groups. For swarms, delay-tolerant networking means that a drone that photographs a possible survivor in a basement, out of contact with everyone, does not lose that information. It stores the detection, tagged with priority and location, and delivers it at the first contact. If the first contact is another searching drone, that drone can carry a copy too, so that whichever drone reaches the ground station first delivers the report. This epidemic spreading, described for ad hoc networks by Amin Vahdat and David Becker in 2000, maximizes delivery probability and minimizes delay at the cost of storage and redundant transmissions; practical systems limit the number of copies or prioritize by importance. Shared state that survives partition The deeper problem in a partitioned swarm is not delivering individual messages but maintaining shared state: which areas have been searched, which tasks have been claimed, where obstacles and detections are. When the swarm splits into groups that cannot communicate, each group continues to update its own copy of this state. When they reconnect, the copies must be merged, and the merge must produce a sensible result regardless of the order in which updates arrive. Computer scientists studying distributed databases have developed a theory for exactly this. Conflict-free replicated data types, formalized by Marc Shapiro, Nuno Preguiça, Carlos Baquero and Marek Zawirski in 2011, are data structures designed so that any two copies can be merged and the result is the same whatever the order of merging. A simple example is a set that only grows: the merge of two copies is their union, which is the same however many times and in whatever order merges occur. A coverage map in which each cell records the most recent time it was searched is another: the merge takes the later time for each cell. Designing a swarm's shared state from such structures means that reconnecting groups can reconcile without negotiation and without the possibility of contradictory outcomes. The Buzz programming language, developed by Carlo Pinciroli and Giovanni Beltrame at Polytechnique Montréal and published in 2016, built a construct called virtual stigmergy directly into a swarm programming language. Virtual stigmergy is a shared key-value store, replicated across robots, in which each entry carries a logical timestamp; robots that meet exchange entries, and newer entries replace older ones. It is the virtual pheromone of the second chapter made concrete, and its value lies in giving swarm programmers a shared memory that tolerates partition without requiring them to manage the merging themselves. Not all state can be made conflict-free. Task claims are the hardest case. If two disconnected drones each claim the same unclaimed task, both will do it, wasting effort. When they reconnect, a rule must decide whose claim stands, perhaps the earlier claim or the drone that is closer. The waste cannot be undone, only limited. The next chapter returns to this problem in detail, because it is at the heart of distributed task allocation. Spending bandwidth wisely Even when the network is connected, it is scarce, and the most effective improvements often come from deciding what not to send. Event-triggered communication, studied by Dimos Dimarogonas, Emilio Frazzoli and Karl Henrik Johansson among others, replaces periodic broadcasting with broadcasting only when something has changed enough to matter. A drone flying a predictable path need not broadcast its position ten times a second; its neighbours can predict it, and the drone need only transmit when its actual position departs from the prediction by more than a threshold. In formation flight, this can cut traffic sharply with little loss in performance. The same principle applies at higher levels. A drone with a thermal camera need not stream video; it can run detection on board and transmit a short message when it detects a likely heat signature, with a small thumbnail and its estimated location. A mapping drone need not transmit every scan; it can transmit compact summaries and send full data only on request or when convenient. Each such decision trades completeness for bandwidth, and the trade is worth making when the alternative is a saturated channel that delivers nothing reliably. Degradation by design All of these techniques can be organized into a set of operating modes that a swarm moves between as its network changes. When the network is fully connected to the ground, the swarm can stream data, accept operator instructions and coordinate tightly. When the swarm is connected internally but cut off from the ground, it can continue coordinating, store findings for later delivery and follow its last instructions. When it fragments into groups, each group can coordinate internally, avoid areas it believes others have claimed, and work to reconnect at planned points. When a single drone is isolated, it can continue its assigned task for a bounded time, fly conservatively, and return to its last known point of contact. Aviation has a long-established concept for single aircraft, the lost-link procedure, which specifies what a drone does when it loses contact with its pilot: typically climb, hold, and return home after a timeout. Swarms need a richer version, defined for the group as well as the individual, and it needs to be specified, tested and understood by operators before a mission. A swarm whose behaviour in each degraded mode is known in advance can be trusted to keep working when its links fail. A swarm whose degraded behaviour emerges unpredictably from interacting rules cannot. The tools in this chapter therefore serve the argument of the whole book. Because the network bounds what the swarm can know and do, the swarm's behaviour must be designed as a function of its network state. Connectivity maintenance, relays, planned disconnection, delay-tolerant transport, conflict-free shared state and disciplined bandwidth use are the means by which behaviour and communication are designed together. The next chapter applies the same discipline to the question of how a swarm divides its work. CHAPTER 7 Who Does What A swarm that can fly together, locate itself and communicate has solved the prerequisites. It has not yet done anything useful. Usefulness comes from dividing work: deciding which drone searches which sector, which investigates a heat signature, which returns to recharge, which stays high as a relay, which flies close to confirm a detection with a second camera. In a centrally controlled fleet, a mission planner makes these decisions with a global view. In a swarm, they must be made by the drones themselves, each knowing only part of the picture, communicating over the unreliable network described in the previous two chapters, while the picture changes underneath them. This is the problem of distributed task allocation, and it is where the network's limits bite hardest, because allocation is fundamentally about agreement, and agreement requires communication. A taxonomy of allocation problems The modern study of multi-robot task allocation is usually dated to a 2004 paper by Brian Gerkey and Maja Matarić in the International Journal of Robotics Research, which proposed a taxonomy along three dimensions. The first is whether each robot can do one task at a time or several. The second is whether each task needs one robot or several working together. The third is whether allocation is instantaneous, assigning current tasks with no thought for the future, or time-extended, planning sequences of tasks over time. The simplest case, one task per robot, one robot per task, instantaneous assignment, is the classic assignment problem: given a table of how well each robot would do each task, find the matching that maximizes total value. It has an exact, efficient solution, the Hungarian method published by Harold Kuhn in 1955, whose running time grows with the cube of the number of robots. Dimitri Bertsekas' auction algorithm, developed from the late 1970s, solves the same problem by having tasks bid up their prices iteratively, and lends itself naturally to distribution. Almost every other case is computationally hard. Time-extended allocation, where each drone must be assigned an ordered sequence of tasks, contains the travelling salesman problem as a special case: deciding the best order in which one drone visits a set of search cells is already hard in general, and dividing cells among several drones and ordering each drone's visits is a version of the vehicle routing problem studied for decades in operations research. Tasks requiring several robots at once, such as triangulating a radio beacon, which needs simultaneous measurements from at least three positions, add coordination constraints that make the problem harder still. G. Ayorkor Korsah, Anthony Stentz and M. Bernardine Dias extended the taxonomy in 2013 to capture dependencies between tasks, such as one task needing to be finished before another can start, which arise constantly in search and rescue. A detection must be confirmed before a supply drop is dispatched; an area must be mapped before it can be searched systematically. Because these problems are hard even with full information and a central computer, distributed task allocation does not aim for optimal solutions. It aims for solutions that are good enough, reached quickly, robust to communication failures, and adaptable as tasks appear and drones fail. The main families of method make different trade-offs among these aims, as Table 3 summarizes. Table 3. Families of distributed task allocation and how they respond to network limits. Method Communication needed Solution quality When the network partitions Representative work Response thresholds None or minimal Approximate, statistical Degrades smoothly Bonabeau et al. 1996; Krieger et al. 2000 Spatial partitioning Neighbour positions Good for coverage tasks Each part covers its own region Cortés et al. 2004 Contract net and auctions Bid and award rounds Good; some bounds Duplicate or orphaned tasks Smith 1980; Dias et al. 2006 Consensus-based bundles Repeated neighbour exchange Conflict-free; bounded optimality Conflicts resolved on reconnection Choi, Brunet and How 2009 Decentralized planning search Compressed plan exchange Strong on coupled tasks Plans on stale beliefs Best et al. 2019 Allocation without negotiation The cheapest form of task allocation needs no negotiation at all. It comes, again, from social insects. Eric Bonabeau, Guy Theraulaz and Jean-Louis Deneubourg proposed in 1996 a response threshold model of division of labour in insect colonies. Each individual has a threshold for each kind of task. Each task emits a stimulus that grows when the task is neglected, such as the hunger of larvae or the amount of debris in the nest. An individual whose threshold for a task is exceeded by the local stimulus takes up the task with high probability. Individuals with low thresholds respond first; if they cannot keep up, the stimulus rises and individuals with higher thresholds join. The colony allocates labour in proportion to demand without anyone counting tasks or workers. Michael Krieger, Jean-Bernard Billeter and Laurent Keller demonstrated the approach with real robots in a 2000 paper in Nature. Robots foraged for objects that represented food, needed to keep a nest's energy level up, and each had a different threshold for leaving the nest to forage. As the nest's energy fell, more robots went out. Because the thresholds differed, the number of active foragers rose and fell with the nest's needs, rather than every robot responding at once and crowding the others, and the group regulated its effort without any robot knowing what the others were doing. For a drone swarm, thresholds are a natural fallback when communication fails. A drone that cannot negotiate can still observe local stimuli: how much unsearched area is near it, how many neighbours it can hear, how much battery it has left. A threshold rule such as "if I can hear fewer than two neighbours and there is unsearched area within reach, search it; if my battery is below a level, head home" needs no agreement. It will produce duplication and gaps, but it will degrade smoothly rather than freezing, which is the property that matters when the network is down. Spatial partitioning is another negotiation-free approach for coverage tasks, which is to say most search tasks. Jorge Cortés, Sonia Martínez, Timur Karataş and Francesco Bullo showed in 2004 how mobile sensors can spread out to cover an area optimally using only neighbours' positions. Each sensor computes its Voronoi cell, the region of space closer to it than to any other sensor, and moves toward the centroid of its cell, weighted by how important each point is. Repeated, this converges to a configuration in which the area is divided evenly according to importance, a distributed version of an algorithm due to Stuart Lloyd. In a search, the importance weighting can be the probability of finding a survivor, so that drones crowd more densely where survivors are more likely. If a drone fails, its neighbours' cells expand to absorb its region automatically. If the network partitions, each part continues to divide its own region among its members. The limitation is that partitioning handles coverage well and discrete tasks poorly: it has nothing to say about which drone should fly to investigate a specific detection. Markets and auctions The most widely studied family of explicit allocation methods treats tasks as goods to be traded. The Contract Net Protocol, described by Reid Smith in 1980 for distributed problem solving among computers, set the pattern: a manager announces a task, contractors submit bids reflecting their cost or suitability, and the manager awards the task to the best bidder. In a robot team, any robot that discovers a task can act as manager for it. M. Bernardine Dias, Robert Zlot, Nidhi Kalra and Anthony Stentz surveyed market-based multi-robot coordination in the Proceedings of the IEEE in 2006 and found that markets offered a good balance: much better solutions than purely reactive methods, much lower communication and computation than centralized optimization, and natural robustness, since a robot that fails simply stops bidding and its tasks can be re-auctioned. The details of auction design matter. In sequential single-item auctions, studied by Michail Lagoudakis, Sven Koenig and colleagues in the mid-2000s, tasks are auctioned one at a time; each robot bids the extra cost of adding the task to its current plan; the lowest bidder wins and the next task is auctioned. This is simple and, for routing problems, provably produces solutions within a constant factor of optimal under certain cost measures. Combinatorial auctions, in which robots bid on bundles of tasks, can capture synergies, such as two search cells that are adjacent and therefore cheap to do together, but the number of possible bundles grows explosively. Auctions have one fundamental dependence on the network: someone must collect the bids and announce the winner, and every bidder must hear the announcement. When the network is partitioned, a task may be auctioned separately in two partitions and awarded twice, or an award may not reach the winner, leaving the task orphaned. Auction protocols include timeouts and confirmations to manage this, but each round of messages adds delay, and in a network with high loss, auctions can stall. Consensus-based bundle algorithms The method that has become a reference point for decentralized allocation among drones is the Consensus-Based Bundle Algorithm, published by Han-Lim Choi, Luc Brunet and Jonathan How at the Massachusetts Institute of Technology in 2009 in the IEEE Transactions on Robotics. Its design addresses precisely the problem that auctions have with unreliable networks: there is no auctioneer. Each drone runs two phases in alternation. In the bundle construction phase, the drone builds an ordered list of tasks it would like to do, adding tasks one at a time, each time choosing the task that most increases its total score given the tasks already in its bundle. For each task it adds, it records its bid, the marginal value it would contribute. In the consensus phase, the drone exchanges with its neighbours its current knowledge of who is winning each task and at what bid, along with timestamps showing how recent that knowledge is. Using a set of rules that compare bids and timestamps, each drone updates its view of the winners. If a drone learns that someone else is winning a task in its bundle with a higher bid, it drops that task and every task it added after it, because their marginal values were computed assuming the dropped task was included. Then it rebuilds. Choi, Brunet and How proved that if the network is connected and the scoring function satisfies a property called diminishing marginal gain, meaning that adding a task to a bundle never makes other tasks more valuable, the algorithm converges to a conflict-free assignment, with each task assigned to at most one drone, within a bounded number of rounds related to the network's diameter and the number of tasks, and that the assignment achieves at least half the optimal value. Crucially, convergence does not require drones to have consistent information about the world: drones can disagree about task locations or values and still reach a conflict-free assignment, because they agree on bids rather than on underlying facts. This matters for swarms, whose members' views of the world are, as the fourth chapter showed, always somewhat inconsistent. Later work relaxed the algorithm's assumptions. An asynchronous version developed by Luke Johnson, Sameera Ponda, Han-Lim Choi and Jonathan How around 2011 removed the requirement that drones exchange messages in synchronized rounds, which is unrealistic over a mesh with variable delay. Other extensions handled tasks that require several drones, tasks with time windows, and uncertainty about task values. The diminishing marginal gain condition is not merely technical: scoring functions that violate it, such as those rewarding drones for visiting tasks in a sequence that saves travel, can cause the algorithm to cycle without converging. Designers must shape scores carefully. The behaviour of consensus-based bundle algorithms under partition is instructive. Each partition converges to its own conflict-free assignment. Across partitions, conflicts can exist: two drones in different partitions may both have claimed the same task. When the partitions reconnect, the consensus phase detects and resolves the conflict, and the losing drone drops the task. Work done in the meantime may be duplicated, but the system recovers to a consistent state without intervention. That is exactly the graceful degradation the book has been arguing for. Planning over coupled tasks Search tasks are often more coupled than auction formulations assume. The value of searching a cell depends on what has been learned elsewhere: if a hiker's backpack is found near a stream, the probability that the hiker is upstream rises and the value of cells downstream falls. The value of a drone's next action depends on what other drones will do. Such problems are naturally framed as joint planning under uncertainty, which is intractable to solve exactly for even modest team sizes. Graeme Best, Oliver Cliff, Timothy Patten, Ramgopal Mettu and Robert Fitch proposed in 2019, in the International Journal of Robotics Research, a decentralized version of Monte Carlo tree search, the planning method behind game-playing programs such as AlphaGo. Each robot plans its own actions by simulating many possible futures, using a probabilistic model of what other robots will do. Robots periodically exchange compressed summaries of their own likely plans, which improves each other's models. Planning continues between exchanges, so that the method degrades gracefully as communication becomes sparse: robots plan with older information about one another, which costs some coordination quality but never stops the planning. This is a general pattern in state-of-the-art methods, which treat communication as something that improves plans when available rather than as a prerequisite for planning at all. Learning-based allocation, using multi-agent reinforcement learning, is an active research area. Agents learn policies that map local observations and messages to actions, sometimes including learning what to communicate. Results in simulation are often impressive. The difficulties are the familiar ones: policies may not transfer from simulation to the field, may behave unpredictably on situations outside their training, and are hard to verify. For safety-critical civilian applications, learned allocation is more likely to appear as a component inside a structured framework, for instance providing value estimates to a bundle algorithm, than as a replacement for it. A worked illustration Consider a hypothetical swarm of six drones searching a wooded hillside for a missing walker, organized as described in this chapter. The search area has been divided by a probability map into forty cells of varying priority. The drones launch together and, using a weighted Voronoi partition, spread across the area with denser coverage over high-probability cells near a footpath and a stream. Within each drone's region, it orders its cells by a bundle construction that balances priority against travel distance. Twelve minutes in, drone 3's thermal camera registers a warm object under trees near the stream. It creates a new task, confirm the detection, which needs a second sensor view from a different angle and lower altitude. It broadcasts the task with its location. Drones 2 and 4 are in range. Each computes a bid: drone 2 is closer but has less battery; drone 4 is further but has more. Through the consensus phase, drone 4's higher bid wins, it inserts the confirmation task at the head of its bundle, and its previous search cells are released. The Voronoi partition shifts as drone 4 moves toward the stream, and drone 5's region expands to absorb some of drone 4's former cells. The ground team is notified through a relay chain when one exists and through stored-and-forwarded messages when it does not. Meanwhile, drone 6, on the far side of a ridge, has lost contact with everyone. It continues searching its assigned cells, records the cells as searched in its own copy of the coverage map, and stores its imagery. Because the coverage map is a conflict-free structure, when drone 6 returns over the ridge and reconnects, its search history merges cleanly into the other drones' maps. Two cells near the ridge had been reassigned to drone 5 while drone 6 was out of contact, and both drones searched them, a small duplication that was the cost of drone 6's isolation. Drone 1's battery reaches its return threshold; it announces it is withdrawing, its remaining cells are released and reabsorbed, and it flies home. Nothing in this illustration is exotic. Each element, the weighted partition, the bundle construction, the consensus on bids, the conflict-free coverage map, the battery threshold, is drawn from methods described above. What makes the system work is that each element degrades gracefully: the partition reshapes when drones leave, the consensus resolves conflicts after the fact, the map merges regardless of order, and the battery rule needs no communication at all. A design that required every drone to agree on every reassignment before acting would have stalled when drone 6 disappeared. Allocation and the network The family of methods in this chapter can be read along a single axis: how much communication each requires before acting, and what it does when that communication is missing. Threshold models act on local observation and never wait. Spatial partitioning acts on neighbours' positions, which can be sensed directly. Auctions require rounds of messages and can stall or duplicate under partition. Consensus-based bundles require repeated exchange but resolve conflicts eventually without a coordinator. Decentralized planning uses communication to improve plans but plans regardless. Real systems combine these. A swarm may use partitioning for baseline coverage, bundle algorithms for discrete tasks, and thresholds as a fallback when isolated. The choice of which mechanism governs at any moment depends on the state of the network, which is the theme this book keeps returning to. A swarm's division of labour is not a fixed property of its software but a function of who can hear whom. The next chapter puts all of this to work in the application where it matters most. CHAPTER 8 Search and Rescue as the Proving Ground Of all the civilian uses proposed for drone swarms, search and rescue is the one that most clearly rewards what swarms do well and most harshly punishes what they do badly. It rewards parallel coverage, because survival probabilities fall with time. It rewards robustness to loss, because the environments are hostile. It rewards independence from infrastructure, because disasters destroy infrastructure and wilderness never had it. And it punishes unpredictability, information overload and any demand on the attention of people who are already stretched. This chapter looks at how search is actually done, what drones already contribute, where swarms add something single drones cannot, and how the components described in the previous chapters fit together in an operation. It ends with a briefer look at mapping, agriculture and scientific research, where the same principles apply under less pressure. The science of looking Search has a mathematical theory, and it predates drones by eighty years. During the Second World War, the operations research group of the United States Navy, led in this area by Bernard Koopman, developed a theory of search to improve the effectiveness of aircraft looking for submarines. Koopman's work, later published as Search and Screening, introduced the concepts that still structure search planning: the probability that the target is in a given area, the probability of detecting it if it is there and that area is searched, and the effective sweep width, a measure of how wide a swath a given sensor and searcher effectively cover. Lawrence Stone's Theory of Optimal Search, published in 1975, put the theory on rigorous foundations and showed how to allocate search effort optimally among areas. The core logic is Bayesian. Before searching, planners estimate, for each area, the probability that the missing person is there, drawing on where they were last seen, their likely behaviour and the terrain. Searching an area and not finding the person does not mean the person is not there; detection is imperfect. It does lower the probability for that area and, correspondingly, raises the probability for areas not yet searched. Effort should flow, at each moment, to where the product of the probability of the person being there and the probability of detecting them there is highest. A well-known demonstration of this approach at large scale was the search for the wreckage of Air France flight 447, which disappeared over the Atlantic in 2009. After two years of unsuccessful searching, analysts at Metron, a firm specializing in search theory, produced a Bayesian probability map that accounted for the evidence from earlier, unsuccessful searches, including the possibility that the aircraft's underwater locator beacons had failed, which would explain why the early acoustic search had heard nothing. The wreckage was found in April 2011 within a week of resuming search in the highest-probability area, as Stone and colleagues later described in Statistical Science. For land search, the main input to the probability map is lost person behaviour. Robert Koester's book Lost Person Behavior, published in 2008 and based on a large international database of incidents, categorizes missing people, hikers, hunters, children of various ages, people with dementia, people who are despondent, and gives statistics on how far each category typically travels and what kinds of terrain features they are found near. Children and people with dementia behave very differently from experienced hikers; some categories tend to follow paths and drainages, others to stop and shelter. Search managers use these statistics to define search areas and weight their probabilities. This theory matters for swarms in a specific way. A swarm's task allocation and coverage algorithms need an objective, and the probability map provides it. The weighted Voronoi partitioning of the previous chapter, with density weighted by probability of the person being in each area, is a direct implementation of optimal search effort allocation. The Bayesian update after unsuccessful search can be performed by each drone on its own copy of the map and merged across drones as they communicate, so that the swarm's shared belief about where the person is evolves as the search progresses. A swarm that searches uniformly, ignoring the probability map, wastes much of its advantage. What drones already do Single drones are already standard equipment for many search and rescue teams. Their thermal cameras detect body heat at night and in cool conditions, and their visual cameras give searchers views of terrain that would take hours to reach on foot. Drones deliver flotation devices to people in water, carry radios and medical supplies to stranded people, and give incident commanders an overview of an operation. Research has also documented the limitations. Michael Goodrich and colleagues at Brigham Young University, working with Utah search and rescue teams, published in 2008 in the Journal of Field Robotics one of the first careful studies of drone use in wilderness search. They found that operating the drone, watching its video and integrating what it saw into the search were distinct roles that one person could not perform well at once, and that the bottleneck was often human attention to the video rather than the drone's coverage. Small figures in aerial imagery are easy to miss, especially when partly hidden by vegetation, and a person reviewing hours of footage tires. Onboard detection with machine-learning models trained to find people in aerial and thermal imagery has improved substantially since then, and it reduces the problem by flagging candidate detections for human review rather than requiring humans to watch everything. But it introduces its own trade-off between missed detections and false alarms, and every false alarm consumes a searcher's time. Some search tasks use sensors other than cameras. Avalanche transceivers, worn by backcountry skiers, transmit at an internationally standardized frequency of 457 kilohertz, and drones carrying receivers can search avalanche debris faster than rescuers on foot. Mobile phones emit signals that drones carrying suitable receivers can detect and locate, an approach under active development in research and by commercial vendors. For radio sources, multiple drones are especially valuable: simultaneous measurements from several positions locate a source far faster than one drone moving around it, which is one of the clearest cases where a swarm outperforms a single drone. Where swarms add value Measured against the single drones already in use, swarms offer four distinct contributions. The first is coverage. A swarm of ten small drones can, in principle, search a given area roughly ten times as fast as one, provided coordination prevents duplication and gaps. That proviso is the whole of the previous chapter. With good task allocation, coverage scales nearly linearly with the number of drones; with poor coordination, much of the additional effort is wasted. The second is reach. Small drones can enter collapsed structures, culverts, dense forest and cave entrances where larger drones cannot. A swarm can push into such places with some members acting as relays and others exploring, using the techniques of the sixth chapter. The DARPA Subterranean Challenge showed that robots can explore and map underground spaces largely autonomously, and the 2024 to 2026 SAPIENCE competition, sponsored by NATO's Science for Peace and Security Programme and explicitly aimed at search and rescue, set teams of universities the task of fielding autonomous drone swarms that could navigate without satellite positioning, divide tasks among themselves, identify survivors and deliver supplies according to urgency, in disaster scenarios combining indoor and outdoor environments. The final round, held in the summer of 2026 at the Unmanned Valley test site in the Netherlands, was won by the team from City St George's, University of London. The third is simultaneous perspective. Several drones viewing a scene at once allow three-dimensional reconstruction of a collapsed structure, triangulation of radio sources, and confirmation of detections from multiple angles, which reduces false alarms. A heat signature seen from two angles at once is much more likely to be a person than one seen from one angle. The fourth is persistence through loss. A search operation using one drone stops when that drone crashes or its battery runs out. A swarm continues, with remaining drones absorbing the lost member's area. In practice, this also means that a swarm can rotate members through battery changes without interrupting coverage, which matters in operations that last many hours. A mission in outline Consider how these elements might combine in an urban disaster, for example after an earthquake that has damaged a neighbourhood of mid-rise buildings, in darkness, with cellular networks partly down. The scenario is hypothetical, but each capability in it has been demonstrated in research or is in operational use. In the first phase, the swarm builds an overview. Drones launched from the edge of the affected area fly above rooftop level, where satellite positioning is usually still available and air-to-air radio links are good. They divide the area by weighted partitioning, giving priority to buildings with the most visible damage and the highest likely occupancy at that hour. Each drone maps its region with cameras and, if carried, laser scanners, and the maps are merged through collaborative mapping. Within minutes, the incident commander has an updated map of the area showing which structures have collapsed, where roads are blocked and where fires are burning. This overview phase is where existing drone mapping practice is strongest, and swarms mainly add speed. In the second phase, the swarm looks for signs of life. Drones with thermal cameras survey exposed rubble and open spaces. Drones with radio receivers listen for mobile phones, and when one detects a signal, nearby drones join to triangulate it. Each detection becomes a task in the allocation system, confirmed by a second view or sensor before being reported. Detections are prioritized and sent to the command post through the mesh, which at this altitude usually reaches the ground station directly or through one relay. In the third phase, small drones enter structures that rescuers have judged worth investigating but too dangerous to enter immediately. Here satellite positioning disappears and radio links degrade rapidly. One or two drones position themselves at openings as relays, or drop small radio nodes at the entrance. Explorer drones move inside using visual-inertial odometry and lights, mapping as they go, and periodically return to relay range to deliver their findings. Their maps are merged with the exterior maps through loop closures at openings. What they find, voids where survivors might be sheltering, a heat signature, a voice detected by a microphone, is reported to the rescue team, whose structural specialists decide where to cut and shore. Throughout, the swarm's behaviour is shaped by its network. Above rooftops it is connected and centrally supervised. Inside structures it is fragmented and largely autonomous. Its task allocation shifts from partition-based coverage outdoors to explorer-relay coordination indoors. Its information flow shifts from streaming to store-and-forward. The design question at every stage is what the swarm does with the communication it has. People in the loop None of this works unless the swarm fits into how rescue operations are actually run. Large incidents are typically managed under a structured command system, such as the Incident Command System widely used in the United States, in which air operations are a defined function with a supervisor who coordinates all aircraft, crewed and uncrewed. Urban search and rescue teams that deploy internationally follow guidelines set by the International Search and Rescue Advisory Group, a United Nations network, including standardized ways of marking searched structures and reporting findings. A swarm's outputs must fit these structures, which means producing information in forms the command system can use, and accepting direction in forms commanders can give. Andreas Kolling, Phillip Walker, Nilanjan Chakraborty, Katia Sycara and Michael Lewis surveyed human interaction with robot swarms in the IEEE Transactions on Human-Machine Systems in 2016. Their review identified a set of persistent difficulties. Operators struggle to perceive swarm state, because many drones generate more information than a person can track. They struggle to predict swarm behaviour, because emergent behaviour is hard to anticipate. And commands have delayed and indirect effects, because they propagate through the network and are interpreted by local rules. The most effective interfaces they reviewed let operators work at the level of the swarm as a whole: drawing areas to search or avoid, adjusting priority maps, assigning goals to an informed minority of drones in the manner of the leadership models of the second chapter, and receiving summarized findings rather than raw data. A recurring finding in this research is that operators need to understand the swarm's communication state as well as its physical state. An operator who sees drones on a map but does not know which of them are currently reachable will issue commands that some drones never receive and will draw wrong conclusions from the absence of reports. An interface that shows which drones are connected, how fresh each drone's last report is, and where communication is reliable gives the operator an accurate sense of what the swarm can do next. This is the book's controlling idea reappearing at the level of human-machine interaction. Beyond rescue: mapping, agriculture and research The same capabilities serve less urgent civilian tasks, where the pressure of time is lower but the economics of coverage are similar. In mapping and surveying, multi-drone operation is already common in the fleet sense: a site is divided among several drones, each flies a planned pattern, and images are combined into maps and three-dimensional models. Swarm methods add value when the environment degrades positioning or communication, such as mapping the inside of a large structure, a mine or the underside of a bridge, or when the area is too large for a single pre-planned operation and drones must adapt as they discover terrain. In agriculture, the most significant regulatory step toward multi-drone operation in the United States was an exemption granted to the spraying-drone company Hylio in early 2024, permitting a single pilot to operate up to three agricultural drones each weighing more than 55 pounds at once, including at night, under the rules for agricultural aircraft operations. The drones remain within the pilot's visual line of sight and fly pre-planned routes, so this is a coordinated fleet rather than a swarm in this book's sense, but it established that the regulator would accept one-to-many operation for heavy drones given adequate safety systems. The National Agricultural Aviation Association, representing crewed agricultural pilots, raised concerns about obstacle avoidance and whether one pilot can adequately supervise several spraying aircraft, concerns that illustrate the debates swarm operations will face more generally. Research on genuine agricultural swarms has focused on tasks such as weed mapping, where drones divide a field and adaptively concentrate on areas where weeds are detected; Dario Albani, Daniele Nardi and Vito Trianni described such a system in 2017 as part of the European SAGA project. In environmental research, drones already sample volcanic gas plumes, count wildlife, measure glaciers and monitor forests. Many such tasks benefit from simultaneous measurement at multiple points: measuring a gas plume's structure, or the movement of an animal herd, requires observations at the same moment from different positions. Swarm coordination can provide this, and the scientific setting is often more tolerant of experimental technology than an active rescue. Research sites can therefore serve as proving grounds for capabilities that later move into emergency use. What search and rescue teaches Search and rescue makes clear that a swarm's value lies not in the number of drones but in coordination quality, and that coordination quality is bounded by communication. A swarm that duplicates effort, misses areas, floods operators with unconfirmed detections or behaves unpredictably when links fail may be worse than a single well-operated drone. A swarm that divides effort according to a shared and updated probability map, confirms detections before reporting, shifts its coordination mode as its network changes, and shows its operators its communication state can find people faster than any single aircraft. The difference between these two swarms is not in their airframes or cameras. It is in how they handle information that is partial, delayed and distributed, which is to say in everything the preceding chapters have described. The final chapter addresses the conditions under which such systems can be trusted to fly at all. CHAPTER 9 Trust, Safety and Governance A swarm that works in a research trial is not yet a swarm that can be used. Between the two lies a set of questions that engineers sometimes treat as afterthoughts and that, in practice, decide whether a technology is ever deployed. How does the swarm fail, and how badly? How can anyone know, before it flies over people, that it will behave as intended? What happens if someone interferes with it, deliberately or by accident? Under what rules may it fly? And how should a society regulate a technology whose civilian and military uses share so many components? This chapter takes up those questions. Its argument is the same as the rest of the book's: because a swarm's behaviour depends on its network, the questions of safety, security and governance are questions about the network as much as about the aircraft. How swarms fail Single drones fail in familiar ways: a motor stops, a battery depletes faster than expected, a sensor gives bad readings, a pilot makes a mistake. Swarms add failure modes that arise from interaction. The first is common-mode failure. Because swarm members typically run identical software on identical hardware, a defect or an unanticipated condition can affect all of them at once. A bug in the flocking code, a sensor that misbehaves in a particular temperature range, a timing assumption violated by an unusually congested channel: any of these can strike every drone simultaneously. Centrally choreographed drone shows provide public examples of what simultaneous failure looks like, because they fly over crowds. In December 2024, during a holiday show in Orlando, Florida, a number of drones fell from the sky into spectators, and a child was seriously injured; the Federal Aviation Administration opened an investigation. The cause in such cases may lie in the central system rather than in decentralized behaviour, but the lesson for swarms is the same: a fleet of identical aircraft can fail together, and redundancy in numbers does not protect against faults that every member shares. The second is propagation of bad information. In a swarm, one drone's error can become everyone's. A drone with a faulty position estimate shares it with its neighbours, who adjust their own estimates or their formation to match. A false loop closure distorts the shared map, as the fourth chapter described. A drone that incorrectly believes it has searched an area marks it as covered in the shared coverage map, and no other drone searches it. The robustness that comes from sharing information has a reverse side: shared errors. The third is emergent misbehaviour. Rules that produce good behaviour under tested conditions may produce poor behaviour under untested ones. A flock that is stable at one communication delay may oscillate at a longer one. A task allocation scheme that converges under normal conditions may cycle when scores change rapidly. These failures are not caused by any individual drone's malfunction; each drone does what its rules say. They arise from the interaction of correctly functioning parts, which makes them hard to anticipate and hard to diagnose after the fact. The fourth is network-induced failure. A swarm that loses connectivity at a critical moment, perhaps while converging on a detection, may behave in ways its designers did not consider: drones that cannot hear each other's trajectories may plan conflicting paths, drones that cannot receive task updates may duplicate effort, and drones that interpret silence as failure may reassign tasks that are actually being done. Because the network's state depends on the drones' positions, failures can be self-reinforcing: drones that move apart to cover more ground lose contact, which degrades their coordination, which may move them further apart. Knowing before flying Aviation safety rests on the ability to show, before an aircraft flies, that it will behave acceptably. For conventional aircraft, this involves analysis, testing and certification of components and systems against defined standards. For swarms, the micro-macro gap described in the first chapter makes this unusually difficult. Engineers can specify and test what each drone does, but the safety-relevant behaviour is collective. Alan Winfield and colleagues at the Bristol Robotics Laboratory began in the mid-2000s to argue that swarm robotics needed a discipline of swarm engineering, including formal methods for specifying and verifying emergent behaviour. Their work, and later work by Clare Dixon, Michael Fisher and others, applied temporal logic and model checking to swarm algorithms: specifying properties such as "the swarm never splits into disconnected groups" or "every robot eventually reaches the goal area," and checking them exhaustively against a model of the system. The difficulty is scale. Exhaustive checking becomes impossible as the number of robots and states grows, so practical verification combines formal checking of small configurations with statistical methods, which run large numbers of randomized simulations and estimate the probability that a property holds. Simulators designed for swarms, such as ARGoS, developed by Carlo Pinciroli and colleagues in Dorigo's group and published in 2012, can simulate thousands of robots faster than real time, which makes such statistical verification feasible. The weakness of all simulation-based verification is fidelity. A property verified in a simulator with idealized communication may fail in the field. The flocking work of Vásárhelyi and colleagues, described in the third chapter, succeeded in part because their simulator modelled delays and noise realistically; the same principle applies to verification. A credible safety case for a swarm must include a communication model validated against measurements from the environments in which the swarm will fly, and evidence about behaviour across the full range of network conditions, from fully connected to fragmented. A complementary approach accepts that the main controller cannot be fully verified and wraps it in a simpler layer that can be. The simplex architecture, proposed by Lui Sha and colleagues in the 1990s and 2000s for safety-critical systems, pairs a complex, high-performance controller with a simple, verified safety controller and a monitor that switches to the safety controller when the system approaches unsafe states. Control barrier functions, described in the third chapter, are a modern form of this idea for multi-robot systems: whatever the mission controller commands, the barrier layer modifies it minimally to preserve separation. Geofences that prevent drones from leaving an approved area, and lost-link behaviours that bring drones to a safe state when communication fails, are other examples. For regulators, this layered approach is attractive because it allows the safety case to rest on the simple, verifiable layer rather than on the complex, adaptive one. When someone interferes Civilian swarms will operate in environments where interference is possible, sometimes from deliberate attackers and more often from accidental sources. Both matter. Satellite navigation is the most obvious target. In 2012, Todd Humphreys and his group at the University of Texas at Austin demonstrated publicly, at the invitation of the United States Department of Homeland Security, that they could take control of a small drone's navigation by transmitting counterfeit satellite signals, causing the drone to believe it was somewhere other than where it was. The wider prevalence of satellite navigation interference since 2022, discussed in the fourth chapter, means that civilian swarm designers must assume their satellite receivers may be misled, not merely blocked. The defences are the same as those for operating without satellites: visual and inertial navigation, relative positioning between drones and consistency checking between independent position sources. A swarm has an advantage here over a single drone, because if drones are spread out, a spoofing signal may affect them differently, and comparing their reports can reveal the deception. The mesh network is a second target. Jamming, flooding the radio band with noise, can partition a swarm or isolate it from the ground. Because civilian swarms use unlicensed bands shared with many other devices, accidental interference is common and deliberate jamming is easy. The defences include frequency agility, changing channels when interference is detected; operation in multiple bands; directional antennas; and, most fundamentally, the graceful degradation designs of the sixth chapter, which allow the swarm to continue its mission and remain safe when communication fails. A swarm that crashes when jammed is dangerous. A swarm that carries on sensibly when jammed is merely inconvenienced. Message integrity is a third concern. In an open network, an attacker might inject false messages: fake position reports to provoke collisions, fake task claims to leave areas unsearched, fake detections to divert resources. Cryptographic authentication, in which each message is signed with a key that only legitimate swarm members hold, defends against outsiders, at the cost of computation, message size and key management. It does not defend against a compromised member, which holds valid keys. Distributed computing has studied this problem since the early 1980s under the name of Byzantine faults, after a paper by Leslie Lamport, Robert Shostak and Marshall Pease describing generals who must agree on a plan despite traitors among them. Their result established that agreement is possible only if fewer than a third of participants are faulty and enough communication occurs. In swarm robotics, Volker Strobel, Eduardo Castelló Ferrer and Marco Dorigo showed in 2018 how blockchain techniques could let a robot swarm reach collective decisions while limiting the influence of Byzantine robots that report false information. Such methods are costly in communication, which on a constrained mesh is precisely the scarce resource, and their practical use in field swarms remains limited. A related risk, identified by John Douceur in 2002, is the Sybil attack, in which a single malicious node pretends to be many, gaining disproportionate influence over any voting or consensus process; defences require some way to bind identities to physical devices. The rules of the air Swarms fly in airspace that is regulated, and the regulations were written for aircraft flown one at a time by pilots. As Table 4 summarizes, the main regimes in the United States and the European Union were in different states of evolution in late 2026. Table 4. Principal civil drone rules relevant to swarm operations, as of late 2026. Instrument Jurisdiction Status Relevance to swarms 14 CFR Part 107 United States In force since 2016 One aircraft per remote pilot and visual line of sight, both waivable 14 CFR Part 89, Remote ID United States Compliance required since 2023 Each drone must broadcast identity and location Proposed Part 108, beyond visual line of sight United States Proposed August 2025; final rule under review in 2026 Would create a routine path for automated operations beyond sight Regulation (EU) 2019/947 European Union Applicable since 2020 Swarms fall in the specific category, assessed case by case Regulation (EU) 2021/664, U-space European Union Applicable since January 2023 Digital traffic services for drones in designated airspace In the United States, small commercial drones fly under Part 107 of the federal aviation regulations, in force since 2016. Two of its provisions bear directly on swarms: drones must generally stay within the visual line of sight of the pilot or a visual observer, and a person may not act as remote pilot of more than one drone at a time. Both can be waived on application with an adequate safety case, and the Hylio exemption for agricultural drones described in the previous chapter is an example of multi-aircraft operation authorized by exemption. Remote identification rules, in Part 89, required most drones to broadcast their identity and location from September 2023, with enforcement beginning in March 2024. Each member of a swarm must comply, which adds a broadcast to every drone's radio load and makes swarm operations visible to anyone with a receiver. The most consequential pending change is the proposed Part 108, published by the Federal Aviation Administration as a notice of proposed rulemaking in August 2025, which would establish a standard regulatory path for operations beyond visual line of sight instead of the current case-by-case waivers. The proposal shifts emphasis from the individual remote pilot toward the operating organization and introduces supervisory roles suited to highly automated operations. An executive order had set a deadline for a final rule in early 2026; that deadline passed, and the final rule was submitted for review to the White House Office of Information and Regulatory Affairs in July 2026, where it remained in the autumn. Whatever its final form, it will define the conditions under which one person can supervise several automated drones beyond their sight, which is the regulatory precondition for most practical civilian swarm operations. In the European Union, drone operations are governed by Implementing Regulation 2019/947, which divides operations into open, specific and certified categories according to risk. Swarm operations almost always fall in the specific category, where the operator must obtain authorization based on a risk assessment, usually following the Specific Operations Risk Assessment methodology published by the Joint Authorities for Rulemaking on Unmanned Systems, an international group of regulators, which issued a revised version 2.5 in 2024. The assessment considers ground risk, the chance of hurting people below, and air risk, the chance of colliding with other aircraft, and requires mitigations proportionate to both. The U-space regulation, applicable from January 2023, establishes a framework for digital traffic management services in designated airspace, including network identification, geo-awareness and traffic information, which in principle allows drones, including swarm members, to share airspace with one another and with crewed aircraft under automated coordination. Emergency operations sometimes receive special treatment. In the United States, public safety agencies can obtain expedited authorizations for emergency operations, and in many jurisdictions state aircraft used by police, fire and rescue services operate under different rules from commercial drones. These provisions matter for search and rescue swarms but do not remove the need to demonstrate safety; they shift who decides and how quickly. Dual use and public policy Drone swarms are a dual-use technology. The algorithms for flocking, mesh networking and task allocation described in this book are equally applicable to military systems, and military research funding has shaped the field, including several of the programmes mentioned here. The war in Ukraine since 2022 has made small drones and the electronic contest over their communications a prominent feature of modern conflict, and governments have responded with large investments in autonomous systems. This book does not discuss military applications or how to build them. But civilian practitioners cannot ignore the policy debate, because it shapes export controls, research funding, public perception and the regulatory environment for civilian work. At the international level, the principal forum has been the Group of Governmental Experts on lethal autonomous weapon systems, established in 2016 under the Convention on Certain Conventional Weapons in Geneva. Its mandate was extended in 2023 to develop elements of a possible instrument, without predetermining its form, and in March 2026 its working text characterized such a system as a combination of weapons and technological components that can identify, select and engage a target without intervention by a human operator. States remained divided: a large and growing group supported negotiating a legally binding instrument, while several major military powers, including the United States and Russia, opposed binding rules. The United Nations General Assembly adopted its first resolution on the subject, Resolution 78/241, in December 2023 and a further resolution, 79/62, in December 2024, both by large majorities, which broadened discussion beyond the Geneva forum. National policies have developed alongside; the United States Department of Defense updated its directive on autonomy in weapon systems, Directive 3000.09, in January 2023, requiring that such systems allow commanders and operators to exercise appropriate levels of human judgment over the use of force. For civilian swarm researchers and operators, the practical implications are threefold. First, export controls and research security rules may apply to some swarm technologies, particularly high-performance mesh radios and autonomy software, and practitioners should understand them. Second, public trust in civilian swarms is affected by the military associations of the word "swarm," and civilian operators benefit from transparency about what their systems do and do not do. Third, the principle that emerged in the policy debate, that meaningful human judgment should be retained over consequential decisions, is a sound principle for civilian systems too. A search swarm can decide autonomously where to look; it should not decide autonomously that a search is over. Privacy and public acceptance Civilian swarms raise civilian concerns as well. A swarm of camera-equipped drones searching a residential area for a missing child is also, inevitably, recording everything it sees. Data protection law in Europe and privacy expectations elsewhere constrain what may be collected, how long it may be kept and who may see it. Search and rescue agencies generally operate under legal bases that permit such collection in emergencies, but the data still needs to be handled carefully, with clear retention limits and access controls. Onboard processing that transmits detections rather than raw imagery, adopted in earlier chapters for bandwidth reasons, also serves privacy: imagery that is never transmitted cannot be leaked. Public acceptance depends on more than law. Noise, the sight of many drones, and uncertainty about who operates them all affect how communities respond. Remote identification helps by making operators identifiable. Clear communication by agencies about why drones are flying, what they are looking for and what happens to the data helps more. A swarm that finds a missing person will be welcomed; a swarm that seems to be watching everyone, for unclear reasons, will not. Trust as an engineering requirement The conclusion of this chapter is that trust is not a public relations problem to be solved after the engineering is done. It is an engineering requirement with specific technical content. A trustworthy swarm has known failure modes and known behaviours in each degraded network state. Its safety rests on simple, verifiable layers rather than on the correctness of complex emergent behaviour. It detects and resists interference, and when it cannot resist, it fails safe. It complies with airspace rules and makes itself identifiable. It retains human judgment over consequential decisions. And it handles the information it collects with restraint. Each of these properties depends on how the swarm's behaviour is coupled to its network, which is why they belong in the design from the beginning rather than being added at the end. Conclusion The chapters of this book have treated flocking, positioning, networking, task allocation, search and governance in turn. Read together, they support a claim stronger than any one of them makes alone: the decisive engineering choices in a drone swarm are choices about what each drone does with partial, delayed and intermittently available information from the others. The aircraft, sensors and algorithms matter, but they are bounded by the network, and the swarms that work in the field are those designed around that bound rather than in denial of it. Several consequences follow, and they are worth stating plainly, because they cut against the way swarms are usually presented and the way their development is usually organized. The unit of design is the coupling, not the component The first consequence is that a swarm cannot be designed one subsystem at a time. A flocking controller tuned with idealized communication will oscillate on a real radio. A collaborative mapping system that ignores bandwidth will saturate the channel that flocking and task allocation depend on. A task allocation algorithm proved optimal under connectivity may duplicate work wastefully when the network fragments, and an interface that shows operators only positions will mislead them about what the swarm can do. Each subsystem's behaviour depends on the network state, and the network state depends on what every subsystem does: where drones fly, what they send, when they send it. The practical implication is organizational. Teams building swarms often divide work into perception, planning, control and communications, with interfaces defined early and rarely revisited. That division mirrors the structure of single-robot systems, where it works well. For swarms it is a source of failure, because the interfaces hide the couplings that matter most. The research systems that have performed best in demanding conditions, from the optimized flocks of Budapest to the forest swarm of Zhejiang and the subterranean teams of the DARPA challenge, were built by groups that treated communication as a design variable in every subsystem rather than as a service provided by one of them. Organizations evaluating or procuring swarm systems should ask how the vendor or research team tested behaviour across the full range of network conditions, not only whether each component meets its specification. Degraded modes are the product The second consequence concerns what a swarm should be judged on. Demonstrations naturally show swarms at their best: connected, well-positioned, under good conditions. But the environments that justify swarms are precisely those where conditions are not good. A swarm's value, therefore, lies largely in how it behaves when links fail, positioning drifts and members are lost. This reverses the usual priority. The fully connected mode should be treated as the special case, a bonus when available, and the degraded modes as the main product. Designers should specify, before writing control code, what the swarm does when it is connected to the ground, when it is connected internally but not to the ground, when it is fragmented, and when individual drones are isolated, and how it moves between these modes. Those behaviours should be tested at least as thoroughly as nominal behaviour, in simulators whose communication models have been validated against measurements in representative environments. Rescue agencies evaluating swarms should ask to see the swarm fragment and recover, not just to see it fly in formation. The scaling argument has been misread The third consequence concerns size. The founding argument for swarm robotics was scalability: local rules whose cost does not grow with the number of members. That argument is correct for computation. It is incorrect, or at least incomplete, for communication. The Gupta-Kumar result shows that per-node throughput in a shared wireless network falls as the network grows, and every attempt to share global information, a full map, a complete task list, a picture of every member's state, runs into that ceiling. Swarms scale only to the extent that their information needs remain local. This suggests that the most useful civilian swarms in the near future will not be the largest. They will be groups of perhaps five to fifty drones, large enough to provide parallel coverage, simultaneous perspective and tolerance of loss, and small enough that their shared state fits within the capacity of a mesh. Very large swarms of hundreds or thousands are feasible for tasks where each drone needs almost nothing from the others, but those are few in civilian practice. Funders, journalists and developers would do well to stop treating the number of drones as a measure of progress. The relevant measures are coverage per unit time, detection reliability, behaviour under partition and burden on the human operator. Humans supervise networks, not aircraft The fourth consequence concerns people. The human factors research on swarm supervision shows that operators struggle to perceive and predict swarm behaviour. Part of the difficulty is inherent in emergent systems, but a substantial part comes from presenting operators with the wrong information. An operator who sees drone positions on a map, but not which drones can currently be reached, how stale each drone's last report is, or where communication is reliable, is supervising a fiction. The swarm's ability to respond to a command, to report a finding or to coordinate a response depends on its network, and an interface that hides the network hides the swarm's true state. For agencies, this argues for including communication state in operational displays and training, and for adapting command procedures accordingly. A commander who knows that the drones inside a building will report only when they return to relay range will interpret silence correctly. One who does not will either assume the worst or, more dangerously, assume that no report means nothing found. Training should teach the swarm's degraded modes explicitly, so that operators know what the drones are doing when they cannot be seen or heard. What remains unsettled Several important questions remain open, and honesty requires naming them. The regulatory path for routine civilian swarm operations is not yet established. The United States rule for operations beyond visual line of sight was still under final review in late 2026, and its treatment of multi-aircraft supervision will shape what is practical for years. European operators can obtain authorizations through risk assessment, but case-by-case approval is slow and costly. Until regulators have a standard way to evaluate swarm safety cases, including evidence about behaviour under network degradation, most civilian swarm operations will remain trials or exemptions. Verification of swarm behaviour at realistic scale and fidelity is an unsolved research problem. Formal methods do not scale to realistic swarm sizes; statistical simulation is only as good as its models; and field testing cannot cover the space of conditions. The layered approach, in which simple verified safety layers wrap complex behaviour, offers a workable path for safety, but it does not establish that a swarm will perform its mission well, only that it will not cause harm while trying. Learning-based methods are advancing quickly and may surpass designed controllers and allocation schemes in performance. Whether they can be made predictable and verifiable enough for safety-critical civilian use is unknown, and the answer may differ between the inner loops of flight control, where learned components can be constrained by barrier functions, and the outer loops of task allocation, where errors manifest as wasted effort rather than collisions. The relationship between civilian and military development is unsettled. Military investment has driven much of the field and continues to grow. International negotiations on autonomy in weapons have not produced binding agreement. Civilian practitioners will need to navigate export controls, research security rules and public suspicion, and they will benefit from being clear and open about what their systems are for. What to do differently For engineers, the implication is to put the network at the centre of the design from the first day: model it realistically, decide what each subsystem needs from it and what it does without it, specify degraded modes before nominal ones, keep information local wherever possible, and test fragmentation and recovery as thoroughly as formation flight. For rescue agencies and other civilian users, the implication is to evaluate swarms on their behaviour under degradation, to demand interfaces that show communication state, to train operators in degraded modes, and to begin with modest swarms on tasks where parallel coverage and simultaneous perspective clearly add value, such as rapid overview mapping, multi-point radio location and confirmation of detections. For regulators, the implication is to build evaluation frameworks that ask about network-dependent behaviour explicitly: what the swarm does when links fail, how it was verified, and what layered safety mechanisms bound its behaviour regardless of what its higher-level logic does. And for anyone thinking about swarms, the implication is to abandon the image of the light show, thousands of drones moving as one under perfect conditions, and replace it with a more accurate one: a handful of small aircraft working in the dark, each knowing only part of the picture, talking when they can, carrying information when they cannot, and still, because they were designed for exactly this, finding the person they were sent to find. Notes Introduction 1. Federal Aviation Administration, "Normalizing Unmanned Aircraft Systems Beyond Visual Line of Sight Operations," notice of proposed rulemaking, Federal Register, August 2025. Chapter 1 1. M. Dorigo, G. Theraulaz and V. Trianni, "Swarm Robotics: Past, Present, and Future," Proceedings of the IEEE 109, no. 7 (2021): 1152–1165. 2. M. Brambilla, E. Ferrante, M. Birattari and M. Dorigo, "Swarm Robotics: A Review from the Swarm Engineering Perspective," Swarm Intelligence 7, no. 1 (2013): 1–41. 3. M. Rubenstein, A. Cornejo and R. Nagpal, "Programmable Self-Assembly in a Thousand-Robot Swarm," Science 345, no. 6198 (2014): 795–799. 4. H. Hamann, Swarm Robotics: A Formal Approach (Springer, 2018). Chapter 2 1. M. Ballerini et al., "Interaction Ruling Animal Collective Behavior Depends on Topological Rather than Metric Distance: Evidence from a Field Study," PNAS 105, no. 4 (2008): 1232–1237. 2. A. Cavagna et al., "Scale-Free Correlations in Starling Flocks," PNAS 107, no. 26 (2010): 11865–11870. 3. T. Vicsek, A. Czirók, E. Ben-Jacob, I. Cohen and O. Shochet, "Novel Type of Phase Transition in a System of Self-Driven Particles," Physical Review Letters 75, no. 6 (1995): 1226–1229. 4. I. D. Couzin, J. Krause, R. James, G. D. Ruxton and N. R. Franks, "Collective Memory and Spatial Sorting in Animal Groups," Journal of Theoretical Biology 218, no. 1 (2002): 1–11. 5. Y. Katz, K. Tunstrøm, C. C. Ioannou, C. Huepe and I. D. Couzin, "Inferring the Structure and Dynamics of Interactions in Schooling Fish," PNAS 108, no. 46 (2011): 18720–18725. 6. I. D. Couzin, J. Krause, N. R. Franks and S. A. Levin, "Effective Leadership and Decision-Making in Animal Groups on the Move," Nature 433 (2005): 513–516. 7. M. Nagy, Z. Ákos, D. Biro and T. Vicsek, "Hierarchical Group Dynamics in Pigeon Flocks," Nature 464 (2010): 890–893. 8. T. D. Seeley, Honeybee Democracy (Princeton University Press, 2010). 9. T. D. Seeley et al., "Stop Signals Provide Cross Inhibition in Collective Decision-Making by Honeybee Swarms," Science 335, no. 6064 (2012): 108–111. 10. S. Goss, S. Aron, J.-L. Deneubourg and J. M. Pasteels, "Self-Organized Shortcuts in the Argentine Ant," Naturwissenschaften 76 (1989): 579–581. 11. M. Dorigo, V. Maniezzo and A. Colorni, "Ant System: Optimization by a Colony of Cooperating Agents," IEEE Transactions on Systems, Man, and Cybernetics, Part B 26, no. 1 (1996): 29–41. 12. D. M. Gordon, Ant Encounters: Interaction Networks and Colony Behavior (Princeton University Press, 2010). Chapter 3 1. C. W. Reynolds, "Flocks, Herds, and Schools: A Distributed Behavioral Model," Computer Graphics 21, no. 4 (1987): 25–34. 2. A. Jadbabaie, J. Lin and A. S. Morse, "Coordination of Groups of Mobile Autonomous Agents Using Nearest Neighbor Rules," IEEE Transactions on Automatic Control 48, no. 6 (2003): 988–1001. 3. M. Fiedler, "Algebraic Connectivity of Graphs," Czechoslovak Mathematical Journal 23 (1973): 298–305. 4. R. Olfati-Saber, "Flocking for Multi-Agent Dynamic Systems: Algorithms and Theory," IEEE Transactions on Automatic Control 51, no. 3 (2006): 401–420. 5. R. Olfati-Saber and R. M. Murray, "Consensus Problems in Networks of Agents with Switching Topology and Time-Delays," IEEE Transactions on Automatic Control 49, no. 9 (2004): 1520–1533. 6. S. Hauert et al., "Reynolds Flocking in Reality with Fixed-Wing Robots: Communication Range vs. Maximum Turning Rate," IEEE/RSJ International Conference on Intelligent Robots and Systems (2011). 7. G. Vásárhelyi et al., "Outdoor Flocking and Formation Flight with Autonomous Aerial Robots," IEEE/RSJ International Conference on Intelligent Robots and Systems (2014). 8. G. Vásárhelyi et al., "Optimized Flocking of Autonomous Drones in Confined Environments," Science Robotics 3, no. 20 (2018). 9. X. Zhou et al., "Swarm of Micro Flying Robots in the Wild," Science Robotics 7, no. 66 (2022). 10. X. Zhou, J. Zhu, H. Zhou, C. Xu and F. Gao, "EGO-Swarm: A Fully Autonomous and Decentralized Quadrotor Swarm System in Cluttered Environments," IEEE International Conference on Robotics and Automation (2021). 11. S. Li, F. Shan, J. Liu, M. Coppola, C. De Wagter and G. C. H. E. de Croon, "Onboard Ranging-Based Relative Localization and Stability for Lightweight Aerial Swarms," arXiv:2003.05853 (2020). 12. E. Tolstaya et al., "Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks," Conference on Robot Learning (2019). 13. S. Batra et al., "Decentralized Control of Quadrotor Swarms with End-to-End Deep Reinforcement Learning," Conference on Robot Learning (2021). 14. Y. Koren and J. Borenstein, "Potential Field Methods and Their Inherent Limitations for Mobile Robot Navigation," IEEE International Conference on Robotics and Automation (1991). 15. J. van den Berg, S. J. Guy, M. Lin and D. Manocha, "Reciprocal n-Body Collision Avoidance," Robotics Research, Springer Tracts in Advanced Robotics 70 (2011): 3–19. 16. L. Wang, A. D. Ames and M. Egerstedt, "Safety Barrier Certificates for Collisions-Free Multirobot Systems," IEEE Transactions on Robotics 33, no. 3 (2017): 661–674. 17. D. Zhou, Z. Wang, S. Bandyopadhyay and M. Schwager, "Fast, On-Line Collision Avoidance for Dynamic Vehicles Using Buffered Voronoi Cells," IEEE Robotics and Automation Letters 2, no. 2 (2017): 1047–1054. 18. J. A. Preiss, W. Hönig, G. S. Sukhatme and N. Ayanian, "Crazyswarm: A Large Nano-Quadcopter Swarm," IEEE International Conference on Robotics and Automation (2017). Chapter 4 1. T. Qin, P. Li and S. Shen, "VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator," IEEE Transactions on Robotics 34, no. 4 (2018): 1004–1020. 2. C. Campos et al., "ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial, and Multimap SLAM," IEEE Transactions on Robotics 37, no. 6 (2021): 1874–1890. 3. W. Xu, Y. Cai, D. He, J. Lin and F. Zhang, "FAST-LIO2: Fast Direct LiDAR-Inertial Odometry," IEEE Transactions on Robotics 38, no. 4 (2022): 2053–2073. 4. V. Walter, M. Saska and A. Franchi, "Fast Mutual Relative Localization of UAVs Using Ultraviolet LED Markers," International Conference on Unmanned Aircraft Systems (2018). 5. P.-Y. Lajoie, B. Ramtoula, Y. Chang, L. Carlone and G. Beltrame, "DOOR-SLAM: Distributed, Online, and Outlier Resilient SLAM for Robotic Teams," IEEE Robotics and Automation Letters 5, no. 2 (2020): 1656–1663. 6. P.-Y. Lajoie and G. Beltrame, "Swarm-SLAM: Sparse Decentralized Collaborative Simultaneous Localization and Mapping Framework for Multi-Robot Systems," IEEE Robotics and Automation Letters 9, no. 1 (2024): 475–482. 7. Y. Tian et al., "Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems," IEEE Transactions on Robotics 38, no. 4 (2022): 2022–2038. 8. J. G. Mangelson, D. Dominic, R. M. Eustice and R. Vasudevan, "Pairwise Consistent Measurement Set Maximization for Robust Multi-Robot Map Merging," IEEE International Conference on Robotics and Automation (2018). 9. S. J. Julier and J. K. Uhlmann, "A Non-Divergent Estimation Algorithm in the Presence of Unknown Correlations," Proceedings of the American Control Conference (1997). 10. K. Ebadi et al., "Present and Future of SLAM in Extreme Environments: The DARPA SubT Challenge," IEEE Transactions on Robotics 40 (2024): 936–959 (published online 2023). 11. M. Tranzatto et al., "CERBERUS in the DARPA Subterranean Challenge," Science Robotics 7, no. 66 (2022). Chapter 5 1. İ. Bekmezci, O. K. Sahingoz and Ş. Temel, "Flying Ad-Hoc Networks (FANETs): A Survey," Ad Hoc Networks 11, no. 3 (2013): 1254–1270. 2. S. Hayat, E. Yanmaz and R. Muzaffar, "Survey on Unmanned Aerial Vehicle Networks for Civil Applications: A Communications Viewpoint," IEEE Communications Surveys and Tutorials 18, no. 4 (2016): 2624–2661. 3. T. Clausen and P. Jacquet, eds., "Optimized Link State Routing Protocol (OLSR)," RFC 3626 (IETF, 2003); T. Clausen et al., "The Optimized Link State Routing Protocol Version 2," RFC 7181 (IETF, 2014). 4. J. Chroboczek and D. Schinazi, "The Babel Routing Protocol," RFC 8966 (IETF, 2021). 5. C. Perkins, E. Belding-Royer and S. Das, "Ad hoc On-Demand Distance Vector (AODV) Routing," RFC 3561 (IETF, 2003). 6. B. Karp and H. T. Kung, "GPSR: Greedy Perimeter Stateless Routing for Wireless Networks," Proceedings of ACM MobiCom (2000). 7. G. Di Caro, F. Ducatelle and L. M. Gambardella, "AntHocNet: An Adaptive Nature-Inspired Algorithm for Routing in Mobile Ad Hoc Networks," European Transactions on Telecommunications 16, no. 5 (2005): 443–455. 8. P. Gupta and P. R. Kumar, "The Capacity of Wireless Networks," IEEE Transactions on Information Theory 46, no. 2 (2000): 388–404. 9. M. Grossglauser and D. Tse, "Mobility Increases the Capacity of Ad Hoc Wireless Networks," IEEE/ACM Transactions on Networking 10, no. 4 (2002): 477–486. 10. S.-Y. Ni, Y.-C. Tseng, Y.-S. Chen and J.-P. Sheu, "The Broadcast Storm Problem in a Mobile Ad Hoc Network," Proceedings of ACM MobiCom (1999). 11. Fraunhofer Institute for Integrated Circuits IIS, "Innovation from Above: How Mesh Networks Help Control Drone Swarms," press release, June 2026. Chapter 6 1. M. M. Zavlanos and G. J. Pappas, "Distributed Connectivity Control of Mobile Networks," IEEE Transactions on Robotics 24, no. 6 (2008): 1416–1428. 2. L. Sabattini, C. Secchi, N. Chopra and C. Fantuzzi, "Distributed Control of Multirobot Systems with Global Connectivity Maintenance," IEEE Transactions on Robotics 29, no. 5 (2013): 1326–1332. 3. P. Yang, R. A. Freeman, G. J. Gordon, K. M. Lynch, S. S. Srinivasa and R. Sukthankar, "Decentralized Estimation and Control of Graph Connectivity for Mobile Sensor Networks," Automatica 46, no. 2 (2010): 390–396. 4. F. Amigoni, J. Banfi and N. Basilico, "Multirobot Exploration of Communication-Restricted Environments: A Survey," IEEE Intelligent Systems 32, no. 6 (2017): 48–57. 5. G. A. Hollinger and S. Singh, "Multirobot Coordination with Periodic Connectivity: Theory and Experiments," IEEE Transactions on Robotics 28, no. 4 (2012): 967–973. 6. J. de Hoog, S. Cameron and A. Visser, "Role-Based Autonomous Multi-Robot Exploration," International Conference on Advanced Cognitive Technologies and Applications (2009). 7. V. Cerf et al., "Delay-Tolerant Networking Architecture," RFC 4838 (IETF, 2007); S. Burleigh, K. Fall and E. Birrane, "Bundle Protocol Version 7," RFC 9171 (IETF, 2022). 8. R. C. Shah, S. Roy, S. Jain and W. Brunette, "Data MULEs: Modeling a Three-Tier Architecture for Sparse Sensor Networks," Ad Hoc Networks 1, nos. 2–3 (2003): 215–233. 9. W. Zhao, M. Ammar and E. Zegura, "A Message Ferrying Approach for Data Delivery in Sparse Mobile Ad Hoc Networks," Proceedings of ACM MobiHoc (2004). 10. A. Vahdat and D. Becker, "Epidemic Routing for Partially Connected Ad Hoc Networks," Technical Report CS-200006, Duke University (2000). 11. M. Shapiro, N. Preguiça, C. Baquero and M. Zawirski, "Conflict-Free Replicated Data Types," Symposium on Self-Stabilizing Systems (2011). 12. C. Pinciroli and G. Beltrame, "Buzz: An Extensible Programming Language for Heterogeneous Swarm Robotics," IEEE/RSJ International Conference on Intelligent Robots and Systems (2016). 13. D. V. Dimarogonas, E. Frazzoli and K. H. Johansson, "Distributed Event-Triggered Control for Multi-Agent Systems," IEEE Transactions on Automatic Control 57, no. 5 (2012): 1291–1297. Chapter 7 1. B. P. Gerkey and M. J. Matarić, "A Formal Analysis and Taxonomy of Task Allocation in Multi-Robot Systems," International Journal of Robotics Research 23, no. 9 (2004): 939–954. 2. H. W. Kuhn, "The Hungarian Method for the Assignment Problem," Naval Research Logistics Quarterly 2 (1955): 83–97. 3. D. P. Bertsekas, "The Auction Algorithm: A Distributed Relaxation Method for the Assignment Problem," Annals of Operations Research 14 (1988): 105–123. 4. G. A. Korsah, A. Stentz and M. B. Dias, "A Comprehensive Taxonomy for Multi-Robot Task Allocation," International Journal of Robotics Research 32, no. 12 (2013): 1495–1512. 5. E. Bonabeau, G. Theraulaz and J.-L. Deneubourg, "Quantitative Study of the Fixed Threshold Model for the Regulation of Division of Labour in Insect Societies," Proceedings of the Royal Society B 263 (1996): 1565–1569. 6. M. J. B. Krieger, J.-B. Billeter and L. Keller, "Ant-Like Task Allocation and Recruitment in Cooperative Robots," Nature 406 (2000): 992–995. 7. J. Cortés, S. Martínez, T. Karataş and F. Bullo, "Coverage Control for Mobile Sensing Networks," IEEE Transactions on Robotics and Automation 20, no. 2 (2004): 243–255. 8. R. G. Smith, "The Contract Net Protocol: High-Level Communication and Control in a Distributed Problem Solver," IEEE Transactions on Computers C-29, no. 12 (1980): 1104–1113. 9. M. B. Dias, R. Zlot, N. Kalra and A. Stentz, "Market-Based Multirobot Coordination: A Survey and Analysis," Proceedings of the IEEE 94, no. 7 (2006): 1257–1270. 10. M. G. Lagoudakis et al., "Auction-Based Multi-Robot Routing," Robotics: Science and Systems (2005). 11. H.-L. Choi, L. Brunet and J. P. How, "Consensus-Based Decentralized Auctions for Robust Task Allocation," IEEE Transactions on Robotics 25, no. 4 (2009): 912–926. 12. L. Johnson, S. Ponda, H.-L. Choi and J. P. How, "Asynchronous Decentralized Task Allocation for Dynamic Environments," AIAA Infotech@Aerospace (2011). 13. G. Best, O. M. Cliff, T. Patten, R. R. Mettu and R. Fitch, "Dec-MCTS: Decentralized Planning for Multi-Robot Active Perception," International Journal of Robotics Research 38, nos. 2–3 (2019): 316–337. Chapter 8 1. B. O. Koopman, Search and Screening: General Principles with Historical Applications (Pergamon, 1980; based on wartime work first issued in 1946). 2. L. D. Stone, Theory of Optimal Search (Academic Press, 1975). 3. L. D. Stone, C. M. Keller, T. M. Kratzke and J. P. Strumpfer, "Search for the Wreckage of Air France Flight AF 447," Statistical Science 29, no. 1 (2014): 69–80. 4. R. J. Koester, Lost Person Behavior: A Search and Rescue Guide on Where to Look for Land, Air and Water (dbS Productions, 2008). 5. M. A. Goodrich et al., "Supporting Wilderness Search and Rescue Using a Camera-Equipped Mini UAV," Journal of Field Robotics 25, nos. 1–2 (2008): 89–110. 6. SAPIENCE (Sense and Avoid, Cooperative Drone Competition), NATO Science for Peace and Security Programme, project information and City St George's, University of London, announcement of final results, August 2026. 7. A. Kolling, P. Walker, N. Chakraborty, K. Sycara and M. Lewis, "Human Interaction with Robot Swarms: A Survey," IEEE Transactions on Human-Machine Systems 46, no. 1 (2016): 9–26. 8. National Agricultural Aviation Association, "Precedent-Setting Part 137 UAS Exemption Granted to Hylio," NAAA eNewsletter, March 2024. 9. D. Albani, D. Nardi and V. Trianni, "Field Coverage and Weed Mapping by UAV Swarms," IEEE/RSJ International Conference on Intelligent Robots and Systems (2017). Chapter 9 1. A. F. T. Winfield, J. Sa, M.-C. Fernández-Gago, C. Dixon and M. Fisher, "On Formal Specification of Emergent Behaviours in Swarm Robotic Systems," International Journal of Advanced Robotic Systems 2, no. 4 (2005): 363–370. 2. C. Dixon, A. F. T. Winfield, M. Fisher and C. Zeng, "Towards Temporal Verification of Swarm Robotic Systems," Robotics and Autonomous Systems 60, no. 11 (2012): 1429–1441. 3. C. Pinciroli et al., "ARGoS: A Modular, Parallel, Multi-Engine Simulator for Multi-Robot Systems," Swarm Intelligence 6, no. 4 (2012): 271–295. 4. L. Sha, "Using Simplicity to Control Complexity," IEEE Software 18, no. 4 (2001): 20–28. 5. A. J. Kerns, D. P. Shepard, J. A. Bhatti and T. E. Humphreys, "Unmanned Aircraft Capture and Control via GPS Spoofing," Journal of Field Robotics 31, no. 4 (2014): 617–636. 6. L. Lamport, R. Shostak and M. Pease, "The Byzantine Generals Problem," ACM Transactions on Programming Languages and Systems 4, no. 3 (1982): 382–401. 7. V. Strobel, E. Castelló Ferrer and M. Dorigo, "Managing Byzantine Robots via Blockchain Technology in a Swarm Robotics Collective Decision Making Scenario," Proceedings of AAMAS (2018). 8. J. R. Douceur, "The Sybil Attack," International Workshop on Peer-to-Peer Systems (2002). 9. Federal Aviation Administration, 14 CFR Part 107, Small Unmanned Aircraft Systems; 14 CFR Part 89, Remote Identification of Unmanned Aircraft. 10. Reporting on the status of the proposed Part 108 rule, including its submission to the Office of Information and Regulatory Affairs in July 2026, from industry press including PickDrones, "FAA Part 108 Delay," September 2026. 11. Commission Implementing Regulation (EU) 2019/947; Commission Implementing Regulation (EU) 2021/664 on a regulatory framework for the U-space. 12. Joint Authorities for Rulemaking on Unmanned Systems, Specific Operations Risk Assessment (SORA), version 2.5 (2024). 13. United Nations General Assembly, Resolution 78/241 (2023) and Resolution 79/62 (2024), "Lethal Autonomous Weapons Systems." 14. United States Department of Defense, Directive 3000.09, "Autonomy in Weapon Systems," January 25, 2023. Glossary Algebraic connectivity. A number computed from a network's structure that measures how well connected it is; it is zero when the network has split into separate pieces, and larger values mean information spreads and agreement forms faster. Collaborative SLAM. Simultaneous localization and mapping performed by several robots together, in which each robot's map and trajectory are tied to the others' through shared observations of the same places. Consensus. A process in which agents repeatedly adjust a shared quantity, such as heading or a task assignment, toward their neighbours' values until all agree. Delay-tolerant networking. A family of protocols in which data is stored by nodes and forwarded when a link becomes available, rather than requiring an end-to-end path at the moment of sending. FANET. Flying ad hoc network: a wireless network formed among airborne drones without fixed infrastructure, in which drones relay traffic for one another. Loop closure. The recognition that a robot, or another robot, has returned to a previously observed place, which provides a constraint used to correct accumulated drift in position estimates. Mesh network. A network in which nodes forward traffic for one another and multiple paths may exist between any two nodes, so that the failure of one link or node need not disconnect the network. Response threshold. In task allocation, an individual's level of sensitivity to a task-related stimulus; when the stimulus exceeds the threshold, the individual is likely to take up the task. Stigmergy. Coordination through modifications to a shared environment, such as pheromone trails or a shared map, rather than through direct messages between individuals. Topological interaction. Interaction with a fixed number of nearest neighbours regardless of their distance, as opposed to metric interaction with all neighbours within a fixed radius. Ultra-wideband ranging. Measuring the distance between two radios by timing very short pulses spread across a wide frequency band, typically accurate to around ten centimetres. Visual-inertial odometry. Estimating a vehicle's motion by fusing camera images with measurements from accelerometers and gyroscopes. Further Reading Heiko Hamann, Swarm Robotics: A Formal Approach (Springer, 2018).** A clear textbook organized around the micro-macro problem, with accessible treatments of modelling, collective decision-making and swarm design. Eric Bonabeau, Marco Dorigo and Guy Theraulaz, Swarm Intelligence: From Natural to Artificial Systems (Oxford University Press, 1999).** The foundational book linking social insect behaviour to algorithms for optimization and robotics; still the best introduction to stigmergy and response thresholds. Thomas D. Seeley, Honeybee Democracy (Princeton University Press, 2010).** A beautifully written account of how honeybee swarms choose a new home, and an indispensable source for anyone designing collective decision-making. Francesco Bullo, Jorge Cortés and Sonia Martínez, Distributed Control of Robotic Networks (Princeton University Press, 2009).** A rigorous, freely available treatment of consensus, coverage and rendezvous for readers comfortable with mathematics. Mehran Mesbahi and Magnus Egerstedt, Graph Theoretic Methods in Multiagent Networks (Princeton University Press, 2010).** The standard reference on how network structure governs multi-agent coordination, including algebraic connectivity and formation control. Marco Dorigo, Guy Theraulaz and Vito Trianni, "Swarm Robotics: Past, Present, and Future," Proceedings of the IEEE (2021).** A concise survey of the field's history and open problems by three of its leading figures. Soon-Jo Chung, Aditya Paranjape, Philip Dames, Shaojie Shen and Vijay Kumar, "A Survey on Aerial Swarm Robotics," IEEE Transactions on Robotics 34, no. 4 (2018).** A comprehensive technical review focused specifically on aerial swarms, covering modelling, control, estimation and planning. Samira Hayat, Evşen Yanmaz and Raheeda Muzaffar, "Survey on Unmanned Aerial Vehicle Networks for Civil Applications: A Communications Viewpoint," IEEE Communications Surveys and Tutorials (2016).** The best single source on the networking requirements of civilian drone applications. Kamak Ebadi et al., "Present and Future of SLAM in Extreme Environments: The DARPA SubT Challenge," IEEE Transactions on Robotics (2024).** A multi-team account of what worked and what did not in mapping and localization under the most demanding conditions yet tested. Robert J. Koester, Lost Person Behavior (dbS Productions, 2008).** The practitioner's reference on where missing people are found; essential context for anyone designing search algorithms. Andreas Kolling et al., "Human Interaction with Robot Swarms: A Survey," IEEE Transactions on Human-Machine Systems (2016).** A thorough review of how people can supervise swarms and where they struggle. Hashtags: #AutonomousDroneSwarms #DroneSwarmTechnology #SwarmRobotics #CollaborativeRobotics #MeshNetworking #FlyingAdHocNetworks #FANET #DecentralizedControl #MultiAgentSystems #CollectiveIntelligence #SwarmIntelligence #AutonomousDrones #UnmannedAerialVehicles #UAVCoordination #DistributedConsensus #FlockingAlgorithms #CollisionAvoidance #MultiRobotSLAM #CollaborativeMapping #VisualInertialOdometry #SensorFusion #GNSSDeniedNavigation #UltraWidebandLocalization #DistributedTaskAllocation #MultiAgentPathPlanning #NetworkResilience #FaultTolerantRobotics #SearchAndRescueRobotics #WirelessCommunication #AutonomousNavigation

Download the Book (PDF): Introduction It is a little after one in the morning in a residence hall. A second-year student has a chemistry exam at nine, a lab report due at noon, and a text thread with a friend that went wrong earlier in the evening. The lights are off. The student is not studying and is not sleeping. Instead, the same few thoughts circle: I am going to fail this. I should have started earlier. Everyone else understands this material. What if I lose the scholarship. Each lap of the circuit takes a few seconds and leaves the heart rate a little higher. By two o'clock, nothing has been learned, nothing has been solved, and the exam is seven hours away. Almost every student knows some version of that night. Worry about grades, deadlines, money, the future and other people is not a sign that something is wrong with the student who feels it. It is the ordinary cost of a demanding stage of life, one in which the stakes feel permanent and the calendar never quite empties. But some of the ways students manage that worry help, and some make it worse. This booklet is about one tool in the first category that costs almost nothing: writing things down. The scale of the problem Student stress is not a niche concern. The Healthy Minds Study, an annual survey run by researchers at the University of Michigan, UCLA and Boston University, collected responses from more than 84,000 students at 135 colleges and universities in the 2024-2025 academic year. Thirty-seven percent screened positive for moderate to severe depression and 32 percent for moderate to severe anxiety. Eleven percent said they had seriously considered suicide in the past year. Those numbers were actually an improvement: all three had fallen for the third year in a row from their 2022 peak, when 44 percent screened positive for depression and 37 percent for anxiety.[1] An improvement from a very high level is still a high level. On a typical campus, a third or more of the people in any lecture hall are carrying something heavy. Counseling centers have expanded, and more students now use them than a decade ago; the same survey found that 37 percent had received therapy or counseling in the past year. Yet professional care is not always available at the moment it is needed. Waitlists are common in busy weeks. Some students are not sure their problem is "bad enough" to justify an appointment. Others cannot afford off-campus care, or come from families and cultures in which talking to a stranger about feelings is unfamiliar. And much of academic stress arrives at hours when no office is open: the night before a deadline, the walk home after a bad grade, the Sunday afternoon when the week ahead looks impossible. Journaling fits into those gaps. It requires a pen and paper or a phone, ten or fifteen minutes, and no appointment. It can be done in a library carrel, on a bus, or in bed. It leaves no bill. That accessibility is exactly why it deserves careful thought rather than casual enthusiasm. A tool that everyone can use should be one that everyone can use well. What this booklet claims, and what it does not The popular story about journaling is simple and warm: write about your feelings and you will feel better. The research story is more interesting and more useful. Several decades of experiments show that some kinds of writing, done in some ways, produce real but modest benefits for many people. Other kinds of writing, done in other ways, do little. And for a minority of people in certain states of mind, writing can deepen the very spirals it was meant to break. The central argument of this booklet is that "journaling" is not one tool but a small kit of distinct tools, each suited to a different kind of stress. Dumping a cluttered head onto a page before bed is a different act from writing about a painful memory. Listing three good things from the day is a different act from testing whether a catastrophic thought is true. Writing about your fears ten minutes before an exam is different again. Each method has its own mechanism, its own evidence base, and its own failure mode. Students get the most from writing when they choose the method that fits the problem in front of them, and when they notice when writing has stopped helping and started circling. The evidence will be presented honestly throughout. When a study is small, that will be said. When a meta-analysis (a study that pools the results of many studies) finds an average effect that is statistically real but small, that will be said too. There is no value in overselling a free tool; overselling only sets up the reader for disappointment and the feeling that they are doing it wrong. A modest, reliable benefit that you can reach for any night of the week is worth a great deal, especially when it sits alongside sleep, movement, friendship and, when needed, professional help. How the booklet is organized Chapter 1 explains what writing actually does to a worried mind: why a thought on paper behaves differently from a thought in your head, and why that matters for students whose main tool, working memory, is the first casualty of anxiety. It also offers a map matching common kinds of academic stress to the methods covered in later chapters. Chapter 2 examines the method that started the science: James Pennebaker's expressive writing, in which people write about their deepest thoughts and feelings for fifteen or twenty minutes over several days. It sets out what the early studies found, what the larger meta-analyses have found since, and how to do the exercise in a way that follows the evidence. Chapter 3 turns to one of the most practically useful findings for students: that ten minutes of writing about test worries immediately before an exam improved the scores of anxious students in both laboratory and classroom experiments. Chapter 4 covers the brain dump and its cousins, including the bedtime to-do list, which in one sleep-laboratory study helped people fall asleep faster than writing about what they had already finished. Chapter 5 looks at gratitude logs and other forms of positive writing, including what the evidence says about how often to do them and why forcing gratitude can backfire. Chapter 6 borrows two structured techniques from cognitive behavioral therapy: the thought record, which puts anxious predictions under examination, and scheduled worry time, which gives worry a place and a boundary. Chapter 7 addresses the particular sting of academic failure and describes self-compassion writing, a method with evidence that it does not make students complacent but instead helps them try again. Chapter 8 is the chapter many books on journaling leave out: when writing backfires, how to tell, and what to do instead. It includes clear guidance on when to seek counseling and when to seek urgent help. Chapter 9 deals with the practical questions that decide whether any of this lasts: how to build a habit that survives midterms, and how to keep a private record private when you share a room, a laptop or a family phone plan. The conclusion draws these threads into a single argument about what a student gains from learning to write their way through stress, beyond the relief of any single night. A note on safety and scope This booklet is written for college students and for the parents, advisers, residence staff and faculty who support them. It is meant to be practical and grounded in evidence, but it is not a substitute for individual assessment or treatment by a qualified professional. Journaling is a self-help tool. It can work alongside counseling and medical care; it cannot replace them when they are needed. If you are in the United States and you are thinking about suicide or are in crisis, you can call or text 988 to reach the 988 Suicide and Crisis Lifeline at any hour. If you are in immediate danger, call 911 or go to the nearest emergency department. Outside the United States, local emergency numbers and national crisis lines serve the same role. Chapter 8 returns to these points in more detail. The student awake at one in the morning does not need a lecture about self-care. They need something they can do in the next fifteen minutes that will make the next seven hours go better. The chapters that follow are an attempt to give them several such things, along with enough understanding of how each one works to choose well. Chapter 1: What Writing Does to a Worried Mind Consider a junior studying for an organic chemistry exam. She opens the textbook to the chapter on reaction mechanisms and reads the same paragraph three times. Nothing sticks. It is not that the material is beyond her; she understood the previous two chapters well. The problem is that part of her mind is busy elsewhere, rehearsing what will happen if she does badly, wondering whether she should drop the course, remembering the look on her lab partner's face when she got an answer wrong last week. She is trying to study with half a brain, and the half that is missing is precisely the half she needs. This is the ordinary experience of academic anxiety, and it points to the most important reason writing can help. Worry is not only unpleasant. It occupies mental space. Understanding what that space is, and what happens when you move worry out of it and onto a page, makes sense of every method in this booklet. Worry uses the same workspace as studying Psychologists use the term working memory for the small mental workspace in which we hold and manipulate information in the moment: the numbers you carry while doing arithmetic in your head, the first half of a sentence you are reading while you get to the second half, the steps of a proof you are trying to follow. Working memory is limited. Most people can hold only a handful of items in it at once, and it is the resource that almost every academic task depends on. Reading comprehension, problem solving, essay planning and exam performance all lean on it heavily. Worry lives in the same workspace. When you are anxious, your mind generates a steady stream of threat-related thoughts, and each one takes up room. Attentional control theory, developed by the psychologist Michael Eysenck and colleagues, proposes that anxiety impairs performance chiefly by disrupting the mind's ability to direct attention where it is wanted and to hold off distraction.[1] Anxious students often work harder to compensate, which is why anxiety sometimes affects the efficiency of work (how much effort it takes) before it affects the quality of the result. But under enough pressure, or when the task demands a lot of working memory, performance itself suffers. This explains a pattern that frustrates many students: they know the material at home and lose it in the exam room. It is not that the knowledge vanished. It is that the workspace needed to retrieve and use the knowledge is crowded with thoughts about failing. Chapter 3 describes experiments by Gerardo Ramirez and Sian Beilock that tested this idea directly, and found that clearing some of that worry by writing about it before a test improved the scores of the most anxious students. There is also evidence that the effect accumulates over longer periods. In a study published in 2001, Kitty Klein and Adriel Boals asked first-year college students to write either about their thoughts and feelings about coming to college or about neutral topics. Students who wrote expressively showed larger gains in working memory capacity over the following weeks, and the gains were associated with a drop in intrusive and avoidant thinking.[2] The study was modest in size and should not be treated as proof that journaling makes you smarter. But its logic is consistent with the rest of this chapter: when unresolved worries stop demanding attention, some of that attention comes back. A thought on paper behaves differently When a worry exists only in your head, it has certain properties that make it hard to manage. It is fast: a whole catastrophe can unfold in a second. It is vague: "everything is falling apart" does not specify what, exactly, is falling apart. It is repetitive: the same thought can return fifty times in an evening without developing. And it is hard to inspect, because you are looking at it from the inside. Writing changes each of these properties. It slows thought down to the speed of a hand or a keyboard, which is much slower than the speed of rumination. It forces a degree of specificity, because it is difficult to write a vague sentence without noticing that it is vague. It makes repetition visible: if you write the same fear three times on one page, you see it three times. And it puts the thought outside you, where you can look at it as you would look at a friend's problem. The research on affect labeling supports part of this picture. In a brain-imaging study led by Matthew Lieberman at UCLA, participants who put a word to the emotion shown in a photograph of a face showed less activity in the amygdala, a brain region involved in threat responses, than participants who matched the face to another face or to a gender label.[3] Naming a feeling, in other words, seems to take some of the heat out of it. Labeling alone is a small intervention, and the laboratory setting is far from a student's desk at midnight. But it suggests a plausible reason why simply writing "I am scared about this exam" can feel different from being scared about the exam. A second mechanism concerns unfinished business. The mind tends to keep incomplete goals active, returning to them unprompted. That is useful when the reminder arrives in time for you to act, and exhausting when it arrives at two in the morning. In a set of experiments published in 2011, E. J. Masicampo and Roy Baumeister found that unfulfilled goals produced intrusive thoughts that interfered with other tasks, but that making a specific plan for the goal, without completing it, largely eliminated the interference.[4] The mind seems to accept a credible plan as a kind of down payment. Written plans are credible in a way that vague intentions are not. Chapter 4 builds on this finding to explain why a bedtime to-do list can help people fall asleep. A third mechanism concerns meaning. James Pennebaker, whose work Chapter 2 describes in detail, developed a computer program that counts categories of words in a text. When he and his colleagues analyzed the writing of people who had taken part in expressive writing studies, they found that those whose health improved most tended to increase their use of words associated with cause and insight over the course of their writing: words such as because, realize, understand and reason.[5] In plain terms, the people who benefited were the ones whose writing moved from raw reaction toward an account of what had happened and why. Writing seemed to help most when it helped people build a story. The same worry, inside and outside It helps to see the difference concretely. Return to the junior with the organic chemistry exam. Inside her head, the worry runs something like this, in less than a second and over and over: I'm going to fail, I don't get mechanisms, I'm not cut out for this, everyone else gets it, what am I doing in this major. Each phrase triggers the next. None of them is examined. The sequence ends where it began, with a sense of doom, and then starts again. Now suppose she takes ten minutes and writes it out by hand. The first lines might be nearly identical to the loop: I am going to fail this exam. I don't understand mechanisms. But writing a sentence invites the next sentence, and the next sentence is usually more specific. Actually I understand substitution reactions fine. It's elimination that I keep mixing up, especially when to expect E1 versus E2. A little later: I got a B on the last exam and I was just as scared then. And later still: The thing I'm really afraid of is telling my parents if I have to drop it. That's not about chemistry. Three things have happened in those ten minutes. The global threat ("I don't understand mechanisms") has shrunk to a specific, fixable gap ("E1 versus E2"). A piece of counter-evidence ("I got a B last time") has surfaced that the loop never allowed to appear. And a hidden layer of the worry, about family expectations, has come into view, where it can be thought about separately rather than contaminating every attempt to study. None of this required special technique. It came from the ordinary demands of writing: one word after another, in sentences that have to make sense. This example is illustrative, and real entries are often messier. Sometimes the writing stays stuck at the level of the loop for a whole page, and that is a signal worth noticing (Chapter 8 discusses what to do when it happens). But the general pattern, in which the act of writing pulls vague dread toward specifics, is common enough that most people who try it recognize it. Why students are a natural fit College students are, in a sense, the population on which much of this research was built. Many of the early expressive writing experiments recruited undergraduates, partly for convenience and partly because the transition to college is itself a well-defined stressor. In one early study, Pennebaker and colleagues asked first-year students to write about their deepest thoughts and feelings about coming to college; over the following months, those students made fewer visits to the campus health center for illness than students who wrote about superficial topics.[6] That does not mean the findings apply only to students, and later chapters discuss why results from one group do not always carry over to another. But it does mean that the typical reader of this booklet looks a good deal like the typical participant in the studies it describes. Students also face a particular mix of stressors that suits writing. Much academic worry is anticipatory, about tests and deadlines and futures that have not happened yet, and anticipatory worry is exactly the kind that thrives in an unorganized head. Much of it is evaluative, about being judged and found wanting, and evaluative worry tends to generate harsh internal commentary that looks different once it is written down in plain view. And much of it arrives in a crowded life, with many small demands competing for attention, which is the condition under which externalizing tasks onto paper most clearly frees up the mind. Organizing, not just venting These mechanisms have a shared implication that runs through the rest of this booklet. Writing helps a worried mind chiefly by organizing experience: sorting it, specifying it, planning around it, making sense of it. It helps much less, and can even hurt, when it simply rehearses distress. This matters because the popular image of journaling is often one of venting, of pouring feelings onto the page until they are gone. Emotional expression is a real part of what makes writing useful, and some of the methods in this booklet deliberately invite strong feeling. But feeling alone does not seem to be the active ingredient. A study by Philip Ullrich and Susan Lutgendorf illustrates the point. Students were asked to journal about a stressful or traumatic event for a month, either focusing only on their emotions or focusing on both their thoughts and their emotions, with the aim of understanding the event. The group that combined cognition and emotion developed a greater sense of positive growth from the experience. The group that focused only on emotion reported more illness symptoms over the study period.[7] Venting without understanding did not help, and may have made people feel worse. The practical lesson is simple. When you write about something difficult, let yourself feel it, but also ask questions of it. What exactly happened? What am I afraid of? What does this remind me of? What would I say to someone else in this situation? What, if anything, can I do? Those questions are what turn a page of distress into a page of processing. Matching the method to the problem If writing works by organizing experience, then different problems call for different kinds of organization. A student whose head is full of dozens of half-remembered tasks needs something different from a student who cannot stop replaying a painful conversation, and both need something different from a student who is about to walk into an exam hall. One of the most common reasons people give up on journaling is that they chose a method that did not fit their problem. They tried to write gratitude lists when what they needed was a plan, or wrote long emotional entries when what they needed was to get to sleep. Table 1 offers a rough map from common forms of student stress to the methods covered in this booklet. It is not a rulebook. Many students will use two or three methods at different times, and many problems have more than one layer. But it is a place to start. Table 1. Matching common student stresses to journaling methods. When the problem is... Try this method Typical dose Chapter A painful event you keep returning to Expressive writing 15-20 min, 3-4 days 2 Dread just before an exam Pre-exam worry writing 10 min, right before 3 A cluttered head, racing thoughts at bedtime Brain dump or to-do list 5-10 min 4 A flat, joyless stretch of the semester Gratitude or good-things log Weekly or a few times a week 5 A catastrophic prediction that feels certain Thought record 10-15 min per thought 6 Worry leaking into the whole day Scheduled worry time 15-30 min, fixed time 6 Harsh self-criticism after a setback Self-compassion letter 10-15 min 7 The doses in the table are approximate and come from the studies discussed in later chapters. They are starting points, not prescriptions. The more important column is the first one. Before you write, it is worth asking a single question: what kind of trouble am I in right now? The answer usually points to the right tool. What writing cannot do It is worth being clear from the start about the limits. Writing does not change the facts of a situation. A student who is failing a course because they have not attended class will not fix that problem in a notebook, though a notebook may help them face it and plan. Writing is not a treatment for major depression, an anxiety disorder, an eating disorder or a trauma-related condition, though it may complement treatment. Writing does not replace people. Much of the benefit people get from expressing emotion comes from being heard, and a page cannot hug you, laugh with you or tell you that you are not alone. And writing is not always the right moment's tool. Sometimes the best response to a spiral of worry is not to engage with it at all but to go for a walk, call a friend, eat something or sleep. Part of using journaling well is recognizing when you have written enough. Nor is writing equally suited to every person. Some students think best by talking, and find that a conversation with a trusted friend does what a page cannot. Some have learning differences that make writing laborious, in which case speaking into a voice memo, or writing in bullet fragments rather than sentences, may capture most of the benefit with less effort. The methods in this booklet are described in terms of writing because that is how they were studied, but their core, which is slowing down, specifying and organizing one's thoughts, can be adapted. Within those limits, writing has a particular advantage. It is always available, it costs nothing, and it trains a skill that outlasts any single entry: the habit of stepping slightly outside your own thoughts and looking at them. The next chapter begins with the experiments that first put that skill under scientific scrutiny. Chapter 2: Expressive Writing and the Honest Size of Its Effects In the mid-1980s, a young psychologist at Southern Methodist University named James Pennebaker became curious about a pattern in his data. People who had gone through traumatic experiences and never told anyone about them seemed to have worse health than people who had gone through similar experiences and talked about them. Keeping secrets, he suspected, took effort, and the effort might be costly. If so, then giving people a structured chance to put their experiences into words might help. The experiment he designed with Sandra Beall to test that idea was almost absurdly simple, and it launched a research field that now includes hundreds of studies. It is worth understanding both what that experiment found and what the field has found since, because the honest answer is more modest than the popular one and, in its way, more useful. The original experiment Pennebaker and Beall recruited undergraduates and asked them to write for fifteen minutes on four consecutive days.[1] The students were randomly assigned to one of four groups. One group wrote about trivial topics, such as describing their room or their shoes. A second group wrote about a traumatic or deeply upsetting experience but was asked to focus only on the emotions it produced. A third group wrote about a traumatic experience but focused only on the facts. The fourth group was asked to write about both: the facts of the experience and their deepest thoughts and feelings about it. Writing about trauma was not pleasant. Students in the trauma groups, and particularly those who combined facts and feelings, reported feeling more upset immediately after writing than those who wrote about trivial topics. But in the months that followed, the group that had written about both facts and feelings made fewer visits to the student health center for illness than the other groups. Something about putting a difficult experience into words, fully and with feeling, seemed to have a lasting effect. Later studies extended the finding. Pennebaker and colleagues found that first-year students who wrote about the transition to college made fewer illness visits than students who wrote about superficial topics.[2] Other research groups reported effects on immune markers, grades, absenteeism from work and self-reported mood. By the late 1990s, expressive writing had become one of the most studied brief psychological interventions in the world. What the standard exercise looks like The instructions used in most expressive writing studies are close to the following, paraphrased here rather than quoted. Over the next several days, write for fifteen to twenty minutes each day about your deepest thoughts and feelings concerning the most important emotional issue that has affected you or your life. Really let go and explore your emotions. You might tie the topic to your relationships with others, to your past, your present or your future, or to who you have been, who you would like to be, or who you are now. You can write about the same general issue every day or about different issues. Do not worry about spelling, grammar or sentence structure. Just keep writing until the time is up. The writing is for you alone. Several features of this procedure are deliberate. The writing is private, which frees people to say things they would not say aloud. It is continuous, which keeps people from stopping to edit themselves. It is time-limited, which makes it tolerable. It is repeated over several days, which gives people a chance to return to a topic and see it differently. And it invites both emotion and reflection, which, as Chapter 1 explained, seems to be the combination that matters. What the meta-analyses found Single studies can mislead. A small experiment can find a large effect by chance, and dramatic findings are more likely to be published than dull ones. The better guide to whether something works is a meta-analysis, which combines the results of many studies into a single estimate. Expressive writing has been the subject of several, and their results tell a story of shrinking but persistent effects. The results are usually reported as effect sizes, which describe how far apart the average outcomes of two groups are, measured in units of how much individuals vary. A common convention treats an effect size (Cohen's d) of about 0.2 as small, 0.5 as medium and 0.8 as large. Another way to read an effect size is to ask: if you picked one person at random from the writing group and one from the comparison group, how often would the writer be doing better? With no effect, the answer would be half the time. With an effect of 0.2, it is roughly 56 percent of the time. With an effect of 0.5, roughly 64 percent. The first major meta-analysis, by Joshua Smyth in 1998, pooled 13 studies of healthy participants and found an average effect size close to 0.5, a medium effect, across outcomes including reported physical health, psychological well-being and physiological functioning.[3] It also confirmed that writing increased distress in the short term, and that this short-term distress did not predict worse outcomes later. Among the factors that influenced the size of the effect was whether participants were college students. A 2004 meta-analysis by Pamela Frisina, Joan Borod and Stephen Lepore looked at studies in clinical populations, people with physical or psychiatric conditions. It found a smaller average effect, around 0.2, with the benefit concentrated in physical health outcomes rather than psychological ones.[4] The largest analysis, by Joanne Frattaroli in 2006, gathered 146 randomized studies. Using more conservative statistical methods than the earlier reviews, she found an overall effect that was positive and statistically significant but small: expressed as a correlation, about 0.075, which corresponds to a d of about 0.15.[5] Frattaroli also identified factors associated with larger effects, including more sessions, longer sessions, writing at home in private, and participants who had more stress or worse health to begin with. Most recently, a 2018 meta-analysis by Mira Reinhold, Paul-Christian Bürkner and Heinz Holling focused specifically on depressive symptoms. Across 39 randomized trials with about 4,000 participants, expressive writing produced a very small reduction in depressive symptoms immediately after the intervention (Hedges' g, a close relative of d, of 0.09) and no meaningful difference at later follow-ups averaging about six months.[6] The authors concluded that expressive writing, as usually delivered, is not an effective way to reduce depressive symptoms in the general population. The pattern across these reviews is summarized in Table 2. Table 2. Major meta-analyses of expressive writing. Review Studies included Population Average effect Plain reading Smyth, 1998 13 Healthy adults d about 0.47 Medium Frisina et al., 2004 9 Clinical samples d about 0.19 Small Frattaroli, 2006 146 Mixed r = 0.075 (d about 0.15) Small Reinhold et al., 2018 39 Healthy adults; depressive symptoms only g = 0.09 after, 0.03 later Very small to none Sources: the meta-analyses cited in the text; effect-size conversions are approximate. How to read a small effect It is tempting to read Table 2 as a debunking. The first big estimate was medium; the later, larger and more careful estimates were small. That is a familiar pattern in psychology, and it usually means the early studies overestimated the effect. It would be wrong to pretend otherwise. But a small average effect is not the same as no effect, and it is not the same as a small effect for everyone. Three points help put it in context. First, an average hides variation. If expressive writing helps some people noticeably, helps many people a little and helps some not at all, the average will be small even though the benefit for a particular person may be substantial. Frattaroli's moderator findings suggest that people under more stress may gain more, which is relevant to students writing during a hard semester. Second, the cost is tiny. Four sessions of fifteen or twenty minutes, with a notebook and a pen, is about as cheap as an intervention can be. A small benefit at a tiny cost is a reasonable bargain, especially when it can be combined with other things that help. Third, the outcome matters. The evidence is weakest for depression, where Reinhold and colleagues found essentially no lasting effect. That is an important finding, and it means expressive writing should not be offered to a depressed student as a substitute for treatment. The evidence is somewhat stronger for other outcomes, such as self-reported physical symptoms and, as Chapter 3 will show, performance under test pressure. The right conclusion is not that expressive writing works or does not work, but that it does some things modestly and other things hardly at all. An honest student using expressive writing should therefore expect not a transformation but a modest shift: perhaps a slightly clearer head, a slightly different understanding of a painful event, a slight loosening of a knot. That is worth having. It is not worth pretending to be more. Who tends to benefit, and when The research offers some guidance on who is most likely to get something out of expressive writing and under what conditions, though the guidance is less precise than one would like. People who have been holding something in seem to be good candidates. The original rationale for the method was that not talking about a significant experience carries a cost, and writing offers a way to express what one has not been able to say. A student who has not told anyone about a failing grade, a family crisis at home or the end of a relationship may find the exercise more useful than one who has already talked the experience through at length with friends. Timing matters. Pennebaker himself has advised against writing about a trauma in its immediate aftermath, when feelings are raw and the experience has not yet settled enough to be thought about. The broader research on psychological debriefing, in which people are encouraged to recount traumatic events soon after they happen, has found that such early, structured retelling does not prevent later problems and in some trials seems to have made outcomes worse.[7] The lesson for writing is to allow some time after a very recent shock, and to let the need to write come from you rather than from a sense of obligation. The kind of writing matters. As Chapter 1 described, people who benefit tend to move over the course of their writing toward explanation and insight, not simply to repeat their feelings. If you find yourself writing the same painful sentences day after day without any change, the exercise may not be doing what it is designed to do. Chapter 8 returns to this in detail. Finally, the topic matters. Pennebaker's instructions invite people to write about whatever is most important to them. They do not require the most traumatic experience of one's life. For many students, the most useful topic will be something current and specific: the fear of disappointing a parent, the loneliness of a first semester away, the shock of a first failing grade after years of success. Variations on the method Researchers have tried many variations on the original instructions, partly to find versions that work better and partly to find versions that are less distressing. Two are worth knowing. The first shifts attention from the past to the future. In a study published in 2001, the psychologist Laura King asked students to write for twenty minutes on four consecutive days about their "best possible future self": a life in which everything had gone as well as it reasonably could and they had achieved their goals. Compared with a control group, students who wrote about their best possible selves reported greater well-being afterward and, over the following months, fewer visits to the health center for illness, a benefit similar to that of students who wrote about trauma, but without the upset that trauma writing produced.[8] For a student who is not carrying a particular painful event but is anxious and directionless about the future, this may be a more natural fit. It also has the advantage of prompting concrete thinking about goals, which connects it to the planning methods in Chapter 4. The second variation changes the angle of view. Instead of writing in the first person, from inside the experience, some people find it easier to write about a difficult event as though they were an observer, using "you" or their own name, or as though describing the experience to a sympathetic friend years later. Research on self-distancing, discussed further in Chapter 7, suggests that stepping back in this way can reduce the emotional reactivity of recalling a painful event while still allowing people to make sense of it. For a student who finds first-person writing overwhelming, a shift in perspective can make the exercise tolerable. Neither variation is guaranteed to outperform the original, and the evidence behind each is thinner than the evidence behind the standard method. But they show that expressive writing is not a single fragile ritual. Its core ingredients, which are privacy, continuity, repetition and a mixture of feeling and thinking, can be arranged in different ways to suit different people. A student version of the protocol Drawing on the research, here is a way for a student to try expressive writing that stays close to the conditions of the studies. Consider a sophomore whose parents announced over winter break that they were separating, and who returned to campus without telling anyone. Three weeks in, she finds it hard to concentrate and feels flat in a way she cannot explain. She chooses four evenings in the same week, when she has a free half hour and a private space. Each evening she sets a timer for twenty minutes. She writes by hand, without stopping, about her deepest thoughts and feelings about her parents' separation and what it means for her. On the first night she mostly describes what happened and how shocked she was. On the second night she writes about anger and then, unexpectedly, about guilt. On the third night she finds herself writing about her own fear of relationships ending, and about what she wants her own life to look like. On the fourth night the writing is calmer; she writes about what she can and cannot control, and about the friend she might tell. After each session she does something ordinary and pleasant for a few minutes, because she knows from the research that writing can leave people feeling temporarily sadder. She does not reread the pages right away. She keeps them in a place her roommate will not find them, and she knows she can throw them away if she wants to; in the research, the benefit comes from the writing, not from keeping the record. The sophomore is a composite, not a real person, and no one should expect their own four evenings to unfold as neatly. But the structure is sound. A few practical rules help: · Choose a topic you can handle. If thinking about a subject makes you feel that you would be overwhelmed by writing about it, choose a different subject or wait. Pennebaker has called this the "flip-out rule": if you feel you would flip out writing about something, don't write about it now. · Keep it private. Write as though no one will ever read it, and make sure no one will. · Give yourself a buffer. Leave a few minutes after writing before you have to be anywhere or do anything demanding. · Stop if it is making things worse. A little sadness after writing is normal. A sustained worsening in mood over several days is a signal to stop and, if it persists, to talk to someone. Expressive writing is the method that launched the science of therapeutic writing, but it is only one of the tools this booklet describes. The next chapter looks at a shorter and more targeted variation, designed not to process a painful past but to clear the way for a demanding present: the exam. Chapter 3: Ten Minutes Before the Exam Every campus has its version of the scene outside the exam hall in the last minutes before the doors open. Some students are flipping frantically through flash cards. Some are comparing notes and discovering, with rising alarm, topics they had not studied. Some are sitting apart, headphones on, trying to shut it all out. A few are gripped by something close to panic, the kind that makes the stomach turn and the mind go blank. The last group is the one this chapter is about. They are often not the students who studied least. Many of them know the material well. What they lack in those minutes is not knowledge but access to it, and a line of research led by the cognitive scientist Sian Beilock suggests that a short, deliberately uncomfortable act of writing can give some of that access back. Why good students choke Beilock's research began with a question about performance under pressure. Why do people who are capable in practice fall apart when the stakes rise? Her answer, developed across many studies and summarized in her book Choke, is that pressure triggers worries about the situation and its consequences, and those worries consume the working memory the task requires.[1] Chapter 1 described this mechanism in general terms. Beilock's contribution was to show it in action on precisely the kind of tasks that exams demand. One consequence of this account is counterintuitive. It is often the students with the most working memory capacity, the ones who usually do best on demanding problems, who suffer most under pressure. They rely heavily on working memory to solve problems, so when worry eats into it, they lose more. Students who rely on simpler shortcuts lose less, because they had less to lose. Pressure, in other words, can erase the advantages of the students who have worked hardest to acquire them. If the problem is that worry takes up space in working memory, then a natural question follows: could students clear that space before the test begins? Earlier research on expressive writing, described in Chapter 2, had suggested that writing about worries can reduce their intrusiveness. Beilock and her graduate student Gerardo Ramirez set out to see whether a single short session of writing, immediately before a test, could do the same. The experiments Ramirez and Beilock published their results in the journal Science in 2011.[2] The paper reported four experiments, two in the laboratory and two in real classrooms. In the first laboratory study, twenty college students took two short math tests. The first was presented as low-stakes practice. Before the second, the researchers turned up the pressure: students were told that they could earn money for good performance, that their performance was tied to a partner who had already done well, and that they would be videotaped so that math teachers could review their work. These are artificial pressures, but they reliably produce the feeling of being evaluated. Before the high-pressure test, half the students sat quietly for ten minutes. The other half spent those ten minutes writing as openly as possible about their thoughts and feelings about the upcoming test. The students who sat quietly choked. Their accuracy fell by about 12 percent from the practice test to the high-pressure test. The students who wrote about their worries did not choke; their accuracy actually rose by about 5 percent. A second laboratory study addressed an obvious objection. Perhaps any writing would help, simply by giving students something to do. So this time a comparison group wrote for ten minutes about an unrelated, unemotional event. Those students still choked. Only writing about worries about the test itself protected performance. The classroom studies moved the experiment into the real world. Ninth-grade students at a Chicago-area high school were preparing for their first high school final exam, in biology. Some weeks before the exam, the researchers measured each student's general level of test anxiety. On the day of the final, students were randomly assigned to spend the first ten minutes either writing about their thoughts and feelings about the exam or writing about a topic unlikely to be on the exam. Among students with low test anxiety, the writing made little difference. But among the most test-anxious students, those who had written about their worries earned an average grade of about B+, while those who had written about an unrelated topic averaged about B-. A second classroom study replicated the pattern with a different comparison task. A later laboratory study by Daeun Park, Ramirez and Beilock extended the work to math anxiety specifically, a fear of mathematics that affects many students regardless of their actual ability.[3] Among 80 university students, highly math-anxious students who sat quietly before a math test did significantly worse on math problems than students low in math anxiety. Among those who first wrote about their thoughts and feelings regarding the test, the gap between high- and low-anxious students was substantially reduced. Echoing Pennebaker's findings, the highly anxious students who benefited most were the ones who used more words related to anxiety, causation and insight in their writing. It was not venting alone that helped; it was writing that engaged with and made sense of the fear. What the replications show A finding published in Science, with striking classroom results, invites enthusiasm. It also deserves the same honest scrutiny that Chapter 2 applied to expressive writing in general, because later attempts to use the method in college classrooms have produced mixed results. The clearest test came from a pair of experiments by Kelsey Myers, Jennifer Davis and Jason Chan at Iowa State University, published in 2021.[4] Students in a cognitive psychology course completed either an expressive writing exercise or a control writing exercise before each of four real course exams, alternating between conditions so that each student served as their own comparison. Across 173 students in the first experiment, exam scores after expressive writing (an average of 73 percent) were essentially identical to scores after control writing (74 percent). The writing did not reduce test anxiety, and it did not weaken the link between anxiety and lower scores. The students did write about their anxieties, as instructed; it simply made no measurable difference. Other studies point in different directions. A small randomized trial with Chinese high school students who had severe test anxiety found large reductions in anxiety after thirty days of daily writing, though that was a longer intervention focused on positive emotion, and very large effects in small samples often shrink when studies are repeated.[5] A classroom implementation in an introductory biology course at the University of Washington could not measure exam scores, because the writing was anonymous, but instructors observed that mentions of anxiety in the writing declined over the term and that students seemed calmer; many chose to crumple up and throw away their pages before the exam, which seemed to release tension in the room.[6] What should a student make of this? The honest summary is that the pre-exam writing effect is real in the laboratory, was striking in the original classroom studies, and has not reliably reproduced in every classroom since. Several reasons are plausible. College students taking their fourth exam of a course may have already developed their own coping strategies, leaving less room for improvement. The effect may depend on how anxious students are, and many studies include a mix. And an exercise imposed by an instructor on a whole class may feel different from one a student chooses for themselves. None of this makes the method worthless. It makes it a reasonable experiment rather than a guaranteed fix. For a student who knows that anxiety regularly sabotages their exam performance, ten minutes of writing costs very little, and the student is in a good position to judge whether it helps them. How to do it The procedure used in the research is simple. In the ten minutes before an exam, or as close to the start as possible, write as openly as you can about your thoughts and feelings about the exam you are about to take. Do not try to calm yourself or argue with your fears. The point is to put them on paper, not to fix them. Write continuously. Do not worry about grammar or neatness. Consider a first-year student who freezes on calculus exams. She studies hard, does well on homework and then watches her mind empty as soon as the exam starts. She decides to try the method before her next midterm. She arrives at the exam building twenty minutes early, finds a bench, sets a timer on her phone and writes in a pocket notebook: I'm scared I'll blank again like last time. My hands are already sweaty. I know how to do related rates but I'm afraid I'll panic on the first problem and then everything will go. I feel like everyone else finds this easy. If I fail this I'll have to retake it and my parents will... After a few minutes, the writing shifts: Last time the problem was I spent too long on number 2. If I get stuck I'll skip and come back. When the timer goes off, she closes the notebook, puts it in her bag and walks in. That shift toward a small plan was not part of the instructions, but it often happens, and it echoes the finding that writing which moves toward understanding tends to help most. Her experience is an illustration, not a promise. But it shows the shape of the practice. A few practical points help: · Timing. The research placed the writing immediately before the test. Writing the night before is a different exercise (Chapter 4 covers bedtime writing), and it may help in its own way, but it is not what the pre-exam studies tested. · Location. If you cannot write in the exam room, write outside it, or in a restroom, or on the bus. Some instructors, having heard of this research, now allow a few minutes of writing at the start of an exam. · Disposal. You do not need to keep the writing. Some students find that throwing the page away is part of the relief. · Duration. Ten minutes is enough. Longer writing may simply eat into time you need, and it risks tipping into rumination. · Judgment. Try it for two or three exams before deciding whether it helps. A single experience is not a test. Pairing the writing with a different reading of arousal Pre-exam writing addresses the worries in an anxious student's head. It does not directly address the pounding heart, the dry mouth and the shaky hands that often come with them, and many students interpret those bodily signals as proof that disaster is coming. A separate line of research suggests that the interpretation matters, and that a few written sentences can change it. In a study led by Jeremy Jamieson and colleagues, students preparing to take the Graduate Record Examination completed a practice test in the laboratory. Before the test, some were given a short passage explaining that feeling anxious during a standardized test does not necessarily hurt performance and can even help, because the body's arousal response mobilizes energy for the task. The rest received no such information. Students who read the reappraisal passage scored higher on the math section of the practice test, and when the researchers later obtained their scores on the real GRE, taken months afterward, the reappraisal group had again scored higher on math.[7] The sample was small and the effect was on one section of one test, but the finding fits with a broader body of work on reappraisal. In a related set of experiments, Alison Wood Brooks at Harvard asked people facing anxiety-provoking tasks, including singing in front of others, giving a speech and solving math problems, to say a simple phrase beforehand. Those who told themselves "I am excited" performed better than those who told themselves "I am calm" or said nothing.[8] Brooks argued that anxiety and excitement are both high-arousal states, so shifting from one to the other is easier than trying to shift from high arousal to calm. Telling a racing heart to slow down rarely works; telling it that it is getting ready for something important sometimes does. These studies suggest a simple addition to the pre-exam routine. After writing about the worries for most of the ten minutes, a student can close with two or three sentences that reinterpret the body's signals: My heart is racing because my body is getting ready. This energy can help me think faster. I care about this exam because it matters to me, and that is not a bad thing. This is not a matter of denying fear or pretending to feel confident. It is a matter of offering the nervous system a more accurate account of what it is doing, since arousal before an exam is a preparation response, not a sign of impending failure. A student should be careful not to turn this into a demand to feel positive. If the sentences feel false, they will not help, and forcing them may add a layer of frustration. The aim is a reading of the situation that is both true and useful, and for many students, "I am nervous because this matters, and nervous energy can be used" meets both tests. Writing in the days before Some students ask whether writing about exam worries in the days leading up to an exam, rather than just before it, can help. The honest answer is that the direct evidence is thinner. The classroom studies that found the biggest effects placed the writing immediately before the test, and the mechanism they proposed, clearing working memory at the moment it is needed, is specific to that timing. But there are other good reasons to write in the run-up to an exam, and they draw on methods covered elsewhere in this booklet. A student who is anxious for days before a test may benefit from a thought record (Chapter 6) to examine the prediction that the exam will go badly, from scheduled worry time (also Chapter 6) to keep anxiety from swallowing every study session, or from a bedtime to-do list (Chapter 4) to protect sleep the night before. Sleep deserves particular emphasis. A student who lies awake worrying until three in the morning arrives at the exam with less working memory to protect, and no amount of pre-exam writing will fully make up for that. The best exam-day writing routine starts the night before. Beyond the exam hall The logic of pre-exam writing applies wherever a performance is threatened by worry about the performance: a class presentation, an audition, a job interview, a difficult conversation with an adviser. In each case, the idea is the same. Name the fears on paper just before the event, so that they take up less space in the mind during it. The method also has limits that are worth stating directly. It does not replace preparation. A student who has not studied will not be rescued by ten minutes of writing, and anxiety that is a reasonable response to being unprepared is better addressed by preparing. It is not a treatment for severe test anxiety that makes a student avoid exams altogether, miss them or feel unable to function; that pattern calls for a conversation with a counselor or the campus disability or accessibility office, which may be able to arrange accommodations or refer the student to structured treatment. And it is not a substitute for the slower work, described in Chapter 6, of examining the beliefs that make exams feel so dangerous in the first place. Within those limits, pre-exam writing is one of the rare interventions that can be tested in a single morning, at no cost, by the person who most needs it. Its evidence is incomplete. Its risks are close to nil. For many anxious students, that combination makes it worth a try. Chapter 4: Brain Dumps, To-Do Lists and the Bedtime Mind It is Sunday evening in the middle of the semester. A senior majoring in economics sits down to plan the week and finds that she cannot think clearly about any of it. There is a problem set due Tuesday, an internship application due Friday, a group project whose other members have gone quiet, a midterm the following Monday, a shift at her campus job, an email from her adviser she has not answered, a birthday present to buy for her brother, laundry, and a vague sense that there is something else she has forgotten. None of these tasks is overwhelming on its own. Together, held loosely in her head, they produce a low hum of dread that makes it hard to start any of them. This is a different kind of stress from the painful memories of Chapter 2 or the acute fear of Chapter 3. It is the stress of a crowded mind, and it calls for a different kind of writing. Instead of exploring feelings, the task is to get everything out of the head and onto a surface where it can be seen, sorted and planned. This chapter covers that practice, commonly called a brain dump, and a close relative with unusually specific evidence behind it: the bedtime to-do list. Why unfinished tasks will not leave you alone The mind is built to keep track of unfinished business. An incomplete goal tends to stay active in memory, surfacing periodically to remind you it is there. Most of the time this is helpful; it is how you remember to buy milk on the way home. But when the number of open tasks grows large, the reminders become a constant background noise, and they arrive whenever there is a lull: in the shower, during a lecture, and especially in bed. Chapter 1 described experiments by E. J. Masicampo and Roy Baumeister showing that unfulfilled goals produce intrusive thoughts that interfere with other tasks.[1] Their key finding is worth restating because it is the foundation of this chapter. The intrusions did not require the goal to be completed in order to fade. They faded when participants made a specific plan for completing the goal. In the language of the productivity writer David Allen, whose book Getting Things Done popularized the brain dump for a general audience, the mind seems to need confidence that each open loop has been captured somewhere trustworthy and has a defined next step.[2] Once it has that confidence, it can stop reminding you. This explains why a vague intention, such as "I need to deal with the group project," does little to quiet the mind, while a specific plan, such as "Monday after lunch, text Jordan and Priya to propose a meeting Wednesday at four," does a great deal. The vague version is still an open loop. The specific version is a credible commitment. The bedtime to-do list The most direct test of this idea for sleep comes from a study by Michael Scullin and colleagues at Baylor University, published in 2018.[3] Fifty-seven healthy young adults aged 18 to 30 spent a night in a sleep laboratory, where their sleep was measured with polysomnography, the same set of brain, eye and muscle recordings used in clinical sleep studies. Before lights out, each participant spent five minutes writing. Half were randomly assigned to write a to-do list of the tasks they needed to remember to do over the next few days. The other half wrote about tasks they had completed over the previous few days. The participants who wrote to-do lists fell asleep significantly faster: on average in about 16 minutes, compared with about 25 minutes for those who wrote about completed tasks. The more specific their to-do lists were, the faster they fell asleep. For the group writing about completed tasks, the pattern ran the other way; writing more about what they had done was associated, if anything, with slightly slower sleep onset. The finding is intuitive once you connect it to the research on unfinished goals. Writing about completed tasks leaves the open loops untouched and may even draw attention to how much remains undone. Writing a to-do list moves the open loops out of the head and onto paper, where they will still be in the morning. The mind can let go. The study has limitations that its authors acknowledged. The participants were healthy young adults without sleep disorders, spending a single night in an unfamiliar laboratory. Nine minutes is a meaningful difference but not a dramatic one, and the findings may not apply to people with clinical insomnia. The comparison group wrote about completed tasks, not nothing, so the study shows that a to-do list beats a done list rather than that a to-do list beats no writing at all. A larger replication would strengthen the case. But the method is so cheap and so easy that a student having trouble falling asleep because of a racing mind has little to lose by trying it. Doing a full brain dump The bedtime to-do list is a compact version of a broader practice. A full brain dump, done once or twice a week, clears the accumulated clutter more thoroughly and turns it into a plan. The first stage is capture. Take a blank sheet of paper, or a blank document, and write down everything that is occupying your mind: tasks, deadlines, worries, errands, ideas, things you have been meaning to do, things you are dreading, questions you need to ask someone. Do not sort, prioritize or judge at this stage. The goal is completeness. Write until you cannot think of anything else, then wait a minute and see whether anything more surfaces; it usually does. For a busy student this list may run to thirty or forty items, and seeing it can be a shock. It is also, for most people, a relief. The list is finite. The dread was not. The second stage is sorting. Go through the list item by item and ask what kind of thing each one is. Most items fall into a handful of categories, and Table 3 describes one useful way to sort them. Table 3. Sorting the items in a brain dump. Category What it is What to do with it Student example Next action A task you can do soon Write the specific first step Email Professor Lee about the extension Scheduled A task tied to a date Put it in a calendar with a time Start lab report Wednesday 2-4 pm Ask or hand off Something someone else must do or decide Note who and how you will contact them Ask roommate about rent split Worry to examine A fear, not a task Move it to a thought record or worry time "I'll never get an internship" Let go Not yours, not now, or not needed Cross it out deliberately Redesign my whole study system The third category deserves special attention. Brain dumps often mix tasks with worries, and the two need different treatment. "Finish problem set" is a task: it has a next action and a completion point. "I'm falling behind everyone" is a worry: it has no next action and no completion point, and trying to put it on a to-do list only makes the list feel more hopeless. Separating worries from tasks is one of the most useful things a brain dump does. The worries do not disappear, but they are moved to methods better suited to them, described in Chapter 6. The last category is also underrated. Many items on a student's list are not really commitments at all. They are vague aspirations ("get fit," "read more"), guilt-driven obligations nobody actually expects, or tasks that were important three weeks ago and no longer matter. Crossing them out, deliberately and with a clear decision, can be one of the most relieving parts of the whole exercise. Making the next step specific The Scullin study found that specificity mattered, and a broader body of research supports the point. The psychologist Peter Gollwitzer has studied what he calls implementation intentions: plans of the form "When situation X arises, I will do Y." A 2006 meta-analysis by Gollwitzer and Paschal Sheeran, pooling 94 independent tests, found that forming implementation intentions had a medium-to-large effect on whether people actually achieved their goals (an average d of about 0.65), compared with simply setting the goal.[4] Specific plans help people start tasks, resist distractions and follow through. For a student, this means that the most important thing to write next to each task is not a deadline but a first step, tied to a time and a place. "Write the internship cover letter" is a goal. "Tuesday at 10 in the library, open the application and draft the first paragraph" is an implementation intention. The second version is more likely to happen, and, if the Masicampo and Baumeister findings apply, it is also more likely to stop the task from nagging at you in the meantime. Return to the economics senior. After her brain dump, which takes about fifteen minutes, she has thirty-one items. She crosses out six. She identifies four worries ("I'm not going to get a job," "Jordan and Priya think I'm bossy," "I should have picked a different major," "I don't know what I'm doing after graduation") and sets them aside for a separate session. The remaining items go into her calendar or onto a short list of next actions, each with a specific first step. The group project item becomes "Monday 1 pm: text group proposing Wednesday meeting." When she goes to bed, the hum of dread is quieter. Nothing has been completed, but everything has a place. Triage for crunch weeks There are weeks when the honest result of a brain dump is that the tasks do not fit into the time available. Finals week, a stretch with three midterms and a paper, a week when illness or a family emergency has cost several days: in these cases, a list that simply records everything can deepen the panic rather than relieve it, because it makes the impossibility visible without offering a way through. The brain dump is still the right starting point, but it needs a further step, which is triage. Triage is a word borrowed from emergency medicine, where it means deciding who gets treated first when not everyone can be treated at once. Applied to a student's week, it means deciding in writing which tasks matter most, which can be done less thoroughly and which will have to be renegotiated. A simple way to do this is to write three short lists beneath the brain dump. The first holds the tasks where full effort really matters this week, usually the ones that carry the most weight in a grade or the most serious consequences if missed. The second holds tasks that can be done adequately rather than excellently: the reading response that can be solid instead of brilliant, the problem set on which a few questions can be left partly done. The third holds tasks that need to be moved, and next to each one, the person you need to contact to move it. That third list is where many students stall, because asking for an extension or a change feels like admitting failure. It helps to write out the request before sending it. A short, factual email that states the situation, proposes a specific new deadline and does not over-apologize is usually received better than students expect, particularly when it is sent before the deadline rather than after. Writing it first in a notebook, where no one will see a clumsy draft, lowers the barrier. Many campuses also have a dean of students office or an academic advising office that can help when several courses are affected at once, for example after an illness or a family crisis. Consider a junior who wakes on the Monday of finals week with a fever. By Wednesday she is well enough to work but has lost two days. Her brain dump shows two exams, a final paper and a lab practical. Triage puts the two exams in the first list, since they cannot be moved and carry heavy weight. The lab practical goes in the second, because she knows the material well and can review it lightly. The paper goes in the third, with a note to email the professor that morning, attaching the note from the campus health center. The email takes ten minutes to draft in her notebook and two to send. The professor grants three extra days. The week is still hard, but it has become possible. This kind of writing is less about emotion than about decision, but its emotional effect can be large. Much of the panic of a crunch week comes from trying to hold every task at full priority at once. Writing the priorities down, and accepting in writing that some things will be done less well or later, turns an impossible week into a difficult one. A bedtime routine that protects sleep Sleep is where the crowded mind does most of its damage, and it is also where students have the most to gain. A student who falls asleep twenty minutes faster, several nights a week, recovers hours of rest over a semester, and sleep supports mood, memory consolidation and the working memory that anxiety erodes. A reasonable bedtime writing routine, drawing on the research, looks something like this: 1. Earlier in the evening, not in bed, spend ten to fifteen minutes on any real planning or problem-solving the day requires. Sleep researchers working with college students with insomnia have tested a version of this, sometimes called "constructive worry," in which students write their concerns and possible next steps early in the evening, and found that it reduced the mental arousal they experienced at bedtime.[5] 1. At bedtime, spend about five minutes writing a specific to-do list for the next day or two. Be concrete. Include the small things. 2. Close the notebook and put it somewhere you will see it in the morning, so the mind knows the list will be there. 3. If new tasks surface after lights out, jot them down in a notebook by the bed without turning on bright lights or picking up a phone, then let them go. Paper has an advantage over a phone here. A phone brings notifications, bright light and the temptation to check one more thing, all of which work against sleep. A cheap notebook and a pen by the bed avoid all three. The bedtime list should be short and practical. It is not the place for a full brain dump, for expressive writing about painful events or for detailed planning of a long project. Writing that opens up big questions at bedtime is more likely to wake the mind than settle it. If a worry keeps returning after the list is written, it is a sign that the worry needs its own time during the day, a practice the next chapters describe. When the list becomes the problem Lists help most people, but they can become a source of stress in their own right. Some students respond to a long list by spending hours organizing it rather than doing any of it, color-coding tasks and redesigning systems as a sophisticated form of procrastination. Others find that every item left undone at the end of the day becomes evidence of failure. And a few, particularly those prone to perfectionism or obsessive worry, find that writing things down becomes a compulsion: they feel they must record everything, and cannot rest until the list is complete. A few guardrails help. Keep the daily list short enough to be realistic; many people find that three to five important items is about right, with smaller tasks grouped separately. Accept that some items will roll over to tomorrow, which is a normal feature of a busy life rather than a failure. And if list-making starts to feel driven by anxiety rather than relieving it, if you find you cannot stop adding items, checking and rewriting, treat that as useful information. It may be a sign that the underlying worry is larger than any list, and that it deserves attention in its own right, whether through the methods in Chapter 6 or through a conversation with a counselor. Chapter 5: Gratitude Logs and the Uses of Good Things The methods in the previous chapters all work directly on the sources of stress: a painful event, an imminent exam, a crowded list of tasks. Gratitude writing works differently. It does not ask students to confront what is going wrong. It asks them to notice what is going right, which, in the middle of a hard semester, can feel either like welcome relief or like an insult. That tension is the right place to start, because gratitude writing is one of the most popular forms of journaling and one of the most easily misused. Its evidence is real but modest, its benefits depend heavily on how it is done, and done badly it can leave students feeling worse. Done well, it offers something none of the other methods do: a counterweight to the mind's natural tendency to spend its attention on threats. Why the mind needs a counterweight The human mind is tilted toward the negative. Psychologists have documented this bias across many domains: bad events tend to have larger and more lasting effects on mood than good events of similar size, criticism tends to be remembered longer than praise, and threats capture attention faster than rewards. There are good evolutionary reasons for the tilt. Missing a threat can be fatal; missing an opportunity usually is not. But in a modern student's life, where most threats are to grades and reputations rather than to survival, the tilt can make a semester feel much worse than it is. Consider a sophomore at the end of an ordinary week. On Monday a professor praised her question in class. On Tuesday she had a long, easy lunch with a friend. On Wednesday she got back a quiz with a mediocre score. On Thursday she ran further than she had managed before. On Friday a group member failed to deliver his part of a project. Asked on Friday night how the week went, she says it was terrible, and she means it. The quiz and the group member have crowded out everything else. Gratitude writing is a deliberate act of rebalancing. It does not deny the quiz or the group member. It simply insists that the praise, the lunch and the run also happened, and gives them a place on the page where they can be seen. Why good feelings matter under stress It might seem that positive emotions are a luxury for students under pressure, something to enjoy once the real work of managing stress is done. The psychologist Barbara Fredrickson has argued the opposite. Her broaden-and-build theory proposes that negative emotions narrow attention toward a threat, which is useful in an emergency, while positive emotions broaden attention and thinking, making people more open to new ideas, more flexible in solving problems and more likely to build relationships and skills that serve them later.[1] Positive emotions, on this view, are not just pleasant. They are resources that accumulate. Fredrickson and colleagues tested one implication of this idea in a study of college students who had completed measures of resilience before the September 11, 2001 attacks and were reassessed in the weeks afterward. Students who were more resilient reported experiencing more positive emotions, such as gratitude, love and interest, alongside their distress, and those positive emotions statistically accounted for part of why the resilient students were less likely to develop depressive symptoms.[2] The study was correlational, so it cannot show that gratitude caused the protection. But it is consistent with the idea that the capacity to notice good things does not compete with coping; it is part of coping. For students, the practical point is that noticing good things is not a distraction from dealing with stress. A stressed mind narrows. A few minutes spent deliberately widening it may make the rest of the week's problem-solving a little more flexible. What the studies found The best-known experiments on gratitude writing were published in 2003 by Robert Emmons and Michael McCullough.[3] In their first study, 192 undergraduates kept weekly records over nine weeks. One group was asked to list up to five things in their lives, large or small, that they were grateful for. A second group listed hassles, the irritants and annoyances of daily life. A third group listed events that had affected them, without any particular emotional slant. At the end of the study, the gratitude group rated their lives as a whole more favorably and were more optimistic about the coming week than the other groups. They also reported fewer physical complaints and spent about an hour and a half more per week exercising than the hassles group. A second study, with 157 students completing daily entries over thirteen days, found that the gratitude group reported more positive emotion than the hassles group and was more likely to report having helped someone with a personal problem or offered emotional support, though the physical health and exercise effects did not appear. A third study, with adults living with neuromuscular disease, found that three weeks of daily gratitude writing increased positive mood and was associated with longer and better-rated sleep, and the participants' spouses independently noticed the difference. Other experiments followed. In a widely cited 2005 study, Martin Seligman and colleagues asked adults recruited online to write down three things that went well each day, along with why they went well, every night for one week. Participants reported increases in happiness and decreases in depressive symptoms that persisted, in that study, for six months.[4] Later attempts to replicate this result found that the benefits were harder to distinguish from those of a comparison exercise, which is a common fate for striking early findings in this field.[5] The most useful summary of the evidence comes from a 2016 meta-analysis by Don Davis and colleagues.[6] They were careful to distinguish between different kinds of comparison. Compared with doing nothing, gratitude interventions improved psychological well-being with an effect size of about 0.31. Compared with a neutral alternative activity, such as listing daily events, gratitude interventions still came out ahead, but by less: an effect of about 0.17 on well-being. Compared with other psychologically active exercises, such as other positive writing tasks, gratitude interventions performed about equally well, but not better. And the meta-analysis found no significant effect of gratitude interventions on anxiety. The last finding deserves emphasis for a booklet about student stress. Gratitude writing does not appear to reduce anxiety directly. Its benefits are to well-being, mood and perhaps a sense of perspective. For a student in the grip of exam dread, a gratitude list is the wrong tool; the methods in Chapters 3, 4 and 6 are better suited. For a student in a flat, gray stretch of the semester, when nothing is acutely wrong but nothing feels good either, gratitude writing may be exactly right. How often, and how One of the more surprising findings in this literature concerns frequency. In a study described by Sonja Lyubomirsky, Kennon Sheldon and David Schkade in 2005, participants were asked to "count their blessings" either once a week or three times a week for six weeks. Those who did it once a week showed increased well-being. Those who did it three times a week did not.[7] The authors suggested that doing the exercise too often made it a chore and drained it of meaning. Not every study has examined frequency, and the result should not be treated as a firm rule. But it matches the experience of many people who start a daily gratitude journal with enthusiasm and find, after two weeks, that they are writing "my friends, my family, my health" on autopilot. A related finding comes from a set of studies by Lyubomirsky, Lorie Sousa and Rene Dickerhoof, published in 2006.[8] They compared writing, talking and thinking about both negative and positive life experiences. For negative experiences, writing and talking were more beneficial than simply thinking, which fits with the structuring effect of language described in Chapter 1. For positive experiences, the pattern reversed: replaying a happy memory in one's mind was more beneficial than writing about it analytically. The researchers argued that analyzing a positive event can explain it away, reducing its wonder and emotional charge, while simply savoring it preserves it. These findings suggest several principles for gratitude writing: · Less can be more. Once or twice a week is a reasonable starting point. If daily writing stays fresh, continue. If it becomes rote, cut back. · Specific beats general. "My roommate made me tea when I was stressed on Tuesday" carries more feeling than "my roommate." Specificity also resists autopilot, because each week's specifics are different. · Savor, don't analyze. When writing about a good thing, try to recall it vividly, what it looked and felt like, rather than to explain why it happened. A sentence of vivid description can do more than a paragraph of reflection. · Variety helps. Rotating between people, experiences, small pleasures and personal strengths keeps the exercise from going stale. · Include your own part. Noting what you did to bring a good thing about ("I ran further because I kept going when I wanted to stop") builds a sense of agency along with gratitude. Letters of gratitude A variation that many students find more powerful than a list is the gratitude letter: a letter to a specific person thanking them for something they did. Seligman's 2005 study included a "gratitude visit," in which participants wrote and delivered a letter to someone they had never properly thanked, and it produced large short-term gains in happiness that faded within a few weeks.[4] A study by Steven Toepfer, Kelly Cichy and Patti Peters, in which college students wrote three letters of gratitude over three weeks, found improvements in happiness and life satisfaction and reductions in depressive symptoms compared with a control group, even though the letters were not necessarily sent.[9] For students, the letter has particular value. College is often a time of distance from people who shaped one's life: a teacher who believed in you, a grandparent who paid for textbooks, a coach, an older sibling. Writing to one of them, whether or not the letter is ever sent, can restore a sense of connection and of being part of a larger story. For a lonely first-year student, it can be a reminder that the people who matter are still there. Positive affect journaling A broader approach, sometimes called positive affect journaling, goes beyond gratitude to include writing about positive experiences, personal strengths, hopes and sources of meaning. In a randomized trial published in 2018, Joshua Smyth and colleagues asked adults with elevated anxiety symptoms, recruited from general medical settings, to write online for fifteen minutes, three times a week, for twelve weeks, using prompts that invited them to reflect on positive experiences and emotions. Compared with a usual-care control group, the journaling group reported less mental distress and greater well-being after one month, and some benefits persisted through the study.[10] The trial was small and preliminary, and its participants were not students, but it suggests that positive writing can be useful for anxious people when it is framed broadly rather than as a narrow list of blessings. A weekly practice in a hard semester Consider a first-generation student in her first year at a large university, several hundred miles from home. She works fifteen hours a week in the dining hall, is finding her introductory economics course far harder than she expected, and calls her mother every Sunday feeling she has little good news to report. A friend suggests gratitude journaling. She tries writing three things every night and gives up within ten days: the entries became repetitive and started to feel like homework. She tries again with a different structure. Every Sunday afternoon, before calling home, she writes for ten minutes. She lists four or five specific good things from the past week, each with one sentence recalling what it felt like rather than why it happened. One week the list includes a dining hall coworker who covered for her when she was late, an economics concept that finally clicked during office hours, a video call with her younger brother in which he showed her his science project, and a cold, clear morning walk to class. She finishes with one sentence about something she did that she is quietly proud of: I went to office hours even though I was embarrassed. Two things change. The entries stay fresh, because a week holds enough variety that the lists rarely repeat. And her Sunday calls become easier, not because her problems have gone away, but because she arrives at them with a clearer sense of what actually happened in her week, good as well as bad. The economics course is still hard. She is still tired. But the week no longer collapses in memory into a single gray impression of struggle. This student is a composite, and her experience is an illustration rather than evidence. But the structure, weekly rather than daily, specific rather than general, savoring rather than analyzing and with a note of personal agency, draws directly on the research described in this chapter. When gratitude backfires Gratitude writing can go wrong in ways that are worth naming directly. The first is invalidation. A student going through something genuinely hard, a family illness, a financial crisis, an experience of discrimination or assault, may find that being told to count their blessings feels like being told that their pain does not matter. Gratitude writing should never be used to talk oneself out of legitimate distress. The painful experience needs its own attention, whether through the methods of Chapter 2, through people who can help or through professional support. Gratitude can sit alongside that attention; it cannot replace it. The second is guilt. Some students, particularly those who know their families have sacrificed for their education, find that gratitude lists turn into lists of debts. Every good thing becomes a reminder of what they owe and how they are failing to repay it. If gratitude writing leaves you feeling more indebted than grateful, shift the focus to small, present pleasures that carry no obligation: a good meal, a song, the light at a particular time of day. The third is performance. Gratitude journaling has become fashionable, and there is a version of it that is really about presenting oneself as grateful, to others or to oneself, rather than about noticing anything. If the exercise feels like a performance, it is worth making it smaller, more private and more specific, or setting it aside for a while. The fourth is using gratitude to stay in a bad situation. There is a difference between appreciating what is good in one's life and using that appreciation to avoid facing something that needs to change. A student in a harmful relationship, a toxic lab, or a major they have come to hate does not need a gratitude list that argues them into staying. They need honest writing, of the kind described in Chapters 2 and 6, about whether and how to leave. Used with these cautions in mind, gratitude writing does something the other methods do not. The other methods work on problems. Gratitude writing works on the frame through which problems are seen, widening it just enough to include what the mind's natural tilt would leave out. For a student who has spent a semester seeing only threats, that widening can be a quiet but real relief. Chapter 6: Thought Records and Worry Time A sophomore opens the course website and sees the grade for his first statistics midterm: 68. Within a few seconds a chain of thoughts has formed, fully assembled and entirely convincing. I'm going to fail this course. If I fail statistics I can't stay in the psychology major. If I can't do psychology I don't know what I'll do. I'm not smart enough for college. My parents are going to be so disappointed. By the time he closes the laptop, he feels sick, and the feeling is attached not to a single disappointing grade but to the collapse of his entire future. The expressive writing of Chapter 2 would let him explore those feelings, and that might help. But there is another, more structured approach, drawn from cognitive behavioral therapy, that treats each of those thoughts as a claim that can be examined. It does not ask him to think positively. It asks him to think accurately. This chapter describes that approach, the thought record, along with a second technique from the same tradition, scheduled worry time, which gives chronic worry a place and a boundary so it stops leaking into the rest of the day. Thoughts are not facts Cognitive behavioral therapy, usually called CBT, grew out of the work of the psychiatrist Aaron Beck in the 1960s. Beck noticed that his depressed patients experienced a stream of quick, habitual, negative thoughts about themselves, their world and their future, which he called automatic thoughts. These thoughts felt like facts to the patients, but when examined they were often distorted in characteristic ways. Beck proposed that changing these thoughts, and the deeper beliefs behind them, could change how people felt. Over the following decades CBT became one of the most thoroughly tested forms of psychotherapy. A 2012 review of meta-analyses by Stefan Hofmann and colleagues found strong support for its effectiveness across anxiety disorders, and good support for many other conditions.[1] That evidence concerns CBT delivered by trained therapists over a series of sessions, not self-help writing, and a student should not assume that filling in thought records alone is equivalent to therapy. But the thought record is one of CBT's core tools, and many therapists teach it precisely so that clients can use it on their own between sessions and after treatment ends. Popular workbooks such as Mind Over Mood by Dennis Greenberger and Christine Padesky are built around it.[2] The central idea is simple. Between an event and an emotion there is usually a thought, an interpretation of what the event means. The 68 on the midterm did not by itself make the sophomore feel sick. His interpretation did: that the grade meant he would fail, lose his major and disappoint his family. A different interpretation of the same grade would produce a different emotion. The thought record is a structured way of catching the interpretation, writing it down and asking whether it holds up. Common thinking traps CBT practitioners have catalogued the typical ways automatic thoughts go wrong. Learning to recognize them makes a thought record much faster to complete, because many anxious thoughts turn out to be versions of the same few patterns. Among those most common in academic stress are the following: · Catastrophizing: jumping from a setback to the worst possible outcome. A 68 means I'll fail the course and my life will fall apart. · All-or-nothing thinking: seeing things in extremes with nothing in between. If I don't get an A, I've failed. · Fortune-telling: predicting the future as though it were certain. I know I'll blank on the final. · Mind-reading: assuming you know what others think, usually something negative. My professor thinks I'm stupid. · Overgeneralization: drawing a sweeping conclusion from a single event. I always mess up exams. · Discounting the positive: dismissing evidence that does not fit. I only did well on the last test because it was easy. · Labeling: turning a behavior into an identity. I'm a failure, rather than I did badly on this test. · Should statements: rigid rules about how things must be. I should be able to do this without help. Looking back at the sophomore's thoughts, nearly every trap appears: catastrophizing (fail the course, lose the major), fortune-telling (my parents are going to be disappointed), overgeneralization and labeling (I'm not smart enough for college). Recognizing these patterns does not make the thoughts disappear. But it makes them visible as patterns, habits of mind rather than accurate reports. Filling in a thought record A thought record is usually laid out in columns, each capturing one step in the process of examining a thought. Versions differ in their details; Table 4 shows a common structure, with the sophomore's entry as an example. Table 4. A thought record, with a worked student example. Column What to write Example entry Situation What happened, briefly and factually Got 68 on first stats midterm, Tuesday evening Emotions Feelings and their intensity, 0-100 Panic 85, shame 70 Automatic thought The thought behind the strongest feeling "I'm going to fail this course and lose my major" Evidence for Facts that support the thought 68 is below my target; stats is required for the major Evidence against Facts that do not fit the thought Midterm is 20% of grade; class average was 71; I got a B+ on homework Balanced thought A fairer summary of all the evidence "This was a weak start, but one midterm won't decide the course; I need to change how I study" Re-rating Emotions after writing, 0-100 Panic 40, shame 35 The most important columns are the two evidence columns. The aim is to be a fair judge, not a defense attorney. Evidence for the thought belongs on the page as much as evidence against it; if you leave out the real grounds for worry, the balanced thought will ring false, and you will not believe it. The 68 really is below where the sophomore wanted to be, and statistics really is required. Those facts deserve a place. Several questions help fill in the evidence against: · What would I say to a friend who had this thought? · Have I been in a similar situation before? What happened? · Is there anything I'm ignoring because it doesn't fit? · What is the worst that could realistically happen, what is the best, and what is most likely? · Even if the worst happened, how would I cope? · Is this thought helping me, or just hurting? The balanced thought is not a positive affirmation. "I'm going to ace this course" would be as unfounded as "I'm going to fail it." A good balanced thought is one you can believe, and it often includes a next step. The sophomore's balanced thought acknowledges the weak start and points toward action: changing how he studies. That, in turn, connects the thought record to the planning of Chapter 4. He might go on to write: Thursday: go to TA office hours with the two problems I got wrong. Try doing practice problems instead of rereading notes. The re-rating at the end is a check on whether the exercise helped. It is common for emotions to fall substantially, but not to zero, after a thought record. That is a realistic outcome. A 68 is disappointing; the goal is to feel appropriately disappointed rather than catastrophically afraid. A shorter version for busy days A full thought record takes ten or fifteen minutes, which is not always available between classes. A compressed version, done in a notebook or on a phone, keeps the essential steps. Write three lines. First, the thought, word for word: I'm going to fail this course. Second, the single strongest piece of evidence against it: The midterm is only 20 percent of the grade. Third, one fairer sentence and one action: It was a weak start; I'll go to office hours Thursday. This takes two minutes, and it interrupts the spiral at the moment it starts rather than hours later. The short version works best once the full version has been practiced a few times, because the full version teaches the habit of looking for evidence on both sides. After a few weeks, many students find that they begin running a quick version in their heads, catching a catastrophic thought and asking, almost automatically, whether it is really true. That internalized habit is one of the main goals of the technique. The notebook is training for a skill that eventually works without it. Testing thoughts in the world A thought record examines a thought on paper. Sometimes the evidence needed to judge a thought is not available on paper and has to be gathered. CBT calls this a behavioral experiment: a planned test of a prediction. Some research comparing the two suggests that behavioral experiments can produce stronger changes in belief than thought records alone, perhaps because direct experience is more convincing than reasoning.[3] For the sophomore, a behavioral experiment might target the thought My professor thinks I'm stupid, so I can't go to office hours. He writes the prediction down: If I go to office hours, the professor will be impatient and make me feel stupid; confidence I'm right, 80 percent. He goes. He writes down what happened: She spent fifteen minutes on the problems, said lots of people struggle with sampling distributions, and suggested a practice set. He writes what he learned: My prediction was wrong. Confidence in it now: 15 percent. Writing is essential to behavioral experiments, because memory is unreliable in exactly the direction anxiety needs. If the sophomore does not write down his prediction beforehand, he may later tell himself that he never really expected it to go badly, or that the professor was only being nice. The written prediction and the written outcome make the lesson hard to explain away. Scheduled worry time Thought records work well for specific, identifiable thoughts triggered by specific events. Some students have a different problem: not a single catastrophic thought but a diffuse, persistent worry that follows them through the day. They worry during lectures, while trying to read, while eating with friends and in bed. No single worry is examined because there is never a moment when worrying stops long enough to examine it. For this pattern, CBT offers a technique that sounds paradoxical: scheduling time to worry. The idea goes back to experiments by Thomas Borkovec and colleagues in the early 1980s, who asked chronic worriers to set aside a thirty-minute worry period at the same time and place each day and to postpone worries that arose at other times until that period.[4] The participants' daily worry decreased. Borkovec interpreted the result in terms of stimulus control: worry had become associated with so many times and places that almost anything could trigger it, and confining it to one time and place weakened those associations. The technique has since been studied under the name worry postponement. A 2023 meta-analysis by Dippel, Brosschot and Verkuil pooled seven randomized trials with 999 participants and found that worry postponement, practiced for between one week and a month, reduced both the duration and the frequency of worry in daily life compared with simply recording worries, with small effects (effect sizes of about 0.3 for duration and 0.2 for frequency).[5] The authors noted that long-term follow-up studies are still lacking. Earlier work by Jos Brosschot and Maria van der Doef had found that a simple worry reduction intervention of this kind was associated with fewer somatic health complaints among participants.[6] The procedure, adapted for students, runs as follows: 1. Choose a worry period. Pick a fixed daily time of fifteen to thirty minutes, ideally in the late afternoon or early evening. Avoid the hour before bed, since the aim is to keep worry away from sleep. 2. Choose a place. Use the same place each day, preferably not your bed or your main study desk, so those places are not associated with worry. 3. Postpone during the day. When a worry arises at another time, jot it down in a word or two, in a notebook or a note on your phone, and tell yourself you will deal with it at worry time. Then return your attention to what you were doing. This is harder than it sounds and takes practice; the goal is not to suppress the worry but to defer it. 4. Worry on schedule. At worry time, look at the list. Write about each worry. Many will have lost their urgency; it is common to look at a worry from the morning and find it no longer matters. For those that remain, sort them into problems you can do something about and problems you cannot. 5. Act or accept. For solvable problems, write a next step, as in Chapter 4. For unsolvable ones, such as what a recruiter thinks of you, whether it will rain on graduation day, or what the future holds in general, practice noting the worry and letting it be. A thought record may help with some of them. 6. Stop on time. When the period ends, stop, even if you have not finished. Leftover worries can wait until tomorrow's period. The written list is central to the technique. It gives the postponed worry a place to go, which makes it easier to let go of in the moment. And it provides a record over time. A student who looks back over two weeks of worry lists often notices that most of the things they worried about never happened, or happened and turned out to be manageable, which is itself a useful piece of evidence for the next thought record. Productive and unproductive worry Not all worry is a problem. Some worry is productive: it identifies a real problem that you can do something about and leads to action. Worrying that you have not started a paper due in three days is productive if it leads you to start the paper. Other worry is unproductive: it circles around problems that cannot be solved, or that cannot be solved now, or that are hypothetical. Worrying about whether you will get into graduate school, when you are a first-year student, is unproductive; there is nothing you can do today that resolves it. The psychologist Robert Leahy, in his self-help book The Worry Cure, suggests asking of any worry whether there is an action you can take on it today.[7] If there is, the worry is productive: take the action, or schedule it, and the worry has done its job. If not, the worry is unproductive, and the aim is to recognize it as such and let it go, rather than trying to solve it through more worrying. Writing helps with this distinction because it forces specificity. "I'm worried about my future" does not suggest an action. "I'm worried I won't have a summer internship" suggests several: check the career center's listings, ask a professor about research positions, update a resume. And "I'm worried that even if I get an internship I won't like it and I'll have chosen the wrong career" is, at least for now, a worry without an action. Written down, the difference is plain. When the thoughts will not budge Thought records and worry time are powerful tools, but they are not always enough. Some students find that no amount of evidence changes how a thought feels. They can write a balanced thought, agree with it intellectually and still feel the full force of the original fear. Others find that worry keeps spreading despite postponement, or that the effort of examining every thought becomes exhausting in its own right. These experiences are not failures. They are information. Persistent worry that does not respond to self-help techniques, that fills much of the day and that interferes with sleep, study or relationships may be a sign of generalized anxiety disorder, a common and treatable condition. Thoughts that are dark, heavy and resistant to evidence, especially thoughts of worthlessness or hopelessness, may be a sign of depression. And a need to check, repeat or neutralize thoughts until they feel right may be a sign of obsessive-compulsive patterns, for which analyzing thoughts can actually make things worse. In each of these cases, a counselor or therapist trained in CBT can offer what a notebook cannot: guidance, feedback, a relationship and a structured treatment. Chapter 8 discusses how to recognize when it is time to seek that help, and campus counseling centers are a good first stop. Chapter 7: Writing Kindly to Yourself After a Setback Imagine two students who receive the same email on the same afternoon: a rejection from a competitive summer research program each had worked hard to get into. The first student reads it, closes the laptop and begins an internal monologue that runs for the rest of the evening. Of course they didn't want me. My application was mediocre. Everyone else in the lab got something this summer. I'm lazy and I'm not good enough, and I should have known better than to apply. The second student feels the same sharp disappointment. But her internal voice sounds different. That really hurts. I wanted this badly and I worked hard on it. Most people who apply don't get in; it's a tough program. I'm allowed to be upset tonight. Tomorrow I'll ask the director whether there's feedback on my application. Most people assume the first student is being hard on himself for a reason: that harsh self-criticism, however painful, keeps standards high and drives improvement, and that the second student risks letting herself off the hook. The research on self-compassion suggests that assumption is largely mistaken. This chapter explains why, and describes a form of writing designed to help students respond to failure the way the second student did. The harsh inner critic Academic life delivers a steady supply of evaluation: grades, feedback, rankings, acceptances, rejections. For many students, each negative evaluation triggers a wave of self-criticism, a voice that interprets the setback as proof of personal inadequacy. That voice can be relentless, and it is often much harsher than anything the student would say to anyone else. There are reasons to think this pattern has become more common. A 2019 meta-analysis by Thomas Curran and Andrew Hill examined perfectionism scores among more than 41,000 American, Canadian and British college students who completed the same questionnaire between 1989 and 2016.[1] All three forms of perfectionism it measured increased over that period. The largest increase was in what psychologists call socially prescribed perfectionism: the belief that others expect you to be perfect and will judge you harshly if you are not. Students increasingly feel that the people around them demand flawlessness, and many turn that perceived demand into internal criticism. Self-criticism has a certain logic. It seems to promise protection: if I criticize myself first and hardest, I will fix my flaws before anyone else notices them, and I will never become complacent. But research on self-criticism generally finds that it is associated with more depression, more anxiety and more procrastination, not with better performance. The inner critic does not motivate so much as frighten, and a frightened mind, as earlier chapters described, is a less capable one. It also tends to make failure so painful that students avoid situations where they might fail, which means avoiding the very challenges that would help them grow. What self-compassion is, and what it is not The psychologist Kristin Neff, whose work did much to bring self-compassion into research psychology, describes it as having three components.[2] The first is self-kindness: treating oneself with warmth and understanding in the face of suffering or failure, rather than with harsh judgment. The second is common humanity: recognizing that failure and imperfection are part of the shared human experience rather than signs that something is uniquely wrong with oneself. The third is mindfulness: holding painful thoughts and feelings in balanced awareness, neither suppressing them nor being swept away by them. In the example that opens this chapter, the second student shows all three. "That really hurts" is mindful acknowledgment. "Most people who apply don't get in" is common humanity. "I'm allowed to be upset tonight" is self-kindness. And then, notably, she moves toward action: asking for feedback. It is worth being equally clear about what self-compassion is not. It is not self-pity, which tends to magnify suffering and isolate the sufferer; common humanity works in the opposite direction. It is not self-esteem, which depends on evaluating oneself positively and therefore tends to collapse precisely when one fails; self-compassion is available whether one has succeeded or not. And it is not self-indulgence or lowering standards. One can hold high standards and still treat oneself decently when falling short of them, in the same way a good coach holds an athlete to high standards without insulting them after a loss. Self-compassion is strongly associated with mental health. A 2012 meta-analysis by Angus MacBeth and Andrew Gumley found a large negative association between self-compassion and symptoms of depression, anxiety and stress: people higher in self-compassion reported fewer of these symptoms.[3] That finding is correlational, and it cannot show that self-compassion causes better mental health. The experimental research is more informative on that question. Does self-compassion make you lazy? The most common objection to self-compassion, especially among ambitious students, is that it will sap motivation. If I am kind to myself after failing, why would I work harder next time? A series of experiments by Juliana Breines and Serena Chen, published in 2012, tested this objection directly.[4] In one experiment, undergraduates took a difficult vocabulary test on which nearly everyone did poorly. Afterward, they were randomly assigned to one of three conditions. In the self-compassion condition, they read a short message encouraging them to be understanding about the difficulty of the test and reminding them that many students find such tests hard. In the self-esteem condition, they read a message reassuring them that they must be intelligent to have gotten into their university. In the control condition, they received no message. All participants were then told they would take a second test and were given as much time as they wanted to study a list of words. Students in the self-compassion condition studied longer than those in either of the other conditions. Far from making them complacent, a kind response to failure made them more willing to put in effort to improve. In other experiments in the same paper, participants who took a self-compassionate view of a personal weakness were more likely to believe it could be changed and more motivated to change it, and participants who wrote self-compassionately about a past wrongdoing were more motivated to make amends and avoid repeating it. The explanation Breines and Chen offer is intuitive. When failure is devastating, people protect themselves from it: they avoid thinking about it, make excuses or disengage from the task. When failure is survivable, people can look at it squarely and learn from it. Self-compassion makes failure survivable. Related research by Mark Leary and colleagues found that people who were prompted to think about a past failure in self-compassionate terms, through a writing exercise, experienced less negative emotion about it than those given other prompts, while still taking responsibility for their role in what happened.[5] Self-compassion, in other words, did not function as an excuse. It allowed people to own their mistakes without being crushed by them. A 2019 meta-analysis by Madeleine Ferrari and colleagues pooled 27 randomized controlled trials of self-compassion interventions and found improvements across a range of outcomes, including self-criticism, rumination, stress, anxiety and depressive symptoms.[6] Many of those interventions were structured programs lasting weeks rather than single writing exercises, so the results do not translate directly to a notebook. But they support the idea that self-compassion is a skill that can be learned, and that learning it helps. The self-compassion letter The most common form of self-compassion writing is a letter to oneself. The exercise has several versions, and Neff describes one in her book Self-Compassion.[2] The core instruction is to write about a failure, flaw or painful situation from the perspective of a friend who is unconditionally kind, accepting and wise, someone who sees your struggle clearly and cares about you anyway. For a student, the exercise might run like this: 1. Name the situation. In a sentence or two, write down what happened and what you are telling yourself about it. I got rejected from the research program. I keep thinking I'm not good enough and I was stupid to apply. 2. Imagine the friend. Picture a friend, real or imagined, who is warm and wise and knows you well. Some students picture a grandparent, a mentor or simply the kind of friend they try to be for others. 3. Write the letter. Write to yourself from that friend's perspective, in the second person or using your own name. What would they say about the situation? How would they respond to the harsh thoughts? What would they want you to know? 4. Include the three components. A good letter usually acknowledges the pain, places it within the common human experience and offers kindness. It may also gently point toward a next step, as a good friend would. 5. Read it later. Set the letter aside and read it again after a few hours or the next day. A student's letter might read, in part: Dear Sam, I know you're hurting. You wanted this and you put real effort into that application; of course it stings. But getting rejected from a program that takes a tiny fraction of applicants doesn't mean you're not good enough. It means you aimed high. Plenty of people you admire got turned down for things at your stage. It's okay to feel disappointed tonight. When you're ready, it might help to ask for feedback. You're going to keep growing, and this is part of it. Writing this kind of letter feels awkward to many people at first. Students who have spent years with a harsh inner critic may find it genuinely difficult to write kindly to themselves, and the first attempts may feel false. That awkwardness is common, and it tends to fade with practice. It can help to start by writing what you would say to a friend in the same situation, then changing the name. The power of stepping back One reason the letter works may be that it changes the writer's position. Instead of experiencing the failure from the inside, the student is looking at it from the outside, as a friend would. Research on self-distancing suggests that this shift in perspective is itself helpful. In a set of studies published in 2014, Ethan Kross and colleagues asked people to reflect on their feelings before a stressful task, such as giving a speech with little preparation time. Some were asked to use first-person pronouns ("Why am I feeling this way?"). Others were asked to use their own name or non-first-person pronouns ("Why is Sam feeling this way?"). Those who used their own name or "you" felt less anxious and less ashamed after the speech, performed better according to independent raters and engaged in less rumination afterward.[7] Referring to oneself by name, the researchers argued, creates a small psychological distance that allows people to reason about their situation more as they would reason about someone else's. This finding suggests a simple adjustment to many forms of journaling. When writing about something painful, try writing about yourself in the third person or addressing yourself as "you." Sam is disappointed about the rejection. What would help Sam right now? The shift feels odd at first, but many people find it creates just enough distance to think clearly without denying the feeling. Forgiving yourself for procrastinating One of the most common sources of student self-criticism is procrastination. A student delays starting a paper, ends up writing it in a panicked all-nighter, turns in something weaker than they could have written and then spends days berating themselves for it. The self-criticism feels like the appropriate response, a way of making sure it never happens again. And then, a few weeks later, it happens again. A study by Michael Wohl, Timothy Pychyl and Shannon Bennett followed first-year university students across two midterm exams.[8] Students reported how much they had procrastinated in studying for the first midterm and how much they had forgiven themselves for it. The students who had forgiven themselves more for procrastinating on the first exam procrastinated less when studying for the second. The researchers' interpretation was that self-forgiveness reduced the negative feelings associated with the first exam, and that those feelings, left unresolved, would otherwise have made students want to avoid thinking about the next one. Avoidance is the engine of procrastination; guilt and shame feed it. The study was correlational, so it cannot rule out other explanations. But it fits well with the Breines and Chen experiments, and it describes a pattern many students will recognize. The student who spends the week after a bad exam feeling terrible about their study habits is often the student who delays starting to study for the next one, because opening the textbook brings the bad feelings back. A short writing exercise can break the cycle. After a bout of procrastination, write a few sentences that acknowledge what happened honestly, without minimizing it: I put off the paper until the last night and it wasn't my best work. Then add a sentence of common humanity and kindness: Procrastinating on hard tasks is incredibly common, especially when I'm anxious about them. I'm not a bad person for doing it. Then add a specific, small plan for next time, in the style of the implementation intentions described in Chapter 4: For the next paper, I'll write the first paragraph on Monday afternoon, even if it's rough. The combination is the point. Self-forgiveness without a plan can drift into excuse-making; a plan without self-forgiveness is made by someone who is still too ashamed to look at the problem. Self-compassion in a competitive culture Self-compassion can be a hard sell in environments that prize achievement. Students in pre-medical programs, in competitive engineering and computer science majors, at selective universities and in performing arts programs often absorb the message that toughness and self-criticism are the price of success. Some worry that becoming kinder to themselves will cost them their edge. The research suggests the opposite, but it is worth taking the worry seriously. Self-compassion does not ask students to lower their goals. It asks them to change how they respond when they fall short. A student who treats each failure as a catastrophe will eventually burn out, avoid risks or quit; a student who can fail, feel the sting, learn and try again is far more likely to sustain the effort that high achievement requires. The most successful people in demanding fields often describe exactly this capacity: not an absence of failure, but a way of recovering from it. There is also something to be said for the effect of self-compassion on others. Students who are harsh with themselves are often, quietly, harsh with others too, and highly self-critical environments can become corrosive for everyone in them. Students who learn to respond to their own failures with understanding may find it easier to respond to others' failures the same way, which makes them better friends, lab partners and eventually colleagues. When kindness is hard to find For some students, self-compassion writing brings up more than awkwardness. A student who has experienced abuse, neglect or chronic criticism from important people in their life may find that trying to write kindly to themselves brings up grief, anger or a sense of unworthiness that feels overwhelming. The inner critic, in these cases, is often an echo of real voices, and softening it is not a matter of a few letters. If self-compassion writing consistently leaves you feeling worse, not in the passing way of touching something tender but in a way that lingers for days, it is worth stopping and talking to a counselor. Therapies specifically designed for high self-criticism and shame, such as compassion-focused therapy, exist for exactly this reason, and a counselor can help decide what is right. As with every other method in this booklet, the writing is a tool, not a test, and struggling with it says nothing about your worth. Chapter 8: When Journaling Backfires Most books about journaling are written by enthusiasts, and they tend to present writing as unambiguously good for you. It is not. The same research that shows modest benefits for many people also shows that some kinds of writing, done by some people in some states of mind, can make things worse. A student who keeps a journal faithfully every night and feels steadily more miserable is not doing it wrong in some small technical way. They may be using a tool that is actively amplifying the problem. This chapter describes how that happens, how to recognize it and what to do instead. It also covers the larger question that sits behind every chapter of this booklet: when a notebook is no longer enough, and it is time to talk to someone. Rumination on paper The main way journaling backfires is by becoming rumination. Rumination is repetitive, passive thinking about one's distress: its causes, its meaning, its consequences. The psychologist Susan Nolen-Hoeksema spent much of her career studying it, and her research showed that people who respond to low mood by ruminating tend to become more depressed, stay depressed longer and are more likely to develop depression in the future than people who respond in other ways.[1] Rumination feels like problem-solving, which is why people do it. It rarely solves anything. Later research distinguished two components of rumination. Brooding is a moody, passive comparison of one's situation with some standard that has not been met: Why do I always react this way? Why can't I handle things better? What is wrong with me? Reflection is a more neutral, purposeful turning inward to understand and solve problems. Brooding is more consistently linked to worse outcomes; reflection is more mixed, and in some studies is associated with less depression over time.[2] The psychologist Edward Watkins has identified a related distinction between two ways of thinking about a problem.[3] Abstract, evaluative thinking asks why things happen and what they mean: Why did I fail? What does this say about me? Concrete, process-focused thinking asks how things happened and what to do: What exactly happened in the hour before the exam? What would I do differently? Abstract thinking about negative events tends to be unhelpful for people who are already distressed, while concrete thinking tends to be helpful. Abstract "why" questions invite answers about one's character; concrete "how" questions invite answers about one's actions. A journal is an ideal place to ruminate. It is private, it is always available, and it can be returned to night after night. A student who writes every evening some version of Why am I like this? Why can't I be happy? Why does everything go wrong for me? is not processing their experience. They are rehearsing it, and writing gives the rehearsal a permanence it would not otherwise have. Reading back over old entries of the same kind can extend the rehearsal further. What the research shows about writing that hurts Several studies have documented conditions under which writing does not help or makes things worse. The clearest example comes from research by David Sbarra and colleagues on adults who had recently gone through a marital separation.[4] Participants were randomly assigned to traditional expressive writing, a narrative version of expressive writing or a control writing task. For most participants, the writing condition made little difference. But among participants who were actively engaged in a search for meaning about their separation, who were, in effect, already ruminating about why it had happened, both forms of expressive writing led to worse emotional outcomes months later than control writing. For people already caught in the question of why, being asked to write deeply about it seemed to deepen the trap. The study was of adults going through divorce, not students, and it should not be over-generalized. But it has an obvious counterpart in student life. A student who has just been through a painful breakup, a friendship that ended badly, or an academic failure that feels like a verdict on their worth, and who is already spending hours asking why, may not benefit from being told to write about it more. They may benefit more from writing that is concrete and forward-looking, or from not writing about it at all for a while. Chapter 1 described the study by Ullrich and Lutgendorf in which students who journaled only about their emotions, without trying to understand the event, reported more illness symptoms than those who combined thoughts and feelings.[5] And the meta-analytic evidence reviewed in Chapter 2 shows that expressive writing reliably increases distress in the short term. For most people this short-term distress is temporary and unrelated to later outcomes. For someone whose mood is already fragile, a string of evenings that each end with a spike in distress may not be temporary at all. Warning signs How can a student tell whether their journaling has tipped from processing into rumination? No single sign is decisive, but several patterns are worth watching for. Table 5 lists common warning signs and what to try instead. Table 5. Signs that journaling has become rumination, and what to try instead. Warning sign What it may mean What to try instead The same themes, in the same words, week after week Rehearsing, not processing Switch to a thought record or a concrete "how" question Mostly "why" questions about yourself Abstract, evaluative brooding Ask what happened and what you could do next Mood reliably worse after writing, for hours or days The method is amplifying distress Stop, change method, or take a break Writing replaces talking to people Isolation Tell a friend or counselor one thing from the journal Rereading old entries to relive them Rumination extended Stop rereading; consider discarding pages Writing late at night that delays sleep Worry at bedtime Move writing earlier; use a short to-do list at bedtime The third row deserves particular attention. Feeling somewhat worse right after writing about something painful is normal, and it usually passes within an hour or so. Feeling worse for the rest of the evening, sleeping badly and waking up still heavy, and having that happen repeatedly, is a sign that the writing is not working. What to do instead When journaling becomes rumination, there are several options, and choosing among them depends on the student and the situation. Change the question. Shifting from abstract "why" questions to concrete "how" and "what next" questions is often enough to turn unproductive writing into productive writing. Instead of Why do I always fall apart in exams?, write What exactly happened in the first five minutes of the exam? What is one thing I could try differently next time? Change the method. A student who has been doing open-ended expressive writing and finds it circling may do better with a more structured method: a thought record (Chapter 6), which forces the evidence for and against a thought into view; worry time (also Chapter 6), which contains worry rather than letting it spread; or a self-compassion letter (Chapter 7), which changes the voice in which the writing speaks. Change the perspective. Writing about oneself in the third person or as "you," as described in Chapter 7, can create enough distance to break the circling. Change the timing. Writing about painful things late at night, when the mind is tired and defenses are low, tends to go worse than writing earlier in the day. Move the writing to the afternoon, and keep the bedtime list short and practical. Stop writing for a while. Sometimes the best response is to put the journal away. Distraction has a bad reputation, but for someone caught in rumination, absorbing activity (exercise, time with friends, a game, a film, cooking) can interrupt the cycle in a way that more thinking cannot. Nolen-Hoeksema's research found that distraction tends to relieve depressed mood in the short term, which can create the space needed to address problems later with a clearer head.[1] Talk to someone. A journal is a conversation with oneself, and there are limits to what that conversation can accomplish. Telling another person, a friend, a family member, a mentor or a counselor, adds something writing cannot: an outside perspective and the experience of being heard. A journal that turned Consider a junior who started journaling in the fall after a friend recommended it. At first it helped. He wrote a few times a week about classes, his part-time job and a new relationship, and he liked the sense of order it gave him. In November the relationship ended, abruptly and without much explanation. He began writing every night, often for an hour or more, usually after midnight. The entries changed. Early ones described what had happened and how he felt. Later ones circled the same questions: Why did she leave? What did I do wrong? Why do I always push people away? What is wrong with me that nobody stays? He reread old entries looking for clues. He wrote less about classes, and then stopped going to some of them. He told himself the writing was helping him process things, but he went to bed later and more upset each week, and he had stopped mentioning the breakup to friends because he felt he had already said everything to the page. What finally made him notice was a practical problem. An adviser emailed about missed assignments, and when he sat down to write about that, he found himself three paragraphs later writing again about the breakup and what it said about him. Looking back over two weeks of entries, he saw that nearly all of them asked the same questions in almost the same words. None of them had produced an answer, and none had led to anything he did differently. He made three changes. He stopped writing after ten at night and kept only a short to-do list at bedtime. He replaced his open-ended evening entries with a fifteen-minute block in the late afternoon, and when the breakup came up he tried writing about it in the third person and asking concrete questions: What does he need this week? Who could he talk to? And he told a close friend, over dinner, some of what he had been writing, which was the first time he had said it aloud. A few weeks later, when his mood still had not lifted, he made an appointment at the counseling center. This junior is a composite, and not every story turns so cleanly. But the pattern is common: a practice that helped at first gradually changes character under pressure, and the change is hard to see from inside because the writing still feels like work on the problem. The signs in Table 5 are one way to see it from outside. So is the simplest test of all: after several weeks of writing, is anything different? Special cautions A few situations call for particular care. Trauma. Students who have experienced serious trauma, such as sexual assault, violence or a life-threatening event, may find that writing about it in detail on their own brings on intrusive memories, nightmares or overwhelming distress. Writing can be part of effective trauma treatment: a structured approach called written exposure therapy, in which people write about a trauma in five guided sessions with a therapist, was found in a randomized trial to be about as effective for post-traumatic stress disorder as a longer standard therapy.[6] But in that approach the writing takes place within a therapeutic relationship, with preparation and support. A student carrying trauma should be cautious about using expressive writing alone on the trauma itself, and should consider reaching out to a counselor or campus sexual assault resource center, many of which offer confidential support. Eating and body concerns. Food diaries, calorie logs and body-focused journaling can feed eating disorders, and some forms of tracking are themselves symptoms. A student who finds their journal filling with records of what they ate, what they weighed or how their body looks should treat that as a signal to seek help rather than as a coping strategy. Obsessive thinking. For students with obsessive-compulsive tendencies, writing can become a form of checking or reassurance-seeking, in which the student writes and rewrites a worry trying to reach certainty that never comes. In this case more analysis is usually counterproductive, and a therapist trained in exposure and response prevention can help. Self-harm and suicidal thoughts. A journal is not a safe place to hold thoughts of suicide or self-harm alone. If you notice yourself writing about wanting to die, wanting to hurt yourself or feeling that others would be better off without you, that is a clear signal to reach out to someone now, as described below. When to seek counseling Journaling is a self-help tool, and self-help has limits. Some signs suggest that it is time to talk to a professional, most often at a campus counseling center, a student health service, a private therapist or a primary care provider. These include: · Low mood, hopelessness or loss of interest in things you usually enjoy that has lasted more than two weeks. · Anxiety or worry that fills much of the day and does not respond to the methods in this booklet. · Changes in sleep, appetite or energy that are persistent and not explained by a single busy week. · Difficulty attending classes, completing work or keeping up with daily tasks. · Panic attacks, or avoiding places and situations because of fear. · Using alcohol, cannabis or other substances to cope with stress. · Pulling away from friends and family. · Anything in your journal that frightens you when you read it back. Many students hesitate because they are not sure their problem is serious enough. Counseling centers generally want students to come in before problems become severe, not after, and most offer an initial consultation to help decide what kind of support fits. Many also offer groups, workshops, drop-in hours and after-hours phone lines. Some students worry about confidentiality; campus counseling records are generally kept separate from academic records, and counselors can explain the limits of confidentiality in the first session. A journal can be useful in counseling. Some students bring entries, or notes drawn from them, to help explain what they have been experiencing. A counselor may also suggest specific forms of writing, such as thought records, as part of treatment, which can build directly on the habits described in this booklet. When to seek help now Some situations call for immediate help rather than an appointment next week: · Thoughts of suicide, especially with a plan or intent. · Urges to harm yourself or recent self-harm. · Feeling unable to keep yourself safe. · Seeing or hearing things others do not, or feeling that reality is slipping. · Being in danger from someone else. In the United States, you can call or text 988 at any time to reach the 988 Suicide and Crisis Lifeline, which offers free and confidential support. The Crisis Text Line is available by texting HOME to 741741. In an emergency, call 911 or go to the nearest emergency department. Most campuses have after-hours counseling lines and campus police or security who can respond quickly. Outside the United States, local emergency numbers and national crisis lines serve the same role. For friends, family and staff: if someone shares writing that mentions suicide or self-harm, take it seriously. Ask them directly and calmly whether they are thinking about suicide; asking does not put the idea in their head. Stay with them if the risk seems immediate, and help them connect with 988, a counselor or emergency services. Respecting a student's privacy is important, but not at the expense of their safety. Journaling at its best helps students understand their own minds. Part of that understanding is recognizing when a mind needs more help than a notebook can give. Knowing when to close the journal and pick up the phone is not a failure of the practice. It may be the most important thing the practice teaches. Chapter 9: Keeping It Going, Keeping It Private Almost every student who reads about journaling feels a flicker of resolve. They buy a notebook, write faithfully for a week or two and then, somewhere around the first round of midterms, stop. The notebook ends up under a stack of papers, a few entries long, and it becomes one more small reminder of something they meant to do and did not. That pattern is not a character flaw. It is what happens to most new habits, especially ones started in a burst of enthusiasm and dropped into an already crowded life. And it is not necessarily a problem: some methods in this booklet, such as pre-exam writing and the bedtime to-do list, are tools to pick up when needed rather than daily practices. But for students who want writing to become a reliable part of how they handle stress, it helps to understand how habits actually form, and to set things up so that the practice survives the weeks when it is most needed and hardest to maintain. This chapter also takes up a question that journaling guides often neglect: privacy. A journal is only useful if it is honest, and it is only honest if the writer trusts that no one else will read it. For students living in shared rooms, using shared devices and relying on apps whose business models they may not understand, that trust has to be earned by some deliberate choices. How habits form The most widely cited study on how long habits take to form was published in 2010 by Phillippa Lally and colleagues at University College London.[1] They asked 96 volunteers to choose a new eating, drinking or exercise behavior and to perform it daily in the same context, such as drinking a glass of water after breakfast. Participants reported each day how automatic the behavior felt. On average, automaticity rose steeply at first and then leveled off, and the median time to reach near-maximum automaticity was 66 days. But the range was enormous, from 18 days to 254, and some participants never reached a plateau at all within the study. More complex behaviors tended to take longer than simple ones. Two findings from this study are especially useful for students. First, habits take longer to form than the popular "21 days" suggests, so a student who has written for three weeks and still has to remind themselves has not failed; they are in the middle of a normal process. Second, missing a single day did not significantly derail habit formation. What mattered was returning to the behavior, not an unbroken streak. A student who misses three days during a busy week and then picks the journal back up is still building the habit. The study also points to the importance of context. Participants performed their behaviors in a consistent situation, and that consistency is what allows a behavior to become automatic: the situation itself begins to prompt the behavior. For journaling, this means choosing a specific cue, a moment in the day that already happens reliably, and attaching the writing to it. Examples include right after dinner, when you first sit down at your desk in the evening, after brushing your teeth or on the bus ride home. The more reliable the cue, the faster the habit. This is the same logic as the implementation intentions described in Chapter 4. "I will journal more" is a goal. "After I get back from dinner on weekdays, I will write for ten minutes before I open my laptop" is a plan with a cue, and it is much more likely to happen. Making it easy The biggest threat to a writing habit is friction: the small obstacles that make it slightly harder to start. If the notebook is in a drawer, the pen is missing, and the student feels they have to write a full page of meaningful reflection, the habit will lose to almost any competing demand. Reducing friction is often more effective than increasing motivation. A few principles help: · Keep the tools where the cue is. If you write after dinner at your desk, keep the notebook and pen on the desk. If you write at bedtime, keep them by the bed. · Lower the minimum. Decide in advance that one sentence counts. On a hard day, one sentence is enough to keep the habit alive, and it often turns into more once you start. · Have prompts ready. A blank page can be intimidating. A short list of prompts inside the cover of the notebook, drawn from the methods in this booklet, removes the question of what to write. What is on my mind right now? What went well today? What am I worried about, and is there anything I can do about it? What would I tell a friend in my situation? · Match the method to the day. Not every entry has to be the same kind of writing. Use the map in Chapter 1: a brain dump on a crowded day, a thought record after a setback, a gratitude entry on a flat day. · Don't reread unless it helps. Some people find rereading useful for spotting patterns; others find it drags them back into old feelings. Neither is required. A semester rhythm Academic stress is not constant. It follows the calendar, and so can journaling. The methods that help most in the first weeks of a semester are often different from those that help during finals, and planning for that rhythm can make writing more useful and more sustainable. Table 6 sketches one way to think about it. Table 6. A semester rhythm for journaling. Point in semester Common stresses Methods that often fit First weeks Adjustment, loneliness, uncertainty Expressive writing about the transition; gratitude letter home Early steady weeks Building routines, early assignments Weekly brain dump; brief good-things log Midterms Exam dread, sleep disruption, first grades Pre-exam writing; bedtime to-do list; thought records After first grades Disappointment, self-criticism Self-compassion letter; thought records; worry time Late semester Accumulated deadlines, fatigue Triage brain dump; bedtime list; short entries Finals and after Peak pressure, then release Pre-exam writing; after finals, a reflective entry on the term The last row points to a practice many students find valuable: a short reflective entry at the end of each term. What was hard? What helped? What would I do differently? Which methods did I actually use? Read at the start of the next term, a few paragraphs like this can be more useful than any general advice, because they are specific to the writer's own experience. Choosing tools Journaling can cost nothing, and nothing in the research requires expensive materials. A cheap spiral notebook and a pen are enough. That said, different formats suit different people and different methods. Paper has real advantages. It is slower than typing, which suits reflective writing. It has no notifications, no internet and no temptation to check something else. It does not glow, which matters at bedtime. It can be torn up. And it cannot be hacked, synced to a cloud or mined for advertising data. Its disadvantages are that it can be found and read by others and cannot be searched. A notes app or word processor is fast, always with you on your phone and easy to search. It suits brain dumps and to-do lists especially well. Its disadvantages are distraction and, as the next section discusses, privacy. Dedicated journaling apps offer prompts, reminders, mood tracking and sometimes encryption. Some are thoughtfully designed. But students should look carefully at who makes the app, what it costs over time and what it does with the data. Index cards or a small pocket notebook work well for pre-exam writing and for jotting postponed worries during the day, as described in Chapter 6. Many students end up using more than one: a paper notebook for reflective writing and a phone for lists and quick notes. The point is to choose what makes it easiest to write honestly. Keeping a journal private The benefits of expressive writing depend on honesty, and honesty depends on privacy. In the research, participants were typically told their writing would be kept confidential or that they could keep or destroy it themselves. A student who suspects a roommate, partner or parent might read their journal will, consciously or not, write for that reader. The writing becomes a performance, and much of its value is lost. For paper journals, the practical risks are other people in shared spaces. A few measures help: keep the notebook somewhere not obvious, take it with you when you travel home, and remember that you do not need to keep what you write. The benefit of expressive writing comes from the act of writing, not from the archive. Many people write about painful subjects on loose paper and then shred or discard it. This is not wasteful; it can be a deliberate part of the practice, as in the classroom where students crumpled up their pre-exam writing and threw it away. For digital journals, the risks are less visible. Documents stored in cloud services sync across devices, and a journal written on a laptop may appear on a shared family tablet. Files stored in accounts provided by a university or an employer are generally governed by that institution's policies, so personal journals are better kept in personal accounts. Phones left unlocked, shared passwords and saved logins can all expose private writing. A passcode on the device and on the journal app, and care about which accounts a journal lives in, go a long way. Apps deserve particular scrutiny. Mental health and wellness apps collect some of the most sensitive data a person can generate, and not all of them treat it carefully. In 2022, the Mozilla Foundation reviewed the privacy practices of popular mental health and prayer apps and found that many of them fell short of basic privacy standards.[2] In 2023, the US Federal Trade Commission took action against the online counseling company BetterHelp, alleging that it had shared consumers' sensitive health information with advertising platforms after promising to keep it private; the company agreed to pay $7.8 million to settle the charges.[3] These cases do not mean every app is untrustworthy. They mean that students should read the privacy policy of any app they use for journaling, prefer apps that store data on the device or offer end-to-end encryption, and be cautious about apps that are free and supported by advertising. The same caution applies to AI chat tools, which some students now use to talk through stress or to help them journal. Such tools can be useful for organizing thoughts or generating prompts, but conversations may be stored by the provider and, depending on settings, used in various ways. They are also not therapists, and they cannot assess risk or provide care in a crisis. Students who use them should check the privacy settings, avoid sharing identifying details they would not want stored and treat them as a supplement to, not a replacement for, people and professional help. Running your own experiment The research described in this booklet reports averages across groups of people. Whether a particular method helps a particular student is a question the research cannot answer. The student can, with a little structure. The simplest approach is to treat each method as an experiment with a defined trial period. Choose one method that fits a current problem. Decide in advance how long to try it: two weeks is reasonable for most methods, or three or four exams for pre-exam writing. Before each session, rate how you feel on a simple scale from 0 to 10, where 0 means no distress and 10 means the worst you can imagine. Rate it again a short while after the session. For methods aimed at sleep, note roughly how long it took you to fall asleep. For methods aimed at exams, note how you felt during the exam, not just the grade. At the end of the trial, look back. Did the ratings tend to fall after writing? Did the pattern change over the two weeks? Did anything else change, such as sleep, concentration or willingness to start difficult tasks? Did the method become easier or harder to do? This is not a scientific study; there is no control group, and a good or bad week for other reasons can swamp the effect. But it is far better evidence about your own experience than a vague impression, and it guards against two common errors: abandoning a method that was quietly helping because the effect was modest, and persisting with one that was quietly hurting because it felt like the right thing to do. Consider a second-year engineering student who tries three methods over a semester. Two weeks of nightly gratitude entries produce almost no change in her ratings; she finds them pleasant but beside the point. Two weeks of worry time produce a clear drop in how much she worries during lectures, although the sessions themselves are uncomfortable. Pre-exam writing before three exams produces mixed results: she feels calmer before one and no different before the other two. She keeps worry time as a regular practice, uses pre-exam writing occasionally, and sets gratitude aside for a future term when it might fit better. Her conclusions would not be the same as anyone else's, and that is the point. The research narrows the options to methods with some evidence behind them. The student's own trial decides which of those methods belong in their kit. Written ratings make that decision more honest, because memory tends to favor whatever we expected to find. For parents and staff Parents and staff who care about a student sometimes wonder whether they should read the student's journal, especially when they are worried. In almost every case, the answer is no. A journal read without permission is no longer private, and the student will likely stop writing honestly, or stop writing at all, and may lose trust in the person who read it. The better route is direct conversation: saying what you have noticed, asking how the student is doing and listening without rushing to fix things. The exception is imminent danger. If you have strong reason to believe a student is at risk of suicide or serious harm, safety comes first, and the steps described in Chapter 8 apply. But ordinary worry about a student's stress is best addressed by talking to them, not by reading what they wrote for themselves. Staff who encourage journaling, such as residence advisers, instructors who assign reflective writing and advisers, should be clear about what will and will not be read. Assigned reflective writing that will be graded or shared is a different kind of writing from a private journal, and students should know which is which. Instructors who use pre-exam writing in class, as some now do, can reassure students by letting them keep or discard their pages. When the habit fades Even with good planning, most students will go through stretches without writing. Sometimes that is because life got busy; sometimes because things got better and the need faded; sometimes because the method stopped fitting. None of these is a failure. The useful question is not Why did I stop? but Would writing help now? If the answer is yes, the way back is the same as the way in: a specific cue, a low minimum and a method that fits the current problem. A single sentence tonight is a restart. The notebook under the stack of papers is still there, and the pages that are already filled do not have to be read before a new one is started. Conclusion Go back to the student awake at one in the morning before a chemistry exam. After nine chapters, it is possible to say something more precise about what that student could do than "try journaling." If the trouble is a crowded mind, with the lab report and the exam and the text thread all competing for attention, the most useful fifteen minutes might be a quick brain dump that sorts the tasks from the worries, followed by a short, specific list for the morning. If the trouble is a single catastrophic thought, I'm going to fail and lose the scholarship, the most useful fifteen minutes might be a thought record that weighs the evidence and ends with a balanced thought and a next step. If the trouble is the argument with the friend, still raw, the best move might be to write it down briefly and schedule proper time for it tomorrow afternoon, or to set it aside entirely until after the exam. And in a few hours, just before walking into the exam room, ten minutes of writing about the fear itself might free up some of the working memory the exam will demand. None of these is the same act. Each fits a different problem. That is the first thing this booklet has tried to establish: journaling is not a single practice but a small set of distinct tools, and much of their value depends on choosing among them well. The second is that the evidence for each tool is real but modest. Expressive writing produces small average effects that shrink under the most careful analysis and barely touch depression. Pre-exam writing produced striking results in its original experiments and has not reliably reproduced in every classroom since. Gratitude writing improves well-being slightly but not anxiety. Worry postponement reduces worry by small amounts. The bedtime to-do list shortened sleep onset by about nine minutes in one laboratory study. Taken one at a time, none of these findings would justify the enthusiasm with which journaling is sometimes promoted. The case for a kit But taking them one at a time is not the right way to judge them, and this is the argument that only becomes visible once the chapters are laid side by side. A student's stress is not one problem. It is a shifting mix: the dread before an exam, the crowded list on a Sunday night, the sting of a rejection, the flat gray stretch in the middle of a semester, the fear that circles at night. Each individual method addresses a slice of that mix, with a modest effect on its slice. A student who has learned several methods, and who knows which slice each one addresses, has something different from any single intervention studied in any single experiment. They have a repertoire, and a way of matching it to what is happening. The skill that makes the repertoire work is the question at the center of Chapter 1: What kind of trouble am I in right now? It sounds simple. It is not. Most stressed students experience their stress as a single undifferentiated weight. Learning to break it into parts — this is a task, this is a worry about the future, this is a harsh judgment about myself, this is a painful memory, this is simply fatigue — is itself a significant change in how a person relates to their own mind. It is the difference between being inside a storm and being able to describe the weather. That skill is not measured in the meta-analyses, because no study has tried to teach it across a whole set of methods and track what happens. It may matter more than any individual effect size. A student who asks the question regularly is practicing, many times a semester, the step back from their own experience that every method in this booklet depends on. The research on self-distancing, on affect labeling and on the insight words that predicted benefit in Pennebaker's studies all point toward the same capacity: the ability to see one's own thoughts as thoughts, and one's own trouble as something with a shape. The notebook as an instrument There is a second argument that emerges only from reading the chapters together. Chapter 8 described the ways journaling can go wrong: rumination on paper, writing that rehearses distress rather than processing it, entries that reveal a problem larger than any notebook can handle. Those risks are real. But they have an unexpected consequence. A student who keeps a journal, and who has learned what healthy and unhealthy writing look like, is carrying an early-warning instrument. The same themes repeated week after week, a steady drift from "how" questions to "why" questions, mood reliably worse after writing, entries that frighten the writer when read back: these are not only signs that the journaling method needs changing. They are often among the first visible signs that something more serious is developing, whether depression, an anxiety disorder or a crisis. Most people have no record of how their mind has changed over a month. A journaling student does. Used honestly, that record can tell them, earlier than they would otherwise know, that it is time to talk to someone. This reframes the relationship between journaling and professional help. They are not alternatives, with the notebook serving as a cheap substitute for the counselor. The notebook is the first layer of a larger system: the thing a student does on an ordinary Tuesday night, which handles a real share of ordinary stress and which also tells them when the stress is no longer ordinary. The counselor, the physician and, in a crisis, the 988 Lifeline are the next layers. A student who uses the first layer well is more likely, not less, to reach the others at the right time. What this asks of the people around students For parents, advisers, residence staff and faculty, this has a practical implication. "You should try journaling" is well-meant advice that often fails, because it offers a single, vague practice for a varied set of problems and sets the student up to conclude that journaling does not work for them. Better advice is more specific: try writing a to-do list before bed, try ten minutes on your fears before the exam, try writing to yourself the way you would write to a friend. Better still is to teach the question behind the methods, and to be honest about the evidence, so that students approach each method as an experiment rather than a promise. It also asks institutions to see journaling as a complement to counseling capacity rather than a way of rationing it. A campus that teaches students to write their way through ordinary stress, and to recognize in their writing the signs that they need more, should expect more students to arrive at the counseling center, and earlier. That is a success, not a failure of the approach. A modest tool, used well The strongest case for journaling is not that it is powerful. It is that it is modest, free, private, always available and, for most students, safe, and that those qualities make it possible to use it often, to adjust it and to learn from it. A tool with a small effect that can be used a hundred times in a semester, well matched to the problem each time, may add up to more than a tool with a larger effect that is used once. The student at one in the morning does not need to become a different person. They need to know what to do in the next fifteen minutes, and then, over time, to understand their own patterns well enough to know what to do in the next fifteen minutes after that. A notebook will not carry the whole weight of a hard semester. But it can make the weight easier to see, easier to sort and easier to carry, and sometimes it can show the writer, clearly and early, that it is time to ask for help carrying it. Notes Introduction 1. Healthy Minds Network. The Healthy Minds Study: 2024-2025 Data Report. University of Michigan, UCLA, Boston University and Wayne State University, 2025. https://healthymindsnetwork.org/wp-content/uploads/2025/09/2024-2025HMS-National-Data-ReportStudent.pdf. Summary figures as reported by the University of Michigan School of Public Health, September 2025. Chapter 1: What Writing Does to a Worried Mind 1. Eysenck, M. W., Derakshan, N., Santos, R., and Calvo, M. G. "Anxiety and cognitive performance: Attentional control theory." Emotion 7, no. 2 (2007): 336-353. 2. Klein, K., and Boals, A. "Expressive writing can increase working memory capacity." Journal of Experimental Psychology: General 130, no. 3 (2001): 520-533. 3. Lieberman, M. D., Eisenberger, N. I., Crockett, M. J., Tom, S. M., Pfeifer, J. H., and Way, B. M. "Putting feelings into words: Affect labeling disrupts amygdala activity in response to affective stimuli." Psychological Science 18, no. 5 (2007): 421-428. 4. Masicampo, E. J., and Baumeister, R. F. "Consider it done! Plan making can eliminate the cognitive effects of unfulfilled goals." Journal of Personality and Social Psychology 101, no. 4 (2011): 667-683. 5. Pennebaker, J. W., Mayne, T. J., and Francis, M. E. "Linguistic predictors of adaptive bereavement." Journal of Personality and Social Psychology 72, no. 4 (1997): 863-871. 6. Pennebaker, J. W., Colder, M., and Sharp, L. K. "Accelerating the coping process." Journal of Personality and Social Psychology 58, no. 3 (1990): 528-537. 7. Ullrich, P. M., and Lutgendorf, S. K. "Journaling about stressful events: Effects of cognitive processing and emotional expression." Annals of Behavioral Medicine 24, no. 3 (2002): 244-250. Chapter 2: Expressive Writing and the Honest Size of Its Effects 1. Pennebaker, J. W., and Beall, S. K. "Confronting a traumatic event: Toward an understanding of inhibition and disease." Journal of Abnormal Psychology 95, no. 3 (1986): 274-281. 2. Pennebaker, Colder and Sharp, "Accelerating the coping process" (see Chapter 1, note 6). 3. Smyth, J. M. "Written emotional expression: Effect sizes, outcome types, and moderating variables." Journal of Consulting and Clinical Psychology 66, no. 1 (1998): 174-184. 4. Frisina, P. G., Borod, J. C., and Lepore, S. J. "A meta-analysis of the effects of written emotional disclosure on the health outcomes of clinical populations." Journal of Nervous and Mental Disease 192, no. 9 (2004): 629-634. 5. Frattaroli, J. "Experimental disclosure and its moderators: A meta-analysis." Psychological Bulletin 132, no. 6 (2006): 823-865. 6. Reinhold, M., Bürkner, P.-C., and Holling, H. "Effects of expressive writing on depressive symptoms: A meta-analysis." Clinical Psychology: Science and Practice 25, no. 1 (2018): e12224. 7. Rose, S., Bisson, J., Churchill, R., and Wessely, S. "Psychological debriefing for preventing post traumatic stress disorder (PTSD)." Cochrane Database of Systematic Reviews, no. 2 (2002): CD000560. 8. King, L. A. "The health benefits of writing about life goals." Personality and Social Psychology Bulletin 27, no. 7 (2001): 798-807. Chapter 3: Ten Minutes Before the Exam 1. Beilock, S. Choke: What the Secrets of the Brain Reveal About Getting It Right When You Have To. New York: Free Press, 2010. 2. Ramirez, G., and Beilock, S. L. "Writing about testing worries boosts exam performance in the classroom." Science 331, no. 6014 (2011): 211-213. Figures for the laboratory and classroom results as reported by the University of Chicago, January 2011. 3. Park, D., Ramirez, G., and Beilock, S. L. "The role of expressive writing in math anxiety." Journal of Experimental Psychology: Applied 20, no. 2 (2014): 103-111. 4. Myers, S. J., Davis, S. D., and Chan, J. C. K. "Does expressive writing or an instructional intervention reduce the impacts of test anxiety in a college classroom?" Cognitive Research: Principles and Implications 6 (2021). https://doi.org/10.1186/s41235-021-00309-x. 5. Shen, L., Yang, L., Zhang, J., and Zhang, M. "Benefits of expressive writing in reducing test anxiety: A randomized controlled trial in Chinese samples." PLOS ONE 13, no. 2 (2018): e0191779. 6. Doherty, J. H., and Wenderoth, M. P. "Implementing an expressive writing intervention for test anxiety in a large college course." Journal of Microbiology and Biology Education 18, no. 2 (2017). https://doi.org/10.1128/jmbe.v18i2.1307. 7. Jamieson, J. P., Mendes, W. B., Blackstock, E., and Schmader, T. "Turning the knots in your stomach into bows: Reappraising arousal improves performance on the GRE." Journal of Experimental Social Psychology 46, no. 1 (2010): 208-212. 8. Brooks, A. W. "Get excited: Reappraising pre-performance anxiety as excitement." Journal of Experimental Psychology: General 143, no. 3 (2014): 1144-1158. Chapter 4: Brain Dumps, To-Do Lists and the Bedtime Mind 1. Masicampo and Baumeister, "Consider it done!" (see Chapter 1, note 4). 2. Allen, D. Getting Things Done: The Art of Stress-Free Productivity. New York: Viking, 2001; revised edition, Penguin, 2015. 3. Scullin, M. K., Krueger, M. L., Ballard, H. K., Pruett, N., and Bliwise, D. L. "The effects of bedtime writing on difficulty falling asleep: A polysomnographic study comparing to-do lists and completed activity lists." Journal of Experimental Psychology: General 147, no. 1 (2018): 139-146. Average sleep-onset times as reported by the authors in public summaries of the study. 4. Gollwitzer, P. M., and Sheeran, P. "Implementation intentions and goal achievement: A meta-analysis of effects and processes." Advances in Experimental Social Psychology 38 (2006): 69-119. 5. Carney, C. E., and Waters, W. F. "Effects of a structured problem-solving procedure on pre-sleep cognitive arousal in college students with insomnia." Behavioral Sleep Medicine 4, no. 1 (2006): 13-28. Chapter 5: Gratitude Logs and the Uses of Good Things 1. Fredrickson, B. L. "The role of positive emotions in positive psychology: The broaden-and-build theory of positive emotions." American Psychologist 56, no. 3 (2001): 218-226. 2. Fredrickson, B. L., Tugade, M. M., Waugh, C. E., and Larkin, G. R. "What good are positive emotions in crises? A prospective study of resilience and emotions following the terrorist attacks on the United States on September 11th, 2001." Journal of Personality and Social Psychology 84, no. 2 (2003): 365-376. 3. Emmons, R. A., and McCullough, M. E. "Counting blessings versus burdens: An experimental investigation of gratitude and subjective well-being in daily life." Journal of Personality and Social Psychology 84, no. 2 (2003): 377-389. 4. Seligman, M. E. P., Steen, T. A., Park, N., and Peterson, C. "Positive psychology progress: Empirical validation of interventions." American Psychologist 60, no. 5 (2005): 410-421. 5. Mongrain, M., and Anselmo-Matthews, T. "Do positive psychology exercises work? A replication of Seligman et al. (2005)." Journal of Clinical Psychology 68, no. 4 (2012): 382-389. 6. Davis, D. E., Choe, E., Meyers, J., Wade, N., Varjas, K., Gifford, A., Quinn, A., Hook, J. N., Van Tongeren, D. R., Griffin, B. J., and Worthington, E. L. "Thankful for the little things: A meta-analysis of gratitude interventions." Journal of Counseling Psychology 63, no. 1 (2016): 20-31. 7. Lyubomirsky, S., Sheldon, K. M., and Schkade, D. "Pursuing happiness: The architecture of sustainable change." Review of General Psychology 9, no. 2 (2005): 111-131. 8. Lyubomirsky, S., Sousa, L., and Dickerhoof, R. "The costs and benefits of writing, talking, and thinking about life's triumphs and defeats." Journal of Personality and Social Psychology 90, no. 4 (2006): 692-708. 9. Toepfer, S. M., Cichy, K., and Peters, P. "Letters of gratitude: Further evidence for author benefits." Journal of Happiness Studies 13, no. 1 (2012): 187-201. 10. Smyth, J. M., Johnson, J. A., Auer, B. J., Lehman, E., Talamo, G., and Sciamanna, C. N. "Online positive affect journaling in the improvement of mental distress and well-being in general medical patients with elevated anxiety symptoms: A preliminary randomized controlled trial." JMIR Mental Health 5, no. 4 (2018): e11290. Chapter 6: Thought Records and Worry Time 1. Hofmann, S. G., Asnaani, A., Vonk, I. J. J., Sawyer, A. T., and Fang, A. "The efficacy of cognitive behavioral therapy: A review of meta-analyses." Cognitive Therapy and Research 36, no. 5 (2012): 427-440. 2. Greenberger, D., and Padesky, C. A. Mind Over Mood: Change How You Feel by Changing the Way You Think. 2nd ed. New York: Guilford Press, 2016. 3. Bennett-Levy, J. "Mechanisms of change in cognitive therapy: The case of automatic thought records and behavioural experiments." Behavioural and Cognitive Psychotherapy 31, no. 3 (2003): 261-277. 4. Borkovec, T. D., Wilkinson, L., Folensbee, R., and Lerman, C. "Stimulus control applications to the treatment of worry." Behaviour Research and Therapy 21, no. 3 (1983): 247-251. 5. Dippel, A., Brosschot, J. F., and Verkuil, B. "Effects of worry postponement on daily worry: A meta-analysis." International Journal of Cognitive Therapy (2024). https://doi.org/10.1007/s41811-023-00193-x. 6. Brosschot, J. F., and van der Doef, M. "Daily worrying and somatic health complaints: Testing the effectiveness of a simple worry reduction intervention." Psychology and Health 21, no. 1 (2006): 19-31. 7. Leahy, R. L. The Worry Cure: Seven Steps to Stop Worry from Stopping You. New York: Harmony Books, 2005. Chapter 7: Writing Kindly to Yourself After a Setback 1. Curran, T., and Hill, A. P. "Perfectionism is increasing over time: A meta-analysis of birth cohort differences from 1989 to 2016." Psychological Bulletin 145, no. 4 (2019): 410-429. 2. Neff, K. D. "Self-compassion: An alternative conceptualization of a healthy attitude toward oneself." Self and Identity 2, no. 2 (2003): 85-101; and Neff, K. Self-Compassion: The Proven Power of Being Kind to Yourself. New York: William Morrow, 2011. 3. MacBeth, A., and Gumley, A. "Exploring compassion: A meta-analysis of the association between self-compassion and psychopathology." Clinical Psychology Review 32, no. 6 (2012): 545-552. 4. Breines, J. G., and Chen, S. "Self-compassion increases self-improvement motivation." Personality and Social Psychology Bulletin 38, no. 9 (2012): 1133-1143. 5. Leary, M. R., Tate, E. B., Adams, C. E., Allen, A. B., and Hancock, J. "Self-compassion and reactions to unpleasant self-relevant events: The implications of treating oneself kindly." Journal of Personality and Social Psychology 92, no. 5 (2007): 887-904. 6. Ferrari, M., Hunt, C., Harrysunker, A., Abbott, M. J., Beath, A. P., and Einstein, D. A. "Self-compassion interventions and psychosocial outcomes: A meta-analysis of RCTs." Mindfulness 10, no. 8 (2019): 1455-1473. 7. Kross, E., Bruehlman-Senecal, E., Park, J., Burson, A., Dougherty, A., Shablack, H., Bremner, R., Moser, J., and Ayduk, O. "Self-talk as a regulatory mechanism: How you do it matters." Journal of Personality and Social Psychology 106, no. 2 (2014): 304-324. 8. Wohl, M. J. A., Pychyl, T. A., and Bennett, S. H. "I forgive myself, now I can study: How self-forgiveness for procrastinating can reduce future procrastination." Personality and Individual Differences 48, no. 7 (2010): 803-808. Chapter 8: When Journaling Backfires 1. Nolen-Hoeksema, S., Wisco, B. E., and Lyubomirsky, S. "Rethinking rumination." Perspectives on Psychological Science 3, no. 5 (2008): 400-424. 2. Treynor, W., Gonzalez, R., and Nolen-Hoeksema, S. "Rumination reconsidered: A psychometric analysis." Cognitive Therapy and Research 27, no. 3 (2003): 247-259. 3. Watkins, E. R. "Constructive and unconstructive repetitive thought." Psychological Bulletin 134, no. 2 (2008): 163-206. 4. Sbarra, D. A., Boals, A., Mason, A. E., Larson, G. M., and Mehl, M. R. "Expressive writing can impede emotional recovery following marital separation." Clinical Psychological Science 1, no. 2 (2013): 120-134. 5. Ullrich and Lutgendorf, "Journaling about stressful events" (see Chapter 1, note 7). 6. Sloan, D. M., Marx, B. P., Lee, D. J., and Resick, P. A. "A brief exposure-based treatment vs cognitive processing therapy for posttraumatic stress disorder: A randomized noninferiority clinical trial." JAMA Psychiatry 75, no. 3 (2018): 233-239. Chapter 9: Keeping It Going, Keeping It Private 1. Lally, P., van Jaarsveld, C. H. M., Potts, H. W. W., and Wardle, J. "How are habits formed: Modelling habit formation in the real world." European Journal of Social Psychology 40, no. 6 (2010): 998-1009. 2. Mozilla Foundation. Privacy Not Included: Mental Health Apps. 2022. https://foundation.mozilla.org/en/privacynotincluded/categories/mental-health-apps/. 3. US Federal Trade Commission. "FTC to Ban BetterHelp from Revealing Consumers' Data, Including Sensitive Mental Health Information, to Facebook and Others for Targeted Advertising." Press release, March 2, 2023. https://www.ftc.gov/news-events/news/press-releases/2023/03/ftc-ban-betterhelp-revealing-consumers-data-including-sensitive-mental-health-information-facebook. Further Reading Pennebaker, J. W., and Smyth, J. M. Opening Up by Writing It Down: How Expressive Writing Improves Health and Eases Emotional Pain. 3rd ed. New York: Guilford Press, 2016.** The two researchers most closely associated with expressive writing explain the science and how to apply it. Readable, balanced and practical. Pennebaker, J. W., and Evans, J. F. Expressive Writing: Words That Heal. Enumclaw, WA: Idyll Arbor, 2014.** A short workbook built around the standard expressive writing exercise and its variations, with guidance on when not to write. Beilock, S. Choke: What the Secrets of the Brain Reveal About Getting It Right When You Have To. New York: Free Press, 2010.** An accessible account of why capable people fail under pressure, including the research behind pre-exam writing. Greenberger, D., and Padesky, C. A. Mind Over Mood. 2nd ed. New York: Guilford Press, 2016.** The standard CBT self-help workbook, widely used by therapists. The best single source for learning thought records properly. Leahy, R. L. The Worry Cure. New York: Harmony Books, 2005.** A practical guide to productive and unproductive worry, written by a senior cognitive therapist. Neff, K. Self-Compassion: The Proven Power of Being Kind to Yourself. New York: William Morrow, 2011.** The researcher who defined self-compassion explains the concept and offers exercises, including the self-compassion letter. Kross, E. Chatter: The Voice in Our Head, Why It Matters, and How to Harness It. New York: Crown, 2021.** A readable account of research on inner speech, rumination and self-distancing techniques such as addressing oneself by name. Emmons, R. A. Thanks! How the New Science of Gratitude Can Make You Happier. Boston: Houghton Mifflin, 2007.** A popular treatment of gratitude research by one of its leading investigators. 988 Suicide and Crisis Lifeline (988lifeline.org). Free, confidential support by call, text or chat, 24 hours a day, in the United States. The Jed Foundation (jedfoundation.org). A nonprofit focused on emotional health and suicide prevention for teens and young adults, with guidance for students, families and campuses. Healthy Minds Network (healthymindsnetwork.org). Publishes the annual Healthy Minds Study of college student mental health and related research. Hashtags: #StudentJournaling #PowerOfJournaling #JournalingForStress #StressManagement #AcademicStress #AcademicAnxiety #StudentMentalHealth #CollegeStudentWellness #BudgetFriendlyWellness #LowCostSelfCare #ExpressiveWriting #TherapeuticWriting #PreExamJournaling #TestAnxiety #ExamStressRelief #BrainDumpJournaling #GratitudeJournaling #SelfCompassionWriting #CognitiveBehavioralTechniques #ThoughtRecords #ScheduledWorryTime #EmotionalProcessing #MindfulnessJournaling #MentalWellbeing #StressReliefTechniques #HealthyStudyHabits #StudentSelfCare #BedtimeJournaling #EmotionalResilience #EvidenceBasedSelfHelp

Download the Book (PDF): Introduction Every prescription drug sold in the United States arrives with a folded sheet of paper so densely printed that almost nobody reads it. Pharmacists staple it to the bag, patients throw it away, and physicians consult the digital version only when something has gone wrong. Yet that sheet, the prescribing information that lawyers and regulators simply call "the label," is the most consequential legal document in American medicine. It is the product of years of clinical research and months of negotiation between a manufacturer and the Food and Drug Administration. It defines what the drug is legally for. It fixes the outer boundary of what the company may say when it sells the product. It is revised when new dangers emerge, and when it is not revised quickly enough it becomes Exhibit A in lawsuits that can run to thousands of plaintiffs and billions of dollars. When a court asks whether federal law shields a manufacturer from those suits, the question it is really asking is who had the power to change the label and when. This booklet follows a drug's legal life from the laboratory to the courtroom, and it argues that the label is the hinge on which the whole system turns. The approval process is, at bottom, a procedure for producing a label that the government is prepared to stand behind. The law of off-label promotion is a law about speech that strays beyond the label. Postmarket safety surveillance and recalls are mechanisms for revising the label, or for taking the product off the market when no label could make it safe. Civil liability for failing to warn is a judgment, rendered by a jury after the fact, about whether the label said enough. And the doctrine of federal preemption, which decides whether those juries may be heard at all, turns on who controlled the words. Seen this way, a body of law that looks sprawling and technical has a single center of gravity, and most of its hardest controversies are arguments about how much weight the label should bear. Why this subject matters now Pharmaceutical law has rarely been as unsettled as it is in the mid-2020s. The Supreme Court's 2024 decision in Loper Bright Enterprises v. Raimondo ended the practice of judicial deference to agency interpretations of ambiguous statutes, and the FDA, which administers one of the oldest and most amended regulatory statutes in the federal code, is among the agencies most exposed. A change of administration in 2025 brought new leadership to the Department of Health and Human Services and the FDA, substantial reductions in agency staff, public promises to speed approvals, new priority review mechanisms, a push to reduce animal testing, a campaign against direct-to-consumer advertising, and a willingness to revisit settled safety judgments. Meanwhile the courts continued to work through mass litigation over heartburn medication, weight-loss drugs, hormonal contraceptives and pain relievers, and to test whether the qui tam provisions of the False Claims Act, the engine of most off-label promotion enforcement, are consistent with the Constitution. These developments make the subject feel volatile, and in some respects it is. But the volatility sits on a structure that has been stable for decades. The Federal Food, Drug, and Cosmetic Act of 1938, as amended in 1962 and many times since, still requires that a new drug be shown safe and effective before it is sold. Manufacturers are still responsible for the content of their labels. Physicians are still free to prescribe approved drugs for unapproved uses. State tort law still supplies the remedy for injured patients, subject to limits the Supreme Court drew between 2009 and 2019. Understanding that structure is the only way to judge what the current turbulence is likely to change and what it will leave alone. Who this book is for The reader I have in mind is intelligent, curious and not a specialist: a journalist covering a drug safety story, a physician who wonders why sales representatives can say some things and not others, a patient advocate, a policy student, an investor trying to understand regulatory risk, or a lawyer from another field who has been handed a pharmaceutical matter and needs the architecture before the details. I assume no legal training. Terms of art are explained where they first appear, and the cases discussed are real decisions, cited by their proper names and courts. This is an educational work, not legal advice for any particular situation. The focus is on prescription drugs and, where the law is materially the same, on biological products such as vaccines, antibodies and gene therapies. Medical devices, dietary supplements, food, tobacco and cosmetics are governed by the same statute but by largely different rules, and they appear only where a contrast illuminates the drug rules. The law described is American law. Comparative notes on Europe and elsewhere appear where they sharpen the point, but the book does not attempt a survey of global regulation. How the argument unfolds The first chapter explains how the modern regulatory system was built, disaster by disaster, and sets out its basic legal machinery: the concepts of adulteration and misbranding, the definition of a "new drug," the agency's enforcement tools, and the constitutional and administrative-law framework within which the FDA now operates. The second and third chapters follow a drug through development and approval. Chapter 2 treats the clinical trial as a legal instrument, examining the investigational new drug application, the ethical and regulatory protections for human subjects, the duty to register and report trials, and the liability that attaches when research goes wrong. Chapter 3 examines the approval decision itself, including the "substantial evidence" standard, the review process funded by industry user fees, the expedited pathways that now account for a large share of novel approvals, the controversies over accelerated approval, and the separate route for generic drugs. The fourth chapter turns to the label as a legal document: what it contains, how it is written, what "intended use" means, and why off-label prescribing by physicians is lawful while off-label promotion by manufacturers has long been treated as a crime. Chapter 5 describes the enforcement regime that grew up around that distinction, a regime built largely on the False Claims Act and responsible for some of the largest corporate settlements in American history. Chapter 6 examines the First Amendment challenge to that regime, from the Supreme Court's decision in Sorrell v. IMS Health through the Second Circuit's decision in United States v. Caronia and the agency's shifting guidance on how companies may share scientific information. The last three chapters address what happens after approval. Chapter 7 covers postmarket safety surveillance, safety-driven labeling changes and the legal mechanics of drug recalls, including the surprising fact that the FDA has no general power to order a drug recalled. Chapter 8 examines civil liability for failing to warn about adverse effects: the learned intermediary doctrine, the problems of proving causation, the role of expert evidence, and the machinery of mass tort litigation. Chapter 9 addresses preemption, the doctrine that decides whether federal approval of a label bars a state-law claim that the label was inadequate, and the stark and much-criticized divide it has produced between brand-name and generic drugs. The conclusion draws these threads together and asks what a better-designed system would look like, and what the reader should watch for as the law continues to move. A note on what the law is for It is easy, in a subject this technical, to lose sight of the stakes. The modern drug statute exists because of dead children: the more than one hundred people, many of them young, killed in 1937 by a sulfanilamide elixir dissolved in diethylene glycol, and the thousands of infants born with severe limb deformities in Europe and elsewhere after their mothers took thalidomide in the late 1950s and early 1960s. It also exists because patients with serious diseases need new treatments, and every month a useful drug is held back by regulatory delay is a month in which some of those patients suffer or die. The law is a continuing, imperfect negotiation between these two truths. Nothing in the chapters that follow resolves that tension, and anyone who claims to have resolved it is selling something. What the law can do, and what this booklet tries to explain, is to assign responsibility for the risks clearly enough that patients, physicians, companies and courts each know what they owe one another. Chapter 1: Built by Disaster: The Architecture of Drug Regulation American drug law was not designed. It was accreted, one crisis at a time, and its structure still bears the marks of the particular catastrophes that produced each layer. A reader who understands those catastrophes understands why the law asks the questions it asks, and why certain features that look odd to a newcomer, such as the central role of the label or the absence of any general recall power, are not accidents but fossils of specific political bargains. From patent medicines to the 1938 Act For most of the nineteenth century, the American market in medicines was essentially unregulated. "Patent medicines," which usually had no patent and often little medicine, were sold with extravagant claims and undisclosed ingredients, including alcohol, opium and cocaine. The muckraking journalism of the early twentieth century, together with the chemist Harvey Washington Wiley's campaign at the Department of Agriculture, produced the Pure Food and Drugs Act of 1906. That statute prohibited the interstate sale of "adulterated" and "misbranded" drugs, and those two concepts remain the foundation of drug law today. The 1906 Act was, however, a truth-in-labeling law rather than a safety law. It required that labels not be false or misleading, and the Supreme Court read even that requirement narrowly. In United States v. Johnson (1911), the Court held that the Act's prohibition on false statements reached only statements about a drug's identity and ingredients, not false claims about its curative power. Congress responded with the Sherley Amendment of 1912, which prohibited false and fraudulent therapeutic claims, but the government had to prove intent to defraud, a burden that made the provision nearly unenforceable. A seller could market a worthless cancer cure with impunity if he sincerely believed in it, or if the government could not prove he did not. The event that transformed the law came in the autumn of 1937. The S.E. Massengill Company of Bristol, Tennessee, sought to produce a liquid form of sulfanilamide, one of the first antibacterial drugs, for patients who had difficulty swallowing tablets. Its chief chemist dissolved the drug in diethylene glycol, a sweet-tasting solvent chemically related to antifreeze, added raspberry flavoring, and shipped the product nationwide as "Elixir Sulfanilamide." No one tested the formulation for toxicity, and no law required anyone to. More than one hundred people died, many of them children, in agonizing kidney failure. The government's only legal hook for seizing the remaining stock was a technicality: an "elixir" was supposed to contain alcohol, and this one did not, so the product was misbranded. Congress had been debating a new food and drug bill for years, and the Massengill deaths broke the logjam. The Federal Food, Drug, and Cosmetic Act of 1938 required, for the first time, that a manufacturer submit a "new drug application" demonstrating that a new drug was safe before it could be marketed. The FDA had a fixed period in which to object; if it did nothing, the application became effective automatically. The Act also required adequate directions for use and warnings on labels, extended the misbranding provisions, and removed the requirement to prove fraudulent intent for false therapeutic claims. The 1938 Act, heavily amended, is still the governing statute. It lives in Title 21 of the United States Code, and its sections are universally referred to by their numbering in the original Act: section 505 for new drug approval, section 502 for misbranding, section 501 for adulteration. Thalidomide and the 1962 amendments The second great layer came from a disaster that the United States largely escaped. Thalidomide, a sedative marketed in West Germany from 1957 and later in dozens of other countries, was prescribed to pregnant women for morning sickness. It caused catastrophic birth defects, most characteristically phocomelia, in which the long bones of the limbs fail to develop. Estimates of the number of affected children worldwide run to more than ten thousand. In the United States, the application to market the drug landed on the desk of Frances Oldham Kelsey, a newly hired FDA medical officer, who repeatedly found the safety data inadequate and refused to let it become effective. By the time the link to birth defects was established in late 1961, American exposure had been limited largely to patients who received the drug as an investigational product from physicians to whom the sponsor had distributed samples. The near miss galvanized Congress, which had been considering drug reform led by Senator Estes Kefauver for other reasons, chiefly high prices and weak evidence of efficacy. The Kefauver-Harris Amendments of 1962 changed the system in three fundamental ways. First, they required proof of effectiveness as well as safety, defining the required proof as "substantial evidence," meaning evidence from "adequate and well-controlled investigations, including clinical investigations," by qualified experts. Second, they converted the premarket system from notification to affirmative approval: a new drug could not be marketed until the FDA approved it. Third, they gave the FDA authority over clinical investigations themselves, requiring that investigational drugs be studied under regulations that included informed consent of subjects. The same amendments transferred jurisdiction over prescription drug advertising from the Federal Trade Commission to the FDA, and required manufacturers to report adverse experiences. The 1962 amendments also required the FDA to review the efficacy of every drug approved on safety grounds alone between 1938 and 1962. This effort, the Drug Efficacy Study Implementation, took decades and removed many products from the market. It also produced important litigation. In Weinberger v. Hynson, Westcott & Dunning (1973), the Supreme Court upheld the agency's power to deny a hearing where a company could not produce adequate and well-controlled studies, effectively confirming that the "substantial evidence" standard meant real clinical trials and not physician testimonials. The later layers The statute's development since 1962 is best understood as a series of adjustments to the balance between access and caution, each responding to a perceived failure in one direction or the other. The major enactments are set out in Table 1, which is meant as a map rather than a history; the chapters that follow return to most of them. Table 1. Major federal enactments shaping U.S. drug regulation. Year Enactment Principal effect 1906 Pure Food and Drugs Act Prohibited adulterated and misbranded drugs 1938 Federal Food, Drug, and Cosmetic Act Premarket proof of safety; new drug applications 1962 Kefauver-Harris Amendments Proof of efficacy; affirmative approval; trial oversight 1983 Orphan Drug Act Incentives for drugs for rare diseases 1984 Hatch-Waxman Act Abbreviated generic pathway; patent term restoration 1992 Prescription Drug User Fee Act Industry fees fund review; performance deadlines 1997 FDA Modernization Act Fast track; limited dissemination of off-label reprints 2007 FDA Amendments Act REMS; postmarket study and labeling powers; trial registry 2010 Biologics Price Competition and Innovation Act Abbreviated pathway for biosimilars 2012 FDA Safety and Innovation Act Breakthrough therapy designation 2016 21st Century Cures Act Real-world evidence; payor communications 2022 Food and Drug Omnibus Reform Act Accelerated approval reforms; trial diversity plans Several themes run through this list. The Hatch-Waxman Act of 1984 (formally the Drug Price Competition and Patent Term Restoration Act) created the modern generic drug industry by allowing a generic manufacturer to rely on the brand manufacturer's safety and efficacy data and to show only that its product is the "same" as the brand drug and bioequivalent to it. That bargain, as Chapter 9 explains, had consequences for tort law that no one in Congress anticipated. The Prescription Drug User Fee Act of 1992, passed in the shadow of the AIDS crisis and complaints about a "drug lag" behind Europe, allowed the FDA to collect fees from applicants in exchange for commitments to review deadlines. It must be reauthorized every five years, and each reauthorization has become a vehicle for other reforms; the current authorization runs through September 2027, so its renewal is a live legislative question as this book is written. The FDA Amendments Act of 2007, passed after the withdrawal of the painkiller Vioxx, pushed in the other direction, strengthening the agency's postmarket powers. The core legal concepts Beneath the procedural apparatus, drug law rests on a small number of concepts that recur throughout this book. A drug is defined in section 201(g) of the Act by reference to its intended use: articles "intended for use in the diagnosis, cure, mitigation, treatment, or prevention of disease," or "intended to affect the structure or any function of the body," along with articles recognized in the official compendia. The definition's reliance on intended use has momentous consequences. The same chemical can be a drug or not depending on how it is marketed, which is why the law of promotion and the law of classification are intertwined, and why the agency's views about how intent is proved have been fought over for decades. A new drug, under section 201(p), is any drug not generally recognized among qualified experts as safe and effective for the uses in its labeling. In practice, virtually every drug introduced since 1962 is a new drug, and section 505 prohibits introducing a new drug into interstate commerce unless an approval is in effect. The term "new" therefore does not mean recent; it means unapproved-until-proven. A drug that has been sold for decades remains a "new drug" in the legal sense, and a new use of an old drug can make it new again. A drug is misbranded under section 502 if, among other things, its labeling is false or misleading in any particular, it fails to bear adequate directions for use and adequate warnings, or its advertising fails to include required information about risks. The concept of "adequate directions for use" is the hidden foundation of the law of off-label promotion. A prescription drug is exempt from the requirement of adequate directions for lay users only if its labeling contains the information physicians need for its intended uses. If a manufacturer intends the drug for a use not described in its labeling, the labeling lacks adequate directions for that use and the drug is misbranded. This chain of reasoning, explained in Chapter 4, is how promotion of an unapproved use became a federal crime. A drug is adulterated under section 501 if, among other things, it is manufactured in conditions that do not conform to current good manufacturing practice, contains filthy or decomposed substances, or differs in strength, quality or purity from what it purports to be. Adulteration is the legal basis for most recalls, and the manufacturing standards, codified at parts 210 and 211 of Title 21 of the Code of Federal Regulations, are the subject of much of the agency's inspection work. The Act enforces these concepts through a set of prohibited acts listed in section 301, including introducing an adulterated or misbranded drug into interstate commerce. Violations are crimes. A first violation is a misdemeanor punishable without proof of intent; a violation committed with intent to defraud or mislead, or a repeat violation, is a felony. The absence of an intent requirement for misdemeanors is a distinctive and controversial feature. In United States v. Dotterweich (1943), the Supreme Court held that the president of a company that shipped misbranded drugs could be convicted even though he had no personal knowledge of the violation, because the Act puts the burden of protecting the public "upon those who have at least the opportunity of informing themselves of the existence of conditions imposed for the protection of consumers." In United States v. Park (1975), the Court upheld the conviction of the chief executive of a food chain whose warehouses were infested with rodents, holding that a corporate officer with "responsible relation" to a violation, who had the power to prevent or correct it and failed to do so, could be held criminally liable. This "responsible corporate officer" doctrine, sometimes called the Park doctrine, has been used against pharmaceutical executives. In 2007, three senior executives of Purdue Frederick pleaded guilty to misdemeanor misbranding in connection with the marketing of OxyContin, on the basis of their positions rather than any allegation that they personally knew of the misconduct, and were later excluded from participating in federal health programs. The enforcement toolkit The FDA does not prosecute crimes itself; it refers cases to the Department of Justice. Its own tools are largely administrative. It inspects manufacturing facilities and clinical trial sites and records observed deficiencies on a Form 483. It issues warning letters, which have no binding legal force but are public and are taken seriously by companies and investors. It can ask the Justice Department to seek seizure of violative products or an injunction against a company, often resolved through a consent decree under which the company agrees to remediation overseen by independent experts. It can refuse admission to imported drugs and place foreign manufacturers on import alert. It can suspend or withdraw approval of an application, though withdrawal ordinarily requires notice and an opportunity for a hearing. It can require labeling changes and postmarket studies under powers granted in 2007. And it can communicate with the public, issuing safety communications and alerts that often do more practical work than any formal order. What the FDA generally cannot do is order a drug recalled. That gap, and how the system works around it, is the subject of Chapter 7. The administrative law frame The FDA is an executive agency within the Department of Health and Human Services, and it acts through regulations, guidance documents, and individual decisions on applications. All three are shaped by general administrative law. Regulations adopted through notice-and-comment rulemaking under the Administrative Procedure Act have the force of law. Guidance documents, which the FDA issues in large numbers, do not; under the agency's own good guidance practices regulation, they represent the agency's current thinking and bind neither the public nor the agency. In practice, companies follow guidance closely because departing from it invites scrutiny, which is why the legal status of guidance has become important in the off-label debate. For forty years, courts reviewing the FDA's interpretation of its statute applied the framework of Chevron U.S.A. v. Natural Resources Defense Council (1984), deferring to any reasonable agency reading of an ambiguous provision. The FDA's history in the Supreme Court under that regime was mixed. In FDA v. Brown & Williamson Tobacco Corp. (2000), the Court held that the agency lacked authority to regulate tobacco as a drug, reasoning that Congress could not have intended to delegate a question of such economic and political significance so obliquely. (Congress granted the authority in 2009.) In June 2024, in Loper Bright Enterprises v. Raimondo, the Court overruled Chevron, holding that courts must exercise independent judgment about the best reading of a statute, although they may give respectful consideration to an agency's views. For the FDA, the consequences are still unfolding. The Act is full of broad terms, including "adequate and well-controlled," "intended use," "false or misleading," and "substantial evidence," whose meaning the agency has elaborated over decades. After Loper Bright, regulated companies have greater room to argue that the agency's readings are wrong, and the agency must defend them on the merits rather than on deference. At the same time, much of what the FDA does involves scientific judgment applied to particular facts, and courts have long given such judgments considerable latitude under the arbitrary-and-capricious standard, a form of deference that Loper Bright did not disturb. The Supreme Court's April 2025 decision in FDA v. Wages and White Lion Investments, which upheld the agency's denial of marketing applications for flavored e-cigarette liquids against a claim that it had changed its standards unfairly, suggests that the Court is not inclined to second-guess product-specific scientific decisions lightly. The agency's legitimacy also depends on politics in a way that its formal structure obscures. The commissioner is a political appointee, and the agency's decisions on contraception, vaccines, opioids, and abortion medication have repeatedly drawn it into national controversy. The Supreme Court's 2024 decision in FDA v. Alliance for Hippocratic Medicine, which rejected on standing grounds a challenge by physicians to the FDA's loosening of restrictions on mifepristone, left the merits of that controversy undecided. The period beginning in 2025, with new departmental leadership openly skeptical of aspects of the agency's past practice, a significant reduction in staff, and public statements about vaccine safety and approval standards, has made the question of how much of the FDA's work rests on durable legal commitments and how much on institutional habit more pressing than at any time in decades. What the architecture implies Three features of this architecture shape everything that follows. First, the system is built around products and their labeling, not around medical practice. The FDA regulates what manufacturers sell and say. It does not regulate what physicians do with approved drugs. The line between the two, which Congress has reaffirmed in section 1006 of the Act by disclaiming any intention to interfere with the practice of medicine, is the foundation of the off-label system. Second, the system relies heavily on the manufacturer's own information and incentives. The sponsor designs and runs the trials, writes the first draft of the label, collects adverse event reports, and decides in the first instance whether to recall a product. The FDA reviews and reacts. That design reflects limited public resources, but it also means that the law's most important function is to structure the manufacturer's incentives, which is why civil liability and fraud enforcement matter so much. Third, the system is layered with other bodies of law that pursue overlapping goals by different means. State tort law compensates injured patients. The False Claims Act polices fraud on federal health programs. Securities law punishes misleading statements to investors about regulatory prospects. Antitrust law polices abuses of the generic drug system. Each of these regimes takes the FDA's decisions as its starting point, and each creates pressure on the label from a different direction. The chapters that follow trace those pressures in the order a drug encounters them, beginning with the clinical trial. Chapter 2: The Clinical Trial as a Legal Instrument A clinical trial is a scientific experiment, but it is also a legal event of unusual density. It requires the government's permission to ship an unapproved drug across state lines. It takes place under federal rules that govern who may be enrolled, what they must be told, and who must oversee the research. It generates data that will be scrutinized for integrity and must be publicly registered and reported. And it exposes the people who run it to liability when participants are harmed. The law of clinical trials is the law of how society permits a small number of people to bear risk for the benefit of many, and its main instrument is the requirement of informed, voluntary consent reviewed by an independent body. Permission to experiment: the IND Because section 505 prohibits introducing an unapproved new drug into interstate commerce, a sponsor that wishes to test a drug in humans needs an exemption. The exemption is the investigational new drug application, or IND, governed by part 312 of Title 21 of the Code of Federal Regulations. The IND contains the sponsor's preclinical data on pharmacology and toxicology, information on manufacturing and chemistry, the proposed clinical protocol, and information on the investigators. Once the IND is submitted, the sponsor may begin the trial after thirty days unless the FDA imposes a "clinical hold," an order that the study not begin or be suspended. Holds are imposed when the agency concludes that subjects would face unreasonable and significant risk, that investigators are not qualified, that the investigator's brochure is misleading, or that the application lacks sufficient information to assess risk. The IND is thus an unusual form of regulation: permission by default, subject to veto. It has proven durable because it combines speed with a real check. Most INDs proceed without objection, but the possibility of a hold gives the agency leverage to shape protocols, and holds are imposed with some regularity, particularly in novel fields like gene therapy. A hold lifted after the sponsor responds is routine; one that persists can end a program. The preclinical data supporting an IND historically included mandatory animal testing. The FDA Modernization Act 2.0, enacted in December 2022, removed the statutory language that had been read to require animal studies, allowing sponsors to rely on alternative methods such as cell-based assays, organ-on-chip systems and computer modeling where scientifically adequate. In 2025 the agency announced a plan to phase out animal testing requirements for certain products, beginning with monoclonal antibodies. How quickly that shift translates into practice depends on whether the alternatives can predict human toxicity as reliably, which is a scientific question the legal change leaves open. Clinical development conventionally proceeds in phases, which the regulations describe though they do not strictly require. Phase 1 studies, usually in a few dozen healthy volunteers or, in oncology, in patients, assess safety, tolerability and how the drug is absorbed and eliminated. Phase 2 studies, in a few hundred patients, explore effectiveness and dosing. Phase 3 studies, often in thousands of patients, are the large confirmatory trials designed to supply the "substantial evidence" of effectiveness that approval requires and to characterize the drug's safety profile. The phases increasingly overlap, and modern protocols often use "seamless" or adaptive designs, but the conceptual progression from safety to dose to confirmation persists. Protecting subjects: consent and oversight The ethical framework of human research is written in the aftermath of abuse. The Nuremberg Code of 1947, formulated in the judgment against Nazi physicians who had conducted lethal experiments on concentration camp prisoners, declared that "the voluntary consent of the human subject is absolutely essential." The World Medical Association's Declaration of Helsinki, first adopted in 1964 and revised many times since, elaborated principles for research combined with medical care. In the United States, the revelation in 1972 of the Public Health Service's syphilis study at Tuskegee, in which hundreds of Black men with syphilis were observed without treatment for forty years, even after penicillin became standard therapy, led to the National Research Act of 1974 and the Belmont Report of 1979. The Belmont Report's three principles, respect for persons, beneficence and justice, remain the conceptual vocabulary of American research ethics. These principles are implemented through two sets of federal rules that overlap but are not identical. The FDA's regulations at parts 50 and 56 of Title 21 govern informed consent and institutional review boards for research on FDA-regulated products, regardless of who funds it. The Federal Policy for the Protection of Human Subjects, known as the Common Rule and codified for HHS at part 46 of Title 45, governs research funded or conducted by federal agencies that have adopted it. The Common Rule was substantially revised in 2018, among other things to require a single IRB for most cooperative research and to require a concise "key information" section at the start of consent forms. The FDA has been harmonizing its rules with the revised Common Rule, but differences remain, and a trial may be subject to both. Informed consent under these rules has required elements: a statement that the study involves research, a description of its purposes and procedures, a description of reasonably foreseeable risks and discomforts, a description of benefits, disclosure of alternatives, a statement about confidentiality, an explanation of compensation and treatment for injury in research involving more than minimal risk, contacts for questions, and a statement that participation is voluntary and may be discontinued without penalty. The consent must be documented, usually in a signed form, and must be sought under circumstances that minimize coercion or undue influence. Special protections apply to children, prisoners, pregnant women and fetuses, and adults who lack capacity, generally requiring the permission of a parent or legally authorized representative and, for children capable of it, their assent. Institutional review boards are the gatekeepers. An IRB must have at least five members of varied backgrounds, including at least one scientist, one nonscientist, and one person not otherwise affiliated with the institution. It must approve a study before it begins and review it at least annually for most ongoing research involving more than minimal risk. It must find, among other things, that risks are minimized and reasonable in relation to anticipated benefits, that selection of subjects is equitable, and that consent will be properly obtained and documented. IRBs may be based at academic institutions or may be independent commercial boards, and the rise of commercial IRBs, paid by sponsors, has raised persistent questions about independence that the regulations address only through conflict-of-interest rules for individual members. For larger trials, sponsors typically also appoint a data monitoring committee, an independent group of clinicians and statisticians who review unblinded interim data and can recommend stopping a trial early for benefit, harm or futility. Their deliberations are confidential, and their role is increasingly important in trials where the sponsor and investigators must remain blinded. When research harms: the Gelsinger case and its lessons The rules are only as good as their observance, and the most important modern American research scandal illustrates how they can fail. In September 1999, Jesse Gelsinger, an eighteen-year-old from Arizona with a mild, diet-controlled form of a metabolic disorder called ornithine transcarbamylase deficiency, died four days after receiving an experimental gene therapy vector in a Phase 1 trial at the University of Pennsylvania. Subsequent investigations found that he should arguably have been excluded under the protocol's criteria because of elevated ammonia levels, that serious adverse events in earlier subjects and deaths of monkeys in related animal studies had not been adequately reported to the FDA or disclosed in the consent form, and that the lead investigator had a financial interest in a company with rights to the technology. The FDA placed the university's human gene therapy trials on hold. Gelsinger's family sued the university, the investigators and others; the case settled. In 2005 the university and a related institution resolved federal civil claims with the Justice Department, and the investigators accepted restrictions on their research. The Gelsinger case is still taught because it combined nearly every failure the regulations are designed to prevent: inadequate disclosure, deviation from protocol, failure to report, and conflicts of interest. It also led to reforms, including stronger conflict-of-interest policies at academic medical centers and greater scrutiny of gene therapy trials. And it illustrates the characteristic legal posture of research injury: the claims were in tort, sounding in negligence, lack of informed consent and fraud, not under the federal regulations themselves, which create no private right of action. The most important judicial decision on research duties came from Maryland. In Grimes v. Kennedy Krieger Institute (2001), the Court of Appeals of Maryland, then the state's highest court, considered a study in which a research institute enrolled families with young children in housing with varying levels of lead-abatement work, to measure how effective partial abatement was. The children's blood lead levels were monitored, but the families were allegedly not promptly informed of hazardous dust readings. The trial court dismissed the families' negligence claims on the ground that researchers owe subjects no legal duty. The Court of Appeals reversed, holding that a research consent agreement can create contractual obligations and that the relationship between researchers and subjects can give rise to a duty of care in tort. The court's opinion was sharply critical of research that exposed healthy children to risk for the benefit of others and went further than necessary in suggesting limits on parental consent for nontherapeutic research, but its central holding, that researchers may owe enforceable duties to subjects, has been influential. Research injury remains a gap in American law. The regulations require that consent forms disclose whether compensation or treatment is available for injury, but they do not require that it be provided, and many sponsors offer only coverage of medical treatment, if that. Several other countries, including many in Europe, require sponsors to carry insurance covering research injury. American subjects who are harmed generally must prove negligence or a consent failure in court, a difficult task when the risk that materialized was one disclosed in the consent form. Integrity and transparency Because clinical trial data are the basis for approval, the law treats their integrity seriously. Sponsors must monitor trials, and investigators must maintain accurate records and permit inspection. The FDA's Bioresearch Monitoring program inspects clinical sites, sponsors and IRBs. Falsifying data submitted to the FDA can be prosecuted under the general federal false statements statute and other laws, and investigators found to have submitted false information can be disqualified from receiving investigational products. The FDA also relies on internationally harmonized standards of "good clinical practice." The International Council for Harmonisation's guideline on good clinical practice was substantially revised in its third version, known as E6(R3), which was adopted in January 2025 and emphasizes risk-proportionate quality management and accommodates digital and decentralized trial methods. The FDA has separately issued guidance on trials with decentralized elements, such as remote visits and home delivery of investigational products, a practice that expanded during the COVID-19 pandemic. A different integrity problem concerns selective publication. For years, sponsors could conduct many trials and publish only the favorable ones, distorting the medical literature. The example most often cited is GlaxoSmithKline's Study 329, a trial of the antidepressant paroxetine in adolescents, published in 2001 with favorable conclusions that later reanalyses found unsupported, while other trials with negative results went unpublished. The New York Attorney General sued the company in 2004 alleging fraud through concealment of unfavorable data, and the company settled and agreed to post trial results. The same year, the International Committee of Medical Journal Editors announced that its journals would publish only trials registered before enrollment began. Congress codified registration in the FDA Amendments Act of 2007. Section 801 of that Act requires the "responsible party," usually the sponsor, to register most controlled clinical investigations of FDA-regulated products other than Phase 1 studies (called "applicable clinical trials") on ClinicalTrials.gov, and to submit summary results, generally within one year of the primary completion date. HHS issued a final rule implementing these requirements in 2016, codified at part 11 of Title 42, which among other things clarified that results must be reported for trials of products that are never approved. Compliance was poor for years. In Seife v. HHS (2020), a federal district court in New York held that the rule's requirements applied to trials completed after 2007 but before the rule took effect, rejecting the agency's narrower reading. The statute authorizes civil penalties for noncompliance, and in 2021 the FDA issued its first notice of noncompliance under the provision, but penalty enforcement has been rare. Transparency depends more on journal policies, public reporting of noncompliance by researchers, and institutional pressure than on federal sanctions. Who is studied: the diversity problem Clinical trials have historically enrolled populations that differ from the patients who will take the drug. Women were excluded from many early-phase trials for years under a 1977 FDA guideline, withdrawn in 1993, that barred women of childbearing potential. Older adults, racial and ethnic minorities, and people with multiple conditions remain underrepresented in many trials. The legal consequence is that labels may carry less information about the populations in which adverse effects are most likely. The Food and Drug Omnibus Reform Act of 2022 required sponsors of Phase 3 trials and certain other pivotal studies to submit "diversity action plans" setting enrollment goals by demographic subgroup and explaining how they would be met. The FDA issued draft guidance in June 2024. In early 2025, amid executive orders directed at diversity programs, the draft guidance was removed from the agency's website, and it was later restored following litigation over the removal of federal health information. The statutory requirement itself remains in the Act, but its implementation has become entangled in broader political disputes, and sponsors face uncertainty about what the agency will expect in practice. The underlying scientific point, that a drug's effects can differ across populations and that a label is only as informative as the trials behind it, does not depend on the politics. Access outside the trial Finally, the law must decide what to do about patients who cannot get into a trial but want an investigational drug. The FDA's expanded access regulations, in subpart I of part 312, allow individual patients, intermediate-size groups and large populations to receive investigational drugs for serious or immediately life-threatening conditions when there is no satisfactory alternative, subject to FDA authorization and IRB review. The agency grants the vast majority of individual requests, often within days; the practical barrier is usually the manufacturer, which is not obliged to provide the drug and may decline because of limited supply or fear that adverse events will complicate the development program. Patients have argued that they have a constitutional right to such drugs. In Abigail Alliance for Better Access to Developmental Drugs v. von Eschenbach (2007), the en banc D.C. Circuit rejected the claim that terminally ill patients have a fundamental right under the Due Process Clause to access investigational drugs that have completed Phase 1 testing, holding that the nation's history of drug regulation did not support such a right. Advocates then turned to legislation. After most states enacted "right to try" laws, Congress passed the federal Right to Try Act in 2018, which allows eligible patients with life-threatening diseases who have exhausted approved options and cannot participate in a trial to obtain, with physician certification, investigational drugs that have completed Phase 1, without FDA authorization or IRB review. The Act limits the use of adverse outcomes against the drug's development and provides some liability protection. It has been used far less than expanded access, largely because the manufacturer's consent remains necessary and the FDA pathway was already permissive. The Right to Try debate is nonetheless revealing: it showed that the bottleneck in access was not primarily legal, and that the law's real leverage lies in the incentives facing manufacturers. The trial and the label Every element of the trial system feeds the label. The population enrolled determines for whom the label can make claims. The endpoints measured determine what benefits it can describe. The adverse events recorded determine what warnings it carries. The integrity of the data determines whether any of it can be trusted, and registration determines whether physicians can see the evidence the label omits. When, years later, a jury is asked whether a manufacturer adequately warned of a risk, the question will often become whether the trials were designed and reported in a way that would have revealed it. The clinical trial is where the label's authority is earned, and where its blind spots are created. Chapter 3: The Approval Decision Approval is the moment at which a private company's product becomes a publicly endorsed medicine. Legally, it is an order by the FDA that an application is approved, accompanied by an agreed label. Practically, it is the outcome of a long negotiation in which the agency weighs evidence of benefit against evidence of risk for a defined population and a defined use. The central claim of this chapter is that approval is never an approval of a drug in the abstract. It is an approval of a drug for the uses, doses and populations described in its labeling, and the entire downstream legal system takes its bearings from that specificity. The application and the standard A sponsor seeking to market a new small-molecule drug submits a new drug application under section 505(b) of the Act. Biological products, such as vaccines, blood products, antibodies and cell and gene therapies, are licensed under a biologics license application pursuant to section 351 of the Public Health Service Act, with a standard phrased in terms of "safe, pure, and potent." The two standards are treated as largely equivalent in practice, and the FDA's Center for Drug Evaluation and Research and Center for Biologics Evaluation and Research apply similar review principles. An application typically runs to hundreds of thousands of pages and includes complete reports of all clinical and nonclinical studies, manufacturing information, proposed labeling, and patent information. Section 505(d) directs the agency to refuse approval if the application fails to show the drug is safe for use under the conditions in the proposed labeling, or if there is a lack of "substantial evidence" that the drug will have the effect it purports to have. Substantial evidence means evidence from adequate and well-controlled investigations on the basis of which qualified experts could fairly and responsibly conclude that the drug will have the claimed effect. The agency historically read the plural "investigations" to require two adequate and well-controlled trials, each convincing on its own, so that a result could not be explained by chance or by the peculiarities of a single study. In 1997 Congress amended the provision to allow the agency to rely on one adequate and well-controlled investigation plus "confirmatory evidence" when it determines that such data are sufficient, and the FDA has increasingly done so, particularly for rare diseases where two large trials may be impossible. In 2023 the agency issued draft guidance describing the kinds of confirmatory evidence it will accept, including mechanistic data, related trials, and real-world evidence. The regulations at section 314.126 describe what makes an investigation adequate and well-controlled: a clear statement of objectives, a design permitting valid comparison with a control, methods of selecting subjects that ensure they have the disease, methods of assigning subjects to groups that minimize bias (typically randomization), measures to minimize bias in assessing outcomes (typically blinding), well-defined and reliable methods of assessing response, and adequate analysis. The controls may be placebo, no treatment, active treatment, dose comparison, or in limited circumstances historical controls. These criteria are the practical definition of good clinical evidence in American law. Safety, by contrast, has no comparably defined standard. No drug is safe in an absolute sense, and the Act has always been read to require a judgment that benefits outweigh risks for the intended population. The agency has formalized this as a structured benefit-risk framework, considering the severity of the condition, the unmet medical need, the magnitude of benefit, the nature and severity of risks, and the ability to manage those risks. A drug with serious toxicity may be approvable for a fatal cancer with no alternatives and unapprovable for a common, benign condition. Inside the review Once an application is submitted, the FDA has sixty days to decide whether it is sufficiently complete to be "filed" for review. If not, it issues a refuse-to-file letter. Filed applications are assigned a review team including medical officers, statisticians, pharmacologists, chemists, clinical pharmacologists and others. Under performance goals negotiated as part of the user fee program, the agency aims to act on standard applications for new molecular entities within ten months of the sixty-day filing date, and on priority applications within six months of filing. Priority review is available for drugs that would provide a significant improvement in the safety or effectiveness of treatment for a serious condition. The review includes inspection of manufacturing facilities and, often, of clinical sites. The agency may convene an advisory committee of outside experts to hold a public meeting and vote on questions such as whether the benefits outweigh the risks. Advisory committee votes are not binding, and the agency has departed from them in both directions. The most prominent recent departure came in 2021, when the FDA granted accelerated approval to aducanumab, a Biogen antibody for Alzheimer's disease, after its advisory committee had voted overwhelmingly that the evidence of clinical benefit was inadequate. Several committee members resigned. The drug's uptake was minimal after federal insurers restricted coverage, and Biogen discontinued it in 2024. The episode prompted congressional investigation and damaged confidence in the agency's process. Since then the agency has convened advisory committees less frequently for novel drugs, a trend that critics regard as a loss of transparency. At the end of the review, the agency either approves the application or issues a "complete response letter" explaining the deficiencies that prevent approval and, often, what would be needed to fix them. Historically these letters were confidential, and companies' public descriptions of them were sometimes incomplete or misleadingly optimistic. In 2025 the FDA began publishing complete response letters, first for past decisions on applications later approved and then more broadly, a transparency measure with implications for securities disclosure as well as scientific understanding. Approval itself comes with conditions. The agency and sponsor negotiate the final label line by line. The agency may require postmarketing studies, which are enforceable requirements under section 505(o)(3), or accept voluntary postmarketing commitments. It may require a Risk Evaluation and Mitigation Strategy, or REMS, under section 505-1, where it determines that such a strategy is necessary to ensure that benefits outweigh risks. A REMS can be as modest as a medication guide for patients or as restrictive as "elements to assure safe use," such as requirements that prescribers be certified, that pharmacies be specially enrolled, or that patients undergo monitoring. The clozapine REMS, which required monitoring of white blood cell counts to guard against a dangerous drop in neutrophils, and the isotretinoin program known as iPLEDGE, designed to prevent pregnancy exposure, are familiar examples. (The FDA eliminated the clozapine REMS in 2025, concluding that the program was no longer necessary to ensure safe use.) The mifepristone REMS, which has been modified several times, has become the most politically contested. The expedited programs Much of the development of drug law since the 1980s has consisted of creating faster pathways for drugs that address serious conditions. There are now four principal programs, which differ in what they offer and what they require, as Table 2 sets out. Table 2. The FDA's principal expedited programs for serious conditions. Program Origin Qualifying criteria Main benefit Fast track 1997 Potential to address unmet need in serious condition Frequent meetings; rolling review Breakthrough therapy 2012 Preliminary clinical evidence of substantial improvement Intensive guidance; senior involvement Accelerated approval 1992 Effect on surrogate endpoint reasonably likely to predict benefit Earlier approval subject to confirmation Priority review 1992 Significant improvement in safety or effectiveness Six-month review goal These programs overlap, and a single drug may qualify for several. According to the agency's own annual summaries, a majority of novel drugs approved in recent years have used at least one expedited program. The programs change timing and intensity of interaction, but with one exception they do not change the evidentiary standard. The exception is accelerated approval, which is a genuine departure and has become the most contested feature of the system. Other designations offer different incentives. The Orphan Drug Act of 1983 grants seven years of marketing exclusivity, tax credits and fee waivers to drugs for diseases affecting fewer than 200,000 people in the United States. Its success in stimulating rare-disease development has been dramatic, but critics note that companies have obtained orphan status for subsets of common diseases and then earned large revenues. Regenerative medicine advanced therapy designation, created by the 21st Century Cures Act, offers breakthrough-like benefits for cell and gene therapies. Priority review vouchers, created in 2007 for tropical diseases and later extended to rare pediatric diseases and medical countermeasures, entitle a sponsor that obtains approval of a qualifying drug to priority review of a different application; the vouchers are transferable and have sold for large sums. In 2025 the agency added a new, commissioner-driven "national priority voucher" program promising dramatically shorter reviews for products that align with stated national priorities such as domestic manufacturing, affordability, or unmet public health needs. That program, created by agency initiative rather than statute, raises questions about criteria and consistency that the courts have not yet addressed. Accelerated approval and its discontents Accelerated approval, created by regulation in 1992 and codified in 2012, allows approval on the basis of an effect on a surrogate endpoint, such as tumor shrinkage or a laboratory measure, that is reasonably likely to predict clinical benefit, or on an intermediate clinical endpoint. The price is a requirement that the sponsor conduct confirmatory trials to verify the predicted benefit. If those trials fail, or are not conducted with due diligence, the agency may withdraw approval through expedited procedures. The program was born in the AIDS crisis, when waiting for mortality data was intolerable, and it has delivered many valuable cancer and HIV drugs. Its weakness has been the confirmatory stage. For years, confirmatory trials were frequently delayed, sometimes for a decade, and withdrawal proceedings were slow and contested. The case that crystallized the problem was Makena, a synthetic progestin approved in 2011 under accelerated approval to reduce the risk of preterm birth, based on a surrogate endpoint. The confirmatory trial, completed in 2019, found no benefit on either preterm birth or neonatal outcomes. The FDA's drug center proposed withdrawal in 2020, the sponsor requested a hearing, and a formal hearing was not held until October 2022. The agency's commissioner withdrew approval in April 2023, more than three years after the failed trial. The Food and Drug Omnibus Reform Act of 2022 responded to this history. It authorizes the agency to require that confirmatory trials be underway before accelerated approval is granted, requires sponsors to report progress every 180 days, and establishes an expedited withdrawal procedure that replaces the formal evidentiary hearing with notice, an opportunity for written comment, and an informal meeting. The agency used the new procedures in subsequent withdrawals, and it has more frequently required confirmatory trials to be underway at approval. Whether these reforms have struck the right balance is contested. Patient groups, especially in rare disease and oncology, argue that tighter requirements will deter development; critics argue that the program still allows products of unproven clinical value to be sold for years at high prices. The deeper issue is that accelerated approval shifts risk from the manufacturer to patients and payers during the period of uncertainty. The label of an accelerated approval product states that continued approval may be contingent on verification of clinical benefit, but that statement means little to a patient told the drug is "FDA-approved." Accelerated approval therefore strains the label's function as a trustworthy summary of evidence. Generics and biosimilars Most prescriptions in the United States are filled with generic drugs, and their approval pathway differs fundamentally. Under section 505(j), added by the Hatch-Waxman Act, a generic manufacturer submits an abbreviated new drug application showing that its product has the same active ingredient, strength, dosage form, route of administration and conditions of use as the brand-name "reference listed drug," and that it is bioequivalent, meaning it delivers the active ingredient to the bloodstream at a comparable rate and extent. The generic does not repeat the clinical trials; it relies on the FDA's prior finding that the reference drug is safe and effective. Critically, the generic must have the same labeling as the reference drug, with limited exceptions. This "sameness" requirement is what makes a generic interchangeable in the eyes of pharmacists and state substitution laws, and it is also what, as Chapter 9 explains, led the Supreme Court to hold that most failure-to-warn claims against generic manufacturers are preempted. A generic manufacturer may also "carve out" patented indications from its label, marketing a so-called skinny label for only unpatented uses. The Federal Circuit's decision in GlaxoSmithKline v. Teva (2021), which held a generic manufacturer liable for inducing infringement of a patented use despite a skinny label, unsettled that practice, and the issue has continued to be litigated in cases involving the line between describing a generic's approval status and encouraging infringing uses. Hatch-Waxman also created a patent-litigation framework, in which a generic filer that certifies a brand's patents are invalid or not infringed (a "Paragraph IV certification") may be sued, triggering an automatic thirty-month stay of approval, with 180 days of exclusivity for the first successful challenger. The system has generated a large body of patent and antitrust law, including the Supreme Court's decision in FTC v. Actavis (2013), which held that "reverse payment" settlements in which brand manufacturers pay generic challengers to delay entry can violate the antitrust laws. For biological products, the Biologics Price Competition and Innovation Act of 2010 created an abbreviated pathway for "biosimilars," which must be "highly similar" to a reference biologic with no clinically meaningful differences, and for "interchangeable" biosimilars that may be substituted at the pharmacy without prescriber intervention. Because biologics are large, complex molecules made in living cells, exact copies are not possible, and the standards are correspondingly more demanding than for small-molecule generics. The FDA has moved in recent years to reduce the clinical data required, reflecting its growing confidence in analytical comparisons. Approval as a legal act Approval has legal consequences beyond permission to sell. It establishes the label that will define the drug's lawful promotion. It triggers periods of regulatory exclusivity, including five years for new chemical entities and three years for new clinical investigations supporting changes, which operate independently of patents. It becomes the baseline against which later claims of inadequate warning are measured, and, in some circumstances, a defense against them. And it is reviewable in court. Competitors, patients and advocacy groups have challenged approvals under the Administrative Procedure Act, though the mifepristone litigation showed that standing to do so is limited: in FDA v. Alliance for Hippocratic Medicine (2024), the Supreme Court unanimously held that physicians who did not prescribe the drug lacked standing to challenge its approval conditions merely because they objected to it and might treat patients who experienced complications. The approval decision also reflects the law's peculiar economy of knowledge. At the moment of approval, the drug has been studied in perhaps a few thousand people, over months or a few years, in populations selected for their suitability for research. Adverse effects that occur in one in ten thousand patients, or only after years of use, or only in people with other diseases or taking other drugs, are unlikely to have been observed. Approval is therefore not a guarantee of safety but a judgment made under structured uncertainty, and the label it produces is necessarily provisional. The law's treatment of promotion, surveillance, recalls and liability can be understood as a set of mechanisms for managing what approval could not know. Chapter 4: The Label as Law To a physician, the label of a prescription drug is a reference document. To a lawyer, it is closer to a constitution: a text negotiated between a private party and the government that defines the scope of permitted conduct, fixes the terms of later disputes, and can be amended only through prescribed procedures. This chapter examines what the label is, how intended use is determined, and why the law draws its most important line not between safe and unsafe drugs but between what a physician may do with a drug and what a manufacturer may say about it. Anatomy of a label The Act distinguishes between the "label," meaning the written material on the immediate container, and "labeling," a much broader category encompassing all written, printed or graphic matter "upon or accompanying" the article. The Supreme Court held in Kordel v. United States (1948) that material need not be physically attached to the product to accompany it; pamphlets shipped separately but used in the sale of the product were labeling. The FDA has long taken the position that virtually all promotional material distributed by or on behalf of a manufacturer, including brochures, sales aids and websites, is labeling. In ordinary usage, however, "the label" of a prescription drug means its official prescribing information, the document the FDA approves. Since the agency's Physician Labeling Rule took effect in 2006, prescribing information for new drugs follows a standardized format. It opens with a half-page "Highlights" summary, followed by a table of contents and the full prescribing information, organized into numbered sections: boxed warning, if any; indications and usage; dosage and administration; dosage forms and strengths; contraindications; warnings and precautions; adverse reactions; drug interactions; use in specific populations; drug abuse and dependence; overdosage; description; clinical pharmacology; nonclinical toxicology; clinical studies; how supplied; and patient counseling information. In 2015 the agency's Pregnancy and Lactation Labeling Rule replaced the old letter categories (A, B, C, D and X) for pregnancy risk with narrative summaries of available data, on the theory that letters conveyed false precision. Several sections carry particular legal weight. The indications and usage section defines the uses for which the drug is approved, and it is the reference point for the concept of off-label use. The boxed warning, sometimes called a black box warning, is the most serious warning the agency requires, reserved for adverse reactions so serious in proportion to benefit that they must be considered in assessing whether to use the drug, or for serious reactions that can be prevented or reduced by appropriate use. The contraindications section lists situations in which the drug should not be used because the risk clearly outweighs any possible benefit. The warnings and precautions section describes clinically significant adverse reactions and other hazards. The distinction between these sections is not merely editorial; as later chapters show, whether a risk appears in the boxed warning, the warnings section or only the adverse reactions table can be decisive in litigation about whether the warning was adequate. The label is also supplemented, for some drugs, by patient-directed documents: medication guides, required when the agency determines that patient information is necessary to prevent serious adverse effects or that patient decision-making should be informed by risks; and patient package inserts, required for some products such as oral contraceptives and estrogens. These documents are written in plain language and dispensed with the drug, and they complicate the traditional assumption that the label speaks only to physicians. Who writes the label, and who can change it The first draft of the label is written by the sponsor. The agency reviews it and negotiates changes, often extensively, and approves the final text. After approval, the label may be changed through supplements to the application. Major changes, including new indications, require a prior approval supplement. Moderate changes may be implemented thirty days after submission. And certain changes may be implemented immediately upon submission, before FDA approval, under the "changes being effected" (CBE) regulation at section 314.70(c) of Title 21. The CBE regulation matters enormously in liability law. It permits a manufacturer to add or strengthen a contraindication, warning, precaution or adverse reaction, or to strengthen a statement about drug abuse, dependence or overdosage, "to reflect newly acquired information." In 2008 the FDA amended the rule to clarify that "newly acquired information" includes new analyses of previously submitted data, and that CBE changes are permitted only where there is reasonable evidence of a causal association. The regulation means that a brand manufacturer does not need the agency's permission to strengthen its warnings, though the agency may later reject the change. That fact, as Chapter 9 explains, is the reason the Supreme Court held in Wyeth v. Levine (2009) that brand-name manufacturers can be held liable under state law for inadequate warnings. Generic manufacturers, by contrast, generally may not use the CBE process to depart from the brand label, because their labels must remain the same as the reference drug's. Congress also gave the FDA its own power to change labels. Under section 505(o)(4), added in 2007, if the agency becomes aware of new safety information that it believes should be included in labeling, it must notify the manufacturer, which must propose changes within thirty days; if the parties cannot agree, the agency may issue an order requiring the change. The provision expressly states that it does not affect the manufacturer's own responsibility to maintain its label, a detail that has been important in preemption cases. The label's status as a manufacturer's responsibility is basic. The FDA approves labels, but it does not write them, and it does not continuously monitor them. The legal duty to keep the label accurate and adequate belongs to the company. Intended use Because a product's regulatory status depends on its intended use, the law must decide how intent is established. The FDA's regulations, at section 201.128 of Title 21 for drugs, define intended use as "the objective intent of the persons legally responsible for the labeling." That intent may be shown by labeling claims, advertising, oral or written statements by those persons or their representatives, and the circumstances surrounding distribution, such as the knowledge that a product is being offered for a purpose for which it is neither labeled nor advertised. For years, the regulation contained a final sentence stating that if a manufacturer knew, or had knowledge of facts that would give it notice, that a product was being used for purposes other than those for which it was offered, it was required to provide adequate labeling for those uses. Read literally, the sentence meant that mere awareness of widespread off-label prescribing could make a drug misbranded unless the company revised its labeling, which it could not do without FDA approval. The agency said it had never applied the provision that way. In 2017 it attempted to revise the regulation, then delayed the revision after industry objections that the new language would still allow intended use to be based on the "totality of the evidence." In a final rule published in August 2021, the agency adopted language stating that intended use may be shown by any relevant source of evidence, while affirming that a firm would not be regarded as intending an unapproved new use based solely on its knowledge that its product was being prescribed or used for that use. This debate is not merely technical. Under the "totality of the evidence" approach, a company's truthful statements about a drug's off-label use could be evidence that the company intended that use, and therefore that the drug was misbranded, even if the statements were not themselves the violation. Critics argue that this approach punishes speech indirectly while claiming to regulate only conduct. The agency argues that intent must be proved by some evidence, and that speech has always been admissible to prove intent; the Supreme Court has recognized that the First Amendment does not prohibit "the evidentiary use of speech to establish the elements of a crime or to prove motive or intent," as it put it in Wisconsin v. Mitchell (1993). That tension is at the center of Chapter 6. Off-label prescribing is lawful The single most important and most often misunderstood point in this area is that physicians may lawfully prescribe approved drugs for unapproved uses. The Act regulates the introduction of drugs into interstate commerce and their labeling and promotion by manufacturers. It does not regulate what licensed practitioners do with drugs lawfully in their possession. Section 1006 of the Act, added in 1997 in connection with medical devices but reflecting a longstanding principle, states that nothing in the Act shall be construed to limit or interfere with the authority of a health care practitioner to prescribe or administer any legally marketed device for any condition or disease within a legitimate practitioner-patient relationship. The FDA has long taken the same position for drugs. The Supreme Court acknowledged it in Buckman Co. v. Plaintiffs' Legal Committee (2001), describing off-label use as an accepted and valuable part of medical practice that the FDA is charged with regulating without directly interfering with. Off-label prescribing is also common, and often good medicine. Oncology relies heavily on off-label regimens, because approved indications lag behind evolving evidence about combinations and new tumor types. Pediatric medicine is largely off-label, because until recent decades few drugs were studied in children. Psychiatry uses many drugs off-label for conditions that were not the subject of pivotal trials. Studies of prescribing patterns have found off-label use to be a substantial fraction of all prescriptions for many drug classes, and a large share of it lacks strong scientific support, which is precisely why its promotion is controversial. The law tolerates off-label prescribing for principled reasons. Approval is expensive and slow, and manufacturers have limited incentive to seek approval for new uses of drugs whose patents are expiring or whose markets for the new use are small. Medical knowledge advances through the published literature faster than labels can be revised. And physicians, not regulators, are positioned to judge the needs of individual patients. Several government programs recognize off-label use explicitly: Medicare's coverage of anticancer drugs extends to off-label uses supported by listed compendia or peer-reviewed literature, and Medicaid's definition of "medically accepted indication" includes compendia-supported uses. The physician's freedom, however, has limits that come from outside the Act. State medical licensing law, malpractice law and institutional policies govern the standard of care. A physician who prescribes off-label without an adequate basis may be negligent, and the fact that a use is off-label is admissible in some jurisdictions as evidence of the standard of care, though most courts hold that off-label use is not itself negligence. Whether a physician must tell a patient that a use is off-label is unsettled; most courts that have considered the question have held that informed consent doctrine requires disclosure of the risks and alternatives of a treatment, not of its regulatory status. Why manufacturers may not promote off-label If physicians may prescribe off-label, why may manufacturers not promote off-label uses? The legal answer is the chain of reasoning described in Chapter 1. A prescription drug is exempt from the requirement that its labeling bear adequate directions for lay use only if its labeling contains adequate information for its intended uses. When a manufacturer promotes a drug for an unapproved use, that use becomes an intended use, and the labeling does not contain adequate information for it, because the approved labeling describes only approved uses. The drug is therefore misbranded. Moreover, a new use of an approved drug may make the drug an unapproved "new drug" for that use. Either way, introducing it into interstate commerce with that intended use violates the Act. The policy answer is more important, and it is what makes the debate hard. If companies could promote any use a physician may prescribe, they would have little reason to seek approval for new uses, and the approval system, which depends on the sponsor's willingness to fund rigorous trials, would be hollowed out. The evidence behind many off-label uses is weak, and manufacturers have obvious incentives to present it favorably. Physicians are busy, rely heavily on information supplied by sales representatives, and are susceptible to selective presentation of evidence. The prohibition on off-label promotion is thus a way of forcing new claims through the approval process, where evidence is tested by a neutral reviewer, before they can be broadcast by the party with the most to gain. The counterargument, pressed by manufacturers and some scholars, is that the prohibition suppresses truthful information that physicians legitimately want, that it is inconsistent to allow physicians to prescribe off-label while forbidding the people with the most knowledge of the drug from discussing it, and that the FDA's approach punishes speech rather than conduct. Chapters 5 and 6 trace how these arguments have fared: first in an enforcement regime that extracted billions of dollars from the industry, and then in courts that have grown increasingly receptive to the First Amendment objection. The permitted zone The prohibition has never been absolute, and the zone of permitted communication has grown. Manufacturers may respond to unsolicited requests from health care professionals for information about off-label uses, provided the response is tailored, balanced and nonpromotional, and is provided by medical rather than sales personnel. They may support independent continuing medical education, provided they do not control its content. They may present scientific findings at scientific meetings and publish in peer-reviewed journals. The 21st Century Cures Act of 2016 permits companies to provide health care economic information to payors and formulary committees, including information related to unapproved uses, if it is based on competent and reliable scientific evidence; the agency's 2018 guidance on payor communications elaborated this safe harbor, including for information about investigational products before approval. And, as Chapter 6 describes, the agency has issued guidance, most recently in final form in January 2025, on when firms may distribute published scientific information about unapproved uses to health care providers. The label, then, is not only the center of a drug's approved identity but also the outer boundary of a zone of protected commercial speech that has been steadily negotiated outward. Where exactly that boundary lies, and whether the government may enforce it with criminal sanctions, is one of the most contested questions in modern regulatory law. Chapter 5: Off-Label Promotion and the Enforcement Era Between roughly 2004 and 2013, the pharmaceutical industry paid the federal and state governments well over ten billion dollars to resolve allegations that it had promoted drugs for unapproved uses. The largest settlements in the history of American health care fraud enforcement were off-label promotion cases. Yet the statute under which most of this money was collected was not the Food, Drug, and Cosmetic Act but a Civil War-era law aimed at fraud on the government. This chapter explains how the off-label prohibition became a fraud prohibition, why that transformation mattered, and what it left unresolved. From misbranding to fraud The Food, Drug, and Cosmetic Act supplies criminal penalties for misbranding, but those penalties, even for felonies, are modest in comparison with the revenues at stake in a blockbuster drug. The criminal fine for an organization is set by the general federal fine statute, which allows fines of up to twice the gross gain or loss from the offense, and corporate guilty pleas in off-label cases have carried substantial criminal fines. But the bulk of the money in most settlements came from civil liability under the False Claims Act. The False Claims Act, first enacted in 1863 to combat fraud by contractors supplying the Union Army, imposes liability on any person who knowingly presents, or causes to be presented, a false or fraudulent claim for payment to the federal government, or knowingly makes a false record or statement material to such a claim. Liability is for treble damages plus a civil penalty for each false claim, adjusted annually for inflation and now in the range of roughly fourteen thousand to twenty-eight thousand dollars per claim. Because Medicare and Medicaid pay for an enormous number of prescriptions, a pharmaceutical company's promotional practices can implicate millions of claims. The theory connecting off-label promotion to false claims runs as follows. Medicaid generally covers outpatient drugs only for "medically accepted indications," defined by statute as FDA-approved uses and uses supported by certain drug compendia. When a manufacturer promotes a drug for uses that are neither approved nor compendia-supported, and physicians prescribe it for those uses, the resulting claims to Medicaid are for uncovered uses and therefore false. The manufacturer did not submit the claims itself, but it "caused" them to be submitted by pharmacies and physicians. Additional theories involved kickbacks, which make resulting claims false under a 2010 amendment to the Anti-Kickback Statute, and false statements to the government. The engine of this enforcement was the qui tam provision, which allows a private person, called a relator, to sue on behalf of the United States and to receive between fifteen and thirty percent of the recovery. Congress strengthened the provision in 1986, and pharmaceutical sales representatives, marketing managers and medical personnel became prolific relators. They had direct knowledge of promotional practices, documents to support their claims, and the prospect of multimillion-dollar awards. The government could intervene and take over a case, and in the major off-label cases it usually did. The landmark cases The first major case set the pattern. David Franklin, a medical liaison at Parke-Davis, a division of Warner-Lambert that Pfizer later acquired, filed a qui tam action in 1996 alleging that the company had promoted its epilepsy drug Neurontin (gabapentin) for a range of unapproved uses, including bipolar disorder, pain and migraine, through paid "consultant" meetings, sponsored medical education and ghostwritten articles. In United States ex rel. Franklin v. Parke-Davis (2001), the federal district court in Massachusetts denied the company's motion to dismiss, accepting the theory that off-label promotion could cause false Medicaid claims. In 2004 Warner-Lambert pleaded guilty to misbranding and agreed to pay $430 million in criminal and civil penalties. The cases that followed grew larger. Table 3 lists several of the most significant resolutions, drawn from the Justice Department's public announcements. The amounts include criminal fines, civil False Claims Act settlements with the federal government and states, and in some cases other claims such as kickbacks or price reporting, so they are not purely attributable to off-label promotion. Table 3. Selected federal resolutions involving off-label promotion allegations. Year Company Principal drugs Total resolution 2004 Warner-Lambert (Pfizer) Neurontin $430 million 2009 Eli Lilly Zyprexa $1.415 billion 2009 Pfizer Bextra and others $2.3 billion 2010 Allergan Botox $600 million 2012 Abbott Depakote $1.5 billion 2012 GlaxoSmithKline Paxil, Wellbutrin, Avandia and others $3 billion 2013 Johnson & Johnson Risperdal, Invega, Natrecor $2.2 billion Several features of these cases stand out. The conduct alleged often went well beyond sharing truthful scientific information. In the Pfizer case, the government alleged that the company promoted the painkiller Bextra for uses and doses the FDA had specifically declined to approve because of safety concerns, and that it paid kickbacks to physicians. The GlaxoSmithKline resolution included allegations that the company promoted the antidepressant Paxil for children and adolescents despite the absence of approval, helped publish a misleading journal article, and failed to report safety data about its diabetes drug Avandia. The Eli Lilly case involved promotion of the antipsychotic Zyprexa for dementia in elderly patients and other uses, although antipsychotics carry a boxed warning about increased mortality in elderly patients with dementia. The Johnson & Johnson case included allegations that its subsidiary promoted Risperdal for elderly dementia patients and children, and that it paid kickbacks to a large pharmacy company serving nursing homes. The cases were almost never litigated to judgment. Companies faced not only enormous potential damages but also the risk of exclusion from Medicare and Medicaid, a penalty imposed by the HHS Office of Inspector General that would be commercially fatal. Settlements therefore typically included a corporate integrity agreement, a five-year compliance program overseen by the Inspector General, requiring independent review, board-level certifications, disclosure of payments to physicians, and restrictions on sales force compensation. In many cases a subsidiary pleaded guilty to a misdemeanor misbranding charge, preserving the parent's eligibility for federal programs. The consequence was that the legal theories underlying off-label enforcement were rarely tested in court. The companies paid, the government declared victory, and the precise boundaries of what was prohibited remained defined largely by settlement documents and corporate integrity agreements rather than judicial decisions. That absence of adjudication would matter when defendants began to press First Amendment arguments. The limits of the false claims theory Where the theory was tested, courts identified significant limits. First, the plaintiff must identify actual false claims. Courts have held that allegations of a fraudulent scheme are insufficient under the heightened pleading standard for fraud unless the relator can connect the scheme to specific claims submitted to the government, or at least provide reliable indicia that such claims were submitted. Relators who worked in sales often lacked access to billing information, and many cases were dismissed on this ground. Second, the plaintiff must show causation: that the promotion caused physicians to write prescriptions that were then submitted for payment. Physicians exercise independent judgment, and courts have required some connection between the promotional conduct and the prescriptions at issue. Third, off-label use is not necessarily uncovered. Because Medicaid covers compendia-supported uses, and Medicare coverage rules vary by program, a prescription for an off-label use is not automatically a false claim. The plaintiff must show that the particular use was not covered. Fourth, and most important in recent years, the Supreme Court tightened the materiality requirement in Universal Health Services v. United States ex rel. Escobar (2016). The Court endorsed the "implied false certification" theory, under which a claim can be false if it makes specific representations about the goods or services provided while failing to disclose noncompliance with material requirements. But it held that materiality is "demanding" and that minor or insubstantial noncompliance is not enough. If the government pays claims in full with actual knowledge that certain requirements were violated, that is very strong evidence that the requirements are not material. After Escobar, defendants have argued, with some success, that the government's continued payment for off-label prescriptions, even after learning of alleged promotional misconduct, shows that the promotion was not material to payment. The Supreme Court has also addressed scienter. In United States ex rel. Schutte v. SuperValu Inc. (2023), the Court held that the Act's knowledge requirement refers to the defendant's actual subjective understanding, so that a defendant who believed its claims were false cannot escape liability by pointing to an objectively reasonable interpretation of an ambiguous requirement it did not actually hold. That decision strengthened the government's hand in cases where internal documents show awareness of wrongdoing. The constitutional challenge to qui tam The qui tam mechanism itself has come under constitutional attack. In United States ex rel. Polansky v. Executive Health Resources (2023), the Supreme Court addressed the government's power to dismiss a qui tam action over the relator's objection. Justice Thomas, dissenting, suggested that the qui tam device might be inconsistent with Article II of the Constitution, which vests executive power, including the power to enforce federal law, in the President and officers accountable to him. Justices Kavanaugh and Barrett, concurring, agreed that the question merited consideration in an appropriate case. In September 2024, in United States ex rel. Zafirov v. Florida Medical Associates, a federal district court in Florida took up that invitation and held that the qui tam provisions violate the Appointments Clause because relators exercise significant executive authority without being appointed as officers of the United States. The decision was appealed to the Eleventh Circuit and has been closely watched, since a ruling against qui tam would drastically reduce the number of cases filed, the majority of which originate with relators. Whatever the appellate outcome, the question is widely expected to reach the Supreme Court. A decision invalidating qui tam would not eliminate False Claims Act enforcement, since the government could still sue, but it would remove the insiders' financial incentive that drove the off-label enforcement era. The decline of pure off-label cases After 2013, large off-label promotion settlements became less common. Several reasons explain the decline. Companies overhauled their compliance programs, often under corporate integrity agreements, restricting sales representatives' discussion of off-label uses and separating medical affairs from commercial functions. The Physician Payments Sunshine Act, enacted as part of the Affordable Care Act in 2010, required public reporting of payments and transfers of value from manufacturers to physicians and teaching hospitals, reducing opportunities for disguised inducements. And, as the next chapter explains, the government's legal position was weakened by First Amendment decisions, making prosecutors more cautious about cases that rested on truthful speech. Enforcement did not disappear; it shifted. The Justice Department continued to pursue cases involving kickbacks, including payments through patient assistance charities and speaker programs, and cases in which promotional claims were false or misleading rather than merely off-label. The HHS Inspector General issued a special fraud alert in 2020 warning that speaker programs, in which physicians are paid to speak to other physicians over meals, carry inherent fraud risk. In 2020 Novartis paid over $678 million to resolve allegations that it used sham speaker programs to induce prescriptions. Opioid manufacturers faced a separate wave of criminal and civil enforcement: Purdue Pharma pleaded guilty in 2020 to federal felonies related to its marketing of OxyContin, including conspiracy to defraud the United States and violations of the Anti-Kickback Statute, although the company's bankruptcy and the Sackler family's contributions generated protracted litigation. In Harrington v. Purdue Pharma (2024), the Supreme Court held that the bankruptcy code did not authorize a plan releasing the Sacklers from third-party claims without the claimants' consent, sending the parties back to negotiate a revised plan. Individual prosecutions also continued. The most significant appellate decision involved two former executives of Acclarent, a medical device company, who were acquitted of felony fraud charges but convicted in 2016 of misdemeanor adulteration and misbranding in connection with promoting a sinus device for an unapproved drug-delivery use. The First Circuit affirmed the misdemeanor convictions in United States v. Facteau (2023), rejecting arguments that the prosecution impermissibly punished truthful speech and that the jury instructions misdescribed how intended use could be proved. The case shows that, whatever the uncertainty about the First Amendment, the misbranding theory remains available to prosecutors when evidence shows a company intended a product for an unapproved use. What the enforcement era accomplished The enforcement era changed industry behavior. Promotional practices that were routine in the 1990s and early 2000s, such as lavish consultant meetings, ghostwritten articles, and sales representatives distributing off-label studies to physicians in the course of detailing, became untenable. Compliance departments grew into powerful institutions within companies. The separation of medical and commercial functions became standard. But the enforcement era also had costs, and they are the costs that animate the First Amendment challenge. Because the law was made through settlements rather than judgments, the boundary between permitted scientific exchange and prohibited promotion remained vague. Companies, fearing enormous liability, became extremely cautious about any communication concerning off-label uses, even truthful, balanced information that physicians wanted. Physicians who wished to know what the manufacturer knew about a drug's use in a population not studied in pivotal trials often could not get a straight answer from a sales representative, and had to rely on a medical information request or on the literature. Critics argued that the law thus kept useful information from physicians while doing little to deter the underlying problem, which was not truthful speech but misleading speech and kickbacks, both independently unlawful. The government's response was that truthful speech was never the target, that the major cases involved deception and inducements, and that clear rules against promotion were necessary to preserve the incentive to seek approval. This debate could not be resolved by enforcement practice alone. It required courts to decide whether the Constitution permits the government to prohibit truthful speech about lawful uses of lawful products, and that is the question to which the next chapter turns. Chapter 6: Speech and the Label: The First Amendment Debate For most of the twentieth century, no one seriously argued that the Constitution protected a drug company's right to promote its products. Commercial speech was thought to lie outside the First Amendment altogether. That assumption collapsed in the 1970s, and over the following four decades the Supreme Court built a body of commercial speech doctrine that increasingly protects truthful advertising. The collision between that doctrine and the FDA's regime for policing off-label promotion produced one of the most intellectually interesting conflicts in modern regulatory law, and one that remains unresolved. The rise of commercial speech protection The modern doctrine began, fittingly, with prescription drugs. In Virginia State Board of Pharmacy v. Virginia Citizens Consumer Council (1976), the Supreme Court struck down a Virginia law that made it unprofessional conduct for pharmacists to advertise prescription drug prices. The Court held that speech proposing a commercial transaction is protected by the First Amendment, reasoning that the free flow of commercial information is indispensable to private economic decisions and to the formation of intelligent opinions about how the economy should be regulated. The consumers who brought the case, many of them elderly and on fixed incomes, had an interest in knowing where drugs were cheapest. Four years later, in Central Hudson Gas & Electric Corp. v. Public Service Commission (1980), the Court announced the four-part test that still governs most commercial speech cases. First, the speech must concern lawful activity and not be misleading; if it is false or misleading, it is not protected. Second, the government's asserted interest must be substantial. Third, the regulation must directly advance that interest. Fourth, it must not be more extensive than necessary to serve the interest, a requirement later interpreted to mean a reasonable fit between means and ends rather than the least restrictive alternative. Over time the Court applied the test with increasing rigor. In 44 Liquormart v. Rhode Island (1996), a plurality suggested that bans on truthful, nonmisleading commercial speech designed to keep people in ignorance for their own good deserve particular skepticism. And in Thompson v. Western States Medical Center (2002), the Court applied Central Hudson to the drug statute itself. The FDA Modernization Act of 1997 had exempted compounded drugs from the new drug approval requirements on condition that pharmacies not advertise or promote specific compounded drugs. The Court struck down the advertising condition, holding that although the government had a substantial interest in preserving the new drug approval process and in permitting compounding for individual patients, it had not shown that restricting speech was necessary to draw the line between them, since it could have used non-speech restrictions such as limits on volume. The Court's opinion noted pointedly that "if the First Amendment means anything, it means that regulating speech must be a last—not first—resort." The first skirmishes: Washington Legal Foundation The first direct challenge to the FDA's off-label policies came from the Washington Legal Foundation, a conservative public interest group, in the 1990s. The foundation challenged FDA guidance documents that restricted manufacturers' dissemination of journal articles and textbooks discussing off-label uses and their support of continuing medical education. In Washington Legal Foundation v. Friedman (1998), the federal district court in the District of Columbia held that the restrictions violated the First Amendment. It reasoned that the government had a substantial interest in encouraging manufacturers to seek approval for new uses, but that a complete ban on distributing peer-reviewed scientific information was more extensive than necessary, since the government could require disclosures, such as statements that the use was not approved and that the manufacturer had a financial interest. Congress had meanwhile enacted a limited safe harbor in the 1997 Modernization Act, permitting distribution of peer-reviewed articles about off-label uses if the manufacturer submitted a supplemental application for the use. On appeal, in Washington Legal Foundation v. Henney (2000), the government took the position that the statute and guidance created only a safe harbor, not an independent prohibition, and that it could still rely on dissemination of such materials as evidence of intended use in a misbranding case. The D.C. Circuit, finding no live constitutional controversy in light of that concession, vacated the district court's injunction to the extent it addressed the statute. The episode left the underlying question open, and the statutory safe harbor expired in 2006. The litigation had another lasting consequence. It established the pattern, which persists, of the agency defending its regime by characterizing speech as merely evidence of intended use rather than the violation itself. The government's position was, and remains, that it does not prohibit truthful speech but regulates the marketing of drugs for unapproved uses, a form of conduct that speech may help prove. Sorrell and the heightened scrutiny The next major development came from a case that had nothing to do with off-label promotion. In the 2000s, pharmaceutical companies purchased "prescriber-identifying information," data showing which physicians prescribed which drugs, from pharmacies through data-mining firms, and used it to target their sales efforts. Vermont enacted a law prohibiting the sale or use of such data for marketing purposes without the prescriber's consent, while permitting its use for other purposes. The state's avowed goal was to reduce the effectiveness of detailing by brand manufacturers and thereby encourage prescribing of generics. In Sorrell v. IMS Health Inc. (2011), the Supreme Court struck down the law by a vote of six to three. Justice Kennedy's opinion for the Court held that the law imposed content-based and speaker-based burdens on protected expression, since it disfavored marketing speech and disfavored speakers, namely pharmaceutical manufacturers. The Court said that such burdens warrant "heightened judicial scrutiny," and it concluded that the law failed even under the Central Hudson standard. It rejected the state's interest in reducing brand-name prescribing through restrictions on persuasive speech, stating that the fear that people would make bad decisions if given truthful information cannot justify content-based burdens on speech. Justice Breyer, dissenting, warned that the majority's approach threatened to return to the discredited era of Lochner v. New York by subjecting ordinary economic regulation to searching judicial review. Sorrell did not address off-label promotion, but its implications were obvious. The FDA's regime is content-based, since it distinguishes between speech about approved and unapproved uses, and speaker-based, since it applies to manufacturers but not to physicians, researchers or journalists who say the same things. If content-based and speaker-based burdens on truthful commercial speech warrant heightened scrutiny, the regime was vulnerable. Caronia The test came within a year. Alfred Caronia was a sales representative for Orphan Medical, the manufacturer of Xyrem, a drug approved to treat excessive daytime sleepiness and cataplexy in patients with narcolepsy. Xyrem's active ingredient is sodium oxybate, the sodium salt of gamma-hydroxybutyrate, or GHB, a substance with a history of abuse as a "date rape" drug, and the product carried a boxed warning. In conversations secretly recorded by the government, Caronia promoted Xyrem to a physician for unapproved uses including insomnia, fibromyalgia, restless leg syndrome and other conditions, and for use in patients younger than sixteen. He was convicted of conspiracy to introduce a misbranded drug into interstate commerce, a misdemeanor. In United States v. Caronia (2012), the Second Circuit vacated the conviction by a vote of two to one. The majority read the trial record as showing that the government had prosecuted Caronia for his speech itself, not merely used his speech as evidence of intended use; the prosecution's summation and the jury instructions had emphasized his promotion of off-label uses as the conduct at issue. To avoid the serious constitutional question that criminalizing such speech would raise under Sorrell, the court construed the misbranding provisions as not prohibiting or criminalizing the truthful off-label promotion of FDA-approved prescription drugs. It reasoned that the government's interests in preserving the approval process and protecting patients were substantial, but that prohibiting truthful promotion of lawful uses did not directly advance those interests, since off-label use itself is legal and the prohibition interfered with physicians' access to information about it. It also suggested less restrictive alternatives, such as guiding physicians to distinguish between approved and unapproved uses, developing warnings or disclaimers, or criminalizing only false or misleading promotion. The dissent argued that the majority's holding called into question the entire approval regime, since a manufacturer could now obtain approval for one narrow use and then promote the drug for any other use without further trials. The government did not seek Supreme Court review, apparently preferring to limit the decision to its facts and the Second Circuit rather than risk a nationwide adverse ruling. The FDA's public position after Caronia was that the decision did not significantly affect its enforcement, because it would continue to use speech as evidence of intended use and to pursue false or misleading promotion. Amarin and Pacira Companies soon went on the offensive. Amarin Pharma had obtained approval of Vascepa, a purified fish oil derivative, for patients with very high triglyceride levels. It then conducted a trial in patients with persistently high but lower triglyceride levels and sought approval for that population. The FDA declined, in part because a separate outcomes trial of a different triglyceride-lowering drug had failed to show cardiovascular benefit, raising doubt about whether lowering triglycerides in this population would help patients. Amarin wished to tell physicians truthfully about its trial results for the unapproved population and sued for a declaration that doing so would not expose it to misbranding liability. In Amarin Pharma, Inc. v. FDA (2015), Judge Paul Engelmayer of the Southern District of New York granted a preliminary injunction. He held that under Caronia, the FDA may not bring a misbranding action based on truthful and nonmisleading speech promoting off-label use, and he reviewed Amarin's proposed statements and disclosures, largely approving them as truthful and nonmisleading. The court emphasized that its ruling protected only speech that was truthful and not misleading, and that the government could still act against misleading promotion. The case settled in 2016 on terms that allowed Amarin to make the proposed communications and established a process for pre-review of future statements. In a postscript that illustrates the stakes, Amarin later conducted a large outcomes trial, REDUCE-IT, which showed a cardiovascular benefit, and the FDA approved Vascepa for cardiovascular risk reduction in 2019. The approved indication vindicated the drug, but it did so through the approval process that the company's speech claims had partially bypassed. A similar suit, Pacira Pharmaceuticals v. FDA, challenged the agency's attempt to restrict promotion of a long-acting local anesthetic, Exparel, which the agency asserted was approved only for particular surgical sites. The case settled in 2015, with the FDA rescinding a warning letter and approving revised labeling. After these cases, the government was notably reluctant to bring off-label enforcement actions premised on truthful speech. The agency's evolving guidance The FDA's response to the constitutional pressure has been to expand the zone of permitted communication through guidance, while preserving its position that it may regulate promotion of unapproved uses. In January 2017, in the final days of the Obama administration, the agency released a memorandum reviewing the public health interests at stake in off-label communications, defending its approach but acknowledging the First Amendment concerns. In 2018 it issued final guidance on communications consistent with FDA-required labeling, allowing companies to communicate information not contained in the label, such as long-term safety data or comparisons in populations within the approved indication, so long as it was consistent with the label. It also finalized guidance on payor communications implementing the 21st Century Cures Act. The most recent step was the agency's final guidance, issued in January 2025, on communications from firms to health care providers regarding scientific information on unapproved uses of approved or cleared medical products. The guidance, which replaced earlier documents on distributing reprints and textbooks, describes the circumstances in which the agency does not intend to treat a firm's proactive distribution of certain scientific information about unapproved uses as evidence of a new intended use. The information must come from scientifically sound studies or analyses, the communication must be truthful, nonmisleading and clinically relevant, and it must be presented with specified contextual disclosures, including that the use is unapproved and describing material limitations of the study. The guidance directs that such communications be separated from promotional materials and that sales personnel not be the ones to present them in a promotional manner. The guidance is a significant accommodation. It effectively concedes that manufacturers may share truthful, balanced science about unapproved uses with physicians. But it is guidance, not a regulation, and it does not purport to bind the agency or bar prosecution. It also depends on distinctions, between scientific exchange and promotion, between medical and commercial personnel, and between "clinically relevant" information and marketing, that may not hold up under a constitutional regime demanding narrow tailoring. After Loper Bright, and with the general deregulatory stance adopted by agency leadership in 2025, companies may press harder, and the agency may be less inclined to defend the traditional regime. At the same time, the government's 2025 campaign against direct-to-consumer advertising, which included a large wave of enforcement letters and a stated intention to revisit the rules governing broadcast advertisements, shows that the agency's appetite for regulating promotional speech has not disappeared; it has changed its targets. The arguments on each side Stripped to essentials, the argument for the traditional off-label regime rests on institutional design. Approval is a mechanism for generating reliable evidence, funded by the party that profits from the drug. If the manufacturer can promote any use that physicians may lawfully prescribe, the incentive to fund confirmatory trials for new uses collapses, and the evidence base for medicine deteriorates. Moreover, the line between truthful and misleading is notoriously difficult to police case by case. A study may be accurately described and yet present a misleading picture because of selective emphasis, omission of contrary studies, or subgroup analyses that would not survive replication. The FDA cannot review every sales call. A prophylactic rule, forbidding promotion of unapproved uses and forcing claims through the approval process, is therefore a reasonable way of protecting the integrity of information in a field where error kills. The argument against the regime rests on the logic of Sorrell and Western States. The government may not suppress truthful information because it fears that listeners, here highly trained physicians, will make poor decisions. Physicians may lawfully prescribe off-label, and patients benefit when they do so with the best available information. The government has alternatives that do not restrict speech: it may require disclosures, prohibit false or misleading statements, fund independent research, offer incentives for supplemental applications, and punish kickbacks. And the regime's reliance on speech as evidence of intent is a device for evading constitutional scrutiny: if the only evidence that a company intended an unapproved use is its truthful speech about that use, then the prosecution is, in substance, punishing speech. Both sides have a point, and the law has settled into an uneasy compromise. False or misleading promotion is unprotected and prosecutable. Kickbacks are unlawful regardless of speech. Truthful, balanced scientific communication to physicians is increasingly permitted, through guidance if not through law. And the government retains the theoretical power to prosecute promotion of unapproved uses when intent is shown by conduct as well as speech, as the Facteau prosecution demonstrated. What remains unsettled is whether the government may ever punish truthful and nonmisleading promotion of an unapproved use standing alone. No appellate court outside the Second Circuit has squarely adopted Caronia's reasoning, and the Supreme Court has not addressed it. A comparative note The American debate is unusual. In the European Union, promotion of medicinal products outside the terms of their authorized summary of product characteristics is prohibited by the Community Code relating to medicinal products for human use, and direct-to-consumer advertising of prescription medicines is prohibited altogether. Enforcement in many member states is conducted through industry self-regulatory codes, backed by national authorities, and constitutional challenges based on freedom of expression have had little traction. The United Kingdom follows a similar model, with the industry's Prescription Medicines Code of Practice Authority adjudicating complaints and publishing rulings. The contrast underlines how much of the American debate is a product of the distinctive strength of American free speech doctrine, and how little it reflects any consensus about what is good for patients. Speech and the label The First Amendment debate returns, in the end, to the label. The label is the government-approved statement of what a drug does. The off-label regime treats the label as the outer limit of a manufacturer's permissible claims, forcing new claims through the process that produced it. The constitutional challenge treats the label as one source of information among many, and asks why the manufacturer should be silenced about evidence that exists outside it. The law has not chosen between these conceptions. It has instead built a thicket of guidance, safe harbors and enforcement discretion around the boundary, which works tolerably well for sophisticated companies with large compliance departments and poorly for anyone trying to understand what the law actually requires. Chapter 7: After Approval: Surveillance, Safety Labeling and Recalls Approval is a judgment made with incomplete information. The trials that support it are too small, too short and too selective to reveal rare or delayed harms, and manufacturing problems can arise at any time in the life of a product. The postmarket system exists to catch what approval missed and to respond. It has two distinct functions, which are often confused. One is pharmacovigilance: detecting and responding to adverse effects inherent in the drug itself, usually by revising the label, restricting use, or, rarely, withdrawing the drug. The other is quality control: detecting and responding to defects in particular batches or manufacturing processes, usually through recalls. This chapter treats both, and emphasizes a feature of American law that surprises most people: the FDA's powers to remove a drug from the market are weaker, and more dependent on the manufacturer's cooperation, than the agency's public image suggests. Detecting harm The foundation of pharmacovigilance is adverse event reporting. Under section 314.80 of Title 21, manufacturers must report to the FDA every serious and unexpected adverse experience associated with their drugs, from any source, within fifteen calendar days of learning of it, and must submit periodic reports of other adverse experiences. These "fifteen-day alert reports" are among the most important obligations a manufacturer bears; failure to make them has been the basis for enforcement actions and is frequently alleged in product liability suits. Health care professionals and consumers may report voluntarily through the MedWatch program. The reports flow into the FDA Adverse Event Reporting System, known as FAERS, a database containing many millions of reports. Spontaneous reporting has well-known limits. Only a small fraction of adverse events are reported, and reporting is influenced by publicity, litigation and the novelty of the drug. Reports do not establish causation, since the patient may have experienced the event for reasons unrelated to the drug. And without knowing how many people took the drug, one cannot calculate the rate of an event. FAERS is best at detecting rare, severe, distinctive events, such as liver failure or a particular type of skin reaction, that are unlikely to occur spontaneously. It is poor at detecting modest increases in common events, such as heart attacks, which are frequent in the population regardless of any drug. That weakness was dramatically exposed by the case that reshaped postmarket law. Vioxx (rofecoxib), a selective COX-2 inhibitor marketed by Merck as a pain reliever with fewer gastrointestinal side effects than older anti-inflammatory drugs, was approved in 1999 and was prescribed to many millions of Americans. A large trial published in 2000 found more heart attacks among patients taking Vioxx than among those taking naproxen, a difference Merck attributed to a protective effect of naproxen. The debate over that interpretation continued until September 30, 2004, when Merck voluntarily withdrew the drug after a placebo-controlled trial of colon polyp prevention showed an increased risk of cardiovascular events after eighteen months of use. The withdrawal prompted congressional hearings, sharp criticism of both Merck and the FDA, and a wave of litigation that Merck resolved in 2007 with a settlement program of $4.85 billion for claims involving heart attacks and strokes. It also led to a report by the Institute of Medicine in 2006 recommending that the FDA be given stronger postmarket authorities. The 2007 powers The FDA Amendments Act of 2007 responded with a package of postmarket tools. The agency may require a manufacturer to conduct postmarket studies or clinical trials to assess a known serious risk, signals of a serious risk, or an unexpected serious risk, under section 505(o)(3); failure to comply can result in civil penalties and misbranding. It may require safety labeling changes under section 505(o)(4), as described in Chapter 4. It may require a Risk Evaluation and Mitigation Strategy at approval or later. And it was directed to establish an active surveillance system, which became the Sentinel System, drawing on electronic health records and insurance claims data covering many millions of patients. Sentinel allows the agency to query large populations to evaluate whether a suspected safety signal reflects a real increase in risk, compensating in part for the limits of spontaneous reporting. The 21st Century Cures Act further directed the agency to develop a framework for using real-world evidence, the product of such data, in regulatory decisions. These tools changed the practical balance of power. Before 2007, the FDA could request label changes and postmarket studies but had limited means to compel them, and manufacturers could resist for months or years. Now the agency can order changes and studies, with defined deadlines and penalties. In practice, most safety labeling changes are still negotiated, but the negotiation takes place in the shadow of the agency's order power. The agency communicates safety information through several channels: Drug Safety Communications, which are public notices describing new risks and the agency's response; quarterly postings of potential signals identified in FAERS; and changes to the label itself, which remain the authoritative expression of what is known. A label change is the characteristic outcome of pharmacovigilance. A boxed warning added to a widely used drug, such as those added to antidepressants in 2004 regarding suicidality in children and adolescents, or to fluoroquinolone antibiotics regarding tendon rupture and other serious effects, can change prescribing practice and supply the basis for litigation. Label changes can also run in the other direction, and they are not immune to political pressure. In September 2025, federal health officials announced that the FDA would begin the process of changing the labeling of acetaminophen to reflect a possible association between prenatal use and neurodevelopmental conditions, including autism. The announcement was controversial because the epidemiological evidence is contested and professional bodies in obstetrics had long regarded acetaminophen as the preferred pain reliever in pregnancy when used appropriately. The episode showed that the label change process, which ordinarily proceeds through scientific review and negotiation with manufacturers, can be driven from the top of the department, and it raised questions, still unresolved, about how such changes will interact with the pending and future litigation over the same risk. Withdrawal of approval When a label change is not enough, the ultimate remedy is withdrawal of approval. Section 505(e) authorizes the Secretary to withdraw approval after notice and opportunity for a hearing on specified grounds, including that new evidence shows the drug is unsafe for use under the conditions of its approval, or that there is a lack of substantial evidence of effectiveness. If the Secretary finds an "imminent hazard to the public health," approval may be suspended immediately, with a hearing to follow; that power has been used only once, in 1977, for the diabetes drug phenformin. In practice, formal withdrawal proceedings are rare, slow and cumbersome, as the Makena case discussed in Chapter 3 illustrated. Most drugs removed from the market for safety reasons are withdrawn voluntarily by the manufacturer, usually after discussion with or pressure from the agency. The diet drug combination known as fen-phen was effectively ended in 1997 when manufacturers withdrew fenfluramine and dexfenfluramine at the FDA's request after reports of heart valve disease. Merck's withdrawal of Vioxx was voluntary. In 2020, the FDA requested that manufacturers withdraw all ranitidine (Zantac) products from the market because of concerns about the probable human carcinogen N-nitrosodimethylamine (NDMA), and manufacturers complied. Voluntary withdrawal is attractive to companies because it avoids a formal finding that the drug is unsafe, which would be powerful evidence in litigation, and to the agency because it is faster than any formal process. The anatomy of a recall Recalls address a different problem. A recall, as defined in part 7 of Title 21 of the Code of Federal Regulations, is a firm's removal or correction of a marketed product that the FDA considers to be in violation of the laws it administers and against which the agency would initiate legal action. Most drug recalls arise from manufacturing and quality defects: contamination, subpotency or superpotency, failed dissolution, mislabeling, packaging errors, or the presence of impurities. They typically concern particular lots, not the drug as a whole. The central legal fact about drug recalls is that they are, with narrow exceptions, voluntary. The Food, Drug, and Cosmetic Act gives the FDA mandatory recall authority for medical devices, infant formula, food (since the Food Safety Modernization Act of 2011), tobacco products and certain other categories, and the Public Health Service Act permits the recall of biological products that present an imminent or substantial hazard. But the Act contains no general authority to order the recall of a prescription drug. When the agency wants a drug recalled, it asks. If the company refuses, the agency's recourse is to seek a court order for seizure of the product or an injunction, to issue public warnings, or to take action against the company's other applications and facilities. Because these tools are powerful and the reputational costs of refusal are high, refusal is rare, and the voluntary system works most of the time. But it depends on cooperation, and it places the initial judgment about whether a problem warrants recall in the hands of the manufacturer. When a recall occurs, the FDA classifies it according to the health hazard presented, as Table 4 sets out. The classification determines the urgency and depth of the recall and the extent of public notice. Table 4. FDA recall classifications under 21 C.F.R. Part 7. Class Definition Typical drug examples Class I Reasonable probability of serious adverse health consequences or death Contaminated sterile products; wrong drug in package Class II May cause temporary or medically reversible adverse consequences, or serious consequences are remote Subpotency; impurities above limits Class III Not likely to cause adverse health consequences Minor labeling or packaging defects The recalling firm must develop a recall strategy addressing the depth of the recall (whether to the wholesale, retail or consumer level), the need for public warnings, and effectiveness checks to verify that recipients received notice and acted on it. The FDA monitors recalls, publishes them in its weekly Enforcement Report, and terminates a recall when it determines that all reasonable efforts have been made to remove or correct the product. Firms must also submit reports to the FDA under separate rules when they learn of certain quality defects in distributed products, known as field alert reports, within three working days. The distinction between recall and withdrawal of approval matters for liability. A recall is not an admission that the drug is unsafe as designed or that anyone was harmed; it is typically a response to a manufacturing deviation. Plaintiffs who allege harm from a recalled lot must prove that the defect caused their injury, which is often difficult when the defect was a matter of impurities at low levels or potency outside specification. The nitrosamine episodes The most significant recall episodes of recent years illustrate both the strengths and the weaknesses of the system. In July 2018, several manufacturers of the blood pressure drug valsartan recalled products after the discovery that active ingredient made by a Chinese manufacturer, Zhejiang Huahai Pharmaceutical, contained NDMA, a nitrosamine classified as a probable human carcinogen. The impurity had apparently been introduced by a change in the manufacturing process years earlier. The recalls spread to other drugs in the same class, losartan and irbesartan, and over the following years to other drugs, including ranitidine and certain forms of the diabetes drug metformin. For ranitidine, the problem appeared to be the instability of the molecule itself, which could degrade into NDMA over time and at higher temperatures, which is why the agency requested withdrawal of the entire product class rather than particular lots. These episodes exposed the globalized supply chain on which American drug supply depends. A large share of active pharmaceutical ingredients, especially for generic drugs, is manufactured in China and India, and FDA inspection of foreign facilities has historically been less frequent than of domestic ones, and often announced in advance. The agency responded by issuing guidance requiring manufacturers to assess their products for nitrosamine risk, and later guidance on nitrosamine drug substance-related impurities, which set acceptable intake limits and compliance timelines. It also increased unannounced foreign inspections. The nitrosamine problem also produced massive litigation, discussed in Chapter 8: a multidistrict litigation over valsartan in the District of New Jersey, and federal and state litigation over ranitidine that turned on whether NDMA exposure from the drug caused cancer. Contamination and compounding Other episodes involved outright contamination. In 2012, contaminated injectable methylprednisolone acetate produced by the New England Compounding Center in Framingham, Massachusetts, caused a nationwide outbreak of fungal meningitis that killed more than sixty people and sickened hundreds more. The pharmacy was a compounder, operating under a regime of state pharmacy regulation and uncertain federal oversight. Its owner and supervising pharmacist were convicted of federal crimes, including racketeering and fraud, though juries acquitted them of second-degree murder charges. Congress responded with the Drug Quality and Security Act of 2013, which created a category of "outsourcing facilities" subject to federal good manufacturing practice requirements and inspection, while preserving traditional pharmacy compounding under state oversight. The same Act established a national system for tracing prescription drugs through the supply chain, with electronic, interoperable tracing requirements phased in over a decade. In 2023, artificial tear eye drops sold under the EzriCare and Delsam Pharma brands were recalled after being linked to an outbreak of a drug-resistant strain of Pseudomonas aeruginosa that caused deaths, blindness and loss of eyes. The products were manufactured in India. The episode led to a broader FDA effort to warn consumers about contaminated eye products and to increase scrutiny of imported over-the-counter drugs. Compounding became a live issue again with the shortage of GLP-1 drugs for diabetes and weight loss. When semaglutide and tirzepatide were on the FDA's drug shortage list, compounders were permitted to produce copies, and a large market in compounded versions, sold through telehealth companies, grew up. After the agency declared the shortages resolved, in December 2024 for tirzepatide and February 2025 for semaglutide, it set transition periods after which such mass compounding of copies would no longer be permitted, and trade groups of compounders sued unsuccessfully to challenge the determinations. The episode illustrates how shortage declarations, a seemingly technical regulatory category, can open and close large markets, and how quality concerns about compounded products can become a significant public health issue. Why the system relies on cooperation The absence of mandatory drug recall authority is often described as a gap, and Congress has considered proposals to fill it. But the existing system reflects a certain logic. Drug manufacturers, unlike many food producers, are few, heavily regulated, dependent on continuing approvals and inspections, and highly sensitive to reputation and liability. The agency's informal leverage over them is enormous. Mandatory authority would matter mainly at the margins, in cases involving small, marginal or foreign firms that care little about their relationship with the agency, and in such cases seizure and import alerts may be more practical than recall orders. The deeper weakness is not the absence of a legal power but the dependence of the system on information the manufacturer controls. The manufacturer designs the process, tests the batches, receives the complaints and adverse event reports, and decides in the first instance whether a deviation warrants a recall or a signal warrants a label change. The FDA's inspections and surveillance systems provide an independent check, but they are limited in resources, and staff reductions in 2025 raised concerns about the agency's capacity to maintain inspection and review functions. Where the manufacturer's incentives diverge from the public's, the postmarket system depends on something outside itself to align them. That something is civil liability, the subject of the next chapter. Chapter 8: Failure to Warn: Civil Liability for Adverse Drug Effects When a patient is injured by a prescription drug, the regulatory system offers no compensation. The FDA can change the label, restrict the drug or ask for its withdrawal, but it cannot award damages. Compensation, and much of the deterrence that shapes manufacturers' behavior, comes from state tort law, administered by juries in state and federal courts. The dominant theory in prescription drug litigation is failure to warn: the claim that the manufacturer knew or should have known of a risk and did not adequately disclose it. This chapter explains how that claim works, why it is harder to win than public perception suggests, and how mass litigation has transformed it. Why warnings, and not design Product liability law generally recognizes three kinds of defect: manufacturing defects, in which a particular unit departs from its intended design; design defects, in which the design itself is unreasonably dangerous; and warning defects, in which the product lacks adequate instructions or warnings. Prescription drugs occupy a special place in this scheme. Manufacturing defect claims are available when a batch is contaminated or otherwise departs from specifications, as in the recall episodes described in Chapter 7. Design defect claims against prescription drugs are difficult. The Restatement (Second) of Torts, in comment k to section 402A, recognized that some products are "unavoidably unsafe," meaning incapable of being made safe for their intended use, and that their sellers are not strictly liable if the products are properly prepared and accompanied by proper warnings. Many courts have applied comment k to prescription drugs generally, and the Restatement (Third) of Torts: Products Liability, adopted in 1997, provides in section 6(c) that a prescription drug is defectively designed only if its foreseeable risks are so great in relation to its benefits that reasonable health care providers would not prescribe it for any class of patients. That standard is almost impossible to meet for an FDA-approved drug, and the Supreme Court held in Mutual Pharmaceutical Co. v. Bartlett (2013) that design defect claims against generic drugs are generally preempted, a subject taken up in Chapter 9. The result is that failure to warn has become the principal theory in drug litigation. The learned intermediary The distinctive feature of failure-to-warn law for prescription drugs is the learned intermediary doctrine. Under this doctrine, adopted in some form by nearly every American jurisdiction, the manufacturer's duty to warn runs to the prescribing physician, not to the patient. The rationale is that prescription drugs are available only through physicians, who are in the best position to evaluate risks and benefits for the individual patient and to communicate them in the context of the patient's condition. The label, written in technical language for physicians, is the vehicle of that warning. The Restatement (Third), in section 6(d), adopts the doctrine, with an exception where the manufacturer knows or has reason to know that health care providers will not be in a position to reduce the risk of harm, for example in mass immunization programs. The doctrine has several important consequences. The adequacy of a warning is judged from the perspective of a reasonable physician, not a layperson. The manufacturer satisfies its duty if the physician is adequately warned, even if the physician fails to relay the warning to the patient. And causation often turns on the prescribing physician: a plaintiff must generally show that a different warning would have changed the physician's decision to prescribe, or the way the drug was prescribed or monitored. If the physician testifies that he was already aware of the risk, or that he would have prescribed the drug anyway, the claim may fail. The doctrine has come under pressure from the rise of direct-to-consumer advertising, which the FDA effectively permitted in broadcast media after a 1997 guidance clarified how television advertisements could satisfy disclosure requirements. If manufacturers speak directly to patients, encouraging them to ask for drugs by name, the premise that the physician stands between manufacturer and patient is weakened. In Perez v. Wyeth Laboratories (1999), the Supreme Court of New Jersey held that the learned intermediary doctrine does not apply when a manufacturer markets directly to consumers, at least where the advertising fails to adequately warn, reasoning that manufacturers who bypass the physician cannot rely on the physician as a shield. Most other jurisdictions have declined to adopt such an exception, but the argument recurs, and the growth of telehealth prescribing models, in which a manufacturer's marketing leads a patient to an online platform where a prescription is quickly issued, has given it new force. Some courts have also recognized other exceptions: for oral contraceptives, where the Supreme Judicial Court of Massachusetts held in MacDonald v. Ortho Pharmaceutical Corp. (1985) that the manufacturer had a duty to warn patients directly, given the FDA's patient package insert requirement and the limited role of physicians after the initial prescription; and for mass vaccinations. The existence of FDA-required medication guides has led some plaintiffs to argue that the manufacturer's duty extends to their content, though courts have generally continued to measure the duty by the physician label. What makes a warning adequate A warning's adequacy is ordinarily a question of fact for the jury. Courts have identified several dimensions along which it may be judged: whether it addresses the specific risk that caused the injury, not merely a related one; whether it conveys the risk's nature, gravity and likelihood with sufficient intensity; whether it is placed where a physician would be expected to see it; and whether it is undercut by the manufacturer's promotional activity, a practice sometimes called "overpromotion." A warning buried in the adverse reactions table may be inadequate for a risk that should have appeared in the warnings section or a boxed warning. A warning that describes a risk as rare, when evidence available to the manufacturer suggested it was common, may be inadequate even though it mentions the risk. The duty to warn is continuing. It extends to risks that the manufacturer knew or should have known of, and the manufacturer is held to the knowledge of an expert in the field. It has a duty to monitor the literature and adverse event reports and to update its warnings as knowledge develops. This is where regulatory and tort law interlock most directly. Evidence that the manufacturer received adverse event reports and did not report them to the FDA, or conducted analyses showing a signal and did not act, or resisted the agency's requests for label changes, is central to failure-to-warn trials. Conversely, evidence that the manufacturer promptly reported everything and that the FDA reviewed the evidence and decided not to require a warning is a powerful defense, and in some circumstances, as the next chapter shows, a complete one. Compliance with FDA requirements is not, in most states, a complete defense. The traditional rule is that regulatory standards are minimum standards and that a manufacturer may be negligent even if it complied with them. A few states have enacted statutes that alter this rule. Michigan, for many years, barred most product liability claims against manufacturers of FDA-approved drugs, subject to exceptions for fraud on the agency and bribery; the state legislature repealed that immunity in recent legislation. Texas has a statutory presumption that warnings approved by the FDA are adequate, rebuttable only on narrow grounds. Several states limit or bar punitive damages for FDA-approved drugs absent fraud on the agency. In most jurisdictions, however, a jury may find an FDA-approved label inadequate. Proving causation The hardest element in most drug cases is causation, and it has two parts. General causation asks whether the drug is capable of causing the injury alleged in the population at large. Specific causation asks whether it caused the injury in this plaintiff. Both typically require expert testimony, and the admissibility of that testimony is often the decisive battle. Under Federal Rule of Evidence 702, as interpreted in Daubert v. Merrell Dow Pharmaceuticals, Inc. (1993), federal judges act as gatekeepers, admitting expert testimony only if it is based on sufficient facts or data, is the product of reliable principles and methods, and reflects a reliable application of those methods to the facts. Daubert itself was a drug case, involving claims that the anti-nausea drug Bendectin caused birth defects, and the Bendectin litigation became a byword for mass tort claims unsupported by epidemiological evidence; large, well-designed studies eventually found no association, but only after years of litigation had led the manufacturer to withdraw the drug from the market in 1983. In General Electric Co. v. Joiner (1997), the Supreme Court held that trial judges' admissibility decisions are reviewed for abuse of discretion, and that a court may exclude opinion testimony connected to existing data "only by the ipse dixit of the expert." In December 2023, Rule 702 was amended to emphasize that the proponent must establish admissibility by a preponderance of the evidence and that each expert opinion must reflect a reliable application of methodology to facts, an amendment intended to correct what the advisory committee regarded as decisions treating these as questions of weight for the jury rather than admissibility. General causation evidence typically consists of epidemiological studies, supplemented by animal studies, mechanistic evidence and case reports. Courts have debated how to treat studies whose results are not statistically significant, whether a relative risk greater than two is required to show that it is more likely than not that the drug caused a particular injury, and how to assess the "weight of the evidence" methodology, in which an expert reasons from multiple lines of evidence to a causal conclusion. The Bradford Hill criteria, a set of considerations for assessing causation from epidemiological associations articulated by the British epidemiologist Austin Bradford Hill in 1965, are frequently invoked and frequently disputed. Recent litigation shows how decisive these questions are. In the federal ranitidine litigation, consolidated in the Southern District of Florida, the court in December 2022 excluded the plaintiffs' general causation experts, concluding that their methodologies were unreliable, and entered judgment against thousands of federal claimants. Parallel litigation continued in state courts, where the admissibility standards and results differed; a Delaware trial court admitted plaintiffs' experts, but the Delaware Supreme Court reversed in 2025, holding that the trial court had applied an overly permissive standard. Before that decision, GlaxoSmithKline announced in 2024 that it would settle the large majority of pending state court cases against it for up to $2.2 billion, without admitting liability. In the acetaminophen litigation, consolidated in the Southern District of New York, the court in December 2023 excluded the plaintiffs' experts on whether prenatal acetaminophen exposure causes autism and attention-deficit/hyperactivity disorder, and the plaintiffs appealed. The federal government's 2025 announcement concerning acetaminophen labeling, described in Chapter 7, injected an unusual element into that litigation, since regulatory statements are not scientific evidence but may be offered as bearing on what a reasonable manufacturer should have known. Mass torts and the MDL Most significant drug litigation today is mass litigation. When thousands of patients claim injury from the same drug, federal cases are typically transferred by the Judicial Panel on Multidistrict Litigation to a single district judge for coordinated pretrial proceedings, under the federal multidistrict litigation statute. Parallel state court cases may be consolidated in state coordinated proceedings, such as New Jersey's multicounty litigation or California's judicial council coordinated proceedings. Multidistrict litigations now account for a large share of the pending federal civil docket, and product liability cases involving drugs and devices make up much of that share. The MDL process typically proceeds through consolidated discovery, rulings on preemption and expert admissibility, and "bellwether" trials of selected cases intended to give the parties information about the value of claims. Most MDLs resolve through global or inventory settlements, often administered through claims programs with eligibility criteria. The Vioxx settlement of 2007 was an early model, requiring a high percentage of eligible claimants to enroll before the settlement became effective. Mass drug litigation has since involved, among many others, diabetes drugs, antipsychotics, testosterone replacement, blood thinners, heartburn medications, and hormonal contraceptives. Litigation over GLP-1 receptor agonists such as semaglutide and tirzepatide, alleging failure to warn of gastrointestinal conditions such as gastroparesis and later of vision loss, was consolidated in a federal MDL in the Eastern District of Pennsylvania in 2024. Litigation over the injectable contraceptive Depo-Provera, alleging failure to warn of a risk of meningioma, a usually benign brain tumor, was consolidated in the Northern District of Florida in 2025. The MDL system has critics on both sides. Defendants argue that it attracts large numbers of weak or unvetted claims, generated by advertising, because the cost of filing is low and settlements rarely distinguish sharply among claims. A new Federal Rule of Civil Procedure, Rule 16.1, which took effect in December 2025, encourages early management of MDLs, including early exchange of information about the factual bases of claims. Plaintiffs argue that the system gives defendants enormous leverage, delays resolution for years, and allows a single judge's expert rulings to extinguish thousands of claims. The tension between efficiency and individual justice is inherent in aggregation, and the drug cases are where it is most acute. Punitive damages and the deterrence question Failure-to-warn cases often seek punitive damages, which require proof of conduct more culpable than negligence, typically malice, fraud, or conscious disregard of safety. Internal documents are the key: evidence that the company knew of a risk and chose to suppress it for commercial reasons. Some of the largest verdicts in drug litigation have been punitive. The Supreme Court's due process decisions, particularly State Farm Mutual Automobile Insurance Co. v. Campbell (2003), which suggested that single-digit ratios of punitive to compensatory damages will usually satisfy due process, have constrained awards. And state caps and FDA-compliance defenses limit them further. Whether tort liability improves drug safety is debated. Its defenders point to documents revealed in litigation, such as the evidence in the Vioxx cases about Merck's handling of cardiovascular data, as information that regulators did not have, and to the incentive that liability creates for manufacturers to investigate and disclose risks. Critics argue that juries are poorly equipped to evaluate scientific evidence, that liability is driven by publicity rather than evidence, as the Bendectin experience suggests, and that the threat of litigation discourages development of drugs for conditions, such as pregnancy-related ones, where liability risks are high. Both views contain truth. The system's defenders are right that manufacturers' knowledge often exceeds the agency's, and that tort law extracts that knowledge. Its critics are right that the causal evidence in many mass torts is weak and that a litigation system in which weak and strong claims are resolved together sends a muddy signal. Liability beyond the manufacturer Pharmacists, physicians and hospitals may also face liability. Physicians who fail to warn patients of risks disclosed in the label, or who prescribe negligently, may be sued for malpractice, and in some cases the learned intermediary doctrine shifts responsibility from manufacturer to physician. Pharmacists have traditionally been held to a narrow duty to fill prescriptions accurately, though some courts have recognized broader duties when a pharmacist has specific knowledge of a danger to a particular patient. A distinctive issue concerns vaccines. The National Childhood Vaccine Injury Act of 1986 created a no-fault compensation program, funded by an excise tax on vaccines, for injuries caused by covered childhood vaccines, and it requires claimants to pursue that program before suing manufacturers. In Bruesewitz v. Wyeth LLC (2011), the Supreme Court held that the Act preempts all design defect claims against vaccine manufacturers. Countermeasures against public health emergencies, including COVID-19 vaccines and treatments, are covered by the Public Readiness and Emergency Preparedness Act, which provides broad immunity and a separate, much more limited compensation program. These regimes show that Congress has been willing, for particular products whose social value depends on broad uptake, to replace tort liability with administrative compensation. Whether that model should be extended is a recurring policy debate, and the political scrutiny of vaccine safety that intensified in 2025 has made the future of the vaccine compensation system itself a subject of controversy. The warning as the measure of liability The pattern that emerges from this chapter is that the label is not only the regulatory statement of a drug's risks but also the standard by which the manufacturer's legal responsibility is judged. Failure-to-warn law asks, in effect, whether the label should have said more, earlier, or more prominently, given what the manufacturer knew or should have known. That question can be answered only by comparing the label's content with the evidence available at the time, which is why litigation revolves around documents and adverse event reports, and why the regulatory history of the label is always at the center of trial. But the question also raises a further problem. If the FDA approved the label, and if the FDA controls what the label may say, can a jury in one state find that label inadequate? That is the problem of preemption. Chapter 9: Who Controls the Words: Federal Preemption The Constitution's Supremacy Clause makes federal law "the supreme Law of the Land," and state law that conflicts with it must yield. When a patient sues a drug manufacturer claiming that its FDA-approved label was inadequate, the manufacturer often responds that the claim is preempted: that state law cannot require a label different from the one federal law approved. Between 2009 and 2019, the Supreme Court decided four cases on this question, and the result is a body of doctrine with a stark and widely criticized consequence. Patients injured by brand-name drugs can usually sue; patients injured by the generic versions of the same drugs, which account for the great majority of prescriptions filled, usually cannot. This chapter explains how that happened and argues that it follows directly from the law's organizing principle: liability follows control of the label. The forms of preemption Preemption comes in several forms. Express preemption occurs when Congress states that federal law displaces state law, as it did for medical devices in the Medical Device Amendments of 1976, which prohibit states from imposing requirements "different from, or in addition to" federal requirements for devices. Field preemption occurs when federal regulation is so pervasive that Congress is inferred to have left no room for the states. Conflict preemption occurs when it is impossible to comply with both state and federal law, or when state law stands as an obstacle to the accomplishment of federal objectives. For prescription drugs, there is no express preemption provision. When Congress amended the Act in 1962, it included a savings clause providing that the amendments should not be construed as invalidating any state law unless there is a direct and positive conflict. And when Congress added express preemption for medical devices in 1976 and for nonprescription drugs in 1997, it declined to do so for prescription drugs. Preemption of drug claims therefore depends on implied conflict preemption, and in particular on the "impossibility" branch: whether the manufacturer could have complied with both the state-law duty to warn and federal law. The contrast with devices is instructive. In Riegel v. Medtronic, Inc. (2008), the Supreme Court held that the express preemption provision for devices bars state-law claims challenging the safety or effectiveness of devices that received premarket approval, the most rigorous form of device review, because the approval imposes device-specific federal requirements with which state tort duties would be "different from, or in addition to." Most device claims involving approved products are thus preempted, except to the extent they allege violations of federal requirements themselves, the so-called "parallel claims." Many observers expected that the Court would reach a similar result for drugs. It did not. Wyeth v. Levine: brand manufacturers can be sued Diana Levine, a professional guitarist in Vermont, went to a clinic for treatment of a migraine headache and received Phenergan, an anti-nausea drug manufactured by Wyeth, by intravenous push injection. The drug was inadvertently injected into an artery, or escaped from the vein into surrounding tissue, and caused gangrene; her forearm had to be amputated. The label warned of the risk of gangrene from intra-arterial injection and cautioned practitioners about the method of administration, but did not prohibit IV push administration in favor of the safer IV drip method. A Vermont jury found that the label was inadequate and awarded her damages. In Wyeth v. Levine (2009), the Supreme Court held, by a vote of six to three, that the claim was not preempted. Justice Stevens's majority opinion rested on the changes-being-effected regulation discussed in Chapter 4. Because the regulation allowed Wyeth to strengthen its warning unilaterally, before FDA approval, it was not impossible for Wyeth to comply with both federal law and a state-law duty to provide a stronger warning. The Court acknowledged that the FDA could reject a CBE change, but held that absent "clear evidence" that the FDA would not have approved the stronger warning, the Court would not conclude that compliance with both laws was impossible. The Court emphasized that the manufacturer, not the FDA, bears primary responsibility for its label at all times. The Court also rejected Wyeth's obstacle-preemption argument, which relied on a preamble the FDA had published in 2006 with its Physician Labeling Rule, asserting that FDA approval of labeling preempts conflicting state law. The Court gave the preamble no weight, noting that it had been issued without notice and comment and reversed the agency's longstanding position. And it observed that Congress, which had been aware of state tort litigation for decades, had never enacted express preemption for drugs, suggesting that it viewed state tort law as a complementary form of drug regulation. State tort suits, the Court said, "uncover unknown drug hazards and provide incentives for drug manufacturers to disclose safety risks promptly." PLIVA v. Mensing: generic manufacturers usually cannot Two years later, the Court confronted the same question for generic drugs. Gladys Mensing and Julie Demahy took generic metoclopramide, a drug for gastrointestinal disorders, and developed tardive dyskinesia, a severe and often irreversible neurological disorder causing involuntary movements. They alleged that the generic manufacturers had failed to strengthen the warnings on their labels despite mounting evidence that long-term use carried a much higher risk than the label indicated. In PLIVA, Inc. v. Mensing (2011), the Court held, five to four, that the claims were preempted. The key difference from Wyeth was the federal duty of sameness. Under the Hatch-Waxman framework, a generic manufacturer's label must be the same as the brand-name reference drug's. The FDA interpreted its regulations to mean that generic manufacturers could not use the CBE process to change their labels unilaterally, and that they could not send "Dear Doctor" letters to physicians conveying new warnings, because such letters are themselves labeling and would have to be consistent with the approved label. The generic manufacturers could only ask the FDA to require the brand manufacturer to change the label. Because state law required a stronger warning and federal law prohibited the generic manufacturers from providing one independently, it was impossible to comply with both, and state law had to yield. The Court rejected the argument that the manufacturers could have asked the FDA for help, holding that the possibility of obtaining the federal government's permission does not defeat impossibility. The majority, in an opinion by Justice Thomas, candidly acknowledged the anomaly, conceding that from the plaintiffs' perspective, finding preemption in their cases but not in Wyeth made little sense. Had they taken the brand-name drug, they could have sued; because pharmacists had filled their prescriptions with generics, as state law often encourages or requires, they could not. The Court said that this result followed from the statutory scheme and that Congress and the FDA could change it. Justice Sotomayor, in dissent, argued that the majority had invented a new and broad impossibility standard, and that the consequences would be arbitrary for consumers. Mutual Pharmaceutical v. Bartlett: design claims too The third case extended Mensing to design defect claims. Karen Bartlett took a generic version of sulindac, a nonsteroidal anti-inflammatory drug, and developed toxic epidermal necrolysis, a devastating reaction in which much of her skin sloughed off; she was left severely disabled and nearly blind. The label at the time she was prescribed the drug did not specifically refer to the condition, though it warned of serious skin reactions. The case went to trial in New Hampshire on a design defect theory, the court having dismissed her failure-to-warn claim because her physician had not read the label, and the jury awarded over $21 million. In Mutual Pharmaceutical Co. v. Bartlett (2013), the Court held, five to four, that the design defect claim was preempted. New Hampshire's design defect law, the Court reasoned, imposed a duty either to make the drug safer or to provide stronger warnings. The generic manufacturer could not redesign the drug, since federal law requires that it be chemically the same as the brand, and it could not change the label. The Court rejected the argument that the manufacturer could comply with both laws by simply ceasing to sell the drug, holding that the "stop-selling" rationale would make impossibility preemption "all but meaningless." Justice Alito's majority opinion expressed sympathy for Bartlett but again said the remedy lay with Congress. Merck v. Albrecht: what counts as clear evidence The final case clarified the "clear evidence" exception in Wyeth. Patients who took Fosamax, Merck's osteoporosis drug, suffered atypical femoral fractures, a rare type of thigh bone break, and alleged that Merck failed to warn of the risk. Merck argued that the FDA would have rejected a stronger warning, pointing to the fact that in 2008 it had submitted a proposed label change discussing stress fractures, and the FDA had rejected the proposed language, saying that Merck's justification was inadequate. Two years later, the FDA required a warning about atypical femoral fractures. In Merck Sharp & Dohme Corp. v. Albrecht (2019), the Court held that preemption is a question of law for the judge, not the jury, and that to establish the "clear evidence" defense a manufacturer must show that it fully informed the FDA of the justifications for the warning required by state law and that the FDA, in turn, informed the manufacturer that it would not approve a change to the label to include that warning. The agency's action must be one taken pursuant to its congressionally delegated authority to approve or reject label changes, such as a formal rejection or notice-and-comment rulemaking. Informal communications or the mere fact that the FDA did not require a warning would not suffice. The Court remanded for the lower courts to apply the standard, and the Fosamax litigation returned to the lower courts, where it continued for years. Albrecht made the preemption defense for brand manufacturers narrow. A manufacturer must show that the specific warning plaintiffs demand was put to the FDA with full information and formally rejected. That will occasionally be possible, but it is rare, because manufacturers seldom propose warnings that the agency rejects, and when they do, plaintiffs typically argue that the proposal was inadequate or the information incomplete. The four decisions are summarized in Table 5. Table 5. The Supreme Court's prescription drug preemption decisions. Case Year Product type Claim Holding Wyeth v. Levine 2009 Brand Failure to warn Not preempted; manufacturer could use CBE PLIVA v. Mensing 2011 Generic Failure to warn Preempted; duty of sameness Mutual Pharmaceutical v. Bartlett 2013 Generic Design defect Preempted; cannot redesign or relabel Merck v. Albrecht 2019 Brand Failure to warn Preemption for judge; clear evidence requires formal FDA rejection The claims that survive, and the claims that are barred Around these holdings, lower courts have developed a detailed body of law about which claims survive. Against generic manufacturers, courts have allowed some claims based on the failure to update the label to match a brand label change, since federal law required the generic to make that change; claims based on manufacturing defects; and in some jurisdictions claims based on failure to communicate approved warnings, though most courts have rejected the last theory because communications would themselves be new labeling. Against brand manufacturers, failure-to-warn claims generally proceed unless the clear evidence defense is established. Design defect claims against brand manufacturers raise their own preemption questions, since a brand manufacturer also cannot change the chemical composition of an approved drug without FDA approval. Several circuits have held such claims preempted to the extent they rest on a theory that the manufacturer should have sold a different drug after approval, while leaving open claims that the manufacturer should have designed the drug differently before seeking approval. A separate line of preemption doctrine, drawn from Buckman Co. v. Plaintiffs' Legal Committee (2001), bars state-law claims that exist solely by virtue of federal requirements. In Buckman, plaintiffs claimed that a consultant had made fraudulent representations to the FDA in obtaining clearance for orthopedic bone screws. The Court held that such "fraud-on-the-FDA" claims are impliedly preempted, because policing fraud against federal agencies is a federal function and state claims would interfere with the agency's own enforcement discretion. After Buckman, plaintiffs must frame claims as violations of traditional state-law duties, not as enforcement of the Act, which contains no private right of action. The interplay between Buckman and the express preemption for devices creates a "narrow gap" through which device claims must pass: they must parallel federal requirements, but may not exist solely because of them. The generic gap and attempts to close it The consequence of Mensing and Bartlett is that the large majority of prescription drug users have little or no tort remedy for warning defects. Several responses have been attempted. The FDA proposed a rule in 2013 that would have allowed generic manufacturers to update their labels through the CBE process, temporarily creating labeling differences between brand and generic products until the FDA resolved them. The agency explained that the proposal would give generic manufacturers the same ability and responsibility as brand manufacturers to update warnings, which would have the effect of exposing them to failure-to-warn liability. Generic manufacturers strongly opposed the rule, arguing that it would create confusion, raise costs and increase liability exposure, and the agency ultimately withdrew it in 2018 without finalizing it. Legislation to the same effect has been introduced in Congress but not enacted. Plaintiffs have also pursued "innovator liability," the theory that a brand manufacturer can be held liable for injuries caused by a generic version of its drug, because the generic's label is a copy of the brand's and the brand manufacturer controls its content. Most courts have rejected the theory, holding that a manufacturer owes no duty to people who did not use its product. But a few courts have accepted it. In Conte v. Wyeth, Inc. (2008), a California appellate court held that the brand manufacturer of metoclopramide could be liable for negligent misrepresentation to a plaintiff who took the generic, because the brand manufacturer could foresee that physicians would rely on its label in prescribing the generic. In T.H. v. Novartis Pharmaceuticals Corp. (2017), the Supreme Court of California held that a brand manufacturer could owe a duty of care to patients injured by the generic version of its drug, even after it had sold its rights to the brand product, where the alleged deficiency existed while it controlled the label. The Supreme Court of Alabama reached a similar conclusion in Wyeth, Inc. v. Weeks (2014), and the Alabama legislature promptly abrogated the decision by statute in 2015. The Supreme Judicial Court of Massachusetts, in Rafferty v. Merck & Co. (2018), rejected liability for negligence but allowed claims for recklessness where a brand manufacturer intentionally failed to update its label despite knowledge of an unreasonable risk of death or grave bodily injury. Innovator liability thus remains a minority rule, dependent on state law. The logic and its critics The preemption decisions have been heavily criticized, and their critics have a point. The distinction between brand and generic users is arbitrary from the perspective of patients, who usually have no control over which product the pharmacy dispenses. It also removes incentives for generic manufacturers, many of which have become large companies with substantial pharmacovigilance capacity, to monitor safety and press for label changes. And it leaves a significant category of injuries uncompensated. But the decisions also follow from a coherent principle, which is the principle this book has traced from the beginning: legal responsibility for a label follows control of the label. The brand manufacturer drafts its label, can strengthen it without prior approval, and bears responsibility for it at all times; therefore it may be held liable when it is inadequate, unless the FDA has formally taken the choice away. The generic manufacturer is required by federal law to copy the brand label and cannot change it on its own; therefore it cannot be held liable for failing to do so. The problem is not the principle but the regulatory structure that allocates control: a structure designed in 1984 to make generics cheap and interchangeable, at a time when no one was thinking about tort liability. That diagnosis also points to the solution. If generic manufacturers were given the power to strengthen warnings, as the 2013 proposal would have done, they would also bear responsibility, and the anomaly would disappear. Alternatively, if brand manufacturers were responsible for the labels on which generic users rely, as the innovator liability cases suggest, patients would have a defendant. Or Congress could adopt an administrative compensation scheme, on the model of the vaccine program. Each option has costs, including higher generic prices in the first case, uncertain fairness in the second, and administrative complexity in the third. None has attracted enough political support to be enacted. The generic gap therefore persists, a vivid illustration of the power of the label and of the way that small regulatory decisions about who may change its words cascade into large questions about who pays when those words turn out to be wrong. Conclusion: Governing Through the Label A drug's legal life, followed from the first human dose to the last lawsuit, turns out to be the life of a document. The clinical trial generates the evidence the label will summarize. The approval decision fixes the label's content and makes it the government's statement as well as the manufacturer's. The law of promotion treats the label as the boundary of what the manufacturer may claim. Postmarket surveillance exists to revise the label, and recalls and withdrawals address what no label can fix. Failure-to-warn litigation measures the manufacturer's conduct against what the label should have said, and preemption doctrine decides whether that measurement is permitted by asking who controlled the words. Having traced each stage, it is possible to say something about what the label-centered design does well, where it fails, and what follows for anyone who deals with it. What the design gets right The label-centered system has a real virtue: it concentrates responsibility. At each stage there is an identifiable party accountable for the accuracy of a specific text. The manufacturer drafts it, must keep it current, and answers for its inadequacy in court. The agency reviews it, can compel changes, and answers politically for its decisions. Physicians rely on it and answer for departures from the standard of care. That concentration gives each actor a reason to invest in the information the label contains. It is why manufacturers maintain pharmacovigilance departments, why sponsors fight over the placement of a warning, and why internal emails about safety signals become the most important documents in a trial. The design also accommodates the unavoidable gap between regulatory knowledge and medical practice. Off-label prescribing lets medicine move faster than the approval process, while the prohibition on off-label promotion, however contested, preserves an incentive to put new claims through rigorous review. The tolerated zone of scientific exchange, enlarged most recently by the agency's 2025 guidance, gives physicians more access to the manufacturer's knowledge without abandoning that incentive entirely. Where it fails The system's failures cluster where control of the label and responsibility for it come apart. The generic gap is the starkest example. Because generic manufacturers must copy the brand label, they cannot be held responsible for its inadequacy, and because the brand manufacturer's responsibility generally extends only to its own customers, most patients have no warning claim against anyone. No one designed this outcome. It emerged from a 1984 bargain about competition that did not consider tort law, and a series of Supreme Court decisions that applied general preemption principles faithfully to a structure that was never built with them in mind. It persists because the parties who could fix it, Congress and the FDA, face concentrated opposition and diffuse support. Accelerated approval is another. A product approved on a surrogate endpoint carries a label that looks, to patients and to many physicians, like any other label, though it reflects a provisional judgment. The 2022 reforms made it easier to withdraw such products when confirmatory trials fail, but they did not change the fact that, during the years of uncertainty, the label's authority exceeds its evidence. A third failure lies in the dependence of the postmarket system on the manufacturer's cooperation. The FDA's inability to order a drug recalled, and its reliance on the manufacturer to detect and report problems, means that the quality of postmarket safety depends heavily on the incentives created by liability and reputation. Where those incentives are weak, as with small foreign manufacturers of low-margin generics, or where litigation has been cut off by preemption, the system has fewer safeguards than its formal structure suggests. And a fourth failure is institutional. The off-label regime was built largely through settlements, the permitted zone of communication through guidance, and much of the approval process through agency practice rather than statute. After Loper Bright, and in a period in which agency leadership has shown a willingness to depart from past practice rapidly, from creating new review vouchers by administrative initiative to driving label changes from the top of the department, the legal foundations of these practices are less secure than they appeared. A system that has relied on the expertise and continuity of a stable agency is exposed when that continuity is disrupted. What remains unsettled Several questions will shape the next decade. The first is whether the Supreme Court will hold the qui tam provisions of the False Claims Act unconstitutional, which would transform the enforcement landscape for promotional misconduct and fraud on federal health programs. The second is whether the Court or a court of appeals outside the Second Circuit will squarely address the constitutionality of prosecuting truthful off-label promotion, resolving the question Caronia avoided. The third is how Loper Bright will affect challenges to the agency's interpretations of its core statutory terms, including "intended use" and "substantial evidence." The fourth is whether the PDUFA reauthorization due in 2027 becomes a vehicle for significant changes to the approval process, including codification of new expedited pathways or constraints on them. The fifth is how courts will treat regulatory statements, such as the 2025 acetaminophen announcement, that are made outside the ordinary scientific process but bear on the same questions juries decide. And the sixth is whether anyone will close the generic gap. None of these questions has an obvious answer, and each could move the system in a different direction. What they have in common is that each concerns the relationship between the label and the institutions that produce, police and rely on it. What follows for the reader For physicians, the lesson is that the label is necessary but not sufficient. It is the authoritative summary of what the agency and manufacturer agreed on at a particular time, not a complete account of what is known. Off-label prescribing is lawful and often wise, but it places more responsibility on the prescriber, who must evaluate the evidence without the benefit of a neutral reviewer. Information from manufacturers about unapproved uses, even when now permitted, comes from a party with interests, and should be read the way one reads any interested source: with attention to what is omitted as well as what is said. For patients, the lesson is that "FDA-approved" is a meaningful but limited statement. It means that a neutral reviewer found substantial evidence of benefit for a defined use and judged the known risks acceptable for that use. It does not mean that every risk is known, that the drug works for uses beyond its label, or that a product approved on an accelerated basis has been shown to improve how patients feel, function or survive. Patients should also understand that whether they have a legal remedy for an injury may depend on whether their pharmacy dispensed a brand or a generic, a fact that few people know when they fill a prescription. For companies, the lesson is that the label is an asset and a liability at once. The investments that make it accurate, including rigorous trials, honest reporting, prompt updating and disciplined promotion, are also the best defense against enforcement and litigation. The documents that reveal a gap between what a company knew and what its label said are the ones that produce the largest verdicts and settlements. And the zone of permitted communication, though larger than it was, remains defined by guidance that can change. For policymakers, the lesson is that the architecture's weaknesses lie where control and responsibility diverge, and that the most effective reforms would reconnect them. Allowing generic manufacturers to update warnings, requiring confirmatory evidence before accelerated approval rather than after, giving the agency clear recall authority for drugs, and strengthening the agency's capacity to detect signals independently of manufacturers would each do so. They would also have costs, which should be weighed honestly. The history recounted in the first chapter of this book is a history of disasters followed by reforms. The better course, always proposed and rarely followed, is to reform before the next disaster. The label, that dense and rarely read document, is where the next disaster will be recorded, in the warnings that were added too late or not at all. It is also where, with sufficient care, it might be prevented. Notes Introduction 1. Loper Bright Enterprises v. Raimondo, 603 U.S. 369 (2024). 1. Federal Food, Drug, and Cosmetic Act, ch. 675, 52 Stat. 1040 (1938), codified as amended at 21 U.S.C. § 301 et seq. Chapter 1 1. Pure Food and Drugs Act, ch. 3915, 34 Stat. 768 (1906). 1. United States v. Johnson, 221 U.S. 488 (1911). 2. Drug Amendments of 1962 (Kefauver-Harris Amendments), Pub. L. No. 87-781, 76 Stat. 780. 3. Weinberger v. Hynson, Westcott & Dunning, Inc., 412 U.S. 609 (1973). 4. United States v. Dotterweich, 320 U.S. 277 (1943). 5. United States v. Park, 421 U.S. 658 (1975). 6. Friedman v. Sebelius, 686 F.3d 813 (D.C. Cir. 2012) (addressing the exclusion of the Purdue Frederick executives following their misdemeanor pleas). 7. Chevron U.S.A. Inc. v. Natural Resources Defense Council, Inc., 467 U.S. 837 (1984). 8. FDA v. Brown & Williamson Tobacco Corp., 529 U.S. 120 (2000). 9. FDA v. Wages and White Lion Investments, LLC, 604 U.S. ___ (2025). 10. FDA v. Alliance for Hippocratic Medicine, 602 U.S. 367 (2024). 11. 21 U.S.C. §§ 321(g), 321(p), 331, 333, 351, 352, 355, 396. Chapter 2 1. 21 C.F.R. Part 312 (investigational new drug applications), Part 50 (protection of human subjects), and Part 56 (institutional review boards). 1. FDA Modernization Act 2.0, enacted as part of the Consolidated Appropriations Act, 2023, Pub. L. No. 117-328. 2. National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, The Belmont Report (1979). 3. Federal Policy for the Protection of Human Subjects (revised Common Rule), 82 Fed. Reg. 7149 (Jan. 19, 2017), codified at 45 C.F.R. Part 46. 4. Grimes v. Kennedy Krieger Institute, Inc., 782 A.2d 807 (Md. 2001). 5. International Council for Harmonisation, Guideline for Good Clinical Practice E6(R3) (2025). 6. Food and Drug Administration Amendments Act of 2007, Pub. L. No. 110-85, § 801, codified at 42 U.S.C. § 282(j); Clinical Trials Registration and Results Information Submission, 81 Fed. Reg. 64982 (Sept. 21, 2016), codified at 42 C.F.R. Part 11. 7. Seife v. U.S. Department of Health and Human Services, 440 F. Supp. 3d 254 (S.D.N.Y. 2020). 8. Food and Drug Omnibus Reform Act of 2022, Pub. L. No. 117-328, div. FF, title III. 9. Abigail Alliance for Better Access to Developmental Drugs v. von Eschenbach, 495 F.3d 695 (D.C. Cir. 2007) (en banc). 10. Trickett Wendler, Frank Mongiello, Jordan McLinn, and Matthew Bellina Right to Try Act of 2017, Pub. L. No. 115-176 (2018). Chapter 3 1. 21 U.S.C. § 355(d); 21 C.F.R. § 314.126. 1. FDA, Draft Guidance for Industry, Demonstrating Substantial Evidence of Effectiveness With One Adequate and Well-Controlled Clinical Investigation and Confirmatory Evidence (2023). 2. 21 U.S.C. § 355-1 (risk evaluation and mitigation strategies). 3. FDA, Guidance for Industry, Expedited Programs for Serious Conditions—Drugs and Biologics (2014). 4. Orphan Drug Act, Pub. L. No. 97-414 (1983). 5. Drug Price Competition and Patent Term Restoration Act of 1984 (Hatch-Waxman Act), Pub. L. No. 98-417. 6. GlaxoSmithKline LLC v. Teva Pharmaceuticals USA, Inc., 7 F.4th 1320 (Fed. Cir. 2021). 7. FTC v. Actavis, Inc., 570 U.S. 136 (2013). 8. Biologics Price Competition and Innovation Act of 2009, enacted as part of the Patient Protection and Affordable Care Act, Pub. L. No. 111-148 (2010). Chapter 4 1. Kordel v. United States, 335 U.S. 345 (1948). 1. Requirements on Content and Format of Labeling for Human Prescription Drug and Biological Products, 71 Fed. Reg. 3922 (Jan. 24, 2006). 2. Content and Format of Labeling for Human Prescription Drug and Biological Products; Requirements for Pregnancy and Lactation Labeling, 79 Fed. Reg. 72064 (Dec. 4, 2014). 3. 21 C.F.R. § 314.70(c); Supplemental Applications Proposing Labeling Changes for Approved Drugs, Biologics, and Medical Devices, 73 Fed. Reg. 49603 (Aug. 22, 2008). 4. Regulations Regarding "Intended Uses," 86 Fed. Reg. 41383 (Aug. 2, 2021), amending 21 C.F.R. § 201.128. 5. Wisconsin v. Mitchell, 508 U.S. 476 (1993). 6. Buckman Co. v. Plaintiffs' Legal Committee, 531 U.S. 341 (2001). 7. 21st Century Cures Act, Pub. L. No. 114-255, § 3037 (2016). Chapter 5 1. 31 U.S.C. §§ 3729–3733. 1. United States ex rel. Franklin v. Parke-Davis, 147 F. Supp. 2d 39 (D. Mass. 2001). 2. U.S. Department of Justice press releases announcing the resolutions listed in Table 3. 3. Universal Health Services, Inc. v. United States ex rel. Escobar, 579 U.S. 176 (2016). 4. United States ex rel. Schutte v. SuperValu Inc., 598 U.S. 739 (2023). 5. United States ex rel. Polansky v. Executive Health Resources, Inc., 599 U.S. 419 (2023). 6. United States ex rel. Zafirov v. Florida Medical Associates, LLC (M.D. Fla. Sept. 30, 2024), appeal to the Eleventh Circuit. 7. HHS Office of Inspector General, Special Fraud Alert: Speaker Programs (Nov. 16, 2020). 8. Harrington v. Purdue Pharma L.P., 603 U.S. 204 (2024). 9. United States v. Facteau, 89 F.4th 1 (1st Cir. 2023). Chapter 6 1. Virginia State Board of Pharmacy v. Virginia Citizens Consumer Council, Inc., 425 U.S. 748 (1976). 1. Central Hudson Gas & Electric Corp. v. Public Service Commission, 447 U.S. 557 (1980). 2. 44 Liquormart, Inc. v. Rhode Island, 517 U.S. 484 (1996). 3. Thompson v. Western States Medical Center, 535 U.S. 357 (2002). 4. Washington Legal Foundation v. Friedman, 13 F. Supp. 2d 51 (D.D.C. 1998); Washington Legal Foundation v. Henney, 202 F.3d 331 (D.C. Cir. 2000). 5. Sorrell v. IMS Health Inc., 564 U.S. 552 (2011). 6. United States v. Caronia, 703 F.3d 149 (2d Cir. 2012). 7. Amarin Pharma, Inc. v. FDA, 119 F. Supp. 3d 196 (S.D.N.Y. 2015). 8. Pacira Pharmaceuticals, Inc. v. FDA (S.D.N.Y., filed 2015; settled Dec. 2015). 9. FDA, Memorandum, Public Health Interests and First Amendment Considerations Related to Manufacturer Communications Regarding Unapproved Uses of Approved or Cleared Medical Products (Jan. 2017). 10. FDA, Guidance for Industry, Medical Product Communications That Are Consistent With the FDA-Required Labeling—Questions and Answers (2018); Drug and Device Manufacturer Communications With Payors, Formulary Committees, and Similar Entities—Questions and Answers (2018). 11. FDA, Guidance for Industry, Communications From Firms to Health Care Providers Regarding Scientific Information on Unapproved Uses of Approved/Cleared Medical Products: Questions and Answers (Jan. 2025). 12. Directive 2001/83/EC of the European Parliament and of the Council on the Community code relating to medicinal products for human use, arts. 87–88. Chapter 7 1. 21 C.F.R. § 314.80 (postmarketing reporting of adverse drug experiences); § 314.81 (field alert reports). 1. Institute of Medicine, The Future of Drug Safety: Promoting and Protecting the Health of the Public (National Academies Press, 2007). 2. 21 U.S.C. § 355(o)(3)–(4). 3. 21 U.S.C. § 355(e). 4. 21 C.F.R. Part 7, Subpart C (recalls). 5. FDA, Guidance for Industry, Control of Nitrosamine Impurities in Human Drugs (2020, as subsequently revised). 6. Drug Quality and Security Act, Pub. L. No. 113-54 (2013). Chapter 8 1. Restatement (Second) of Torts § 402A cmt. k (1965); Restatement (Third) of Torts: Products Liability § 6 (1998). 1. Perez v. Wyeth Laboratories Inc., 734 A.2d 1245 (N.J. 1999). 2. MacDonald v. Ortho Pharmaceutical Corp., 475 N.E.2d 65 (Mass. 1985). 3. Tex. Civ. Prac. & Rem. Code § 82.007. 4. Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993). 5. General Electric Co. v. Joiner, 522 U.S. 136 (1997). 6. Fed. R. Evid. 702 (as amended Dec. 1, 2023). 7. A. Bradford Hill, "The Environment and Disease: Association or Causation?," Proceedings of the Royal Society of Medicine 58 (1965): 295–300. 8. In re Zantac (Ranitidine) Products Liability Litigation, 644 F. Supp. 3d 1075 (S.D. Fla. 2022). 9. In re Acetaminophen—ASD-ADHD Products Liability Litigation, 707 F. Supp. 3d 309 (S.D.N.Y. 2023). 10. 28 U.S.C. § 1407; Fed. R. Civ. P. 16.1. 11. State Farm Mutual Automobile Insurance Co. v. Campbell, 538 U.S. 408 (2003). 12. National Childhood Vaccine Injury Act of 1986, 42 U.S.C. § 300aa-1 et seq.; Bruesewitz v. Wyeth LLC, 562 U.S. 223 (2011). Chapter 9 1. Riegel v. Medtronic, Inc., 552 U.S. 312 (2008). 1. Wyeth v. Levine, 555 U.S. 555 (2009). 2. PLIVA, Inc. v. Mensing, 564 U.S. 604 (2011). 3. Mutual Pharmaceutical Co. v. Bartlett, 570 U.S. 472 (2013). 4. Merck Sharp & Dohme Corp. v. Albrecht, 587 U.S. 299 (2019). 5. Buckman Co. v. Plaintiffs' Legal Committee, 531 U.S. 341 (2001). 6. Supplemental Applications Proposing Labeling Changes for Approved Drugs and Biological Products, 78 Fed. Reg. 67985 (proposed Nov. 13, 2013). 7. Conte v. Wyeth, Inc., 168 Cal. App. 4th 89 (2008). 8. T.H. v. Novartis Pharmaceuticals Corp., 4 Cal. 5th 145 (2017). 9. Wyeth, Inc. v. Weeks, 159 So. 3d 649 (Ala. 2014); Ala. Code § 6-5-530. 10. Rafferty v. Merck & Co., 479 Mass. 141 (2018). Further Reading Daniel Carpenter, Reputation and Power: Organizational Image and Pharmaceutical Regulation at the FDA (Princeton University Press, 2010).** The definitive scholarly history of the modern FDA, explaining how the agency's reputation became the source of its authority over the pharmaceutical industry. Philip J. Hilts, Protecting America's Health: The FDA, Business, and One Hundred Years of Regulation (Knopf, 2003).** An accessible narrative history covering the Elixir Sulfanilamide disaster, thalidomide and the growth of the agency's powers. Peter Barton Hutt, Richard A. Merrill, Lewis A. Grossman, Nathan Cortez, Erika Lietzan and Patricia J. Zettler, Food and Drug Law: Cases and Materials (Foundation Press, 5th ed., 2022).** The standard law school casebook, with extensive materials on approval, promotion and enforcement. Institute of Medicine, The Future of Drug Safety: Promoting and Protecting the Health of the Public (National Academies Press, 2007).** The report, commissioned after Vioxx, that shaped the postmarket reforms of 2007. Jerry Avorn, Powerful Medicines: The Benefits, Risks, and Costs of Prescription Drugs (Knopf, 2004).** A Harvard pharmacoepidemiologist's account of how evidence about drugs is generated, distorted and used, still valuable on the gap between labels and knowledge. James H. Jones, Bad Blood: The Tuskegee Syphilis Experiment (Free Press, rev. ed., 1993).** The standard history of the study whose exposure produced the modern federal framework for protecting research subjects. Aaron S. Kesselheim and Jerry Avorn, writings in the New England Journal of Medicine and JAMA on off-label promotion, accelerated approval and the First Amendment.** Kesselheim's program at Brigham and Women's Hospital has produced much of the empirical work on these questions. Food and Drug Administration, Expedited Programs for Serious Conditions—Drugs and Biologics (Guidance for Industry, 2014), and the agency's annual New Drug Therapy Approvals reports.** Primary sources for how the expedited pathways work and how often they are used. ClinicalTrials.gov and the FDA's Drugs@FDA database. Free public databases where readers can find trial registrations and results, approved labels, and review documents for individual drugs. Hashtags: #PharmaceuticalRegulation #FDALaw #PharmaceuticalLaw #HealthcareLaw #DrugRegulation #FDACompliance #ClinicalTrials #ClinicalResearch #DrugDevelopment #DrugApproval #NewDrugApplication #InvestigationalNewDrug #GoodClinicalPractice #ResearchEthics #InformedConsent #DrugLabeling #PrescriptionDrugLabeling #OffLabelPrescribing #OffLabelPromotion #FalseClaimsAct #PharmaceuticalLitigation #DrugSafety #Pharmacovigilance #AdverseEventReporting #DrugRecalls #RiskEvaluationAndMitigation #FDAEnforcement #FailureToWarn #ProductLiability #FederalPreemption

Download the Book (PDF): Introduction Every large language model in wide use today descends from a single architectural decision made in 2017: that a network reading a sequence should be allowed to look, at every step, at every earlier element of that sequence. This is what self-attention does. When a transformer produces the ten-thousandth word of a document, it compares a representation of its current position against stored representations of all 9,999 positions before it, weighs them by relevance, and blends them into its next prediction. Nothing is forgotten. Nothing is summarised. The whole past is kept on the table, in full, and consulted afresh for every new token. That decision is the source of the transformer's power and the source of its most stubborn cost. Comparing every position with every other position means the work grows with the square of the sequence length. Double the context and you quadruple the attention computation during training. During generation the arithmetic is slightly kinder but the memory is not: the model must keep a cache of keys and values for every token it has seen, so the memory needed to hold a conversation grows in direct proportion to its length, and the time to produce each new token grows with it. For a document of a few thousand words this is invisible. For a codebase, a genome, an hour of audio, a long legal record or a million-token context window, it becomes the dominant fact of the system's economics. This booklet is about the family of architectures that refuses that bargain. State space models, and in particular the line of work running from HiPPO and S4 to Mamba, Mamba-2 and Mamba-3, process a sequence by maintaining a fixed-size internal state that is updated as each new element arrives. They do not keep the whole past. They keep a compressed, continuously revised summary of it. As a result their cost grows linearly with sequence length during training, and during generation each new token costs the same small, constant amount no matter how long the sequence has become. A model of this kind can, in principle, read a million tokens with the same memory footprint it uses to read a hundred. The argument Put that way, the idea sounds old, and it is. Recurrent neural networks did exactly this for decades, and the transformer displaced them for good reasons. Recurrent networks were slow to train because each step had to wait for the one before it, and they were poor at remembering things over long distances. The interesting question is therefore not whether a fixed-size state is cheaper than attention. Obviously it is. The question is how a fixed-size state was made good enough to compete, and where it still falls short. The answer this booklet develops has three parts, and they form its controlling argument. First, the modern state space model succeeded because it borrowed a mathematical discipline from continuous-time systems theory. By treating a sequence as samples of an underlying signal, and by designing the state update as the solution of a carefully chosen linear differential equation, researchers obtained models whose memory was not an accident of training but a deliberate approximation of the past, with known properties. That discipline is what solved the long-range memory problem that defeated earlier recurrent networks. Second, linearity alone was not enough, and the decisive step in Mamba was to break it in exactly one place. Early structured state space models were time-invariant: they applied the same dynamics to every input, which made them fast but left them unable to decide what was worth remembering. Mamba made the dynamics depend on the input, so that the model could choose, token by token, whether to absorb new information or preserve what it already held. This selectivity destroyed the convolutional shortcut that made earlier models fast, and the work that made Mamba practical was the design of a parallel scan algorithm arranged around the memory hierarchy of a modern GPU. The point is general and worth holding onto: in this field, an algorithm's asymptotic complexity matters less than whether its data movement matches the hardware. Third, a fixed-size state imposes a real and provable ceiling on what a model can recall exactly. Compression is lossy. A model that summarises the past cannot always retrieve an arbitrary detail from it, and a growing body of theory and experiment shows that pure state space models fall behind attention on tasks that demand precise lookup, such as copying long strings or finding a single fact buried in a large context. The practical response, now adopted in production systems from several large laboratories, is not to choose between the two mechanisms but to combine them: mostly recurrent layers for cheap, continuous compression, with a small number of attention layers kept for exact retrieval. Million-token modeling, as it is actually being built, is a division of labour. So the booklet argues that the quadratic cost of attention is not a law of sequence modeling but the price of one particular memory strategy, that state space models offer a different and principled strategy whose efficiency depends as much on hardware-aware algorithm design as on mathematics, and that the most capable long-context systems combine both because each strategy fails where the other succeeds. Who this is for and how it proceeds The intended reader is technically literate but not necessarily a specialist in sequence modeling: an engineer who has used transformers and wants to understand what the alternatives are actually doing; a researcher in a neighbouring field; a technical manager deciding whether a hybrid architecture is worth adopting; a student who has read the Mamba paper and found that it assumes a great deal. Some mathematics is unavoidable, because the central ideas are mathematical, but every equation is explained in words and the argument never depends on the reader carrying out a derivation. Where a formula appears it is written plainly, with subscripts for time steps, and the surrounding prose says what it means. The chapters build in order. The first sets out precisely what attention costs, in compute, memory and memory bandwidth, and why the many attempts to approximate it cheaply did not displace it. The second returns to recurrent networks, explains why they lost, and shows that a linearised form of attention is itself a recurrent network, which reframes the whole debate. The third introduces the continuous-time state space model from control theory, the three equivalent ways of computing it, and the discretisation step that turns a differential equation into a sequence layer. The fourth explains HiPPO, the theory of how a small state can optimally remember a long history, which is the intellectual core of the field. The fifth follows S4 and its many simplifications and rivals, from diagonal models to long convolutions, and the benchmarks on which they first proved themselves. The sixth chapter is about selection: why time-invariant models cannot perform certain basic tasks, and how Mamba's input-dependent parameters fix this. The seventh explains the parallel scan and the hardware-aware implementation that makes selection affordable, with enough detail about GPU memory to show why the design looks the way it does. The eighth covers Mamba-2 and the discovery that state space models and a family of linear attention mechanisms are two views of the same mathematical object, along with the convergence of the field around a small set of gated, delta-rule recurrences. The ninth confronts the limits: what a fixed-size state provably cannot do, how those limits show up in practice, and what recent designs such as Mamba-3 do about them. The tenth turns to deployment, to the hybrid architectures now shipping in production, to the arithmetic of million-token contexts, and to domains such as genomics and audio where very long sequences are the norm rather than the exception. The conclusion draws out what all this implies for how sequence models should be chosen and built. A brief glossary and an annotated list of further reading close the book. The papers named in the text are real and are listed there, and the reader who wants to go deeper should treat them as the primary sources they are. A note on what is settled This is a fast-moving field, and some care is needed in separating what is established from what is still contested. Several things are now well established. Structured state space models can match or exceed transformers of similar size on standard language modeling benchmarks at the scales studied so far. They are substantially cheaper at inference for long sequences. They are clearly weaker than attention at exact retrieval from long contexts when used alone. Hybrids that combine a minority of attention layers with a majority of recurrent ones have been shown by multiple independent groups to match or beat pure transformers on quality while using far less memory. Other things are not settled. Nobody yet knows how pure recurrent models behave at the very largest scales of training compute, because the largest experiments have mostly been hybrids. The best ratio of attention to recurrence, and where in the network the attention layers should sit, varies from paper to paper. Benchmarks for genuinely long-context reasoning remain immature, and a model that passes a simple retrieval test at a million tokens may still fail to use that context well. Where the text describes these open questions it says so plainly, and it avoids attributing more certainty to any claim than the evidence supports. The reward for working through the material is a clear picture of a genuine alternative to attention: not a marginal efficiency trick but a different theory of what it means for a model to remember, with its own strengths, its own failure modes and its own engineering logic. Understanding that alternative changes how one sees the transformer itself. Attention stops looking like the natural way to process sequences and starts looking like what it is: one very expensive, very effective point in a much larger space of designs. Chapter 1. The Quadratic Wall To understand why anyone would want to replace attention, it helps to be exact about what attention costs and where those costs land. Talk of "quadratic complexity" is usually left at the level of a slogan, and slogans hide the interesting part. Attention's cost is not one number. It is a set of different pressures, on arithmetic, on memory capacity and on memory bandwidth, that bite at different stages of a model's life and at different sequence lengths. Some of them have been engineered away. One of them cannot be, without changing what attention is. What attention actually computes A transformer layer receives a sequence of vectors, one per token, each of some width d, which in a large model might be four or eight thousand. Self-attention projects each vector three ways, producing a query, a key and a value. For a given position, the query is compared with the key of every position the model is allowed to see, by taking a dot product. Those dot products are scaled, passed through a softmax so that they become positive weights summing to one, and used to form a weighted average of the corresponding values. The result is that position's output. In a language model the comparison is causal: each position sees itself and everything before it, never anything after. Written compactly, if Q, K and V are the matrices whose rows are the queries, keys and values for a sequence of length n, the output is softmax(QKᵀ / √d) V with a mask applied so that no position attends to the future. The matrix QKᵀ has n rows and n columns. Every entry is a dot product of length d. That matrix is where the quadratic cost lives: there are n² pairs of positions, and attention computes a score for every pair. The design is extraordinarily expressive. Because each output is a data-dependent mixture of all earlier values, a single attention layer can route information from any position to any other in one step. It can copy a token from far back, match a pattern against an earlier occurrence, or retrieve the definition of a variable named three thousand lines ago. The weights are recomputed for every input, so the routing adapts to content. Nothing in the mechanism privileges nearby tokens over distant ones; distance enters only through whatever positional information the model has been given. This is precisely why transformers are so good at in-context learning, and why they displaced recurrent networks that had to carry information forward step by step. Three different costs The first cost is arithmetic. Forming QKᵀ for a sequence of length n takes on the order of n²d multiply-add operations per layer, and multiplying the resulting weights by V takes the same again. For a causal model roughly half the matrix is masked, which halves the work but does not change its quadratic shape. It is worth placing this against the rest of the layer, because for short sequences attention's quadratic term is not where most of the arithmetic goes. Each token also passes through the four projections that produce queries, keys, values and the output, and through a feed-forward network that typically expands the width by a factor of four and contracts it again. Counting multiply-adds per token per layer, the projections cost about 8d² and the feed-forward network about 16d², for a total around 24d² that does not depend on sequence length at all. The attention scores and the weighted sum cost about 4nd per token. Setting 4nd equal to 24d² gives a crossover at n equal to about six times d. For a model of width 4,096, attention's quadratic term only overtakes the rest of the layer once the context passes roughly twenty-five thousand tokens. Below that, the transformer's arithmetic is dominated by its linear parts. This calculation explains a good deal of history. For years, contexts of two to eight thousand tokens were standard, and at those lengths the quadratic term was a minority of the compute. Efficient attention was an academic concern. It became a practical concern when contexts reached into the hundreds of thousands and millions of tokens, because at a million tokens, with the same width, the attention term is around forty times larger than everything else in the layer combined. The second cost is memory capacity, and it divides into two cases. During training, a naive implementation materialises the full n-by-n score matrix for every head in every layer so that it can be used again in the backward pass. That is quadratic memory, and for long sequences it was historically the binding constraint: it is not hard to construct a configuration where the attention matrices alone would exceed the memory of any accelerator. This cost, unlike the arithmetic, turned out to be removable. FlashAttention, introduced by Tri Dao and colleagues in 2022, computes attention in tiles that fit in a GPU's fast on-chip memory, never writing the full score matrix out, and recomputes what it needs during the backward pass. Training memory for attention became linear in sequence length. The arithmetic stayed quadratic, but because the computation now made far fewer trips to slow memory, it also ran several times faster in wall-clock terms. FlashAttention is part of this story for a second reason: its author went on to co-design Mamba, and the same instinct, that the real cost of an algorithm is its data movement, runs through both. During inference the memory question is different and, for long contexts, more severe. A language model generates text one token at a time. To avoid recomputing the keys and values for the entire prefix at every step, it stores them in a key-value cache. Each new token computes its own query, key and value, appends the key and value to the cache, and attends over everything stored. The cache grows by a fixed amount for every token, in every layer, for every sequence being served. A worked example of the cache The size of that cache is easy to calculate and sobering to see. Consider a model with the published shape of Llama 3 8B: 32 layers, eight key-value heads (it uses grouped-query attention, in which several query heads share each key-value head), and a head dimension of 128. For every token, each layer stores a key vector and a value vector for each of the eight key-value heads. That is 2 × 8 × 128 = 2,048 numbers per layer, or 65,536 numbers across 32 layers. In 16-bit precision, two bytes each, the cache costs 131,072 bytes, or 128 kibibytes, per token. At a context of eight thousand tokens the cache is about one gigabyte, which is manageable. At 128,000 tokens it is about sixteen gigabytes, already as large as the model's weights. At one million tokens it is about 131 gigabytes for a single sequence, more than the memory of an 80-gigabyte accelerator before a single weight has been loaded. And this is for a modest model that already uses grouped-query attention to shrink its cache by a factor of four relative to standard multi-head attention. A model of the same family at seventy billion parameters, with 80 layers, would need two and a half times as much. The third cost is bandwidth, and it is the one that determines how fast tokens come out. Generating a token requires reading the entire key-value cache from the accelerator's main memory, because the new query must be compared with every stored key and must combine every stored value. The arithmetic involved in that read is tiny: a handful of operations per number fetched. Modern accelerators can perform hundreds of operations in the time it takes to fetch one number from high-bandwidth memory, so decoding with a long context is limited almost entirely by how fast the cache can be streamed in. On an accelerator with around three terabytes per second of memory bandwidth, reading a 131-gigabyte cache takes over forty milliseconds, for every single token, for every sequence. Serving many users at once means holding many caches, which fills memory and limits how many requests can be batched together, which in turn reduces the efficiency of the whole system. So the attention bill has three lines. Arithmetic grows quadratically in training and in the initial processing of a prompt, which practitioners call prefill. Memory grows linearly per sequence during generation, but with a large constant. Bandwidth per generated token grows linearly with context, and since decode is bandwidth-bound, so does latency. Table 1 sets these costs beside those of two alternatives that recur through this book. Table 1. Asymptotic costs of three sequence-mixing strategies for a sequence of length n and model width d. Strategy Training compute Compute per generated token Memory held during generation Exact access to any past token Full causal attention Quadratic in n Linear in n Linear in n (key-value cache) Yes Sliding-window attention, window w Linear in n (times w) Constant (depends on w) Constant (w tokens) Only within the window Recurrent state, size fixed Linear in n Constant Constant No; only what the state retains Source: standard asymptotic analysis of the respective mechanisms. The final column is the one that matters for the rest of the book, and it is the one slogans leave out. Attention's costs buy something concrete: the ability to reach back and retrieve any earlier token exactly. Every cheaper alternative gives up some or all of that ability. The question is never only how much an alternative saves, but what kind of memory it offers in exchange. A decade of cheaper attention The research community did not ignore these costs. Between 2019 and 2021 there was a flood of so-called efficient transformers, and their fates are instructive. One family restricted which pairs of positions could interact. The Sparse Transformer of Child and colleagues at OpenAI in 2019 used fixed strided and local patterns. Longformer, from the Allen Institute in 2020, combined a sliding local window with a few global tokens that attend everywhere. BigBird, from Google in the same year, mixed local, global and random connections and proved that such patterns retain important theoretical properties of full attention. Sliding-window attention in its plain form, in which each token sees only the previous few thousand, later appeared in production models such as Mistral 7B. These approaches cut cost by declaring most pairs irrelevant in advance. They work well when the important dependencies are local or structured, and poorly when the crucial piece of information sits at an unpredictable distance outside the pattern. A second family approximated the attention matrix itself. Linformer projected keys and values down to a fixed number of rows, assuming the attention matrix is approximately low rank. Reformer used locality-sensitive hashing to bucket similar queries and keys so that each query compares only with keys in its bucket. Performer replaced the softmax with a random-feature approximation whose kernel trick allowed the n-by-n matrix never to be formed. Each had a clean mathematical story. Each traded some fidelity for speed. A third family, which this book will return to at length, dropped the softmax altogether. The linear transformer of Katharopoulos and colleagues, published in 2020 under the provocative title "Transformers are RNNs", replaced the exponential similarity of softmax attention with a dot product of feature maps. With that change, the order of operations can be rearranged so that the keys and values are summarised in a fixed-size matrix that is updated token by token. Attention became a recurrence, with constant memory during generation. It also became noticeably worse at language modeling, for reasons that later chapters will make precise. To compare these proposals, Yi Tay and colleagues at Google released the Long Range Arena benchmark in 2020: a suite of tasks with sequences from one thousand to sixteen thousand elements, including long text classification, document matching, hierarchical list operations and image classification from flattened pixel sequences. Its hardest task, Path-X, asked a model to determine whether two points in a 128-by-128 image were connected by a path, given the image as a sequence of 16,384 pixels. The results were humbling. The efficient transformers clustered together with modest scores, none clearly dominant, and none of them, nor the full transformer, did better than chance on Path-X. The benchmark revealed that long-range reasoning over very long sequences was not solved by any approximation of attention then available. Why attention survived By 2022 most of the efficient-attention variants had faded from frontier practice, and it is worth being clear about why, because the same forces later shaped which state space models succeeded. The first reason was quality. Approximations that looked harmless on benchmarks lost measurable perplexity on large-scale language modeling, and in a field where small perplexity gains translate into visibly better models, a few percent was disqualifying. The second was hardware. Many of the efficient methods replaced dense matrix multiplication, which GPUs perform at enormous speed, with gathers, scatters, hashing and irregular sparsity, which they perform poorly. A method with lower asymptotic cost could easily run slower in practice at the sequence lengths people actually used. The third was FlashAttention itself. By making exact attention two to four times faster and linear in training memory, it pushed the point at which approximations paid off further out, and it removed the most acute pain without any loss of quality. The fourth was the crossover arithmetic above: at contexts of a few thousand tokens, attention's quadratic term was simply not where most of the cost lay. Inference-side engineering then took care of much of the rest. Multi-query attention, proposed by Noam Shazeer in 2019, shares a single key-value head across all query heads, shrinking the cache dramatically at some cost in quality. Grouped-query attention, from Ainslie and colleagues in 2023, interpolates between that and standard attention and became the default in open models. DeepSeek's multi-head latent attention, introduced with DeepSeek-V2 in 2024, compresses keys and values into a low-dimensional latent vector, reducing the cache further. Paged memory management in serving systems such as vLLM reduced waste. Quantising the cache to eight or four bits halves or quarters it again. For distributed training on very long sequences, ring attention splits a sequence across devices and passes key-value blocks around a ring, so that context length scales with the number of accelerators. All of these help. None of them changes the shape of the problem. Grouped-query and latent attention shrink the constant in front of the cache's linear growth; they do not stop it growing. Ring attention spreads the quadratic arithmetic across more hardware; it does not reduce it. A transformer that reads a million tokens must still, for every new token, consult a record of all million. As long as the model keeps the whole past, it pays to keep it. That is the wall. The alternative is to stop keeping the whole past, and instead to keep a summary of it that is fixed in size and updated as each new token arrives. Recurrent networks did exactly this, long before transformers existed. The next chapter asks why they lost, and why the reasons they lost turned out not to be permanent. Chapter 2. Recurrence, Lost and Found Before the transformer, the standard way to build a neural network for sequences was to give it a memory and let it read. A recurrent neural network holds a vector, its hidden state, and at each step it combines that state with the next input to produce a new state. The state is the network's entire knowledge of the past. Whatever the network will need later, it must have written into that vector by now, because nothing else survives. This is the opposite of attention's strategy. Attention keeps everything and decides later what matters. A recurrent network must decide as it goes. This chapter explains why that strategy lost its dominance, and why the specific reasons it lost were not intrinsic to recurrence itself. The distinction matters, because the state space models that occupy the rest of this book are recurrent networks. They succeed by keeping recurrence's central virtue, a fixed-size memory updated in constant time, while removing the particular defects that sank the earlier designs. The classical recurrent network The simplest recurrent network, often associated with Jeffrey Elman's work in 1990, updates its state by multiplying the previous state by a matrix, adding a matrix times the current input, and passing the sum through a squashing nonlinearity such as the hyperbolic tangent: hₜ = tanh(W hₜ₋₁ + U xₜ + b) An output is then read off the state, often by another matrix multiplication. The weights W and U are the same at every step, so the network can process sequences of any length with a fixed number of parameters. The idea is elegant, and in principle such a network is very powerful: with enough units and the right weights it can simulate any finite computation over its input. In practice it was hard to train. The standard training method, backpropagation through time, unrolls the network across the sequence and pushes error signals backwards from the output to every earlier step. Each backward step multiplies the signal by the transpose of W and by the derivative of the nonlinearity. Over many steps, those repeated multiplications cause the signal either to shrink towards zero or to grow without bound, depending on whether the relevant magnitudes are below or above one. Sepp Hochreiter identified this in his 1991 diploma thesis, and Yoshua Bengio, Patrice Simard and Paolo Frasconi analysed it formally in 1994. Their conclusion was stark: a network of this kind, trained by gradient descent, has great difficulty learning dependencies that span more than a few tens of steps, because the gradient that would teach it to do so either vanishes or explodes. Gates and the long short-term memory The long short-term memory, introduced by Hochreiter and Jürgen Schmidhuber in 1997, attacked the vanishing gradient directly. Its core is a cell state that is updated additively rather than by repeated multiplication through a nonlinearity. In the version that became standard, after Felix Gers and colleagues added a forget gate in 2000, the cell state at each step is the previous cell state multiplied element by element by a forget gate, plus new content multiplied by an input gate. The gates are small neural networks that look at the current input and the previous hidden state and output numbers between zero and one. The effect is that the network can learn to hold a value nearly constant for a long time, by setting its forget gate close to one and its input gate close to zero, and the gradient flowing back through that held value is multiplied by numbers close to one rather than by an arbitrary matrix. The gated recurrent unit of Kyunghyun Cho and colleagues in 2014 simplified the arrangement while keeping the same principle. It is worth pausing on what the gates do, because the same mechanism reappears, in a different mathematical dress, at the heart of Mamba. A gate is a learned, input-dependent decision about how much of the past to keep and how much of the present to let in. When the forget gate is near one, the state persists; when it is near zero, the state is wiped. Because the gate depends on the input, the network can decide, token by token, which inputs are worth remembering and which should be ignored. This is selectivity, and it is the property that later chapters will show is necessary for a recurrent model to perform well on language. LSTMs became the workhorse of sequence modeling through the mid-2010s. Sequence-to-sequence translation, introduced by Ilya Sutskever, Oriol Vinyals and Quoc Le in 2014, used one LSTM to read a sentence into a vector and another to write its translation. In the same year, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio added an attention mechanism that let the decoder look back at all of the encoder's states rather than relying on a single summary vector. That addition was a confession: a single fixed-size state was too narrow a channel for a long sentence, and allowing the decoder to consult the whole input directly helped substantially. Why recurrence lost The transformer, introduced by Ashish Vaswani and colleagues in 2017, removed the recurrence altogether and kept only attention. It won for two reasons, and it is important to keep them separate. The first was quality on tasks that need long-range access. Even an LSTM's gated memory degrades over hundreds or thousands of steps, and whatever it fails to write into its state is permanently lost. Attention never has to decide in advance what to keep. The second reason, at least as important, was training speed. A recurrent network's state at step t depends on its state at step t − 1 through a nonlinear function. To compute the state at step one thousand, one must first compute the states at steps one through 999, in order. During training, this means the network cannot process the positions of a sequence in parallel. A GPU, which achieves its speed by doing thousands of operations simultaneously, is left processing one time step at a time, with only the batch and hidden dimensions to parallelise over. A transformer, by contrast, computes all positions of a training sequence at once, as large matrix multiplications that keep the hardware fully occupied. On the same accelerator budget, transformers could be trained on far more data, and more data won. Now consider which of these reasons is fundamental to recurrence. The first is partly fundamental: any fixed-size state must discard information, and this book will return to the consequences in Chapter 9. But the severity of the problem in LSTMs was a property of how their states were designed and trained, not a theorem about all recurrent models. The second reason is not fundamental at all. It is a consequence of the nonlinearity in the state update. If the state at step t were a linear function of the state at step t − 1, the whole sequence of states could be computed in parallel, by an algorithm called a scan, in a number of sequential stages that grows only with the logarithm of the sequence length. Chapter 7 explains that algorithm in detail. The essential point here is that the training bottleneck which sank recurrent networks came from one design choice that could be changed. People noticed this before state space models became fashionable. The quasi-recurrent neural network of James Bradbury and colleagues in 2016 and the simple recurrent unit of Tao Lei and colleagues in 2017 both used recurrences that were linear in the state, with gates computed from the input alone, precisely so that the expensive parts could be parallelised across time. Eric Martin and Chris Cundy showed in 2018 that linear recurrences of this kind could be computed with a parallel scan across the sequence length on GPUs, with large speedups. These models were fast and respectable, but they did not overtake transformers on quality, and they did not come with a theory of how to make a linear state remember well over long distances. That theory would come from a different direction. Attention, linearised, is a recurrence The strongest evidence that recurrence and attention are not opposites came from inside attention research. Recall that softmax attention computes, for each query, a weighted average of values, where the weight on each value is proportional to the exponential of the dot product between the query and the corresponding key. In 2020, Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas and François Fleuret asked what happens if the exponential similarity is replaced by an ordinary dot product of some feature map applied to queries and keys. Write φ for that feature map. The unnormalised output at position t becomes a sum over earlier positions s of the quantity φ(qₜ)ᵀφ(kₛ) times vₛ. Because this is a sum of products, the order of operations can be changed. The query φ(qₜ) can be pulled outside the sum, leaving the sum over s of φ(kₛ)vₛᵀ, which is a matrix. Call it Sₜ. Then the output is simply φ(qₜ)ᵀSₜ, and Sₜ obeys a recurrence: Sₜ = Sₜ₋₁ + φ(kₜ) vₜᵀ yₜ = φ(qₜ)ᵀ Sₜ with a similar running sum used for normalisation. This is a recurrent network. Its state is a matrix whose size depends on the key and value dimensions but not on the sequence length. Each step adds an outer product of the current key and value to the state, and the output reads the state using the current query. During generation the model needs constant memory and constant time per token. During training, the same computation can be done in parallel, either by rearranging it back into an attention-like matrix form with the softmax removed or by a scan, because the recurrence is linear in the state. The title of their paper, "Transformers are RNNs", made the point directly. A transformer without the softmax is a recurrent network with a matrix-valued state. Conversely, softmax attention can be understood as a recurrent network whose state is the entire growing list of keys and values: an infinitely large, uncompressed state. Seen this way, the difference between a transformer and a recurrent network is not that one has memory and the other does not. It is how the memory is organised and how much of it is kept. Why linear attention fell short Linear attention was fast but noticeably worse than softmax attention at language modeling, and the reasons are a map of the problems that later models had to solve. The first problem is that the state never forgets. Every key-value outer product is added to the state with equal weight, forever. After a long sequence the state is a sum of thousands of contributions, with recent and relevant ones given no more prominence than ancient and irrelevant ones. An LSTM's forget gate exists precisely to prevent this, and plain linear attention has no equivalent. The second problem is capacity, and it is best understood through an older idea. In the early 1990s Schmidhuber had proposed fast weight programmers, networks in which one part writes into a weight matrix that another part then uses. Imanol Schlag, Kazuki Irie and Schmidhuber showed in 2021 that linear attention is exactly such a system. The state matrix S is an associative memory. Writing φ(k)vᵀ stores an association from a key to a value; reading with φ(q) retrieves the stored value whose key best matches the query. An associative memory of this kind, with keys of dimension m, can store at most about m associations without interference, because once there are more keys than dimensions they cannot all be orthogonal, and retrieving one returns a blend contaminated by others. A sequence of a hundred thousand tokens may contain far more than m things worth remembering. The state overflows and retrieval becomes noisy. Schlag and colleagues also proposed a remedy, which they called the delta rule. Rather than simply adding a new association, the update first retrieves what the memory currently associates with the new key, and writes only the difference between the desired value and the retrieved one. This lets the memory overwrite stale associations rather than piling new ones on top of them. The idea lay relatively dormant for a few years and then returned, in a highly optimised form, as one of the most important ingredients in the recurrent architectures of 2024 and 2025. Chapter 8 takes it up again. The third problem is the feature map. Softmax's exponential produces sharp, nearly one-hot attention patterns when one key matches much better than the others, which is exactly what precise lookup needs. Simple feature maps produce flatter, more diffuse patterns, which blur what is retrieved. So by 2020 the field had two incomplete insights. Recurrent networks with linear state updates could be trained in parallel and run in constant memory. And attention itself, when linearised, was such a network, but one that forgot nothing, overflowed easily and retrieved poorly. What was missing was a principled answer to the question every recurrent model must answer: given a fixed-size state, what exactly should it remember about the past, and how should each new input be folded in so that the memory remains useful over very long spans? The answer that unlocked the field came from a body of mathematics far older than deep learning: the theory of continuous-time linear systems, developed for control engineering and signal processing in the middle of the twentieth century. It is to that theory that the next chapter turns. Chapter 3. Thinking in Continuous Time The phrase "state space model" comes from engineering, not from machine learning. In the late 1950s and early 1960s, Rudolf Kálmán and others reformulated the analysis of dynamical systems, such as aircraft, circuits, chemical plants and orbiting satellites, around a small number of linear differential equations describing how an internal state evolves in response to inputs. That formulation became the foundation of modern control theory and of the Kalman filter, which estimates the hidden state of a system from noisy measurements and still guides spacecraft and phones today. Deep learning borrowed the formalism decades later, and the borrowing is not merely a matter of vocabulary. The mathematical properties of these systems, which engineers had studied for sixty years, are exactly the properties a sequence model needs. This chapter introduces the continuous-time state space model, explains what its pieces mean, shows how it is turned into something that can process a sequence of discrete tokens, and sets out the three equivalent ways of computing it. Those three views, continuous, recurrent and convolutional, are the conceptual toolkit for everything that follows. A state that evolves Imagine a signal u(t) arriving continuously over time: a sound pressure wave, a voltage, a stream of measurements. A state space model maintains an internal state x(t), a vector of N numbers, and evolves it according to the equation x′(t) = A x(t) + B u(t) y(t) = C x(t) + D u(t) The first equation says that the rate of change of the state depends linearly on the state itself, through an N-by-N matrix A, and on the current input, through a vector B. The second says that the output y(t) is a linear readout of the state, through a vector C, plus a direct pass-through of the input scaled by D. In the deep learning setting the D term simply acts as a skip connection, and it is usually set aside in the analysis because it does nothing interesting. A useful way to read these equations is as a description of how the system forgets and how it listens. The matrix A governs forgetting: left alone, with no input, the state evolves as x′ = Ax, and the solution decays, oscillates or grows according to the eigenvalues of A. The vector B governs listening: it determines how the current input is written into each dimension of the state. The vector C governs speaking: it determines which combination of state dimensions is reported as output. Consider the simplest possible case, with a one-dimensional state, A equal to some negative number −a, B equal to a and C equal to one. The equation becomes x′ = −a x + a u. If the input is held constant at some value, the state moves towards that value exponentially, with a time constant of 1/a. If the input changes, the state follows it with a lag. This is a low-pass filter, the mathematical description of an electrical circuit made of a resistor and a capacitor, and also of a thermometer settling to the temperature of a room. Its state at any moment is a weighted average of the input's recent history, with weights that decay exponentially into the past. Larger a means faster decay and shorter memory; smaller a means slower decay and longer memory. Now let the state have many dimensions and let A be a general matrix. Each eigenvalue of A corresponds to a mode of the system. A real negative eigenvalue gives a mode that decays exponentially at a characteristic rate. A pair of complex eigenvalues with negative real part gives a mode that oscillates while it decays, like a struck bell. A system with N modes at many different rates and frequencies can hold a rich, multi-timescale summary of its input history in just N numbers. Provided every eigenvalue has a negative real part, the system is stable: its response to any bounded input remains bounded, and the influence of the distant past fades rather than accumulating without limit. The solution and the impulse response Linear differential equations of this form have an explicit solution. Starting from a zero state, the state at time t is x(t) = ∫ e^{A(t−s)} B u(s) ds where the integral runs over all past times s up to t, and e^{A(t−s)} is the matrix exponential, the natural generalisation of the ordinary exponential to matrices. The output is C times this. The formula has a direct interpretation. The input at each past moment s is injected through B, then propagated forward to the present by the matrix exponential, which applies the system's dynamics for the elapsed time t − s. The present state is the sum of all those propagated contributions. The output can therefore be written as y(t) = ∫ K(t − s) u(s) ds, where K(τ) = C e^{Aτ} B The function K is the system's impulse response: the output it would produce, τ time units later, in response to a single sharp spike of input. The output for any input is the convolution of that input with the impulse response. In other words, a linear time-invariant state space model is nothing more than a particular kind of convolution, one whose kernel is generated by a small set of parameters A, B and C. A state of size N encodes a kernel that can extend infinitely far into the past. This is the first of the three views. The second and third arise when the model is made to work on discrete sequences. From differential equation to sequence layer Text, audio samples and pixels do not arrive continuously. They arrive as a sequence u₀, u₁, u₂ and so on. To apply a continuous-time model, one imagines that these samples were taken from an underlying signal at regular intervals of some step size, written Δ, and asks what the system's state would be at each sample time. This step is called discretisation, and there are several ways to perform it, each corresponding to a different assumption about what the signal does between samples. The most common in modern state space models is the zero-order hold, which assumes the input stays constant over each interval and then jumps to the next sample. Under that assumption, the differential equation can be solved exactly over one interval, giving a discrete recurrence hₖ = Ā hₖ₋₁ + B̄ uₖ yₖ = C hₖ with discrete parameters Ā = exp(ΔA), B̄ = (ΔA)⁻¹ (exp(ΔA) − I) ΔB The matrix Ā is the system's dynamics applied for one step of length Δ. The vector B̄ is the accumulated effect of holding the input constant over that step. An alternative, the bilinear or Tustin transform, uses Ā = (I − ΔA/2)⁻¹(I + ΔA/2), which approximates the exponential by a ratio and was used in the earliest structured state space models. A cruder option, the forward Euler method, sets Ā = I + ΔA, which is accurate only for very small steps and can turn a stable continuous system into an unstable discrete one if Δ is too large. Exact or well-behaved discretisations preserve stability: if every eigenvalue of A has a negative real part, every eigenvalue of Ā has magnitude below one, and the recurrence never blows up. A concrete example makes the role of Δ clear. Take the scalar system with A = −1 and B = 1. With step size Δ = 0.1, the zero-order hold gives Ā = e^{−0.1}, about 0.905, and B̄ = 1 − e^{−0.1}, about 0.095. The recurrence is hₖ = 0.905 hₖ₋₁ + 0.095 uₖ. This is an exponential moving average: each step keeps about ninety per cent of the old state and mixes in about ten per cent of the new input. The influence of an input falls to about a third of its original weight after ten steps, so the memory spans roughly ten tokens. Now shrink the step to Δ = 0.01. Then Ā is about 0.990 and B̄ about 0.010, and the memory spans roughly a hundred tokens. Enlarge it to Δ = 1 and Ā is about 0.368, with a memory of only a step or two. The same continuous system, sampled at different rates, therefore has very different memory lengths measured in tokens. The step size is a timescale knob. A small Δ means each token is treated as a brief moment in a slowly varying process, so the state changes little and remembers far back. A large Δ means each token is treated as a long interval, so the state largely resets to reflect the current input. This interpretation will matter enormously in Chapter 6, where Mamba makes Δ depend on the input and thereby lets the model decide, for each token, whether to remember or to reset. The continuous-time framing also gives a property with no counterpart in an ordinary recurrent network. Because the model's parameters describe an underlying continuous process, the same trained model can be applied to data sampled at a different rate simply by changing Δ. The original S4 work demonstrated this on speech: a model trained on audio at one sampling rate could be evaluated at half that rate, without retraining, by doubling the step size. A discrete recurrent network, whose weights implicitly encode a single sampling rate, has no such option. The recurrent view The discrete recurrence hₖ = Ā hₖ₋₁ + B̄ uₖ is the second view of the model, and it is the one used for generation. To produce each new output, one multiplies the previous state by Ā, adds B̄ times the new input, and reads off C times the result. The cost per step depends on N, the state size, and not at all on how many steps have come before. The memory required is the state itself, N numbers per channel. This is the constant-time, constant-memory inference that motivates the whole field. A point of terminology helps here, because papers use the letters inconsistently. In control theory, u is the input, x the state and y the output. Many deep learning papers, including the Mamba papers, rename the input x and the state h, following the convention of recurrent networks. This book follows the latter convention from now on: the input at step t is xₜ, the hidden state is hₜ, and the output is yₜ. The convolutional view The third view comes from unrolling the recurrence. Starting from a zero state, the first state is B̄x₀. The second is ĀB̄x₀ + B̄x₁. The third is Ā²B̄x₀ + ĀB̄x₁ + B̄x₂. In general, the output at step k is yₖ = C B̄ xₖ + C Ā B̄ xₖ₋₁ + C Ā² B̄ xₖ₋₂ + … which is a discrete convolution of the input sequence with the kernel K̄ = (C B̄, C Ā B̄, C Ā² B̄, C Ā³ B̄, …) This is the discrete counterpart of the impulse response. In the scalar example above, with Δ = 0.1 and C = 1, the kernel begins 0.095, 0.086, 0.078, 0.070 and decays geometrically. For a sequence of length L, one computes the first L entries of the kernel and convolves it with the input. The convolutional view is what made structured state space models fast to train. A convolution of two sequences of length L can be computed using the fast Fourier transform in time proportional to L log L, and the computation is fully parallel across positions. There is no sequential dependence from one step to the next, so a GPU can process the whole sequence at once, just as it does for a transformer. The catch is that one must compute the kernel K̄, whose entries involve successively higher powers of Ā, and doing that naively for large N and long L is expensive. Making this computation efficient was the central technical contribution of S4, and Chapter 5 describes how it was done. Why the three views matter Having three mathematically equivalent ways to compute the same model is not a curiosity. It is the structural reason state space models can be trained like transformers and run like recurrent networks. During training, the whole input sequence is available at once. The convolutional view lets the model process it in parallel, using fast and hardware-friendly algorithms, with total cost growing only slightly faster than linearly in sequence length. During generation, tokens arrive one at a time. The recurrent view lets the model update its fixed-size state in constant time per token, with no growing cache. The continuous view sits behind both, supplying principled initialisations, a natural timescale parameter, stability guarantees and the ability to change sampling rate. In a deep network, this machinery is applied channel by channel. A layer receives a sequence of vectors of width D. For each of the D channels, it runs an independent single-input, single-output state space model with its own state of size N, so the total state per layer is D times N numbers. The outputs are then mixed across channels by ordinary linear layers, nonlinearities and gating, just as a transformer mixes channels with its feed-forward network. The state space model provides the mixing along the sequence; the rest of the network provides mixing across features. The layer is linear in its input along time, but the network as a whole is highly nonlinear because of everything placed around it. There is one condition on which all three views depend. The equivalence between recurrence and convolution holds because the parameters Ā, B̄ and C are the same at every step. The system is time-invariant. That property is what makes the convolution kernel a fixed object that can be computed once and applied everywhere. It is also, as Chapter 6 will show, the property that prevents these models from handling language well. Before reaching that tension, though, there is a prior question. Given a state of N numbers, what should A be so that the state remembers the past well over thousands of steps? A random choice works badly. The answer, a theory of optimal online memory called HiPPO, is the subject of the next chapter. Chapter 4. Memory as Approximation A state space model with a state of N numbers can, in principle, carry information about its input across arbitrarily long spans. Whether it actually does so depends entirely on the matrix A, which governs how the state evolves and therefore what it retains. Choose A badly and the state either forgets everything within a few steps or amplifies noise until it explodes. Choose it well and a few dozen numbers can hold a faithful sketch of thousands of past inputs. The question of how to choose A is the question of what a fixed-size memory ought to remember, and the answer the field converged on is one of the more beautiful ideas in modern machine learning. That answer is HiPPO, short for high-order polynomial projection operators, introduced by Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra and Christopher Ré at NeurIPS in 2020. It reframes memory as a problem in function approximation, solves that problem exactly, and finds that the solution is a linear state space model with a specific, structured A matrix. Everything in the S4 and Mamba lineage descends from this move. Memory as online function approximation Begin with the problem a recurrent memory faces. A signal f has been arriving over time, and at the present moment t the model wants to hold a compact representation of the entire history of f up to t. The representation must have a fixed size, N numbers, regardless of how long the history has become. And it must be maintained online: as new input arrives, the representation must be updated incrementally, without going back to reread the past. Phrased this way, the problem has a natural mathematical form. Pick a family of basis functions, such as polynomials, and represent the history as the best approximation to it that can be built from the first N of those functions. The N coefficients of that approximation are the memory. If the basis is rich enough and the history is reasonably smooth, a modest N will reconstruct the history well. The approximation needs one more ingredient: a measure, which says how much the approximation should care about each part of the past. One might want the history over the last fixed window to be reproduced uniformly well, or the recent past to be reproduced better than the distant past, or the entire history since the beginning to be weighted equally. This is the same idea as representing a sound by its Fourier coefficients, or an image by the coefficients of a compression transform. What is new is the online requirement. The history keeps growing, so the approximation problem changes at every instant, and one needs the coefficients of the best approximation to be updated continuously as the window slides or expands. The HiPPO result The central result of the HiPPO paper is that, for a family of natural measures and with orthogonal polynomials as the basis, the vector of optimal coefficients c(t) evolves according to a linear differential equation of exactly the state space form: c′(t) = A(t) c(t) + B(t) f(t) The matrices A and B are not learned. They are derived in closed form from the choice of measure and basis. In other words, if one asks "what is the optimal way to compress the history of a signal into N polynomial coefficients, updated online?", the answer is "run a linear state space model with this particular A and B." The memory problem and the state space model turn out to be the same object. Different measures give different matrices. The paper worked out several. With a translated Legendre measure, which weights the last θ units of time uniformly and ignores everything earlier, the result is a sliding-window memory; this case had been derived independently a year earlier by Aaron Voelker, Ivana Kajić and Chris Eliasmith in their Legendre Memory Unit, published at NeurIPS in 2019, and deserves credit as a precursor. With a translated Laguerre measure, which weights the past with exponential decay, the result is a memory that emphasises the recent past and fades smoothly. And with what the authors called the scaled Legendre measure, abbreviated LegS, which weights the entire history from the start up to the present uniformly, the result is a memory that stretches as time goes on, always summarising everything seen so far on an equal footing. The scaled Legendre case is the important one. Its equation takes the form c′(t) = −(1/t) A c(t) + (1/t) B f(t) where A is a lower-triangular N-by-N matrix with entries determined by simple formulas. For row n and column k, counting from zero, the entry is the square root of (2n + 1) times the square root of (2k + 1) when n is greater than k, it is n + 1 on the diagonal, and it is zero above the diagonal. The vector B has entries equal to the square root of (2n + 1). The factor 1/t means the dynamics slow down as time passes, which is what allows the memory to cover an ever-longer history with the same N coefficients. What the coefficients mean The formula becomes intuitive when one looks at its smallest cases. Take N equal to one. Then A is the single number one, B is the single number one, and the equation reads c′(t) = (f(t) − c(t))/t. The solution of that equation is the running average of f from time zero to time t. So the one-dimensional HiPPO-LegS memory is the mean of everything seen so far. That is the best constant approximation to the history under a uniform weighting, which is exactly what the theory promises: the zeroth Legendre polynomial is a constant, and the best constant fit to a function is its average. Take N equal to two. The second coefficient now tracks the projection of the history onto the first-degree Legendre polynomial, a straight line across the history. Together the two coefficients describe the best straight-line fit to everything seen so far: its average level and its overall trend. With N equal to three, the memory adds a quadratic term, capturing curvature; with more coefficients it captures progressively finer structure. A memory of a few dozen coefficients can reconstruct a smooth signal over thousands of steps with small error, and the reconstruction error falls quickly as N grows when the signal is smooth. It is worth being clear about what such a memory is good at and what it is not. It is good at retaining the broad shape of a long history: trends, slow oscillations, the gist of what has happened. It is less good at retaining a single isolated spike, such as one specific token that appeared once ten thousand steps ago, because a sharp spike is precisely the kind of feature a low-order polynomial approximation smooths away. That limitation does not go away in later models, and it returns as a central theme in Chapter 9. Properties that matter for learning The HiPPO-LegS memory has several properties that were strikingly valuable for deep learning. The paper proved that it is robust to timescale: because the memory scales itself to the length of the history, stretching or compressing the input in time does not change the coefficients in any essential way. It has bounded gradients: the derivative of the memory with respect to past inputs neither vanishes nor explodes as the history grows, because the 1/t scaling and the structure of A balance one another. That property directly addresses the vanishing gradient problem that plagued recurrent networks. And its update can be computed efficiently, in time proportional to N per step, thanks to the structure of A. To demonstrate this, the authors replaced the memory cell of a gated recurrent network with a HiPPO-LegS update. On permuted sequential MNIST, a standard test in which the 784 pixels of a handwritten digit are fed one at a time in a fixed scrambled order so that the model must integrate information across the whole sequence, the HiPPO-based network reached accuracy around 98 per cent, a state-of-the-art result for recurrent models on that task at the time. More importantly, it did so with a memory mechanism that was derived rather than learned. From memory unit to deep layer HiPPO was presented as a better memory cell for recurrent networks. The next step, taken by Gu and colleagues in the Linear State-Space Layer, published at NeurIPS in 2021, was to use the state space model itself as a layer in a deep network, with HiPPO supplying the initial value of A and with the other parameters, including B, C and the step size Δ, learned by gradient descent. This step brought the three views of Chapter 3 into play. A linear state space layer could be run as a recurrence at inference or as a convolution during training. By stacking such layers with nonlinear mixing between them, the authors obtained a model that combined continuous-time memory with deep representation learning, and it performed strongly on long-sequence benchmarks. The step also made a change that later theory would need to explain. HiPPO-LegS is time-varying: the 1/t factor means the dynamics depend on the absolute time since the sequence began. A convolution, however, requires time-invariant dynamics. The linear state space layer, and the S4 model that followed, therefore dropped the 1/t factor and used a fixed step size Δ, giving time-invariant dynamics with the LegS matrix A. It was not obvious at first that this simplification retained any of HiPPO's guarantees. The 2023 paper "How to Train Your HiPPO", by Gu, Isys Johnson, Aman Timalsina, Atri Rudra and Christopher Ré, resolved the question. It showed that the time-invariant LegS system also performs an optimal polynomial projection, but onto Legendre polynomials under an exponentially warped measure, so that the memory remembers the whole past with exponentially decaying emphasis whose rate is set by Δ. The step size, in that light, controls how far back the memory effectively reaches, and a spread of learned step sizes across channels gives the model a spread of memory horizons. Why the initialisation matters so much A natural question is whether any of this is necessary. Gradient descent is powerful. Why not initialise A randomly and let training find a good memory? The empirical answer, from ablations in the S4 paper and subsequent work, is that random initialisation performs dramatically worse on tasks requiring long-range memory. The reasons follow from Chapter 3. A random matrix has eigenvalues scattered without regard to timescale. Those with large negative real parts forget within a few steps; those near zero or positive cause instability once the model is discretised and run for thousands of steps. Training must then discover, through noisy gradients flowing back over long sequences, a configuration of eigenvalues that spans the right timescales, and the very gradient pathologies the model is meant to avoid make that discovery difficult. Starting from HiPPO places the model in a region where it already remembers well, and learning refines the memory rather than having to invent it. There is a deeper lesson in this. The transformer's attention mechanism needs no special initialisation to remember, because it does not compress: it stores every token and lets the learned query-key similarity decide what to retrieve. A compressing memory has no such luxury. Its quality depends entirely on what it chooses to keep, and a good default choice, one grounded in approximation theory, is the difference between a model that can bridge thousands of steps and one that cannot. HiPPO's contribution was to make that choice principled. The shape of the problem that remained By 2021 the pieces were in place. A state space model with HiPPO initialisation provided a memory with strong long-range properties. The recurrent view provided cheap inference; the convolutional view promised parallel training. But the linear state space layer was impractical at scale. Computing its convolution kernel required forming powers of the dense N-by-N matrix Ā for every position up to the sequence length, which was expensive in both compute and memory. The naive approach to the recurrence was numerically unstable, because the HiPPO matrix has entries that grow rapidly with N, and repeated multiplication by its discretisation magnified rounding errors. The layer demonstrated what continuous-time memory could do, but it did not yet run fast enough to compete with transformers on large problems. Solving those computational problems, while keeping HiPPO's memory, was the achievement of S4, the structured state space sequence model. It is the subject of the next chapter, together with the cascade of simplifications and rivals that followed once researchers understood which parts of S4's elaborate machinery were essential and which were not. Chapter 5. S4 and Its Descendants The structured state space sequence model, universally called S4, was introduced by Albert Gu, Karan Goel and Christopher Ré in a paper titled "Efficiently Modeling Long Sequences with Structured State Spaces", presented at ICLR in 2022. Its name puts the emphasis in the right place. The memory theory already existed in HiPPO. What S4 added was structure: a way of representing the HiPPO matrix so that the model's convolution kernel could be computed quickly and stably for long sequences and large states. The result was the first architecture to make continuous-time memory practical at scale, and the first model of any kind to solve the hardest task on the Long Range Arena benchmark. S4 was also, in retrospect, more complicated than it needed to be. Within about eighteen months, a series of papers stripped away most of its mathematical machinery, discovered which parts mattered, and in doing so set the stage for Mamba. This chapter follows that arc: what S4 did, what it proved, and how its descendants simplified it. The computational problem S4 solved Recall the convolutional view from Chapter 3. For a sequence of length L, a time-invariant state space model computes its output by convolving the input with a kernel whose k-th entry is C Āᵏ B̄. Once the kernel is known, the convolution can be done with fast Fourier transforms in time proportional to L log L. The difficulty is computing the kernel. Done naively, it requires L successive multiplications by the N-by-N matrix Ā, costing on the order of N²L operations and requiring storage that grows the same way. With N in the hundreds and L in the tens of thousands, per channel, per layer, this was prohibitive. The earlier linear state space layer had run into exactly this wall, and had also suffered numerical instability, because powers of the discretised HiPPO matrix contain entries that grow exponentially and swamp the arithmetic. The obvious fix in linear algebra is to diagonalise. If A can be written as V Λ V⁻¹ with Λ diagonal, then every power of A is V Λᵏ V⁻¹, and powers of a diagonal matrix are trivial to compute. The HiPPO-LegS matrix can in principle be diagonalised, but the matrix V that does so has entries exponentially large in N, so the diagonalisation is numerically useless. Any computation that went through V would lose all precision. S4's insight was that the HiPPO matrices, though not stably diagonalisable, have a special structure: each can be written as a normal matrix plus a low-rank correction. Normal matrices are exactly those that can be diagonalised by a unitary transformation, which is perfectly stable. So the HiPPO matrix is, up to a stable change of basis, a diagonal matrix plus a matrix of rank one or two. The authors called this the normal plus low-rank form, and its practical counterpart the diagonal plus low-rank form. With that structure in hand, S4 computed the kernel not directly but through its generating function, the polynomial whose coefficients are the kernel entries. Evaluating that generating function at the L complex roots of unity gives precisely the discrete Fourier transform of the kernel, from which the kernel can be recovered with an inverse transform. For a diagonal-plus-low-rank matrix, the generating function reduces, via a classical identity for inverting a matrix plus a low-rank correction known as the Woodbury identity, to a sum of simple fractions in which each term involves one diagonal entry. Such sums are known as Cauchy kernels, and fast, stable algorithms exist for evaluating them. The overall cost of computing the kernel dropped to roughly linear in N plus L, up to logarithmic factors, and the computation was numerically stable because it never formed a large matrix power. The technical apparatus was formidable, and it is one reason S4 was considered difficult to understand and implement. But its purpose was simple: to compute a long convolution kernel, determined by a structured state matrix, cheaply and without numerical blow-up. What S4 demonstrated The results were striking enough to change many researchers' minds about recurrent models. On the Long Range Arena, introduced in Chapter 1, S4 outperformed every efficient transformer variant by a wide margin on average. The headline result was Path-X, the task of deciding whether two points in a flattened 128-by-128 image are connected, presented as a sequence of 16,384 pixels. Every model evaluated in the original benchmark, including the full transformer, had scored at the level of chance, fifty per cent. S4 scored around 88 per cent in its original paper and higher in later refinements. It was the first model to demonstrate that a neural network could learn a reasoning task requiring integration of information across more than sixteen thousand steps. On sequential CIFAR-10, where images are fed pixel by pixel with no two-dimensional structure available, S4 reached accuracy above 90 per cent, comparable to some convolutional networks that see the image in two dimensions. On raw speech classification, using audio sequences of sixteen thousand samples, it matched or beat specialised models that worked on hand-crafted spectrogram features, and as noted in Chapter 3 it could be evaluated at a different sampling rate without retraining. On language modeling it came close to, but did not match, transformers of similar size on WikiText-103, while generating text far faster because each new token required only a recurrent state update. That last result contained the seed of the next phase. S4 was excellent on continuous, signal-like data and on synthetic long-range reasoning tasks. On language, the domain that mattered most commercially, it was good but not quite good enough. Understanding why would require looking at what kind of computation language demands, which is the subject of Chapter 6. First, though, the field simplified S4 dramatically. Diagonal is enough In 2022, Ankit Gupta, Albert Gu and Jonathan Berant published "Diagonal State Spaces are as Effective as Structured State Spaces". The title states the finding. If one simply drops the low-rank correction and uses a purely diagonal state matrix, initialised from the diagonal part of the HiPPO matrix's normal-plus-low-rank decomposition, performance on the Long Range Arena is essentially unchanged. The elaborate Cauchy kernel machinery becomes unnecessary: with a diagonal A, the kernel entries are sums of powers of individual complex numbers, which can be computed with a simple matrix product known as a Vandermonde product. Gu, Gupta, Goel and Ré followed with a systematic study of diagonal state space models, published as S4D later the same year. They explained why the diagonal approximation works, by showing that the diagonal part of the LegS decomposition produces, in the limit of large N, the same kernels as the full HiPPO matrix. They also proposed even simpler initialisations. The S4D-Lin initialisation, for instance, sets the n-th diagonal entry to −1/2 plus iπn: every mode decays at the same rate, and the modes oscillate at evenly spaced frequencies, much like the components of a Fourier series. With such initialisations, a diagonal state space model could be implemented in a few dozen lines of code and matched S4 on most tasks. This was an important clarification. The continuous-time framing and the principle of initialising for long memory were essential. The specific normal-plus-low-rank structure of HiPPO was not. What mattered was a diagonal state matrix with eigenvalues spread sensibly across timescales and frequencies, discretised properly with a learned step size. Scans instead of convolutions A second simplification changed how the model was computed. Jimmy Smith, Andrew Warrington and Scott Linderman introduced S5 at ICLR in 2023. Where S4 ran an independent single-input, single-output state space model for each channel and computed each one's convolution kernel, S5 used a single multi-input, multi-output state space model with a shared diagonal state of moderate size, and it computed the model not by convolution but directly through its recurrence, using a parallel scan. A parallel scan, explained fully in Chapter 7, is an algorithm for computing all the intermediate results of a linear recurrence in parallel, with a number of sequential stages proportional to the logarithm of the sequence length. Because S5's state matrix was diagonal, each step of the recurrence was an element-wise operation, and the scan was efficient. S5 matched S4's results on the Long Range Arena while being conceptually much simpler. More importantly, by computing the recurrence directly rather than going through a convolution kernel, it opened a door that the convolutional approach kept shut. A convolution requires time-invariant parameters. A scan does not. Any linear recurrence, even one whose parameters change at every step, can be computed by a scan. S5 did not yet exploit this freedom, but Mamba would. Do we even need continuous time? The third line of simplification asked whether the continuous-time story was needed at all. In 2023, Antonio Orvieto, Samuel Smith, Albert Gu and colleagues at Google DeepMind published "Resurrecting Recurrent Neural Networks for Long Sequences", introducing the linear recurrent unit. They started from an ordinary recurrent network and applied a sequence of changes, checking at each stage what was necessary to match S4 on the Long Range Arena. They removed the nonlinearity from the recurrence, making it linear. They diagonalised the recurrent matrix, using complex numbers. They parameterised the diagonal entries in an exponential form so that their magnitudes stayed below one, and initialised them close to the unit circle so that memory would decay slowly. They added a normalisation factor so that the state's magnitude did not grow uncontrollably at initialisation. With those changes, a recurrent network with no continuous-time derivation and no discretisation matched the performance of S4 and its variants. The authors' interpretation was that the essential ingredients were linearity, diagonal complex-valued dynamics, stable parameterisation and careful initialisation near the boundary of stability, which is where long memory lives. The continuous-time theory, on this view, is one good route to finding such parameters, not the only one. The two perspectives are compatible, and they converged. Discretising a diagonal continuous system with eigenvalues of small negative real part produces exactly a diagonal discrete system with magnitudes slightly below one. The continuous view supplies intuition about timescales and the step size; the discrete view shows that what ultimately matters is where the eigenvalues of the recurrence sit. Closing the gap on language While these simplifications were under way, a parallel effort tried to make state space models competitive on language. The most influential was H3, from Daniel Fu, Tri Dao and colleagues, presented at ICLR in 2023 under the playful title "Hungry Hungry Hippos". The authors began by diagnosing what state space models were failing to do. They built small synthetic tasks capturing operations that attention performs easily: associative recall, in which the model sees a list of key-value pairs and must output the value for a queried key, and induction, in which the model must notice that a token appeared earlier and predict whatever followed it then. On these tasks, S4 and its variants struggled, while attention solved them easily. The H3 layer added two state space models with different roles. The first, a shift state space model, effectively remembered the last few tokens. The second, a diagonal state space model, accumulated a longer-term summary. Between them, H3 inserted multiplicative interactions, element-wise products of projections of the input, which played a role resembling the query-key comparison in linear attention. This let the layer perform a crude form of lookup. A pure H3 model came within a small margin of transformers on language modeling perplexity, and a hybrid that kept just two attention layers outperformed a transformer of comparable size on the OpenWebText corpus. The paper also introduced FlashConv, a hardware-aware implementation of the long convolutions, foreshadowing the emphasis on kernels that would define Mamba. Hyena, by Michael Poli, Stefano Massaroli and colleagues at ICML in 2023, took a different route. It abandoned the state space parameterisation of the kernel and instead used long convolutions whose kernels were generated implicitly by a small neural network, interleaved with element-wise gating by projections of the input. The gating made the overall operator data-dependent, even though each convolution was fixed. Hyena reached transformer quality on language modeling at moderate scale with lower training compute, and its descendants, such as StripedHyena, were later used for very long sequences in genomics, a thread Chapter 10 returns to. Table 2 summarises how these models differ on the dimensions that turned out to matter. Table 2. Structured state space models and close relatives before Mamba. Model (year) State matrix Training computation Parameters depend on input? Main contribution S4 (2022) HiPPO, normal plus low-rank Long convolution via Cauchy kernel and FFT No Stable, fast kernel; solved Path-X DSS and S4D (2022) Diagonal, complex Long convolution via Vandermonde product No Showed diagonal suffices; simple initialisations S5 (2023) Diagonal, shared multi-input multi-output Parallel scan over the recurrence No Replaced convolution with scan LRU (2023) Diagonal, complex, near unit circle Parallel scan No Matched S4 without continuous-time derivation H3 (2023) Shift plus diagonal state space models Long convolution (FlashConv) Only through multiplicative gating Diagnosed recall gap; near-transformer perplexity Hyena (2023) Implicit long convolution, no explicit state FFT convolution Only through multiplicative gating Transformer-level perplexity at moderate scale Sources: the respective papers, as listed in Further Reading. The penultimate column of Table 2 is the one that matters. Every model in this lineage computed its sequence mixing with parameters that were the same at every position. Where data dependence entered at all, it did so only through gating applied around a fixed, time-invariant core. The time-invariance was what enabled the convolutional shortcut, and it had seemed to be the price of efficiency. Yet the synthetic tasks introduced by H3, associative recall and induction, kept pointing at the same deficiency. To recall a specific earlier token, a model must treat that token differently from its neighbours at the moment it arrives, holding it in memory while letting less important tokens pass. A system whose dynamics are identical for every input cannot do that. It applies the same filter to everything. The next chapter explains this limitation precisely and describes the step that removed it: making the state space model's own parameters depend on the input. It is the step that gave Mamba its name and its results. Chapter 6. Selection: The Idea That Made Mamba In December 2023, Albert Gu and Tri Dao released a paper titled "Mamba: Linear-Time Sequence Modeling with Selective State Spaces". It reported that a model built entirely from a new kind of state space layer, with no attention at all, matched or exceeded transformers of the same size on language modeling, while generating text with several times higher throughput and scaling linearly in sequence length. Mamba models of around three billion parameters, trained on 300 billion tokens of the Pile, outperformed transformers of the same size on standard downstream evaluations and matched transformers twice as large. For the first time, a recurrent architecture was a credible replacement for attention in the domain where attention was strongest. The core idea fits in a sentence: make the state space model's parameters depend on the current input. This chapter explains why that change was necessary, what exactly it alters, how it relates to the gates of older recurrent networks, and what it costs. What time-invariance cannot do Chapter 5 ended with a puzzle. Structured state space models were excellent on audio, images and synthetic long-range tasks, yet persistently weaker on language and on simple recall tasks. Gu and Dao sharpened the diagnosis with two synthetic problems. The first is a variant of copying. In the plain copying task, a model sees a short sequence of tokens, then a long stretch of blank filler, and must reproduce the original tokens at the end. A time-invariant model can solve this easily, because the gap between the tokens to be copied and the point where they must be reproduced is always the same. A convolution kernel that simply shifts its input by the right fixed distance does the job. In the selective copying task, by contrast, the tokens to be remembered are scattered at random positions among filler tokens, and the model must output them in order while ignoring the filler. Now no fixed kernel works, because the relevant tokens arrive at different times in each example. The model must recognise, from the content of each token, whether it is something to keep or something to discard. The second problem is the induction head task, named after the circuit that researchers at Anthropic found transformers use for in-context learning. The model sees a sequence in which a special token appears, followed by some other token, and later sees the special token again. It must predict that the token which followed the special token the first time will follow it again. Solving this requires noticing a match between the current token and an earlier one, and retrieving what came after that earlier occurrence. Again, which positions matter depends on content, not on fixed offsets. A linear time-invariant system applies exactly the same dynamics to every input. Its state evolves by the same matrix Ā at every step and absorbs each input through the same vector B̄. It cannot decide to store one token and skip the next, because it treats them identically. It can weight inputs differently by how long ago they arrived, which is what a convolution kernel does, but it cannot weight them differently by what they are. The models of Chapter 5 that attached data-dependent gating outside a time-invariant core, such as H3 and Hyena, recovered some content dependence, but the memory itself, the state, still evolved without regard to content. Language is full of situations where the right response depends on what a token is rather than where it sits: a name introduced once and referred to later, a bracket that must be closed, an instruction that changes how everything after it should be read. A memory that cannot filter by content will fill with irrelevant material and lose the few items that matter. Making the parameters depend on the input Mamba's selective state space model removes time-invariance in a targeted way. Three of its parameters become functions of the current input token: the step size Δ, the input vector B and the output vector C. The state matrix A stays fixed, but because the discretised transition is Ā = exp(ΔA), it too becomes input-dependent through Δ. Concretely, for an input vector xₜ at position t, the model computes Bₜ = Linear(xₜ), Cₜ = Linear(xₜ), Δₜ = softplus(bias + Linear(xₜ)) where each "Linear" is a learned linear projection, and softplus is a smooth function that maps any real number to a positive one, ensuring the step size is always positive. In the original design, the projection producing Δ is low-rank, a cheap way of letting a small summary of the input control the step size for every channel. The recurrence then runs as before, but with parameters that change at every step: hₜ = Āₜ hₜ₋₁ + B̄ₜ xₜ, yₜ = Cₜ hₜ with Āₜ = exp(Δₜ A) and, in Mamba's simplified discretisation, B̄ₜ = Δₜ Bₜ. The state matrix A is diagonal and real, with entries initialised to negative values such as −1, −2, −3 and so on up to −N, a choice the S4D work called S4D-Real. The default state size per channel in the original Mamba is N equal to 16, smaller than in some earlier models; the expressive power now comes from selectivity rather than from a large time-invariant state. The authors found that complex-valued states, which were helpful for continuous signals such as audio, were unnecessary for language, where real values worked as well or better. That finding would be revisited in Mamba-3, as Chapter 9 explains. What each selective parameter does Each selective parameter controls a distinct aspect of how the model handles the current token, and they are worth taking one at a time. The step size Δ controls the balance between the past and the present. Recall the scalar example from Chapter 3: a small step makes the transition close to one, so the state is preserved and the new input contributes little; a large step makes the transition close to zero, so the old state is wiped and replaced by the current input. When Δ depends on the input, the model can choose, token by token, which of these regimes to be in. Faced with a filler token in the selective copying task, it can produce a tiny Δ, effectively letting the token pass without changing the state, so that the important tokens already stored are preserved. Faced with a token it must remember, it can produce a larger Δ and write it in. Faced with a boundary, such as the start of a new document in a concatenated training batch, it can produce a very large Δ and reset its memory entirely. There is an attractive way to think about this. In the continuous-time picture, Δ is the length of the interval each token represents. An input-dependent Δ means the model stretches time for important tokens and compresses it to nearly nothing for unimportant ones. The model's internal clock runs at a rate determined by content. A stream of irrelevant tokens passes in an instant of internal time, while a crucial token occupies a long interval during which it reshapes the state. The input vector B controls what is written. Because it varies with the token, the model can decide which dimensions of its state each token should affect, and how strongly. This is a content-dependent write address. The output vector C controls what is read. Because it too varies with the token, the model can query different parts of its state depending on what it currently needs. This is a content-dependent read address. Readers who recall Chapter 2 will notice something. A content-dependent write vector, a content-dependent read vector and a fixed-size state: that is the structure of linear attention, in which the key determines the write, the query determines the read and the state is a matrix of accumulated associations. Selection has moved the state space model towards attention. Chapter 8 shows that, in a precise sense, it arrives there. Gating rediscovered The Mamba paper includes a short theorem that makes the connection to older recurrent networks exact. Consider the simplest selective model: a single channel, a state of size one, A equal to −1 and B equal to 1, with Δ given by softplus of a linear function of the input. Then the discretised transition is exp(−softplus(z)), where z is that linear function, and a short calculation shows this equals 1 − σ(z), where σ is the logistic sigmoid. The recurrence becomes hₜ = (1 − gₜ) hₜ₋₁ + gₜ xₜ, where gₜ = σ(zₜ) This is precisely the gating mechanism of the gated recurrent unit and, in essence, of the LSTM's forget and input gates. A selective state space model with a one-dimensional state is a gated recurrent network. Selection is gating, rediscovered from the continuous-time side. That observation clarifies what Mamba is and is not. It is not a wholly new principle. The idea that a recurrent memory should decide, from each input, how much to keep and how much to admit is as old as the LSTM. What is new is the combination: gating applied inside a linear recurrence, so that it can be parallelised; a multi-dimensional state with HiPPO-inspired initialisation, so that memory spans many timescales; per-channel step sizes, so that different channels can remember over different horizons; and content-dependent read and write vectors, so that the state behaves like an associative memory. Earlier gated linear recurrences, such as the quasi-recurrent network, had some of these ingredients but not all, and none had an implementation that made them competitive at scale. On the synthetic tasks, the effect was decisive. On selective copying, time-invariant models such as S4 performed poorly, while the selective version solved the task almost perfectly. On the induction head task, Mamba, trained on sequences of 256 tokens, solved the task perfectly and extrapolated to test sequences of around a million tokens, some four thousand times longer than anything seen in training. Attention-based models trained the same way failed to generalise beyond a modest multiple of their training length. The Mamba block The selective state space model is one component of a larger layer. The Mamba block combines elements of the H3 layer and of the gated feed-forward networks used in modern transformers into a single, homogeneous unit that is simply stacked, replacing both the attention and the feed-forward sublayers. An input vector of width D is first normalised and projected up to a wider dimension, typically twice D, along two parallel branches. On the main branch, the expanded signal passes through a short causal convolution, typically spanning four tokens, which gives the model cheap access to the immediate local context. It then passes through a nonlinearity, the SiLU function, and into the selective state space model, which operates independently on each of the expanded channels, each with its own state of size N. On the second branch, the expanded signal passes through a SiLU nonlinearity alone, forming a gate. The two branches are multiplied element by element, and the result is projected back down to width D and added to the residual stream. The design is deliberately simple. There are no separate attention and feed-forward sublayers, just one kind of block repeated. In parameter terms, two Mamba blocks are roughly equivalent to one transformer layer with its attention and feed-forward network. Most of the parameters sit in the input and output projections, which are ordinary matrix multiplications; the selective state space model itself has comparatively few parameters but does the work of mixing information along the sequence. The results were reported across several domains. On language modeling with the Pile, Mamba matched the scaling curve of a strong modern transformer recipe, with rotary position embeddings, gated feed-forward layers and other improvements, from about 125 million to 1.3 billion parameters, and was the first attention-free model to do so. On DNA sequence modeling using the human reference genome, Mamba's performance continued to improve as the context length grew to a million base pairs, whereas a comparable Hyena-based model's performance degraded with longer contexts. On raw audio waveforms, Mamba improved on prior state-of-the-art models for both modeling and generation. In inference throughput, Mamba generated tokens about five times faster than a transformer of similar size, because it had no key-value cache to read and could therefore batch many more sequences. The price of selection All of this came at a cost that is easy to overlook. The models of Chapter 5 were fast to train because their time-invariance meant the whole state space model collapsed into a single convolution kernel. The kernel could be computed once and applied with fast Fourier transforms, and the state itself never had to be materialised during training. A selective model has no such shortcut. Because Āₜ and B̄ₜ change at every step, there is no single kernel; the influence of input s on output t depends on every transition in between. The convolution view is gone. What remains is the recurrent view, which is linear in the state and therefore parallelisable by a scan, but which requires computing the state at every position. That state is large. For a batch of B sequences of length L, with an expanded width of E times D channels each carrying a state of size N, the full set of states has B × L × E × D × N numbers. For a modest configuration, with D of 2,048, an expansion of two, a state size of 16 and a single sequence of 2,048 tokens, that is over 130 million numbers per layer, sixteen times larger than the layer's input and output combined. Writing all of it to GPU memory and reading it back, layer after layer, would make the model slow and memory-hungry, erasing the efficiency gains that motivated the whole enterprise. Mamba's second contribution, as important as the first, was an implementation that computes the selective scan without ever writing that expanded state to main memory. It required understanding exactly where the time goes on a modern GPU, and arranging the computation around that knowledge. The next chapter describes how it works. Chapter 7. The Parallel Scan and the Shape of the Hardware A selective state space model is a linear recurrence whose coefficients change at every step. Training it efficiently means computing all of its states, across sequences of thousands or millions of tokens, on hardware designed for something rather different. Two ideas make this possible. The first is mathematical: a linear recurrence, even a time-varying one, can be computed in parallel by an algorithm called a scan. The second is architectural: on a modern GPU, the cost of a computation is governed less by how many arithmetic operations it performs than by how much data it moves between different levels of memory. Mamba's implementation combines the two, and the combination is what makes selection affordable. This chapter explains both ideas from the ground up, because they recur in every efficient sequence model and because they show why the field's progress has depended as much on systems engineering as on theory. Why a linear recurrence can be parallelised Consider the recurrence for a single channel and, for simplicity, a single state dimension: hₜ = aₜ hₜ₋₁ + bₜ where aₜ is the discretised transition at step t and bₜ is the discretised input contribution. Computed directly, each state requires the one before it, so computing L states takes L sequential steps. On a GPU, which achieves its speed by performing thousands of operations at once, a computation with a long chain of sequential dependencies leaves most of the hardware idle. The way out is to notice that each step of the recurrence is a function, and that these functions can be composed in any grouping. Step t takes a state h and returns aₜ h + bₜ. Represent that step by the pair (aₜ, bₜ). Applying step s and then step t gives aₜ(aₛ h + bₛ) + bₜ, which equals (aₜ aₛ) h + (aₜ bₛ + bₜ). So the composition of the two steps is itself a step of the same form, represented by the pair (aₜ aₛ, aₜ bₛ + bₜ). Define an operation that combines two pairs in this way. Because function composition is associative, this operation is associative: combining three steps gives the same result whether the first two are combined first or the last two. Associativity is the key. Any associative operation can be accumulated over a sequence, producing all the running results, by a parallel prefix algorithm, usually called a scan. The canonical example is the running sum of a list of numbers, where the operation is addition. The idea was developed in the 1970s and 1980s in the theory of parallel computation, with influential formulations by Richard Ladner and Michael Fischer in 1980, by Daniel Hillis and Guy Steele in 1986, and by Guy Blelloch, whose 1990 report "Prefix Sums and Their Applications" set out the work-efficient version most often used today. How the scan works A small example shows the structure. Suppose the sequence has eight steps, represented by eight pairs. In the first stage, combine adjacent pairs in parallel: steps one and two, three and four, five and six, seven and eight. That produces four composite steps, each spanning two original steps, and all four combinations happen at once. In the second stage, combine adjacent composites: the first two and the last two, producing two composites each spanning four steps. In the third stage, combine those two, producing a single composite spanning all eight. This up-sweep takes three stages for eight elements, and in general the logarithm to base two of the sequence length. At this point the algorithm knows the total effect of the whole sequence and of various power-of-two blocks within it, but not the running state at every position. A second pass, the down-sweep, propagates partial results back down the tree, combining each block's composite with the accumulated effect of everything to its left, until every position has its correct running state. This also takes a logarithmic number of stages. In total, the scan performs a number of combinations proportional to the sequence length, the same order of work as the sequential loop, but arranges them into about twice the logarithm of the length in sequential stages. For a sequence of a million tokens, that is around forty stages instead of a million. To make this concrete with numbers, take four steps with transitions a of 0.5, 0.5, 0.5 and 0.5, and inputs b of 1, 2, 3 and 4, starting from a zero state. The sequential recurrence gives states 1, then 0.5 times 1 plus 2 equals 2.5, then 0.5 times 2.5 plus 3 equals 4.25, then 0.5 times 4.25 plus 4 equals 6.125. The scan instead first combines steps one and two into the pair (0.25, 0.5 × 1 + 2), which is (0.25, 2.5), and steps three and four into (0.25, 0.5 × 3 + 4), which is (0.25, 5.5). Combining those gives (0.0625, 0.25 × 2.5 + 5.5), which is (0.0625, 6.125). Applied to a zero initial state, this yields 6.125 for the final state, matching the sequential answer. The down-sweep recovers the intermediate states 2.5 and 4.25 the same way. The same reasoning extends directly to the full model. When the state is a vector and the transition matrix is diagonal, as in Mamba, each state dimension is an independent scalar recurrence, so the scan runs element-wise over all dimensions at once. When the transition is a general matrix, the composition multiplies matrices, which is far more expensive; this is one practical reason why diagonal transitions dominate. S5 and the linear recurrent unit, described in Chapter 5, already used scans, implemented with general-purpose associative scan primitives such as the one provided in the JAX library. Mamba's innovation was not the scan itself but how it was executed. Where the time actually goes To see why execution matters, one needs a rough picture of a GPU's memory. A modern data-centre accelerator such as the NVIDIA A100 or H100 has a large pool of high-bandwidth memory, tens of gigabytes, where model weights, activations and inputs live. Reading from and writing to that memory runs at somewhere between about 1.5 and 3.4 terabytes per second, depending on the generation. The chip itself is divided into many streaming multiprocessors, over a hundred on these parts, and each has a small amount of very fast on-chip memory, called shared memory or SRAM, of the order of a couple of hundred kilobytes, along with registers. Collectively the on-chip memory amounts to some tens of megabytes, and its bandwidth is roughly an order of magnitude higher than that of the main memory. The FlashAttention paper estimated around nineteen terabytes per second for the A100's on-chip memory, against about 1.5 to 2 for its main memory. The arithmetic units are faster still. An A100 can perform hundreds of trillions of low-precision multiply-adds per second on its tensor cores, the specialised units for matrix multiplication. If a computation must fetch each number from main memory and does only a few operations with it, the arithmetic units sit idle waiting for data. Such a computation is said to be memory-bound. Matrix multiplications of large matrices reuse each number many times and are compute-bound; element-wise operations, such as adding two vectors or applying an exponential, are almost always memory-bound. A state space scan is made almost entirely of element-wise operations. Discretising involves an exponential and a few multiplications per element. The scan's combination step is a multiply and an add. The output is a multiply by C and a sum over the state dimension. If each of these were performed as a separate operation, reading inputs from main memory and writing results back, the computation would spend nearly all its time moving data. And as Chapter 6 calculated, the biggest piece of data, the expanded state tensor of size batch × length × channels × state size, is many times larger than the layer's input and output. Fusing the scan Mamba's implementation, written as custom GPU kernels, avoids those memory trips through three techniques. The first is kernel fusion. Instead of running the discretisation, the scan and the output projection as separate operations, each reading and writing main memory, the kernel performs all of them in one pass. It loads the inputs it needs, namely the input sequence, the step sizes, the selective vectors B and C, and the fixed matrix A, from main memory into on-chip memory. It computes the discretised transitions and inputs there. It runs the scan there. It multiplies by C and sums over the state dimension there. And it writes back only the final output, which has the same size as the input. The large intermediate state tensor, with its extra factor of N, never touches main memory at all. Since the main memory traffic is reduced roughly by that factor, the speedup on a memory-bound computation is correspondingly large. The second technique is recomputation. Training requires a backward pass, which normally needs the intermediate values from the forward pass. Storing the full state tensor for the backward pass would reintroduce exactly the memory cost the fusion avoided. Mamba's kernel does not store it. During the backward pass, it reloads the same small inputs from main memory and recomputes the states on chip, then computes gradients. Recomputation costs extra arithmetic, but arithmetic is cheap and memory traffic expensive, so the trade is favourable. FlashAttention uses the same technique for attention. With it, the selective scan layer's memory requirement during training is comparable to that of an optimised transformer layer using FlashAttention. The third is careful partitioning of work. Different channels of a Mamba layer are completely independent recurrences, so they can be distributed across the GPU's many processors without any communication. Within a processor, the sequence is processed in chunks small enough to fit on chip, with a parallel scan within each chunk and the running state carried from one chunk to the next. Batch and channels supply abundant parallelism; the scan exploits the remainder along the sequence. The authors reported that their fused selective scan was many times faster than a straightforward implementation in a standard deep learning framework, and faster than FlashAttention-2 for sequences beyond about two thousand tokens, with the advantage growing as sequences lengthened, since the scan's cost is linear and attention's quadratic. Inference: the recurrence pays off At inference time the picture is simpler and more favourable. A language model processes a prompt and then generates tokens one by one. Processing the prompt, the prefill, can use the same parallel scan as training, since the whole prompt is available at once. Generation then switches to the recurrent view: for each new token, the model updates its state with a single step of the recurrence, computes the output and moves on. It needs to keep only the current state for each layer, along with the last few inputs for the short causal convolution in the Mamba block. That memory is constant, and it is small. Take a model with the published shape of the original Mamba-2.8B: 64 layers, a model width of 2,560, an expansion factor of two giving 5,120 inner channels, and a state size of 16 per channel. The recurrent state per layer is 5,120 × 16, or 81,920 numbers, plus a convolution buffer of three previous inputs per channel, about 15,000 numbers. Across 64 layers that is roughly six million numbers, or about twelve megabytes in 16-bit precision. That figure does not change whether the model has read a hundred tokens or a million. A transformer of similar size, with 32 layers of width 2,560 and standard multi-head attention, stores two vectors of width 2,560 per layer per token, about 330 kilobytes per token in 16-bit precision. At a hundred thousand tokens of context, its cache is around 33 gigabytes, nearly three thousand times larger than the Mamba model's state. This difference in memory compounds into throughput. Because generation is bandwidth-bound, reading a small state is faster than reading a large cache, so each token is produced more quickly. And because each sequence occupies so little memory, many more sequences can be processed in a single batch, which raises the arithmetic intensity of the weight multiplications and lets the hardware run closer to its peak. Those two effects together account for the five-fold generation throughput advantage reported in the Mamba paper. The constant state also simplifies distributing very long sequences across devices. To split a sequence of millions of tokens across several accelerators, each device can process its segment and pass only its final state, a few megabytes, to the next device, which uses it as the initial state for its own segment. A corrected pass can then propagate exact states within each segment. Attention, by contrast, requires every device to see the keys and values of all earlier segments, which is why schemes such as ring attention must circulate key-value blocks among all participants. What the first implementation left on the table The fused scan solved the memory-traffic problem, but it left one inefficiency unaddressed, and that inefficiency shaped the next generation of the architecture. Mamba's scan is built from element-wise operations, so it runs on the GPU's general-purpose arithmetic units rather than on its tensor cores. On an A100, the tensor cores deliver about 312 trillion operations per second in 16-bit precision, against about 19.5 trillion for 32-bit arithmetic on the general units, a gap of roughly sixteen times. On the H100 the gap is similar. A transformer spends most of its time in matrix multiplications that use the tensor cores. The original Mamba scan, however well it managed memory, could not. This mattered most during training, where throughput determines how much data a model can see for a given budget. It also constrained the state size: since the scan's cost grows with N, and the computation could not use the fastest units, Mamba's state size was kept small, at 16 per channel. A larger state would remember more, but at a cost the hardware made steep. The obvious question was whether a selective state space model could be reformulated so that most of its computation became matrix multiplication. Answering it led Gu and Dao to a theoretical result that reframed the whole relationship between state space models and attention, and to the second version of Mamba. That result, and the family of related architectures that emerged around it, is the subject of the next chapter. Chapter 8. Duality: When State Space Models Became Attention In May 2024, Tri Dao and Albert Gu published a paper with a title designed to provoke: "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality". It introduced Mamba-2, but its larger contribution was conceptual. It showed that a selective state space model, under one mild restriction, computes exactly the same function as a particular form of masked linear attention, and that the two descriptions correspond to two different algorithms for the same computation. The recurrent algorithm is linear in sequence length but uses slow element-wise operations. The attention-like algorithm is quadratic but uses fast matrix multiplications. A hybrid algorithm that switches between them at the right scale gets the best of both. That result dissolved the apparent opposition between state space models and attention that had framed the field since S4. It also revealed that a whole cluster of architectures developed independently in 2023 and 2024, under names such as RetNet, gated linear attention, RWKV, xLSTM and DeltaNet, were members of a single family of linear recurrences differing mainly in how they forget and how they write. This chapter explains the duality, the algorithm it enables, and the family it unified. A sequence layer as a matrix Every sequence-mixing layer considered in this book, attention included, can be written in the same abstract form: the output sequence is a matrix times the input sequence. For a single channel, if x is the vector of inputs at all L positions and y the vector of outputs, then y = M x for some L-by-L matrix M. For a causal model, M is lower triangular, since no output may depend on a future input. The entry in row j and column i says how much input i contributes to output j. For softmax attention, M is the attention matrix, the softmax of the masked query-key scores; it depends on the input, and it is computed explicitly, at quadratic cost. For a selective state space model, unrolling the recurrence gives an explicit formula for M too. The contribution of input i to output j is the product of three things: the input vector Bᵢ, which writes input i into the state; the transitions Āᵢ₊₁ through Āⱼ, which carry that contribution forward from step i to step j; and the output vector Cⱼ, which reads the state at step j. So Mⱼᵢ = Cⱼᵀ Āⱼ Āⱼ₋₁ ⋯ Āᵢ₊₁ Bᵢ Matrices with this structure have a name in numerical linear algebra: they are semiseparable. Every block lying entirely below the diagonal has rank at most N, the state size, because every path from an earlier input to a later output passes through the N-dimensional state. This low-rank structure below the diagonal is precisely what allows the matrix to be applied in linear time by a recurrence instead of quadratic time by explicit multiplication. A recurrent model is a structured matrix, and its state size is the rank of that structure. The duality Mamba-2 makes one restriction relative to Mamba. Instead of a diagonal transition with a separate decay for each state dimension, the transition at each step is a single scalar times the identity. All dimensions of the state decay at the same rate at a given step, though that rate still depends on the input. Call that scalar aₜ. Then the product of transitions from step i + 1 to step j is just the product of scalars aᵢ₊₁ ⋯ aⱼ, and the matrix becomes Mⱼᵢ = (aᵢ₊₁ ⋯ aⱼ) · Cⱼᵀ Bᵢ The second factor, Cⱼᵀ Bᵢ, is a dot product between a vector computed from token j and a vector computed from token i. Call C the query and B the key, and that is exactly the query-key score of attention, without the softmax. The first factor is a scalar that depends only on the positions and on the decays between them: a causal mask that, instead of being simply one or zero, decays smoothly with the cumulative product of the gates. So the whole matrix is a masked attention matrix, with a learned, input-dependent decaying mask, applied to the input as the value. Written in matrix form, M = L ∘ (C Bᵀ), where L is the decaying causal mask and ∘ denotes element-wise multiplication. This is linear attention, as in Chapter 2, with a forgetting mechanism built into the mask. Dao and Gu called the correspondence structured state space duality, abbreviated SSD. The same model can be computed in two ways: as a recurrence over an N-dimensional state, at linear cost, or as a masked attention matrix, at quadratic cost. It is worth absorbing how much this unifies. The query, key and value of attention correspond to the C, B and x of a state space model. The state size N corresponds to the dimension of the queries and keys. The selective step size Δ corresponds to a data-dependent forget gate on an attention-like memory. Linear attention without decay is the special case in which all the gates aₜ are one. The connection that Chapter 6 noticed informally, that content-dependent read and write vectors made Mamba resemble linear attention, is here made exact. The chunked algorithm The practical payoff is an algorithm that runs mostly on tensor cores. Divide the sequence into chunks of moderate length, say 64 to 256 tokens. Within each chunk, use the attention form: compute the small chunk-sized masked attention matrix and multiply it by the inputs. These are matrix multiplications of modest size, exactly the kind of work tensor cores excel at, and their quadratic cost is confined to the chunk length, which is fixed. Between chunks, use the recurrent form: compute the state at the end of each chunk, again with matrix multiplications, then run a short recurrence over the chunk-boundary states to propagate information from each chunk to all later ones. Finally, convert each chunk's incoming state into its contribution to that chunk's outputs, once more by matrix multiplication. The total work grows linearly with sequence length, since the number of chunks does, and almost all of it is matrix multiplication. Only the short inter-chunk recurrence, over a sequence that is shorter than the original by a factor of the chunk length, remains sequential, and it is cheap. The authors reported that this core layer ran two to eight times faster than Mamba's fused selective scan. More importantly, it allowed much larger state sizes. Mamba used N of 16; Mamba-2 commonly used 64 to 256, with only modest slowdown, because increasing N now mainly increased the size of matrix multiplications that the hardware handled efficiently. What else changed in Mamba-2 The block design changed as well. Mamba-2 is organised in heads, like multi-head attention: the inner channels are split into groups of typically 64, and each head shares a single scalar decay and a single pair of B and C vectors across its channels. In the attention analogy, B and C are shared across heads in a way resembling multi-query attention, so that many value channels are read and written with the same keys and queries. The input projections that produce x, B, C and Δ are all computed in parallel at the beginning of the block, rather than sequentially as in Mamba, which simplifies splitting a large layer across multiple GPUs. An extra normalisation is added before the output projection to stabilise training at scale. The scalar decay is a genuine restriction. Mamba's diagonal decay could forget at different rates in different state dimensions within a single channel; Mamba-2's cannot. In exchange, Mamba-2 gets much larger states and much faster training. On the Pile, the authors reported that Mamba-2 matched or exceeded both Mamba and a strong transformer recipe at equal compute across the model sizes studied, and on synthetic multi-query associative recall, in which a model must retrieve many key-value pairs from context, the larger state gave substantially better results than Mamba's. Capacity for recall, as Chapter 2 suggested, grows with the state. A family of linear recurrences The duality also explained something that had been happening across the field. In 2023 and 2024 many groups, starting from linear attention rather than from state space theory, had developed architectures that addressed linear attention's failure to forget. Placed side by side, they turn out to be the same recurrence with different choices for two things: how the state decays, and how new information is written. Write the state as a matrix S, of size key dimension by value dimension, as in linear attention. Every one of these models updates it in the form "decay the old state, then add a new association built from the current key and value", and reads it with the current query. RetNet, introduced by Yutao Sun and colleagues at Microsoft in 2023, uses a fixed decay per head: the old state is multiplied by a constant less than one at every step, so older associations fade at a set rate regardless of content. Gated linear attention, from Songlin Yang, Bailin Wang and colleagues at ICML in 2024, makes the decay data-dependent and allows a different decay rate for each key dimension, which makes it very close in spirit to Mamba's diagonal selective transition; its authors also developed hardware-efficient chunkwise training algorithms, released in a library called Flash Linear Attention, which has become a shared toolkit for this whole family. Mamba-2 uses a data-dependent scalar decay. The RWKV line of models, developed by Bo Peng and an open community, moved in its sixth version to data-dependent decays as well. The matrix-memory variant of xLSTM, from Maximilian Beck, Sepp Hochreiter and colleagues in 2024, uses exponential gating on a similar matrix state. Griffin and its pure-recurrent sibling Hawk, from Soham De and colleagues at Google DeepMind in 2024, use a gated diagonal linear recurrence with a vector rather than matrix state, combined in Griffin with local attention; Griffin's architecture became the basis of the RecurrentGemma models. Each of these makes the same move Mamba made: add input-dependent forgetting to a linear recurrence. They differ in the granularity of the forgetting, per head, per channel or per dimension, and in parameterisation details. All of them can be trained with chunked algorithms that resemble the one described above. The delta rule returns A second axis of variation concerns the write. All the models above write additively: the new association, an outer product of key and value, is added to the decayed state. Additive writing has a weakness, identified in Chapter 2. If a key similar to an existing one arrives with a different value, the memory ends up holding a blend of both. Decay helps by fading old entries, but it fades all of them, relevant or not. The delta rule, revived from Schlag, Irie and Schmidhuber's 2021 work, writes differently. Before storing a new association, it retrieves the value the memory currently associates with the new key, and it writes only the correction needed to replace that value with the new one, scaled by a learning rate βₜ that depends on the input. The effect is targeted overwriting: an outdated association for a key is replaced rather than piled upon. There is a clean interpretation of this. If one thinks of the state as the weights of a linear map that should send each key to its value, the delta rule is exactly one step of gradient descent on the squared error between the map's prediction and the true value, using the current key-value pair as a training example. The recurrent state is a small model being trained online, at inference time, on the sequence it is reading. For several years the delta rule was considered impractical for large models, because its update multiplies the state by a matrix that depends on the key, which seemed to rule out efficient parallel training. In 2024, Songlin Yang and colleagues showed how to parallelise DeltaNet across the sequence by representing products of those key-dependent matrices compactly, using a technique from numerical linear algebra for accumulating products of Householder-like matrices. Gated DeltaNet, by Yang, Jan Kautz and Ali Hatamizadeh, presented at ICLR in 2025, combined the delta rule's targeted writing with Mamba-2-style data-dependent decay, so that the model could both erase everything quickly, through the gate, and overwrite specific associations precisely, through the delta rule. On language modeling and on retrieval-heavy benchmarks it outperformed both Mamba-2 and DeltaNet at the scales studied. Table 3 sets out the family. Table 3. A family of linear recurrent sequence layers, written as updates to a matrix state S. Layer How the old state decays How new information is written Data-dependent forgetting? Linear attention (2020) No decay Add key-value outer product No RetNet (2023) Fixed scalar per head Add key-value outer product No Gated linear attention (2024) Learned vector, per key dimension Add key-value outer product Yes Mamba-2 (2024) Learned scalar per head, via Δ Add key-value outer product, scaled by Δ Yes DeltaNet (2024) No decay beyond the overwrite Replace value for key by delta rule Only through overwrite Gated DeltaNet (2025) Learned scalar per head Replace value for key by delta rule Yes Sources: the respective papers, as listed in Further Reading. The online-learning interpretation of the delta rule has been pushed further. The test-time training layers of Yu Sun and colleagues in 2024 treat the state as the weights of a small model, potentially a nonlinear multilayer network, updated by gradient steps on a self-supervised loss as the sequence is read. Titans, from Ali Behrouz and colleagues at Google, uses a neural long-term memory updated in a similar spirit, with a measure of surprise determining how strongly each input is written. These designs blur the boundary between recurrence and learning. In each, the question of what a fixed-size memory should hold is answered by an optimisation process running inside the model. What the duality means The deepest consequence of structured state space duality is that the distinction between state space models and attention is no longer a distinction of kind. Both compute a sequence transformation given by a lower-triangular matrix built from queries, keys and some form of mask. Softmax attention uses an exponential similarity, which makes the matrix full rank and forces it to be computed explicitly, at quadratic cost, but allows sharp, precise retrieval. State space models and their linear-attention relatives use a dot-product similarity with a structured decaying mask, which gives the matrix low rank below the diagonal and allows linear-time recurrence, but limits how sharply the matrix can single out one past position among many. Seen this way, the choice between architectures is a choice about the structure of the mixing matrix, and therefore about the kind of memory the model has. A low-rank structure means compression into a state of size N. A full-rank structure means keeping everything. The decaying mask means forgetting by recency, modulated by content. The delta rule means overwriting by key. Each choice makes some computations easy and others hard. That raises the question that the next chapter takes up directly. If the structure of a state space model's matrix has rank at most N below the diagonal, then there are things that matrix cannot represent, however it is trained. What are they, how much do they matter in practice, and can anything be done about them? Chapter 9. What a Fixed State Cannot Do Everything good about state space models follows from one design choice: the model's memory of the past is a fixed-size state. That choice delivers constant memory during generation, linear-time training, small serving footprints and long-sequence efficiency. It also imposes limits, and those limits are not artefacts of immature engineering. Some of them can be proved. Others show up consistently in experiments across independent research groups. A clear understanding of them is the most important practical knowledge a user of these models can have, because it explains where pure recurrent models can be trusted, where they cannot, and why the most capable long-context systems now combine recurrence with a small amount of attention. Two different limits need to be kept apart. The first concerns recall: the ability to retrieve an arbitrary piece of information from a long context exactly. Here state space models are weaker than attention, and the weakness follows directly from compression. The second concerns state tracking: the ability to follow a sequence of operations whose effects compose, such as tracking which cup the ball is under after a series of swaps. Here, surprisingly, both state space models and transformers are limited, and the interesting comparison is with older nonlinear recurrent networks. The counting argument The recall limit begins with a simple observation. A state of fixed size can hold a fixed amount of information. Suppose a model's entire recurrent state, across all layers, consists of S numbers, each stored with b bits of precision. Then the state can be in at most 2^(Sb) distinct configurations, and so it can distinguish at most that many different histories. Now consider a task that requires the model, after reading a random string of L tokens drawn from a vocabulary of size V, to reproduce the string exactly. There are V^L possible strings. If the state is to determine which one was seen, it must be able to take at least V^L distinct values, so Sb must be at least L log₂V. For any fixed state, copying must fail once L exceeds Sb divided by log₂V. A transformer has no such ceiling, because its key-value cache grows with the sequence. Samy Jelassi, David Brandfonbrener, Sham Kakade and Eran Malach made this formal in a 2024 paper titled "Repeat After Me: Transformers are Better than State Space Models at Copying", presented at ICML. They proved that a two-layer transformer can copy strings of length exponential in its size, while any model with a fixed-size latent state is bounded as the counting argument requires. They then showed the gap empirically. When trained to copy, transformers learned the task far faster than state space models and generalised to longer strings. When pretrained models were tested on retrieval from context, such as looking up a person's number in a long synthetic phone book provided in the prompt, transformers of a given size substantially outperformed Mamba models of the same size, and the gap widened as the phone book grew. It is important not to over-read the counting argument. The theoretical ceiling for a real model is very high. A model with six million state numbers at sixteen bits holds about a hundred million bits, which in principle could encode several million tokens. The practical limit arrives far sooner because a learned compression is nothing like an optimal code. The state is trained to be useful for predicting the next token on typical text, not to store arbitrary strings losslessly, and it holds information in a distributed, approximate form that degrades gradually as more is written. The counting argument establishes that a limit exists. Experiments show where it bites, and it bites at context lengths that matter. Recall in practice The most systematic study of the recall gap came from Simran Arora, Sabri Eyuboglu and colleagues at Stanford. In a 2023 paper titled "Zoology", they trained many efficient architectures, including Hyena, H3, RWKV and others, alongside attention, and asked what accounted for the remaining gap in language modeling quality. Their analysis found that most of the gap could be attributed to a single capability: associative recall, the ability to retrieve a token that was associated with another earlier in the context. They introduced a harder synthetic test, multi-query associative recall, in which the model sees many key-value pairs and must answer many queries about them, and showed that models without attention needed a state size growing with the number of pairs to solve it, whereas attention solved it at any size. A follow-up study by the same group, presented at ICML in 2024 with a model named Based, framed the issue as a trade-off between recall and memory. Across many architectures, they found that a model's recall ability was tied closely to the size of its recurrent state: larger states recalled better, and attention, with its unbounded state, recalled best, at the highest memory cost. No efficient architecture escaped this frontier. What architectures could do was sit at better points along it, achieving more recall for a given state size. Based itself combined a linear attention layer with a large feature map, giving a large state, and short sliding-window attention for precise local lookups. The same picture appeared at production scale. In June 2024, Roger Waleffe and colleagues at NVIDIA published "An Empirical Study of Mamba-based Language Models", comparing 8-billion-parameter Mamba, Mamba-2, transformer and hybrid models trained on identical data of up to 3.5 trillion tokens. Pure Mamba and Mamba-2 models matched or exceeded the transformer on many standard benchmarks. They lagged, however, on tasks that require in-context retrieval or precise use of prompt content, including the five-shot version of the MMLU knowledge benchmark, where the model must use worked examples in the prompt to determine the answer format, and synthetic phone-book lookup. A similar phenomenon had been reported by AI21 in the Jamba paper earlier the same year: a pure Mamba model often failed to follow the output format demonstrated in a few-shot prompt, producing a plausible but wrong label because it had not retained the exact form of the examples. On a sentiment classification task scored by exact label match, the pure Mamba model scored around 49 per cent against 84 per cent for pure attention, while the hybrid scored about 91 per cent. These results converge on a consistent account. A pure state space model is good at absorbing the gist of a long context: topic, style, broad facts, trends. It is less reliable at reproducing a specific detail exactly, especially one that seemed unimportant when it was read and so may not have been written strongly into the state. This is not a minor defect for applications. Answering questions about a long document, following a format shown in a prompt, retrieving a function signature from a large codebase, and citing an exact passage are all recall tasks. Strategies within the recurrent paradigm Several responses keep the model fully recurrent while improving its recall. The most direct is a larger state, which Mamba-2's matrix-multiplication algorithm made affordable. Better writing rules help as well: the delta rule's targeted overwriting, described in Chapter 8, uses a state of a given size more efficiently than additive writing, which is one reason Gated DeltaNet improves on Mamba-2 in retrieval tests at equal size. A third response addresses the order of information rather than the model. A recurrent model must decide what to store when it reads each token, before it knows what will be asked. If the question comes after the document, the model may have discarded exactly the detail the question requires. Arora and colleagues showed in 2024 that simply presenting the context twice, a strategy they called just-read-twice, substantially improves recall for recurrent models: on the second pass, the model already knows what the document contains and can store what is relevant to it. Placing the question before the document has a similar effect. These are practical techniques a user of a pure recurrent model can apply immediately, and they illustrate the underlying issue well. A compressing memory does best when it knows in advance what to keep. The illusion of state The second limit runs in a different direction and overturns a natural assumption. One might expect that a recurrent model, which maintains an explicit state and updates it step by step, would be naturally good at tasks that require tracking a changing state: following a sequence of moves in a game, evaluating a program line by line, or tracking which entity holds which object after a series of transfers. Classical nonlinear recurrent networks can do this; an LSTM can in principle simulate any finite-state machine. Transformers, by contrast, are known from theoretical work to struggle, because a fixed-depth transformer performs a bounded number of sequential steps of computation regardless of input length. William Merrill, Jackson Petty and Ashish Sabharwal showed, in a 2024 ICML paper titled "The Illusion of State in State-Space Models", that linear state space models such as S4 and Mamba share the transformer's limitation rather than the LSTM's capability. They used the tools of circuit complexity theory, which classify computations by the depth of logical circuits needed to perform them. Under realistic assumptions about numerical precision, transformers fall into a class of computations that can be done by constant-depth circuits, and the authors proved that linear state space models with diagonal or similarly structured transitions fall into the same class. A canonical problem believed to lie outside it is composing a sequence of permutations, such as tracking the arrangement of five objects through a series of swaps. The authors showed empirically that Mamba, like transformers, fails to learn such tasks at lengths beyond its depth, whereas a simple nonlinear recurrent network learns them easily. The reason is the linearity that made parallel training possible. A linear recurrence composes its transitions by multiplication, and when the transitions are diagonal with positive real entries, as in Mamba and Mamba-2, those multiplications commute and can only shrink or preserve magnitudes. They cannot represent operations whose order matters, such as permutations, and they cannot even represent a sign flip. Riccardo Grazzi and colleagues, in a paper presented at ICLR in 2025, pointed out that this means Mamba-style models with transitions confined to the interval between zero and one cannot solve parity, the problem of tracking whether an even or odd number of ones has appeared, because doing so requires multiplying the state by minus one. Allowing the transitions to take values between minus one and one, they showed, lets such models solve parity and improves state tracking more generally. Similar reasoning applies to delta-rule models, whose transitions can be made to reflect rather than merely shrink. The illusion of state is that a model which carries an explicit state looks as though it should be good at tracking state, when its linearity actually prevents it. The irony is sharp. The nonlinearity that made classical recurrent networks impossible to train in parallel was also what gave them their expressive power for state tracking. Modern state space models gave up that power to gain parallelism. They are, in this respect, no worse than transformers, but no better either. Mamba-3's response The most recent revision of the Mamba line takes both limits seriously. Mamba-3, by Aakash Lahoti, Kevin Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao and Albert Gu, was posted in March 2026 and presented at ICLR the same year. It makes three changes, each traceable to the analyses in this chapter. The first is a more expressive discretisation. The authors observed that the discretisation used in Mamba and Mamba-2 corresponds to a simple exponential-Euler scheme, and replaced it with what they call an exponential-trapezoidal scheme, which weights both the previous and the current input in each update. This makes the recurrence itself act as a short implicit convolution over its inputs, and the authors report that it can replace the short causal convolution that every earlier Mamba block had treated as indispensable. The second change returns to complex numbers. Chapter 6 noted that the original Mamba found complex-valued states unnecessary for language. Mamba-3 reintroduces them for a different reason: a complex transition can rotate the state rather than only decay it, and rotation is exactly what the positive real transitions of Mamba-2 cannot do. The authors show that this complex update is equivalent to applying a data-dependent rotary embedding, the same kind of position-dependent rotation transformers use to encode position, and that it can be computed efficiently. On synthetic state-tracking tasks of the kind described above, the rotational version solves problems on which Mamba-2 and a non-rotational Mamba-3 perform at chance. The third change targets efficiency at decode time. Generating a token with a state space model is memory-bound: the model reads its state, performs a little arithmetic and writes it back. The authors introduce a multi-input, multi-output formulation in which the state update becomes a small matrix multiplication rather than an outer product, so that more useful arithmetic is done per byte read. They report up to four times more decoding computation than Mamba-2 at fixed state size with similar wall-clock latency, and that a multi-input, multi-output model with state size 64 matches the quality of Mamba-2 with state size 128. At the 1.5-billion-parameter scale, the paper reports that Mamba-3 improves average downstream accuracy over Gated DeltaNet, Mamba-2 and a transformer baseline, with the multi-input, multi-output variant adding a further gain. These changes improve the frontier, but they do not abolish it. A complex-valued transition helps with state tracking; it does nothing about the counting argument. A more efficient use of the state raises the amount that can be recalled for a given memory budget; it does not make the budget unbounded. The recall limit is structural, a consequence of the decision to compress. Where this leaves the pure recurrent model The evidence supports a balanced conclusion. Pure state space models are excellent at what compression does well: modeling the statistical structure of long sequences, absorbing broad context, generating fluently and cheaply, and handling continuous signals such as audio and genomic data where exact token-level recall is rarely the point. They are weaker, by a margin that grows with context length, at retrieving specific information exactly. On state tracking they share the transformer's limits, which recent designs are beginning to address. There is an obvious engineering response to a model that compresses well but recalls poorly, alongside a model that recalls perfectly but at linear memory cost per token. Use both, in proportions that give most of the efficiency of the first and most of the recall of the second. The next chapter shows that this is exactly what the industry has done. Chapter 10. Hybrids and Million-Token Contexts in Practice By the middle of 2024 the evidence from the previous chapter had produced an unusual degree of agreement. Pure state space models were cheap and fluent but unreliable at exact recall. Pure transformers recalled perfectly but paid for it in memory and bandwidth that grew with every token. Several groups, working independently, tried the obvious combination and reached the same conclusion: a network made mostly of recurrent layers, with a small fraction of attention layers interspersed, matched or beat a pure transformer on quality while keeping most of the recurrent model's efficiency. Within eighteen months, that design had moved from research papers into production models released by AI21, NVIDIA, IBM, Alibaba's Qwen team, Moonshot AI and others. This chapter describes how those hybrids are built and why they work, sets out the arithmetic of serving contexts of a million tokens, examines how long-context ability is actually measured, and surveys the domains outside language, especially genomics and audio, where very long sequences are routine. Why a few attention layers are enough The reasoning behind hybrids follows from the division of labour that Chapter 9 implied. Most of what a language model does at each layer, combining nearby context, building up representations of syntax and meaning, tracking topic and style, does not require exact retrieval from arbitrarily far back. A recurrent layer handles it well and cheaply. Occasionally the model needs to look something up: the exact wording of an instruction, a name introduced pages earlier, the output format demonstrated in a prompt. For that, one attention layer with access to the full context suffices, because a single attention layer can retrieve any token in one step. The recurrent layers can then build on what it retrieves. The economics follow directly. A hybrid's key-value cache grows only in its attention layers. If one layer in eight is attention, the cache is one-eighth the size of an equivalent transformer's, and the bandwidth needed per generated token falls by a similar factor, plus the small constant cost of reading the recurrent states. Prefill arithmetic still contains a quadratic term, but only in the attention layers, so its coefficient falls by the same factor. The first hybrids The first large hybrid released openly was Jamba, from AI21 Labs, in March 2024. Its architecture interleaved Mamba layers with attention layers in a ratio of one attention layer to seven Mamba layers, and it used mixture-of-experts feed-forward layers, in which each token is routed to a small subset of many expert networks, to increase capacity without increasing per-token compute. The released model had 52 billion total parameters, of which about 12 billion were active for any given token. It supported contexts of 256,000 tokens. The paper reported that at that length its key-value cache was about four gigabytes in 16-bit precision, against about thirty-two gigabytes for the similarly sized Mixtral mixture-of-experts transformer, and that its throughput at long contexts was about three times Mixtral's. In ablations at smaller scale, the authors found that ratios of one to three and one to seven performed about equally well, and chose the latter for efficiency. They also reported, as noted in Chapter 9, that the hybrid succeeded at in-context learning tasks where the pure Mamba model failed, even though only one layer in eight was attention. And they found that the hybrid needed no explicit positional encoding in its attention layers, apparently because the Mamba layers supply positional information implicitly through their ordered recurrence. NVIDIA's controlled study of June 2024, described in Chapter 9, provided the cleanest comparison. Its 8-billion-parameter hybrid consisted of about 43 per cent Mamba-2 layers, 7 per cent attention layers and 50 per cent feed-forward layers. Trained on the same 3.5 trillion tokens as a pure transformer of the same size, it exceeded the transformer on all twelve standard tasks evaluated, by an average of 2.65 points, and the authors projected it to be up to eight times faster at generating tokens. It also matched or exceeded the transformer on long-context evaluations when both were extended to longer contexts. Other designs explored the space. Zyphra's Zamba models shared a single attention block, with one set of weights, across several points in a Mamba backbone, reducing the parameter cost of attention further. Microsoft's Samba combined Mamba with sliding-window attention rather than full attention, aiming for constant memory overall at the cost of exact long-range recall. Google DeepMind's Griffin, described in Chapter 8, paired gated linear recurrences with local attention. Some groups also released pure recurrent models at substantial scale: Mistral's Codestral Mamba, a 7-billion-parameter Mamba-2 model for code, appeared in July 2024, and the Technology Innovation Institute's Falcon Mamba 7B, a pure Mamba model, in August 2024. These demonstrated that pure state space models could be competitive at the seven-billion scale on many benchmarks, while the recall results of Chapter 9 explained why most subsequent production efforts chose hybrids. Hybrids in production The pattern that emerged in 2025 is summarised in Table 4, which lists several hybrid families whose layer compositions have been published. Table 4. Selected published hybrid architectures combining recurrent and attention layers. Model (developer, year) Recurrent layer type Recurrent to attention ratio Size (total / active parameters) Notes Jamba (AI21, 2024) Mamba 7 : 1 52B / 12B Mixture of experts; 256K context Nemotron-H-8B (NVIDIA, 2025) Mamba-2 24 : 4, plus 24 feed-forward layers 8B dense About 8 per cent of layers are attention Nemotron-H-56B (NVIDIA, 2025) Mamba-2 54 : 10, plus 54 feed-forward layers 56B dense Reported up to about 3 times faster than comparable transformers at long inputs Granite 4.0-H (IBM, 2025) Mamba-2 9 : 1 Up to 32B / 9B Reported over 70 per cent less memory for long inputs Qwen3-Next (Alibaba Qwen, 2025) Gated DeltaNet 3 : 1 80B / 3B Gated full attention in every fourth layer Kimi Linear (Moonshot AI, 2025) Kimi Delta Attention About 3 : 1 48B / 3B Delta-rule recurrence with fine-grained gating Sources: the developers' technical reports and announcements (AI21, March 2024; NVIDIA, April 2025; IBM, 2 October 2025; Qwen, September 2025; Moonshot AI, October 2025). NVIDIA's Nemotron-H family, published in April 2025, scaled the controlled study's recipe. The 8-billion-parameter model had 52 layers, of which only four were self-attention, with 24 Mamba-2 and 24 feed-forward layers; the 56-billion-parameter model had 118 layers including ten attention layers. NVIDIA reported that, on inputs of 65,536 tokens with 1,024 generated tokens, the 8-billion model was 1.8 times faster than Qwen-2.5-7B and three times faster than Llama-3.1-8B, while matching or exceeding their accuracy. Later Nemotron releases continued the pattern, combining Mamba-2 with mixture-of-experts layers. IBM's Granite 4.0, released on 2 October 2025, made the hybrid design the default for an enterprise model family. Its hybrid models combine Mamba-2 layers and transformer blocks in a nine-to-one ratio, and IBM reported a reduction of more than 70 per cent in the memory required for long inputs and many concurrent requests compared with conventional transformers. The largest, Granite-4.0-H-Small, is a mixture-of-experts model with 32 billion total and 9 billion active parameters. Tellingly, IBM also released a conventional transformer variant of its smallest model, for platforms whose software did not yet support hybrid layers. The Qwen team's Qwen3-Next, released in September 2025, and Moonshot AI's Kimi Linear, released in October 2025, represent the delta-rule branch of the family described in Chapter 8. Qwen3-Next interleaves Gated DeltaNet layers with gated full-attention layers in a three-to-one ratio, so that 36 of its 48 layers are recurrent, and combines this with a sparse mixture of experts that activates about 3 billion of its 80 billion parameters per token; the same layout was carried into the larger Qwen3.5. Kimi Linear uses a refined delta-rule layer called Kimi Delta Attention, with gating at the level of individual channels, in a similar ratio to full attention; its developers reported large reductions in key-value cache size and substantially faster decoding at million-token contexts. MiniMax's MiniMax-01 models, released in January 2025, used a linear attention variant called lightning attention with a full softmax attention layer after every seven linear layers, at a scale of 456 billion total parameters, and were described as handling contexts of up to four million tokens at inference. The convergence is striking. Groups starting from state space models and groups starting from linear attention arrived at the same structure: a majority of linear recurrent layers with data-dependent forgetting, a minority of full attention layers, and usually mixture-of-experts feed-forward layers. The ratio varies, from three-to-one in the delta-rule models to nine-to-one in Granite, and the optimal choice appears to depend on the strength of the recurrent layer, the target context length and the balance between recall and efficiency that a deployment needs. Where the attention layers go also matters. Several reports find that placing attention in the middle of the network works better than placing it at the very beginning, and that a few attention layers spread evenly through the depth outperform the same number clustered together, though the evidence is not yet systematic. A further practical development is conversion. Training a large hybrid from scratch is expensive, and many organisations already have strong transformers. Junxiong Wang, Daniele Paliotta, Avner May, Alexander Rush and Tri Dao showed in a 2024 paper titled "The Mamba in the Llama" that a pretrained transformer can be distilled into a hybrid by initialising the recurrent layers from the attention layers' projection weights, exploiting the duality of Chapter 8, and then fine-tuning to match the original model's outputs. The approach retained much of the original model's quality while delivering hybrid-level inference efficiency, and related methods have since been explored by several groups. The arithmetic of a million tokens It is worth returning to the worked example of Chapter 1 to see what a hybrid buys at the extreme. A transformer with the shape of Llama 3 8B needs about 131 gigabytes of key-value cache to hold a single million-token context. Now imagine a model of similar size in which only four of 32 layers are attention, keeping the same key-value head configuration, and the other 28 are Mamba-2 layers. The cache now grows only in those four layers: about 16 kilobytes per token, or about 16 gigabytes at a million tokens. The recurrent layers add a constant state, perhaps a few hundred megabytes depending on configuration, independent of length. The total fits comfortably on a single 80-gigabyte accelerator alongside the model's weights, with room for more than one sequence. A pure transformer of the same shape would need two such accelerators for the cache of one sequence alone. Bandwidth falls by a similar factor. Each generated token now reads about 16 gigabytes of cache instead of 131, and generation runs correspondingly faster on a bandwidth-bound accelerator. Prefill is where the hybrid's remaining quadratic cost lives. Reading a million-token prompt still requires each of the four attention layers to compute its full causal attention matrix. Using the per-layer estimate from Chapter 1, at a million tokens the attention term in each attention layer is many times larger than all the other arithmetic in that layer. The hybrid reduces the total attention arithmetic by a factor of eight relative to the full transformer, which is substantial, but at a million tokens the four attention layers can still dominate the prefill time. That is one reason some designs use sliding-window attention for some of their attention layers, and why systems that must routinely process multi-million-token inputs push the attention fraction lower still. It is also why a pure recurrent model, whose prefill cost is strictly linear, remains attractive for applications that care more about throughput on enormous inputs than about exact recall. Serving infrastructure has had to adapt as well. Transformer serving systems rely on techniques that assume a growing key-value cache: sharing the cache across requests with a common prefix, truncating it, or rolling it back when a speculative guess about upcoming tokens turns out wrong. A recurrent state cannot be truncated or rolled back to an arbitrary earlier position, because it is a compressed summary rather than a list. Supporting prefix sharing and speculative decoding therefore requires checkpointing states at chosen positions, which adds engineering complexity. Major open-source serving engines and inference libraries added support for Mamba-family layers during 2024 and 2025, but support has lagged behind that for transformers, and IBM's decision to ship a pure transformer variant alongside its hybrids reflects that reality. Measuring long context honestly A model that accepts a million tokens is not necessarily a model that uses them well, and measuring the difference has proved harder than expected. The most widely cited early test, the needle in a haystack introduced by Greg Kamradt in 2023, hides a single out-of-place sentence at some depth in a long document and asks the model to retrieve it. Many models now pass it nearly perfectly at their full advertised context length, which shows that the test is too easy: it requires only one retrieval of a distinctive item. More demanding benchmarks reveal much more. RULER, from Cheng-Ping Hsieh and colleagues at NVIDIA in 2024, extends the needle test to many needles, multiple values per key, distractors that resemble the target, tracing chains of variable assignments through the context, and aggregating information spread across the whole input. Its authors found that the effective context length of many models, defined as the length at which performance stays above a threshold, was far shorter than the advertised length. Pure recurrent models tend to degrade steadily on such tests as length grows, consistent with the recall limits of Chapter 9. Hybrids do considerably better, and NVIDIA and others have reported strong results for hybrid models on RULER at long lengths. Length generalisation is a separate issue. A model trained on sequences of a certain length may not work on longer ones, and for state space models the reasons are specific. The synthetic induction-head result in Chapter 6, where Mamba extrapolated to a million tokens, is genuine but concerns a single, simple retrieval. Studies in 2024, including the DeciMamba work of Assaf Ben-Kish and colleagues, found that Mamba models trained on modest lengths often fail to retrieve information from beyond their training length in more realistic settings, because the learned step sizes imply an effective memory horizon matched to the training distribution. Training at the target length, or adjusting the step sizes, mitigates this. The broader lesson is that long-context capability must be trained for, not assumed from the architecture's asymptotic properties. Beyond language Language is where attention is strongest and where the comparison is most demanding. In several other domains, very long sequences are the norm and exact token-level recall matters less, and there state space and related long-convolution models have been especially influential. Genomics is the clearest case. A DNA sequence is a string of four letters, and biologically meaningful interactions span tens or hundreds of thousands of base pairs, as when a regulatory element influences a gene far along the chromosome. Tokenising DNA into larger units shortens sequences but loses single-nucleotide resolution, which matters because single-letter mutations can be decisive. HyenaDNA, from Eric Nguyen and colleagues in 2023, used Hyena operators to model sequences of up to a million nucleotides at single-character resolution. Caduceus, from Yair Schiff and colleagues at ICML in 2024, built a bidirectional Mamba model that respects the fact that DNA has two complementary strands read in opposite directions. Evo, published in Science in 2024 by Nguyen, Brian Hie and colleagues at the Arc Institute and Stanford, used the StripedHyena hybrid architecture at 7 billion parameters with a context of about 131,000 tokens to model and generate genomic sequences across bacteria and their viruses. Its successor Evo 2, released in early 2025 at up to 40 billion parameters, extended the context to a million nucleotides. In these models, the long-range efficiency of convolutional and recurrent mixing is not a convenience but a precondition: a transformer at single-nucleotide resolution with a million-token context would be prohibitively expensive. Audio was the domain where S4 first excelled, and it remains one where state space models are natural. Raw audio at 16 kilohertz yields 16,000 samples per second, so a minute of sound is nearly a million steps. SaShiMi, from Karan Goel, Albert Gu, Chris Donahue and Christopher Ré in 2022, used S4 layers in a multi-scale architecture to generate raw audio waveforms with better quality and faster sampling than earlier autoregressive models. Several of the researchers behind S4 went on to found Cartesia, a company building real-time speech systems on state space architectures, where constant-time generation per audio frame and small memory footprints are precisely the requirements of low-latency voice applications on modest hardware. Vision is more contested. Vision Mamba and VMamba, both from 2024, adapted Mamba to images by flattening patches into sequences and scanning them in several directions, to compensate for the fact that a one-dimensional recurrence imposes an arbitrary order on a two-dimensional image. They reported competitive results on standard benchmarks with favourable scaling to high resolutions. A pointed rejoinder, titled "MambaOut" and published by Weihao Yu and Xinchao Wang in 2024, argued that for image classification at standard resolutions the selective state space model was unnecessary and that simpler gated convolutional networks performed as well, while acknowledging that state space models might help for long-sequence vision tasks such as detection and segmentation at high resolution. That debate captures a general principle: the advantage of a linear-time sequence model grows with sequence length, and where sequences are short the advantage disappears. The shape of current practice Taken together, the practical picture as of late 2026 is clear in outline. For language at the frontier, hybrid architectures, with a large majority of linear recurrent layers and a small minority of attention, have become a mainstream choice for models that must serve long contexts efficiently, and the recurrent layers themselves have converged on gated, data-dependent linear recurrences in the Mamba-2 and delta-rule families. For domains of very long, signal-like sequences, such as genomics and audio, recurrent and long-convolution models are often the default rather than the alternative. For short sequences and for tasks dominated by exact retrieval, the transformer remains hard to beat. What links these choices is the argument of this book: the right architecture depends on what kind of memory a task needs, and modern systems increasingly provide different kinds of memory in different layers of the same network. Conclusion: Choosing a Memory The story told in this book began with a cost and ends with a choice. The cost was attention's: quadratic arithmetic in training and prefill, a key-value cache that grows with every token, and generation speed limited by the bandwidth needed to read that cache. The choice is about memory. Having followed the argument from control theory through HiPPO, S4, selection, the parallel scan, duality and the limits of compression, it is now possible to say something sharper than "state space models are efficient" or "attention is expressive". Each is a particular answer to the question of what a sequence model should remember, and the answers have different consequences that can be stated precisely. What the preceding chapters establish Three conclusions follow from the material, and none of them could have been stated with confidence at the outset. The first is that the quadratic cost of attention is a consequence of a memory policy, not a property of sequence modeling. Attention keeps everything; its mixing matrix is full and must be computed explicitly. State space models keep a compressed summary; their mixing matrix is semiseparable, with rank bounded by the state size below the diagonal, and can therefore be applied by a linear-time recurrence. Structured state space duality shows that these are two members of one family, distinguished by the structure of a single lower-triangular matrix. Once that is understood, "attention versus recurrence" stops being a contest between paradigms and becomes a design parameter: how much of the past to keep exactly, how much to compress, and how to decide which is which. The second is that, in this field, the algorithm and the hardware cannot be separated. The convolutional shortcut of S4, the fused selective scan of Mamba, the chunked matrix-multiplication algorithm of Mamba-2 and the multi-input, multi-output decode of Mamba-3 are each a response to a specific fact about GPUs: that memory traffic costs more than arithmetic, and that matrix multiplication runs an order of magnitude faster than other arithmetic. A selective state space model with a naive implementation would have been a curiosity, slower than the transformer it was meant to replace. Its success was as much an achievement of systems engineering as of mathematics. The same point applies in reverse to attention, whose dominance was prolonged by FlashAttention. The practical ranking of architectures is set by the hardware they run on, and could change if the hardware does. The third is that compression has a price that cannot be engineered away entirely. A fixed-size state can hold only a bounded amount of information, and learned compressions hold far less than the theoretical bound. The consequences show up exactly where theory predicts: in exact copying, in retrieval from long contexts, in following formats demonstrated in a prompt. Separately, the linearity that allows parallel training limits the kinds of state tracking a model can perform, a limit shared with transformers and only now being addressed with complex-valued and sign-flipping transitions. These limits explain why the field's practical answer has been hybrid rather than revolutionary. Why the hybrid is not a compromise It would be easy to read the rise of hybrids as a retreat, as though state space models had promised to replace attention and settled for a supporting role. That reading misunderstands what the hybrids show. In every controlled comparison described in Chapter 10, the hybrid did not merely approach the transformer's quality at lower cost. It exceeded the transformer's quality, on standard benchmarks, at a fraction of its memory. The combination is better than either part. The reason is that the two mechanisms are complementary rather than redundant. Recurrent layers with data-dependent forgetting are good at the continuous, cumulative work of building context: integrating information over long spans, tracking topic, smoothing over noise. Attention is good at the discrete work of lookup: finding the one token that matters among thousands. A transformer uses attention for both kinds of work, spending its expensive mechanism on tasks that a cheaper one would do as well. A pure recurrent model uses compression for both, failing at lookup. A hybrid assigns each kind of work to the mechanism suited to it. That is not a compromise but a division of labour, and it is the reason a model in which only one layer in eight or ten attends to the full context can outperform one in which every layer does. What remains unsettled Several important questions are open, and honest use of these architectures requires keeping them in view. Nobody yet knows how pure recurrent models behave at the very largest training scales. Most frontier-scale experiments with these architectures have been hybrids, and the scaling behaviour of a pure state space model at a trillion parameters and tens of trillions of tokens has not been publicly established. It is possible that larger states and better writing rules will narrow the recall gap enough to make attention unnecessary for many tasks; it is also possible that the gap will widen as tasks demand more retrieval. The design space of hybrids is poorly mapped. Published ratios range from three recurrent layers per attention layer to nine or more. Placement, the use of sliding-window versus full attention, positional encoding in the attention layers, and the interaction with mixture-of-experts layers all vary between reports, and few studies vary them systematically at scale. It is likely that the right answer depends on the target context length and on how recall-heavy the intended workload is, which means there will not be a single best hybrid. Long-context evaluation remains immature. A model that passes a single needle-in-a-haystack test at a million tokens may fail at multi-hop reasoning over the same context, and benchmarks that measure the latter are still being developed. Claims about context length, from any architecture, should be read as claims about what the model accepts rather than what it reliably uses, until tested on tasks that resemble real workloads. Finally, the relationship between recurrence and learning is being rethought. The delta rule's interpretation as online gradient descent, test-time training layers and neural long-term memories all treat the recurrent state as the parameters of a small model trained on the sequence as it is read. This reframing may lead to memories that compress far more intelligently than current designs, choosing what to keep by something closer to learning than to gating. It is too early to say whether it will. What to do differently For practitioners, the argument has concrete implications. When choosing an architecture, start from the memory the task needs, not from the architecture's reputation. If the workload involves long inputs and the answer depends on the gist rather than on exact details, such as summarisation, classification, continuous signal modeling, or genomic and audio sequences, recurrent and hybrid models offer large efficiency gains with little loss. If the workload depends on exact retrieval from long contexts, such as question answering over contracts, citation, code search or following formats demonstrated in long prompts, make sure some attention is present, and test retrieval explicitly at the lengths that will be used in production. When evaluating a model, use tests that probe recall at the target length, with many needles, distractors and multi-step reasoning, rather than a single retrieval. Measure throughput and memory at realistic batch sizes and contexts, because the efficiency advantage of recurrent layers appears mainly in generation and at long contexts and can be invisible in short benchmarks. When using a pure or mostly recurrent model, arrange inputs so that the model knows what to keep. Put the question before the document, or present important context twice. A compressing memory does best when it knows in advance what will be asked. When deploying, check that the serving software supports the architecture's needs, including state checkpointing for prefix sharing and speculative decoding. The efficiency of a hybrid on paper is realised only if the infrastructure around it is built for recurrent state rather than for a growing cache. For researchers, the most useful contributions are likely to be careful ones: systematic studies of hybrid design at scale, evaluations that separate accepting a long context from using it, and theory that characterises which tasks require which kinds of memory. The field has moved quickly by building and testing models. It would benefit now from understanding why its best designs work. A different picture of sequence modeling The transformer has been so successful that it is easy to mistake it for the natural way to process sequences, with every alternative an approximation of it. The work described in this book supports a different picture. There is a large space of sequence models, parameterised by how they remember: what they keep, what they compress, how they decide, and how their memory maps onto the hardware that computes it. Softmax attention is one point in that space, at the extreme of keeping everything. The first structured state space models were another, at the extreme of fixed, content-blind compression. Mamba and its relatives occupy the territory between, with compression that responds to content, and the hybrids span both regions at once. The question a model designer faces is therefore not whether attention or recurrence is better. It is which kind of memory each part of a computation needs, and how to provide it at a cost the hardware can bear. That question has no single answer, but it is the right question, and asking it is what made linear-time sequence modeling possible. Glossary Associative recall. The ability to retrieve a value that was paired with a given key earlier in a sequence. The capability on which pure recurrent models most clearly trail attention. Associative scan. A parallel algorithm that computes all running results of an associative operation over a sequence in a number of sequential stages proportional to the logarithm of its length. Used to compute linear recurrences in parallel. Delta rule. A rule for writing to a matrix memory that first retrieves the value currently stored for a key and writes only the correction needed to replace it. Equivalent to one step of gradient descent on a squared-error regression objective. Discretisation. The conversion of a continuous-time state space model into a discrete recurrence over sequence steps, using a step size Δ. Common methods include the zero-order hold, the bilinear transform and, in Mamba-3, an exponential-trapezoidal scheme. HiPPO. High-order polynomial projection operators: a framework that derives the state matrix of a linear system so that its state holds the optimal polynomial approximation of the input history under a chosen weighting of the past. Hybrid architecture. A network that interleaves recurrent or state space layers with a smaller number of attention layers, combining compressed memory with exact retrieval. Key-value cache. The stored keys and values of all previous tokens that a transformer keeps during generation. Its size grows linearly with context length. Linear time-invariant system. A linear system whose parameters are the same at every step. Such a system is equivalent to a convolution, which made S4 and its successors fast to train but unable to filter inputs by content. Memory-bound. Describes a computation whose speed is limited by the rate at which data can be moved to and from memory rather than by arithmetic throughput. Token generation and element-wise operations are typically memory-bound. Prefill. The processing of a prompt before generation begins, during which all prompt tokens are available at once and can be processed in parallel. Selective state space model. A state space model whose step size and input and output vectors depend on the current input, allowing it to choose what to store and what to ignore. The core of Mamba. Semiseparable matrix. A matrix whose submatrices lying entirely below the diagonal have low rank. The mixing matrix of a state space model is semiseparable, with rank bounded by the state size. State size. The number of values a recurrent layer keeps per channel or head. It bounds how much information the layer can carry forward and therefore its recall capacity. Step size (Δ). The discretisation interval of a state space model, interpretable as how much time each token represents. Small values preserve the state; large values reset it towards the current input. Structured state space duality. The correspondence, introduced with Mamba-2, between a state space model with scalar transitions and a form of masked linear attention, allowing the same layer to be computed either recurrently or in an attention-like matrix form. Tensor cores. Specialised GPU units for matrix multiplication, roughly an order of magnitude faster than the general-purpose arithmetic units. Mamba-2's chunked algorithm was designed to use them. Further Reading The works below are the primary sources for the arguments in this book, grouped by theme. Most are freely available on arXiv or in the proceedings of the conferences named. Attention and its costs Vaswani, A., et al. "Attention Is All You Need." NeurIPS, 2017. The paper that introduced the transformer. Still the clearest statement of what self-attention computes and why it parallelises. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." NeurIPS, 2022. The best introduction to thinking about GPU algorithms in terms of memory traffic. Essential background for understanding Mamba's implementation. Tay, Y., et al. "Long Range Arena: A Benchmark for Efficient Transformers." ICLR, 2021. The benchmark that exposed the weakness of efficient-attention approximations and on which S4 first made its name. Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention." ICML, 2020. Shows that removing the softmax turns attention into a recurrence with a matrix state. The conceptual bridge between the two halves of this field. Schlag, I., Irie, K., and Schmidhuber, J. "Linear Transformers Are Secretly Fast Weight Programmers." ICML, 2021. Recasts linear attention as an associative memory, explains its capacity limit and revives the delta rule. Continuous-time memory and structured state spaces Gu, A., Dao, T., Ermon, S., Rudra, A., and Ré, C. "HiPPO: Recurrent Memory with Optimal Polynomial Projections." NeurIPS, 2020. The theory of online memory as polynomial approximation. The intellectual foundation of everything that follows. Voelker, A., Kajić, I., and Eliasmith, C. "Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks." NeurIPS, 2019. An important precursor deriving a sliding-window Legendre memory. Gu, A., Goel, K., and Ré, C. "Efficiently Modeling Long Sequences with Structured State Spaces." ICLR, 2022. The S4 paper. Mathematically demanding, but its introduction and experiments repay reading even without the derivations. Gu, A., Gupta, A., Goel, K., and Ré, C. "On the Parameterization and Initialization of Diagonal State Space Models." NeurIPS, 2022. The S4D paper, which shows how simple a working structured state space model can be. A good place to start for implementers. Smith, J. T. H., Warrington, A., and Linderman, S. W. "Simplified State Space Layers for Sequence Modeling." ICLR, 2023. The S5 paper, which replaced convolutions with parallel scans. Orvieto, A., et al. "Resurrecting Recurrent Neural Networks for Long Sequences." ICML, 2023. Shows that a carefully parameterised linear recurrent network matches S4 without a continuous-time derivation. Albert Gu's doctoral thesis, "Modeling Sequences with Structured State Spaces" (Stanford University, 2023). A unified, more leisurely account of HiPPO, S4 and their relatives by their principal author. Selection and Mamba Fu, D. Y., Dao, T., et al. "Hungry Hungry Hippos: Towards Language Modeling with State Space Models." ICLR, 2023. Diagnoses the recall gap with synthetic tasks and narrows it with the H3 layer. Poli, M., et al. "Hyena Hierarchy: Towards Larger Convolutional Language Models." ICML, 2023. The long-convolution alternative to explicit state space parameterisation. Gu, A., and Dao, T. "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." arXiv, December 2023; COLM, 2024. The central paper of this book. Its sections on selection and on the hardware-aware scan are both readable and important. Dao, T., and Gu, A. "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality." ICML, 2024. The Mamba-2 paper. Its treatment of semiseparable matrices is the clearest statement of the relationship between attention and recurrence. Lahoti, A., Li, K. Y., Chen, B., Wang, C., Bick, A., Kolter, J. Z., Dao, T., and Gu, A. "Mamba-3: Improved Sequence Modeling using State Space Principles." ICLR, 2026. Exponential-trapezoidal discretisation, complex-valued state updates and multi-input, multi-output decoding. Blelloch, G. E. "Prefix Sums and Their Applications." Carnegie Mellon University technical report, 1990. The classic account of the parallel scan. The linear-recurrence family Yang, S., Wang, B., Shen, Y., Panda, R., and Kim, Y. "Gated Linear Attention Transformers with Hardware-Efficient Training." ICML, 2024. Data-dependent gating for linear attention and the chunkwise training algorithms behind the Flash Linear Attention library. Yang, S., Kautz, J., and Hatamizadeh, A. "Gated Delta Networks: Improving Mamba2 with Delta Rule." ICLR, 2025. Combines gating and the delta rule; the recurrent layer adopted in Qwen3-Next. De, S., et al. "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models." arXiv, 2024. Google DeepMind's hybrid design, the basis of RecurrentGemma. Sun, Y., et al. "Retentive Network: A Successor to Transformer for Large Language Models." arXiv, 2023. The RetNet paper, an early linear recurrence with fixed decay. Limits of fixed-size state Arora, S., Eyuboglu, S., et al. "Zoology: Measuring and Improving Recall in Efficient Language Models." arXiv, 2023; ICLR, 2024. Introduces multi-query associative recall and attributes most of the quality gap to recall. Arora, S., et al. "Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff." ICML, 2024. The Based paper, which frames the recall-memory frontier. Jelassi, S., Brandfonbrener, D., Kakade, S. M., and Malach, E. "Repeat After Me: Transformers are Better than State Space Models at Copying." ICML, 2024. The formal and empirical case that fixed-state models cannot copy long strings. Merrill, W., Petty, J., and Sabharwal, A. "The Illusion of State in State-Space Models." ICML, 2024. Shows that linear state space models share the transformer's limits on state tracking. Grazzi, R., et al. "Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues." ICLR, 2025. Explains why positive-only transitions cannot solve parity and how to fix it. Hybrids and applications Lieber, O., et al. "Jamba: A Hybrid Transformer-Mamba Language Model." arXiv, 2024. The first large open hybrid, with useful ablations on ratio and in-context learning. Waleffe, R., et al. "An Empirical Study of Mamba-based Language Models." arXiv, 2024. NVIDIA's controlled comparison of Mamba, Mamba-2, transformers and hybrids at 8 billion parameters. NVIDIA. "Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models." arXiv, 2025. Production-scale hybrids with detailed layer compositions and inference measurements. Wang, J., Paliotta, D., May, A., Rush, A. M., and Dao, T. "The Mamba in the Llama: Distilling and Accelerating Hybrid Models." NeurIPS, 2024. How to convert a pretrained transformer into a hybrid. Hsieh, C.-P., et al. "RULER: What's the Real Context Size of Your Long-Context Language Models?" arXiv, 2024. A benchmark that separates accepting a long context from using it. Nguyen, E., et al. "Sequence Modeling and Design from Molecular to Genome Scale with Evo." Science, 2024. Long-sequence modeling of genomes with a hybrid convolutional architecture. Schiff, Y., et al. "Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling." ICML, 2024. A Mamba-based DNA model respecting the two-strand structure of the genome. Hashtags: #StateSpaceModels #Mamba #Mamba2 #Mamba3 #SequenceModeling #LinearTimeSequenceModeling #DeepLearning #ArtificialIntelligence #MachineLearning #SelectiveStateSpaceModels #StructuredStateSpaceModels #S4 #HiPPO #RecurrentNeuralNetworks #LinearAttention #SelfAttention #TransformerAlternatives #LongContextModeling #LongSequenceProcessing #EfficientLanguageModels #LargeLanguageModels #ParallelScanAlgorithms #HardwareAwareAlgorithms #GPUOptimization #ConstantMemoryInference #SelectiveScan #StateSpaceDuality #HybridNeuralArchitectures #EfficientInference #NeuralSequenceModels

Download the Book (PDF): Introduction Watch a software engineer for an afternoon and count the minutes spent waiting. They change three lines of code and start a build. The build takes four minutes, so they open a chat window. The chat window leads to an email, the email to a document, and by the time they return to the terminal the build has been finished for six minutes and has failed for a reason unrelated to their change: a test that fails one run in fifty, on a part of the system they have never touched. They run it again. It passes. They push the change, and the continuous integration server spends twenty-five minutes rebuilding and retesting a codebase that is, apart from those three lines, identical to the one it built an hour earlier. Then a reviewer asks for a small edit, and the cycle begins again. Nothing in that afternoon is dramatic. No one is incompetent and no system is broken in the ordinary sense. Yet most of the engineer's day has gone to waiting for machines to repeat work they have already done, and to deciding whether a red signal means anything. Multiply that afternoon across a few hundred engineers and a few years and the cost is enormous, larger in most organizations than the cost of the build infrastructure itself. It is also largely invisible, because it shows up nowhere as a line item. It shows up as features that ship a quarter late, as engineers who stop running the full test suite locally because it takes too long, and as a quiet culture of re-running failed jobs until they go green. Developer Productivity Engineering, usually shortened to DPE, is the name that has settled on the discipline of attacking this cost directly. The term was popularized by Gradle, Inc., the company behind the open-source Gradle Build Tool, whose commercial platform was sold for years as Gradle Enterprise. That product is now called Develocity, and in mid-2026 the company itself adopted the same name, while Gradle Build Tool remains free and open source. But DPE is not a product category. Google practised it at scale long before the term existed, building the internal system it calls Blaze and later releasing a version of it as the open-source Bazel. Large organizations such as Meta, Uber and Apple have published research on the same problems. What these efforts share is a premise: that the speed and trustworthiness of the feedback an engineer receives from builds and tests is an engineering problem, with measurable inputs and outputs, and deserves dedicated engineers who treat it that way. The argument of this book The central claim of this book is simple to state and takes the rest of the book to earn. Fast feedback comes far less from doing work faster than from not doing work at all, and an organization can only safely skip work when its builds and tests are deterministic enough to be trusted. Build caching, flaky test control and a fast local deployment loop look like three separate topics. They are one discipline. A build cache works by recognizing that a piece of work has been done before with exactly the same inputs and handing back the earlier result instead of repeating it. That is only correct if the work is a pure function of its declared inputs. A flaky test is, by definition, work whose result is not a function of its inputs. An inner development loop that takes minutes rather than seconds is almost always a loop that rebuilds and redeploys things that have not changed, because nothing in the toolchain can prove they have not changed. Seen this way, the hard part of DPE is not buying a faster machine or a bigger cache server. The hard part is the slow, unglamorous work of making a build honest: declaring every input, removing timestamps and absolute paths from outputs, isolating tests from each other and from the clock, and building the observability to notice when any of this regresses. Once a build is honest, the speed follows almost automatically, because enormous amounts of work turn out to be repetition that can be avoided. Until it is honest, every attempt at speed is fragile, and some attempts make things worse by serving wrong results quickly. This framing also explains why the two dominant toolchains in this space look so different. Bazel was designed from the start around hermeticity, the property that a build step can see only the inputs it has declared. It enforces honesty through sandboxing and pays for it with a steep migration cost. Gradle, and the Develocity platform that grew up around it, took the opposite path: start with the builds people already have, make caching opt-in and observable, and provide the tools to find and fix the places where a build is not honest. Neither approach is simply better. Each is a different answer to the question of how to get from where an organization is to a build that can be trusted enough to skip. Who this book is for The book is written for intelligent readers who are not necessarily build specialists: engineering managers deciding whether to fund a productivity team, senior engineers who suspect their build is slower than it should be but are not sure why, platform and infrastructure engineers inheriting a cache they did not design, and technical leaders trying to understand why developer experience has become a board-level topic in an era when artificial intelligence writes an increasing share of code. It assumes familiarity with the general idea of compiling code and running tests, but not with the internals of any particular build system. Where it describes specific features of Develocity, Gradle or Bazel, it does so to illustrate a principle; tools change faster than principles, and the reader who understands why a cache key must include the compiler version will adapt quickly to whatever the next release renames. The book is deliberately narrow. It does not survey every build tool, though it mentions Maven, npm and others where the comparison is useful. It does not treat developer productivity as a matter of individual output or try to rank engineers, which is a different and much more doubtful enterprise. It does not cover code review practice, team topology or product management, all of which affect how quickly software ships but none of which is a build problem. Its subject is the machinery between an engineer's keystroke and the moment they learn whether that keystroke worked. How the book proceeds Chapter 1 sets out why waiting is so costly, drawing on the research on developer experience that identifies feedback loops as one of its core dimensions, and distinguishes the inner loop on an engineer's workstation from the outer loop of continuous integration. Chapter 2 builds the theory of work avoidance from first principles: what a cache key is, why content addressing matters, and the ladder from incremental builds through local and remote caching to remote execution. Chapters 3 and 4 apply that theory to the two toolchains. Chapter 3 covers Gradle and Develocity, including the configuration cache that became Gradle's preferred mode of execution in version 9, the build cache node, and a structured set of experiments for finding cache problems. Chapter 4 covers Bazel, whose design makes hermeticity a rule rather than an aspiration, and the open Remote Execution API that has produced a small ecosystem of caching and execution services. Chapter 5 turns to the practical work of diagnosing cache misses and protecting a cache from being poisoned with wrong results, a problem that is as much about security as about speed. Chapter 6 takes on flaky tests, the largest single threat to trust in a fast build, using published data from Google and the academic literature to explain where flakiness comes from and how to manage it without simply re-running jobs until they pass. Chapter 7 addresses the outer loop: running fewer tests through predictive selection, running the rest in parallel across machines, and designing continuous integration so that it feeds the cache rather than competing with it. Chapter 8 is about the last stretch, from a compiled change to a running service on the engineer's own machine or development cluster, where minutes most often hide in container image builds and redeploys. Chapter 9 describes how to run a DPE practice: what to measure, how to make the economic case, and how the arrival of AI coding assistants changes the bottleneck. The Conclusion draws out what all of this implies for how organizations should think about the time their engineers spend waiting. A note on numbers This is a field full of impressive figures. Vendors publish case studies with large percentages, conference talks report dramatic speedups, and internal dashboards show hit rates that look excellent until someone asks what they measure. Wherever this book cites a number, it names the source, and it prefers published research and official documentation to marketing. Where reliable figures do not exist, it says so and reasons from mechanism instead. Readers should apply the same discipline to their own organizations: the most important number in DPE is the one measured on your own builds, by your own engineers, before and after a change. A practice that cannot produce that number is not yet engineering. The waiting engineer in the opening paragraph is not a caricature. Most people who have worked on a large codebase will recognize the afternoon. The good news, and the reason this field exists, is that a large fraction of that waiting is avoidable, and the methods for avoiding it are well understood. They are simply not easy, and they reward organizations that treat them as a discipline rather than a purchase. Chapter 1: The Cost of Waiting Every piece of software is built by a person making a change and then finding out whether the change worked. The time between those two events is the feedback loop, and its length shapes almost everything about how engineers work: how large their changes are, how often they test, how willing they are to refactor, and how much of their attention survives the day. This chapter makes the case that feedback latency is the master variable of developer productivity, explains why its cost is so much larger than the minutes on the clock, and sets out the vocabulary the rest of the book uses. Feedback loops and the cost of waiting For a long time, discussions of developer productivity oscillated between two unsatisfying poles. One counted outputs: lines of code, commits, tickets closed. Those numbers are easy to collect and easy to game, and they say little about whether valuable software is being produced. The other pole treated productivity as an ineffable matter of talent and culture, beyond measurement. The research of the last decade has carved out a more useful middle ground by asking what the experience of doing the work is like, and which features of that experience predict good outcomes. The most cited framing is the article "DevEx: What Actually Drives Productivity," published in ACM Queue in 2023 by Abi Noda, Margaret-Anne Storey, Nicole Forsgren and Michaela Greiler. It identifies three core dimensions of developer experience: feedback loops, cognitive load and flow state. The three are not independent. A slow feedback loop raises cognitive load, because the engineer must hold the context of a change in their head across the wait. It also breaks flow, because the wait invites a switch to some other task, and returning from that task has its own cost. The authors observe that developers spend considerable time waiting on compilation, test runs and review, and that slow loops cause frustration and task switching. Among the three dimensions, feedback loops are the one most directly under the control of tooling. Cognitive load depends on architecture and documentation; flow depends on meetings and interruptions; but the time from a keystroke to a test result is set by the build system, the test suite and the machines they run on. Earlier frameworks point the same way. The DORA research program, which grew out of the State of DevOps reports and the book Accelerate by Nicole Forsgren, Jez Humble and Gene Kim, identified lead time for changes as one of four key measures of software delivery performance. Lead time is the time from a commit to that commit running in production, and it is in large part a sum of build, test and deployment latencies. The SPACE framework, published by Forsgren and colleagues in 2021, warned against reducing productivity to any single metric, but it too lists efficiency and flow, the ability to complete work with minimal delays and interruptions, as one of its five dimensions. Different research groups, using different methods, keep arriving at the same conclusion: the speed of feedback is not a convenience. It is a structural property of how well an engineering organization works. The naive model of build cost multiplies build duration by the number of builds and by an engineer's hourly cost. A four-minute build, run twenty times a day by two hundred engineers, consumes about 267 engineer-hours per day. That arithmetic is correct and it already produces large numbers, but it understates the true cost in three ways. The first is task switching. Human attention does not resume instantly. When a wait is long enough to invite another activity, the engineer often takes it, and returning costs both the time to notice the build has finished and the time to rebuild the mental context of the change. Developers know this intuitively. Below some threshold, which varies by person and task but is commonly described as a few seconds to perhaps half a minute, an engineer will simply wait, eyes on the terminal, and the cost of the build is roughly its duration. Above that threshold, the build becomes an interruption, and its cost is its duration plus a switching penalty that can easily be larger than the build itself. This is why reducing a build from four minutes to two produces a modest gain while reducing it from forty seconds to five can transform how someone works. The relationship between build time and lost productivity is not linear; it has a cliff. The second hidden cost is behavioural adaptation. Engineers respond rationally to slow feedback by doing less of whatever is slow. If the full test suite takes twenty minutes, they run a subset locally and let continuous integration catch the rest. If a local deployment takes ten minutes, they batch several changes before trying them. Larger batches mean that when something breaks, there are more candidate causes, so debugging takes longer. Deferred testing means defects are found later, after the author has moved on, when they are more expensive to fix. The organization has not merely lost the minutes of waiting; it has shifted its whole way of working toward larger, riskier changes discovered later. Few of these costs ever appear in a build-time dashboard. The third hidden cost is the erosion of trust. A build that is slow but reliable is frustrating. A build that is slow and sometimes wrong is corrosive. When a test fails, the engineer must decide whether the failure is real. If experience has taught them that one failure in several is noise, they will re-run before investigating, and each re-run is another full wait. Worse, they will sometimes re-run a real failure into a pass, because the flaky behaviour masks the true defect, and the defect reaches production. Chapter 6 deals with flakiness in detail, but its place in the economics of waiting belongs here: unreliable feedback multiplies the cost of slow feedback, because it forces repetition and because it teaches engineers to discount the signal. Two loops and five kinds of slowness It helps to divide the feedback an engineer receives into two loops, because they have different economics and different solutions. The inner loop is everything that happens on the engineer's own workstation, or on a development environment they control, before they share a change with anyone. Edit, compile, run a few tests, start the service, try it in a browser or with a request, look at a log, edit again. The inner loop runs dozens or hundreds of times a day. Its latency is felt directly, second by second, and it is where the attention cliff matters most. A good inner loop gives feedback in seconds. The tools that matter here are incremental compilation, the local build cache, a remote cache that can supply results other people have already computed, fast test selection, and a deployment path that can put a changed service in front of the engineer without rebuilding everything around it. The outer loop begins when the change leaves the engineer's hands: it is pushed to a shared repository, built and tested by continuous integration, reviewed by colleagues, merged and eventually deployed. The outer loop runs far less often per engineer, perhaps a handful of times a day, but each iteration is much more expensive in machine time, because continuous integration typically runs the full build and test suite on fresh machines. Outer-loop latency is felt less as an interruption and more as a queue: engineers move on to other work while their changes wait, and the cost appears as long lead times, merge conflicts and the overhead of context switching back when a review comment or a red build arrives. The two loops are coupled. A change that passes in the inner loop but fails in the outer loop forces the engineer to return to the inner loop, often hours later. Differences between the environments are a common reason: the engineer's machine has a different compiler version, a different operating system, a different set of environment variables. Much of the value of hermetic builds and shared caches is that they make the two loops agree, so that a result produced on a laptop means the same as a result produced in continuous integration, and can be reused by either. When the loops agree, continuous integration can populate a cache that every engineer reads from, and the engineers' own builds become, in effect, a series of lookups against work the shared infrastructure has already done. When engineers complain that a build is slow, they usually mean that it takes a long time. That is a symptom, and there are only a few underlying causes. Distinguishing them is the first diagnostic step in any DPE effort. Some work is necessary and unavoidable: the change really does require recompiling a module and rerunning its tests, and that work takes real computation. The remedies are parallelism, faster machines and faster tools. These remedies have limits. A build's critical path, the longest chain of steps that must happen in sequence, bounds how much parallelism can help. Faster hardware gives a constant-factor improvement that is quickly absorbed by codebase growth. Much more work is repeated: the build is doing something it has done before, with the same inputs, and producing the same outputs. Compiling an unchanged module after switching branches. Rerunning the tests of a library no one has touched in a month. Rebuilding a container image layer whose contents are identical to yesterday's. This is where most of the opportunity lies, and the remedy is work avoidance: incremental builds, caching, and test selection. The theory of work avoidance is the subject of Chapter 2. Some work is wasted: it produces nothing anyone needs. Generating documentation on every local build. Running static analysis that no one reads. Downloading every intermediate output from a remote build when only the final binary is wanted. The remedy is simply to stop, but finding waste requires visibility into what the build is actually doing, which many organizations lack. Some time is not work at all but overhead: starting a virtual machine, resolving dependencies over a slow network, waiting in a queue for a continuous integration agent, configuring a build that has not changed. Gradle's configuration phase and Bazel's analysis phase both fall in this category, and both toolchains now cache their results, as later chapters describe. Finally, some time is spent on repetition caused by unreliability: re-running a build because a flaky test failed, or because a cached result was wrong and someone had to clean and rebuild. This category is the most damaging because it combines the cost of the work with the cost of lost trust. A useful habit is to take any slow build and ask, for each major phase, which of these categories it belongs to. In most large codebases the answer is surprising. Engineers tend to assume their builds are slow because the necessary work is heavy, when in fact the necessary work is a small fraction of what runs. Measuring the right thing A DPE effort that cannot measure its baseline cannot show that it has changed anything, and it will lose the argument for funding the first time a budget is cut. Measurement in this field has a few traps. The first trap is averages. Build times are rarely normally distributed. A typical distribution has a large mass of fast incremental builds and a long tail of slow ones: clean builds after a dependency upgrade, builds that miss the cache, builds that hit a flaky test and retry. The mean is dominated by the tail and describes almost no actual build. Medians and high percentiles describe the experience better, and the distribution should be broken down by kind of build: local versus continuous integration, incremental versus clean, by project and by task. The second trap is measuring only continuous integration. Continuous integration is easy to instrument, because its builds run on infrastructure the organization controls and its logs are centralized. Local builds happen on hundreds of laptops and are usually invisible. Yet the inner loop is where the attention cliff matters most. One of the distinctive contributions of Develocity, and of the Build Scan that Gradle introduced before it, was to capture rich data about every build, including local ones, and send it to a central server where it can be aggregated. Bazel offers a comparable stream through its Build Event Protocol, which services such as BuildBuddy and EngFlow consume to provide dashboards of local and continuous integration builds alike. Without such telemetry, a productivity team is guessing about the inner loop. The third trap is optimizing a proxy. A cache hit rate is a proxy for time saved, but a high hit rate on cheap tasks and a low hit rate on expensive ones can hide a poor outcome. A count of flaky tests is a proxy for wasted time, but one flaky test in a module everyone depends on costs more than a hundred in a module no one touches. The quantity that matters is engineer time, and the best measurements estimate it directly: time spent waiting on builds, time lost to failures that turned out to be flaky, time saved by cache hits computed from the original duration of the cached work. The fourth trap is forgetting that the measure itself changes behaviour. If a team is rewarded for reducing average build time, it may exclude slow builds from the measurement, or push work from the build into some unmeasured step. Good DPE metrics are reported as distributions, are explained alongside the mechanism that changed them, and are owned by the people who can fix them rather than imposed on the people who suffer them. Developer Productivity Engineering as a discipline What turns these observations into a discipline is a decision about ownership. In many organizations, the build belongs to everyone and therefore to no one. Each team adds tasks and plugins to suit its needs; no one is responsible for the aggregate; and the build slows by small increments until it is intolerable, at which point someone spends a heroic month fixing the worst of it before the slow decline resumes. DPE replaces this pattern with a dedicated team, or at least dedicated people, whose job is the feedback loop itself. They own the build telemetry, the caches, the continuous integration configuration and the policies for flaky tests. They treat regressions in build time the way a site reliability team treats regressions in service latency: as incidents with causes to be found. Such a team needs a small number of capabilities. It needs observability: data about every build and test run, local and remote. It needs acceleration technologies: caching, remote execution, test selection and distribution. It needs the authority to change shared build logic and to set standards for new code, because an honest build cannot be maintained if any team can add an undeclared input. And it needs an economic argument, refreshed regularly, that connects its work to the engineer time it saves, because without one it will be seen as an overhead rather than as the leverage it is. None of this is new to the largest technology companies, which have run such teams for many years. What has changed is that the tooling has become available to everyone. Bazel, its Remote Execution API and the open-source caches that implement it are freely available; Gradle's local and remote build caches are part of the open-source build tool; Develocity and several Bazel-oriented services offer the observability and acceleration layers commercially. The obstacle for most organizations is no longer access to technology. It is the discipline of making builds honest enough for the technology to work, which is where the next chapter begins. Chapter 2: The Theory of Work Avoidance The fastest build step is the one that does not run. That sentence sounds like a slogan, but it rests on a precise idea, and getting the idea precise is the difference between a cache that saves hours and a cache that silently ships wrong binaries. This chapter builds the theory from the ground up: what a build step is, when its result can be reused, how a cache recognizes that it has seen a step before, and the ladder of techniques, from incremental builds to remote execution, that apply the idea at increasing scale. Builds as functions Strip away the syntax of any build system and a build is a graph of steps. Gradle calls them tasks; Bazel calls them actions; Maven calls them goals, bound to phases of a lifecycle. Each step reads some files and settings and writes some files. Compiling a Java module reads source files, a classpath of dependencies, and compiler options, and writes class files. Running a test suite reads compiled classes, test resources and configuration, and writes a report. Packaging a container image reads a base image and some binaries and writes a set of layers. Steps are connected because the outputs of one are the inputs of another, and the build system schedules them so that each runs after the steps it depends on. The key observation is this: if a step is a pure function of its inputs, meaning that the same inputs always produce the same outputs and the step has no other effect on the world, then there is never a reason to run it twice with the same inputs. The second run can only produce what the first run produced. Whoever ran it first, on whatever machine, at whatever time, the answer is the same, and it can be copied rather than recomputed. Two conditions hide in that definition, and every problem with build caching is a violation of one of them. The first is completeness: the build system must know every input that affects the output. If a compiler reads an environment variable that the build system does not track, two runs with different values of that variable will produce different outputs while appearing, to the build system, to have identical inputs. A cache will then return the wrong answer. The second is determinism: the step must produce the same output from the same inputs. If a packaging step writes the current time into an archive, two runs with identical inputs will produce byte-different outputs. That does not usually cause wrong answers directly, but it defeats caching downstream, because every step that consumes the archive now sees a different input each time. Completeness violations cause incorrect builds. Determinism violations cause slow builds. Both are, at root, the same problem: the build's model of itself does not match what it actually does. The work of DPE is largely the work of closing that gap. The oldest form of work avoidance is the incremental build. The classic Unix tool make decides whether to rebuild a target by comparing file modification times: if any input is newer than the output, rebuild. This is cheap and works surprisingly often, but it is fragile. Timestamps change when files are touched without being edited, when version control checks out a branch, when a file is copied. They do not change when a tool's version or a command-line flag changes. A timestamp-based incremental build both rebuilds too much and, more dangerously, sometimes rebuilds too little. Modern build tools replace timestamps with content. Gradle records, for each task, a fingerprint of its inputs and outputs from the last execution. Before running the task again, it computes the current fingerprint and compares. If nothing has changed, the task is marked up to date and skipped. The fingerprint is based on file content hashes and the values of declared input properties, not on timestamps, so checking out a different branch and back again does not invalidate anything. Bazel does the same thing for actions, keeping a local action cache keyed on a digest of the action's command line, environment and input files. Incremental builds have one fundamental limitation: they remember only the last execution in one place. If you build branch A, switch to branch B and build, then switch back to A, the incremental state now describes branch B, and the A outputs must be rebuilt. If a colleague has already built exactly your revision, your incremental build cannot know. Incrementality is memory with a capacity of one. Caching generalizes it. Cache keys and content addressing A build cache is a store that maps a description of a step's inputs to the outputs that step produced. The description is called the cache key. To look up a step, the build computes the key from the step's current inputs and asks the cache whether it has an entry. If it does, a cache hit, the build downloads or copies the stored outputs and skips execution. If not, a cache miss, the build runs the step and, if permitted, stores the result under the key for next time. The cache key must capture every input that could affect the output, and nothing else. Including too little produces wrong results. Including too much, such as an absolute file path that differs between machines, produces needless misses. Getting this boundary right is the central design problem of every build cache. In practice, keys are computed by hashing. Each input file is reduced to a cryptographic digest of its content. The digests of all input files, the identity and version of the tool that runs the step, its configuration and arguments, the names of its outputs and anything else that matters are combined, in a defined order, into a single digest. Two steps with the same key are, as far as the build system can tell, the same computation. Outputs, too, are usually stored by content. A content-addressable store saves each file under the digest of its own bytes. This has several useful consequences. Identical files produced by different steps are stored once. A client can verify that a downloaded file is intact by rehashing it. And the cache can be split into two parts with different properties: a small index mapping step keys to lists of output digests, and a large blob store mapping digests to bytes. Bazel's remote caching protocol makes this split explicit. Its documentation describes an action cache, a map from action hashes to action result metadata, and a content-addressable store of output files keyed by their hashes. Gradle's build cache stores each task's outputs as a single archive under the task's key, a simpler model with the same underlying logic. Content addressing also makes caching compose across steps. If step A is a cache hit, its outputs are byte-identical to what A produced before. Step B, which consumes A's outputs, therefore sees inputs whose digests are unchanged, so B's key is unchanged, so B is also likely a hit. One honest step makes the next one cacheable. Conversely, one nondeterministic step, say a code generator that writes a timestamp in a comment, poisons the chain: its outputs differ every time, so every downstream key differs every time, and nothing after it is ever reused. This is why a single misbehaving task can destroy the effectiveness of caching for an entire build, and why finding such tasks is the most valuable diagnostic skill in the field. From local to shared to distributed, and what it costs Once a build can compute honest cache keys, the same mechanism can be applied at widening scope. Table 1 summarizes the rungs of the ladder. Table 1. The ladder of work avoidance techniques. Technique What is reused Scope of reuse Main prerequisite Incremental build Last outputs of each step One workspace, last run only Accurate input fingerprints Local build cache Any earlier outputs on this machine One machine, many branches Complete, deterministic cache keys Remote build cache Outputs produced by anyone Whole team and CI Relocatable keys; trusted writers Remote execution Execution capacity, plus cache Shared cluster of workers Hermetic steps with fully declared inputs A local build cache keeps outputs from many past executions on one machine. It solves the branch-switching problem: build A, build B, return to A, and A's outputs come back from the local cache. Gradle enables a directory-based local cache once build caching is turned on, and Bazel offers a --disk_cache flag that serves the same purpose. A remote build cache is shared over the network. Every engineer and every continuous integration agent can read from it, and some or all of them write to it. This is where the large wins appear. A team of fifty engineers working on the same codebase is, most of the time, building revisions that differ in only a few modules. If continuous integration builds every commit on the main branch and pushes the results to a remote cache, an engineer who pulls the latest main and builds will find nearly everything already computed. Their build becomes a sequence of cache lookups, plus real work only for the modules they have changed. A clean build that would take twenty minutes from scratch might take a minute or two, most of it spent downloading. Remote caching adds a requirement that local caching can sometimes ignore: relocatability. The cache key must not depend on where the workspace lives. If one engineer's checkout is at /home/alice/src/app and another's is at /Users/bob/work/app, and the cache key for a compile step includes the absolute paths of source files, the two will never share results. Build tools address this by normalizing paths relative to the project root, and by giving task authors ways to declare how much of a path matters. It adds a second requirement too: trust. A remote cache is a mechanism for running code produced on one machine on another. Who is allowed to write to it is a security question, treated in Chapter 5. Remote execution goes further. Rather than run a cache-missed step locally and upload the result, the build sends the step to a cluster of worker machines, which runs it and stores the outputs directly in the shared cache. The local machine becomes a coordinator, and the build can use hundreds of cores for highly parallel work such as compiling many independent files or running many test shards. Remote execution requires hermeticity in a strict form: the worker must be able to reconstruct everything the step needs from its declared inputs alone, because it has none of the local machine's environment. That is a natural fit for Bazel, whose actions are designed to declare their full inputs, and a much harder fit for build systems whose tasks can read anything on the file system. A cache hit is not free. It requires computing the key, which means hashing inputs; querying the cache, which means a network round trip for a remote cache; and transferring the outputs, which can mean downloading large files. Whether a hit saves time depends on the ratio between the cost of executing the step and the cost of retrieving its result. For an expensive step with small outputs, such as compiling a module or running a slow test suite whose output is a short report, caching is an obvious win. For a cheap step with large outputs, such as copying a set of files into an archive, retrieving from a remote cache can be slower than just doing the work. Good build tools let task authors mark such steps as not worth caching, and good productivity teams look at the data on how long each cached step took originally against how long its retrieval took. Network topology matters. A remote cache in a different region from the engineers who use it adds latency to every lookup, and a build with thousands of steps performs thousands of lookups. This is why caching infrastructure is often deployed as a set of nodes close to users: a node in each office or cloud region, replicating from a central store. Develocity's build cache node supports this kind of replication, and Bazel cache services generally offer regional deployments. It is also why Bazel introduced what it calls Build without the Bytes: when results are available remotely, the client need not download every intermediate output, only those it actually needs. Since Bazel 7, the default download mode has been to fetch only the outputs of top-level targets, leaving intermediate artifacts in the remote store unless a later step on the local machine requires them. The economics also explain why work avoidance beats raw speed. Suppose a build has a hundred steps of which ninety are unchanged since someone else built them. Doubling the speed of every step halves the build time. Caching the ninety unchanged steps cuts the work to ten steps plus ninety lookups, which, if lookups are cheap, reduces build time far more than doubling the hardware would. And the advantage grows with the size of the codebase, because in a larger codebase a typical change touches a smaller fraction of it. Hardware gains are bounded and linear; avoidance gains scale with the stability of the code. Correctness first All of this rests on the purity assumption. A cache that returns the wrong output is worse than no cache, because it destroys the one thing a build must provide: confidence that the binary reflects the source. Engineers who encounter a single cache-induced wrong result will start running clean builds "to be safe," and a cache that is bypassed whenever it matters is worthless. This is the point at which the two major toolchains diverge in philosophy, and it frames the next two chapters. Bazel enforces purity by construction. Actions run in sandboxes that expose only declared inputs, so an undeclared input is not silently ignored but causes the action to fail. The burden falls on the author of build rules, who must declare everything, and on any organization migrating to Bazel, which must restate its build in these terms. Gradle takes the opposite stance. Its task model allows tasks to read anything; caching is opt-in per task type; and correctness depends on task authors declaring inputs and outputs accurately. The burden falls instead on verification: on tools that detect when a task's outputs vary despite unchanged declared inputs, and on teams that act on what those tools find. Both approaches can produce fast, correct builds. Both can fail. The Bazel failure mode is that the migration never finishes, or that teams work around the sandbox with escape hatches until it no longer enforces anything. The Gradle failure mode is that undeclared inputs accumulate silently until someone gets a wrong cache hit and the organization loses faith in the cache. Understanding the theory in this chapter is what lets a team recognize either failure early. It is worth noticing how much of the burden falls on tools that the build system does not control. Compilers, code generators, archivers and test runners were mostly written without caching in mind, and many of them embed volatile information by default: timestamps in archive entries, absolute paths in debugging information, build host names in generated metadata. The reproducible builds movement, a cross-project effort that grew up around Linux distributions, has spent years persuading tool authors to offer deterministic modes and documenting the flags that enable them. Build caching benefits directly from that work. The more of a build's tools behave as pure functions out of the box, the less a build team has to patch, wrap or configure around them, and the more of the build becomes cacheable without anyone having to think about it. There is one more rung beyond those in the table, and it changes the question rather than answering it faster. Work avoidance asks whether a step's inputs have changed. Test selection, discussed in Chapter 7, asks a bolder question: even if a test's inputs have changed, is it likely to fail? That is a probabilistic judgment rather than a logical one, and it is only safe to make once the logical machinery beneath it is sound. An organization whose cache cannot be trusted has no business letting a model decide which tests to skip. Chapter 3: Gradle and Develocity in Practice Gradle is the build tool of most Android applications and a large share of server-side Java and Kotlin projects. It began in 2008 as an alternative to Ant and Maven that combined Maven's dependency management with the flexibility of a real programming language for build logic. That flexibility is both its strength and the source of most of its performance problems. A Gradle build can do almost anything, which means that, unless someone takes care, it will. This chapter describes how work avoidance is implemented in Gradle, how Develocity extends it with shared caching and observability, and a disciplined procedure for finding out why a particular build is not getting the benefits it should. How Gradle avoids work A Gradle build runs in three phases. In the initialization phase, Gradle determines which projects make up the build. In the configuration phase, it executes the build scripts of those projects, which create and configure tasks and wire them together into a graph. In the execution phase, it runs the tasks the user requested and everything they depend on. Each phase has its own form of work avoidance, and the slowest builds usually lose time in a phase their owners were not watching. Execution-phase avoidance has two layers. The first is the up-to-date check. For each task, Gradle records the fingerprints of its declared inputs and outputs. If they match the previous execution, the task is skipped and reported as up to date. The second is the build cache. If a task is not up to date but is cacheable, Gradle computes its cache key and looks it up, first in the local cache and then, if one is configured, in the remote cache. A hit restores the outputs and reports the task as FROM-CACHE. The build cache is disabled by default and is enabled either with the --build-cache command-line flag or by setting org.gradle.caching=true in the project's gradle.properties file. Most teams want the property, so that every build, local or remote, participates. Gradle's documentation lists what goes into a task's cache key: the task type and its classpath; the names of the output properties; the names and values of the declared input properties; inputs added through the task inputs API; the classpath of the Gradle distribution, the buildSrc directory and plugins; and the content of build scripts where it affects task execution. The practical consequence is that a change to a plugin version or to build logic invalidates every task the plugin touches, which is correct, because the plugin might change what those tasks do. Caching is opt-in at the level of task types. A task type must carry the @CacheableTask annotation to be eligible, and the annotation is not inherited by subclasses. Gradle's built-in tasks that are expensive and deterministic, among them Java compilation, test execution and Checkstyle analysis, are cacheable. Most custom tasks written by teams for their own builds are not, unless someone has deliberately made them so. This design choice is conservative and correct: Gradle cannot know whether an arbitrary custom task declares its inputs completely, so it does not assume it. Declaring inputs well is the core skill of writing cacheable Gradle tasks. File inputs are annotated with their path sensitivity, which tells Gradle how much of each file's path matters to the output. A compiler cares about the content of source files and their path relative to the source root, because the relative path determines the package. It does not care where the project lives on disk. Declaring such an input with relative path sensitivity makes the cache key relocatable: two checkouts in different directories produce the same key. Declaring it with absolute sensitivity, or failing to declare sensitivity at all where the default is absolute, silently makes the task uncacheable across machines. Classpath inputs have their own annotations, which normalize away irrelevant details such as the order of entries in a JAR file. For Java compilation specifically, Gradle goes further with compile avoidance: it fingerprints the application binary interface of upstream dependencies rather than their full content, so changing a private method body in a library does not force recompilation of everything that depends on it. For years, Gradle builds paid a fixed cost before any task ran: configuration. Every build, even one where every task was up to date, executed every build script, applied every plugin and created every task object. In a build with hundreds of projects, configuration alone could take tens of seconds, and it was the most common complaint from engineers whose actual change compiled in under a second. The configuration cache addresses this by caching the result of the configuration phase. Gradle's documentation describes it as storing the task graph, each task's configured state and dependency information. On a subsequent build with the same requested tasks and the same build logic and inputs, Gradle skips configuration entirely, loads the task graph from the cache and proceeds directly to execution. Because the loaded tasks are isolated from the mutable build model, it also allows more tasks to run in parallel safely. The configuration cache imposes discipline on build logic. Tasks cannot reach back into the project model at execution time; build scripts that read environment variables or system properties must declare those reads so that Gradle can invalidate the cache when they change; plugins must be written with these rules in mind. Many older plugins were not, which is why adoption was gradual. In Gradle 9.0, released on 31 July 2025, the configuration cache became the preferred mode of execution. Gradle now recommends enabling it when it detects no incompatible features, new projects generated by gradle init enable it by default, and the documentation states that it will be enabled by default in Gradle 10. It is enabled with --configuration-cache or the property org.gradle.configuration-cache=true. Gradle 9 brought another change of direct interest to caching: archive tasks now produce reproducible output by default. JAR files, ZIP files and similar archives are written with fixed timestamps, a consistent file order and preconfigured permissions, so that building the same inputs twice produces byte-identical archives. Before this change, every team that wanted reproducible archives had to configure each archive task to ignore file timestamps and use a stable order, and teams that did not paid for it with downstream cache misses. The change illustrates a broader trend in the field: determinism moving from something experts configure to something the tool provides. Two further pieces of Gradle infrastructure matter for the inner loop. The Gradle daemon is a long-lived background process that keeps the build tool's code loaded and its in-memory caches warm between builds, so that the second build in a session starts much faster than the first. Continuous build, invoked with the -t or --continuous flag, watches the file system and reruns the requested tasks whenever inputs change, which turns the build into something closer to an always-on compiler. Neither is a cache in the strict sense, but both remove overhead that otherwise surrounds every incremental build. Develocity: observability and shared acceleration Gradle's open-source build tool includes a local build cache and an HTTP client for a remote one. What it does not include is a fleet-scale view of how builds behave, or the infrastructure to serve a cache to a large organization. That is the space Develocity occupies. The platform was originally sold as Gradle Enterprise; it was renamed Develocity, and the Gradle plugin that connects builds to it now carries the identifier com.gradle.develocity. In mid-2026 the company behind both, previously known as Gradle Technologies, also took the Develocity name, while stating that Gradle Build Tool remains open source and free. Develocity today supports Gradle, Maven, Bazel, sbt, npm, pnpm and Python builds, though the depth of support varies by tool. The foundation of the platform is the Build Scan. A build connected to Develocity publishes a detailed record of its execution: every task or goal, its duration, its outcome, whether it was up to date or served from cache, its cache key and the inputs that went into it, the test results, the dependency resolution, the environment, and the console output. Build Scans are produced for local builds as well as continuous integration builds, which is what makes them valuable for the inner loop. When an engineer says "my build is slow," a productivity engineer can open that specific build and see exactly what ran. On top of individual scans, Develocity aggregates. Dashboards show build times by project and by kind of build, cache hit rates and the time those hits saved, failure rates grouped by cause, and test outcomes over time. The build cache node serves the remote cache, can be deployed in several regions, and replicates entries between nodes so that engineers in different locations each read from a nearby copy. The platform also includes test acceleration features, Test Distribution and Predictive Test Selection, discussed in Chapter 7, and flaky test detection, discussed in Chapter 6. One capability deserves emphasis because it turns caching from a black box into something debuggable: comparing Build Scans. Given two builds of what should be the same task, one a cache miss and one a hit, or two misses on different machines, Develocity can compare their task inputs and show precisely which input differed. The answer is usually mundane: a file whose content includes a timestamp, a system property that differs between laptops and continuous integration agents, a dependency resolved to a different snapshot version. But finding it by hand, in a build of thousands of tasks, can take days. Finding it from a comparison takes minutes. Maven builds deserve a word. Maven has historically had no build cache of its own, and every build recompiles and retests everything in the modules it builds. Develocity's Maven extension adds build caching for common plugin goals, and the Apache Maven project has published its own build cache extension. Both face the same challenge as Gradle custom tasks: Maven plugins were not written with declared, normalized inputs in mind, so caching each goal requires specific knowledge of what it reads. Finding out why a build misses the cache The most common experience on enabling a remote build cache is disappointment. The theory promises that most tasks should be hits, and the data shows that most are misses. This is almost never a problem with the cache infrastructure. It is the build telling the truth about itself: somewhere, inputs are varying that should not vary, or outputs are varying that should be stable. The productivity engineers at Gradle published a structured procedure for finding these problems, released as the open-source Develocity Build Validation Scripts. Rather than starting with a full, messy production setup, the procedure isolates one variable at a time through a sequence of controlled experiments. Each experiment runs the same build twice under carefully chosen conditions and examines which tasks failed to be avoided the second time. Because the only thing that varies between the two runs is the condition under test, any task that reruns points to a specific class of problem. Table 2 lists the five Gradle experiments, in the order the project recommends running them. Table 2. The five Gradle experiments in the Develocity Build Validation Scripts. Experiment Condition varied What a rerun task reveals 01 Incremental building Nothing; same checkout run twice Undeclared outputs or volatile inputs 02 Local caching, same location Clean outputs, same directory Tasks that are not cacheable or not deterministic 03 Local caching, different locations Checkout moved to a new directory Absolute paths in cache keys 04 Remote caching, CI to CI Different CI agents Agent-specific environment leaking into keys 05 Remote caching, CI to local CI agent versus developer machine Differences between CI and local environments The sequence is the point. The first experiment asks the most basic question: if I run the same build twice in the same place without changing anything, does every task come back up to date? A task that reruns has a problem with its own inputs or outputs, such as writing into a directory another task also writes into, or reading a value that changes on every run. There is no sense diagnosing remote caching until this passes. The second experiment cleans the build outputs between the runs, so that up-to-date checks cannot help, and asks whether the local cache restores everything. Tasks that rerun here are either not marked cacheable or produce nondeterministic outputs. The third moves the project to a different directory between runs, exposing tasks whose cache keys include absolute paths. The fourth and fifth move to the remote cache, first between two continuous integration agents, which are usually configured alike, and finally between continuous integration and a developer machine, which is where environmental differences such as operating system, Java version, locale and line endings tend to surface. The Maven version of the scripts follows a similar progression with four experiments. Each experiment produces a pair of Build Scans, and for each task that was not avoided, the comparison shows which inputs differed. A team working through the sequence fixes problems at each stage before moving on. In practice, the first three experiments surface most of the issues, and many of them are in a handful of custom tasks or third-party plugins. Fixing a single nondeterministic code generator near the root of the dependency graph can turn a build that was almost entirely misses into one that is almost entirely hits, because of the chaining effect described in Chapter 2. Operating the cache and governing the build A common and sound configuration for a Gradle remote cache is to let every build read from it but only continuous integration builds write to it. Developer machines are varied, sometimes misconfigured, and occasionally running uncommitted changes, and allowing them to write results that other people will consume is an invitation to subtle errors and, in the worst case, deliberate tampering. Continuous integration agents are controlled, built from known images and run only committed code. In Gradle's build logic, this is often expressed by enabling push on the remote cache only when an environment variable set by the continuous integration system is present. This arrangement has a consequence that surprises teams at first: the cache is only as good as continuous integration's coverage. If continuous integration builds only on merge to the main branch, an engineer working on a feature branch will get hits for everything their branch shares with main and misses for everything they have changed, which is exactly the desired behaviour. But if continuous integration runs on a different operating system from developer machines, or with a different Java vendor, then its results may never match a developer's keys, and the fifth experiment above will show it. The fix is either to align environments, often by using Gradle's toolchain support to pin a specific Java version independent of what is installed locally, or to run additional continuous integration builds on the platforms developers use. Continuous integration agents themselves pose the opposite problem. Ephemeral agents, which are created fresh for each job, start with an empty local cache, an empty Gradle user home and no downloaded dependencies. Every job pays the cost of fetching everything. Develocity's newer caching products, which it markets under names such as Setup Cache and Artifact Cache, target this overhead, and generic continuous integration caching of the Gradle user home is a common partial remedy. The underlying principle is the same as everywhere in this book: work that has been done before, with the same inputs, should not be done again, and the effort lies in making that sameness provable. The theme of this chapter is that Gradle trusts its users and gives them tools to verify that trust. That works when someone is actually responsible for verification. In organizations where build logic is copied between projects, where plugins are added without review and where no one looks at Build Scans, Gradle builds degrade in predictable ways: custom tasks without declared outputs that cannot be cached, configuration-time work that defeats the configuration cache, dependencies resolved dynamically to whatever is newest, and environment reads scattered through build scripts. The remedy is not heroics but governance. Shared build logic belongs in convention plugins, maintained by a small number of people who understand caching. New plugins are evaluated for configuration cache compatibility and cacheability before adoption. Build validation experiments run regularly, not once, because a build that was honest last quarter can lose that property with a single commit. And the Build Scan data is watched by someone whose job includes noticing when the cache hit rate drops. Gradle and Develocity supply the mechanisms. Only an organization can supply the attention. Chapter 4: Bazel and Hermeticity by Design Gradle starts from the builds people already have and gives them tools to become honest. Bazel starts from honesty and asks people to rebuild their builds around it. It descends from Blaze, the build system Google developed for its enormous single repository, and Google released it as open source in 2015. Its design reflects the problems of building software at a scale where tens of thousands of engineers commit to one codebase and where any build step that is not a pure function will, eventually, produce a wrong answer for somebody. This chapter explains how Bazel enforces the purity that caching requires, how its remote caching and execution work, and what an organization pays to get those properties. The model and its enforcement A Bazel workspace is divided into packages, each a directory containing a BUILD file. A BUILD file declares targets: libraries, binaries, tests, generated files. Each target is an instance of a rule, such as a Java library or a C++ binary, and each names its sources and its dependencies explicitly. Rules are written in Starlark, a deliberately restricted dialect of Python that cannot perform arbitrary input and output or produce nondeterministic results. A rule's job is to translate its target's attributes into actions: concrete commands, such as an invocation of a compiler with specific arguments, each with a declared set of input files and a declared set of output files. A Bazel build proceeds in three phases. In the loading phase, Bazel reads the BUILD files needed for the requested targets. In the analysis phase, it runs the rules to produce the full graph of actions. In the execution phase, it runs the actions that are needed, in dependency order and in parallel where the graph allows. Bazel keeps the results of loading and analysis in memory in a long-running server process, so that a subsequent build that changes only a few source files does not repeat analysis for the parts of the graph that are unaffected. Its internal evaluation framework tracks dependencies between all these computations at fine grain, so that a change invalidates only what depends on it. The restriction that makes Bazel distinctive is that the action graph is the complete truth. An action may read only the files declared as its inputs, and must write exactly the files declared as its outputs. Everything that affects an action's result must be visible in its definition: the command line, the environment variables Bazel passes to it and the content of its input files. Because of this, Bazel can compute a cache key for every action from information it already holds, without trusting the action's author to declare anything separately. The key is a digest of the action itself. Declaring inputs is not enough if an action can read undeclared files anyway. A compiler invoked on a developer's laptop can see the whole file system. If it happens to pick up a header file from a system directory, the build will work on that laptop and fail, or worse, behave differently, elsewhere. Bazel addresses this with sandboxing. On Linux and macOS, actions run by default in a sandbox that constructs a private view of the file system containing only the action's declared inputs, plus the tools the action needs. An action that tries to read an undeclared file finds that it does not exist and fails. This converts the most dangerous kind of build bug, a silent undeclared input, into a loud build failure at the point where it is introduced. A team adding a new dependency who forgets to declare it learns immediately, rather than months later when a cache returns a stale result. Sandboxing is not perfect hermeticity. Local sandboxes typically still expose system tools and libraries unless toolchains are configured to avoid them, and actions can still read the clock or generate random numbers. Network access can be blocked for actions, but whether it is depends on configuration. A build that auto-detects a C++ compiler installed on the host machine, for example, depends on that host's compiler in a way that sandboxing does not capture, and two machines with different compilers will compute the same action keys for different outputs. Serious Bazel users close these gaps by registering hermetic toolchains: compilers, language runtimes and other tools that are themselves downloaded as declared, version-pinned inputs to the build, so that every machine uses byte-identical tools. The closer a build gets to this ideal, the more safely its results can be shared, and the more readily it can move to remote execution. External dependencies are a frequent source of hermeticity problems in any build system. If a build resolves a library to "whatever version is newest" at build time, two builds of the same commit can differ. Bazel's original mechanism for external dependencies was the WORKSPACE file, which declared remote repositories to fetch. It was flexible but difficult to manage transitively: if two dependencies each needed a third at different versions, resolution depended on the order of declarations. Bazel's replacement is a module system known as Bzlmod, in which each project declares its dependencies in a MODULE.bazel file with explicit versions, and Bazel resolves the full dependency graph using a published registry, the Bazel Central Registry, or a private one. Resolution is deterministic and recorded in a lockfile. Bazel's migration documentation states plainly that the WORKSPACE file was disabled by default in Bazel 8, released in late 2024, and is removed in Bazel 9. Bazel 9 is now the active long-term-support release, with Bazel 7 and 8 in maintenance. Organizations still on WORKSPACE-based builds face a migration whether or not they want one, and the reward is a dependency graph that is stable enough to cache across machines. Remote caching and remote execution Because every Bazel action has a key computed from its full definition, caching is not an opt-in feature to be enabled per task. Every action is cacheable by default, and individual targets can opt out with a no-remote-cache tag when there is a reason. Bazel's remote cache, as its documentation describes, stores two kinds of data. The action cache maps action hashes to action result metadata: the digests of the output files, the exit code, and the standard output and error. The content-addressable store holds the output files themselves, each under the digest of its content. A build with remote caching enabled checks each action against the action cache, and on a hit fetches the result metadata and, if needed, the output blobs. On a miss, it executes the action locally and uploads the result. The remote cache can be reached over HTTP, using simple GET and PUT requests to paths under /ac/ for the action cache and /cas/ for the content store, or over gRPC. The flags tell the story. --remote_cache points Bazel at a cache server. --remote_upload_local_results=false makes the cache read-only for the client, which is how most organizations configure developer machines, so that only continuous integration writes. --disk_cache adds a local on-disk cache that is particularly useful when switching branches. The documentation lists several ways to run a cache: a generic web server with a WebDAV module, the dedicated open-source bazel-remote server, which adds garbage collection, or a cloud object store that can speak HTTP. Test results are also cached. A Bazel test target is an action like any other, and if its inputs, meaning the test binary and its declared data files, have not changed, its previous result is reused. This is one of Bazel's most consequential features. In a large repository, a change to one library reruns only the tests whose transitive inputs include that library. The rest report cached results from whoever last ran them. Tests that depend on external systems can be tagged external to prevent caching, and an engineer can force reruns with --cache_test_results=no when needed. Chapter 6 returns to the hazard this creates: a flaky test that happened to pass will have its pass cached, and a flaky failure may never be reproduced. Bazel's remote execution builds on the same protocol. Rather than executing a cache-missed action locally, Bazel sends it to a remote execution service, which assembles the declared inputs from the content-addressable store on a worker machine, runs the command, stores the outputs in the content-addressable store and records the result in the action cache. The local Bazel client becomes a scheduler. Because the action definition is complete, the worker needs nothing from the client's environment beyond what is in the store, and because many actions can be sent at once, a build can use far more parallelism than a single workstation provides. The protocol is defined by the open Remote Execution API, a set of gRPC service definitions maintained in the bazelbuild/remote-apis repository. Because it is an open specification, it has implementations from many sources, and it is not limited to Bazel: other build tools, including Meta's Buck2 and the Pants build system, speak the same protocol. Bazel's documentation lists several self-hosted implementations and commercial services, summarized in Table 3. Table 3. Remote caching and execution implementations listed in Bazel's documentation. Name Kind Notes from Bazel's listing Buildbarn Self-service, open source Modular cache and execution components Buildfarm Self-service, open source Hosted under the bazelbuild organization BuildGrid Self-service, open source Remote execution implementation NativeLink Open source and commercial Execution, caching, analytics, simulation BuildBuddy Commercial Remote execution, caching and results UI EngFlow Commercial Remote execution and caching with build and test UI The list also includes Aspect Build, which offers self-hosted remote cache and execution, Bitrise, which offers remote build caching aimed at mobile development, and Hermetiq, which offers managed remote cache and execution with cache-miss debugging. Develocity, too, now supports Bazel builds. The market is active and changes from year to year, so the specific names matter less than the fact that the protocol is shared: an organization can change cache or execution vendors without changing its build definitions. Remote execution raises a new problem. If a build runs ten thousand actions remotely, should the client download all their outputs? For intermediate files such as object files that will only be consumed by other remote actions, the answer is clearly no: downloading them wastes bandwidth and time, and for large builds the downloads can dominate. Bazel's answer is a family of modes known as Build without the Bytes, controlled by the --remote_download_outputs flag. In the minimal mode, Bazel downloads nothing it does not strictly need. In the top-level mode, it downloads only the outputs of the targets the user asked for. Starting with Bazel 7, top-level became the default, replacing the earlier behaviour of downloading everything. This seemingly technical change matters for the inner loop. An engineer who asks Bazel to build one service, in a repository where continuous integration has already built and cached almost everything that service depends on, now downloads only the final service artifact. A build that once required gigabytes of transfers can complete with a few lookups and one download. Making the strict model fast and livable Hermeticity has a performance cost that Bazel has had to work around. If every compilation action starts a fresh compiler process in a fresh sandbox, the start-up cost of the compiler is paid on every action. For tools that run on the Java Virtual Machine, such as the Java, Kotlin and Scala compilers, or for the TypeScript compiler, that start-up cost can exceed the cost of compiling a small file. Bazel's answer is persistent workers: long-lived processes that receive a stream of compilation requests and keep their internal state, including just-in-time compiled code and loaded libraries, warm between them. Each request still declares its inputs and outputs, and the results are still cached by action key, but the overhead of starting the tool disappears. Persistent workers are one of the main reasons Bazel's incremental builds of JVM languages are competitive with native build tools despite the stricter model. Bazel also keeps its own server process alive between commands, holding the loaded and analysed build graph in memory. A second build in the same workspace, after a small edit, skips most loading and analysis because the server already knows the graph and can see precisely which parts of it are affected by the changed files. Combined with the ibazel file watcher, which rebuilds affected targets whenever files are saved, this gives the inner loop a responsiveness that a cold invocation of any build tool cannot match. A practical obstacle to Bazel's model is the maintenance of BUILD files. Every new import in a source file is, in principle, a new dependency that must be declared. If engineers must update BUILD files by hand for every change, they will resent the tool and make mistakes. The remedy adopted by most successful Bazel users is to generate dependency declarations from source code automatically. Gazelle, a BUILD file generator maintained in the Bazel ecosystem, does this for Go and has been extended to other languages through plugins, and similar tools exist for other ecosystems. Generation turns Bazel's explicitness from a burden on every engineer into a property maintained by tooling, while preserving the guarantee that every dependency is declared. Organizations that skip this step usually conclude that Bazel is too much work; organizations that invest in it often find that engineers rarely look at BUILD files at all. Bazel exposes a structured stream of everything that happens during a build through its Build Event Protocol. With the --bes_backend flag, the stream can be sent to a server that records it, analogous to Gradle's Build Scans. Services built on this stream show the targets built, the actions executed or served from cache, timing, test results and logs, for local and continuous integration builds alike. Bazel can also write a detailed execution log of every action it ran, including its inputs and their digests, which is the raw material for diagnosing cache misses, discussed in Chapter 5. What Bazel costs Bazel's guarantees come at a price, and organizations that underestimate it are the source of most of the disappointed Bazel stories. The first cost is migration. A codebase built with Maven, Gradle, npm or a collection of shell scripts must be redescribed as Bazel targets with explicit dependencies. Tools can generate much of this, and community rule sets cover the major languages, but every special case, every code generator, every test that reaches outside its sandbox must be handled. For a large codebase this takes months to years of dedicated work, and during that time two build systems must be maintained. The second cost is ergonomics. Bazel's explicitness means more lines of build configuration, and many developers find it less familiar than the tools of their language ecosystem. Integrated development environment support has historically lagged native build tools, though it has improved. Teams that do not invest in tooling to keep BUILD files current, often by generating them automatically from source code imports, find that engineers resent the overhead. The third cost is that enforcement can be bypassed. Bazel has escape hatches: actions can be marked to run without a sandbox, tests can be tagged to run locally only, rules can use local tools. Each escape is reasonable in isolation. Accumulated, they quietly reintroduce the undeclared inputs that Bazel was adopted to eliminate. A Bazel build is only hermetic if someone polices the exceptions. The benefits are correspondingly large where the conditions fit: large, multi-language codebases; many engineers; a high ratio of unchanged to changed code; and an organization willing to fund a build team. In such places, Bazel's combination of enforced hermeticity, universal action caching, test caching and remote execution can make the cost of building scale with the size of a change rather than the size of the codebase. That is the property every DPE effort is ultimately pursuing, and Bazel is the most direct way to get it by design. The choice between Bazel and an improved Gradle or Maven build is therefore not a question of which tool is faster. It is a question of where an organization wants to pay for honesty: up front, through migration and enforcement, or continuously, through verification and repair. Both payments are real. The mistake is to believe that either tool lets an organization avoid paying. Chapter 5: Keeping the Cache Honest A build cache has two ways to fail. It can miss when it should hit, which makes builds slow. Or it can hit when it should miss, which makes builds wrong. The first failure is visible, irritating and common. The second is rare, invisible and catastrophic, because it destroys trust in the entire mechanism. This chapter is about both: how to find the causes of unnecessary misses, how to prevent incorrect hits, and how to protect a shared cache from being turned into an attack vector. It is the most practical chapter in the book, and the one that most separates organizations whose caches work from those whose caches merely exist. Reading the hit rate and finding the causes The first number anyone looks at is the cache hit rate, and it is easy to misread. A build that reports ninety percent of its tasks served from cache may still be slow, if the ten percent that missed include the most expensive compilation and test steps. A build that reports forty percent may be fast, if the misses are all cheap steps that are not worth caching. The meaningful quantity is not the fraction of steps that hit but the fraction of avoidable work time that was avoided: the sum of the original execution times of the steps served from cache, compared with the total execution time the build would have needed without the cache. Both major toolchains give the raw material for this calculation. Develocity's Build Scans record, for each task served from cache, how long that task took when it originally ran, so the time saved can be reported directly, and its dashboards aggregate this across builds. Bazel summarizes each build with a line reporting how many processes ran and how they were satisfied, for example how many were remote cache hits, how many ran in the local sandbox and how many ran remotely. That line is a good first signal, but it counts actions, not time. Teams using Bazel generally rely on a Build Event Protocol service or on their own analysis of execution logs to estimate time saved. The hit rate also needs context about what kind of build is being measured. On continuous integration, building a commit that differs from its parent by a few files, a very high hit rate is expected. On a developer machine after pulling the latest main branch, a high hit rate is also expected, provided continuous integration has already built that commit. On a developer machine building a branch with substantial changes to a widely used library, a low hit rate is correct, because the work genuinely must be done. A dashboard that blends these situations hides the cases where something is broken. The useful question is not "what is our hit rate?" but "for builds where we expected hits, did we get them?" Cache misses that should not happen nearly always come from a small number of causes. Knowing the list shortens most investigations to minutes. Timestamps and build metadata. A task writes the current date, a build number, the host name or the user's name into an output file. Typical culprits include generated version files, manifests in archives, code generators that stamp a header comment and documentation generators that record when they ran. Every run produces different bytes, so every downstream step misses. The remedy is to remove volatile data from outputs that other steps consume, or to isolate it in a small file that is injected as late as possible, ideally only into final release artifacts. Bazel's convention of separating "stamped" builds, which embed version information, from unstamped ones, which do not, is a codified version of this remedy. Absolute paths. A task's cache key includes a file's absolute path, or a tool writes absolute paths into its output. Compilers that embed source file paths for debugging, test reports that list working directories and scripts that pass absolute paths on a command line are the usual sources. Cache entries produced in one checkout location are then useless in any other. In Gradle the fix is usually a correct path sensitivity annotation on the input; in compiled languages it is often a compiler flag that remaps path prefixes in debug information. Environment leakage. A tool reads an environment variable, a system property, the locale or the time zone, and the build system does not know. Differences between developer machines and continuous integration agents surface here, which is why the final experiment of the Develocity validation sequence compares a continuous integration build with a local one. Bazel's documentation on debugging remote cache misses lists environment variables and host properties leaking into builds, and inconsistent build configurations across machines, among the common causes. In Bazel, actions see only the environment variables the rule passes to them, but the set passed can be widened with flags, and a single such flag that differs between machines changes every action key. Tool and platform differences. Two machines compile with different versions of the Java Development Kit, or different C compilers, or on different operating systems. Sometimes this correctly changes the key and causes a miss, which is safe but wasteful. Sometimes it does not change the key, because the tool version is not recorded as an input, and that is a correctness problem. The durable fix is to pin toolchains: Gradle's Java toolchain support can provision a specific Java version regardless of what is installed locally, and Bazel's hermetic toolchains download exact tool versions as build inputs. Unstable dependency resolution. A build declares a dependency on a range of versions or on a snapshot that is republished frequently, so two builds of the same commit resolve to different artifacts. The fix is lockfiles and exact versions. Bazel's module system records resolved versions in a lockfile; Gradle supports dependency locking and verification. Overlapping outputs. Two tasks write to the same directory. Neither can be safely cached because neither knows which files in the directory are its own. Gradle detects this and disables caching for the affected tasks, reporting the reason in the Build Scan. The fix is to give each task its own output location. Nondeterministic ordering. A tool iterates over a hash map or a directory listing and writes entries in whatever order it encounters them. The output is semantically identical but byte-different. Archive tools were a classic example until Gradle 9 made archives reproducible by default; serializers and code generators remain common offenders. Debugging misses and preventing wrong hits Lists of causes are useful, but each investigation ends by identifying exactly which input differed for a specific step between two specific builds. The method is the same in every toolchain: capture a detailed record of two builds that should have matched, line up the corresponding steps, and compare their inputs. In the Gradle and Develocity world, the record is the Build Scan, and the comparison is built into the product. Selecting two Build Scans and comparing them shows each task whose cache key differed, and for each, which input properties or files differed and how. In a disciplined team, an engineer who notices an unexpected miss captures both scans and compares them before speculating. In Bazel, the record is the execution log. Bazel can write one in several formats, selected with --execution_log_compact_file, --execution_log_binary_file or --execution_log_json_file; its documentation recommends the compact format for size. Each entry describes an executed action with its inputs and their digests, its environment, its command line and its outputs. To debug a miss, an engineer captures logs from two builds, converts them to text with the parser tool in the Bazel source tree, and compares the text with an ordinary diff. The first action whose entry differs is usually the culprit; everything after it differs as a consequence. Bazel-oriented services increasingly perform this comparison automatically, which is a material time saver: a manual diff of execution logs from a large build is tedious work. A practical rule of thumb applies to both: always look for the earliest divergence in the dependency graph. Because of the chaining effect described in Chapter 2, one nondeterministic step near the root causes thousands of downstream misses, all of which will appear in any comparison. Fixing the downstream symptoms is pointless; fixing the root fixes them all. A composite example, assembled from patterns that recur across many investigations, shows how this plays out. A team enables a remote cache for a Gradle build of some forty modules and finds that developer builds after pulling the main branch still recompile almost everything. The dashboards show a low hit rate on compilation tasks across the board, which suggests a problem everywhere. The engineer investigating does not start with the dashboards. She runs the same build twice in the same directory, cleaning outputs between runs, and finds that even locally most compile tasks miss. Comparing the two Build Scans shows that every miss traces back to one module near the root of the graph, a small module containing generated code. Its generator, a custom task written years earlier, writes a header comment into each generated source file that includes the time of generation. Every run produces different source files, so the module's compile task has different inputs, so its compiled classes differ in their embedded source information, so every module that depends on it sees a changed classpath. The fix is a two-line change to the generator to drop the timestamp. After it, the local experiment shows nearly every task served from cache, and the remote hit rate for developer builds rises accordingly. The remaining misses, examined the same way, turn out to be a test task that reads the machine's time zone and a documentation task with absolute paths, each fixed in turn. Three lessons from this pattern generalize. The dashboards correctly showed that something was wrong but could not show what; only a controlled comparison could. The root cause was not where the symptoms were most visible, but at the earliest point of divergence. And the fix was trivial once found. Most cache problems are like this: hours of searching, minutes of repair. That ratio is why investment in comparison tooling, and in teaching engineers to use it, pays back so quickly. Unnecessary misses are a performance problem. Wrong hits are a correctness problem, and they deserve stricter treatment. A wrong hit occurs when two executions of a step have the same cache key but should have produced different outputs. By construction, this means some input that affected the output was not part of the key. In Bazel, sandboxing makes this rare for file inputs, though tools taken from the host and actions run outside the sandbox remain risks. In Gradle, it most often arises from a custom cacheable task that reads a file or a value it did not declare. Three practices keep wrong hits rare. The first is conservative opt-in: do not mark a custom task as cacheable until its inputs have been reviewed by someone who understands what it reads. The second is enforcement where it is available: Bazel's sandbox, and in Gradle the configuration cache's requirement that build logic declare its reads of environment variables and system properties, both convert some silent errors into visible ones. The third is periodic verification. A small fraction of continuous integration builds can run with the cache disabled, or with cache reads disabled but writes compared, and their outputs checked against what the cache would have returned. Any discrepancy is a defect. This kind of shadow verification is cheap relative to the cost of a wrong binary reaching users, and it is the only way to catch wrong hits that no one has yet noticed. When a wrong hit is found, the response matters as much as the fix. Engineers will want to disable the cache entirely. The better response is to identify the undeclared input, fix the step, and invalidate affected entries, which in practice usually means changing something that forces new keys, such as a version marker in the task's inputs. A cache that has been fixed and is visibly monitored keeps its users' trust. A cache that has been disabled in a panic rarely gets turned back on. The cache as an attack surface A shared build cache is a mechanism by which code produced on one machine is executed, or shipped, from another. That makes it part of the software supply chain, and it can be attacked. The scenario is straightforward. An attacker who can write an entry to the cache under a key that a legitimate build will later look up can substitute their own output, for instance a compiled library with a backdoor, for the genuine one. The legitimate build will report a cache hit, skip compilation and package the attacker's code. Because the source is untouched, code review will see nothing. This is called cache poisoning, and it is recognized explicitly by the SLSA framework for supply-chain security. Its requirements for the highest build level state that it must not be possible for one build to inject false entries into a build cache used by another build, and that the output of the build must be identical whether or not the cache is used. The defences follow from the threat. First, restrict writes. The configuration already recommended for performance reasons in earlier chapters, in which developer machines read from the remote cache but only continuous integration writes to it, is also the most important security control. A compromised laptop cannot poison what everyone else downloads. Second, trust only builds of trusted code. Continuous integration jobs that build untrusted input, such as pull requests from forks of an open-source project, should not write to the cache used by release builds; they can use a separate, isolated cache or none at all. Third, separate release builds. Many organizations build release artifacts with the cache disabled, or with a cache dedicated to the release pipeline, accepting slower releases in exchange for a guarantee that every shipped byte was produced from source in a controlled environment. Fourth, authenticate and audit access to the cache, so that writes can be traced to the job that made them. Content addressing helps but does not solve the problem. Blobs in a content-addressable store are verified against their digest, so an attacker cannot alter a stored file without changing its address. But the action cache, which maps action keys to output digests, is not content-addressed; it is a lookup table, and whoever can write to it can point a legitimate key at a malicious blob. The weak point is always the map from "what I asked for" to "what I got," and control over who can write that map is what protects it. Making honesty routine The lesson of this chapter is that cache health is not a project but a practice. A build that has been made honest will not stay honest on its own. New tasks are added, plugins are upgraded, a team introduces a code generator that stamps the date, a continuous integration image changes its default locale. Each of these can silently erode hit rates or, worse, introduce a wrong hit. Organizations that keep their caches healthy share a few habits. They track avoided work time, not raw hit rates, and they segment it by build type so that a regression is visible. They run validation experiments on a schedule and in continuous integration, so that a commit that breaks cacheability is noticed within hours rather than months. They review new cacheable tasks and new Bazel rule exceptions as carefully as security-sensitive code. They restrict cache writes and treat the release pipeline separately. And they make the debugging tools, Build Scan comparisons or execution log diffs, familiar to ordinary engineers rather than the private skill of a build specialist, because the engineer who introduced a problem is the one best placed to understand it. These habits make the cache trustworthy. The next chapter turns to the other great threat to trust in fast feedback, and the one that most directly attacks the purity assumption: tests whose results are not determined by their inputs. Chapter 6: Flaky Tests and the Tax on Trust A test is supposed to be a function from code to verdict. Given the same code, it should pass every time or fail every time. A flaky test is one that, given the same code, sometimes passes and sometimes fails. The definition sounds like a minor technical nuisance. In practice, flakiness is among the most expensive problems in software engineering at scale, because it attacks the thing all fast feedback depends on: the engineer's belief that a signal means something. It also attacks the purity assumption that makes caching possible. This chapter explains how common flakiness is, where it comes from, how to detect it, and how to manage it without sliding into the habit of re-running until green. How common, and how costly The best-known public data on flakiness at scale comes from Google. In a post on the Google Testing Blog in May 2016, John Micco reported that about 1.5 percent of all test runs at Google reported a flaky result, that almost 16 percent of Google's tests had some level of flakiness associated with them, and that about 84 percent of the transitions Google observed from a test passing to failing involved a flaky test. The last figure is the striking one. When a test that was passing starts failing, the most natural interpretation is that someone broke something. At Google's scale, that interpretation was wrong most of the time. Those numbers describe one company's codebase at one point in time, and no one should assume their own organization matches them. But the mechanism they reveal is general. A test suite of thousands of tests, each of which fails spuriously only rarely, will produce at least one spurious failure in a large fraction of runs. The arithmetic is unforgiving: if each test is independently reliable to one in a thousand, a suite of a thousand tests will contain a spurious failure in more than half of its runs. Individual tests can be quite reliable while the suite as a whole is not. The cost appears in several forms. The direct cost is reruns: engineers and continuous integration systems rerun failed jobs, and each rerun is another wait of full duration. The investigative cost is the time engineers spend looking at failures that turn out not to be theirs, often discovering this only after reading logs and code they did not write. The deferred cost is the real defects that are missed, because a team accustomed to spurious failures will rerun a real failure until it happens to pass, if the defect is itself intermittent, and ship it. And the cultural cost is the gradual collapse of the norm that a red build must be fixed. Once engineers believe that red often means nothing, a red build no longer stops anyone, and the protective value of the test suite disappears. Flakiness interacts with caching in ways that make both worse. Consider a test that fails five percent of the time. On a given commit, continuous integration runs it; ninety-five times in a hundred it passes, and the pass is written to the remote cache. Every subsequent build of the same commit, on any machine, will report the cached pass without running the test. That is the correct behaviour of the cache, and it has a useful side effect: once a pass is cached, the flaky test stops bothering anyone for that commit. But five times in a hundred the first run fails. Failures are generally not cached, because a build tool should not remember a failure as the definitive answer, so the next build reruns the test, and may pass. The engineer sees a failure, then a pass, with no change in between, and learns once again that the test suite is unreliable. Worse, caching can hide flakiness. A test that is flaky but usually passes will, in a well-cached build, run only once per change to its inputs. The rare failure becomes rarer to observe, and the flakiness is harder to diagnose because there are fewer runs to learn from. Teams that adopt aggressive test caching sometimes find that their flaky test counts appear to fall, when what has fallen is only the number of times flaky tests are executed. And flakiness that comes from undeclared inputs is, by the definitions of Chapter 2, a completeness violation. A test that fails when run after another particular test, or when the machine is under load, or when the clock crosses midnight, is reading an input the build system does not know about: shared state, timing, the date. Such a test cannot be correctly cached, because its result is not a function of its declared inputs. Bazel's sandbox removes some of these hidden inputs by isolating tests from undeclared files, but it cannot remove timing, concurrency or the clock. There is a practical response to this interaction. Teams that cache test results aggressively should also, on a schedule, run tests without the cache: for example, a nightly job that reruns the full suite on the current main branch with cached results ignored, and perhaps several times over. Such runs restore the observations that caching removes, so that flaky behaviour shows up in the data rather than hiding behind a lucky first pass. They cost machine time, but they convert the cache's blind spot into a known, measured quantity. Where flakiness comes from The foundational academic study of flaky tests is "An Empirical Analysis of Flaky Tests" by Qingzhou Luo, Farah Hariri, Lamyaa Eloussi and Darko Marinov, presented at the ACM SIGSOFT International Symposium on Foundations of Software Engineering in 2014. The authors examined commits that fixed flaky tests in open-source projects of the Apache Software Foundation and classified the root causes. Their three most common categories were asynchronous waiting, concurrency and test order dependency, and they identified several smaller categories as well. Subsequent research has refined these categories, but the taxonomy has held up well and remains the standard starting point. Table 4 summarizes the main categories and typical fixes. Table 4. Common root causes of flaky tests and typical remedies. Root cause Typical symptom Typical remedy Asynchronous wait Test sleeps a fixed time, then checks Wait on an explicit condition or event Concurrency Race between threads in code or test Synchronize; make ordering deterministic Test order dependency Passes alone, fails after another test Reset shared state; isolate fixtures Resource leak Fails late in a long test run Release resources; isolate processes Network and external services Fails when a remote system is slow Fake the dependency; mark test external Time and dates Fails near midnight or in another zone Inject a controllable clock Randomness and unordered data Fails on some seeds or orderings Fix seeds; compare as sets, not lists Asynchronous waits are the classic case. A test starts an operation that completes on another thread or in another process, sleeps for a fixed interval, and then checks the result. On a quiet machine the operation finishes in time; on a loaded continuous integration agent, sometimes it does not. The fix is never a longer sleep, which makes the test slower without making it correct. It is to wait for an explicit signal that the operation has completed, with a generous timeout that is only reached when something is genuinely wrong. Concurrency flakiness arises from races in the code under test or in the test itself. Some of these are real bugs that the test is correctly, if intermittently, detecting, which is a reason never to dismiss flaky tests as test problems by default. Test order dependency arises when one test leaves behind state, such as a modified static variable, a record in a shared database or a file on disk, that another test depends on, positively or negatively. The test passes when run in one order and fails in another. Build tools that run tests in parallel or in randomized order expose these dependencies; tools that always run tests in the same order hide them until someone changes the order. Researchers have built tools specifically for detecting order-dependent tests by rerunning suites in different orders, such as iDFlakies, introduced by Wing Lam and colleagues in 2019. The remaining categories, including resource leaks, dependence on external network services, dependence on the current time or time zone, and dependence on randomness or the iteration order of unordered collections, each have recognizable signatures and standard remedies. What they share is that the test has an input, such as available memory, network latency, the clock or a random seed, that is not under its control and not part of what the build system considers its inputs. Detecting flakiness and the limits of retry A test cannot be identified as flaky from a single run. Detection requires observing the same test, on the same code, producing different results. There are two basic ways to get that observation. The first is retry within a build. When a test fails, the build tool immediately reruns it, once or a few times. If a rerun passes, the test is flaky by definition, since the code has not changed. Develocity defines flakiness this way: its documentation states that a test outcome is marked as flaky if it fails and succeeds within the execution of a single Gradle task, Maven goal, Bazel target, sbt task, or npm or Python test process. For Gradle, retry is provided by the Test Retry Gradle plugin or by the test retry functionality built into the Develocity Gradle plugin, configured with a maximum number of retries and an option controlling whether a test that passed only on retry should fail the build. For Maven, the Surefire and Failsafe plugins rerun failing tests through the rerunFailingTestsCount setting. Bazel provides a flaky attribute on test targets that causes a failing test to be retried up to three times, marking it failed only if every attempt fails, and a --flaky_test_attempts flag to set retry counts more broadly. Bazel reports tests that passed only after retry with a distinct flaky status. The second is analysis across builds. A test that fails on one build and passes on another build of the same code, perhaps on a different machine or at a different time, is also flaky. Detecting this requires a central record of test outcomes keyed by code revision, which is exactly what build observability platforms collect. Develocity applies cross-build analysis as well as within-build detection, and Bazel's results services can do similar analysis using test results from the Build Event Protocol. Deliberate stress testing complements both. Bazel's --runs_per_test flag runs a test many times, which can reveal low-rate flakiness before it reaches the main branch. Some organizations run new or modified tests many times in continuous integration as a gate before they are allowed to join the main suite. Detection data is most useful when aggregated and ranked. A list of flaky tests sorted by name is of little use. A list sorted by the engineer time each test has cost, through reruns and failed builds, tells a team where to work first. Apple's engineers described a version of this approach in "Modeling and Ranking Flaky Tests at Apple," presented at the International Conference on Software Engineering in 2020 by Emily Kowalczyk and colleagues, which scored tests by both how often and how unpredictably they flip between outcomes. Retrying failed tests is the most common response to flakiness, and it is double-edged. Used as a diagnostic, it is invaluable: it identifies flaky tests automatically and stops a single spurious failure from blocking a merge. Used as a cure, it is corrosive: the flaky test is never fixed, its retries add time to every build where it misfires, and, most dangerously, a real intermittent defect can pass on retry and ship. The useful middle path treats a pass-after-retry as a distinct outcome that is recorded, reported and owned, not as a pass. The build may be allowed to proceed, so that one engineer's flaky test does not block another engineer's unrelated change, but the flaky result is logged against the test, the test's owner is notified, and the flakiness counts toward a budget. Gradle's retry configuration supports exactly this distinction through its option to fail or not fail a build on a passed-after-retry result, and the reporting in Develocity and Bazel-oriented tools surfaces these outcomes separately. Quarantine, ownership and design Beyond retry, the most effective organizational tool is quarantine. A test known to be flaky is removed from the set of tests that block merges, but it continues to run, its results are collected, and it is assigned to an owner with an expectation that it will be fixed or deleted within a defined time. Quarantine protects the rest of the organization from the test while preserving the information it provides. Google's continuous testing infrastructure has long applied policies of this kind, and the Micco post describes a range of mitigations, including tracking flakiness per test and notifying owners. A quarantine policy only works with three rules. First, quarantine must be cheap and quick to enter: if it takes a week of process to quarantine a test that is blocking everyone, engineers will resort to retries and disabled tests instead. Second, quarantine must have an exit: a test that sits in quarantine indefinitely is a deleted test that still consumes resources. Teams should be required, after a fixed time, to fix the test, delete it, or justify keeping it with a documented reason. Third, quarantine must be visible: the count of quarantined tests per team, and how long they have been there, should be reported as prominently as build times. A growing quarantine is a sign that a team's tests or code are degrading. Ownership is the hinge on which all of this turns. A flaky test with no clear owner will not be fixed. Many organizations assign test ownership through the same mechanism as code ownership, so that a test in a team's directory belongs to that team, and route flakiness notifications accordingly. Google's research has gone further, with automated tools that attempt to locate the root cause of a flaky test in the code; Celal Ziftci and Diego Cavalcanti described one such system in "De-Flake Your Tests," presented in 2020. Such tools reduce the effort of fixing, but they do not remove the need for someone to act on what they find. Every remedy described so far treats flakiness after it appears. The more durable response is to design tests and systems that are less prone to it. That means giving code under test controllable dependencies: an injectable clock instead of direct calls to the system time, an injectable random source, interfaces to external services that can be replaced by deterministic fakes in tests. It means preferring many small, isolated tests over a few large ones that start whole systems, because large end-to-end tests combine every source of nondeterminism in the stack. It means running tests in randomized order and in parallel by default, so that order dependencies and shared state are exposed early. And it means treating hermeticity of tests, in the same sense in which Bazel treats hermeticity of build actions, as a goal: a test should depend only on what it declares. These practices are not new; most experienced engineers know them. What DPE adds is measurement and incentive. When the cost of each flaky test is visible, its owner is known and the organization has a policy for dealing with it, the old advice gets followed. When flakiness is invisible and unowned, it accumulates until the test suite becomes a ritual that everyone performs and no one believes. A fast feedback loop is only valuable if the feedback is true. Caching makes feedback fast by reusing results; it is safe only if results are deterministic. Flakiness is the main source of nondeterminism in test results, and so the work of controlling it is not a separate quality initiative. It is a precondition for every speed gain the rest of this book describes. Chapter 7: Running Less, Running Wider: Tests and the Outer Loop In most mature codebases, tests, not compilation, dominate build time. Compilation is well served by incremental compilers and caching; a typical change recompiles a few modules. Tests are harder. A change to a widely used library invalidates the cache keys of every test that depends on it, and those tests must run. A suite that takes thirty minutes on one machine will take thirty minutes every time anything near its root changes. This chapter looks at the three strategies available once work avoidance has done what it can: running fewer tests, by selecting the ones likely to matter; running tests more widely, by distributing them across machines; and organizing continuous integration so that the outer loop feeds, rather than duplicates, the inner one. From exact to predictive selection The safest way to run fewer tests is to run only those whose inputs have changed. This is exact selection, and it is what test caching provides. In Bazel, every test target is an action whose result is cached against its inputs; a change reruns exactly the tests whose transitive dependencies include changed files, and the rest report cached results. Several open-source tools for Bazel compute the set of affected targets between two revisions so that continuous integration can build and test only those. In Gradle, test tasks are cacheable, so a module whose code and dependencies have not changed will have its test task served from cache. Exact selection is sound, but its effectiveness depends on the granularity of the dependency graph. If a codebase is organized into a few large modules, each depending on most of the others, then almost any change invalidates almost every test. A change to a logging utility used everywhere will rerun the entire suite, even though very few tests could plausibly be affected by it. Fine-grained modules, with narrow and explicit dependencies, make exact selection far more effective. Bazel encourages such granularity by design; Gradle and Maven projects often have to be deliberately split to achieve it. Restructuring a build for finer granularity is one of the highest-value, and least glamorous, investments a DPE team can make, because it improves caching and test selection at the same time. But even with fine granularity, exact selection is conservative in a way that is often wasteful. It reruns every test that could be affected, not every test that is likely to be. A test that depends transitively on a changed library may exercise none of the changed code. Exact selection cannot know that; it can only see the dependency. Predictive test selection takes a different approach. Rather than asking which tests could possibly be affected by a change, it asks which tests are likely to fail given the change, and runs those. The question is answered with a statistical model trained on history: which files changed, which tests failed in the past when similar files changed, how often each test fails at all, how recently it was modified, and similar features. The best-known published description comes from Facebook, now Meta. In the paper "Predictive Test Selection," presented at the International Conference on Software Engineering's Software Engineering in Practice track in 2019, Mateusz Machalica, Alex Samylkin, Meredith Porth and Satish Chandra described a machine-learning system used in Facebook's continuous integration to choose a subset of tests for each code change. They reported that it reduced the infrastructure cost of testing by a factor of two while still reporting over 95 percent of individual test failures and over 99.9 percent of faulty changes to developers. The second figure is the more important one. The aim is not to catch every failing test on every change but to ensure that every bad change is caught by at least one failing test, which is a much easier target, because bad changes usually break several tests. Develocity offers predictive test selection as a product feature for Gradle and Maven builds that run tests on the JUnit Platform. According to its documentation, the model is trained on code changes and test outcomes from an organization's Build Scan data, combined with training across many projects, and predicts the relevant subset of tests for a change. It offers selection profiles, described as conservative, standard and fast, that trade the number of tests selected against the likelihood of catching failures. The documentation recommends using the "relevant tests" mode in early, frequent stages such as local builds and pre-merge continuous integration, and running the "remaining tests" in later, less frequent stages such as nightly or pre-release builds, so that no test is executed more than once for a given change and every test is eventually run. Between exact selection and statistical prediction sits a third approach, often called test impact analysis, which uses code coverage. The idea is to record, for each test, which source files or methods it executed during a previous run, and then, given a change, run only the tests whose recorded coverage touches the changed code. Coverage-based selection is more precise than dependency-based selection, because it reflects what a test actually exercised rather than everything it could reach. It has well-known blind spots, however. Coverage maps go stale as code changes. They miss effects that do not flow through executed code, such as a change to a configuration file, a resource or a reflection-based lookup. And collecting coverage slows test execution, so maps are usually refreshed periodically rather than on every run. Coverage-based selection is a useful signal, and some predictive systems use it as one feature among several, but on its own it is neither as safe as exact selection nor as adaptive as a trained model. That staging pattern is the key to using predictive selection safely. A model will sometimes be wrong. If skipped tests are never run, its errors become escaped defects. If they are run later, its errors become delayed feedback, which is a much smaller cost. Predictive selection therefore changes when a test runs, not whether it runs. Fast, probable feedback arrives in the inner loop and at merge time; complete, certain feedback arrives later in the outer loop. Two cautions apply. First, a predictive model is only as good as its training data, and its training data comes from test results. If a significant fraction of failures in the history are flaky, the model learns to predict flakiness, not defects. This is another reason to control flakiness first, and Meta's paper explicitly addressed the effect of flaky tests on its training. Second, predictive selection must be layered on top of a sound build. If the cache returns wrong results, or if test results depend on undeclared inputs, a probabilistic layer added above will amplify confusion. The order of operations matters: make the build honest, make the tests deterministic, and only then let a model decide what to skip. Distributing tests across machines The tests that remain must run, and the remaining lever is parallelism. Running tests in parallel on one machine is limited by its cores, memory and the tests' tolerance for sharing a machine. Running them across many machines removes the first two limits and, because each machine is isolated, often relaxes the third. Bazel supports this natively. A test target can declare a shard_count, and Bazel will split the test into that many shards, each running a subset of test cases, as separate actions. With remote execution, those actions run on separate workers, and an entire repository's test suite can be spread across a large cluster. Because each shard is an action with declared inputs, its results are cached individually. Gradle and Maven builds can achieve similar effects through Develocity's Test Distribution. According to its documentation, it uses historical test execution data to create balanced partitions of tests with similar expected execution times, transfers the tests and their supporting files to remote agents, runs them, and streams the results back to the build in real time. A build can use local and remote executors at the same time; if an agent disconnects, the affected partition is rescheduled on another executor; and builds configured with local executors can always progress even if remote agents are unavailable. Test Distribution supports Gradle 5.4 and later and Maven with recent versions of the Surefire and Failsafe plugins, for tests running on the JUnit Platform. Two features of distribution deserve attention. One is balancing: if tests are split naively, for example by test class count, one partition may contain a slow integration test and take far longer than the others, so the total time is the slowest partition's time. Using historical durations to balance partitions is what makes distribution effective. The other is the local option. Distribution is not only for continuous integration. An engineer running a module's full test suite on a laptop can, if remote agents are available, have most of it run elsewhere, which turns a coffee-break wait into a short one. This is one of the few ways to make large test suites part of the inner loop. Distribution has costs. Remote agents must be provisioned and maintained, and they must provide an environment the tests will run in correctly. Tests that depend on things specific to a developer's machine, such as a locally running database or a particular file path, will fail remotely, which is another way that hermeticity pays off. Network transfer of test inputs adds overhead, so distribution helps most for suites that are long in total but composed of many independent pieces. Designing and measuring the outer loop Continuous integration is usually set up before anyone thinks about caching, and its shape often works against it. Each pipeline was built to answer a question, such as "does this pull request build?", in isolation, on a clean machine, with no reference to work done elsewhere. A DPE-oriented design starts from a different principle: continuous integration is the producer of shared results, and every other build, local or remote, is primarily a consumer of them. Several consequences follow. Build the main branch promptly: when changes merge, continuous integration should build and test the new main revision quickly, writing results to the remote cache, so that engineers who pull the latest main get hits. If main is built only nightly, every engineer who pulls during the day pays for their own clean build. Align environments: continuous integration agents should use the same toolchain versions, operating system settings and relevant environment as developer machines, or the cache will not bridge them. Give ephemeral agents a warm start: agents created fresh for each job lose local caches, downloaded dependencies and daemon state; provisioning these from a shared source, rather than rebuilding them every time, is a large and frequently neglected saving. Separate trust domains: as Chapter 5 argued, jobs that build untrusted code must not write to caches that trusted builds read. The merge process itself is part of the outer loop. In a busy repository, two changes that each pass continuous integration individually can break the main branch when combined. Organizations at scale have built systems to manage this. Google's continuous integration system, described by Atif Memon and colleagues in "Taming Google-Scale Continuous Testing" in 2017, deals with the volume of changes by not running every affected test on every commit and instead using information about change history to decide when and what to run. Uber described a system it calls SubmitQueue in "Keeping Master Green at Scale," presented at the EuroSys conference in 2019 by Sundaram Ananthanarayanan and colleagues, which predicts which pending changes are likely to conflict and orders their verification accordingly. Merge queues, now offered by mainstream code hosting platforms, give smaller organizations a simpler version of the same guarantee: changes are tested in the order they will be merged, against the state they will actually merge into. All of these systems depend on fast, cache-backed builds; a merge queue in front of a thirty-minute uncached build becomes a bottleneck that throttles the whole organization. The outer loop has its own measures, and the most useful of them are not the ones most continuous integration dashboards display. Pipeline duration, the time from the start of the first job to the end of the last, is the familiar number. It omits two things engineers feel directly. The first is queue time: how long a job waits for an available agent before it starts. In organizations whose continuous integration capacity is sized for average load, queue time can be negligible at night and dominant at the busiest hours of the working day, exactly when engineers most need fast answers. A pipeline that runs in ten minutes but waits twenty for an agent is a thirty-minute pipeline from the engineer's point of view. Queue time is a capacity problem, and it is one of the places where work avoidance pays twice: every job that is mostly cache hits finishes sooner and returns its agent to the pool sooner, which shortens the queue for everyone else. The second is time to first meaningful signal: how long after pushing a change the engineer learns of the first failure, if there is one. A pipeline that runs a twenty-minute integration suite before a thirty-second style check will tell an engineer about a formatting error after twenty minutes. Ordering stages so that the fastest checks with the highest failure rates run first, and failing fast when they fail, can cut the effective feedback time for failing changes dramatically without making any individual step faster. The same logic applies within a test suite: if historical data shows which tests fail most often for which kinds of change, running those first means a failing change is reported in the first minutes of the run rather than the last. Develocity's test history and Bazel's results services both provide the data needed for this ordering, and some predictive selection systems apply it automatically. A third measure is worth tracking for its diagnostic value: the rerun rate, the fraction of pipelines that are retried by hand or by automation without any change to the code. Every rerun is either an infrastructure failure or a flaky test, and a rising rerun rate is usually the earliest visible sign that trust in the outer loop is eroding. Fast feedback first, complete feedback eventually The strategies in this chapter combine into a layered design, and it is worth stating it plainly because many organizations implement the layers in the wrong order or skip the ones that make the others safe. At the bottom is the honest build: complete inputs, deterministic outputs, a remote cache populated by trusted continuous integration builds. On top of that sits exact selection, through caching and fine-grained modules, which ensures that unchanged tests do not rerun. Above that comes predictive selection, which runs the tests most likely to catch a defect early, and defers the rest to a later stage. Alongside both, distribution provides the parallelism to make whatever does run finish quickly. And around all of it, flakiness controls ensure that the signals produced at each layer can be believed. The engineer's experience of this design is simple. A local change compiles in seconds because almost everything comes from cache. The relevant tests run in a minute or two, partly on remote agents. The pull request receives a verdict quickly enough that the engineer has not yet moved on. And the complete suite runs later, catching the rare defect the faster stages missed, before anything reaches users. Each layer trades a little certainty for a lot of speed, and the layers beneath it make that trade safe. What the design does not do is make the last step of the inner loop fast: getting the changed code running, in something resembling its real environment, where the engineer can try it. That step lies outside the build and test systems most organizations have optimized, and it is where minutes most often still hide. It is the subject of the next chapter. Chapter 8: Seconds, Not Minutes: The Local Deployment Loop A build that finishes in five seconds is a fine thing, but engineers do not ultimately want builds. They want to see their change working: the endpoint returning the new field, the page rendering the new button, the service handling the request it used to reject. For many engineers, especially those working on services that run in containers on an orchestration platform, the path from a compiled change to a running one is longer than the build itself. It involves packaging, building an image, pushing it somewhere, deploying it, waiting for it to start and warming it up. Each step was designed for production, where correctness and reproducibility matter more than speed, and each is repeated in full for a one-line change. This chapter applies the principle of work avoidance to that path, with the aim of reducing the time from saving a file to seeing the result from minutes to seconds. Anatomy of the slow loop Consider a typical service written in Java or Kotlin, packaged as a container image and run on Kubernetes, with a few dependent services and a database. The production-shaped way to try a change looks like this. The engineer compiles the code and runs its tests. The build packages the compiled classes and dependencies into an archive. A container build copies the archive into an image on top of a base image. The image is pushed to a registry. A deployment manifest is updated to reference the new image, and the cluster pulls the image, stops the old container and starts a new one. The application starts, which for a large Java service with a dependency injection framework can itself take a substantial time, and then warms up, populating caches and compiling hot code paths. Finally the engineer sends a request. Every step is reasonable in production. In the inner loop, almost every step repeats work. Table 5 walks through the stages, what usually makes each slow, and the techniques that avoid it. Table 5. Stages of the local deployment loop and how each can be shortened. Stage What usually makes it slow Work-avoidance technique Compile and test Rebuilding unchanged modules Incremental compile, build cache, test selection Package Rebuilding archives of unchanged dependencies Skip packaging; run from compiled output Image build Invalidated layers; dependencies in same layer as code Layer by change frequency; daemonless image builders Push and pull Transferring whole images to a remote registry Local registry or cluster; sync files instead Deploy and restart Replacing the whole container for a small change Live file sync with in-process restart Start and warm up Full application startup on each change Hot reload; restart only application classes The table makes a point that is easy to miss: most of the steps are not slow because the computation is hard. They are slow because they repeat work whose inputs have not changed. The dependencies did not change, but the archive containing them was rebuilt. The base image did not change, but a layer above it was invalidated. The framework did not change, but it was reinitialized from scratch. The same diagnosis applies here as everywhere else in this book, and the same cure. Avoiding packaging, rebuilding and redeploying The single most effective step is often to skip packaging and image building in the inner loop altogether. A compiled class file or a transpiled JavaScript module is already a runnable artifact. Packaging it into an archive and then into an image serves distribution, which the inner loop does not need. For services that can run directly on the engineer's machine, that means running the application from the build's output directories, as most integrated development environments do by default, and using the language runtime's reload mechanisms. The Java Virtual Machine has supported limited hot swapping of method bodies through its debugging interface for many years; changes to method signatures, fields or class hierarchies are beyond it. Frameworks fill the gap with restart strategies that are much faster than a cold start. Spring Boot's developer tools, for example, use two class loaders: one for third-party libraries, which rarely change, and one for the application's own classes. When application classes change, only the second loader is discarded and rebuilt, so the restart avoids reloading the bulk of the dependencies. Quarkus offers a development mode with live reload on request. In the JavaScript ecosystem, development servers with hot module replacement, such as Vite, swap changed modules into a running page without a full reload, preserving application state. The pattern in every case is the same: split what changes often from what changes rarely, and rebuild only the former. Continuous build modes reinforce this. Gradle's continuous build reruns tasks when their inputs change; for Bazel, the ibazel watcher from the Bazel project does the same, rebuilding and optionally restarting a target whenever its source files change. With either, the engineer saves a file and the build has already started, often finishing before they switch windows. Some services cannot reasonably run outside a container, and some teams want the inner loop to exercise the real image. In that case the goal is to make the image build itself a matter of work avoidance. Container images are stacks of layers, and container build tools cache layers by the content of their inputs. A layer is reused if the instruction that created it and everything it depends on are unchanged. Any change invalidates that layer and every layer above it. This is the chaining effect of Chapter 2 in a different setting, and it has the same lesson: put the things that change rarely at the bottom. A Dockerfile that copies the entire application, dependencies and code together, in one step, rebuilds the whole layer on every code change. One that copies the dependency manifest, installs dependencies, and only then copies the source code will reuse the dependency layer until the manifest changes. Modern builders also support cache mounts that persist package manager caches between builds without baking them into the image. Purpose-built tools take this further. Google's Jib builds container images for Java applications directly from Maven or Gradle, without a Docker daemon and without a Dockerfile, and splits the application into separate layers for dependencies, resources and classes, so that a code change rebuilds and pushes only the small classes layer. In Bazel, the rules_oci rule set builds images from Bazel targets as ordinary actions, which means image layers are cached and reproducible like any other output and can be served from the remote cache. Reproducibility matters here as much as in compilation: an image builder that embeds the build time in image metadata will produce a different image digest on every build, defeating any downstream cache or deployment step that keys on digests. Image transfer is the next cost. Pushing an image to a remote registry and having a cluster pull it back is slow when the cluster is remote and the network is constrained. Local development clusters, such as those created by kind, minikube or k3d, can load images directly from the engineer's machine without a registry round trip. Where the cluster is remote, pushing only changed layers helps, and the layered builders above ensure that the changed layers are small. The largest gains in container-based inner loops come from not rebuilding the image at all for most changes. Tools designed for Kubernetes development, among them Skaffold, Tilt, DevSpace and Garden, watch the source tree and, for changes that do not require a new image, copy the changed files directly into the running container. Skaffold calls this file sync; Tilt calls it live update. Combined with an in-process restart mechanism like those described above, the result is that saving a file triggers an incremental compile, a copy of a few class files or scripts into the container and a fast restart of application code, without touching the image, the registry or the deployment. Only when something structural changes, such as a new dependency or a modified base image, does the tool fall back to a full rebuild and redeploy. This is a deliberate, controlled break from production fidelity. The running container no longer exactly matches any image; it is an image plus a set of synchronized changes. That is acceptable in the inner loop, where speed matters most, as long as the outer loop builds and tests the real image before anything is merged. The two loops have different purposes, and it is a mistake to make the inner loop pay for the outer loop's guarantees. Dependencies, startup and other loops Services rarely run alone. A change to one service may need several others, a message broker and a database to be exercised meaningfully. Running all of these on a laptop is often impossible; running none of them makes testing unrealistic. Two broad approaches address this. One is to run a slimmed-down environment locally, with real versions of the service under development and its closest dependencies and lightweight fakes or shared remote instances for the rest. The other is to run the service under development locally while connecting it to a shared development cluster where everything else runs. Tools such as Telepresence and mirrord take the second approach: they route traffic intended for a service in the cluster to a process on the engineer's machine, and give that process access to the cluster's network, configuration and dependencies, so that the locally running code behaves as if it were deployed. The engineer gets the fast local loop, with a debugger and hot reload, and the realistic surroundings of the cluster, without deploying anything. A third approach moves the engineer's whole development environment into the cloud. Cloud development environments, offered by several vendors and by code hosting platforms, give each engineer a powerful remote machine, close to the build cache and the cluster, preconfigured with the project's toolchain. This trades the latency of typing over a network for much shorter build and deploy times and an environment that matches continuous integration more closely. Whether the trade is worthwhile depends on the size of the codebase and the quality of the network, but the logic is consistent with the rest of this book: bring the work close to where its cached inputs already are. When restarts cannot be avoided, the application's own startup time becomes the limit. Large services built on reflection-heavy frameworks can take a long time to start, and in an inner loop that restarts on every change, that time dominates. The Java platform has been addressing startup for several years. Class data sharing lets the runtime map previously parsed class metadata from an archive rather than loading it from scratch, and the OpenJDK project known as Leyden has been extending this toward broader ahead-of-time caching of work the runtime would otherwise repeat on every start, with initial pieces delivered in recent JDK releases. Frameworks have added options such as lazy initialization of components, which defers work until it is needed. Native image compilation, which produces a standalone executable that starts very quickly, is excellent for production startup but generally a poor fit for the inner loop, because the native compilation step itself is slow; it trades a fast start for a slow build, the opposite of what the inner loop needs. The general principle, once again, is work avoidance. Startup is repeated work: parsing the same classes, initializing the same frameworks, building the same internal caches. Anything that allows that work to be done once and reused, whether an archive of class metadata, a long-lived parent process with a restartable child or a lazily initialized component, shortens the loop. Not every inner loop ends in a container. For mobile engineers, the equivalent of deployment is installing a changed application on a device or emulator, and the same principles apply in a different form. Android builds use Gradle, so everything in Chapter 3 about caching, the configuration cache and avoiding work across modules applies directly; Android applications with large numbers of modules are among the heaviest users of Gradle's build cache. On top of the build, Android Studio's Apply Changes feature pushes code and resource changes to a running application without reinstalling it, where the nature of the change permits, falling back to a full reinstall when it does not. The pattern is the familiar one: a fast path for the common small change, a correct but slower path for structural changes, and a tool that decides between them. Modularization plays an even larger role in mobile codebases than in services, because a mobile application is usually a single deployable binary built from all of its modules. If those modules are coarse and tightly coupled, any change recompiles much of the application. If they are fine-grained, with feature modules depending on narrow interfaces rather than on each other's implementations, a change to one feature recompiles that feature and little else, and the rest comes from the cache. Several large mobile teams have described multi-year efforts to restructure their applications into hundreds of modules for exactly this reason; it is the mobile equivalent of the granularity argument made for test selection in Chapter 7. Keeping the fast loop honest A fast inner loop can drift away from reality. Configuration used in local development diverges from configuration in production. A synchronized container accumulates changes that a fresh image would not contain. A fake dependency behaves differently from the real one in some edge case. The faster and more specialized the inner loop becomes, the more important it is that the outer loop catches what the inner loop misses, and that engineers understand which guarantees each loop provides. A few habits keep this drift in check. Development environments should be defined in code, alongside the service, so that they are versioned and reviewed rather than accumulated on individual laptops. Fakes of external dependencies should be tested against the real dependencies' contracts, so that they fail when the real behaviour changes. Development tools should rebuild from a clean image periodically, or on demand with a single command, so that a long-lived synchronized container does not become the only place a change has ever worked. And the outer loop must always build the real artifact from source, with the real image and the real deployment, before anything merges. The inner loop is allowed to cut corners precisely because the outer loop does not. A team that applies these techniques can transform its inner loop without changing what it ships. The service is compiled incrementally, with unchanged modules served from the remote cache. A watcher triggers the build on file save. In the common case of a code-only change, a development tool syncs the changed class files into the running container, where a restart mechanism reloads only the application's classes. Dependencies, base image, registry and deployment are untouched. The engineer sends a request and sees the change, typically within seconds of saving the file rather than the minutes the production-shaped path required. When a dependency changes, the tool falls back to rebuilding the affected image layer, which is small because the image was layered by change frequency, and redeploying, which still takes far less time than a full rebuild because everything below the changed layer is reused. None of these steps is exotic, and most are supported by mature open-source tools. What prevents most organizations from achieving such a loop is not technology but ownership. The inner deployment loop sits between teams: the build belongs to one group, the container images to another, the cluster to a platform team and the framework configuration to each service team. No one is responsible for the whole path, so no one optimizes it. Treating that path as a product, with an owner, a measured latency and a target, is a DPE responsibility. The measure is simple and every engineer understands it: how long, after I save this file, until I can see whether it worked? Chapter 9: Running a Developer Productivity Engineering Practice The techniques in the preceding chapters are well documented, and most are available in open-source tools. Yet many organizations that adopt them see disappointing results, while others, with the same tools, transform how their engineers work. The difference is rarely technical. It lies in whether someone owns the feedback loop, whether that owner measures the right things, whether the work is sequenced so that each step makes the next one safe, and whether the organization understands what it is buying. This chapter is about those questions: how to set up and run a DPE practice, how to make its economic case, how to choose between toolchain strategies, and how the spread of AI-assisted development changes the stakes. A charter and a team A DPE practice begins with a charter that says, in one or two sentences, what the team is for. A good charter names the outcome, not the tools: something like "reduce the time engineers spend waiting on builds, tests and deployments, and increase the reliability of the signals those systems produce." It does not say "operate the build cache" or "maintain continuous integration," although the team may do both, because a charter phrased in terms of tools invites the team to optimize the tools rather than the outcome. The team needs a mix of skills that is unusual in one group. It needs people who understand build systems deeply, at the level of how a cache key is computed and how a plugin interacts with the configuration phase. It needs infrastructure engineers who can operate caches, remote execution clusters and test agents reliably. It needs people comfortable with data, who can turn telemetry from thousands of builds into a clear account of where time goes. And it needs people who can work across teams, because almost every fix requires changing code that belongs to someone else. In small organizations these may be the same two or three people. In large ones, the practice may grow to dozens, often organized by the loops they serve: inner loop, continuous integration, test infrastructure. Placement matters. A DPE team buried inside one product group will optimize that group's builds and lack the authority to set standards for others. A team placed in a central platform or engineering effectiveness organization has the reach to change shared build logic, define conventions and set policies for flaky tests. It also needs a senior sponsor who will back it when its standards inconvenience a team in a hurry. Measuring and making the case Chapter 1 described the traps of measurement in general terms. In practice, a DPE team needs a small set of measures, refreshed continuously and segmented by kind of build. The first is feedback time: the distribution of build and test durations, separately for local builds and continuous integration, and for incremental and full builds. Medians describe the common experience; high percentiles describe the frustrating one. The second is avoided time: the execution time saved by caching and up-to-date checks, estimated from the original durations of the avoided work. This is the measure that most directly connects caching investment to engineer time. The third is failure cost: the time lost to failed builds, broken down by cause, with flaky test failures, infrastructure failures and genuine verification failures distinguished. Only the last category represents the build doing its job. The fourth is inner loop latency: the time from saving a change to seeing it run, which is harder to instrument automatically but can be sampled. The fifth is perceived experience, gathered through short, regular developer surveys of the kind recommended by the DevEx research, asking engineers how often they wait, how much they trust test results and where they feel friction. The survey deserves special mention because technical teams often dismiss it. Telemetry tells a DPE team what happens; surveys tell it what matters. A build that takes ninety seconds may be tolerated in one team and resented in another, depending on how often it runs and what engineers are doing while they wait. Engineers will also report problems that no telemetry captures, such as a deployment step that requires a manual ticket, or a test environment that is unavailable every Monday morning. The combination of the two kinds of data, objective and subjective, is far more reliable than either alone. A DPE team spends money: on engineers, on cache and execution infrastructure, on commercial platforms. To survive budget cycles, it must show that it saves more than it spends, in terms that people outside engineering accept. The core calculation is simple. Estimate the engineer time saved per week by an intervention, multiply by a loaded cost of engineer time, and compare with the cost of the intervention. The time saved can be estimated from the avoided-time measure for caching, from reduced waiting for build and test acceleration, and from reduced reruns and investigations for flaky test work. The difficulty is in the multipliers discussed in Chapter 1. A minute saved below the attention threshold, where engineers would have waited without switching tasks, is worth roughly a minute. A minute saved by bringing a build back below that threshold is worth more, because it also removes a task switch. Conservative analysts count only the direct minutes and present the indirect effects qualitatively; this understates the benefit, but it produces a number that finance teams will not dispute, and the direct savings alone usually justify the investment. Two further arguments carry weight with leadership. One is capacity: build and test infrastructure costs money, and work avoidance reduces the machine hours consumed by continuous integration, often enough to pay for the caching infrastructure. The other is risk: flaky tests and untrustworthy builds increase the chance of shipping defects, and a cache poisoning attack is a supply-chain risk with obvious consequences. Framing DPE as a contributor to delivery speed, infrastructure efficiency and software supply-chain integrity gives it allies beyond the engineering organization. Avoid one argument: the claim that DPE will make engineers write more code. It invites the output-counting view of productivity that the research has rejected, it is hard to demonstrate, and it alarms engineers who suspect that productivity measures will be used against them. DPE's honest claim is narrower and more defensible: it removes waiting and noise from engineers' days, and what engineers do with the recovered time is a matter for their managers and themselves. Measures become useful when they are attached to targets, and the most effective form of target borrows from site reliability engineering. Rather than an aspiration such as "builds should be faster," a DPE team can agree with the engineering organization on service-level objectives for the feedback loop: for instance, that a stated high percentile of incremental local builds of a given project complete within a stated number of seconds, or that pull request pipelines deliver a first verdict within a stated number of minutes. The specific values should come from the organization's own baseline and its engineers' tolerance, not from another company's figures. What matters is that the objective is explicit, measured continuously and owned. When a change to the build pushes a project outside its objective, that is treated as a regression to be investigated, in the same way that a latency regression in a production service would be. The same idea applies to reliability. A flakiness budget sets the maximum tolerable rate of spurious failures for a project or a team's tests. While a team is within budget, flaky tests are handled through the ordinary quarantine process. When a team exceeds it, fixing flakiness takes priority over new test-dependent work until the rate comes back down. Budgets of this kind turn an endless, guilt-laden backlog into a clear rule about when the work must be done, and they give team leads a defensible reason to schedule it. Objectives and budgets also discipline the DPE team itself. They tell it which projects need attention most, they make progress visible to sponsors, and they prevent the common drift in which a productivity team spends its time on the technically interesting problem rather than the one costing engineers the most. A team whose highest-traffic project is outside its objective has no business building a new caching feature for a project that is comfortably within its own. Sequencing the work and choosing a toolchain The most common mistake in DPE programs is doing things in the wrong order. Teams buy remote execution before their builds are hermetic, turn on predictive test selection while a quarter of their test failures are flaky, or enable a remote cache that developers write to and then spend months chasing wrong results. Each step in the chain of this book depends on the steps below it, and the order matters. The first step is observability. Before changing anything, collect data on every build, local and continuous integration. Build Scans in the Gradle and Maven world and Build Event Protocol services in the Bazel world provide this. Without it, every later decision is a guess, and every later claim of improvement is unprovable. The second is removing waste and fixing overhead: unneeded tasks, misconfigured parallelism, slow dependency resolution, configuration-phase costs. These fixes are usually cheap and immediately visible, and they build credibility for the program. The third is making the build honest and then enabling caching: local first, then remote, with writes restricted to continuous integration. The validation experiments described in Chapter 3 and the execution-log comparisons of Chapter 5 guide the work. This is the longest phase and the one with the largest payoff. The fourth is controlling flakiness: detection, quarantine, ownership and a budget. It can proceed in parallel with caching work but must be well advanced before the fifth step. The fifth is test acceleration: predictive selection and distribution, layered on a build whose results are trustworthy. The sixth is the deployment loop: the techniques of Chapter 8, which usually require cooperation from platform and service teams and benefit from the credibility the earlier phases have built. Throughout, the practice should publish its results, including failures. Engineers trust a productivity team that admits when a change did not help, and distrust one whose dashboards always go up. Organizations often ask whether they should move to Bazel or improve their existing Gradle or Maven builds with Develocity or similar tools. The question is less binary than it seems, and the answer depends on a few factors. Bazel is most compelling when the codebase is large, spans several languages, lives in one repository or a small number of them, and is maintained by enough engineers that the cost of a dedicated build team is small relative to the time it can save. Its enforced hermeticity, universal action and test caching, and remote execution through an open protocol give it a ceiling no other mainstream approach matches. Its costs are migration effort, ergonomics for engineers used to their language's native tools, and the ongoing discipline of keeping exceptions under control. Improving an existing Gradle or Maven build is most compelling when the codebase is primarily on the Java platform, when the existing build is serviceable, and when the organization cannot justify a migration of months or years. The ceiling is lower, because hermeticity is verified rather than enforced and remote execution of arbitrary tasks is not part of the model, but the path is incremental: each fixed task and each improved cache hit rate delivers value immediately. Develocity adds observability, a managed cache infrastructure and test acceleration on top, and it now supports Bazel builds as well, which means the observability layer need not be tied to the build tool decision. Many large organizations end up with a mix: Bazel for a central monorepo, native tools for peripheral projects, with a shared observability and caching layer across both. The worst outcome is a half-finished migration that leaves two build systems both poorly maintained. Before committing to Bazel, an organization should be confident it can fund the migration to completion; if it cannot, making its existing build honest is the better investment. Feedback loops in the age of AI-assisted development The arrival of AI coding assistants and autonomous coding agents changes the economics of the feedback loop in ways that strengthen the case for everything in this book. The first change is volume. An assistant that drafts code quickly increases the rate at which changes are produced, and each change still has to be built, tested and reviewed. If the build and test system was a constraint before, it becomes a tighter one. The 2024 report from Google's DORA research program found that, while AI adoption was associated with gains in individual productivity, flow and job satisfaction, it was also associated with negative effects on software delivery throughput and stability. One plausible reading is that faster code generation does not automatically translate into faster delivery when the rest of the pipeline, including verification, has not kept pace. The company behind Develocity made the same argument in explaining its 2026 rebrand, saying that AI has shifted the bottleneck from writing code to reviewing and validating it. The second change is that agents are themselves consumers of feedback loops. An autonomous agent that edits code, runs the build, reads the failures and edits again is running an inner loop, often many more iterations per task than a human would. Every second of build latency is multiplied across those iterations, and every flaky failure sends the agent chasing a problem that does not exist, possibly making unnecessary changes in the process. A human engineer learns to recognize a known flaky test and ignore it. An agent, unless told, treats every red signal as real. Deterministic, fast, well-instrumented builds are therefore even more valuable to agents than to people. Build observability platforms have begun to expose their data to coding agents directly; Develocity, for example, now advertises integrations through the Model Context Protocol, so that an agent can query build and test history rather than parse console logs. The third change concerns trust. When a growing share of code is written by machines, the build and test system becomes a more important line of defence, because human review may become less thorough as volume increases. A test suite that can be relied upon, a cache that cannot be poisoned and a build whose outputs are reproducible from source are the mechanisms by which an organization knows that what it ships is what it intended. DPE was conceived as a way to make engineers faster. In an era of machine-generated code, it is equally a way to keep verification credible. The healthiest DPE teams think of their work as a product whose users are the organization's engineers and, increasingly, its agents. They maintain a roadmap, publish release notes when they change the build, run user research through surveys and interviews, and measure adoption and satisfaction. They provide documentation and office hours. They treat a regression in build time as a product defect with a root cause to be found, not as weather. This framing also protects the team from a common failure: becoming the department of "no." A DPE team that enforces standards without providing value will be routed around. One that makes the right way also the easy way, by providing convention plugins that are cacheable by default, templates for new services with fast inner loops built in and tooling that explains a cache miss in one click, will find its standards adopted willingly. Honesty in builds is, in the end, a habit spread across hundreds of engineers. A productivity team cannot impose it. It can only make it the path of least resistance. Conclusion The engineer in the opening pages of this book spent an afternoon waiting: for a build that repeated work already done, for a test that failed for no reason, for a continuous integration pipeline that rebuilt an unchanged codebase from scratch. The chapters since have taken that afternoon apart. What remains is to say what follows from the analysis, because the implications reach beyond build engineering into how organizations think about speed, trust and the work of their engineers. Speed is a property of honesty The first implication is the one this book has argued throughout: an organization cannot buy fast feedback directly. It can buy faster machines, larger caches and more test agents, and each will help a little. But the large gains, the ones that turn minutes into seconds, come from not repeating work, and work can only be skipped when the build can prove that its inputs have not changed. That proof depends on completeness and determinism: every input declared, every output reproducible. The organization that invests in honesty gets speed as a consequence. The organization that pursues speed without honesty gets a cache it cannot trust, a test suite it reruns until green, and engineers who quietly run clean builds to be safe. This reframes what a slow build is. It is not primarily a performance problem. It is an epistemological one: the build does not know enough about itself to avoid work. Every undeclared input, every timestamp in an output, every test that reads the clock is a gap in the build's self-knowledge. Closing those gaps is the real work of DPE, and the speed is a measurement of how much of it has been done. It also explains why Bazel and Gradle, so different in design, converge on the same outcomes when used well. Bazel enforces honesty at the point of authorship; Gradle, with Develocity, verifies it after the fact and makes violations visible. The choice between them is a choice about when to pay for honesty, not whether. An organization that understands this will evaluate toolchains by asking how each would surface its build's dishonesties, and how much each would cost to fix, rather than by comparing benchmark times on someone else's codebase. Trust is the scarce resource The second implication is that trust, not computation, is the scarce resource in a feedback system. Machine time is cheap and getting cheaper. Engineer attention is expensive and finite, and it is spent most wastefully on deciding whether a signal is real. A flaky test consumes far more than its runtime; it consumes the attention of every engineer who sees it fail and must decide whether to care. A cache that has once returned a wrong result consumes trust that will take months to rebuild. That is why the chapter on flaky tests sits at the centre of this book rather than at its edge. Flakiness is the place where the purity assumption behind caching meets the reality of tests that touch time, concurrency, networks and shared state. An organization that tolerates flakiness has decided, in effect, that its feedback is allowed to be wrong some of the time. Every acceleration technique built on top inherits that tolerance and amplifies it: a predictive model trained on noisy failures, a cache that records lucky passes, an agent that rewrites working code in response to a spurious failure. Controlling flakiness is not a quality initiative that competes with speed for resources. It is the foundation that makes speed worth having. Ownership decides the outcome The third implication concerns organization. Nothing in this book is secret. The theory of content-addressed caching, the causes of flaky tests and the techniques for fast local deployment are documented in official guides, academic papers and open-source repositories. The tools are available, many of them free. Yet most organizations have slow, noisy builds. The difference between those that do and those that do not is almost always whether someone owns the feedback loop as a whole. The feedback loop crosses team boundaries at every step. Build logic belongs to one group, continuous integration to another, test infrastructure to a third, container images and clusters to a platform team, and the tests themselves to every product team. In the absence of an owner, each part is optimized locally or not at all, and the aggregate degrades by small increments. DPE as a discipline is, at bottom, the decision to give the loop an owner with the authority to set standards, the data to see the whole path and the mandate to treat latency and unreliability as defects. Organizations that make that decision get results from tools that others found disappointing. Feedback is the bottleneck now The fourth implication is about the near future. For most of the history of software, the scarce step was writing code. Builds and tests were overhead around the expensive activity of a person thinking and typing. That balance has shifted. Code is now produced faster, by people working with AI assistants and increasingly by agents operating with limited supervision. The steps that remain stubbornly bound by time are verification: building, testing, reviewing and deploying. The research on AI adoption and delivery performance, and the reasoning that led the company behind Develocity to rename itself around this shift, point in the same direction. The organizations that benefit most from faster code generation will be those whose feedback loops can absorb the extra volume without slowing down or becoming less reliable. This changes the status of DPE. It was once a specialist concern of large technology companies with enormous monorepos. It is becoming a general condition for making use of the tools every engineering organization is adopting. An agent iterating against a five-minute build with a flaky test suite will waste most of its effort. The same agent against a build that answers in seconds and tells the truth can do useful work. The value of a coding assistant is bounded by the quality of the loop it runs in. What to do on Monday An argument about principles should end with something actionable, and the actions follow directly from the analysis. Start by seeing. Collect data on every build, local and remote, and put it somewhere people can look at it. Most organizations are surprised by what they find, and the surprises set the agenda better than any outside framework. Then measure the right things: distributions rather than averages, avoided time rather than hit rates, the cost of failures by cause, and how engineers experience the loop. Ask engineers directly where they wait and what they do not trust. Then make the build honest, one experiment at a time: the same build twice in the same place, then with outputs cleaned, then from a different directory, then on a different machine. Fix the earliest divergence first. Restrict cache writes to trusted builds. Run some builds without the cache and compare. Treat flaky tests as defects with owners, deadlines and a budget, and never let retry substitute for repair. Only then layer on prediction and distribution, and attack the deployment loop that sits between a compiled change and a running one. Through all of it, keep asking the question every engineer understands: after I save this file, how long until I know whether it worked, and can I believe the answer? Everything in this discipline is an attempt to make the first half of that answer smaller and the second half yes. Glossary Action cache. In Bazel's remote caching protocol, the index that maps the digest of an action to the metadata of its result, including the digests of its output files. Build Scan. A detailed record of a single build's execution, published to Develocity, showing tasks or goals, timings, cache outcomes, inputs and test results. Cache key. A digest computed from everything that affects a build step's output, used to look up previously produced results. Configuration cache. A Gradle feature that stores the result of the configuration phase, including the task graph, so that later builds can skip configuration. Content-addressable store. A store in which each file is saved under a cryptographic digest of its own content. Flaky test. A test that both passes and fails on the same code without any change to its inputs. Hermeticity. The property that a build step depends only on its declared inputs and is isolated from the rest of the machine's environment. Inner loop. The cycle of editing, building, testing and running code on an engineer's own machine or development environment before sharing a change. Outer loop. The cycle that begins when a change is shared: continuous integration, review, merging and deployment. Predictive test selection. Choosing which tests to run for a change by using a statistical model of which tests are likely to fail, deferring the rest to later stages. Relocatability. The property that a cache key does not depend on where a workspace is located on disk, so results can be shared between machines. Remote execution. Running build steps on a shared cluster of worker machines instead of locally, with results stored in a shared cache. Notes 1. Abi Noda, Margaret-Anne Storey, Nicole Forsgren and Michaela Greiler, "DevEx: What Actually Drives Productivity," ACM Queue 21, no. 2 (2023). Source for the three dimensions of developer experience: feedback loops, cognitive load and flow state (Chapter 1). 1. Nicole Forsgren, Jez Humble and Gene Kim, Accelerate: The Science of Lean Software and DevOps (IT Revolution, 2018). Source for lead time for changes as a key delivery measure (Chapter 1). 1. Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck and Jenna Butler, "The SPACE of Developer Productivity," ACM Queue 19, no. 1 (2021). Source for the SPACE framework (Chapter 1). 1. Gradle documentation, "Build Cache" and "Configuration Cache," docs.gradle.org (accessed October 2026). Source for cache key composition, @CacheableTask, path sensitivity, local and remote caches, and the status of the configuration cache as preferred in Gradle 9 and planned default in Gradle 10 (Chapters 2, 3 and 5). 1. Gradle, "What's new in Gradle 9," gradle.org. Source for reproducible archives by default and configuration cache changes in Gradle 9; release date of Gradle 9.0 (31 July 2025) from the Gradle version history (Chapter 3). 1. Develocity, product overview at develocity.ai and announcement "Gradle Technologies is now Develocity" (30 June 2026). Source for supported build tools, product features and the 2026 company rebrand, including the statement that AI has shifted the bottleneck from writing code to reviewing and validating it (Chapters 3 and 9). 1. Gradle, Develocity Build Validation Scripts, github.com/gradle/develocity-build-validation-scripts, including the Gradle experiment descriptions. Source for Table 2 (Chapter 3). 1. Bazel documentation, "Remote Caching," "Remote Execution Overview," "Remote Execution Services," "Debugging Remote Cache Hits for Remote Execution" and "Migrating to Bzlmod," bazel.build (accessed October 2026). Source for the action cache and content-addressable store, cache flags and server options, the list of implementations in Table 3, execution log flags, and the disabling and removal of WORKSPACE (Chapters 2, 4 and 5). 1. Bazel release documentation, bazel.build/release (accessed October 2026). Source for Bazel 9 as the active long-term-support release (Chapter 4). 1. Bazel blog, "Build without the Bytes in Bazel 7" (6 October 2023). Source for the change of default download mode to top-level outputs (Chapters 2 and 4). 1. SLSA, "Supply-chain Levels for Software Artifacts: Requirements," version 1.0, slsa.dev. Source for the Build L3 isolation requirement concerning cache poisoning (Chapter 5). 1. John Micco, "Flaky Tests at Google and How We Mitigate Them," Google Testing Blog, 27 May 2016. Source for flakiness statistics at Google (Chapter 6). 1. Qingzhou Luo, Farah Hariri, Lamyaa Eloussi and Darko Marinov, "An Empirical Analysis of Flaky Tests," Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). Source for the root-cause taxonomy in Table 4 (Chapter 6). 1. Wing Lam, Reed Oei, August Shi, Darko Marinov and Tao Xie, "iDFlakies: A Framework for Detecting and Partially Classifying Flaky Tests," IEEE International Conference on Software Testing, Verification and Validation (ICST 2019) (Chapter 6). 1. Emily Kowalczyk, Karan Nair, Zebao Gao, Leo Silberstein, Teng Long and Atif Memon, "Modeling and Ranking Flaky Tests at Apple," ICSE Software Engineering in Practice (2020) (Chapter 6). 1. Celal Ziftci and Diego Cavalcanti, "De-Flake Your Tests: Automatically Locating Root Causes of Flaky Tests in Code at Google," IEEE International Conference on Software Maintenance and Evolution (ICSME 2020) (Chapter 6). 1. Develocity documentation, "Flaky Test Detection" and "Predictive Test Selection" guides, and Test Distribution documentation, docs.develocity.ai (accessed October 2026). Source for the definition of flaky outcomes, retry mechanisms, selection profiles and staging recommendations, and Test Distribution behaviour (Chapters 6 and 7). 1. Mateusz Machalica, Alex Samylkin, Meredith Porth and Satish Chandra, "Predictive Test Selection," ICSE Software Engineering in Practice (2019); preprint arXiv:1810.05286 (Chapter 7). 1. Atif Memon, Zebao Gao, Bao Nguyen, Sanjeev Dhanda, Eric Nickell, Rob Siemborski and John Micco, "Taming Google-Scale Continuous Testing," ICSE Software Engineering in Practice (2017) (Chapter 7). 1. Sundaram Ananthanarayanan, Masoud Saeida Ardekani, Denis Haenikel, Balaji Varadarajan, Simon Soriano, Dhaval Patel and Ali-Reza Adl-Tabatabai, "Keeping Master Green at Scale," Proceedings of the Fourteenth EuroSys Conference (2019) (Chapter 7). 1. DORA, Accelerate State of DevOps Report 2024, dora.dev. Source for the association of AI adoption with individual gains alongside negative effects on delivery throughput and stability (Chapter 9). Further Reading Titus Winters, Tom Manshreck and Hyrum Wright, Software Engineering at Google (O'Reilly, 2020).** The chapters on build systems, testing and continuous integration explain the reasoning behind artifact-based builds like Bazel and Google's approach to large test suites, from the people who ran them. Abi Noda, Margaret-Anne Storey, Nicole Forsgren and Michaela Greiler, "DevEx: What Actually Drives Productivity," ACM Queue (2023).** A short, readable framework for thinking about developer experience, with feedback loops as one of three core dimensions and practical advice on measuring them. Nicole Forsgren, Jez Humble and Gene Kim, Accelerate (IT Revolution, 2018).** The research foundation for the DORA delivery measures, and the clearest argument that delivery speed and stability go together rather than trade off. Bazel documentation, bazel.build. The pages on remote caching, remote execution, hermeticity and debugging cache hits are unusually good, and the Bzlmod migration guide is essential for anyone still on WORKSPACE. Gradle User Manual, docs.gradle.org, especially "Build Cache" and "Configuration Cache." The authoritative account of how Gradle computes cache keys, how to make custom tasks cacheable and relocatable, and how to adopt the configuration cache. Develocity Build Validation Scripts, github.com/gradle/develocity-build-validation-scripts. The step-by-step experiments for diagnosing cache problems in Gradle and Maven builds, usable as a training exercise as well as a diagnostic. Qingzhou Luo, Farah Hariri, Lamyaa Eloussi and Darko Marinov, "An Empirical Analysis of Flaky Tests" (FSE 2014). The paper that established the standard taxonomy of flaky test causes; still the best place to begin understanding why tests flake. John Micco, "Flaky Tests at Google and How We Mitigate Them," Google Testing Blog (2016). A brief, frank account of flakiness at scale and the mitigations Google used. Mateusz Machalica and colleagues, "Predictive Test Selection" (ICSE-SEIP 2019). A clear description of how a machine-learned test selector was built and evaluated in production, including how it handled flaky tests. SLSA specification, slsa.dev. The supply-chain security framework whose build requirements make explicit why cache isolation and reproducibility matter for security as well as speed. Hashtags: #DeveloperProductivityEngineering #DPE #DeveloperProductivity #DeveloperExperience #DevEx #BuildCaching #RemoteBuildCaching #LocalBuildCache #IncrementalBuilds #FastFeedbackLoops #BuildOptimization #BuildPerformance #Gradle #Bazel #Develocity #HermeticBuilds #ReproducibleBuilds #DeterministicBuilds #ContentAddressableStorage #CacheOptimization #RemoteExecution #ContinuousIntegration #CICD #FlakyTests #TestAutomation #PredictiveTestSelection #DistributedTesting #BuildObservability #InnerDevelopmentLoop #PlatformEngineering

Download the Book (PDF): Introduction Most people meet PyTorch the same way. They copy a training script from a tutorial or a repository, run it, watch a loss number fall, and conclude that they know PyTorch. Then something goes wrong. The validation accuracy is suspiciously high, or the loss turns into nan after six hours, or the model trains well on one GPU and diverges on four, or the exported file that worked on the laptop refuses to load on the server. At that point the copied script stops being useful, because the person running it does not know which of its forty lines are load-bearing and which are habit. This booklet is written for that moment, and ideally for just before it. It is a working manual for people who write PyTorch code for a living or intend to: engineers moving into machine learning, researchers whose prototypes need to survive contact with production, and data scientists who have used higher-level libraries and want to understand what sits underneath. It assumes you can read and write Python comfortably and that you know, at least informally, what a neural network is and what gradient descent does. It does not assume you have read the PyTorch source code, and it will not ask you to. The argument of this book The controlling idea is simple to state. PyTorch is easy to start with because it is ordinary Python executed line by line, and it is hard to master because almost every serious problem you will meet comes from not knowing what that ordinary-looking Python is quietly recording, allocating or assuming on your behalf. A tensor carries a device, a dtype and a memory layout that decide whether an operation is fast, slow or wrong. Every arithmetic operation on a tensor that requires gradients is written onto a tape that autograd will later play backwards. A module silently tracks which tensors are parameters and which are buffers, and it changes behaviour when you call .train() or .eval(). A data loader forks worker processes that each inherit a copy of your dataset object. None of this is hidden in the sense of being secret; all of it is documented. It is hidden in the sense that the code you write does not mention it. The practitioner's skill, then, is to see those implicit contracts while reading explicit code. Once you can, three things follow. You can write custom components, such as datasets, layers, losses and training loops, with confidence instead of by imitation. You can debug quickly, because you know where the small set of usual suspects live. And you can deploy deliberately, because deployment in PyTorch is precisely the act of taking that dynamic, recorded-as-you-go Python program and freezing it into something that can run without Python at all. That last point deserves emphasis because it is where the ecosystem has moved most in the last few years. The flexibility of eager execution, where each line runs immediately and you can drop into a debugger anywhere, is the reason researchers adopted PyTorch. It is also exactly what a production system does not want. A server wants a fixed graph, known input shapes, no interpreter overhead and no surprises. For most of PyTorch's history the bridge between those worlds was TorchScript, and the standard way to put a model behind an HTTP endpoint inside the PyTorch family was TorchServe. Both are covered in this book because both are still found in real codebases and real job descriptions. But the honest picture as of October 2026 is that TorchScript has been formally deprecated in favour of torch.export, with PyTorch 2.14 now surfacing the deprecation warnings to users, and that TorchServe has been placed in limited maintenance, with its GitHub repository archived and its last release, version 0.12.0, dating from September 2024. A guide that taught either as the recommended path for new work would be doing you a disservice. A guide that ignored them would leave you unable to maintain the many systems that still use them. The final two chapters therefore treat both as they are: important, widely deployed, and on their way out, with their successors explained alongside them. How the book is organised The eight chapters move from the inside out. The first three cover the core objects you cannot avoid. Chapter 1 deals with tensors as data structures: shape, dtype, device, strides and the difference between a view and a copy, which is the source of a surprising number of bugs. Chapter 2 takes apart autograd, the automatic differentiation engine, so that loss.backward() stops being a magic incantation and becomes a call whose behaviour you can predict, including when gradients accumulate, when they silently vanish, and how to write your own differentiable operation. Chapter 3 is about nn.Module: how parameters are discovered, how state is saved and restored, and why a model that works in training can misbehave in evaluation. The middle chapters are about getting data in and learning out. Chapter 4 is devoted to the Dataset and DataLoader machinery, with particular attention to writing custom dataset classes for data that does not arrive as a tidy folder of images, and to the multiprocessing behaviour that makes the loader fast and occasionally baffling. Chapter 5 builds a complete training loop by hand, piece by piece, including validation, checkpointing, mixed precision, gradient accumulation and clipping, and explains what each piece protects you from. Chapter 6 is about the things that go wrong: shape errors, exploding losses, silent data leakage, slow input pipelines and memory exhaustion, and the tools PyTorch gives you for finding them, including the profiler and torch.compile. The last two chapters are about leaving the notebook. Chapter 7 explains how a Python model becomes a portable artefact, first through TorchScript's tracing and scripting and then through torch.export, AOTInductor and the other modern routes, with a frank comparison of what each gives up. Chapter 8 covers serving: how TorchServe packages and hosts a model, what you need to know to keep an existing TorchServe deployment safe, and what teams are using instead for new systems. How to read the code Because this is a programming manual, the text includes Python code. The examples are deliberately small. Each is meant to isolate one idea so that you can type it into an interpreter and poke at it, not to serve as a framework you paste into production. They are written against current PyTorch 2.x releases, and where an API has changed in a way that matters, such as the default behaviour of torch.load or the location of the mixed-precision utilities, the text says so. Where an example needs a GPU, it falls back to the CPU so that it still runs on a laptop. A few conventions are worth knowing up front. torch is always imported as torch, the neural network package as nn, and the functional interface as F. Shapes are written as tuples in the order PyTorch stores them, so a batch of 32 colour images of 224 by 224 pixels is (32, 3, 224, 224), batch first, channels next. When the prose says "the device" it means whichever of CPU, CUDA GPU, Apple's MPS backend or another accelerator the tensor lives on. PyTorch now releases on a faster cadence than it once did, roughly every two months during 2026, so some details will drift after publication. The underlying contracts this book concentrates on, such as how autograd records operations, how modules register parameters and how the data loader distributes work, have been stable for years and are the part worth internalising. When a specific version matters, check the release notes for the version you are actually running; torch.__version__ will tell you which that is, and it is a better first debugging step than most people realise. What is left out A short book earns its usefulness by what it declines to cover. There is no survey of architectures here: no chapter on transformers versus convolutional networks, no tour of diffusion models. Those are subjects in their own right, and the techniques in this book apply to all of them. Large-scale distributed training appears only briefly, in the context of what changes in your training loop when you add a second GPU; the mechanics of sharding a model across hundreds of accelerators deserve more space than a booklet can give. The higher-level training libraries, such as PyTorch Lightning, Hugging Face Accelerate and fastai, are mentioned where they are relevant but not taught, because their value is precisely that they write the loop for you, and this book's purpose is to make sure you could write it yourself. Once you can, those libraries become tools you choose rather than crutches you depend on. The aim, in the end, is modest and practical. When you finish, a PyTorch script should no longer look like a sequence of rituals. It should look like a program whose every line you could justify, change or delete with confidence. Chapter 1: Tensors Are Not Just Arrays Every computation in PyTorch begins and ends with a tensor, and most of the confusing behaviour that newcomers meet traces back to a property of the tensor they were not looking at. A tensor looks like a NumPy array with a different import name, and for a while you can treat it that way. But it carries more metadata than an array does, and that metadata decides where the computation runs, how much memory it uses, whether the result can be differentiated, and whether two tensors that look independent are in fact the same block of memory. Learning to read that metadata at a glance is the first practitioner's skill. The four attributes you should always know A tensor has four attributes worth checking before you trust any operation on it: its shape, its dtype, its device, and whether it requires gradients. The quickest way to see them is to print them. import torch x = torch.randn(4, 3) print(x.shape, x.dtype, x.device, x.requires_grad) # torch.Size([4, 3]) torch.float32 cpu False Shape is the size along each dimension. torch.Size is a subclass of Python's tuple, so you can index it, unpack it and compare it to a tuple. Most bugs in model code are shape bugs, and the most dangerous of them are the ones that do not raise an error, because broadcasting quietly made two incompatible shapes compatible. A prediction of shape (32, 1) subtracted from a target of shape (32,) does not fail; it broadcasts into a (32, 32) matrix, and the mean of that matrix is a perfectly finite, perfectly meaningless loss. The model will train against it, and the curve will even go down. The habit that prevents this is to assert shapes at the boundaries of every function you write until you are sure of them, and to call .squeeze(-1) or .view(-1) explicitly rather than hoping dimensions line up. Dtype is the element type. The default floating type is torch.float32, and integer tensors created from Python integers default to torch.int64. The distinction matters more than it does in NumPy, because many PyTorch operations insist on a particular type. Classification targets passed to F.cross_entropy must be integer class indices of type int64 (or probabilities in floating point, a newer option); a float tensor of class labels raises an error. Embedding lookups need integer indices. Conversely, a loss that involves division or a logarithm on an integer tensor will either fail or round in ways you did not want. Lower-precision types, float16 and bfloat16, halve memory and speed up matrix multiplication on modern accelerators, but they have far less range or precision than float32, which is why they are normally used through automatic mixed precision rather than by casting a whole model by hand. Chapter 5 returns to this. The common types and what they are for are set out in Table 1. The ranges are properties of the IEEE 754 and bfloat16 formats themselves, not of PyTorch. Table 1. Common PyTorch dtypes and their typical uses. Dtype Bits Approx. max value Typical use float32 32 3.4e38 Default for weights and activations float16 16 65,504 Mixed-precision compute on GPUs bfloat16 16 3.4e38 Mixed precision with float32's range int64 64 9.2e18 Class labels, indices, token IDs uint8 8 255 Raw image pixels before conversion bool 8 per element n/a Masks The row that surprises people is float16. Its largest representable value is 65,504, so an intermediate value such as a sum of squares or an unnormalised attention score can overflow to infinity in a way that would never happen in float32. bfloat16 keeps float32's exponent range and sacrifices precision instead, which is why it has become the preferred half-precision type on hardware that supports it. Device is where the tensor's memory lives: cpu, cuda:0, mps on Apple silicon, or another backend. Operations between tensors require them to be on the same device, and PyTorch will not move data for you. The rule is to decide the device once, near the top of the program, and pass it down. if torch.cuda.is_available(): device = torch.device("cuda") elif torch.backends.mps.is_available(): device = torch.device("mps") else: device = torch.device("cpu") x = torch.randn(4, 3, device=device) # allocate directly on the device y = torch.zeros(4, 3).to(device) # or move an existing tensor Allocating directly on the target device, as in the first line, avoids a temporary CPU tensor and a copy. Moving with .to(device) returns a new tensor and leaves the original where it was; it does not modify in place. The asymmetry with modules, which .to() does modify in place and also return, catches many people. Writing x.to(device) on a line by itself does nothing useful. Recent versions of PyTorch have also introduced a more general notion of an "accelerator", accessible as torch.accelerator, which abstracts over CUDA, MPS, Intel's XPU and other backends. Code that wants to be portable across hardware vendors can use torch.accelerator.current_accelerator() instead of branching on each backend by name. The explicit branching above remains common and perfectly correct. requires_grad says whether autograd should record operations involving this tensor. It is False for tensors you create yourself and True for the parameters of nn.Module layers. Chapter 2 explains what that recording entails. For now, the point is that this flag spreads: any result computed from a tensor that requires gradients will itself require gradients, and will hold a reference to the computation that produced it. Creating tensors and the cost of conversion There are three common ways to get data into a tensor, and they differ in whether they copy. torch.tensor(data) always copies its input, whether that is a Python list or a NumPy array, and infers the dtype. torch.from_numpy(array) does not copy: the tensor and the array share memory, so writing to one changes the other. torch.as_tensor(data) copies only when it has to, sharing memory with a NumPy array of a compatible type on the CPU and copying otherwise. import numpy as np a = np.zeros(3, dtype=np.float32) t = torch.from_numpy(a) t[0] = 5.0 print(a) # [5. 0. 0.] the array changed too Memory sharing is a feature when you are converting large arrays in a data pipeline and want to avoid a copy. It is a hazard when you hold on to the array and modify it later, perhaps by reusing a buffer for the next batch, without realising the tensor will see the change. In the other direction, tensor.numpy() shares memory with a CPU tensor and raises an error if the tensor requires gradients or lives on a GPU. The idiom tensor.detach().cpu().numpy() exists to handle all three conditions in one line, and it is worth understanding each step rather than typing it as a charm: detach() produces a tensor outside the autograd graph, cpu() copies to host memory if needed, and numpy() exposes it as an array. Python floats are a quieter source of trouble. torch.tensor(0.1) creates a float32 tensor, but NumPy's default float is float64, so torch.from_numpy(np.array([0.1])) produces a float64 tensor. Mix that with a model's float32 weights and you will get an error about mismatched dtypes, or, in operations that promote types, a silent upgrade to double precision that runs far more slowly on a GPU. Data pipelines that start from NumPy or pandas should cast explicitly, typically with .float() for features and .long() for labels. Views, strides and contiguity The concept that separates people who are comfortable with tensors from those who are not is the view. A tensor is a description of how to read a block of memory, called its storage. The description consists of an offset into the storage, a shape and a stride for each dimension, which says how many elements to skip in storage to move one step along that dimension. A freshly created (3, 4) tensor has strides (4, 1): moving one row skips four elements, moving one column skips one. Many operations produce a new tensor that shares the original storage with a different description. Transposing a (3, 4) tensor gives a (4, 3) tensor with strides (1, 4) and no data movement at all. Slicing, view, expand, permute, narrow, squeeze, unsqueeze and t() all behave this way. They are fast because they copy nothing, and they are dangerous for the same reason. x = torch.arange(12).view(3, 4) row = x[0] # a view of the first row row[0] = 100 print(x[0, 0]) # tensor(100) xt = x.t() print(xt.stride(), xt.is_contiguous()) # (1, 4) False A tensor is contiguous when its elements are laid out in storage in the same order you would visit them by iterating over its last dimension fastest. After a transpose it no longer is. Some operations, most notably view, require a compatible memory layout and will raise an error on a non-contiguous tensor; the message suggests using reshape instead. The difference is that reshape returns a view when it can and silently copies when it cannot, while view refuses to copy. That makes view the better choice when you want a guarantee that no memory is being duplicated, and reshape the better choice when you simply want the right shape. Calling .contiguous() forces a copy into the standard layout when you need one. Contiguity also matters for speed. Many GPU kernels are written for contiguous inputs, and a non-contiguous tensor may either be copied internally or processed with a slower code path. Images are a notable example. Convolutions on modern NVIDIA hardware are often faster in the "channels last" memory format, where the channel dimension varies fastest in memory, even though the logical shape remains (N, C, H, W). PyTorch supports this through x.to(memory_format=torch.channels_last) and the same call on a model, and it is one of the cheapest performance wins available for convolutional networks. The logical shape you print does not change at all; only the strides do. This is the clearest illustration of why shape alone does not tell you everything about a tensor. In-place operations and their price Methods whose names end in an underscore, such as add_, mul_, zero_ and clamp_, modify the tensor they are called on. So do indexed assignments like x[mask] = 0 and augmented operators like += on tensors. In-place operations save memory, which can matter on a GPU, and they are used throughout PyTorch's own optimizers. In model code, however, they collide with autograd. When a tensor's value is needed to compute a gradient later, overwriting it in place would make the gradient wrong. Autograd tracks this with a version counter on each tensor, and if it notices during the backward pass that a saved tensor has been modified, it raises an error that mentions "one of the variables needed for gradient computation has been modified by an inplace operation". That error is annoying but protective: it prevents a silently incorrect gradient. The practical guidance is to avoid in-place operations on tensors that participate in a forward pass you intend to differentiate unless you have a measured memory reason, and to prefer the out-of-place form (x = x + 1 rather than x += 1) in custom layers. The inplace=True option on activation functions like nn.ReLU is a holdover from an era of tighter memory budgets and is rarely worth the risk in new code. Broadcasting and reductions Broadcasting is the rule that lets you add a (32, 10) tensor and a (10,) tensor: dimensions are compared from the right, and a dimension of size one, or a missing dimension, is stretched to match. It removes a great deal of explicit looping and tiling, and PyTorch implements it without copying data by using a stride of zero along the stretched dimension. It is also the mechanism behind the silent (32, 1) versus (32,) bug described earlier. When you write an elementwise operation between tensors of different shapes, say to yourself which dimensions you expect to broadcast. If you cannot, the shapes are probably wrong. Reductions such as sum, mean, max and argmax collapse a dimension. Passing dim chooses which, and keepdim=True keeps the reduced dimension with size one, which is usually what you want if the result will be broadcast back against the original. Normalising rows of a matrix is the canonical example. x = torch.randn(5, 3) row_sums = x.sum(dim=1, keepdim=True) # shape (5, 1) normalised = x / row_sums # broadcasts across columns Without keepdim=True, row_sums would have shape (5,), which broadcasts against the last dimension of size 3 and fails, or, in the unlucky case where the sizes happen to match, succeeds with the wrong meaning. It is worth noting that max along a dimension returns both values and indices as a named tuple, whereas amax returns only the values; the former trips up people who expect a single tensor. Randomness and reproducibility Random tensors come from a generator whose state you can seed with torch.manual_seed. Seeding makes most CPU code reproducible from run to run. It is not, on its own, enough to make GPU code reproducible, because some CUDA kernels use non-deterministic algorithms, such as atomic additions whose order varies, for speed. PyTorch exposes torch.use_deterministic_algorithms(True), which makes operations either use a deterministic implementation or raise an error if none exists, and some CUDA operations additionally require an environment variable, CUBLAS_WORKSPACE_CONFIG, to be set; the error message tells you the value. Determinism usually costs some speed, so the normal practice is to enable it when debugging or when a result must be exactly reproducible, and to accept small run-to-run variation otherwise. Data loading adds another source of randomness. Each worker process in a DataLoader has its own random state, and if your dataset applies random augmentations using NumPy, older versions of PyTorch could give every worker the same NumPy seed, producing duplicated augmentations. Modern PyTorch seeds Python's and PyTorch's generators per worker automatically, and recent NumPy usage patterns help, but if your augmentation code uses its own generator you should seed it in a worker_init_fn. Chapter 4 shows how. Reading a tensor before trusting it All of this reduces to a habit. Before you trust a tensor that has come from somewhere you did not write, such as a data loader, a pretrained model's output or a library function, look at its shape, dtype, device and gradient flag, and check its range with x.min(), x.max() and x.isnan().any(). Images that should be normalised floats sometimes arrive as uint8 values from 0 to 255. Labels that should be indices sometimes arrive as floats. Masks that should be boolean sometimes arrive as integers, and indexing with an integer tensor does something entirely different from indexing with a boolean one: the former gathers elements at those positions, the latter selects elements where the mask is true. A minute spent printing these attributes saves hours spent interpreting a loss curve that was never measuring what you thought. The tensor is the data structure. What makes PyTorch more than an array library is what happens when a tensor requires gradients, and that is the subject of the next chapter. Chapter 2: Autograd Without the Magic The line loss.backward() is the most consequential and least examined call in any PyTorch program. It computes the gradient of a scalar with respect to every tensor that contributed to it and required gradients, and it does so for any program you can write in Python, including loops, conditionals and recursion. That generality is why PyTorch felt so different when it appeared in 2016 and 2017: frameworks of the time typically required you to declare a fixed computation graph before running it, whereas PyTorch let you write ordinary code and differentiate whatever actually happened. Understanding how it achieves that is the difference between using autograd and being used by it. The tape that builds itself Autograd is reverse-mode automatic differentiation implemented with a dynamically built graph. Whenever an operation runs on a tensor that requires gradients, PyTorch does two things: it computes the result as usual, and it creates a node recording which backward function to call and which inputs, or saved intermediate values, that function will need. The result tensor gets a grad_fn attribute pointing at that node. Tensors you created directly, called leaf tensors, have no grad_fn; they are the places where gradients finally land. import torch w = torch.tensor([2.0, -1.0], requires_grad=True) # leaf x = torch.tensor([3.0, 4.0]) # data, no grad y = (w * x).sum() # non-leaf print(y.grad_fn) # print(y.grad_fn.next_functions) # ((, 0),) y.backward() print(w.grad) # tensor([3., 4.]) When you call y.backward(), the engine walks this graph from y back to the leaves in reverse topological order, calling each node's backward function with the incoming gradient and passing the outputs on to the nodes that produced its inputs. For y = sum(w * x), the gradient with respect to w is simply x, which is what appears in w.grad. Three properties of this design explain most of autograd's behaviour in practice. First, the graph is rebuilt on every forward pass. There is no persistent model graph; there is only the record of what happened this time. That is why a model can take a different branch for different inputs, and why the gradient correctly reflects only the branch that was taken. It is also why debugging is straightforward: the forward pass is just Python, and you can print or step through it. Second, by default the graph is freed as soon as backward() finishes. The saved intermediate values, which can be most of the memory a training step uses, are released. Calling backward() a second time on the same graph raises an error saying the graph has already been freed. If you genuinely need to backpropagate through the same graph twice, for example to compute two losses separately from a shared trunk, you pass retain_graph=True to the first call. In most code, needing this is a sign that the losses should simply be added together before one backward call. Third, gradients accumulate. Each backward() adds into the .grad attribute of each leaf rather than overwriting it. This is deliberate. It is what lets you accumulate gradients across several small batches to simulate a larger one, and it makes the sum of losses and the sum of their separate backward passes equivalent. But it means that if you forget to clear gradients between optimisation steps, every step uses the sum of all previous gradients, and training goes badly in a way that does not raise any error. This is why every training loop calls optimizer.zero_grad(). Since PyTorch 2.0 that call sets gradients to None by default rather than filling them with zeros, which saves a memory write and lets the next backward pass allocate fresh. Code that reads .grad immediately after zeroing must allow for None. Only scalars, unless you say otherwise backward() with no arguments only works on a scalar, a tensor with a single element. That is because reverse-mode differentiation computes a vector-Jacobian product: given the gradient of some final scalar with respect to an output, it produces the gradient with respect to the inputs. When the output is the scalar itself, the starting gradient is implicitly 1. When the output is a vector, you must supply that starting vector yourself through the gradient argument, and what you get back is the product of that vector with the Jacobian, not the full Jacobian. People occasionally call .backward() on a per-example loss vector and are surprised by the error. The fix is almost always to reduce it with .mean() or .sum() first, which is what the built-in loss functions do by default through their reduction argument. The choice between mean and sum is not cosmetic. A mean makes the gradient magnitude independent of batch size, so the learning rate does not need to change when the batch does. A sum makes it proportional to batch size. Most published hyperparameters assume a mean. When you really do want a full Jacobian or Hessian, the torch.func module, which grew out of the functorch project, provides composable function transforms such as jacrev, jacfwd, hessian, grad and vmap. These work in a functional style on pure functions rather than through the tape, and they are the right tool for per-example gradients, meta-learning and scientific computing problems where higher-order derivatives are central. Switching recording off Recording operations costs memory and time. When you do not need gradients, which is during evaluation, inference and any bookkeeping arithmetic on model outputs, you should switch recording off. There are three ways to do it, with different strengths. tensor.detach() returns a new tensor that shares data with the original but is cut off from the graph. Gradients will not flow back through it. Use it when you want to use a value computed by the model as a constant, for example a target produced by a slowly updated copy of the network, or when you want to log a value without keeping its graph alive. Logging is the classic memory leak in PyTorch training code: appending loss to a Python list for plotting keeps every iteration's entire graph in memory. Appending loss.item(), which returns a Python float, or loss.detach() avoids it. torch.no_grad() is a context manager under which no operations are recorded at all. Results computed inside it have requires_grad=False. It is what every validation loop should be wrapped in. torch.inference_mode() is a stricter, faster version of no_grad. Tensors created inside it are marked as inference tensors and cannot later be used in operations that autograd records, which allows PyTorch to skip some bookkeeping, such as version counters. It is the recommended choice for pure inference code. The restriction rarely matters in practice, but if you produce a tensor in inference mode and later try to use it in training you will get an error, and no_grad is the more forgiving alternative. model.eval() with torch.inference_mode(): logits = model(batch) preds = logits.argmax(dim=1) A related tool is requires_grad_(False) on parameters, which freezes them. Freezing the early layers of a pretrained network while fine-tuning the last few is a common pattern, and it saves both computation and optimizer memory, provided you also exclude the frozen parameters from the optimizer or at least understand that they will receive no gradient. When gradients go missing The most common autograd bug is not a wrong gradient but an absent one: a parameter whose .grad is None after backward(), or a model that does not learn because some part of it is disconnected from the loss. There is a short list of causes, and checking it systematically is faster than staring at the code. The first is an accidental break in the graph. Any trip through NumPy, a Python float or a non-differentiable operation severs the connection. Calling .item(), .numpy() or .tolist() in the middle of a forward pass produces a value autograd cannot see through. So does creating a new tensor with torch.tensor(...) from values computed by the model, which copies the data and discards the history. Integer conversion and argmax have no useful gradient either; if your loss depends on a hard choice, gradients will not flow through that choice, which is why techniques such as the Gumbel-softmax relaxation or the straight-through estimator exist. The second is a detach() or no_grad() in the wrong place, often inherited from code that was originally written for inference. The third is a parameter that was never registered with the model, so the optimizer never received it. That is a module problem, and Chapter 3 covers it. The fourth is a parameter that is genuinely unused on this forward pass, for example the weights of a branch that was not taken. Its gradient stays None, which is correct but may confuse distributed training wrappers that expect every parameter to receive a gradient. DistributedDataParallel has a find_unused_parameters option for exactly this situation. A quick diagnostic after the first backward pass is to list the parameters with no gradient: missing = [n for n, p in model.named_parameters() if p.requires_grad and p.grad is None] print(missing) If that list is not what you expect, the graph is broken somewhere between those parameters and the loss. Gradients that are present but wrong When gradients exist but training misbehaves, the usual suspects are numerical. Gradients can explode, growing by orders of magnitude across layers until a step throws the weights into a region where activations overflow, after which the loss becomes inf and then nan. They can vanish, shrinking until early layers effectively stop learning. Both were more common before normalisation layers, residual connections and careful initialisation became standard, but they still appear in recurrent models, very deep networks and anything trained in reduced precision. Autograd offers two tools for these situations. torch.autograd.set_detect_anomaly(True) makes the engine check every backward function's output for nan and, when one appears, raise an error with a traceback pointing at the forward operation that created the failing node. It slows training considerably and should be switched on only while hunting a problem. torch.autograd.gradcheck compares the analytical gradient of a function against a numerical estimate from finite differences. It is the way to test a custom backward function, and it should be run with double-precision inputs, because finite differences in float32 are too noisy to be meaningful. For explosions in normal training, the standard remedy is gradient clipping, applied between backward() and optimizer.step(), which Chapter 5 covers. Clipping is a guard rail, not a cure. If gradients need to be clipped on most steps, the learning rate, initialisation or architecture usually needs attention. Writing your own differentiable operation Sometimes you need an operation whose gradient autograd cannot derive, or should not. The operation might call an external library, use a numerically unstable formula whose gradient you can write more stably by hand, or need a deliberately altered gradient, such as the straight-through estimator, which passes the incoming gradient through a rounding step as though it were the identity. For these cases you subclass torch.autograd.Function and define forward and backward static methods. class RoundSTE(torch.autograd.Function): @staticmethod def forward(ctx, x): return torch.round(x) @staticmethod def backward(ctx, grad_output): return grad_output # pretend round() was the identity x = torch.tensor([0.3, 1.7], requires_grad=True) RoundSTE.apply(x).sum().backward() print(x.grad) # tensor([1., 1.]) The ctx object is how forward passes information to backward. Tensors needed for the gradient are stored with ctx.save_for_backward(...) and retrieved from ctx.saved_tensors, which lets autograd check that they were not modified in place in the meantime. Non-tensor values can be stored as ordinary attributes on ctx. backward must return one gradient per input to forward, in order, with None for inputs that do not need one, such as integer arguments or configuration flags. You call the function through .apply, never by instantiating it. A more principled example is a numerically stable operation. Suppose you need log(1 + exp(x)), the softplus function. Computing it naively overflows for large x. PyTorch provides F.softplus, which handles this, but the same reasoning applies to functions it does not provide: implement the stable forward pass, save what the derivative needs, and write the derivative directly. class StableSoftplus(torch.autograd.Function): @staticmethod def forward(ctx, x): out = torch.where(x > 20, x, torch.log1p(torch.exp(x))) ctx.save_for_backward(x) return out @staticmethod def backward(ctx, grad_output): (x,) = ctx.saved_tensors return grad_output * torch.sigmoid(x) x = torch.randn(5, dtype=torch.double, requires_grad=True) print(torch.autograd.gradcheck(StableSoftplus.apply, (x,))) # True The gradcheck call at the end is the important habit. A hand-written backward function is a place where a sign error or a missing factor can hide for months, and checking it against finite differences takes one line. Note also that torch.where evaluates both branches. In the forward pass that is harmless here because the overflowing branch is discarded, but if a discarded branch can produce nan and you let autograd differentiate through torch.where itself, the nan can leak into the gradient. That is a well-known trap and one of the reasons hand-written backward functions are sometimes necessary. Custom functions interact with the compiler and export machinery described in Chapter 7. torch.compile can generally trace through them. For operations implemented outside PyTorch, such as a call into a C++ library or a hand-written kernel, the modern route is the custom operator API in torch.library, which registers the operation together with its gradient formula and a "fake" implementation describing output shapes, so that compilers and exporters can reason about it without running it. If you expect to deploy a model that uses a custom operation, it is worth reading that documentation before writing the operation rather than after. Gradients you do not want: hooks and checkpointing Two further autograd features are worth knowing because they solve problems that come up repeatedly in practice. Hooks let you observe or modify gradients as they flow. tensor.register_hook(fn) calls fn with the gradient of that tensor during the backward pass, and if fn returns a tensor, that replaces the gradient. Module-level hooks, covered in Chapter 3, do the same for a layer's inputs and outputs. Hooks are useful for logging gradient norms per layer, for implementing techniques such as gradient reversal in domain adaptation, and for debugging where nan values first appear in the backward pass. Activation checkpointing addresses memory. A forward pass saves intermediate activations for use in backward, and for deep models those saved tensors dominate memory consumption. torch.utils.checkpoint.checkpoint wraps a segment of the forward computation so that its intermediate activations are not saved; during backward the segment is run again to regenerate them. You trade extra computation, roughly one more forward pass through the checkpointed segments, for a large reduction in memory, often enough to double the batch size or fit a model that otherwise would not. The current recommendation is to pass use_reentrant=False, which selects the newer implementation that supports more features and handles edge cases better; recent versions warn if you omit the argument. Autograd, in short, is not magic. It is a recorder that writes down what you did and a player that replays it backwards. Nearly every practical question about gradients can be answered by asking what was recorded, what was saved, and what happened to it before the replay. Chapter 3: Modules, Parameters and State Autograd will differentiate any function of tensors, but a neural network is more than a function. It has weights that persist between calls, it has statistics such as the running mean of a batch normalisation layer that must be saved with the weights but never trained, it behaves differently during training and evaluation, and it is assembled from smaller networks that themselves have all of these properties. nn.Module is PyTorch's answer to that bookkeeping problem. Most of what it does happens through attribute assignment, which is convenient and, once again, invisible. What happens when you assign an attribute A minimal module subclasses nn.Module, calls the parent constructor, creates its layers in init and describes the computation in forward. import torch from torch import nn import torch.nn.functional as F class MLP(nn.Module): def init(self, d_in, d_hidden, d_out, p_drop=0.1): super().__init__() self.fc1 = nn.Linear(d_in, d_hidden) self.fc2 = nn.Linear(d_hidden, d_out) self.drop = nn.Dropout(p_drop) def forward(self, x): x = F.relu(self.fc1(x)) x = self.drop(x) return self.fc2(x) model = MLP(20, 64, 3) print(sum(p.numel() for p in model.parameters())) # 1539 The interesting work happens on the line self.fc1 = nn.Linear(...). nn.Module overrides setattr, so assigning a module to an attribute registers it as a submodule, and assigning an nn.Parameter registers it as a parameter. Everything else, such as integers, strings and plain tensors, is stored as an ordinary attribute and ignored by the machinery. Calling model.parameters() then walks the tree of registered submodules and yields every registered parameter. That is how the optimizer finds the weights it must update, how .to(device) knows what to move, and how state_dict() knows what to save. This mechanism has a few sharp edges that produce some of the most common bugs in PyTorch code. The first is forgetting super().__init__(), or calling it after assigning layers. The registration dictionaries are created by the parent constructor, so assigning a submodule before they exist raises an error. The error message is clear, and you will make this mistake at most twice. The second is storing submodules in a plain Python list or dictionary. A list of layers assigned as self.layers = [nn.Linear(...) for in range(4)] is just a list attribute. The layers inside it are invisible to parameters(), so the optimizer never updates them, .to(device) never moves them, and statedict() never saves them. The model may still run, if the layers happen to be on the right device, and will simply fail to learn those weights. The fix is nn.ModuleList or nn.ModuleDict, which are containers that register their contents. The same applies to parameters: a list of tensors must be an nn.ParameterList. The third is creating a tensor that should be a parameter without wrapping it. A plain torch.zeros(10, requires_grad=True) assigned to self.scale will receive gradients but will not be in parameters(), so the optimizer ignores it. Wrapping it as nn.Parameter(torch.zeros(10)) registers it and sets requires_grad=True. The quickest check on any custom module is to print [n for n, in model.namedparameters()] and confirm that every weight you expect is there. Parameters, buffers and the state dictionary Some tensors belong to a model's state without being trained. Batch normalisation keeps running estimates of the mean and variance of its inputs, which it updates during training by a moving average and uses during evaluation. Positional encodings in a transformer are often fixed tensors computed once. A model that counts how many batches it has seen needs a counter. These should move with the model when it changes device and should be saved with its weights, but the optimizer must leave them alone. PyTorch calls these buffers, and you create them with register_buffer. class RunningNorm(nn.Module): def init(self, dim, momentum=0.1): super().__init__() self.momentum = momentum self.register_buffer("mean", torch.zeros(dim)) self.register_buffer("var", torch.ones(dim)) def forward(self, x): if self.training: m, v = x.mean(0), x.var(0, unbiased=False) with torch.no_grad(): self.mean.lerp_(m, self.momentum) self.var.lerp_(v, self.momentum) else: m, v = self.mean, self.var return (x - m) / torch.sqrt(v + 1e-5) Buffers are included in state_dict() by default. Passing persistent=False to register_buffer keeps a buffer on the right device without saving it, which is appropriate for values that can be recomputed, such as a precomputed attention mask. The distinction between parameters, buffers and plain attributes is worth fixing firmly in mind, and Table 2 sets it out. Table 2. How nn.Module treats the things you assign to it. Kind Created with Optimised Moved by .to() Saved in state_dict Parameter nn.Parameter(...) Yes Yes Yes Buffer register_buffer(...) No Yes Yes Non-persistent buffer register_buffer(..., persistent=False) No Yes No Plain tensor attribute self.t = torch.zeros(...) No No No Submodule nn.Module instance Its parameters Yes Yes, with prefix The state_dict is an ordered dictionary mapping dotted names, such as fc1.weight or encoder.layers.3.norm.running_mean, to tensors. It is the format in which PyTorch models are almost always saved, and it is deliberately just data: no code, no class definitions. To restore a model, you construct it in code and then call load_state_dict, which matches names and copies values. By default the match is strict; any missing or unexpected key raises an error. Passing strict=False loads what matches and returns a report of the keys that did not, which is useful when adapting a pretrained network but dangerous as a habit, because a typo in a layer name then silently leaves that layer at its random initialisation. If you use strict=False, always print the returned missing and unexpected keys. Saving and loading safely The standard idiom saves the state dictionary, not the model object. torch.save(model.state_dict(), "model.pt") model = MLP(20, 64, 3) state = torch.load("model.pt", map_location="cpu") model.load_state_dict(state) Saving the whole object with torch.save(model, ...) also works, but it pickles a reference to the class, so loading requires the exact same class to be importable from the same module path. Renaming a file or refactoring a package then breaks every old checkpoint. Saving state dictionaries avoids this. The map_location argument controls where tensors are loaded. Without it, tensors saved from a GPU are restored to the same GPU index, which fails on a machine without one. Loading to CPU first and then moving the model is the portable pattern. There is a security dimension that every practitioner should understand. torch.save uses Python's pickle format, and unpickling arbitrary data can execute arbitrary code. For years, torch.load would happily run whatever a malicious checkpoint contained. PyTorch 2.6 changed the default of torch.load's weights_only argument to True, which restricts unpickling to tensors, primitive types and a small allow-list of safe classes. Ordinary state dictionaries load fine under this restriction. Checkpoints that contain arbitrary Python objects, such as a custom configuration class or an optimizer with unusual state, may now fail to load with an error naming the class that was rejected. The right response is usually to add that specific class to the allow-list with torch.serialization.add_safe_globals, or to restructure the checkpoint to hold only plain data. Passing weights_only=False restores the old behaviour, and you should only do that for files you created yourself or fully trust. An alternative that avoids pickle altogether is the safetensors format, developed by Hugging Face, which stores only tensors and metadata and is widely used for distributing model weights. PyTorch 2.13 added native support in torch.load for reading .safetensors files; on earlier versions the safetensors library provides save_file and load_file functions. For any model weights you download from the internet, a format that cannot execute code is the sensible default. Training mode and evaluation mode Every module has a boolean training attribute. model.train() sets it to True on the module and all its submodules; model.eval() sets it to False. The flag changes the behaviour of a small number of layers, but they are layers that appear in almost every model. Dropout randomly zeroes elements during training and scales the survivors so the expected value is unchanged; in evaluation mode it does nothing. Batch normalisation normalises with the statistics of the current batch and updates its running estimates during training; in evaluation mode it uses the stored running estimates and does not update them. Forgetting to call model.eval() before validation therefore gives you noisy, pessimistic accuracy from dropout and, worse, corrupts the batch normalisation statistics with validation data. Forgetting to call model.train() afterwards means the rest of training runs with dropout disabled and batch normalisation frozen, which usually shows up as a model that suddenly overfits or stops improving after the first validation run. It is essential to separate two ideas that are often confused. model.eval() changes layer behaviour; it does not turn off gradient recording. torch.no_grad() turns off gradient recording; it does not change layer behaviour. Validation needs both. Inference in production needs both. Fine-tuning with frozen batch normalisation statistics, a common technique when batches are small, needs the model in training mode except for the normalisation layers, which you can set to evaluation mode individually after calling model.train(). Batch normalisation deserves one more warning. Its statistics are computed per batch, so with very small batches, which happen when large models or high-resolution inputs leave room for only two or four examples per GPU, the estimates become noisy and training suffers. Group normalisation and layer normalisation do not depend on the batch and are the usual substitutes. In multi-GPU training, ordinary batch normalisation computes statistics separately on each device; nn.SyncBatchNorm synchronises them at a communication cost. Initialisation, composition and inspection PyTorch's built-in layers initialise their weights with sensible defaults, generally variants of the schemes proposed by Glorot and Bengio and by He and colleagues, scaled to keep the variance of activations roughly constant across layers. You rarely need to override them. When you do, perhaps because a paper specifies an initialisation or because you want the final layer of a classifier to start near zero, the idiom is to write a function and apply it to every submodule with model.apply(fn), using the functions in nn.init inside torch.no_grad() semantics, which those functions handle for you. def init_weights(m): if isinstance(m, nn.Linear): nn.init.kaiming_normal_(m.weight, nonlinearity="relu") nn.init.zeros_(m.bias) model.apply(init_weights) For composing layers, nn.Sequential chains modules whose outputs feed directly into the next, and it is the right choice for simple stacks. As soon as a model needs skip connections, multiple inputs or conditional logic, write a custom forward. There is no penalty for doing so; a custom module is as fast as a Sequential one. The forward method should never be called directly. Call the module itself, as model(x), which invokes call. That wrapper runs any registered hooks before and after forward, and some features, including certain distributed training wrappers and compile-time instrumentation, depend on it. Calling model.forward(x) skips them silently. Hooks, mentioned in Chapter 2 for tensors, also exist at module level. register_forward_hook calls a function with the module, its inputs and its output every time it runs, which is the cleanest way to extract intermediate features from a pretrained network without modifying its code. features = {} def save_output(name): def hook(module, inputs, output): features[name] = output.detach() return hook handle = model.fc1.register_forward_hook(save_output("fc1")) _ = model(torch.randn(8, 20)) print(features["fc1"].shape) # torch.Size([8, 64]) handle.remove() Keeping the returned handle and removing the hook when finished matters; hooks persist on the module and accumulate if registered repeatedly, a subtle leak in notebooks where cells are rerun. For inspection, print(model) shows the module tree, named_modules() walks it, and named_parameters() lists every weight with its full dotted name. These names are what you use to build parameter groups for the optimizer, for example to exclude biases and normalisation weights from weight decay, which is standard practice in transformer training. decay, no_decay = [], [] for name, p in model.named_parameters(): if not p.requires_grad: continue (no_decay if p.ndim < 2 else decay).append(p) optimizer = torch.optim.AdamW([ {"params": decay, "weight_decay": 0.05}, {"params": no_decay, "weight_decay": 0.0}, ], lr=3e-4) The heuristic of excluding one-dimensional parameters catches biases and the scale and shift parameters of normalisation layers in one rule. A worked example: wrapping a pretrained layer The ideas in this chapter come together in a pattern that has become common in practice: adding a small trainable component to a large frozen one. Low-rank adaptation, introduced by Hu and colleagues in 2021 and usually called LoRA, fine-tunes a pretrained linear layer by freezing its weight matrix and learning a low-rank update to it, expressed as the product of two thin matrices. The number of trainable parameters falls by orders of magnitude, and so does the memory needed for gradients and optimizer state. Production implementations live in libraries such as Hugging Face's PEFT, but writing a minimal version by hand is an excellent exercise in module mechanics, because every line depends on something covered above. import math import torch from torch import nn class LoRALinear(nn.Module): def init(self, base: nn.Linear, rank=8, alpha=16): super().__init__() self.base = base # registered submodule self.base.requires_grad_(False) # frozen self.A = nn.Parameter(torch.empty(rank, base.in_features)) self.B = nn.Parameter(torch.zeros(base.out_features, rank)) nn.init.kaiming_uniform_(self.A, a=math.sqrt(5)) self.scale = alpha / rank # plain attribute def forward(self, x): return self.base(x) + (x @ self.A.t() @ self.B.t()) * self.scale layer = LoRALinear(nn.Linear(512, 512), rank=8) print([n for n, p in layer.named_parameters() if p.requires_grad]) # ['A', 'B'] print(sum(p.numel() for p in layer.parameters() if p.requires_grad)) # 8192 Assigning the pretrained layer to self.base registers it as a submodule, so its weights move with the wrapper when it is sent to a device and appear in its state dictionary under the prefix base.. Calling requires_grad_(False) on it freezes those weights, so they receive no gradients and, if the optimizer is built only from trainable parameters, occupy no optimizer state. The two new matrices are wrapped in nn.Parameter so that they are registered and trainable. The scaling factor is a plain Python float, which is correct, since it is configuration rather than state. Initialising B to zeros means the product B @ A starts at zero, so at the beginning of fine-tuning the wrapped layer computes exactly what the pretrained layer did; training departs from the pretrained behaviour gradually rather than with a random jolt. The parameter count confirms the saving: the base layer has 262,656 parameters, while the adapter trains 8,192. The optimizer should then be built from the trainable parameters only, with torch.optim.AdamW(p for p in model.parameters() if p.requires_grad). Passing every parameter would also work, since the frozen ones never receive gradients and are skipped, but filtering makes the intent explicit and guards against an accidental unfreeze. Two further module techniques make the adapter practical. Because only the adapter is trained, only the adapter needs saving; filtering the state dictionary down to the keys that contain the adapter's parameter names produces a checkpoint measured in kilobytes rather than gigabytes, which can be loaded on top of the original pretrained weights with load_state_dict(..., strict=False), checking the reported missing keys to confirm that only base weights were absent. And for deployment, the low-rank update can be folded into the base weight once training is finished, so that inference pays no extra cost. @torch.no_grad() def merge(lora: LoRALinear) -> nn.Linear: lora.base.weight += (lora.B @ lora.A) * lora.scale return lora.base The no_grad decorator is required because the update modifies a parameter in place; without it, autograd would object, or, worse, record the operation. After merging, the returned nn.Linear is an ordinary layer that produces the same outputs as the adapter-wrapped version, to within floating-point rounding, and can be exported like any other. Moving a model, and what moves with it model.to(device) moves every parameter and buffer in place and returns the model. It also converts dtypes, so model.to(torch.bfloat16) casts all floating-point parameters. Two ordering rules follow from the fact that optimizers hold references to parameter tensors. Move the model to its device before creating the optimizer, so that the optimizer holds references to the device tensors from the start. And when loading a checkpoint that includes optimizer state, load the model weights, move the model, create the optimizer, and only then load the optimizer state, which PyTorch moves to match the parameters. For very large models, even constructing them on the CPU can exhaust memory, because every layer allocates and initialises its weights before anything is moved. PyTorch offers the "meta" device for this: a model created under torch.device("meta") has parameters with shapes and dtypes but no storage. It can then be materialised directly on the target device with model.to_empty(device=...) and filled from a checkpoint with load_state_dict(..., assign=True). This is how most large-model loading utilities work under the hood, and it is worth knowing exists even if you never need it for the models you train yourself. A module, then, is a container that keeps track of three kinds of state, moves them together, saves them together and switches some of them between two modes. Most bugs in custom modules come from state that did not get registered, state that was saved without the code to interpret it, or a mode that was not switched. With that understood, the next problem is feeding the module data. Chapter 4: Custom Datasets and the DataLoader Tutorials tend to begin with a dataset that someone else has already prepared: a folder of images sorted into subfolders by class, or a built-in loader that downloads a benchmark and hands back tensors. Real projects rarely start that way. The images are listed in a CSV file with labels from a separate annotation export, some of the paths are wrong, the text is in a line-delimited JSON file too large for memory, the audio clips are of different lengths, and the validation split must be grouped by patient so that no person appears in both. The model code may be fifty lines; the data code is often several hundred, and it is where many projects quietly lose most of their accuracy and much of their speed. PyTorch's data utilities divide the problem cleanly. A Dataset knows how to produce a single example. A Sampler decides the order in which examples are requested. A DataLoader requests examples, possibly in parallel worker processes, groups them into batches with a collate function, and hands those batches to the training loop. Understanding where each responsibility lives lets you write custom components that fit the machinery instead of fighting it. The map-style Dataset The most common kind of dataset is map-style: an object that supports len() and integer indexing. Subclass torch.utils.data.Dataset and implement len and getitem. Everything else is optional. from pathlib import Path import csv import torch from torch.utils.data import Dataset from PIL import Image class LabelledImages(Dataset): def init(self, csv_path, image_root, transform=None): self.root = Path(image_root) self.transform = transform with open(csv_path, newline="") as f: rows = list(csv.DictReader(f)) self.classes = sorted({r["label"] for r in rows}) self.class_to_idx = {c: i for i, c in enumerate(self.classes)} self.items = [(r["filename"], self.class_to_idx[r["label"]]) for r in rows] def len(self): return len(self.items) def getitem(self, idx): filename, label = self.items[idx] image = Image.open(self.root / filename).convert("RGB") if self.transform is not None: image = self.transform(image) return image, label Several design choices in this class are deliberate and generalise to almost any custom dataset. The constructor does the cheap, global work: reading the index file, building the class mapping and storing a list of lightweight records. It does not load images. Loading every image in init would take minutes for a large dataset, consume memory proportional to its size, and, as explained below, be duplicated in every worker process. The expensive work, decoding a file and applying transforms, happens lazily in getitem, one example at a time, which is exactly the unit of work the loader parallelises. The class mapping is built from a sorted set, so it is deterministic. Building it in the order labels first appear in the file means that a reordered CSV silently permutes your class indices, and a model trained on one ordering will produce nonsense when evaluated with another. Saving self.classes alongside the model checkpoint is a good habit for the same reason; a deployed classifier is useless without the list that maps its output indices back to names. The convert("RGB") call guards against a common surprise. Real image collections contain greyscale images, images with an alpha channel and palette images. Without the conversion, most examples produce three-channel tensors and a few produce one or four, and the batch fails to stack with an error that appears only when one of those images happens to land in a batch, perhaps hours into training. The transform is passed in rather than hard-coded, so that the same class can serve training, with random augmentation, and validation, with deterministic resizing and normalisation. In torchvision, the transforms.v2 module is now the recommended transform API; it handles images together with bounding boxes, masks and keypoints, so that a random crop moves the box with the pixels, which the older API could not do. The return value is a tuple of an image tensor and an integer. The default collate function will stack the images into a batch tensor and turn the integers into an int64 tensor, which is exactly what cross_entropy wants. Returning a dictionary instead, such as {"image": ..., "label": ...}, is equally well supported and becomes more readable as the number of fields grows; the default collate function recurses into dictionaries, lists and tuples and stacks the tensors it finds at the leaves. When the data does not fit the default Much of the craft of dataset writing is in handling data that does not arrive as fixed-size tensors. Three situations come up constantly. The first is variable-length sequences: sentences of different lengths, audio clips of different durations, molecules with different numbers of atoms. The default collate function cannot stack tensors of different shapes. The solution is a custom collate_fn, passed to the DataLoader, which receives a list of examples and decides how to combine them. For sequences, that usually means padding to the longest example in the batch and returning a mask or the original lengths so the model can ignore the padding. from torch.nn.utils.rnn import pad_sequence def collate_tokens(batch, pad_id=0): seqs, labels = zip(*batch) lengths = torch.tensor([len(s) for s in seqs]) padded = pad_sequence(seqs, batch_first=True, padding_value=pad_id) mask = padded != pad_id return padded, mask, lengths, torch.tensor(labels) Padding to the longest sequence in each batch, rather than to a global maximum, saves a great deal of computation when lengths vary widely. Going further, sorting or bucketing examples by length so that each batch contains similar lengths reduces padding still more; this is done with a custom sampler, discussed below. Note that inferring the mask from padded != pad_id assumes the pad value never occurs as a real token; if it can, build the mask from the lengths instead. The second is examples that may be missing or corrupt. A path in the index points to a file that was deleted, or a JPEG is truncated. If getitem raises, the whole training run stops. Silently returning None is tempting, but then the collate function must filter it out, and the batch size varies unpredictably. The better course is to validate the index once, before training, and drop or log bad entries in the constructor, or in a separate cleaning script whose output is the index the dataset reads. Data problems found at construction time cost seconds; found at step 40,000 they cost the run. The third is data that does not fit in memory and cannot be indexed randomly, such as a stream of records from a message queue, a set of compressed shards in object storage, or a single enormous text file. For these, PyTorch offers the iterable-style dataset. Iterable datasets and sharding An IterableDataset implements iter instead of getitem. The loader simply iterates over it and batches whatever comes out. There is no length requirement and no random access, which fits streaming data naturally. import json from torch.utils.data import IterableDataset, get_worker_info class JsonlStream(IterableDataset): def init(self, paths): self.paths = list(paths) def iter(self): info = get_worker_info() paths = self.paths if info is not None: # split files among workers paths = paths[info.id::info.num_workers] for path in paths: with open(path) as f: for line in f: rec = json.loads(line) yield torch.tensor(rec["tokens"]), rec["label"] The sharding logic in the middle is the part people miss. When a DataLoader uses several workers with an iterable dataset, each worker gets its own copy of the dataset object and calls iter on it independently. Without the sharding step, every worker reads every file, and every example appears once per worker in each epoch. Nothing raises an error; the model simply sees duplicated data, and an epoch takes several times longer than it should. get_worker_info() returns None in the main process and an object with id and num_workers inside a worker, which is enough to divide the work. In multi-GPU training, you must shard again across processes, typically by rank, so each GPU sees a distinct slice. Shuffling is also your responsibility with iterable datasets, because the loader's shuffle=True option applies only to map-style datasets. The usual approach is to shuffle the order of shards each epoch and maintain a small in-memory buffer from which examples are drawn at random, which gives approximate shuffling with bounded memory. Libraries such as WebDataset, which stores examples in tar archives, and the streaming dataset tooling used by large-scale training frameworks implement this pattern and are worth using rather than rebuilding for production pipelines. (PyTorch's own TorchData project, which once offered composable "DataPipes" for this purpose, deprecated that design in 2024 and refocused on a smaller set of loader utilities, so older tutorials that use DataPipes should not be followed for new code.) What the DataLoader actually does With a dataset in hand, the DataLoader turns it into batches. Its most important arguments are few, but each has a real effect, and Table 3 summarises them with the defaults as of current releases. Table 3. DataLoader arguments that most affect correctness and speed. Argument Default What it controls Practical guidance batch_size 1 Examples per batch Set explicitly; tune with learning rate shuffle False Random order each epoch True for training, False for evaluation num_workers 0 Parallel loading processes Start near CPU cores per GPU; measure pin_memory False Page-locked host buffers True when training on CUDA drop_last False Discard final short batch True if batch norm or fixed shapes need it persistent_workers False Keep workers between epochs True when worker startup is slow collate_fn default How examples become a batch Custom for variable-size data With num_workers=0, the default, all loading happens in the main process, in sequence with training. The GPU sits idle while each batch is read and decoded. For anything beyond small in-memory tensors, this is the most common reason training is slower than it should be. Setting num_workers to a positive number starts that many worker processes, each of which receives a copy of the dataset and a queue of batch indices, loads and collates batches independently, and sends finished batches back to the main process. With enough workers, the next batch is ready by the time the GPU finishes the current one. The right number of workers depends on how expensive each example is and how many CPU cores are available. A starting point is the number of physical cores divided by the number of GPUs on the machine, adjusted by measurement. Too few and the GPU waits; too many and the workers compete for CPU and memory, and startup becomes slow. The test is simple: time an epoch of just iterating the loader with no model. If that alone takes nearly as long as training, data loading is the bottleneck. pin_memory=True makes workers place batches in page-locked host memory, from which the GPU can copy directly by DMA. Combined with tensor.to(device, non_blocking=True) in the training loop, this lets the copy of the next batch overlap with computation on the current one. It is a small change with a measurable benefit on CUDA systems and no benefit on CPU-only training. persistent_workers=True keeps worker processes alive between epochs. By default they are shut down and restarted each epoch, which is cheap for simple datasets but costly when each worker must open file handles, load a tokenizer or build an index on startup. prefetch_factor, which applies only when workers are used, sets how many batches each worker loads ahead. The default of two is usually adequate. Worker processes and the copy problem Because workers are separate processes, a few things that work in single-process code behave differently, and the differences explain several otherwise mysterious problems. Each worker receives a copy of the dataset object. On Linux, where workers are started by forking, the copy is initially shared with the parent through copy-on-write, so a large read-only structure built in init costs little memory at first. But Python's reference counting writes to every object it touches, which gradually forces the operating system to copy the pages. A dataset that holds a Python list of millions of small objects, such as strings or dictionaries, can therefore see memory usage grow steadily across every worker until the machine runs out. The standard remedy is to store such indices as NumPy arrays or other flat buffers rather than lists of Python objects, since an array is one object whose contents are never reference-counted individually. On macOS and Windows, workers are started by spawning a fresh interpreter, so the dataset is pickled and sent to each worker; anything that cannot be pickled, such as an open database connection or a lambda used as a transform, will fail. This is also why scripts that use workers on those platforms need the if name == "__main__": guard around their entry point. Resources that cannot be shared across processes, such as file handles, database connections and some third-party readers for formats like HDF5, should be opened lazily inside the worker, typically on the first call to getitem, rather than in init. A common pattern stores None in the constructor and opens the handle the first time it is needed, so each worker gets its own. Random number generation in workers was a long-standing source of a subtle bug. If augmentation code used NumPy's global random generator and workers were forked, every worker began with the same NumPy state and produced identical "random" augmentations. PyTorch now seeds Python's random module and its own generator differently in each worker, and current NumPy recommends explicit generator objects rather than global state, but the defensive pattern is still worth knowing. import random import numpy as np def seed_worker(worker_id): seed = torch.initial_seed() % 2**32 np.random.seed(seed) random.seed(seed) g = torch.Generator() g.manual_seed(0) loader = DataLoader(ds, batch_size=64, shuffle=True, num_workers=4, worker_init_fn=seed_worker, generator=g) Passing a seeded generator to the loader makes the shuffling order reproducible; the worker initialisation function derives each worker's NumPy and Python seeds from the per-worker seed PyTorch has already chosen. Samplers: controlling the order The sampler is the component that produces indices. shuffle=True is shorthand for a RandomSampler, and shuffle=False for a SequentialSampler. You can pass your own through the sampler argument, or a batch_sampler that yields whole lists of indices at once. Two custom samplers come up often. The first handles class imbalance. When one class outnumbers another by a hundred to one, a model can reach high accuracy by ignoring the rare class. WeightedRandomSampler draws examples with probability proportional to supplied weights, typically the inverse of each example's class frequency, so batches are balanced in expectation. Because it samples with replacement by default, rare examples are repeated within an epoch, which risks overfitting to them; the alternative of reweighting the loss through the weight argument of cross_entropy changes the gradient instead of the data and is often worth trying first. The second is length bucketing for sequence data, mentioned earlier. A batch sampler that groups indices of similar length, shuffles within and across buckets, and yields batches of indices can cut the padding overhead of a language or speech model dramatically. In multi-GPU training with DistributedDataParallel, each process must see a different subset of the data. DistributedSampler handles this for map-style datasets by giving each rank an interleaved slice of a shared random permutation. It has one notorious requirement: you must call sampler.set_epoch(epoch) at the start of every epoch, because the shuffling seed is derived from the epoch number. Forget it, and every epoch uses the same order. The model still trains, just slightly worse, and nothing tells you why. Splitting data without leaking it The last responsibility of the data layer is the split between training, validation and test data, and it is the place where the most consequential errors in applied machine learning are made. torch.utils.data.random_split divides a dataset into random subsets of given sizes, and Subset wraps a dataset with a list of indices. Both are convenient, and both make it easy to split at the wrong level. The rule is that the split must reflect how the model will be used. If the model will be applied to new patients, then all images from a given patient must be on the same side of the split; otherwise the model can learn to recognise patients rather than conditions, and validation accuracy will overstate real performance. If the model will forecast the future, the validation data must come after the training data in time. If near-duplicate images exist, as they do in most scraped collections, they must be grouped. Random splitting at the level of individual examples is correct only when examples are genuinely independent, which is less often than one would hope. The same principle applies to preprocessing. Statistics used for normalisation, vocabularies built from text and any feature scaling must be computed on the training split alone and then applied to validation and test data. Computing a vocabulary on all the data lets information about the test set into the model's inputs. The effect is often small but always in the flattering direction, which is why it survives review. A practical pattern is to perform the split once, in a script, writing out separate index files for each split along with any statistics computed from the training portion, and to have the dataset class read an index file rather than split anything itself. The split is then a versioned artefact that can be inspected, reused and reported, not a side effect of a random seed in the training script. With data flowing correctly, the next step is to do something with it. Chapter 5: Writing the Training Loop PyTorch does not have a fit method. That absence is a design decision, not an oversight. The training loop is where research ideas live, such as a new loss, a curriculum, an adversarial step or a second model trained against the first, and a framework that hid the loop would have to anticipate all of them. Instead, PyTorch hands you the pieces and expects you to assemble them. Higher-level libraries such as PyTorch Lightning and Hugging Face's Trainer and Accelerate write the loop for you, and they are good tools. But every one of them is built from the same few calls, and when something goes wrong inside them, the debugging happens at the level of those calls. This chapter builds a loop from the minimum upward, adding one piece at a time and explaining what each piece protects against. The five lines at the core Stripped to its essentials, one step of training is five operations. for inputs, targets in train_loader: optimizer.zero_grad() outputs = model(inputs) loss = loss_fn(outputs, targets) loss.backward() optimizer.step() Clear the gradients left over from the previous step, run the forward pass, compute a scalar loss, backpropagate to fill each parameter's .grad, and let the optimizer update the parameters using those gradients. Everything else in a production training loop is an elaboration of these five lines: making them run on the right device, measuring whether they are working, keeping them numerically stable, making them faster, and making their results recoverable. The order matters in two places. zero_grad must happen before backward, since gradients accumulate, although it can be placed either at the start of the step or immediately after step(); both are common and equivalent. And step() must follow backward(), since the optimizer reads .grad. Moving zero_grad to the wrong place, for instance between backward and step, erases the gradients before they are used and produces a model that does not learn at all. The optimizer and the learning rate schedule The optimizer holds references to the parameters it updates and any state it keeps per parameter, such as momentum buffers or Adam's running estimates of the first and second moments of the gradient. Two optimizers cover most modern practice. Stochastic gradient descent with momentum remains strong for convolutional networks trained on images, with careful tuning. AdamW, the variant of Adam with decoupled weight decay described by Loshchilov and Hutter, is the default for transformers and a reasonable default almost everywhere else. Adam's state doubles the memory used by the parameters themselves, since it keeps two extra tensors per parameter, which matters for large models. A common error is to use torch.optim.Adam with a weight_decay argument and assume it behaves like AdamW. It does not: in Adam, weight decay is added to the gradient and then scaled by the adaptive learning rate, which weakens it for parameters with large gradient variance. AdamW applies decay directly to the weights. If a paper specifies AdamW, use torch.optim.AdamW. Recent PyTorch releases include fused and multi-tensor ("foreach") implementations of the common optimizers, which update all parameters in a small number of kernel launches rather than one per parameter. The foreach implementation is used by default on CUDA where supported, and fused=True can be requested explicitly for further speed. The learning rate is the single most important hyperparameter, and in practice it is almost never constant. A schedule changes it over the course of training, and PyTorch provides schedulers in torch.optim.lr_scheduler. Transformer training almost universally uses a short linear warmup, during which the learning rate rises from near zero, followed by a decay, often cosine-shaped, to a small final value. Warmup matters because Adam's second-moment estimates are poor in the first few hundred steps, and large updates based on them can destabilise training permanently. from torch.optim.lr_scheduler import LinearLR, CosineAnnealingLR, SequentialLR total_steps = len(train_loader) * num_epochs warmup = 500 scheduler = SequentialLR( optimizer, schedulers=[ LinearLR(optimizer, start_factor=0.01, total_iters=warmup), CosineAnnealingLR(optimizer, T_max=total_steps - warmup, eta_min=1e-6), ], milestones=[warmup], ) The scheduler must be stepped at the right frequency. A schedule defined in steps must have scheduler.step() called after every optimizer step; one defined in epochs, once per epoch. Mixing the two is a classic bug: a cosine schedule meant to span 100,000 steps but stepped once per epoch barely moves, while one meant to span 50 epochs but stepped every batch hits its minimum within the first epoch. Printing scheduler.get_last_lr() at intervals catches either mistake immediately. ReduceLROnPlateau is the odd one out among schedulers: it watches a metric and lowers the learning rate when it stops improving, so its step takes the metric as an argument and is normally called after validation. A complete loop Putting device placement, evaluation mode, gradient control, metrics and scheduling together gives a loop that is worth writing out in full once, because every line has a reason. import torch import torch.nn.functional as F def train_one_epoch(model, loader, optimizer, scheduler, device): model.train() total_loss, total_n = 0.0, 0 for inputs, targets in loader: inputs = inputs.to(device, non_blocking=True) targets = targets.to(device, non_blocking=True) optimizer.zero_grad(set_to_none=True) logits = model(inputs) loss = F.cross_entropy(logits, targets) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) optimizer.step() scheduler.step() total_loss += loss.item() * inputs.size(0) total_n += inputs.size(0) return total_loss / total_n @torch.no_grad() def evaluate(model, loader, device): model.eval() total_loss, correct, total_n = 0.0, 0, 0 for inputs, targets in loader: inputs, targets = inputs.to(device), targets.to(device) logits = model(inputs) total_loss += F.cross_entropy(logits, targets, reduction="sum").item() correct += (logits.argmax(dim=1) == targets).sum().item() total_n += inputs.size(0) return total_loss / total_n, correct / total_n A few of these lines deserve comment. model.train() at the start of the training function and model.eval() at the start of the evaluation function make each function responsible for its own mode. Placing the mode switches inside the functions, rather than in the outer loop, means that no sequence of calls can leave the model in the wrong mode. The evaluation function is decorated with torch.no_grad(), which works as a decorator as well as a context manager. Without it, evaluation would build a graph for every batch and hold activations in memory until the graph was discarded, which roughly doubles memory use and slows evaluation for no benefit. The loss used for reporting is accumulated as loss.item() * batch_size and divided by the total number of examples at the end. Averaging per-batch means instead gives the final, possibly short, batch the same weight as a full one, which is slightly wrong. In evaluation, reduction="sum" gives the exact total directly. loss.item() forces a synchronisation between the CPU and the GPU, because the CPU must wait for the value to be computed before it can read it. Calling it every step is acceptable for most models, but for small models where each step is very fast, the stalls add up. Accumulating the loss as a tensor on the device and calling .item() only when logging avoids them. F.cross_entropy expects raw logits, not probabilities. It applies log-softmax internally, which is more numerically stable than computing softmax and then taking its logarithm. Applying a softmax layer at the end of the model and then passing its output to cross_entropy is a common mistake; it does not raise an error, but it squashes the gradients and slows learning noticeably. The same applies to binary classification, where F.binary_cross_entropy_with_logits should be preferred to a sigmoid followed by binary_cross_entropy. The outer loop that calls these two functions should track the best validation metric and save a checkpoint when it improves. best_acc = 0.0 for epoch in range(num_epochs): train_loss = train_one_epoch(model, train_loader, optimizer, scheduler, device) val_loss, val_acc = evaluate(model, val_loader, device) print(f"epoch {epoch}: train {train_loss:.4f} val {val_loss:.4f} acc {val_acc:.3f}") if val_acc > best_acc: best_acc = val_acc save_checkpoint("best.pt", model, optimizer, scheduler, epoch) save_checkpoint("last.pt", model, optimizer, scheduler, epoch) Checkpoints that can actually resume A checkpoint containing only the model weights is enough for deployment but not for resuming training. To continue exactly where a run stopped after a crash, preemption or deliberate pause, you need the optimizer state, since Adam's moment estimates and SGD's momentum are part of the trajectory, the scheduler state, the epoch or step counter, and ideally the random number generator states. If you use mixed precision with a gradient scaler, its state belongs in the checkpoint too. import os def save_checkpoint(path, model, optimizer, scheduler, epoch, scaler=None): state = { "model": model.state_dict(), "optimizer": optimizer.state_dict(), "scheduler": scheduler.state_dict(), "epoch": epoch, "rng_cpu": torch.get_rng_state(), } if torch.cuda.is_available(): state["rng_cuda"] = torch.cuda.get_rng_state_all() if scaler is not None: state["scaler"] = scaler.state_dict() tmp = path + ".tmp" torch.save(state, tmp) os.replace(tmp, path) # atomic rename on the same filesystem Writing to a temporary file and renaming it means that a crash in the middle of saving cannot leave a corrupted checkpoint in place of a good one, a small precaution that matters most on exactly the long runs where checkpoints are most valuable. When resuming, restore the model, move it to the device, construct the optimizer and scheduler as usual, then call load_state_dict on each and continue from the saved epoch plus one. Because these checkpoints contain only tensors, dictionaries and primitive values, they load under the default weights_only=True restriction described in Chapter 3. Optimizer and scheduler state dictionaries are designed to be safe in this sense, although a scheduler holding a custom Python object, such as a LambdaLR with a lambda function, will not round-trip through state_dict regardless. Mixed precision Modern accelerators perform matrix multiplication much faster in 16-bit formats than in 32-bit ones, and 16-bit activations halve the memory used by the forward pass. Automatic mixed precision, or AMP, captures most of that gain without the numerical problems of converting the whole model. It runs selected operations, chiefly matrix multiplications and convolutions, in reduced precision while keeping others, such as reductions, softmax and losses, in float32. The weights themselves remain in float32, so small updates are not lost to rounding. The current API lives in torch.amp. The older torch.cuda.amp names still appear in many tutorials but are deprecated in favour of the device-generic versions. use_bf16 = torch.cuda.is_available() and torch.cuda.is_bf16_supported() amp_dtype = torch.bfloat16 if use_bf16 else torch.float16 scaler = torch.amp.GradScaler("cuda", enabled=(amp_dtype == torch.float16)) for inputs, targets in train_loader: inputs, targets = inputs.to(device), targets.to(device) optimizer.zero_grad(set_to_none=True) with torch.autocast(device_type="cuda", dtype=amp_dtype): logits = model(inputs) loss = F.cross_entropy(logits, targets) scaler.scale(loss).backward() scaler.unscale_(optimizer) torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) scaler.step(optimizer) scaler.update() scheduler.step() The gradient scaler exists because of float16's narrow range. Small gradients underflow to zero in float16, so the scaler multiplies the loss by a large factor before backward, which scales every gradient by the same factor, and divides it out before the optimizer step. If any gradient overflows to infinity, the scaler skips that step and reduces the factor; if many steps pass without overflow, it increases the factor. The unscale_ call before clipping matters: without it, you would clip the scaled gradients against a threshold meant for unscaled ones. With bfloat16, which has float32's exponent range, underflow is rarely a problem and the scaler is unnecessary, which is why the code above disables it in that case. This simplicity, together with the absence of overflow failures, is the main reason bfloat16 has become the default choice on hardware that supports it, including NVIDIA's Ampere generation and later and Google's TPUs. Only the forward pass and loss computation go inside the autocast context. The backward pass runs outside it and automatically uses the same precision as the corresponding forward operations. Gradient accumulation and clipping When the batch size you want does not fit in memory, gradient accumulation simulates it. Because gradients accumulate by default, you can run several smaller forward and backward passes and call optimizer.step() only after the last. accum = 4 optimizer.zero_grad(set_to_none=True) for i, (inputs, targets) in enumerate(train_loader): loss = F.cross_entropy(model(inputs.to(device)), targets.to(device)) (loss / accum).backward() if (i + 1) % accum == 0: torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() scheduler.step() optimizer.zero_grad(set_to_none=True) Dividing the loss by the number of accumulation steps keeps the gradient magnitude equal to that of a single large batch with a mean loss. Forgetting the division multiplies the effective learning rate by the accumulation factor. The scheduler steps once per optimizer step, not once per micro-batch. The equivalence with a true large batch is exact for most layers but not for batch normalisation, whose statistics are still computed over each small micro-batch. Gradient clipping limits the size of an update. clip_grad_norm_ computes the norm of all gradients concatenated together and, if it exceeds max_norm, scales them all down proportionally, preserving the direction of the update. It returns the norm before clipping, which is one of the most informative numbers you can log: a steady gradient norm indicates healthy training, while spikes often precede loss spikes and divergence. A threshold of 1.0 is a common default for transformers. Clipping individual values with clip_grad_value_ is cruder, since it changes the direction of the update, and is rarely preferable. Adding a second GPU The introduction promised to show what changes in the loop when training moves from one GPU to several. The answer is surprisingly little, because PyTorch's standard tool, DistributedDataParallel, or DDP, is designed to leave the loop alone. DDP runs one process per GPU. Each process holds a full copy of the model, reads a different slice of the data, and computes its own forward and backward pass. During the backward pass, DDP averages the gradients across all processes with an all-reduce operation, overlapping the communication with the remaining gradient computation, so that by the time optimizer.step() runs every process holds identical gradients and therefore makes an identical update. The copies of the model never drift apart, and none of them needs to be told about the others. The older nn.DataParallel, which used one process driving several GPUs from multiple threads, still exists but is slower and is not recommended even on a single machine; the documentation says to use DDP instead. A complete DDP script differs from the single-device version in only a handful of places. import os import torch import torch.distributed as dist import torch.nn.functional as F from torch.nn.parallel import DistributedDataParallel as DDP from torch.utils.data import DataLoader from torch.utils.data.distributed import DistributedSampler def main(): dist.init_process_group("nccl" if torch.cuda.is_available() else "gloo") rank = dist.get_rank() local_rank = int(os.environ["LOCAL_RANK"]) if torch.cuda.is_available(): torch.cuda.set_device(local_rank) device = torch.device("cuda", local_rank) else: device = torch.device("cpu") sampler = DistributedSampler(train_ds, shuffle=True) loader = DataLoader(train_ds, batch_size=32, sampler=sampler, num_workers=4, pin_memory=True) model = build_model().to(device) ddp_ids = [local_rank] if device.type == "cuda" else None model = DDP(model, device_ids=ddp_ids) optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3) for epoch in range(num_epochs): sampler.set_epoch(epoch) model.train() for x, y in loader: x, y = x.to(device), y.to(device) optimizer.zero_grad(set_to_none=True) loss = F.cross_entropy(model(x), y) loss.backward() # gradients are averaged here optimizer.step() if rank == 0: torch.save(model.module.state_dict(), "checkpoint.pt") dist.barrier() dist.destroy_process_group() if name == "__main__": main() The script is launched with torchrun --nproc_per_node=4 train.py, which starts four processes and sets the environment variables, RANK, LOCAL_RANK, WORLD_SIZE and the rendezvous address, that init_process_group reads. The changes from the single-device loop are worth listing because each one prevents a specific failure. The process group must be initialised before anything else communicates. Each process must bind to its own GPU with torch.cuda.set_device, or all of them will allocate on GPU 0. The data must be split with DistributedSampler, with set_epoch called every epoch, as Chapter 4 explained. The model must be moved to its device before being wrapped. Checkpoints should be written by one process only, conventionally rank 0, and should save model.module.state_dict(), the underlying model's state, because the wrapper adds a module. prefix to every key that a single-device model will not expect when loading. The barrier call makes the other processes wait until the checkpoint is written. Two subtler consequences follow from the arithmetic. The effective batch size is the per-process batch size multiplied by the number of processes, so four GPUs with a batch size of 32 each train with a global batch of 128, and the learning rate may need to change accordingly; a linear increase with a warmup is the common heuristic, though it does not hold indefinitely. And every metric computed in the loop is local to one process. A validation accuracy printed by rank 0 reflects only rank 0's slice of the validation data unless you combine the counts from all processes with dist.all_reduce before dividing. When gradients are accumulated over several micro-batches under DDP, each backward call would normally trigger an all-reduce, wasting communication on intermediate results. The model.no_sync() context manager suppresses synchronisation for the micro-batches before the last, so that gradients are averaged across processes only once per optimizer step. DDP requires every process to hold a full copy of the model, its gradients and its optimizer state. When that no longer fits, the next step is fully sharded data parallelism, FSDP, which splits parameters, gradients and optimizer state across processes and gathers each layer's parameters only while that layer is computing. Its current implementation, often called FSDP2, is applied with the fully_shard function on each block of a model. It changes how models are constructed, checkpointed and loaded more than DDP does, and it is the point at which most teams begin to lean on a higher-level library. The training loop itself, however, still has the same five lines at its centre. Validation, early stopping and what to log A training loop is only as good as its feedback. At a minimum, log the training loss, the validation loss and a task-relevant validation metric each epoch, the learning rate, and the gradient norm at regular intervals. Plot them; TensorBoard, through torch.utils.tensorboard.SummaryWriter, and hosted tools such as Weights and Biases are the common choices. The shape of these curves is the main diagnostic of what to change. A training loss that does not fall indicates a bug, a learning rate that is far too high or too low, or data that does not contain the signal. A training loss that falls while validation loss rises indicates overfitting, calling for more data, more augmentation or more regularisation. A validation loss that tracks training loss closely while both stay high indicates underfitting, calling for a larger model or longer training. Early stopping, which halts training when the validation metric has not improved for a set number of epochs, saves time on runs that have plateaued. It requires nothing more than a counter in the outer loop. Combined with saving the best checkpoint, it means the model you keep is the one that generalised best, not simply the last one trained. One sanity check deserves special mention because it catches so many bugs at once: before training on the full dataset, train on a single small batch, perhaps eight examples, for a few hundred steps with regularisation turned off. A correct model and loop will drive the training loss close to zero, memorising the batch. If it cannot, something is broken, whether in the labels, the loss, the gradient flow or the optimizer setup, and it will be much easier to find on eight examples than on eight million. The next chapter turns from building the loop to keeping it healthy. Chapter 6: Debugging, Profiling and Making It Fast A training loop that runs without raising an error is not the same as a training loop that works. Deep learning code fails quietly more often than it fails loudly. A misaligned label, a missing eval() call, a normalisation computed on the wrong split or a learning rate one order of magnitude off will each produce a model that trains, reports numbers and is wrong. Andrej Karpathy's widely read 2019 essay "A Recipe for Training Neural Networks" made the point memorably: neural network training fails silently, and the practitioner's defence is a slow, paranoid, step-by-step process of verifying each component. This chapter turns that attitude into specific techniques, then turns to the separate question of speed, where PyTorch's profiler and compiler do most of the work. Loud failures: reading the error The failures that raise exceptions are the easy ones, and a handful of messages account for most of them. A shape mismatch in a matrix multiplication, reported as something like "mat1 and mat2 shapes cannot be multiplied (32x512 and 256x10)", almost always means the input size declared for a linear layer does not match what the previous layer produces. The fastest fix is to run a single example through the model with a print of x.shape before the failing layer, or to use nn.LazyLinear, which infers its input size from the first batch it sees. Lazy layers are handy for prototyping but make the model's parameter count unknowable until the first forward pass, which complicates optimizer creation, so many practitioners replace them with explicit sizes once the architecture settles. "Expected all tensors to be on the same device" means exactly what it says. Usually the model has been moved and a tensor created inside forward has not, such as a mask built with torch.ones(n) rather than torch.ones(n, device=x.device). Creating new tensors inside forward with device=x.device and dtype=x.dtype, or with helpers such as torch.zeros_like(x) and x.new_zeros(...), avoids this class of bug permanently. Registering constant tensors as buffers, as described in Chapter 3, is the other half of the solution. "Expected scalar type Long but found Float", or its reverse, is a dtype mismatch, typically labels of the wrong type passed to a loss. CUDA errors are harder because GPU execution is asynchronous. The CPU queues kernels and moves on, so when a kernel fails, for instance because an index is out of range in an embedding lookup, the error surfaces at some later line, often an unrelated one. Setting the environment variable CUDA_LAUNCH_BLOCKING=1 forces each kernel to complete before the next is queued, so the error is reported at the operation that caused it. It slows execution dramatically and should be used only for diagnosis. An even quicker route is often to run the failing batch on the CPU, where the same out-of-range index produces a clear Python exception naming the offending value. The most common cause of "device-side assert triggered" is a class label that is greater than or equal to the number of output classes, which happens when labels are numbered from one rather than zero, or when the class list used to build the model differs from the one used to encode the labels. "CUDA out of memory" has its own section below. Quiet failures: a checklist that finds most of them For the failures that raise nothing, a fixed routine is more productive than inspiration. The following checks, roughly in order of cost, find the large majority of silent bugs. Look at the data as the model sees it. Take a batch from the loader, after all transforms, and render a few examples. Decode the labels back to class names and compare them with what you see. This takes five minutes and catches mislabelled data, transforms that destroy the signal, channels in the wrong order, images normalised twice and augmentations that are far too aggressive. Most practitioners who skip this step eventually lose a week to something it would have shown. Check the loss at initialisation. For a classifier with C balanced classes and a sensibly initialised final layer, the initial cross-entropy should be close to the natural logarithm of C, about 2.30 for ten classes. A much larger value means the initial logits are too large or the labels are wrong; a value of zero means the task is trivially solvable, usually because the label leaked into the input. Overfit one batch, as described at the end of Chapter 5. If the model cannot drive the loss on eight examples to near zero, the bug is in the model, the loss or the optimizer, not in the data volume or the hyperparameters. Establish a baseline. Before celebrating a validation accuracy, compute what a trivial model would achieve: always predicting the majority class, or predicting the previous value in a time series. A model at 95 percent accuracy is unremarkable if 95 percent of examples belong to one class. Check gradient flow. After one backward pass, inspect the gradient norm of each parameter. Parameters with None gradients are disconnected; parameters with zero gradients may be behind a dead activation or a saturated nonlinearity; parameters with norms many orders of magnitude larger than others are candidates for instability. for name, p in model.named_parameters(): if p.grad is None: print(f"{name:40s} NO GRAD") else: print(f"{name:40s} {p.grad.norm().item():.3e}") Test for leakage. If a validation result looks too good, it probably is. Shuffle the labels of the training set and train again. A model that still performs well on validation data is exploiting something other than the relationship between inputs and labels, which usually means duplicates across splits or a feature that encodes the label. Exploding losses and nan When the loss becomes nan, the sequence of events is almost always the same: some value grew too large, overflowed to infinity, and then an operation such as infinity minus infinity, or zero times infinity, produced nan, which then contaminated everything downstream. The task is to find the first value that went wrong. The usual causes are a learning rate that is too high, particularly without warmup; a division by a quantity that can be zero, such as a standard deviation computed over a constant input; a logarithm of zero or of a negative number, typically from computing log(softmax(x)) by hand instead of using log_softmax; a square root of zero in a function whose derivative is infinite at zero, as with a Euclidean norm of a zero vector; and overflow in float16, where anything above 65,504 becomes infinity. The remedies are correspondingly specific: add a small epsilon inside divisions, square roots and logarithms; use the fused, numerically stable functions PyTorch provides, such as log_softmax, logsumexp and the withlogits losses; add warmup and clipping; and prefer bfloat16 to float16. To locate the first bad value, check for non-finite values with torch.isfinite(x).all() at the output of each block in the forward pass, or register forward hooks that do so on every module. For problems that arise only in the backward pass, torch.autograd.set_detect_anomaly(True), described in Chapter 2, points at the forward operation whose gradient produced the first nan. A small practical tip: when the loss becomes nan, save the batch that caused it, together with the model weights from the previous step. Reproducing the failure in isolation is far easier than waiting for it to recur. Memory: where it goes and how to get it back GPU memory during training is consumed by four things: the model's parameters, their gradients, the optimizer state and the activations saved for the backward pass. For a model with P parameters trained in float32 with Adam, the first three alone take sixteen bytes per parameter: four for the weight, four for its gradient and eight for Adam's two moment estimates. A model with a billion parameters therefore needs about sixteen gigabytes before a single activation is stored. Activations scale with batch size, sequence length or image resolution, and depth, and for most models of moderate size they are the dominant term. When you hit an out-of-memory error, the remedies in rough order of convenience are: reduce the batch size and use gradient accumulation to keep the effective batch size the same; switch to mixed precision, which roughly halves activation memory; use activation checkpointing, described in Chapter 2, to trade computation for activation memory; and, for models whose parameters and optimizer state alone are too large, shard them across devices using fully sharded data parallelism, which PyTorch provides as FSDP. PyTorch's CUDA memory allocator caches freed blocks rather than returning them to the driver, so tools such as nvidia-smi show memory as in use even after tensors are freed. The accurate numbers come from PyTorch itself: torch.cuda.memory_allocated() reports memory held by live tensors, torch.cuda.max_memory_allocated() reports the peak, and torch.cuda.memory_summary() gives a detailed breakdown. For hard cases, torch.cuda.memory._record_memory_history() records every allocation with its stack trace, and the resulting snapshot can be explored in PyTorch's online memory visualiser to see exactly which operations hold memory at the peak. Calling torch.cuda.empty_cache() releases cached blocks back to the driver, which is useful when sharing a GPU with another process but does not make more memory available to PyTorch itself. Memory leaks during training, where usage creeps upward step after step, have one overwhelmingly common cause: holding references to tensors that are attached to the computation graph, so that each step's graph is kept alive. Accumulating total_loss += loss instead of total_loss += loss.item() is the classic example. Storing model outputs in a list for later analysis without detach() is another. Measuring before optimising Making training faster starts with finding out where the time goes, and intuition is unreliable. A model that seems compute-bound may in fact spend half its time waiting for data, or launching thousands of tiny kernels each too small to keep the GPU busy, or synchronising with the CPU on every step. The first measurement needs no tools. Time an epoch of iterating the data loader alone, without the model. Then time an epoch of training with synthetic data, a fixed random tensor reused every step, without the loader. If the first is close to total training time, the bottleneck is data; increase num_workers, move expensive preprocessing offline, use faster decoders, or store data in a format that is cheaper to read. If the second is close, the bottleneck is the model, and the profiler is the next tool. When timing GPU code by hand, remember that kernel launches are asynchronous. Python's time.time() around a forward pass measures how long it took to queue the work, not to do it. Call torch.cuda.synchronize() before reading the clock, or use torch.cuda.Event objects with enable_timing=True. The torch.utils.benchmark module handles synchronisation, warm-up and repetition for you and is the right tool for comparing two implementations of an operation. PyTorch's built-in profiler records the time spent in every operator on both CPU and GPU. from torch.profiler import profile, ProfilerActivity, schedule activities = [ProfilerActivity.CPU] if torch.cuda.is_available(): activities.append(ProfilerActivity.CUDA) with profile(activities=activities, schedule=schedule(wait=1, warmup=1, active=3), record_shapes=True) as prof: for step, (inputs, targets) in enumerate(train_loader): train_step(inputs, targets) prof.step() if step >= 5: break print(prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=15)) prof.export_chrome_trace("trace.json") The schedule skips the first step, which includes one-off costs such as memory allocation and kernel selection, warms up for one more, and then records three. The table lists the operators that consumed the most time; the exported trace can be opened in Perfetto or Chrome's trace viewer to show a timeline of CPU and GPU activity. The timeline is often the more revealing view. Gaps in the GPU row where nothing is running mean the GPU is waiting for something: data, a CPU-side computation, a .item() call or a host-to-device copy. A dense band of very short kernels means launch overhead dominates, which is exactly the situation the compiler is designed to address. Cheap speedups Before reaching for anything elaborate, a handful of one-line changes routinely deliver large improvements on NVIDIA GPUs, and they should be standard. Mixed precision, covered in Chapter 5, is the largest single gain on modern hardware for most models. Allowing TensorFloat-32 for remaining float32 matrix multiplications on Ampere and newer GPUs, with torch.backends.cuda.matmul.allow_tf32 = True or torch.set_float32_matmul_precision("high"), trades a little precision for substantial speed. For convolutional networks, torch.backends.cudnn.benchmark = True lets cuDNN time several algorithms for each input shape and choose the fastest, which helps when input sizes are fixed and hurts slightly when they vary from batch to batch. The channels-last memory format, described in Chapter 1, often helps convolutions further. Using fused optimizers and zero_grad(set_to_none=True) trims per-step overhead. Using PyTorch's F.scaled_dot_product_attention rather than a hand-written attention computation selects fused implementations such as FlashAttention automatically where the hardware supports them, which cuts both memory and time for transformer models. And making sure the data loader uses enough workers with pinned memory removes the most common bottleneck of all. torch.compile PyTorch 2.0, released in March 2023, introduced torch.compile, and it has since become the principal tool for making PyTorch models faster without rewriting them. Wrapping a model is one line. model = MyModel().to(device) compiled = torch.compile(model) out = compiled(batch) # first call compiles; later calls reuse the result Under the hood, three components cooperate. TorchDynamo intercepts the execution of Python bytecode in the model's forward and extracts sequences of PyTorch operations into graphs, falling back to ordinary Python for anything it cannot capture. AOTAutograd traces the backward pass ahead of time, so the gradient computation is compiled too. TorchInductor, the default backend, takes the resulting graphs and generates optimised code, Triton kernels on GPUs and C++ on CPUs, fusing chains of elementwise operations into single kernels so that intermediate results stay in fast on-chip memory instead of making round trips to the GPU's main memory. The speedup depends heavily on the model. Models dominated by large matrix multiplications already run near peak on cuBLAS and gain less; models with many elementwise operations, normalisations and activations between those multiplications gain more, often substantially. The first call is slow, since compilation can take from seconds to minutes for a large model, so torch.compile pays off for training runs and long-lived inference servers rather than short scripts. Three concepts explain most of the surprises people encounter. A graph break occurs when Dynamo meets something it cannot trace, such as a call to an unsupported library, printing a tensor's value, or converting a tensor to a Python number and branching on it. It ends the current graph, runs that piece in ordinary Python and starts a new graph afterwards. The model still works, but each break limits how much can be fused. Running with the environment variable TORCH_LOGS="graph_breaks" lists every break and its cause, and torch._dynamo.explain gives a summary. A guard is a condition Dynamo records alongside each compiled graph, such as the shapes and dtypes of its inputs, which must hold for the compiled code to be reused. When a guard fails, Dynamo recompiles. By default, after seeing a second distinct shape for a dimension, it marks that dimension as dynamic and compiles a version that works for a range of sizes, which avoids endless recompilation for variable sequence lengths. Too many recompilations, which are logged, mean something in the input changes every call, such as a Python integer argument, and should be made stable or marked dynamic. The mode argument trades compile time for runtime performance. The default is a sensible balance. mode="reduce-overhead" additionally uses CUDA graphs, which record a sequence of kernel launches once and replay it with a single launch, eliminating per-kernel CPU overhead and benefiting small models and small batches most. mode="max-autotune" benchmarks multiple kernel configurations and can be noticeably faster at the cost of long compile times. The practical advice is to get a model working correctly in eager mode first, then compile it, check that the outputs match within floating-point tolerance, and measure the speedup on your actual workload rather than assuming one. If compilation fails or produces a regression, torch.compile can be applied to individual submodules rather than the whole model, which often isolates the problem. Compilation also connects to deployment. The same capture technology that powers torch.compile, turning a Python model into a graph of PyTorch operations, is what underlies the modern export path described in the next chapter. A model that compiles cleanly, without graph breaks, is a model that will usually export cleanly too, and that is a good reason to treat graph breaks as defects worth fixing even when the speedup from removing them is small. Chapter 7: From Python to a Portable Model: TorchScript and torch.export Everything that makes PyTorch pleasant during research works against it in production. Eager execution means the model is a Python program, so running it requires a Python interpreter, the right versions of every library it imports, and the source code of its classes. Each operation is dispatched individually from Python, which costs microseconds per call that add up for small models and low-latency services. And because the computation is discovered as it runs, nothing about the model's structure is known in advance, so nothing can be optimised ahead of time. Deployment, for PyTorch, means turning a program into an artefact: a description of the computation that can be saved, loaded somewhere else, optimised and executed without the original Python. For most of PyTorch's history the official tool for that job was TorchScript. It is still found in a great many production systems, mobile applications and model repositories, and anyone maintaining PyTorch models in industry will meet it. But its time as the recommended path is over. PyTorch 2.10, released in January 2026, formally deprecated TorchScript and directed users to torch.export as the replacement for torch.jit.trace and torch.jit.script, and to ExecuTorch for on-device deployment. In PyTorch 2.14, released in September 2026, calling torch.jit.trace, torch.jit.script or torch.jit.load emits a FutureWarning advising users to switch to torch.compile or torch.export. TorchScript has not been removed, and existing models continue to load and run, but it receives no new features and new deployments should not be built on it. This chapter explains TorchScript first, because you need to understand it to maintain what exists and to migrate away from it, and then the export-based tools that replace it. Tracing: recording one run TorchScript offers two ways to convert a model, and the difference between them is the key to understanding both its usefulness and its pitfalls. The first is tracing. torch.jit.trace runs the model once with an example input and records the tensor operations that were executed, producing a ScriptModule that replays exactly those operations. import torch from torch import nn import torch.nn.functional as F class Net(nn.Module): def init(self): super().__init__() self.fc = nn.Linear(8, 4) def forward(self, x): return F.relu(self.fc(x)) model = Net().eval() example = torch.randn(2, 8) traced = torch.jit.trace(model, example) # FutureWarning on PyTorch 2.14 traced.save("net_traced.pt") loaded = torch.jit.load("net_traced.pt") # no Python class needed print(loaded(torch.randn(5, 8)).shape) # torch.Size([5, 4]) The saved file is a zip archive containing the recorded program and the weights, and it can be loaded without the Net class, both from Python and from C++ through the LibTorch library's torch::jit::load. That ability to run without Python is what made TorchScript attractive for servers, mobile apps and embedded systems. Tracing's weakness follows directly from how it works. It records what happened on one run, so anything that depends on the values of the input, rather than just its operations, is frozen to the path taken during tracing. A conditional such as if x.sum() > 0: is evaluated once, and the trace contains only the branch that was taken; the other branch is gone. A loop whose length depends on the input is unrolled to the number of iterations that happened to run. Python values computed from shapes may be baked in as constants. PyTorch does warn about some of these cases, emitting a TracerWarning that converting a tensor to a Python boolean "might cause the trace to be incorrect", but warnings scroll past in build logs and the resulting model runs without error, silently computing the wrong thing for inputs that would have taken a different path. Tracing is therefore safe only for models whose computation does not depend on input values: most convolutional networks and many feed-forward models fit this description; models with data-dependent control flow do not. Scripting: compiling a subset of Python The second method, scripting, avoids this problem by compiling the source code of the model rather than recording a run. torch.jit.script parses the forward method, and every method and submodule it calls, into TorchScript, a statically typed subset of Python, preserving control flow as real branches and loops in the resulting program. class Gate(nn.Module): def init(self): super().__init__() self.fc = nn.Linear(8, 4) def forward(self, x: torch.Tensor) -> torch.Tensor: if bool(x.sum() > 0): return F.relu(self.fc(x)) return self.fc(x) scripted = torch.jit.script(Gate().eval()) # both branches survive scripted.save("gate_scripted.pt") The price is that the model's code must be written in that subset. TorchScript needs to know the type of every variable, so function arguments that are not tensors must be annotated, and a variable cannot change type. Many Python features are unsupported or restricted: arbitrary third-party library calls, most uses of args and *kwargs, dynamic attribute access, generators, and many standard-library functions. Making a large, real-world model scriptable often meant rewriting significant parts of it, adding annotations, replacing unsupported constructs and marking methods with decorators such as @torch.jit.ignore or @torch.jit.unused to exclude them. Teams sometimes maintained two versions of a model, one for research and one for scripting, and the effort of keeping them in sync was a major reason the PyTorch developers decided a new approach was needed. In practice, mature TorchScript deployments mix the two: tracing the bulk of a model, scripting the small parts with genuine control flow, and combining them, since scripted functions can be called from traced code and vice versa. Followed by torch.jit.freeze, which inlines weights as constants and enables further optimisations, this produced fast, Python-free models that served a large share of PyTorch's production traffic for several years. Maintaining TorchScript models today If you are responsible for existing TorchScript models, the deprecation does not demand an emergency rewrite, but it does call for a plan. A few practical observations help. Existing .pt files produced by TorchScript continue to load in current releases, with a warning. Pinning the PyTorch version of a stable inference service is a reasonable short-term measure, but a pinned framework accumulates unpatched vulnerabilities and drifts away from the drivers and hardware it must run on, so pinning is a way to buy time, not a destination. The warnings themselves can be filtered in production logs using Python's warnings module, but they should not be silenced in development or continuous integration, where they serve as a reminder of what needs migrating. C++ applications that load models with torch::jit::load have the most migration work, because their replacement, described below, has a different loading API and produces a different artefact. Mobile applications that used PyTorch Mobile, the TorchScript-based on-device runtime, should move to ExecuTorch, which reached its 1.0 release in October 2025 and is now the PyTorch project's supported route for phones, microcontrollers and other edge devices. The migration itself usually begins by going back to the original eager-mode model, not the TorchScript one. A scripted model's code was often contorted to satisfy the script compiler; the export tools work best with clean, ordinary PyTorch code, and the rewrites made for TorchScript can frequently be undone. torch.export: capturing a whole graph torch.export is the replacement. It uses the same capture technology as torch.compile, described in Chapter 6, but with a different contract. torch.compile is allowed to fall back to Python whenever it meets something it cannot capture, inserting a graph break and carrying on. torch.export is not: it must capture the entire model as a single graph of PyTorch operations, and if it cannot, it fails with an error explaining why. The result is an ExportedProgram, a self-contained representation of the computation consisting of a graph of operations drawn from PyTorch's core operator set, the model's parameters and buffers, and a signature describing its inputs and outputs, including any constraints on their shapes. from torch.export import export, Dim model = Net().eval() example = (torch.randn(2, 8),) batch = Dim("batch", min=1, max=1024) ep = export(model, example, dynamic_shapes={"x": {0: batch}}) print(ep.graph_module.code) # def forward(self, p_fc_weight, p_fc_bias, x): # linear = torch.ops.aten.linear.default(x, p_fc_weight, p_fc_bias) # relu = torch.ops.aten.relu.default(linear) # return (relu,) torch.export.save(ep, "net.pt2") restored = torch.export.load("net.pt2") print(restored.module()(torch.randn(5, 8)).shape) # torch.Size([5, 4]) The printed graph is worth looking at, because it shows what export produces: a flat list of calls to ATen operators, PyTorch's internal tensor library, with the parameters lifted into explicit inputs. There are no Python classes left, no nn.Linear, no forward methods calling other forward methods. That flatness is what makes the artefact portable and optimisable. Shapes are where export differs most visibly from tracing. By default, export specialises on the shapes of the example inputs, so the program above would accept only batches of exactly two rows if the dynamic_shapes argument were omitted. Marking a dimension as dynamic with Dim tells export to treat it as a symbol and to record whatever constraints the model's code implies, such as a minimum and maximum or a relationship between two inputs' dimensions. If the code turns out to require a dimension to be a particular value, export reports the conflict rather than silently specialising. Newer releases also offer Dim.AUTO, which lets export infer which dimensions can be dynamic, and PyTorch 2.14 added a declarative @dynamic_spec decorator in torch.fx.experimental.dynamic_spec for declaring dynamic dimensions once and sharing them across torch.compile and torch.export. Data-dependent control flow, the problem that broke tracing, is handled explicitly. A Python if on a tensor's value cannot be captured into a static graph, and export does not pretend otherwise: applied to the Gate model above, it fails with an error explaining that it "could not guard on data-dependent expression". The fix is to express the branch with torch.cond, a control-flow operator that takes a predicate and two functions and captures both branches into the graph. class GateExportable(nn.Module): def init(self): super().__init__() self.fc = nn.Linear(8, 4) def forward(self, x): return torch.cond( x.sum() > 0, lambda x: F.relu(self.fc(x)), lambda x: self.fc(x), (x,), ) ep_gate = export(GateExportable().eval(), (torch.randn(2, 8),)) Both branches must return tensors of matching shapes and types. For loops with data-dependent trip counts, torch.while_loop plays the equivalent role, and PyTorch 2.14 added torch.switch for multi-way branching. These operators are less convenient than plain Python, but they are honest: the exported program contains exactly the logic the model needs, and nothing has been silently discarded. In current releases, export defaults to a "non-strict" mode (strict=False) in which the model's Python code actually runs, with tensors replaced by tracing proxies, rather than being analysed bytecode by bytecode as torch.compile does. Non-strict mode accepts a wider range of Python, since anything that does not involve tensors simply executes normally. The strict=True argument selects the bytecode-analysis mode, which catches more cases where Python-level side effects would be lost. The default is the right starting point. When export fails, the errors are usually specific about what went wrong and where. torch.export also provides draft_export, which attempts to export while collecting every problem it encounters, rather than stopping at the first, and produces a report that is a useful starting point for migrating a large model. What to do with an exported program An ExportedProgram is an intermediate artefact, not a runtime. It can be executed in Python through ep.module(), which is useful for testing, but its value is as input to tools that turn it into something deployable. Three are important. AOTInductor compiles an exported program ahead of time using the same TorchInductor backend that torch.compile uses, producing a package containing compiled machine code, Triton kernels on GPUs and C++ on CPUs, and the weights. The package can be loaded and run from Python or from a C++ application without a Python interpreter, which makes AOTInductor the direct successor to the "TorchScript plus LibTorch" deployment pattern for servers. pkg_path = torch._inductor.aoti_compile_and_package(ep, package_path="net_aoti.pt2") runner = torch._inductor.aoti_load_package(pkg_path) print(runner(torch.randn(3, 8)).shape) # torch.Size([3, 4]) The leading underscore in torch._inductor signals that the Python entry points still live in a namespace PyTorch treats as internal, even though they are documented and widely used. On the C++ side, the package is loaded with the AOTIModelPackageLoader class from LibTorch. Because the package contains compiled code, it is specific to the platform and, on GPUs, to the class of device it was compiled for; it is not a portable format in the way an exported program is. Each release continues to extend AOTInductor; PyTorch 2.14, for instance, added the ability for several models to share one copy of their weights in memory. ExecuTorch takes an exported program and lowers it, through further transformations and hardware-specific "delegates", into a compact .pte file executed by a small C++ runtime on mobile phones, embedded processors and specialised accelerators. Its 1.0 release in October 2025 listed production-ready backends including XNNPACK for Arm and x86 CPUs, Apple's Core ML, Qualcomm's AI Engine and Arm's Ethos-U neural processors. If your target is a device rather than a server, ExecuTorch is where to go. ONNX export, through torch.onnx.export, converts a model into the Open Neural Network Exchange format, which can be executed by ONNX Runtime, NVIDIA's TensorRT and many vendor toolchains. In recent releases, torch.onnx.export uses the torch.export capture path by default (the dynamo=True argument), replacing the older TorchScript-based exporter. ONNX remains the most widely supported interchange format outside the PyTorch ecosystem and is often the pragmatic choice when the serving infrastructure is not PyTorch-based. The main choices are compared in Table 4. The status column reflects the state of each tool as of the PyTorch 2.14 release, with "current" marking the routes the project now recommends. Table 4. Routes from an eager PyTorch model to a deployable artefact. Route Captures control flow Needs Python to run Main target Status (2.14) torch.jit.trace No, one path recorded No Servers, mobile Deprecated torch.jit.script Yes, via Python subset No Servers, mobile Deprecated torch.export Via torch.cond and friends Program needs a runtime Input to the tools below Current AOTInductor As exported No Servers, C++ apps Current ExecuTorch As exported No Phones, edge devices Current ONNX (dynamo=True) As exported No ONNX Runtime, TensorRT Supported A migration checklist Moving a model from TorchScript, or from an eager-mode deployment, to the export path tends to follow the same sequence, and doing it in order avoids most of the frustration. Begin with the eager model in evaluation mode, and confirm that it produces correct outputs on a fixed set of reference inputs. Save those inputs and outputs; they are your test oracle for every later step. Run torch.compile with fullgraph=True on the model, which turns graph breaks into errors. Each error points to a construct that will also obstruct export: a call into NumPy, a .item() used in Python logic, a print statement, a dictionary keyed by tensor values. Fix them in the model's code. Export with the shapes you need marked dynamic. Fix any data-dependent branching with torch.cond or by restructuring the computation so that it does not depend on values at all, which is often possible; a mask applied with torch.where frequently replaces an if. Compare the exported program's outputs against the reference outputs within a tolerance appropriate to the dtype. Then compile with AOTInductor, or lower to ExecuTorch or ONNX, and compare again. Numerical differences of the order of 1e-5 in float32 are normal when kernels are fused or reordered; differences of the order of 1e-1 are bugs. Finally, benchmark the result on the real target hardware with realistic batch sizes. Ahead-of-time compilation usually reduces latency and removes Python overhead, but the magnitude varies with the model, and the only number that matters is the one measured on your workload. Exporting a model produces a file. Serving it, which means accepting requests, batching them, running the model, returning results and doing so reliably under load, is a separate problem, and the subject of the final chapter. Hashtags: #DeepLearning #PyTorch #PyTorchTutorial #MachineLearning #ArtificialIntelligence #NeuralNetworks #PythonProgramming #TensorOperations #Autograd #AutomaticDifferentiation #NeuralNetworkTraining #CustomNeuralNetworks #PyTorchDataLoader #CustomDatasets #GradientDescent #Backpropagation #ModelOptimization #MixedPrecisionTraining #GradientAccumulation #GradientClipping #GPUComputing #CUDAP rogramming #ModelDebugging #PyTorchProfiler #TorchCompile #ModelCheckpointing #TransferLearning #LoRAFineTuning #ModelDeployment #MLOps

Download the Book (PDF): Introduction In the mid-1990s a team of researchers set out to build a model that would help hospitals decide which patients with pneumonia could safely be sent home and which needed to be admitted. The data were good, the outcome was clear, and the models were accurate. One of them, a set of rules learned from the records, contained a pattern that should have stopped everyone in the room: patients with a history of asthma appeared to have a lower risk of dying from pneumonia than patients without it. Rich Caruana, who later described the episode in a widely read 2015 paper on intelligible models, explained what had happened. The pattern was real. Asthmatic patients who arrived with pneumonia were, as a matter of hospital policy, rushed to intensive care and treated aggressively. That treatment was so effective that their recorded outcomes were better than average. The model had faithfully learned a true association in the data and drawn exactly the wrong conclusion for the question it was meant to answer. Deployed as a triage tool, it would have sent the most vulnerable patients home. Nothing about that model was broken in the sense that a data scientist is trained to detect. It would have scored well on a held-out test set drawn from the same hospitals. It was not overfit, nor was it trained on mislabeled data. Its fault was more fundamental and more common: it had been asked a question about what would happen if a decision were changed, and it had answered a different question about what tends to happen alongside what. The first is a causal question. The second is a predictive one. Machine learning, as most practitioners learn and practise it, is extraordinarily good at the second and has nothing to say, on its own, about the first. This booklet is about the gap between those two questions and the tools that have been built to cross it. The question behind the question Most of the work that data scientists are asked to do is described in predictive language: forecast churn, score credit risk, estimate the probability of readmission, rank products by likely click. But almost none of that work is wanted for its own sake. A churn score exists because someone intends to intervene on the customers it flags. A readmission model exists because a hospital plans to allocate follow-up care. A recommendation ranking exists because a platform will change what people see. In each case the prediction is a means to a decision, and a decision is an intervention on the world. Whether the decision is a good one depends on what would happen under each available action, which is a causal quantity that the predictive model never estimated. Sometimes the gap does no harm. If a model predicts which machines will fail and engineers simply replace those parts first, the prediction and the decision are aligned closely enough that the distinction rarely bites. But the gap widens in exactly the settings where the stakes are highest: where the treatment being considered is itself part of the data the model learned from, where people respond to the system that is scoring them, where the population the model is deployed on differs from the one it was trained on, and where the model's output will be used to explain, justify or contest a decision about a person. In those settings, a model that is only predictive will mislead its users with complete confidence, and no amount of additional data or model capacity will fix it, because the problem is not one of estimation error. It is a problem of asking the data a question it cannot answer without further assumptions. What the booklet argues The controlling idea of what follows is simple to state: causal questions cannot be answered by data alone; they require explicit assumptions about how the data were generated, and the discipline of causal inference is the craft of making those assumptions visible, checking what can be checked, and estimating what follows from them. Directed acyclic graphs are the most practical tool yet invented for making assumptions visible. The potential-outcomes framework and the structural causal model give precise meaning to what an "effect" is. Modern machine learning, far from being displaced by causal inference, turns out to be one of its most powerful engines once the causal question has been correctly posed. That last point matters, because a common misreading of the causal revolution is that it is somehow opposed to machine learning, a return to small models and hand-specified equations. The opposite is true. The most important methodological advances of the last decade, from double machine learning to causal forests to the meta-learners that estimate how effects vary across people, all use flexible predictive models as components. What changed is that the predictive models are now pointed at the right targets, inside an estimation strategy whose validity rests on a causal argument rather than on a leaderboard score. Who this is for and what it covers The reader I have in mind is a working data scientist or machine learning engineer, or someone who manages them, who is fluent with regression and classifiers and has heard the phrase "correlation is not causation" more times than they can count, but who has never been shown what to do about it in a systematic way. No prior training in econometrics or epidemiology is assumed. The mathematics is kept to what is needed to state ideas precisely, and every formal idea is introduced through an example before it is named. The first chapter dissects how predictive models fail when used for decisions, through a set of real cases, and isolates the handful of mechanisms responsible. The second introduces the language that makes causal questions precise: potential outcomes, the ladder that separates seeing from doing from imagining, and the fundamental problem that every causal method must work around. The third and fourth chapters are the heart of the book. They teach the reader to draw directed acyclic graphs, to read off which variables must be adjusted for and which must be left alone, and to recognise the specific traps, such as conditioning on a collider or a mediator, that catch even experienced analysts. The fifth chapter turns an identified causal effect into a number, covering regression adjustment, matching, weighting, and the doubly robust and double machine learning estimators that are now standard. The sixth deals with the uncomfortable but common case in which an important confounder cannot be measured, and surveys the designs that work around it: instrumental variables, regression discontinuity, difference-in-differences and synthetic control, together with sensitivity analysis for when no design is available. The seventh moves from average effects to individual ones, which is where most commercial applications live: uplift modelling, the family of meta-learners, causal forests and the evaluation of targeting policies. The eighth chapter climbs to the top rung of the causal ladder, counterfactual reasoning about specific cases, and shows how it underpins fairness criteria, algorithmic recourse, explanation, and robustness to distribution shift. It also takes an honest look at causal discovery from data and at what large language models can and cannot do with causal questions. The ninth chapter assembles all of this into a working method that a team can adopt: how to frame a question as a hypothetical experiment, how to defend a graph, how to try to break one's own estimate before a stakeholder does, and how to decide when an observational analysis is not good enough and a randomised experiment must be run instead. What it leaves out A booklet earns its usefulness by what it declines to cover. There is no treatment here of the philosophy of causation, of time-series causality in the Granger sense beyond a passing remark, or of the large literature on causal mediation analysis beyond what is needed to avoid its most common mistake. Proofs are omitted; the do-calculus is described, not derived. Software is mentioned where it helps a reader get started, but this is not a tutorial for any particular library, because libraries change faster than the ideas they implement. What the reader should have at the end is a changed habit of mind. Faced with a modelling request, they should be able to ask what decision the model will inform, write down the causal question that decision depends on, draw the assumptions under which that question can be answered from the data available, and say plainly whether those assumptions are credible. That habit, more than any single technique, is what separates a model that describes the past from one that can be trusted to guide action. Chapter 1: The Prediction Trap A predictive model is a compressed description of a joint distribution. Given features, it returns the outcome that tends to accompany those features in the data it was trained on. That is a powerful thing to have, and for many tasks it is exactly the right thing. The trouble starts when the model is used not to anticipate the world but to change it, because the joint distribution the model learned was produced by a particular set of decisions, habits and mechanisms, and the moment you act on the model you alter some of them. This chapter looks at how that happens. The aim is not to catalogue failures for their own sake but to isolate the small number of recurring mechanisms behind them, so that the reader can recognise them before a model ships rather than after. Accurate models, wrong answers Start with a case that has nothing to do with machine learning, because it shows the problem in its purest form. In 1986 a group of urologists published a comparison of treatments for kidney stones in the British Medical Journal. Across all patients, percutaneous nephrolithotomy, the newer and less invasive procedure, had a higher success rate than open surgery: roughly 83 per cent against 78 per cent. But when the patients were split by stone size, open surgery did better for small stones (about 93 against 87 per cent) and better for large stones (about 73 against 69 per cent). The aggregate comparison and the stratified comparisons pointed in opposite directions. This reversal, Simpson's paradox, is often presented as a statistical curiosity. It is better understood as a causal one. Surgeons tended to use open surgery on the harder cases, the large stones, and large stones are harder to treat by any method. Stone size influenced both which treatment a patient received and how likely the treatment was to succeed. A model trained on the pooled data to predict success from treatment would learn, correctly, that patients who received the newer procedure tended to do better. A clinician who used that model to choose a treatment would be choosing worse. Nothing in the data tells you which comparison is the right one for a decision. That depends on knowing that stone size is a cause of treatment choice rather than, say, a consequence of it, and that piece of knowledge comes from outside the data. The pneumonia model described in the introduction is the same structure with the arrows rearranged. A history of asthma caused aggressive treatment, aggressive treatment caused survival, and the model, seeing only asthma and survival, learned that asthma was protective. The treatment was a hidden pathway between a feature and an outcome. Any model trained on outcomes that are themselves shaped by past decisions will absorb those decisions into its picture of risk, and will then recommend undoing the very interventions that made the risk look low. This is the most common way a predictive model misleads a decision-maker, and it has a name: confounding. A confounder is a variable that influences both the thing you might change and the outcome you care about. When it is present and unaccounted for, the association between action and outcome mixes the effect of the action with the effect of whatever drove the action to be taken. The proxy and the target A second mechanism is subtler and arguably more damaging, because it can hide inside a model that is both accurate and free of confounding in the ordinary sense. It arises when the outcome the model is trained to predict is not the outcome the decision is meant to improve. In 2019 Ziad Obermeyer and colleagues published a study in Science of a commercial algorithm used across American health systems to identify patients for high-risk care management programmes. The algorithm predicted future health care costs, on the reasonable-sounding grounds that sicker patients cost more. It predicted costs well. But at a given level of predicted cost, Black patients in the study's data were considerably sicker than white patients, because less money had historically been spent on Black patients with the same conditions, owing to unequal access to care. The algorithm was accurate about the proxy and biased about the target. When the researchers retrained it to predict a direct measure of health need rather than cost, the share of Black patients flagged for extra help rose substantially. The lesson the authors drew, and the one that matters here, is that the choice of label is a causal choice. Costs are produced by a process in which illness is one input and access to care is another. Predicting costs therefore predicts a mixture of need and access. A decision-maker who wants to allocate care by need has asked a question about one cause in a multi-cause system and received an answer about the sum. No regularisation scheme, fairness constraint bolted on afterwards, or larger training set can fix that, because the model is optimising exactly what it was told to optimise. Data scientists encounter versions of this constantly. A fraud model trained on confirmed fraud learns what the investigators chose to investigate. A hiring model trained on past performance ratings learns what past managers rewarded. A content model trained on clicks learns what headlines provoke, not what readers value. In each case the label is the output of a process that includes human decisions, and the model inherits that process wholesale. When people respond to the model A third mechanism appears once a model is deployed into a system that reacts to it. A predictive model treats the world as fixed. Many worlds are not. Consider a subscription business that builds a churn model, ranks customers by their predicted probability of cancelling, and sends a retention offer to the top decile. This is one of the most common uses of machine learning in commerce, and it rests on an unstated assumption: that customers most likely to leave are the ones the offer will most help. That assumption is false in general. Practitioners of uplift modelling divide customers into four rough groups. Some will stay whether or not they receive an offer; spending money on them is waste. Some will leave whatever you do; spending money on them is also waste. Some will stay only if they receive the offer; these are the customers the campaign exists for. And some, the group marketers sometimes call "sleeping dogs", will be more likely to leave if contacted, because the offer reminds them that they have a subscription they had forgotten about or prompts them to shop around. A churn model ranks people by their baseline risk, which mixes all four groups. The customers at the top of the list are disproportionately the lost causes. A campaign targeted by churn score can therefore show disappointing results not because the offer is bad but because the targeting was aimed at the wrong quantity. The quantity that matters for the decision is the difference between what each customer would do with and without the offer, which is an individual causal effect. Chapter 7 is devoted to estimating it. Feedback loops make the problem compound over time. A recommender system shows users the items it predicts they will engage with; users engage with what they are shown; the engagement data are used to retrain the recommender. Items that were never shown never accumulate evidence, and items that were shown early accumulate it disproportionately. The logged data reflect the old policy's choices as much as the users' preferences, and a model trained naively on those logs will mistake the policy's choices for preferences. Predictive policing tools raise a similar concern: patrols sent where a model predicts crime will record more crime there, which the next model reads as confirmation. In all of these cases, the data-generating process includes the model, and the model is blind to its own influence. Researchers have begun to treat this as a problem in its own right. In 2020 Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünner and Moritz Hardt introduced the term performative prediction for settings in which a model's predictions change the distribution of the outcomes they predict. A credit score that determines who gets loans changes who defaults; a traffic forecast that routes drivers away from a road changes the traffic on it. They showed that repeatedly retraining a model on the data its own deployment produces can converge to a stable point that is not the point of best performance, and sometimes fails to converge at all. The predictive workflow has no concept for this. A model is evaluated on a frozen snapshot of a world that, once the model is live, no longer exists. Causal language supplies the missing concept: the model is an intervention, and its effects belong in the analysis. When the world moves The fourth mechanism is the one most familiar to engineers, though it is rarely described in causal terms: a model that worked in one setting fails in another. Google Flu Trends, launched in 2008, estimated influenza activity from search queries. For a while it tracked official surveillance data closely. Then, as David Lazer and colleagues documented in Science in 2014, it substantially overestimated flu prevalence for long stretches, including the 2012–13 season, when its estimates were roughly double the reported level of doctor visits for influenza-like illness at the peak. The model had learned correlations between search terms and flu activity, but the reasons people searched for those terms were not fixed. Media coverage, changes to the search engine itself and seasonal terms that happened to co-occur with winter all moved the searches without moving the illness. The model had no way to distinguish search behaviour caused by being ill from search behaviour merely associated with flu season. Image classifiers show the same weakness at a smaller scale. In a 2018 study, John Zech and colleagues found that convolutional networks trained to detect pneumonia from chest radiographs could identify which hospital an image came from with very high accuracy, and that performance dropped when models were tested at hospitals not seen in training. Because pneumonia prevalence differed across hospitals, a model could improve its scores by learning hospital-specific signatures, such as markers or equipment artefacts, rather than anything about the lungs. Researchers studying camera-trap images have reported analogous problems, with classifiers leaning on background scenery that tends to accompany a species rather than the animal itself, and failing when the same animals appear in unfamiliar locations. In causal language, these models latched onto features that were associated with the label in the training environment through paths other than the mechanism of interest. A radiograph's hospital marker does not cause pneumonia; it is associated with pneumonia because hospitals differ in their patient mix. When the environment changes, those non-causal paths change, and the associations they produced dissolve. Features that cause the label, or are caused by it through a stable mechanism, tend to keep their relationship across environments. This observation, that causal relationships are the invariant ones, turns out to be one of the most productive bridges between causal inference and machine learning, and Chapter 8 returns to it. The mechanisms, named Strip away the details and four mechanisms remain. 1. Confounding. A common cause of an action and an outcome creates an association between them that is not the action's effect. The kidney stones and the asthmatic pneumonia patients are cases of it. 2. Target mismatch. The label being predicted is produced by a process that includes the quantity of interest alongside other causes, so accuracy on the label is not accuracy on the target. The cost-based health algorithm is the canonical case. 3. Intervention and feedback. Acting on a prediction changes the system that generated the data, so the prediction is no longer about the world that will exist after the action. Retention campaigns and recommender loops live here. 4. Non-invariance under shift. Associations that run through unstable, non-causal paths break when the environment changes. Flu Trends and the hospital-identifying image models illustrate it. What these share is that each is invisible to the standard predictive workflow. A held-out test set drawn from the same distribution as the training data cannot reveal confounding, because the confounded association is real in that distribution. It cannot reveal target mismatch, because the model is scored on the very proxy that is mismatched. It cannot reveal feedback, because the test data were collected before the model was deployed. And it cannot reveal non-invariance, because by construction the test environment is the training environment. This is why the problem cannot be solved by better predictive practice. Cross-validation, larger models, more careful hyperparameter search and richer features all improve the estimate of the joint distribution. None of them changes the fact that the joint distribution, however well estimated, underdetermines the answer to a causal question. Two very different causal structures can produce identical observational data. The kidney stone data are compatible with a world in which stone size drives treatment choice and also, in principle, with a world in which treatment choice somehow influenced how stone size was recorded. In the first world you should stratify; in the second you should not. The numbers alone cannot tell you which world you are in. Why "add more features" is not the answer The natural instinct of a machine learning practitioner, on hearing that confounding is the problem, is to include the confounder as a feature. Sometimes that is exactly right. In the kidney stone case, conditioning on stone size recovers the correct comparison. But the instinct generalises badly. Chapter 4 shows that including the wrong variable can create bias where there was none: conditioning on a consequence of the treatment strips out part of the effect you are trying to measure, and conditioning on a common consequence of two variables can manufacture an association between them that does not exist in the population. The predictive habit of throwing every available feature into the model, which is often harmless or helpful for prediction, is actively dangerous for causal estimation. There is a second reason. A model that includes the confounder and the treatment as features will, if the model is flexible enough, give a good prediction of the outcome. But the coefficient or feature importance attached to the treatment in that model is not, in general, an estimate of the treatment's effect. Regularisation shrinks it, correlated features share credit with it, and nonlinear models smear it across interactions. Reading causal effects off a predictive model's internals is one of the most common mistakes in applied data science, and feature attribution methods such as SHAP values, which explain how a model uses its inputs, do not change this. They describe the model, not the world. What a causal question looks like If the predictive workflow cannot answer causal questions, the first task is to recognise when one is being asked. The test is straightforward. A question is causal if answering it requires knowing what would happen under a change that has not, in the data, been made in a controlled way. "Which customers are likely to churn?" is predictive. "Which customers would stay if we offered them a discount?" is causal. "What is the risk of readmission for this patient?" is predictive. "Would a follow-up call reduce this patient's risk of readmission?" is causal. "Which features does the model rely on?" is a question about the model. "Which features, if changed, would change the outcome?" is a question about the world. Most business and policy questions, on inspection, turn out to be causal ones expressed in predictive vocabulary. A data scientist who learns to translate them back, and to say clearly which assumptions are needed to answer them, has already avoided most of the damage described in this chapter. The rest of this booklet supplies the language for that translation and the tools for carrying it out. The next chapter begins with the language, because the first obstacle to answering a causal question is stating it precisely enough that it has an answer at all. Chapter 2: The Language of Effects The phrase "the effect of X on Y" sounds as if it needs no definition. It does. Without a precise definition, two analysts can disagree for weeks about a number without realising that they are estimating different things, and a single analyst can produce an estimate without being able to say what assumptions it depends on. This chapter supplies the vocabulary. It draws on two traditions that developed largely in parallel and are now understood to be closely related: the potential-outcomes framework associated with Jerzy Neyman and Donald Rubin, and the structural approach associated with Judea Pearl. Practitioners today use both, often in the same analysis, and the reader needs both. Potential outcomes Take a single customer, call her Ana, and a single decision: whether to send her a retention discount. Imagine two versions of the next month. In one, Ana receives the discount; in the other, she does not. In each version she either renews her subscription or does not. Write her outcome under the discount as Y(1) and her outcome without it as Y(0). These are her potential outcomes. The effect of the discount on Ana is the difference between them: Y(1) minus Y(0). If she would renew with the discount and cancel without it, the effect is plus one. If she would renew either way, it is zero. If, being reminded of her subscription, she would cancel with the discount and renew without it, it is minus one. This definition is deliberately concrete. It says that an effect is a comparison between two states of the world that differ only in the action taken. It does not mention regression coefficients, correlations or models. It applies to one unit at a time, and it makes the effect a property of that unit rather than of a population. The definition also exposes, immediately, the central difficulty. We will only ever see one of Ana's two potential outcomes. If we send the discount, we observe Y(1) and never learn Y(0); if we do not, the reverse. This is what Paul Holland, in an influential 1986 paper, called the fundamental problem of causal inference: individual causal effects are never directly observed, because each unit experiences only one of its possible treatments. Every causal method is, at bottom, a strategy for filling in the missing potential outcomes using information from other units or other times, and every such strategy rests on an assumption about why the filled-in values can be trusted. From individuals to populations Because individual effects are unobservable, most causal analysis targets averages. The average treatment effect, or ATE, is the average of Y(1) minus Y(0) over a population: the difference between the average outcome if everyone were treated and the average outcome if no one were. The average treatment effect on the treated, or ATT, restricts the average to those who actually received the treatment, and answers a question such as "how much did the discount help the customers we sent it to?" The conditional average treatment effect, or CATE, is the average effect among units sharing some set of characteristics, such as customers in their first year with a particular usage pattern. As the conditioning set becomes richer, the CATE approaches the individual effect, though it never quite reaches it. These distinctions are not pedantry. The ATE and the ATT can differ sharply when the people who were treated are different from the people who were not, which in observational data they almost always are. A job-training programme that enrolls the most motivated applicants may have a large effect on those it enrolled and a small one on the population at large. A decision-maker deciding whether to expand a programme needs a different number from one deciding whether the existing programme was worth its cost. Stating which estimand is wanted, before any data are touched, is the first discipline of causal work. Now consider what the naive comparison actually estimates. Suppose we compare the renewal rate among customers who received a discount with the rate among those who did not. That difference equals the ATT plus a term that is easy to write down and often large: the difference between the treated and untreated groups in what their outcomes would have been without treatment. If the company sent discounts to customers it already thought were at risk, that baseline difference is negative, because the treated group would have renewed less often even without the discount, and the naive comparison will understate the discount's effect, possibly to the point of making it look harmful. This extra term is selection bias, and the whole of observational causal inference can be read as a set of strategies for driving it to zero. A small hypothetical makes the arithmetic concrete. Imagine a company with two kinds of trial customers: engaged ones, who renew 80 per cent of the time without a discount, and disengaged ones, who renew 30 per cent of the time. Suppose the discount raises renewal by ten percentage points for both groups, so the true effect is ten points everywhere. Now suppose the retention team, sensibly from its own point of view, sends the discount to 90 per cent of disengaged customers and only 10 per cent of engaged ones, and that the two groups are the same size. Among customers who received the discount, nine in ten are disengaged, and their renewal rate works out at 45 per cent. Among those who did not, nine in ten are engaged, and their renewal rate is 75 per cent. The naive comparison says the discount reduces renewal by thirty points. Every number in that comparison is accurate, and the conclusion is the opposite of the truth. Splitting by engagement and comparing within each group recovers the ten-point effect in both, and averaging the two within-group effects over the population gives the right answer. Notice what made the correction possible: engagement was recorded, and it was the only thing driving both who received the discount and who renewed. If the team had also used a judgement about each customer's mood on a support call, unrecorded, no amount of stratification on the data would remove the remaining bias. And notice what the correction required that the data could not supply: the knowledge that engagement came before the discount and influenced it, rather than being, say, measured after the discount and partly caused by it. The arithmetic is trivial. The assumptions behind it are the substance. Why randomisation works A randomised experiment drives selection bias to zero by construction. If a coin decides who gets the discount, the treated and untreated groups differ only by chance in every respect, including in their potential outcomes. The average Y(0) in the treated group is, in expectation, the same as the average Y(0) in the control group. The naive comparison then estimates the ATE without bias, and the uncertainty around it can be quantified with ordinary statistics. The technical name for the property that randomisation guarantees is exchangeability: the treated and untreated groups could be swapped without changing the expected outcomes under either treatment. Observational methods try to recover exchangeability by other means. The most common is conditional exchangeability, the assumption that once we account for a set of measured variables, treatment is as good as random. In the kidney stone example, the hope is that within each stone-size group, the choice between procedures was unrelated to how well a given patient would have fared under either. Whether that hope is justified cannot be checked from the data. It is an assumption about the process that assigned treatments, and the next two chapters are about how to reason about it. Two further assumptions sit quietly beneath almost every causal analysis. The first is positivity or overlap: every kind of unit for which we want an estimate must have some chance of receiving each treatment. If the company always sends discounts to customers in their first month and never to anyone else, the data contain no information about what discounts do for older customers, and no method can conjure it. The second is consistency together with the absence of interference, often bundled under Rubin's name as the stable unit treatment value assumption. It says that the treatment is well defined, so that "receiving a discount" means one thing rather than a dozen different offers, and that one unit's treatment does not affect another unit's outcome. That second clause fails in many settings data scientists care about: vaccinations protect the unvaccinated, a price change on one product shifts demand for its substitutes, and a feature launched to some users of a social network alters the experience of their friends. Where interference is present, the potential-outcomes notation must be extended, and the estimand changes. The ladder of causation Pearl, writing with Dana Mackenzie in The Book of Why (2018), organised causal questions into three levels of increasing difficulty, which he called the ladder of causation. The ladder is a useful map, because it makes clear that the gap between prediction and causation is not one gap but two, and that crossing the first does not cross the second. Table 1 lays out the three rungs. Table 1. The three rungs of the ladder of causation. Rung Activity Typical question Formal object What it needs beyond data 1. Association Seeing What does observing X tell me about Y? P(Y given X) Nothing 2. Intervention Doing What would Y be if I set X to x? P(Y given do(X = x)) Assumptions about causal structure, or an experiment 3. Counterfactual Imagining Given what happened, what would Y have been had X been different? P(Y_x given X = x', Y = y') A structural model of the mechanisms The first rung is the territory of standard machine learning. A classifier that predicts readmission from patient characteristics is answering a rung-one question. The second rung introduces the do-operator. P(Y given X = x) is the distribution of Y among units for which X happens to equal x. P(Y given do(X = x)) is the distribution of Y if we intervened to set X to x for everyone, overriding whatever would otherwise have determined it. The two coincide only when nothing that influences X also influences Y through another route, which is precisely the situation that randomisation creates and that observational data rarely provide. The average treatment effect is a rung-two quantity: it is the difference between the mean of Y under do(X = 1) and under do(X = 0). The third rung is the counterfactual about a specific case. "This patient was not given the follow-up call and was readmitted. Would she have been readmitted had she received it?" This question conditions on what actually happened, including the outcome, and then asks about an alternative. It cannot in general be answered from experimental data alone, because a randomised trial tells us about the distribution of outcomes under each treatment but not about how one individual's outcome under treatment relates to the same individual's outcome under control. Rung-three questions arise whenever we assign responsibility, offer an explanation, give someone advice about what they could have done differently, or assess whether a decision about a particular person was fair. Chapter 8 is devoted to them. Structural causal models The potential-outcomes framework defines effects; it says less about how to reason about which assumptions make them estimable. Pearl's structural causal model fills that role. A structural causal model consists of a set of variables, a set of equations that say how each variable is generated from its direct causes plus an unobserved disturbance, and a distribution over those disturbances. For a toy version of the kidney stone case, one equation would say that treatment is determined by stone size and some unmeasured factors such as surgeon preference, and another would say that success is determined by treatment, stone size and unmeasured factors such as the patient's overall health. An intervention, in this model, is surgery on the equations. To compute what would happen under do(treatment = open surgery), we delete the equation that determines treatment and replace it with the constant "open surgery", leaving every other equation intact. The resulting modified model generates the interventional distribution. This is the formal content of the idea that intervening on a variable cuts it off from its usual causes while leaving its consequences in place. It also explains why prediction and intervention diverge: the observational distribution is generated by the full set of equations, including the one for treatment, while the interventional distribution is generated by a model in which that equation has been removed. Potential outcomes fall out of this machinery naturally. Ana's potential outcome Y(1) is the value her outcome equation produces when the treatment is set to one and her own disturbance terms are held at their actual values. Seen this way, the two frameworks are two views of the same object, and modern practice moves between them freely: graphs and structural models to decide what can be identified and how, potential outcomes to define estimands and to build estimators. Identification and estimation are different problems One more distinction organises everything that follows. A causal quantity is identified if it can be written as a function of the distribution of observed variables, given the assumptions we are willing to make. Identification is a question about logic, not about data: it asks whether, with infinite data, we could compute the answer. Estimation is the separate problem of approximating that function from a finite sample. The order matters. If an effect is not identified under credible assumptions, no estimator can rescue the analysis, however sophisticated. A gradient-boosted model with a million rows will produce a confident and precise estimate of the wrong quantity. Conversely, once an effect is identified, the estimation problem is a statistical one, and modern machine learning has a great deal to offer it. Much confused applied work comes from collapsing the two steps: choosing a method, running it, and only afterwards asking what it estimated and why that should be the causal effect. The workflow this booklet teaches keeps them apart. First, state the causal question and the estimand. Second, write down the assumptions about how the data were generated, as a graph. Third, use the graph to decide whether the estimand is identified and, if so, which variables to adjust for. Fourth, estimate. Fifth, probe the estimate for sensitivity to the assumptions that could not be checked. The next two chapters address the second and third steps, which are where most of the intellectual work, and most of the mistakes, are located. Chapter 3: Drawing Your Assumptions Every causal estimate rests on assumptions about how the data were generated. The only choice an analyst has is whether to state those assumptions or leave them implicit. A directed acyclic graph, or DAG, is a way of stating them that is precise enough to calculate with and simple enough for a domain expert with no statistical training to read, criticise and correct. That combination is rare, and it is why DAGs have spread from epidemiology and computer science into economics, the social sciences and, more slowly, industrial data science. This chapter teaches how to read and draw them. The next uses them to decide what to adjust for. Nodes, arrows and what the arrows mean A DAG consists of variables, drawn as nodes, and arrows between them. An arrow from A to B asserts that A is a direct cause of B relative to the other variables in the graph: if we intervened on A while holding every other variable fixed, B could change. "Directed" means every edge has a direction. "Acyclic" means that following the arrows never leads back to where you started: no variable is, through any chain of arrows, a cause of itself. The most important thing to understand about a DAG is that the information is in the arrows that are missing. Drawing an arrow from A to B is a weak claim; it says only that A might affect B, and it allows the effect to be large, small, nonlinear or even zero. Leaving an arrow out is a strong claim; it says that A has no direct effect on B at all. Similarly, leaving out a common cause of two variables asserts that no such cause exists. A graph with every possible arrow drawn makes no assumptions and therefore licenses no conclusions. Causal inference from observational data is possible only to the extent that some arrows can be confidently omitted. DAGs say nothing about functional form. They do not say whether effects are linear, whether they interact, or how large they are. This is a feature: it separates the qualitative structural assumptions, which are usually where domain knowledge is strongest, from the quantitative modelling choices, which can be handled flexibly at the estimation stage. Unmeasured variables belong in the graph too. If you believe that some unrecorded factor, such as a customer's underlying satisfaction or a patient's frailty, influences two variables in the analysis, you draw it as a node, mark it as unobserved, and draw arrows from it to both. A common shorthand is a dashed double-headed arrow between the two affected variables, indicating that they share an unmeasured common cause. Leaving such a factor out of the graph because it cannot be measured is the single most common way that DAGs become dishonest. The graph should describe the world, not the dataset. Three building blocks Any DAG, however large, is built from three elementary patterns connecting three variables. Understanding how information flows through each of them is almost all one needs to reason about the whole graph. The first is the chain: A causes M, which causes B. A fertiliser increases soil nitrogen, which increases yield. In a chain, A and B are associated, because changes in A propagate through M to B. If we condition on M, by looking only at plots with the same nitrogen level, the association between A and B disappears: once we know the nitrogen, knowing about the fertiliser tells us nothing more about the yield. M is a mediator, and conditioning on it blocks the path. The second is the fork: C causes both A and B. Hot weather increases both ice cream sales and swimming. In a fork, A and B are associated even though neither causes the other, because both share a cause. Conditioning on C, by comparing days with the same temperature, removes the association. C is a confounder, and conditioning on it also blocks the path. The third is the collider: A and B both cause K. Talent and luck both contribute to success. In a collider, A and B are not associated through K; information does not flow through a node where two arrows collide. But conditioning on K, or on anything K causes, opens the path and creates an association between A and B. Among successful people, those with less talent must have had more luck, so within that group talent and luck are negatively related, even if they are entirely independent in the population at large. The collider is the pattern that surprises people, and it is responsible for a large family of errors. Joseph Berkson described one in 1946: among hospital patients, two diseases can appear associated simply because having either one increases the chance of being admitted. Selecting on hospitalisation is conditioning on a collider. A similar effect explains why, among published studies, small sample size and large effect size tend to go together: both make a result more likely to clear the bar for publication. Data scientists meet it when they analyse only users who completed a sign-up flow, only loans that were approved, or only patients who survived long enough to be measured. The rules can be summarised in one sentence. Chains and forks transmit association unless you condition on their middle node; colliders block association unless you condition on the collider or one of its descendants. Reading independence off a graph Those three rules generalise to arbitrary graphs through a criterion called d-separation, introduced by Pearl in the 1980s. A path between two variables is any sequence of arrows connecting them, ignoring direction. A path is blocked by a set of conditioning variables if it contains a chain or fork whose middle node is in the set, or a collider that is not in the set and has no descendant in the set. Two variables are d-separated given a set if every path between them is blocked. The payoff is a guarantee. Under the causal Markov condition, which holds for data generated by any structural model matching the graph with independent disturbances, d-separation implies statistical independence. If the graph says A and B are d-separated given C, then A and B must be independent conditional on C in the data, whatever the functional forms involved. This gives DAGs testable consequences. A graph is not a mere sketch of beliefs; it makes predictions about which conditional independences should appear in the data. If the graph implies that region and campaign response are independent given customer tenure, and the data show a strong dependence, the graph is wrong somewhere. Software such as the dagitty package developed by Johannes Textor and colleagues will list all the conditional independences a graph implies, and they can be checked with standard tests. Passing these checks does not prove the graph correct, because many different graphs imply the same independences, but failing them is informative. The converse direction requires an additional assumption, faithfulness: that the independences in the data arise only from the graph's structure, not from coincidental cancellation of effects. If a drug lowers blood pressure directly but raises it by the same amount through a side effect, the two paths cancel and the data will show independence that the graph does not imply. Faithfulness rules such knife-edge cases out. It is usually reasonable, but systems engineered to maintain an equilibrium, such as thermostats or homeostatic biology, can violate it routinely. A worked example Consider a software company that wants to know whether attending a live onboarding webinar increases the probability that a trial user converts to a paid plan. The data contain, for each trial user, whether they attended the webinar, whether they converted, their company size, the acquisition channel they came from, the number of product sessions in their first week, and whether they contacted support. Drawing the graph forces a series of concrete questions, each of which a product manager or sales lead can help answer. Does company size affect webinar attendance? Plausibly yes: larger companies have dedicated staff for evaluating tools. Does it affect conversion? Yes: larger companies have bigger budgets. So company size is a fork between attendance and conversion, a confounder. Does acquisition channel affect attendance? Users from a partner referral may be invited to webinars by the partner. Does it affect conversion directly? Perhaps, through the intent with which users arrive. Another confounder. What about first-week sessions? Here the order of events matters. If sessions are counted over the week in which the webinar took place, then the webinar may well cause more sessions, and sessions cause conversion. Sessions would be a mediator. But a user's underlying enthusiasm for the product, which is not measured, affects both their sessions and their decision to attend. Now sessions lie on a path that runs attendance to sessions to conversion, and also on a path from attendance back to enthusiasm, then forward to sessions. Sessions is a mediator for the first and a collider on the path between attendance and enthusiasm. Support contact may be caused by both attendance (the webinar prompts questions) and difficulties with the product (which also affect conversion), making it another collider. Even this small example makes several points. The graph cannot be drawn from the dataset; it requires knowledge about the timing of events, the incentives of the people involved and the processes that generated each column. The questions it raises are ones that a domain expert can answer and a modeller alone cannot. And the graph already suggests, before any analysis, that adjusting for every available covariate would be a mistake. Including first-week sessions in a regression would strip out part of the webinar's effect and open a biasing path through the unmeasured enthusiasm. The next chapter makes this reasoning systematic. Time, feedback and cycles Real systems contain feedback. Price affects demand and demand affects price. Engagement affects recommendations and recommendations affect engagement. The acyclicity requirement seems to rule such systems out, but in practice it rarely does, because feedback unfolds over time. Price this week affects demand this week, which affects price next week. Unrolled in time, the cycle becomes a chain of time-indexed variables, and the graph is acyclic again. The cost is that the graph becomes larger and that the data must record the timing of events finely enough to respect it. When they do not, as with a monthly aggregate that mixes cause and consequence, the honest response is to acknowledge that the causal order within the period cannot be resolved. Time ordering is also the most reliable source of omitted arrows. A variable measured after the outcome cannot cause it. A variable fixed before the treatment was assigned cannot be caused by it. Building a graph by first laying the variables out on a timeline, and then asking which earlier variables plausibly affect each later one, is the most practical method most teams will find. Patterns that recur in operational data A few structures appear so often in the data that organisations collect that they are worth recognising on sight. The first is the proxy. Many important variables are not measured directly but through an imperfect indicator: income through postcode, health through prior spending, engagement through logins. In a graph, the true variable is an unobserved node and the proxy is its child. Adjusting for the proxy only partly blocks the paths that run through the true variable, so some confounding remains, and the amount depends on how noisy the proxy is. Drawing the true variable and its proxy as separate nodes keeps this residual confounding visible rather than letting the proxy's presence in the model suggest the problem has been solved. The second is selection into the dataset. Almost every analytical table has been filtered: active users, completed transactions, patients with a recorded follow-up visit. It is good practice to add a node for "included in the data", draw the arrows into it, and remember that every analysis implicitly conditions on it. If treatment or its causes, and outcome or its causes, both affect inclusion, that node is a collider and the dataset itself is biased before any modelling begins. The third is treatment–confounder feedback over time. A patient's blood pressure influences whether a drug is prescribed this month; the drug influences blood pressure next month; next month's blood pressure influences the next prescription and also the eventual outcome. Blood pressure is a confounder for later treatment decisions and a mediator of earlier ones. No single adjustment set handles this correctly in an ordinary regression: including blood pressure blocks part of the earlier treatment's effect, and excluding it leaves the later treatment confounded. Robins developed the g-methods, including marginal structural models fitted with inverse probability weights, precisely for this structure. Data scientists meet it in any system where a model's past decisions shape the features that inform its future ones, such as dynamic pricing, adaptive tutoring and credit-limit management. Recognising the pattern in the graph is the step that tells an analyst that standard tools will fail. Where graphs come from and how to defend them The commonest objection to DAGs is that they are subjective. The graph encodes the analyst's beliefs, and different analysts will draw different graphs. This is true, and it is the point. Every causal analysis of observational data makes structural assumptions; a regression with a chosen set of controls is equivalent to a graph in which those controls block all confounding paths. The difference is that a regression hides its assumptions in the choice of covariates, while a graph displays them so that they can be disputed. An analyst who presents a DAG is inviting criticism of exactly the right kind: not "your model is too simple" but "you have missed a common cause of these two variables" or "this arrow points the wrong way because the measurement was taken earlier". Good graphs are built collaboratively. A data scientist brings knowledge of what is recorded and how; a domain expert brings knowledge of how decisions are actually made. The sales lead knows which customers get invited to webinars. The clinician knows what drives a referral. The operations manager knows which warehouse uses which supplier and why. Sessions in which a graph is drawn on a whiteboard with these people present are among the highest-value hours a causal analysis contains, because they expose assumptions that would otherwise be buried in a covariate list. When experts disagree, the disagreement can often be localised to a particular arrow, and the analysis can be run under each version. If the estimate is similar either way, the disagreement does not matter for the decision. If it is very different, the team has learned precisely which fact about the world it needs to establish, and can sometimes find data or run a small experiment to settle it. Either outcome is more useful than an estimate whose dependence on unstated assumptions is unknown. A graph is never proved. It is defended: by appeal to timing, to known mechanisms, to the way the data were collected, to the testable independences it implies, and to the robustness of conclusions across plausible alternatives. That is a lower standard than proof and a higher one than the implicit assumptions behind most predictive models used for decisions. Having drawn a graph, the analyst is ready for the question that matters most in practice: given this structure, what should be adjusted for? Chapter 4: What to Adjust For, and What to Leave Alone The question that most often decides whether an observational analysis is right or wrong is deceptively mundane: which variables should be included as controls? In predictive modelling, the answer is "whatever improves out-of-sample accuracy". In causal estimation, that answer is not just unhelpful but frequently harmful. Some variables must be included to remove bias. Some must be excluded because including them creates bias. Some are harmless either way, and a few improve precision without affecting bias at all. The graph tells you which is which. This chapter shows how. The backdoor criterion The effect of a treatment T on an outcome Y travels along the directed paths that start at T and follow arrows forward to Y. These are the causal paths, and an analysis that wants the total effect of T should leave them all open. Every other path between T and Y is a source of non-causal association. The most important class of such paths are backdoor paths: paths that begin with an arrow pointing into T. A backdoor path exists whenever something that causes T is also connected to Y by a route that does not pass through T. The fork through a confounder is the simplest case. Pearl's backdoor criterion says that a set of variables Z is sufficient to identify the causal effect of T on Y if two conditions hold. First, no variable in Z is a descendant of T. Second, Z blocks every backdoor path between T and Y. When both hold, conditional exchangeability holds: within levels of Z, the treatment is as good as randomised with respect to the outcome, and the interventional distribution can be computed by stratifying on Z and averaging the stratum-specific comparisons over the population distribution of Z. This averaging step is called the adjustment formula, and it is what regression adjustment, matching and weighting all try to approximate. The two conditions correspond to the two errors that analysts most often make. Failing to block a backdoor path leaves confounding in place. Including a descendant of the treatment either blocks part of the causal effect or opens a new non-causal path. Both are made visible by the graph. The criterion does not say that there is one correct adjustment set. Often there are several, and they can be chosen on practical grounds. If a confounder is poorly measured, a different set that blocks the same paths through better-measured variables may be preferable. Tools such as dagitty enumerate the minimal sufficient adjustment sets for a graph, which is especially useful when the graph is too large to reason about by eye. The variable roles It helps to classify each candidate covariate by its position relative to the treatment and outcome, because each position carries a clear recommendation. Table 2 sets out the roles that matter most in practice. Table 2. Roles a covariate can play and whether to adjust for it when estimating a total effect. Role Position in the graph Adjust? Consequence of getting it wrong Confounder Common cause of treatment and outcome Yes Omitting it leaves confounding bias Mediator On a causal path from treatment to outcome No Including it removes part of the effect Collider Common effect of treatment (or its causes) and outcome (or its causes) No Including it creates spurious association Descendant of outcome Caused by the outcome No Including it biases the estimate, often severely Outcome predictor Causes outcome, unrelated to treatment Optional, usually yes Omitting it costs precision only Treatment predictor Causes treatment, unrelated to outcome except through it Usually no Including it inflates variance and can amplify residual bias The confounder row is the familiar one. The mediator row is the one most often violated in industrial settings. If the question is whether a new onboarding flow increases twelve-month retention, and the analysis controls for first-month engagement, it will estimate only the part of the flow's effect that does not work through engagement. Since improving engagement is probably the main way an onboarding flow works, the adjusted estimate may be close to zero even when the total effect is large. The analysis has answered a different question, about the direct effect, which is rarely the one anyone asked. The outcome-predictor row explains a common source of confusion. A variable that strongly predicts the outcome but has nothing to do with treatment assignment, such as a customer's region in a randomised test, does not need to be adjusted for to remove bias. Including it is nonetheless worthwhile, because it explains outcome variance and so tightens confidence intervals. This is why adjusting for pre-experiment covariates in A/B tests, as in the CUPED method popularised by Microsoft's experimentation team, can substantially reduce the sample size needed to detect a given effect without introducing bias. The last row is the least intuitive. A variable that affects treatment but not outcome, such as a random-looking quirk in which sales representative handled a lead, is not a confounder. Including it does not reduce bias. It reduces the variation in treatment left over after adjustment, which increases the variance of the estimate. Worse, if there is any unmeasured confounding, conditioning on a strong predictor of treatment tends to amplify the bias that remains, a phenomenon studied by Pearl and others under the name bias amplification. These are the same variables that, used differently, can serve as instruments, as Chapter 6 explains. Collider bias in the wild Collider bias deserves its own section because it is so common and so often disguised. It appears in three forms. The first is explicit conditioning on a collider as a covariate. Return to the webinar example from Chapter 3. Support contact is caused by webinar attendance and by product difficulties, and product difficulties also affect conversion. Including support contact in the model opens the path from attendance through support contact to product difficulties and on to conversion. Among users who contacted support, those who did not attend the webinar are more likely to have had product difficulties, because something had to prompt their call. The adjusted comparison will therefore favour the webinar, even if it does nothing. The second form is selection. If the analysis dataset includes only units that passed some filter, and the filter depends on both the treatment and the outcome or their causes, the analysis is conditioning on a collider whether or not anyone intended it. Data scientists encounter this whenever they analyse logged-in users, approved applications, completed purchases, retained employees or patients who reached a particular stage of care. Studies of the effect of a credit policy that use only approved loans, for example, can find associations between applicant characteristics and default that are artefacts of the approval rule. A celebrated example from epidemiology shows how strange the consequences can look. Infants of mothers who smoked during pregnancy have, on average, lower birth weight and higher mortality. Yet among infants with low birth weight, those born to smokers have lower mortality than those born to non-smokers. For years this "birth weight paradox" invited speculation that smoking was somehow protective for small babies. In 2006 Sonia Hernández-Díaz, Enrique Schisterman and Miguel Hernán gave the causal explanation in the American Journal of Epidemiology. Low birth weight can be caused by smoking or by other, often more serious, conditions such as birth defects, which also raise mortality. Conditioning on low birth weight, a collider, induces a negative association between smoking and those other causes. A low-birth-weight baby of a non-smoker is more likely to be small because of something more dangerous. The paradox dissolves once the graph is drawn. The third form is conditioning on a descendant. Even if the collider itself is not in the model, a variable caused by the collider carries some of its information and partially opens the path. A post-treatment measure of customer satisfaction, which is influenced by both treatment and the customer's underlying experience, is a typical example. M-bias and the limits of "control for everything pre-treatment" A popular heuristic says that any variable measured before treatment is safe to adjust for, because it cannot be a mediator or a descendant of treatment. This heuristic is mostly right, which is why it persists, but it is not entirely right. The exception is called M-bias, after the shape of the graph that produces it. Suppose an unmeasured variable U1 affects treatment, and a separate unmeasured variable U2 affects the outcome, and both U1 and U2 affect a measured pre-treatment variable M. There is no confounding between treatment and outcome: the path from treatment back to U1, forward to M, back to U2 and forward to outcome is blocked at M, which is a collider. Adjusting for M opens that path and introduces bias, even though M was measured before treatment and is associated with both treatment and outcome. How much this matters in practice is debated. Simulation studies suggest that M-bias is usually small relative to the confounding bias one risks by omitting a genuine pre-treatment confounder, and many methodologists, including Rubin, have argued that adjusting for pre-treatment covariates remains the right default. The fair summary is that the heuristic is a reasonable fallback when the graph is uncertain, but it does not replace drawing the graph, and an analyst who has specific reason to think a pre-treatment variable is a collider should not include it. When the backdoor is shut: the front door and beyond Sometimes a backdoor path cannot be blocked because the confounder on it is unmeasured. That does not always make the effect unidentifiable. Pearl's front-door criterion offers an alternative when the treatment affects the outcome entirely through a measured mediator, the mediator is not affected by the unmeasured confounder, and the treatment-to-mediator and mediator-to-outcome relationships can themselves be identified. Pearl's textbook illustration is smoking, tar deposits and lung cancer, under the hypothetical that an unmeasured genotype affects both smoking and cancer but not tar: one could estimate the effect of smoking on tar, the effect of tar on cancer adjusting for smoking, and chain them together. Real cases that satisfy the front-door conditions exactly are uncommon, but the criterion matters for two reasons. It shows that the backdoor criterion is not the only route to identification, and it occasionally fits practical problems well, for instance when a treatment can only act through a logged intermediate step such as a user actually viewing a recommended item. More generally, Pearl's do-calculus is a set of three rules for transforming expressions containing the do-operator into expressions containing only ordinary conditional probabilities. In 2006, Ilya Shpitser and Pearl, and independently Yimin Huang and Marco Valtorta, proved that the do-calculus is complete: if a causal effect is identifiable from a given graph, the rules will find an expression for it, and if they cannot, no method can. This result turns identification into an algorithmic question. Given a graph, software can determine whether an effect is identified and produce the formula to estimate. Libraries such as DoWhy, maintained under the open-source PyWhy organisation, expose this capability to practitioners without requiring them to manipulate the rules by hand. The table two fallacy One final error belongs in this chapter because it arises from correctly adjusting for confounders and then misreading the result. In a regression of outcome on treatment and an adjustment set chosen by the backdoor criterion, the coefficient on treatment can be interpreted causally, subject to the usual modelling caveats. The coefficients on the adjustment variables generally cannot. Daniel Westreich and Sander Greenland named this the table two fallacy in 2013, after the habit in medical papers of presenting a single table of adjusted coefficients and interpreting each one as an effect. The reason is straightforward once the graph is in view. The adjustment set was chosen to block backdoor paths from the treatment. It was not chosen to block backdoor paths from each covariate, and it may well contain mediators of a covariate's effect. In the webinar analysis, company size is a confounder of attendance and conversion, but attendance is a mediator of company size's effect on conversion, so the coefficient on company size in a model that includes attendance is a partial, direct effect at best. Each causal question needs its own adjustment set. A model built to estimate one effect should not be mined for others. The same caution applies with even more force to feature importance scores from machine learning models. A gradient-boosted model of conversion that includes every available variable is not an adjustment for anything in particular, and its importance rankings reflect predictive usefulness, which bears no fixed relationship to causal influence. Mediators and colliders can be among the most important predictors precisely because they carry information about the outcome through non-causal routes. Working through a covariate list It helps to see the classification applied to the kind of feature table a data scientist actually inherits. Suppose a subscription company wants the total effect of a price increase, applied last quarter to some customers on legacy plans, on twelve-month retention. The feature store offers a long list of columns. Going through them with the graph in mind sorts them quickly. Plan type, tenure at the time of the price change, region and historical usage up to the change date are pre-treatment variables that influenced which customers were moved to the new price, because the company targeted legacy plans in certain markets, and they plausibly affect retention. They are confounders and belong in the adjustment set. Support tickets filed in the month after the price change are caused partly by the change, as customers write in to complain, and partly by underlying dissatisfaction, which also drives cancellation. They are a collider and must be excluded. Usage in the quarter after the change is a mediator, since customers who reduce usage in response to the higher price are more likely to cancel later; including it would remove the part of the effect that runs through disengagement. Whether the customer accepted a downgrade offer is a consequence of the price change and strongly predicts retention; it, too, is a mediator. A churn propensity score computed by an existing model last month uses only pre-change features and can be treated as a summary of confounders, provided its inputs are known. A churn score recomputed this month cannot, because it incorporates post-change behaviour. The exercise usually takes an hour with someone who knows how the columns were built, and it often removes a third of the candidate features from the adjustment set. Those removed features are frequently the strongest predictors of the outcome, which is exactly why a predictive mindset would have kept them. A procedure Putting this together gives a procedure that any data scientist can follow before fitting a model for a causal question. Write down the treatment, the outcome and the estimand. Draw the graph, including unmeasured variables, with timing as the backbone. Identify all backdoor paths from treatment to outcome. Choose an adjustment set that blocks them without including descendants of the treatment, preferring well-measured variables and including strong predictors of the outcome for precision. Check that the set does not open any collider path. If no such set exists among measured variables, look for a front-door structure or one of the designs in Chapter 6, and if none applies, say so plainly. Only then move on to estimation, which is the subject of the next chapter. Chapter 5: From Identification to Estimate Once the graph has done its work, the analyst knows which variables must be adjusted for and has a formula, the adjustment formula, that expresses the causal effect in terms of observable quantities. What remains is statistical: turning that formula into a number from a finite and imperfect dataset. This is the stage at which machine learning, having been warned off answering causal questions directly, returns as an indispensable tool. But it returns in a particular role, and the way it is used matters as much as which algorithm is chosen. This chapter covers the main families of estimators, the reasons modern practice has converged on combining them, and the diagnostics that tell an analyst whether an estimate deserves to be believed. Modelling the outcome The most direct approach models the outcome. Fit a model that predicts the outcome from the treatment and the adjustment set. Then, for every unit in the population, predict its outcome twice: once with the treatment set to one and once with it set to zero, leaving each unit's covariates as they are. The average difference between the two predictions estimates the average treatment effect. This procedure is known in epidemiology as g-computation or the parametric g-formula, from James Robins's work in the 1980s, and in machine learning circles it is sometimes called the S-learner, for reasons Chapter 7 will make clear. When the outcome model is a linear regression with no interactions, g-computation reduces to reading off the treatment coefficient, which is why ordinary regression adjustment is the method most analysts already know. The generalisation matters because the linear model's assumptions are strong. If the treatment effect differs across subgroups, or the covariates affect the outcome nonlinearly, a misspecified linear model can produce a biased average effect even when every confounder has been included. Replacing it with a flexible learner such as gradient boosting or a neural network reduces that risk. It introduces a different one. Flexible learners are tuned to predict well, and regularisation is how they do it. Regularisation shrinks coefficients and simplifies functions, and it does not know that one input, the treatment, is the object of interest. A tree ensemble may rarely split on the treatment if other features explain more variance, producing an effect estimate shrunk towards zero. This regularisation bias is a general hazard of plugging predictive models into causal formulas, and the more sophisticated estimators described below were developed largely to neutralise it. Modelling the treatment The alternative is to model how treatments were assigned. The propensity score, introduced by Paul Rosenbaum and Donald Rubin in 1983, is the probability that a unit receives treatment given its covariates. They proved a result that made it central to applied work: if adjusting for a set of covariates removes confounding, then adjusting for the propensity score alone also removes it. A high-dimensional adjustment problem collapses to a one-dimensional one. The score can be used in several ways. Matching pairs each treated unit with one or more untreated units with a similar score and compares their outcomes. Stratification divides units into bins by score and averages within-bin comparisons. Inverse probability weighting, or IPW, reweights every unit by the inverse of the probability of the treatment it actually received. A treated unit that had only a 10 per cent chance of being treated receives a weight of ten, because it stands in for nine similar units that were not treated. The reweighted sample behaves, in expectation, like a population in which treatment was assigned at random, and a simple difference in weighted means estimates the effect. The appeal of modelling the treatment is that the assignment mechanism is sometimes better understood than the outcome. A company knows the rules its marketing system used to choose whom to target, and those rules can be modelled closely even when the drivers of customer behaviour are murky. The weakness is fragility when scores approach zero or one. A handful of units with tiny estimated propensities can receive enormous weights and dominate the estimate, inflating variance and magnifying any error in the propensity model. Common remedies include stabilised weights, which normalise by the marginal probability of treatment, and trimming or truncating extreme weights. Each trades a little bias for a lot of variance reduction, and each changes the population to which the estimate applies. Checking overlap and balance Extreme propensities are more than a numerical nuisance; they are a signal that the positivity assumption from Chapter 2 is under strain. If some kinds of customers almost always receive the discount and others almost never do, the data contain little or no information about what would have happened to them otherwise, and every estimator, not only IPW, must extrapolate. Outcome models do this silently by projecting their fitted function into regions with no data; weighting estimators do it noisily through huge weights. The honest response is to look before estimating. Two diagnostics are standard. The first is to plot or tabulate the distribution of estimated propensity scores separately for treated and untreated units and to inspect where they fail to overlap. Units outside the region of common support may need to be excluded, with the estimand redefined accordingly, a choice that should be reported rather than buried. The second is to check covariate balance after matching or weighting. For each covariate, compute the standardised mean difference between treated and untreated groups in the adjusted sample. A common rule of thumb treats absolute differences below 0.1 as acceptable. Balance checking has a virtue that is easy to overlook: it evaluates the propensity model on the criterion that actually matters, whether the adjusted groups look alike, rather than on predictive accuracy. A propensity model that predicts treatment extremely well can be a warning sign, since near-perfect prediction implies near-zero overlap. Doubly robust estimation Outcome modelling is biased if the outcome model is wrong. Propensity weighting is biased if the propensity model is wrong. A family of estimators developed by James Robins, Andrea Rotnitzky and Lue Ping Zhao in the 1990s combines the two in a way that is consistent if either model is correctly specified. These are called doubly robust estimators, and the most widely used is augmented inverse probability weighting, or AIPW. The intuition is a correction. Start with the outcome model's prediction of each unit's potential outcomes. Then, for units whose actual treatment matches the potential outcome being estimated, compute the residual between the observed outcome and the prediction, and add it back, weighted by the inverse propensity. If the outcome model is right, the residuals average to zero and the correction does nothing harmful. If the outcome model is wrong but the propensity model is right, the weighted residuals exactly repair the outcome model's systematic error. Only if both models are wrong does bias remain, and even then the bias depends on the product of the two models' errors, which tends to be smaller than either alone. That product structure turns out to be the key to combining causal estimation with machine learning, and it leads directly to the most important methodological development in the field over the past decade. Double machine learning In 2018, Victor Chernozhukov and colleagues published "Double/debiased machine learning for treatment and structural parameters" in The Econometrics Journal, setting out a general recipe for using arbitrary machine learning models to estimate causal parameters while still obtaining valid confidence intervals. The recipe, now usually called double machine learning or DML, rests on two ideas. The first is Neyman orthogonality. Some estimating equations are insensitive, to first order, to small errors in the nuisance models: the outcome and propensity models whose job is to remove confounding but which are not themselves of interest. The AIPW estimating equation is one of these. Because errors in the nuisance models enter only as products, an estimator built on an orthogonal equation can tolerate nuisance models that converge relatively slowly, as flexible machine learning models typically do in high dimensions, and still produce an effect estimate that converges at the usual parametric rate with an approximately normal distribution. The second is cross-fitting. Flexible models overfit, and if the same data are used to fit the nuisance models and to compute the effect, the overfitting leaks into the estimate. Cross-fitting splits the data into folds, fits the nuisance models on all folds but one, computes the contribution of the held-out fold using those models, rotates through the folds, and averages. Every unit's contribution is computed from models that never saw it. In the simplest partially linear version, DML is easy to describe without formulas. Use any learner to predict the outcome from the covariates, and take the residuals. Use any learner to predict the treatment from the covariates, and take those residuals too. Regress the outcome residuals on the treatment residuals. The slope is the effect estimate. The procedure removes from both treatment and outcome everything the covariates can explain, then asks how the unexplained part of the treatment relates to the unexplained part of the outcome. This "residual-on-residual" construction descends from a classical result in econometrics, the Frisch–Waugh–Lovell theorem, with the linear projections replaced by machine learning and protected by cross-fitting. DML has become a default for many applied teams because it lets them use the flexible models they already know, applied to the parts of the problem where flexibility helps, without losing the ability to report a confidence interval they can defend. Microsoft's EconML library and the DoubleML package, both open source, implement it with interfaces that will feel familiar to anyone who has used scikit-learn. Several practical choices affect how well DML works. The nuisance learners should be chosen on the criterion of predicting their own targets well, outcome from covariates and treatment from covariates, and they can be tuned by ordinary cross-validation; there is no need for the same algorithm to be used for both. Five folds is a common default for cross-fitting, and because the split is random, repeating the procedure with several different splits and taking the median estimate reduces sensitivity to an unlucky partition. When the treatment is binary, the partially linear model can be replaced by the fully interactive AIPW form, which does not assume the effect is constant across units. And the diagnostics from earlier in this chapter still apply: a propensity model that predicts treatment almost perfectly is a sign that overlap is poor, and DML offers no protection against that. The residual-on-residual regression will simply have very little residual variation in the treatment to work with, and the confidence interval will widen accordingly, which is the method honestly reporting that the data contain little information. It is also worth being clear about what DML does not do. It does not choose the adjustment set. It does not detect unmeasured confounding. It does not make an observational study equivalent to an experiment. It solves one specific problem, the bias and invalid inference that arise when flexible predictive models are used to adjust for many covariates, and it solves that problem well. A related approach, targeted maximum likelihood estimation, or TMLE, developed by Mark van der Laan and colleagues, reaches similar theoretical guarantees by a different route: it fits an initial outcome model and then updates it in a targeted step that uses the propensity score to remove bias for the specific parameter of interest. TMLE is especially common in biostatistics, often paired with Super Learner, an ensemble method from the same group that combines many candidate algorithms by cross-validated weighting. Comparing the estimators The families differ in what they model, what they assume and how they fail. Table 3 summarises the trade-offs. Table 3. Main estimator families for an identified average effect. Estimator What it models Valid if Main failure mode Typical use Regression adjustment / g-computation Outcome Outcome model correct Silent extrapolation; regularisation bias Strong outcome knowledge, good overlap Matching Treatment (via score or distance) Matching variables remove confounding; good matches exist Poor matches, discarded data Small studies, ATT, transparent reporting Inverse probability weighting Treatment Propensity model correct Extreme weights, high variance Known assignment rules AIPW (doubly robust) Both Either model correct Both models wrong; poor overlap General default Double machine learning Both, with cross-fitting Nuisance models converge reasonably well Poor overlap; weak learners High-dimensional covariates, flexible models TMLE Both, with targeted update Similar to DML Poor overlap Biostatistics, with ensemble learners None of these estimators can compensate for a wrong adjustment set. Every row of the table assumes that the identification step was done correctly. Choosing among them is a matter of statistical efficiency and robustness to modelling error; it is not a way to repair a missing confounder or an included collider. The most sophisticated estimator in the world, applied to a model that controls for a mediator, will estimate the wrong thing very precisely. A cautionary benchmark The value of these methods, and their limits, were sharply illustrated by a line of research that began in 1986, when Robert LaLonde compared experimental and non-experimental estimates of the effect of a job-training programme, the National Supported Work Demonstration. The experiment provided a credible benchmark. LaLonde then discarded the experimental control group, replaced it with comparison samples drawn from large national surveys, and applied the econometric adjustment methods then in common use. The resulting estimates varied widely and were often far from the experimental benchmark. His paper became one of the most cited arguments for randomised evaluation. In 1999 Rajeev Dehejia and Sadek Wahba revisited a subset of the data and showed that propensity score methods, applied with careful attention to overlap, could recover estimates close to the experimental ones. Later work, notably by Jeffrey Smith and Petra Todd, showed that this success was sensitive to the choice of sample and specification. The fair reading of the whole exchange is neither that observational methods work nor that they fail. It is that they work when the adjustment variables genuinely capture the selection process, here prior earnings histories, which strongly predicted both enrolment and later earnings, and when the comparison group overlaps the treated group, and that they fail otherwise. Method choice mattered far less than whether the right variables had been measured on comparable populations. That lesson generalises. An analyst who spends a week tuning a DML pipeline and an hour on the graph has the priorities reversed. Uncertainty and reporting A causal estimate without a measure of uncertainty is incomplete. For doubly robust and DML estimators, analytic standard errors based on the influence function are available and are usually reported by the software. For matching and for more elaborate pipelines, the bootstrap is the general-purpose fallback, provided the whole procedure, including propensity estimation and any trimming, is repeated within each bootstrap sample. Statistical uncertainty, however, is only the part of the uncertainty that the data can measure. The larger part usually lies in the assumptions: an unmeasured confounder, a misplaced arrow, a positivity violation papered over by trimming. A confidence interval says nothing about these. A good report therefore pairs the estimate and its interval with an explicit statement of the identifying assumptions, the diagnostics for overlap and balance, and a sensitivity analysis that asks how strong a violation of those assumptions would have to be to change the conclusion. That last tool belongs to the next chapter, which takes up the situation every observational analyst eventually faces: the confounder that cannot be measured. Chapter 6: When the Confounder Is Missing Every method in the previous chapter depends on one assumption that cannot be tested from the data: that all the important confounders have been measured. Sometimes that assumption is reasonable. A company that assigned discounts using a known rule based on recorded customer attributes has, in effect, measured the whole assignment mechanism. More often it is not. Patients are treated on the basis of clinical impressions that never reach the database. Customers opt into programmes for reasons no form recorded. Managers deploy new tools first to the teams they expect to succeed. In such cases adjustment alone cannot remove the bias, and the analyst must either find a different source of variation in the treatment or quantify how much the unmeasured confounding could matter. The designs in this chapter come mostly from economics, where the inability to run experiments on national policies forced researchers to become ingenious about finding natural ones. Their development was recognised in 2021, when the Nobel memorial prize in economics was awarded to David Card for empirical work using natural experiments, and jointly to Joshua Angrist and Guido Imbens for their methodological contributions to the analysis of causal relationships. Each design trades the assumption of no unmeasured confounding for a different assumption, and the analyst's job is to judge whether the trade is a good one in the case at hand. Instrumental variables An instrument is a variable that shifts the treatment but has no other route to the outcome. If such a variable exists, the portion of variation in the treatment that it induces is free of confounding, because the instrument itself is unrelated to the confounders. Comparing how the outcome moves with the instrument to how the treatment moves with the instrument isolates the treatment's effect. In graph terms, an instrument Z must satisfy three conditions. It must have an arrow to the treatment; this is relevance, and it can be checked in the data. It must share no unblocked path with the outcome other than through the treatment; there must be no common causes of instrument and outcome, a condition sometimes called independence. And it must have no direct arrow to the outcome; this is the exclusion restriction. The last two are assumptions about the world, and they carry the entire weight of the analysis. Classic examples show both the power and the fragility of the idea. Angrist used the Vietnam-era draft lottery, in which birth dates were drawn at random to determine draft eligibility, as an instrument for military service when estimating its effect on later earnings. The lottery was genuinely random, which secured independence, and it plainly affected the probability of serving. The exclusion restriction required that draft eligibility affected earnings only through service, which some critics questioned on the grounds that men with low lottery numbers might have stayed in education longer to obtain deferments. Angrist and Alan Krueger later used quarter of birth as an instrument for years of schooling, exploiting the interaction between birthday cut-offs for school entry and compulsory schooling laws. John Bound, David Jaeger and Regina Baker showed in 1995 that because quarter of birth was only weakly related to schooling, tiny violations of the exclusion restriction or small-sample biases could produce large errors in the estimates. Weak instruments have been a central concern of the method ever since. A second subtlety concerns what an instrumental variable estimate means when effects vary across people. Imbens and Angrist showed in 1994 that under a monotonicity assumption, that the instrument never pushes anyone away from treatment, the IV estimate recovers the local average treatment effect: the average effect among compliers, the units whose treatment status is changed by the instrument. It does not, in general, recover the ATE. If those induced to serve by the draft lottery differed systematically from career soldiers or from those who avoided service under any lottery number, the IV estimate applies to the former group only. This is not a flaw but a precise statement of what the data can reveal, and it matters when the estimate is used for a decision affecting a different population. In industry, the most useful instruments are usually manufactured. A randomised encouragement design randomises an invitation, a nudge or a default rather than the treatment itself, because the treatment cannot be forced. A software company that wants the effect of using a new feature, which users choose freely, can randomly vary whether the feature is highlighted on launch. The highlight is an instrument for use. Randomisation secures independence, and the exclusion restriction requires only that the highlight does not affect the outcome except by changing whether people use the feature, which is often plausible and can be argued explicitly. The same logic underlies the analysis of A/B tests with non-compliance, where some users assigned to treatment never see it. In genetics, the equivalent is Mendelian randomisation, which uses genetic variants that are allocated at conception and influence an exposure, such as cholesterol levels, as instruments for that exposure's effect on disease. Regression discontinuity Many decisions are made by applying a threshold to a score. Students above a test score receive a scholarship. Loan applicants above a credit score are approved. Patients with a measured value above a guideline cut-off receive a drug. Accounts with more than a given number of flagged events are sent to a fraud review team. Wherever a rule like this is applied, there is a natural experiment at the threshold. The regression discontinuity design, first proposed by Donald Thistlethwaite and Donald Campbell in 1960 in a study of merit awards, compares units just above the cut-off with units just below it. If units cannot precisely control their position relative to the threshold, those just either side should be similar in every respect except treatment, so a jump in the outcome at the threshold estimates the treatment's effect there. In the sharp version, the threshold determines treatment completely. In the fuzzy version, it changes the probability of treatment, and the threshold serves as an instrument. The key assumption is that nothing else changes discontinuously at the threshold, and in particular that units are not sorting themselves across it. If applicants know the credit cut-off and can game their scores, or if loan officers nudge borderline cases over the line, the units just above the cut-off are no longer comparable to those just below. Researchers check this by examining whether the density of the running variable jumps at the threshold, a test associated with Justin McCrary, and whether pre-treatment covariates are smooth across it. A regression discontinuity estimate also applies only to units near the threshold. It says what the scholarship did for students who just qualified, not for those who qualified easily. For data scientists, regression discontinuity is under-used. Business rules with hard thresholds are everywhere in production systems, and every one of them is a candidate natural experiment for evaluating the treatment it triggers. The main practical obstacles are that thresholds are often changed, that several rules may share the same cut-off, and that there may be too few observations near the threshold for precise estimates. Difference-in-differences When a treatment is applied to some groups at a point in time and not to others, the difference-in-differences design compares how outcomes changed in the treated groups with how they changed in the untreated ones. The first difference, before versus after, removes stable differences between groups. The second difference, treated versus untreated, removes changes over time that affected everyone. What remains is attributed to the treatment. The canonical modern example is Card and Krueger's 1994 study of a minimum wage increase in New Jersey. They surveyed fast-food restaurants in New Jersey and in neighbouring eastern Pennsylvania, where the minimum wage did not change, before and after the increase, and found no evidence that employment fell in New Jersey relative to Pennsylvania. The finding was contentious and provoked a long exchange about the data, but the design became a template. Difference-in-differences rests on the parallel trends assumption: in the absence of treatment, the treated and untreated groups would have followed the same trajectory. This cannot be tested for the post-treatment period, but it can be probed by examining whether the groups moved in parallel before treatment, ideally over several periods. Diverging pre-trends are a warning that the comparison group is poorly chosen. The design has seen important technical revisions in recent years. When different units adopt treatment at different times, the standard two-way fixed-effects regression that most analysts use can produce misleading averages, because it implicitly uses already-treated units as controls for later-treated ones. Andrew Goodman-Bacon's decomposition of this estimator, and new estimators from Brantly Callaway and Pedro Sant'Anna and from others, published around 2021, have changed recommended practice. An analyst evaluating a feature rolled out market by market, or a policy adopted by different regions in different years, should use one of these newer estimators rather than the textbook regression. Synthetic control Sometimes only one unit is treated: a single state passes a law, a single country adopts a policy, a single market receives a new product. Difference-in-differences needs a comparison group, and no single untreated unit may resemble the treated one closely. The synthetic control method, developed by Alberto Abadie and colleagues, constructs a comparison by taking a weighted combination of untreated units chosen so that the combination tracks the treated unit's outcome closely before the intervention. The divergence afterwards is the estimated effect. The method's best-known application, by Abadie, Alexis Diamond and Jens Hainmueller in 2010, estimated the effect of California's 1988 tobacco control programme, Proposition 99, on cigarette consumption, using a weighted mix of other states that reproduced California's pre-1988 consumption trajectory. Inference is done by placebo tests: applying the method to each untreated unit as if it had been treated and asking whether California's estimated effect is unusual in that distribution. Synthetic control has found a natural home in technology companies that launch products or policies market by market and need to evaluate a single launch against a counterfactual built from untreated markets. Its assumptions are that the weighted combination would have continued to track the treated unit absent treatment, and that no shock unique to the treated unit coincided with the intervention. Interrupted time series and structural forecasts A close relative of synthetic control is used heavily in industry for evaluating interventions on a single time series, such as a marketing campaign's effect on daily sales or a site redesign's effect on traffic. The idea is to model the outcome series before the intervention, using its own history and a set of control series that were not affected, and to forecast what would have happened afterwards. The gap between the forecast and the observed series estimates the effect. In 2015 Kay Brodersen and colleagues at Google described a Bayesian structural time-series version of this approach in the Annals of Applied Statistics and released it as the CausalImpact package, which has been widely adopted. The method is only as good as its forecast, and the forecast is only as good as the assumption that the control series were unaffected by the intervention and that their relationship with the outcome series would have continued unchanged. Both can fail quietly. A campaign in one region may shift sales from neighbouring regions used as controls, contaminating them. A competitor's promotion that happens to coincide with the campaign will be attributed to it. And with a single treated series and no replication, there is no direct way to check the counterfactual forecast after the fact. The best defences are the same as for synthetic control: placebo analyses on periods with no intervention, careful choice of controls that are plausibly unaffected, and a healthy suspicion of effects that are small relative to the forecast's historical error. Choosing a design Each design replaces the assumption of no unmeasured confounding with a different assumption that may be more defensible in particular circumstances. Table 4 summarises them. Table 4. Designs that do not require measuring every confounder. Design Source of variation Key assumption Estimate applies to Common industrial setting Instrumental variables A variable that shifts treatment only Exclusion restriction; no instrument–outcome confounding Compliers Encouragement designs, A/B tests with non-compliance Regression discontinuity A threshold rule on a score No sorting or other jump at the threshold Units near the threshold Credit, fraud, eligibility and alert thresholds Difference-in-differences Timing of treatment across groups Parallel trends without treatment Treated groups Staggered regional or feature rollouts Synthetic control Weighted combination of untreated units Pre-period fit carries over; no coincident shock The single treated unit Market-level launches, policy changes The table should not be read as a menu from which an analyst chooses whichever design is most convenient. Each design is available only when the world has supplied the variation it needs. The skill lies in noticing that variation, which is why many of the best natural experiments in industry are found by people who understand how operational decisions are actually made: which rule fired, which market launched first, which users were exposed by accident of timing. Sensitivity analysis When no design is available and unmeasured confounding cannot be ruled out, an analyst can still say something useful: how strong would an unmeasured confounder have to be to overturn the conclusion? This is the purpose of sensitivity analysis, and it is one of the most underused tools in applied work. Paul Rosenbaum developed an early and influential version for matched studies, which asks how much the odds of treatment would have to differ between matched units, because of an unmeasured factor, before the finding lost statistical significance. Tyler VanderWeele and Peng Ding introduced the E-value in 2017, which reports the minimum strength of association, on the risk ratio scale, that an unmeasured confounder would need to have with both treatment and outcome to fully explain away an observed association. Carlos Cinelli and Chad Hazlett, in 2020, gave a framework for regression estimates expressed in terms of partial R-squared values, with a robustness value summarising how much residual variation in both treatment and outcome a confounder would need to explain. The great advantage of these approaches is that they translate an untestable assumption into a quantity that domain experts can reason about. Cinelli and Hazlett's method in particular allows the strength of a hypothetical unmeasured confounder to be benchmarked against the strength of observed covariates. A statement such as "to explain away this effect, an unmeasured confounder would have to be three times as strongly related to both treatment and outcome as customer tenure, the strongest measured predictor" gives a decision-maker something concrete to evaluate. If no plausible unmeasured factor comes close, the conclusion is robust. If a familiar factor could easily reach that strength, it is not. Sensitivity analysis does not remove uncertainty about confounding. It makes that uncertainty explicit and calibrated, which is what a decision-maker needs. It would be misleading to suggest that the methods in this chapter routinely deliver certainty. They do not. They deliver estimates whose dependence on assumptions is known, which is a different and more valuable thing than estimates whose dependence is hidden. Chapter 7: Effects That Differ: Uplift, Meta-Learners and Targeting An average treatment effect answers the question "should we do this for everyone?" Most decisions in applied machine learning ask something more granular: for whom should we do this? A retention team does not choose between sending the offer to all customers or none; it chooses which customers to send it to. A clinic does not choose whether a follow-up programme is good on average; it chooses which patients to enrol, given limited nurse time. An advertising platform does not decide whether ads work; it decides which impression to show to which user. In each case the quantity that drives the decision is the conditional average treatment effect: the expected effect for units with a given set of characteristics. Estimating it is harder than estimating an average, and it is where causal inference and machine learning meet most directly. This chapter covers why the obvious approach goes wrong, the family of methods that has replaced it, and how to evaluate a targeting policy when no individual's true effect can ever be observed. Why risk is not responsiveness The churn example in Chapter 1 already showed the core problem. Ranking customers by predicted churn risk targets those most likely to leave, but the offer's value depends on how much it changes each customer's behaviour. Those two quantities can be unrelated or even inversely related. Customers at the highest risk may have already decided to leave, and customers at moderate risk may be the most persuadable. Marketing analysts recognised this decades before the modern causal machine learning literature, under the name uplift modelling. Nicholas Radcliffe and Patrick Surry, among others, argued through the 1990s and 2000s that campaign targeting should be driven by the incremental effect of the campaign rather than by the response rate, and developed methods and evaluation curves for doing so. The insight generalises well beyond marketing. In medicine, the patients at highest risk of a bad outcome are not always those who benefit most from a given treatment; a frail patient may be at high risk but unable to tolerate an aggressive therapy. In public policy, the job seekers least likely to find work unaided may not be the ones a training programme helps most. The data requirement is also different. To learn who responds to a treatment, the data must contain both treated and untreated units across the range of characteristics of interest, and the treatment must be either randomised or adjustable using the methods of the previous chapters. A dataset in which every high-value customer received the offer contains no information about how high-value customers respond to it. This is why uplift modelling works best on data from randomised experiments, and why organisations that want to target well often deliberately hold out a random control group from every campaign. The meta-learners A meta-learner is a recipe for estimating heterogeneous effects by combining standard supervised learning models. The term was introduced by Sören Künzel, Jasjeet Sekhon, Peter Bickel and Bin Yu in a 2019 paper in the Proceedings of the National Academy of Sciences, and the letters by which the variants are known have become standard vocabulary. Table 5 compares the main ones. Table 5. Meta-learners for estimating conditional average treatment effects. Learner Construction Strength Weakness S-learner One model of outcome on covariates plus treatment; effect is difference in predictions Simple; shares data across arms Regularisation can shrink effect towards zero T-learner Separate outcome models for treated and control; effect is difference Flexible; no shrinkage of treatment Noisy when one arm is small; differences of errors X-learner T-learner, then imputes individual effects and models them, weighted by propensity Good when arms are unbalanced More moving parts R-learner Residualises outcome and treatment, then fits effect model to residuals Orthogonal; robust to nuisance error Requires good nuisance models and cross-fitting DR-learner Constructs doubly robust pseudo-outcomes, then regresses them on covariates Doubly robust; flexible final model Pseudo-outcomes can be high-variance The S-learner, for "single", fits one model of the outcome using the covariates and the treatment as inputs, then estimates each unit's effect as the difference between the model's predictions with treatment switched on and off. It is g-computation from Chapter 5 applied unit by unit. Its weakness is the one already noted: flexible learners may largely ignore the treatment variable if it is a weak predictor relative to the covariates, which biases effects toward zero. The T-learner, for "two", fits separate models for treated and control units and takes the difference in their predictions. It avoids shrinkage but introduces a different problem. Each model is fit to minimise its own prediction error, and the difference between two noisy functions can be noisier still. If the treated group is small, its model will be poorly estimated, and its errors will masquerade as heterogeneity. The X-learner was proposed by Künzel and colleagues to address unbalanced arms. It starts with a T-learner, uses each arm's model to impute the missing potential outcome for units in the other arm, models these imputed individual effects as a function of covariates, and blends the two resulting effect models using the propensity score. When one arm is much larger than the other, as in many marketing campaigns with small holdouts, it tends to outperform both S and T. The R-learner, introduced by Xinkun Nie and Stefan Wager, applies the residual-on-residual logic of double machine learning to heterogeneous effects. It first estimates the outcome and propensity as functions of covariates, with cross-fitting, then fits a model for the treatment effect by minimising a loss defined on the residuals. Because the loss is orthogonal in the sense described in Chapter 5, errors in the nuisance models have only second-order effects on the effect estimate. The DR-learner, analysed in detail by Edward Kennedy, constructs a doubly robust pseudo-outcome for each unit, an AIPW-style estimate of its individual effect that is unbiased on average but noisy individually, and regresses these pseudo-outcomes on covariates with any learner. Both have strong theoretical properties and are the recommended starting points in much of the recent literature. No single meta-learner dominates. Their relative performance depends on sample size, the balance of treated and control units, how complex the outcome surface is compared to the effect surface, and how good the nuisance models are. Libraries such as EconML and Uber's CausalML implement most of them, which makes it practical to compare several on the same problem, using the evaluation methods described below. Causal trees and forests A parallel line of work adapted tree-based methods directly to the estimation of heterogeneous effects. Susan Athey and Guido Imbens introduced causal trees in 2016. A standard regression tree chooses splits that reduce prediction error. A causal tree chooses splits that maximise the difference in treatment effects between child nodes, so that the leaves group units with similar responses to treatment. Their key methodological contribution was honesty: using one subsample to choose the splits and a separate subsample to estimate the effects within each leaf. Without honesty, the same noise that guided the split would inflate the estimated differences between leaves, producing heterogeneity that would not replicate. Stefan Wager and Athey extended this in 2018 to causal forests, ensembles of honest causal trees, and proved that their estimates are asymptotically normal, which allows confidence intervals for individual-level effect estimates. Athey, Julie Tibshirani and Wager then generalised the approach as generalised random forests, a framework for estimating any quantity defined by a local estimating equation, including heterogeneous effects with instruments. The grf package in R, from the same authors, has become one of the most widely used tools for heterogeneous effect estimation in academic and applied work. Causal forests have a practical advantage beyond their theory: they provide an interpretable measure of which covariates drive heterogeneity, by tracking how often each is used for splitting. This is not the same as saying those covariates cause the heterogeneity, and the warnings of Chapter 4 about reading causal meaning into secondary variables apply. But it gives analysts a useful starting point for asking why effects differ. Evaluating without ground truth The hardest part of heterogeneous effect estimation is evaluation. In ordinary supervised learning, a model's predictions can be compared with held-out labels. Here, the label, the individual treatment effect, is never observed. Standard metrics such as mean squared error cannot be computed directly. Several workarounds are in use, and all of them require data in which treatment was randomised, or at least in which propensities are known. The most common in industry is the uplift curve or the closely related Qini curve. Units in a randomised test set are ranked by their predicted effect. For each fraction of the population, from the top-ranked downward, the curve plots the cumulative incremental outcome achieved by treating that fraction, computed as the difference between treated and control outcomes within the selected group. A model with no ability to rank effects produces a straight diagonal; a good model produces a curve that rises steeply at first, as it captures the most responsive units, and flattens later. The area between the curve and the diagonal summarises ranking quality. More recent work has formalised and refined these ideas. Steve Yadlowsky and colleagues proposed the rank-weighted average treatment effect, a family of metrics with valid confidence intervals for testing whether a prioritisation rule identifies units with larger-than-average effects. A simpler and very useful diagnostic is to group test units into quantiles of predicted effect and estimate the actual average effect within each quantile using the randomised comparison. If the estimated effects rise across quantiles in step with the predictions, the model is capturing real heterogeneity. If they do not, the apparent heterogeneity was noise. That last outcome is common. Estimating heterogeneous effects requires far more data than estimating average effects, because the signal is a difference between differences. A campaign test large enough to estimate the average uplift with reasonable precision may be far too small to estimate how uplift varies. Analysts should be prepared to report that the data do not support personalised targeting, and that treating everyone or no one is the best available policy. From effects to decisions Estimating effects is a means to an end, which is choosing whom to treat. With a cost per treatment and a value per unit of outcome, the natural rule is to treat a unit if its estimated effect, valued in money, exceeds the cost. This plug-in approach is simple and often adequate. But it optimises the wrong thing in a subtle way: it tries to estimate every unit's effect accurately, when what matters is only getting the sign of benefit minus cost right, and only for units near the decision boundary. Policy learning addresses this by optimising the decision rule directly. Athey and Wager, in a 2021 paper in Econometrica, showed how to learn a treatment assignment rule from observational or experimental data by maximising a doubly robust estimate of the rule's value, within a class of rules such as shallow decision trees. Restricting the class of rules has practical virtues. A depth-two tree that says "offer the discount to customers with more than a year's tenure whose usage has declined" can be audited, explained to a business owner and checked for fairness in a way that a black-box effect model cannot. A worked illustration shows how much the choice of target matters. Suppose a retention offer costs £20 per customer in discount and a retained customer is worth £200 over the following year. The offer is worth sending to a customer only if it raises their probability of staying by more than ten percentage points. A churn model might identify a segment with a 60 per cent probability of leaving; an uplift model, estimated from a randomised holdout, might show that the offer moves this segment's retention by only three points, because most of these customers have already decided. Sending the offer to them loses money. Another segment, with only a 25 per cent churn probability, might show a fifteen-point uplift because these customers are wavering and price-sensitive. Sending the offer to them makes money. The churn ranking and the profitable targeting rule select different people, and the difference is invisible until effects, rather than risks, are estimated. Two further considerations belong in any targeting decision. The first is uncertainty: an estimated uplift of twelve points with a confidence interval running from minus two to twenty-six is not good evidence that the threshold has been cleared, and a rule that acts on point estimates alone will treat noise as signal. The second is fairness. A targeting rule that allocates a benefit, such as a discount, a follow-up call or an educational intervention, distributes it across groups in a way that should be examined explicitly, because effect estimates can differ across groups for reasons that reflect historical disadvantage. The causal framing of fairness discussed in the next chapter applies directly. A related problem arises in systems that make decisions continually, such as recommenders and ad servers, which log the actions they took and the outcomes that followed. Evaluating a new policy on such logs, without deploying it, is called off-policy evaluation. The logged data are confounded by the old policy's choices, exactly as observational data are confounded by human decisions, and the solutions are the same: inverse propensity weighting using the old policy's known action probabilities, outcome modelling, and doubly robust combinations of the two, as Miroslav Dudík, John Langford and Lihong Li showed in 2011. The practical requirement is that the logging policy records the probability with which it chose each action and retains some randomisation, so that every action has a non-zero chance of being observed. Systems built without this, deterministic rankers that always show the top-scored item, cannot be evaluated offline on any policy that would have chosen differently, because the positivity assumption fails by design. The organisational lesson The methods in this chapter are powerful, but their most important requirement is not statistical. It is that the organisation must be willing to collect data in which treatment varies for reasons it understands. That usually means randomised holdouts, logged propensities and deliberate exploration, all of which have a short-term cost: some customers who would have benefited from an offer will not receive it, and some impressions will be spent on items the current model thinks are suboptimal. Teams that are not willing to pay that cost will find that their data cannot answer the targeting question, however sophisticated the model. Teams that are willing find that the cost is small relative to the value of knowing, rather than guessing, whom their interventions help. Chapter 8: Counterfactuals, Fairness and Robust Models The methods of the last three chapters live on the second rung of the causal ladder. They estimate what would happen, on average or for a type of unit, under an intervention. The third rung asks something more personal: given everything we know about what actually happened to this particular person, what would have happened if one thing had been different? That question sounds philosophical, but it sits underneath several of the most pressing demands placed on deployed AI systems. People denied a loan want to know what they could have done differently. Regulators want to know whether a decision would have been the same had the applicant belonged to a different group. Engineers want models whose behaviour will not collapse when the world shifts. Each of these is, on inspection, a counterfactual or causal demand, and each is handled badly by tools that operate only on associations. This chapter explains how counterfactuals are computed, and then examines four places where they matter in machine learning practice: explanation and recourse, fairness, robustness to distribution shift, and the discovery of causal structure from data. It ends with an assessment of what large language models bring to causal reasoning. How a counterfactual is computed A rung-two query needs a causal graph. A rung-three query needs more: a structural causal model, with equations that specify how each variable is produced from its causes and an unobserved disturbance. The disturbance terms are what make a unit an individual. Two applicants with the same recorded attributes may still differ in unrecorded ways, and those differences are summarised in their disturbances. Pearl describes counterfactual computation as three steps. Abduction: use the observed facts about the individual to infer the values, or the distribution, of their disturbance terms. Action: modify the model by setting the variable of interest to its counterfactual value, cutting it off from its usual causes. Prediction: run the modified model forward, using the inferred disturbances, to compute the counterfactual outcome. A simple example shows why the steps matter. Suppose a person's salary is determined by their education and a disturbance representing everything else, and the person in question has more education than average and an unusually high salary. To ask what their salary would have been with less education, we first infer from their high salary that their disturbance is large, perhaps reflecting ability or connections. We then set education to the lower value and recompute salary, keeping the large disturbance. The answer is still above what an average person with that lower education would earn, because this individual's other advantages remain. A purely interventional answer, the average salary among people with lower education, would miss this. The cost of this power is that counterfactuals depend on functional forms and on the distribution of disturbances in ways that interventional quantities do not. Two structural models can agree on every observational and interventional distribution and still give different counterfactual answers. Rung-three conclusions therefore rest on stronger assumptions than rung-two conclusions, and they should be reported with that in mind. In some cases, bounds rather than point values are all the data can support: Jin Tian and Pearl showed in 2000 how to bound quantities such as the probability that a treatment was necessary for an outcome by combining experimental and observational data. Explanations and recourse One of the most influential ideas in explainable AI is the counterfactual explanation, proposed by Sandra Wachter, Brent Mittelstadt and Chris Russell in a 2017 article in the Harvard Journal of Law and Technology. Rather than explaining the internal workings of a model, a counterfactual explanation tells a person the smallest change to their inputs that would have changed the decision: "your loan would have been approved if your income had been £5,000 higher". Such explanations are intuitive, require no disclosure of the model, and seemed well matched to legal demands for meaningful explanations of automated decisions. They also have a structural weakness, which Amir-Hossein Karimi, Bernhard Schölkopf and Isabel Valera set out clearly in work presented in 2021. A counterfactual explanation computed on the model alone treats inputs as independent knobs that can be turned one at a time. In the world, they are not. If a person is told to increase their income, and in reality increasing income would also change their debt-to-income ratio and possibly their employment category, the suggested change and its consequences may not lead to approval at all. Conversely, a change that looks large in the model's input space may be achievable through a single action that moves several inputs together. Karimi and colleagues argued that useful algorithmic recourse must be framed as interventions in a causal model of how the inputs relate to each other and to the actions a person can take, not as perturbations of the model's feature vector. Recourse is a rung-two or rung-three question about the world, not a question about the classifier. The point extends to feature attribution generally. Methods that assign importance scores to inputs describe how the model uses them. A person who reads such a score as advice about what to change is making a causal inference the method does not support. Counterfactual fairness Fairness criteria in machine learning were first framed mostly in terms of observable statistics: equal positive rates across groups, equal error rates, calibration within groups. These criteria are useful, but they cannot be satisfied simultaneously in general, and they say nothing about why outcomes differ across groups. Causal approaches to fairness ask instead which pathways from a protected attribute to a decision are acceptable. Matt Kusner, Joshua Loftus, Chris Russell and Ricardo Silva proposed counterfactual fairness in 2017. A decision is counterfactually fair towards an individual if it would have been the same in the counterfactual world where that individual's protected attribute had been different, with everything not caused by the attribute held fixed. This definition is demanding and, crucially, requires a causal model of how the protected attribute influences the other features. It forces the analyst to say, for instance, whether a candidate's postcode or school is partly a consequence of their race in the population being modelled, and if so, to remove that influence before using those features. Other work has refined the idea into path-specific fairness, which allows some causal paths from a protected attribute to the outcome while forbidding others. A university might decide that the effect of gender on admission through choice of department is acceptable while the direct effect is not, a distinction made famous by the 1973 Berkeley graduate admissions data, in which an aggregate gap in admission rates between men and women largely disappeared within departments because women had applied disproportionately to more competitive ones. These frameworks do not settle normative questions. Deciding which paths are legitimate is a matter of values and law, not statistics. What the causal framing does is make the normative question explicit and well posed. It also explains the Obermeyer finding from Chapter 1 with precision: the cost-based algorithm let race influence predictions through the path from race to access to cost, a path no one had decided was legitimate, and the correction was to change the target to one that did not run through it. Invariance and distribution shift The fourth failure mechanism from Chapter 1 was non-invariance: models that rely on associations which hold in the training environment and break in deployment. Causal thinking suggests a remedy. If the mechanism generating the outcome from its direct causes is stable across environments, while the relationships among other variables shift, then a model that relies only on the outcome's causes will transfer, and a model that relies on spurious correlates will not. Jonas Peters, Peter Bühlmann and Nicolai Meinshausen formalised this in 2016 as invariant causal prediction. Given data from several environments, for example different hospitals, time periods or experimental conditions, the method searches for sets of predictors whose relationship to the outcome is the same in every environment, and treats the variables common to all such sets as likely causes. Martin Arjovsky, Léon Bottou, Ishaan Gulrajani and David Lopez-Paz extended the idea to deep learning in 2019 with invariant risk minimisation, which tries to learn a representation on which the optimal classifier is the same across training environments. The idea is compelling, and the empirical record has been sobering. In 2021, Gulrajani and Lopez-Paz evaluated many domain generalisation algorithms, including invariance-based ones, under a common protocol and found that ordinary empirical risk minimisation with careful tuning performed about as well as the specialised methods on the benchmarks studied. Later theoretical work identified conditions under which invariant risk minimisation fails to find invariant predictors at all. The lesson is not that the causal intuition is wrong, but that turning it into a reliable training procedure for high-dimensional inputs such as images is hard, and that having data from several genuinely different environments is a prerequisite that many datasets do not meet. In tabular settings with known structure, the causal approach is more directly useful. If a graph indicates which features are causes of the label and which are effects or spurious correlates, a model can be restricted to causal features, or the analyst can anticipate which shifts will hurt and monitor for them. A model of hospital readmission that relies on a coding practice specific to one hospital system can be expected to degrade when deployed elsewhere, and the graph tells you this before the deployment rather than after. A more modest and often more practical use of counterfactual thinking for robustness is counterfactual data augmentation. Divyansh Kaushik, Eduard Hovy and Zachary Lipton, in work presented at ICLR in 2020, asked human annotators to make minimal edits to movie reviews that flipped their sentiment label, changing only what was needed to reverse the meaning. Models trained on the original and edited reviews together relied less on spurious cues, such as genre words or actor names that happened to correlate with sentiment in the training set, and generalised better to some out-of-domain data. The edited examples act as constructed interventions: they hold fixed everything except the causal feature, so the model is pushed to learn that feature. The same idea appears in computer vision, where images are edited to change backgrounds while keeping the object, and in fairness work, where names or pronouns are swapped to test and reduce sensitivity to protected attributes. Augmentation does not require a full structural model, only a judgement about which features should and should not affect the label, which is a causal judgement made locally. Learning the graph from data If drawing a graph from domain knowledge is the hard part of causal analysis, it is natural to ask whether the graph can be learned from data. This is causal discovery, and it is an active and genuinely useful field, though one whose limits need to be understood. The classical constraint-based approach, embodied in the PC algorithm of Peter Spirtes, Clark Glymour and Richard Scheines, uses conditional independence tests to rule out edges and then orients some of the remaining edges using colliders, which leave a distinctive statistical signature. Its extension, FCI, allows for unmeasured confounders. Score-based methods, such as greedy equivalence search, associated with David Maxwell Chickering, search over graphs to maximise a score that rewards fit and penalises complexity. Both families can, under their assumptions, recover only an equivalence class of graphs: a set of graphs that imply the same independences and are therefore indistinguishable from observational data alone. Many arrow directions remain unresolved. Stronger assumptions buy more. If the relationships are linear and the disturbances non-Gaussian, the LiNGAM method of Shohei Shimizu and colleagues can identify the full graph. Additive noise models exploit asymmetries in nonlinear relationships. In 2018 Xun Zheng and colleagues introduced NOTEARS, which recast structure learning as a continuous optimisation problem amenable to gradient methods, and spawned a large literature of differentiable discovery methods. A sobering 2021 paper by Alexander Reisach, Christof Seiler and Sebastian Weichwald, titled "Beware of the Simulated DAG!", showed that in many benchmark simulations the true causal order could be read off simply by sorting variables by variance, because the simulation procedures inflated variance along the causal order. Some continuous methods were exploiting this artefact, and their performance dropped when data were standardised. The practical upshot is that causal discovery is best used as a tool for generating and checking hypotheses, not as a replacement for domain knowledge. It can propose edges that experts missed, flag conflicts between an expert-drawn graph and the data, and narrow the space of plausible structures. Its output should be treated as one input to the graph-drawing process described in Chapter 3, with arrows that conflict with known timing or mechanisms overridden. What language models can and cannot do Large language models have added a new and unexpected participant to causal analysis. Because they are trained on vast amounts of text that describes causal relationships, they can often state plausible causal links between named variables: that smoking causes cancer, that altitude affects temperature, that a price change affects demand. Emre Kıcıman, Robert Ness, Amit Sharma and Chenhao Tan, in a 2023 study, reported that GPT-4-class models performed strongly on several benchmarks of this kind, including pairwise causal direction questions, and argued that such models could help analysts draft graphs and surface overlooked confounders. Other work points to the limits. Zhijing Jin and colleagues introduced the Corr2Cause benchmark, published at ICLR in 2024, which tests whether models can infer causal relationships from stated correlational facts using the formal rules of causal discovery; the models they tested performed poorly, often close to chance, and fine-tuned models failed to generalise when the problems were paraphrased or variable names changed. Their CLadder benchmark, presented at NeurIPS in 2023, tested formal reasoning across the three rungs of the ladder and also found substantial weaknesses. Matej Zečević and co-authors described such models as "causal parrots", reproducing causal knowledge from their training text rather than reasoning to it. The reasonable position, as of late 2026, combines both findings. Language models are useful as a source of domain knowledge, much as a well-read colleague is: they can suggest variables, propose arrows, recall known mechanisms and draft a first graph for experts to criticise. They are not reliable as causal reasoners in their own right, and they have no access to the specific data-generating process of an organisation's own systems, which is where most of the important assumptions live. An analyst who uses one to draft a DAG should treat the draft exactly as they would treat a draft from a new team member: as a starting point to be checked against timing, mechanism and data, never as an authority. The common thread Explanations, fairness, robustness and structure learning look like separate concerns, and in most organisations they are handled by separate teams. The causal view shows that they share a root. Each asks a question about the mechanisms that generated the data rather than about the patterns the data contain, and each is answered badly by methods that see only patterns. An organisation that builds the habit of writing down its causal assumptions finds that the same graph serves all four: it says which explanations are actionable, which paths are legitimate, which features will transfer and which edges need checking. The final chapter turns that habit into a working method. Chapter 9: A Causal Workflow for Data Science Teams The ideas in this booklet are of little use if they remain ideas. A team that reads about DAGs and doubly robust estimators but continues to answer every request by training a classifier on whatever data are at hand will make the same mistakes as before, only with more sophisticated vocabulary. What changes outcomes is a workflow: a sequence of steps that the team follows by default whenever a question involves a decision, with checkpoints at which the work can be reviewed and challenged. This chapter sets out such a workflow, illustrates it with an extended example, and addresses the organisational question that eventually confronts every team doing observational causal work: when is it not good enough, and when must an experiment be run instead? Start from the hypothetical experiment The most useful single habit in causal analysis is to begin by describing the randomised experiment that would answer the question, even when it cannot be run. Miguel Hernán and James Robins formalised this as target trial emulation in a 2016 paper in the American Journal of Epidemiology, and it has since become standard in clinical epidemiology. The idea is to specify, before touching the data, every element of the ideal trial: who would be eligible, what treatment strategies would be compared, how participants would be assigned, when follow-up would start and end, what outcome would be measured, and what causal contrast would be estimated. The observational analysis is then designed to mimic that trial as closely as the data allow. The discipline catches errors that more sophisticated estimation cannot. The most notorious is immortal time bias. Suppose an analysis compares patients who started a drug at any time during a year with patients who never started it, counting follow-up from the start of the year for everyone. The treated group is guaranteed to have survived until they started the drug, so the period between the start of follow-up and the start of treatment is "immortal" for them. They will appear to have lower mortality even if the drug does nothing. In a real trial this cannot happen, because eligibility, assignment and the start of follow-up coincide. Writing down the target trial makes the misalignment obvious. Hernán and colleagues have shown that several prominent observational findings that conflicted with later randomised trials, for example on the effects of hormone therapy in postmenopausal women, were largely reconciled when the observational data were reanalysed to emulate the trial design properly. The same discipline applies directly to industrial questions. "Does using the mobile app increase customer lifetime value?" becomes: among customers who have an account but have not yet installed the app, compare those who install it within the next thirty days with those who do not, starting follow-up at the moment of eligibility, over the following twelve months. Stated this way, the question already excludes the customers who installed the app because they were already highly engaged, defines a treatment window that avoids immortal time, and makes explicit that the estimand concerns customers who could plausibly be nudged to install. The five steps With the target trial specified, the workflow proceeds through five steps. The DoWhy library, mentioned earlier, organises its interface around a similar sequence of model, identify, estimate and refute, which is a sign of how far the field has converged on the shape of good practice. 1. Frame. Write down the decision the analysis will inform, the treatment and its alternatives, the outcome, the population, the time frame and the estimand: average effect, effect on the treated, conditional effects, or an optimal policy. 2. Model. Draw the causal graph, including unmeasured variables, built around the timeline of events and reviewed with people who understand how treatment decisions are actually made. 3. Identify. Use the graph to determine whether the estimand is identified by adjustment, by a front-door argument, or by one of the designs of Chapter 6. Choose the adjustment set or design. If nothing works, stop and say so. 4. Estimate. Choose an estimator suited to the data, typically a doubly robust or double machine learning approach with flexible nuisance models. Check overlap and balance. Report the estimate with its interval. 5. Refute. Try to break the result. Run sensitivity analyses, negative controls and placebo tests. Report what would have to be true for the conclusion to be wrong. The first three steps involve no statistics at all, and they are where the most consequential decisions are made. A useful norm is that no estimation code is written until the first three steps have been written up in a short document and reviewed by at least one person other than the analyst. That document, which might be called a causal analysis plan, need not be long. A page with the target trial, the graph and the chosen adjustment set is enough. Its value is that it fixes the analysis before results are seen, which protects against the temptation to adjust the specification until the answer looks right. Trying to break your own estimate The refutation step deserves elaboration, because it is the step most often skipped and the one that most distinguishes credible causal work from the rest. A negative control outcome is an outcome that the treatment cannot plausibly affect but that shares the same confounding structure as the real outcome. Marc Lipsitch, Eric Tchetgen Tchetgen and Ted Cohen set out the logic in a 2010 paper in Epidemiology. If an analysis of a flu vaccine finds that vaccinated people are also less likely to be hospitalised for injuries, the vaccine has not prevented injuries. The finding instead reveals that vaccinated people are healthier or more cautious in ways the adjustment missed, and the same bias is probably inflating the vaccine's estimated effect on flu. In a commercial setting, if a loyalty programme appears to increase purchases in product categories that the programme does not touch, the estimate for the targeted categories should be viewed with suspicion. A negative control exposure works the other way: a "treatment" that cannot affect the outcome but would be subject to the same confounding, such as a reminder email that bounced and was never received. Placebo tests replace the real treatment with a randomly permuted version and rerun the analysis; the estimated effect should be near zero. Pre-period tests, in difference-in-differences and synthetic control designs, apply the method to a period before treatment, where again no effect should appear. Subset stability checks whether the estimate holds up across random subsamples and across meaningful segments. Specification curves rerun the analysis under every reasonable alternative adjustment set, estimator and sample definition, and show the full distribution of results, so that a reader can see whether the conclusion depends on a particular choice. None of these proves that an estimate is correct. Each can detect particular kinds of error. Taken together, they shift the burden of proof in the right direction: from a reviewer who must guess what might be wrong to an analyst who has demonstrated what has been checked. An extended example Consider a hospital system that wants to know whether a nurse follow-up telephone call within forty-eight hours of discharge reduces thirty-day readmission. The calls are currently made at the discretion of ward staff, who tend to call patients they consider at risk, and about a third of discharged patients receive one. A predictive readmission model already exists. The request, as it reaches the data science team, is to "use the model to show whether the calls work". The team first reframes. The decision is whether to expand the call programme, and to whom. The target trial is: among adult patients discharged home from medical wards, randomise to a call within forty-eight hours or no call; follow up from discharge for thirty days; compare readmission rates. The estimand of first interest is the average effect among all eligible patients, with conditional effects as a second objective for targeting. Next, the graph. Through a session with two ward nurses and a discharge coordinator, the team learns that calls are prioritised for patients with complex medication regimes, those living alone and those whom staff judged anxious at discharge. Medication complexity and living situation are recorded. Staff judgement of anxiety is not, but it is plausibly related to recorded variables such as prior admissions and length of stay, and it may affect readmission directly. The graph therefore includes an unmeasured node for staff concern, with arrows to the call and to readmission. Calls are made by a limited pool of nurses, and on weekends fewer calls are made because of staffing, regardless of patient characteristics. That fact will matter. The team also identifies traps. Whether the patient attended a follow-up clinic appointment is recorded and strongly predicts readmission, but the call itself often prompts patients to keep their appointment. Clinic attendance is a mediator and must be excluded from the adjustment set. Patients who died within thirty days without readmission are in the data; excluding them would condition on a post-discharge outcome, so they are retained with an appropriate handling of the competing event. And the readmission model's risk score is a function of recorded pre-discharge variables, so it can be included as a summary of them, but it cannot stand in for the unmeasured staff concern. Identification by adjustment alone is doubtful because of the unmeasured concern. But the weekend effect offers a second route: patients discharged on a Friday afternoon are less likely to receive a call within forty-eight hours for reasons that have nothing to do with their own condition. If day of discharge does not otherwise affect readmission, an assumption the team must examine, since weekend discharges may differ in other ways, it is an instrument. The team plans two analyses: a doubly robust adjusted estimate under the assumption that recorded variables capture most of the confounding, and an instrumental variable estimate using discharge timing. The second applies only to patients whose receipt of a call depends on discharge timing, the compliers, which the team notes. Estimation and refutation follow. The team checks overlap and finds that patients on certain specialist wards almost always receive calls, so they are excluded from the primary analysis and the estimand is restricted accordingly. As a negative control outcome, they use emergency attendances for unrelated injuries in the thirty days after discharge, which the call should not affect. They run an E-value calculation on the adjusted estimate and benchmark it against the strongest recorded confounder. Suppose the adjusted estimate suggests a modest reduction in readmission, the IV estimate is in the same direction but imprecise, and the negative control shows a small, non-zero association suggesting some residual confounding. The honest conclusion is that the evidence is suggestive but not decisive. And that conclusion leads naturally to the final question of this chapter. When to stop and run an experiment Observational causal inference is not a substitute for experimentation. It is what one does when experimentation is impossible, unethical, too slow or too expensive, and it is a way of deciding which experiments are worth running. In the hospital example, the right recommendation may well be a randomised rollout: assigning calls by random allocation among patients for whom staff have no strong preference, or randomising the order in which wards adopt an expanded programme. The observational analysis has done its job by establishing that the question is worth answering properly and by identifying the subgroups where uncertainty is greatest. Online businesses are in an unusually fortunate position, because experimentation is cheap. Ron Kohavi, Diane Tang and Ya Xu, drawing on experience at Microsoft, LinkedIn and Google, document in Trustworthy Online Controlled Experiments (2020) both how routinely large technology companies run controlled experiments and how many apparently promising ideas show no improvement once tested. That experience is a useful corrective to confidence in any single observational estimate. Experiments have their own causal pitfalls, which the tools of this booklet help to diagnose. Interference between units, discussed in Chapter 2, violates the assumptions of a standard A/B test whenever treated and control users interact, as in marketplaces and social networks, and requires cluster-level randomisation or specially designed estimators. Sample ratio mismatch, where the observed split between arms differs from the intended one, signals that something in the assignment or logging pipeline is broken and that the arms may no longer be exchangeable. Novelty and learning effects mean that a short test may not estimate the long-run effect. And an experiment that filters its analysis population on post-assignment behaviour, such as analysing only users who opened the changed page, has conditioned on a collider and lost the protection that randomisation provided. A reasonable decision rule for a team is this. If an experiment is feasible, ethical and can be completed in time to inform the decision, run it, and use observational analysis to design it well. If an experiment is not feasible, do the observational analysis using the workflow above, and report its result with the strength of evidence it actually provides. If the decision is high-stakes and the observational evidence is weak, invest in making an experiment feasible: build the randomisation infrastructure, negotiate a holdout, or design a staggered rollout that can be analysed as a natural experiment later. The infrastructure that makes causal work possible Most of the obstacles in the hospital example came from the way the data had been produced: discretionary assignment, an unrecorded reason for treatment, and no randomisation. A team that wants causal questions to be answerable in future can change those conditions, often cheaply, and the changes are worth listing because they are so rarely made. Record the reason for every decision a system or a person makes, not only the decision itself. If a rule fired, log which rule and the score that triggered it. If a person chose, offer a short structured field for why. Record the probability with which an automated policy chose each action, and retain a small amount of randomisation in policies wherever the cost is tolerable, so that every action has some chance of being observed in every context. Keep randomised holdout groups for recurring campaigns and programmes, rotating them so that no customer is permanently excluded. When launching something across markets or teams, randomise or at least deliberately plan the order, and write down the plan before it starts. Version every threshold and business rule, with the dates on which each version applied, so that discontinuities can be found later. And keep the timestamps of events precise enough that the order of cause and effect can be reconstructed. None of this requires new technology. It requires someone to argue, at the moment systems and programmes are designed, that the organisation will one day need to know whether they worked. The data science team is usually the only group with both the motive and the knowledge to make that argument. Communicating causal results The last element of the workflow is how results are presented to the people who will act on them. A causal estimate is easy to over-sell and easy to under-sell. Over-selling presents a point estimate from an observational study as if it were an experimental finding. Under-selling buries a useful conclusion under so many caveats that the decision-maker ignores it. Three practices help. First, present the graph alongside the estimate, so that the assumptions are visible and can be challenged by people who know the domain. Second, express sensitivity in terms the audience understands, as in "this conclusion would be overturned only if an unrecorded factor were twice as strongly related to both the call and readmission as prior admissions are", rather than as an abstract statistic. Third, connect the estimate to the decision: say what the result implies for the choice at hand, how confident one should be, and what further evidence would change the recommendation. A team that follows this workflow will produce fewer headline numbers and more conditional conclusions. It will also produce conclusions that hold up when acted upon, which is the only test that ultimately matters. Conclusion It is tempting to read the story of this booklet as a story about better methods: predictive models could not answer causal questions, so the field invented graphs, doubly robust estimators, causal forests and counterfactual fairness criteria, and now they can. That reading is not wrong, but it misses the more important conclusion, which only becomes visible once the methods have been laid side by side. Look back at where each method drew its power. The backdoor criterion worked only when the confounders had been recorded. Propensity weighting worked only when every kind of unit had some chance of each treatment. Instrumental variables worked only when something in the world shifted the treatment for reasons unrelated to the outcome. Regression discontinuity needed a threshold rule; difference-in-differences needed staggered timing; synthetic control needed untreated units that tracked the treated one. Uplift models needed holdout groups. Off-policy evaluation needed logged probabilities and some exploration. Invariant prediction needed data from genuinely different environments. Even the refutation tests in the final chapter needed negative control outcomes that someone had thought to record. In every case, the binding constraint was not the estimator. It was whether the process that generated the data had left enough of the right kind of variation, recorded in enough of the right detail, for the causal question to be answerable. The estimators differ in efficiency and robustness, and those differences matter at the margin. The data-generating process decides whether there is anything to estimate at all. The data are a product of decisions That observation reverses a habit of mind that predictive machine learning encourages. In the predictive workflow, data are given. The analyst receives a table, splits it, fits models and reports accuracy. Questions about how the table came to exist are treated as preliminaries, to be dispatched by a few cleaning steps. The causal workflow cannot treat data this way, because every causal conclusion depends on the reasons why some units were treated and others were not, and those reasons are facts about the organisation's own decisions. This has a practical implication that is easy to state and harder to act on. The most valuable thing a data science team can do to improve its future causal inferences is not to adopt a new library or hire a specialist in econometrics. It is to change how the organisation makes and records decisions, so that the data its operations produce can answer the questions it will later need to ask. That means logging not only what action was taken but why, and with what probability. It means preserving randomised holdouts from campaigns and programmes even when the business is confident they work. It means preferring rollouts that are staggered in a planned order over simultaneous launches. It means writing business rules with explicit thresholds and recording when they change. It means, where a model drives decisions, building in a small amount of exploration so that the model's choices do not erase the evidence needed to evaluate alternatives. None of these practices is technically difficult. All of them cost something in the short term, and that is why they are rarely adopted without someone arguing for them. The argument is the one this booklet has built: without them, the organisation's data will support predictions but not decisions, and every question about whether an intervention works will be answered by models that confound the intervention with the reasons it was chosen. The role of the data scientist changes A second conclusion follows. If causal answers depend on the data-generating process, then the people best placed to secure them are those who understand that process and can shape it. In practice, that shifts the data scientist's role from one who analyses data to one who also designs it, working upstream with the product, clinical or operational teams who make the decisions that become the data. This is a change in professional identity as much as in skills. The predictive data scientist is judged by model accuracy, and the work is largely self-contained. The causal data scientist is judged by whether decisions improve, and the work is inherently collaborative, because the graph cannot be drawn without the people who know why treatments are assigned, and the experiment cannot be run without the people who control the levers. Teams that recognise this organise accordingly. They put analysts in the room when programmes are designed, not only when they are evaluated. They treat a causal analysis plan as a document for review by domain experts, not as an internal note. They measure success by the quality of decisions informed rather than by the number of models deployed. What remains unsettled It would be misleading to end on the suggestion that the hard problems have been solved. Several remain open, and practitioners should know where the ground is soft. Counterfactual reasoning about individuals rests on assumptions about functional form and disturbance distributions that are, in most applications, untestable. Explanations, recourse and fairness criteria built on structural models are only as good as those models, and the field has not yet developed routine ways of quantifying how sensitive their conclusions are to plausible alternatives. Interference between units, ubiquitous in networked and marketplace systems, still lacks methods as mature and easy to use as those for independent units. Causal discovery remains a tool for generating hypotheses rather than a dependable way to learn structure, and its benchmarks have proved easier to game than anyone expected. Large language models offer real help in drafting graphs and recalling mechanisms but cannot reason reliably about causal structure from data, and nothing in their design gives them knowledge of the private processes inside a particular organisation. The deepest open problem is the meeting point of causal inference and representation learning: how to learn, from raw high-dimensional inputs such as images, text and sensor streams, variables that correspond to the causal factors of the world, so that the methods of this booklet can be applied to them. Researchers including Bernhard Schölkopf and his collaborators have framed this as causal representation learning and argued that it is central to building systems that generalise as people do. The framing is persuasive, but in 2026 it remains more a research programme than a set of techniques a working team can adopt. What to do on Monday For the reader who wants to act on this booklet, three changes are worth more than any others. First, translate every modelling request into the decision it serves and the causal question that decision depends on. If the question is predictive, proceed as before. If it is causal, say so, and refuse to let a predictive model answer it by default. Second, draw the graph before writing estimation code, with the people who know how treatments are assigned, and keep it with the analysis as its most important piece of documentation. The graph is where the analysis can be most usefully criticised, and where the most important mistakes are caught. Third, argue for data that can answer next year's questions: holdouts, logged propensities, planned rollouts, recorded thresholds. This is where the largest gains lie, and it is the work that no one else in the organisation is likely to do. Correlation will always be the raw material. The data an organisation collects are associational by nature, and predictive models will remain indispensable for describing them. What changes, once the causal habit is in place, is that the organisation stops expecting associations to answer questions about action, and starts arranging its own decisions so that the evidence about action is there when it is needed. Notes Introduction 1. Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M. and Elhadad, N. (2015). "Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission." Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1721–1730. Chapter 1 1. Charig, C. R., Webb, D. R., Payne, S. R. and Wickham, J. E. A. (1986). "Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy." British Medical Journal, 292, 879–882. 2. Obermeyer, Z., Powers, B., Vogeli, C. and Mullainathan, S. (2019). "Dissecting racial bias in an algorithm used to manage the health of populations." Science, 366(6464), 447–453. 3. Lazer, D., Kennedy, R., King, G. and Vespignani, A. (2014). "The parable of Google Flu: Traps in big data analysis." Science, 343(6176), 1203–1205. 4. Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J. and Oermann, E. K. (2018). "Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study." PLOS Medicine, 15(11), e1002683. 5. Beery, S., Van Horn, G. and Perona, P. (2018). "Recognition in terra incognita." Proceedings of the European Conference on Computer Vision (ECCV). 6. Perdomo, J., Zrnic, T., Mendler-Dünner, C. and Hardt, M. (2020). "Performative prediction." Proceedings of the 37th International Conference on Machine Learning (ICML). Chapter 2 1. Holland, P. W. (1986). "Statistics and causal inference." Journal of the American Statistical Association, 81(396), 945–960. 2. Rubin, D. B. (1974). "Estimating causal effects of treatments in randomized and nonrandomized studies." Journal of Educational Psychology, 66(5), 688–701. 3. Pearl, J. and Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect. New York: Basic Books. 4. Pearl, J. (2009). Causality: Models, Reasoning, and Inference, 2nd edition. Cambridge: Cambridge University Press. Chapter 3 1. Berkson, J. (1946). "Limitations of the application of fourfold table analysis to hospital data." Biometrics Bulletin, 2(3), 47–53. 2. Textor, J., van der Zander, B., Gilthorpe, M. S., Liśkiewicz, M. and Ellison, G. T. H. (2016). "Robust causal inference using directed acyclic graphs: The R package 'dagitty'." International Journal of Epidemiology, 45(6), 1887–1894. Chapter 4 1. Hernández-Díaz, S., Schisterman, E. F. and Hernán, M. A. (2006). "The birth weight 'paradox' uncovered?" American Journal of Epidemiology, 164(11), 1115–1120. 2. Westreich, D. and Greenland, S. (2013). "The table 2 fallacy: Presenting and interpreting confounder and modifier coefficients." American Journal of Epidemiology, 177(4), 292–298. 3. Shpitser, I. and Pearl, J. (2006). "Identification of joint interventional distributions in recursive semi-Markovian causal models." Proceedings of the 21st National Conference on Artificial Intelligence (AAAI). 4. Huang, Y. and Valtorta, M. (2006). "Pearl's calculus of intervention is complete." Proceedings of the 22nd Conference on Uncertainty in Artificial Intelligence (UAI). 5. Deng, A., Xu, Y., Kohavi, R. and Walker, T. (2013). "Improving the sensitivity of online controlled experiments by utilizing pre-experiment data." Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (WSDM). Chapter 5 1. Rosenbaum, P. R. and Rubin, D. B. (1983). "The central role of the propensity score in observational studies for causal effects." Biometrika, 70(1), 41–55. 2. Robins, J. M., Rotnitzky, A. and Zhao, L. P. (1994). "Estimation of regression coefficients when some regressors are not always observed." Journal of the American Statistical Association, 89(427), 846–866. 3. Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W. and Robins, J. (2018). "Double/debiased machine learning for treatment and structural parameters." The Econometrics Journal, 21(1), C1–C68. 4. van der Laan, M. J. and Rubin, D. (2006). "Targeted maximum likelihood learning." The International Journal of Biostatistics, 2(1). 5. LaLonde, R. J. (1986). "Evaluating the econometric evaluations of training programs with experimental data." American Economic Review, 76(4), 604–620. 6. Dehejia, R. H. and Wahba, S. (1999). "Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs." Journal of the American Statistical Association, 94(448), 1053–1062. 7. Smith, J. A. and Todd, P. E. (2005). "Does matching overcome LaLonde's critique of nonexperimental estimators?" Journal of Econometrics, 125(1–2), 305–353. Chapter 6 1. Angrist, J. D. (1990). "Lifetime earnings and the Vietnam era draft lottery: Evidence from Social Security administrative records." American Economic Review, 80(3), 313–336. 2. Angrist, J. D. and Krueger, A. B. (1991). "Does compulsory school attendance affect schooling and earnings?" Quarterly Journal of Economics, 106(4), 979–1014. 3. Bound, J., Jaeger, D. A. and Baker, R. M. (1995). "Problems with instrumental variables estimation when the correlation between the instruments and the endogenous explanatory variable is weak." Journal of the American Statistical Association, 90(430), 443–450. 4. Imbens, G. W. and Angrist, J. D. (1994). "Identification and estimation of local average treatment effects." Econometrica, 62(2), 467–475. 5. Thistlethwaite, D. L. and Campbell, D. T. (1960). "Regression-discontinuity analysis: An alternative to the ex post facto experiment." Journal of Educational Psychology, 51(6), 309–317. 6. McCrary, J. (2008). "Manipulation of the running variable in the regression discontinuity design: A density test." Journal of Econometrics, 142(2), 698–714. 7. Card, D. and Krueger, A. B. (1994). "Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania." American Economic Review, 84(4), 772–793. 8. Goodman-Bacon, A. (2021). "Difference-in-differences with variation in treatment timing." Journal of Econometrics, 225(2), 254–277. 9. Callaway, B. and Sant'Anna, P. H. C. (2021). "Difference-in-differences with multiple time periods." Journal of Econometrics, 225(2), 200–230. 10. Abadie, A., Diamond, A. and Hainmueller, J. (2010). "Synthetic control methods for comparative case studies: Estimating the effect of California's tobacco control program." Journal of the American Statistical Association, 105(490), 493–505. 11. VanderWeele, T. J. and Ding, P. (2017). "Sensitivity analysis in observational research: Introducing the E-value." Annals of Internal Medicine, 167(4), 268–274. 12. Brodersen, K. H., Gallusser, F., Koehler, J., Remy, N. and Scott, S. L. (2015). "Inferring causal impact using Bayesian structural time-series models." Annals of Applied Statistics, 9(1), 247–274. 13. Cinelli, C. and Hazlett, C. (2020). "Making sense of sensitivity: Extending omitted variable bias." Journal of the Royal Statistical Society: Series B, 82(1), 39–67. Chapter 7 1. Künzel, S. R., Sekhon, J. S., Bickel, P. J. and Yu, B. (2019). "Metalearners for estimating heterogeneous treatment effects using machine learning." Proceedings of the National Academy of Sciences, 116(10), 4156–4165. 2. Nie, X. and Wager, S. (2021). "Quasi-oracle estimation of heterogeneous treatment effects." Biometrika, 108(2), 299–319. 3. Kennedy, E. H. (2023). "Towards optimal doubly robust estimation of heterogeneous causal effects." Electronic Journal of Statistics, 17(2), 3008–3049. 4. Athey, S. and Imbens, G. (2016). "Recursive partitioning for heterogeneous causal effects." Proceedings of the National Academy of Sciences, 113(27), 7353–7360. 5. Wager, S. and Athey, S. (2018). "Estimation and inference of heterogeneous treatment effects using random forests." Journal of the American Statistical Association, 113(523), 1228–1242. 6. Athey, S., Tibshirani, J. and Wager, S. (2019). "Generalized random forests." Annals of Statistics, 47(2), 1148–1178. 7. Yadlowsky, S., Fleming, S., Shah, N., Brunskill, E. and Wager, S. (2025). "Evaluating treatment prioritization rules via rank-weighted average treatment effects." Journal of the American Statistical Association, 120(549), 38–51. 8. Athey, S. and Wager, S. (2021). "Policy learning with observational data." Econometrica, 89(1), 133–161. 9. Dudík, M., Langford, J. and Li, L. (2011). "Doubly robust policy evaluation and learning." Proceedings of the 28th International Conference on Machine Learning (ICML). Chapter 8 1. Tian, J. and Pearl, J. (2000). "Probabilities of causation: Bounds and identification." Annals of Mathematics and Artificial Intelligence, 28, 287–313. 2. Wachter, S., Mittelstadt, B. and Russell, C. (2017). "Counterfactual explanations without opening the black box: Automated decisions and the GDPR." Harvard Journal of Law and Technology, 31(2), 841–887. 3. Karimi, A.-H., Schölkopf, B. and Valera, I. (2021). "Algorithmic recourse: From counterfactual explanations to interventions." Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT). 4. Kusner, M. J., Loftus, J., Russell, C. and Silva, R. (2017). "Counterfactual fairness." Advances in Neural Information Processing Systems 30. 5. Bickel, P. J., Hammel, E. A. and O'Connell, J. W. (1975). "Sex bias in graduate admissions: Data from Berkeley." Science, 187(4175), 398–404. 6. Peters, J., Bühlmann, P. and Meinshausen, N. (2016). "Causal inference by using invariant prediction: Identification and confidence intervals." Journal of the Royal Statistical Society: Series B, 78(5), 947–1012. 7. Arjovsky, M., Bottou, L., Gulrajani, I. and Lopez-Paz, D. (2019). "Invariant risk minimization." arXiv:1907.02893. 8. Gulrajani, I. and Lopez-Paz, D. (2021). "In search of lost domain generalization." International Conference on Learning Representations (ICLR). 9. Kaushik, D., Hovy, E. and Lipton, Z. C. (2020). "Learning the difference that makes a difference with counterfactually-augmented data." International Conference on Learning Representations (ICLR). 10. Spirtes, P., Glymour, C. and Scheines, R. (2000). Causation, Prediction, and Search, 2nd edition. Cambridge, MA: MIT Press. 11. Shimizu, S., Hoyer, P. O., Hyvärinen, A. and Kerminen, A. (2006). "A linear non-Gaussian acyclic model for causal discovery." Journal of Machine Learning Research, 7, 2003–2030. 12. Zheng, X., Aragam, B., Ravikumar, P. and Xing, E. P. (2018). "DAGs with NO TEARS: Continuous optimization for structure learning." Advances in Neural Information Processing Systems 31. 13. Reisach, A. G., Seiler, C. and Weichwald, S. (2021). "Beware of the simulated DAG! Causal discovery benchmarks may be easy to game." Advances in Neural Information Processing Systems 34. 14. Kıcıman, E., Ness, R., Sharma, A. and Tan, C. (2023). "Causal reasoning and large language models: Opening a new frontier for causality." arXiv:2305.00050. 15. Jin, Z., Liu, J., Lyu, Z., Poff, S., Sachan, M., Mihalcea, R., Diab, M. and Schölkopf, B. (2024). "Can large language models infer causation from correlation?" International Conference on Learning Representations (ICLR). 16. Jin, Z., Chen, Y., Leeb, F., Gresele, L., Kamal, O., Lyu, Z., Blin, K., Gonzalez Adauto, F., Kleiman-Weiner, M., Sachan, M. and Schölkopf, B. (2023). "CLadder: Assessing causal reasoning in language models." Advances in Neural Information Processing Systems 36. 17. Zečević, M., Willig, M., Dhami, D. S. and Kersting, K. (2023). "Causal parrots: Large language models may talk causality but are not causal." Transactions on Machine Learning Research. Chapter 9 1. Hernán, M. A. and Robins, J. M. (2016). "Using big data to emulate a target trial when a randomized trial is not available." American Journal of Epidemiology, 183(8), 758–764. 2. Lipsitch, M., Tchetgen Tchetgen, E. and Cohen, T. (2010). "Negative controls: A tool for detecting confounding and bias in observational studies." Epidemiology, 21(3), 383–388. 3. Kohavi, R., Tang, D. and Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge: Cambridge University Press. Conclusion 1. Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A. and Bengio, Y. (2021). "Toward causal representation learning." Proceedings of the IEEE, 109(5), 612–634. Further Reading Miguel A. Hernán and James M. Robins, Causal Inference: What If (Chapman & Hall/CRC, 2020).** The clearest rigorous introduction to potential outcomes, graphs, confounding, selection bias and the g-methods, written by two of the field's leading figures. The authors have made it freely available online. Read Part I first. Judea Pearl and Dana Mackenzie, The Book of Why (Basic Books, 2018).** A non-technical account of the ladder of causation, the do-operator and the history of the causal revolution. Opinionated and occasionally combative, but the best way to absorb the structural view quickly. Judea Pearl, Madelyn Glymour and Nicholas P. Jewell, Causal Inference in Statistics: A Primer (Wiley, 2016).** A short, technical bridge between the popular book and Pearl's Causality, with exercises on d-separation, the backdoor and front-door criteria, and counterfactuals. Guido W. Imbens and Donald B. Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences (Cambridge University Press, 2015).** The authoritative treatment of the potential-outcomes framework, with particular depth on randomised experiments, propensity scores and matching. Joshua D. Angrist and Jörn-Steffen Pischke, Mostly Harmless Econometrics (Princeton University Press, 2009).** The econometric view of instrumental variables, regression discontinuity and difference-in-differences, written with wit and an eye on practical application. Supplement with recent literature on staggered difference-in-differences. Scott Cunningham, Causal Inference: The Mixtape (Yale University Press, 2021).** An accessible tour of the econometric designs, with code in R and Stata, and a free online edition. Good for the material in Chapter 6. Matheus Facure, Causal Inference in Python (O'Reilly, 2023).** A practitioner's guide aimed squarely at industry data scientists, covering meta-learners, double machine learning and evaluation of heterogeneous effects with working Python code. Jonas Peters, Dominik Janzing and Bernhard Schölkopf, Elements of Causal Inference: Foundations and Learning Algorithms (MIT Press, 2017).** The machine learning perspective on structural causal models, causal discovery and invariance, freely available online. Demanding but rewarding for readers interested in Chapter 8. Victor Chernozhukov, Christian Hansen, Nathan Kallus, Martin Spindler and Vasilis Syrgkanis, Applied Causal Inference Powered by ML and AI (online, 2024).** A free textbook from some of the authors of double machine learning, linking modern estimation theory to practical code. Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020).** The standard guide to running A/B tests well, including the pitfalls that turn an experiment into an observational study without anyone noticing. Software. DoWhy (model, identify, estimate, refute) and EconML (double machine learning and heterogeneous effects), both under the PyWhy open-source organisation; CausalML for uplift modelling; grf for causal forests in R; dagitty for drawing graphs and computing adjustment sets, available as an R package and a web tool. Glossary Average treatment effect (ATE). The difference between the average outcome if every unit in a population received the treatment and the average outcome if none did. Backdoor path. A path between treatment and outcome that begins with an arrow pointing into the treatment. Unblocked backdoor paths carry confounding. Collider. A variable on a path that is caused by both of its neighbours on that path. A collider blocks association unless it, or one of its descendants, is conditioned on. Conditional average treatment effect (CATE). The average treatment effect among units sharing specified characteristics; the target of uplift modelling and targeting. Confounder. A common cause of treatment and outcome, which creates an association between them that is not the treatment's effect. Counterfactual. A statement about what would have happened to a specific unit, given what actually happened, had some variable taken a different value. Directed acyclic graph (DAG). A diagram of variables connected by arrows, with no directed cycles, used to encode assumptions about which variables directly cause which. Doubly robust estimator. An estimator that combines an outcome model and a propensity model and remains consistent if either one is correctly specified. Exchangeability. The condition that treated and untreated groups would have had the same distribution of potential outcomes; guaranteed by randomisation and assumed, conditionally on covariates, by adjustment methods. Identification. The property that a causal quantity can be expressed in terms of the distribution of observed data, given stated assumptions. Mediator. A variable on a causal path from treatment to outcome, through which part of the treatment's effect is transmitted. Positivity. The requirement that every type of unit for which an effect is estimated has a non-zero probability of receiving each treatment. Potential outcome. The outcome a unit would experience under a specified treatment, whether or not that treatment was actually received. Propensity score. The probability that a unit receives treatment given its covariates. Hashtags: #CausalInference #CausalAI #ArtificialIntelligence #MachineLearning #CorrelationVsCausation #CausalReasoning #CausalMachineLearning #CausalDiscovery #CausalGraphs #DirectedAcyclicGraphs #StructuralCausalModels #PotentialOutcomes #CounterfactualReasoning #CausalEffectEstimation #AverageTreatmentEffect #ConfoundingBias #ColliderBias #CausalIdentification #DoCalculus #PropensityScoreMatching #InverseProbabilityWeighting #DoublyRobustEstimation #DoubleMachineLearning #CausalForests #HeterogeneousTreatmentEffects #UpliftModeling #InstrumentalVariables #DifferenceInDifferences #AlgorithmicFairness #CausalDecisionMaking

Download the Book (PDF): Introduction On the afternoon of 17 June 2025, two tankers collided in the Gulf of Oman, about fifteen nautical miles off the port of Fujairah. One was the Front Eagle, a very large crude carrier loaded with roughly two million barrels of Iraqi oil bound for China. The other was the Suezmax tanker Adalynn. Both caught fire. Nobody was hurt, which was luck as much as anything. For the previous week, since open conflict between Israel and Iran began on 12 June, satellite positioning across the Gulf and the Strait of Hormuz had been subject to heavy interference. The maritime analytics firm Windward reported that close to a thousand ships had been affected. On tracking screens, vessels appeared to sit inland, at airports, in places no ship could be. The investigation into that particular collision will settle its precise causes, but the circumstance is not in dispute: two professionally crewed ships, with radar, with lookouts, with every navigational aid the twenty-first century offers, came together in open water while the system that tells every modern bridge where it is had been made unreliable. That incident is a useful place to begin a book about ships that have no one on the bridge at all, because it removes two comfortable illusions at once. The first is that crewed ships are safe and autonomous ships are the risky novelty. Ships collide, ground and burn every year under human command, and the investigation reports that follow them return again and again to the same themes: a lookout that was not kept, a radar target that was noticed too late, an assumption about the other ship's intentions that turned out to be wrong, a fatigued officer at the end of a long watch. The second illusion runs the other way. It is the belief that a machine, once it has enough sensors and a fast enough computer, will simply see everything and calculate its way out of trouble. The Gulf of Oman shows that the inputs themselves can lie, and that a navigation system is only as good as its ability to notice when they do. This book is about the engineering that sits between those two illusions. It covers how an uncrewed or lightly crewed cargo ship builds a picture of the world from marine radar, the Automatic Identification System, cameras and other sensors; how it decides what to do when another vessel is on a collision course, under rules written in 1972 for human seamen; how it chooses a route across an ocean that burns as little fuel as possible; and how the satellite links that tie it to a control centre ashore are protected against interference and attack. The ambition is to explain the mechanisms clearly enough that a reader with no maritime background can follow how they work, where they fail, and why the people building them make the choices they do. The argument of this book The controlling idea is simple to state. The hard part of taking crews off cargo ships is not steering. Ships have steered themselves along a set course for a century; autopilots and track-keeping systems are mature, and a computer can follow a planned route more precisely than any helmsman. The hard part is judgement under uncertainty: forming a trustworthy picture of the surroundings from sensors that are each partial and fallible, interpreting rules that were written to be applied by experienced seafarers with common sense, and remaining safe when the communication links and positioning systems that the whole arrangement depends on degrade or fail. From that follows a design principle that runs through every chapter. A safe autonomous ship is not one that never needs help. It is one that knows the limits of its own knowledge, says so, and has a well-rehearsed way of making itself safe when it reaches them. That principle is easy to agree with and hard to engineer. It shapes how sensor data is fused, so that the system carries an honest estimate of its own uncertainty rather than a single confident answer. It shapes how collision avoidance is designed, favouring early, large and readable manoeuvres that other ships can understand. It shapes routing, where the cleverest fuel-saving plan is worthless if it puts the ship somewhere it cannot be supported. And it shapes cybersecurity, where the defensive question is not only how to keep intruders out but how the ship behaves when a link is lost, a signal is spoofed or a system is compromised. What changed, and why now For most of the past decade, autonomous shipping was a field of demonstrations. A Finnish car ferry, the Falco, sailed autonomously through the Turku archipelago in December 2018 with Rolls-Royce's marine business, later absorbed into Kongsberg. The Norwegian fertiliser company Yara commissioned the Yara Birkeland, a small battery-electric container ship designed to sail without crew between its plant at Porsgrunn and the export port at Brevik. The research vessel Mayflower crossed the Atlantic without people aboard in 2022. In the same year an LNG carrier fitted with HD Hyundai's Avikus navigation system made roughly half of a Pacific crossing under autonomous control. Each of these was real and instructive, and each was hedged with qualifications: crew on board as backup, special permissions, short protected routes, research rather than commerce. Two developments in 2026 changed the character of the field. In January and March, two ships under Japan's MEGURI2040 programme, run by the Nippon Foundation, entered commercial service with certification from both the classification society ClassNK and Japan's transport ministry for what the programme calls Level 4-equivalent autonomous navigation in designated waters: the container ship Genbu, serving ports between Kobe and Tokyo, and the roll-on roll-off cargo ship Hokuren Maru No. 2, running between Hitachi and Kushiro. And in May, the International Maritime Organization's Maritime Safety Committee, at its 111th session, adopted the International Code of Safety for Maritime Autonomous Surface Ships, the MASS Code. It is non-mandatory for now. The IMO's stated plan is to gather experience, develop a mandatory version, adopt it around July 2030 and bring it into force on 1 January 2032. But the existence of an agreed international framework means that the questions this book addresses are no longer speculative. They are the questions regulators, class societies, shipowners and insurers are now asking in earnest. How the book is organised The first chapter sets the stage: why anyone would want to take crews off ships, what autonomy actually means in a maritime context, and what the real projects have and have not demonstrated. Chapters 2 and 3 deal with perception. Chapter 2 explains marine radar and AIS, the two systems that every commercial bridge already relies on, and their characteristic failures. Chapter 3 adds cameras and machine vision and explains how a ship fuses these streams into a single tracked picture of the world, with an honest estimate of its own uncertainty. Chapters 4 and 5 deal with decisions. Chapter 4 takes the International Regulations for Preventing Collisions at Sea, the COLREGs, and asks what it takes to turn rules written for human judgement into something a machine can follow. Chapter 5 examines the algorithms that actually choose manoeuvres, from velocity obstacles to model predictive control, and walks through the special difficulty of congested ports and traffic separation schemes. Chapter 6 turns from minutes to days and weeks: voyage optimisation, weather routing and speed management, and how autonomy changes the economics of fuel and emissions. Chapter 7 examines the link between ship and shore, the satellite communications that make remote supervision possible, and the defensive cybersecurity that the industry's standards now require, including the classification societies' unified requirements UR E26 and E27. Chapter 8 brings the threads together around the question of failure: how an autonomous ship reaches a safe state when things go wrong, how such a system is verified, and how responsibility is assigned between machine, remote operator and master. The conclusion argues for what follows from all this for the people who will build, regulate, insure and work alongside these ships. Throughout, the book uses real systems, real incidents and real regulatory instruments, and is candid about where the evidence is thin. Autonomous shipping is a young field with a long tail of unresolved questions, and the most useful thing an expert can offer a newcomer is a clear sense of which problems are solved, which are being solved, and which remain genuinely open. Chapter 1: Why Take the Crew Off Ask a shipowner why they are interested in autonomous ships and the first answer is rarely the one outsiders expect. It is not usually the wage bill. On a large container ship or bulk carrier, the cost of a crew of twenty or so is real but modest beside the cost of fuel, capital and port charges. The answers that come up instead are about people who cannot be found, accidents that keep happening for the same reasons, and the chance to redesign the ship itself once the assumption that two dozen humans must live aboard for months at a time is relaxed. This chapter sets out those motives, defines what autonomy means in the maritime world, and takes stock of what the real projects have shown so far. The case for autonomy, honestly stated Shipping carries more than four-fifths of world trade by volume, according to the United Nations Conference on Trade and Development, and it does so with a workforce that is small relative to the work it does. The most recent detailed census of that workforce, the Seafarer Workforce Report published by BIMCO and the International Chamber of Shipping in 2021, estimated the global seafarer population at about 1.89 million. It also estimated a shortfall of around 26,000 certificated officers at that time, and projected that the industry would need roughly 89,500 additional officers by 2026 to keep pace with fleet growth. Projections of this kind always carry uncertainty, and the shortfall varies by rank and by ship type, but the underlying trend has been consistent across successive editions of the report: experienced officers, particularly engineers and senior deck officers, are hard to recruit and harder to keep. Long tours away from home, the isolation that the COVID-19 crew-change crisis made painfully visible, and better-paid jobs ashore all pull in the same direction. The second motive is safety. Marine casualty investigations, from national bodies such as the United Kingdom's Marine Accident Investigation Branch, the United States National Transportation Safety Board and the Danish Maritime Accident Investigation Board, return repeatedly to human factors: fatigue, poor lookout, misreading of radar, poor communication between bridge team and pilot, or an officer who did not challenge a plan he had doubts about. Marine insurers have for years cited estimates that human error is a factor in somewhere between three-quarters and almost all marine incidents. Those figures need careful handling. "Human error" is often the label applied to whatever happened last in a chain that began with design, training, manning or commercial pressure, and a machine can only remove the errors it does not replace with errors of its own. But the pattern points to something real. A great deal of maritime risk sits in the routine, repetitive, attention-demanding task of keeping watch, and that is precisely the kind of task at which a well-designed machine, which does not get tired at four in the morning, ought to excel. The third motive is design freedom, and it is underrated. A conventional cargo ship carries an accommodation block, life-saving appliances for its crew, fresh water and sewage systems, galley and stores, air conditioning, and a bridge positioned high enough for a human to see over the deck cargo. Every one of these costs weight, space, power and maintenance. Remove the need for people to live aboard and the ship can be reshaped. The European Union's MUNIN project, which ran from 2012 to 2015 and was one of the first serious feasibility studies of an unmanned bulk carrier, examined exactly these trade-offs, along with the operational concept of a shore control centre supervising ships in deep-sea passage. Its conclusions were cautious, and they remain relevant: the economic case improved most when the ship was designed from the start for unmanned operation, rather than when an existing design was simply relieved of its crew. There is also a fourth motive, which is less often stated and more important than it looks: autonomy as decision support for crewed ships. Many of the technologies in this book, from camera-based target detection to automated collision-avoidance advice to fuel-optimised routing, are being sold first to ordinary crewed vessels as aids to the officer on watch. Israeli firm Orca AI sells camera-based situational awareness systems for this purpose; Avikus, the autonomy subsidiary of HD Hyundai, sells its HiNAS system as navigation assistance on crewed ships. For the foreseeable future, most of the commercial value of maritime autonomy will be delivered to ships that still have people on board. The fully crewless ship is the end of a gradient, not a separate technology. It is fair, finally, to state the case against. Ships at sea are a long way from help. A crew does far more than navigate. Engineers repair pumps, clear blocked filters, change fuel purifiers, chase down electrical faults and keep a large and complicated plant running for weeks without outside assistance. On 26 March 2024, the container ship Dali lost electrical power twice while leaving Baltimore and struck the Francis Scott Key Bridge, bringing it down and killing six road workers. The US National Transportation Safety Board's final report, released in November 2025, traced the blackout to a single loose wire in the ship's electrical system. A crew could not prevent that accident once the power was gone. But it is a sharp reminder that the hardest part of operating a ship without people may not be navigation at all. It may be the thousand small mechanical and electrical interventions that crews perform every voyage, which a crewless ship must either design out, monitor remotely or simply do without. What "autonomous" means at sea The word autonomy is used loosely, and in shipping the loose usage has caused real confusion. The IMO's Maritime Safety Committee, in the regulatory scoping exercise it completed in 2021, used four degrees of autonomy to sort the problem. In the first degree, the ship has automated processes and decision support, but seafarers are on board to operate and control the systems, and some operations may be automated and unsupervised. In the second, the ship is remotely controlled from another location but seafarers are still aboard and can take over. In the third, the ship is remotely controlled and no seafarers are aboard. In the fourth, the ship is fully autonomous: its operating system is able to make decisions and determine actions by itself. The degrees are not a ladder that a ship climbs once. A single vessel may operate at different degrees in different phases of a voyage, perhaps with a remote operator actively in control during port departure, autonomously in open sea, and with a pilot boarding for the final approach. Many practical designs blend remote control and onboard autonomy: the onboard system handles routine navigation and collision avoidance, a remote operations centre ashore supervises several ships at once, and the remote operator intervenes when the system requests help or when something unusual happens. That arrangement, sometimes called supervised autonomy, is the dominant concept in current projects, and it is the one assumed by most of this book. Industry projects also use their own level scales, which do not always map cleanly onto the IMO degrees. When the MEGURI2040 programme describes its ships as operating at "Level 4-equivalent" autonomy, it means fully automated navigation within designated waters, under a framework approved by the Japanese authorities and ClassNK. When Avikus describes HiNAS 2.0 as a Level 2 system, it means navigation assistance that can steer and manoeuvre with an officer still responsible on the bridge. Readers of press releases should always ask three questions: who is legally in command, where are they, and what happens if the system stops? The MASS Code adopted by the IMO in May 2026 answers the first of these questions firmly. Under the code, the master retains overall responsibility for the ship at all times, even when not on board. That single principle has large consequences. It means a crewless ship still has a master, who may sit in a remote operations centre, and it means the onboard system is treated not as a legal actor but as equipment, however sophisticated, operated under that master's authority. The code is goal-based: rather than prescribing particular technologies, it requires that autonomous and remotely controlled ships achieve a level of safety at least equivalent to conventional ships, and it sets out requirements on risk assessment, system design, connectivity, remote operations centres, navigation, fire safety and search and rescue that designers must show they meet. What has actually been demonstrated It is worth being precise about the real projects, because the gap between headline and substance is often wide. Table 1 summarises several of the most significant, with their status as publicly reported. Table 1. Selected autonomous ship projects and what they demonstrated. Vessel or project Type and route Year What was shown Falco (Finferries, Rolls-Royce) Car ferry, Turku archipelago, Finland 2018 Autonomous transit and automatic docking, with remote supervision Yara Birkeland (Yara, Kongsberg) 120 TEU battery-electric feeder, Porsgrunn to Brevik, Norway 2022 onward Commercial short-sea service with progressively reduced bridge crew Mayflower Autonomous Ship (ProMare, IBM) Research trimaran, Plymouth to North America 2022 Uncrewed Atlantic crossing, with technical diversions Prism Courage (SK Shipping, Avikus) 180,000-tonne LNG carrier, US Gulf to Korea 2022 About half of a 20,000 km voyage under autonomous navigation Genbu and Hokuren Maru No. 2 (MEGURI2040) Container ship and ro-ro, Japanese domestic routes 2026 Certified commercial service with Level 4-equivalent navigation in designated waters The Yara Birkeland is the most-cited example and the most instructive. It is a small ship, about 80 metres long with capacity for 120 twenty-foot containers, powered by a battery bank of roughly seven megawatt-hours, and it sails a short, sheltered, well-surveyed route of a few hours between Yara's fertiliser plant and the container port at Brevik. Kongsberg reported in 2024 that at full operation it would remove some 40,000 truck journeys a year from Norwegian roads. It entered commercial service in 2022 with crew aboard, and as of Kongsberg's 2024 account still carried three people on the bridge at all times, down from five at launch, while the system's autonomy was tested under supervision from a shore control centre and the operator sought regulatory approval for crewless operation. The direction of travel is clear, and so is the pace: years of supervised operation on a short protected route, with each reduction in crew earned by evidence. The Avikus voyage of the Prism Courage in May and June 2022 was a very different kind of demonstration. The LNG carrier sailed from Freeport, Texas, through the Panama Canal to the Boryeong terminal in South Korea, a journey of some 20,000 kilometres, with the HiNAS 2.0 system navigating for roughly half of it under the watch of the classification societies ABS and the Korean Register. Avikus reported that the system recognised and avoided nearby ships about a hundred times and that optimised routing improved fuel efficiency by around seven per cent and reduced greenhouse gas emissions by about five per cent. Those are vendor figures from a single voyage, and the comparison baseline matters, but the demonstration showed that the technologies of perception, collision avoidance and route optimisation could run together on a very large ship in real ocean traffic. The MEGURI2040 ships of 2026 mark a further step: commercial cargo service, on regular domestic routes, with certification from both the class society and the national maritime authority for automated navigation in designated waters. The programme ran an earlier phase of demonstrations in 2022, in which several vessels, including the container feeder Suzaku in Tokyo Bay, conducted autonomous voyages through busy coastal waters. The current phase has turned those demonstrations into certified service. As with the Yara Birkeland, the important words are "designated waters". Every credible autonomous ship operating today does so inside an operational envelope: a defined route, defined weather limits, defined traffic conditions and defined fallback arrangements. Much of the engineering, and much of this book, is about how that envelope is defined, monitored and enforced. Where autonomy arrives first The pattern of these projects suggests where crewless and lightly crewed operation is likely to become routine first, and it is not on the largest ships on the longest routes. Three kinds of operation stand out. The first is short-sea and domestic shipping on fixed routes: feeders, ferries, ro-ro services and coastal cargo runs that repeat the same passage many times. A fixed route can be surveyed in detail, its traffic patterns learned, its fallback anchorages and safe areas mapped in advance, and its communications coverage engineered, often with terrestrial mobile networks for much of the way. Repetition also generates evidence quickly: a ship making several passages a day accumulates in a year the kind of operational record that would take a deep-sea vessel a decade. The Norwegian and Japanese projects both fit this pattern, as do the various small autonomous passenger ferries trialled in Scandinavian cities. The second is specialised work where the crew is a cost and a risk without being central to the job. Offshore survey is the clearest case. Mapping the seabed for cables, pipelines and wind farms involves long, slow, repetitive lines in often rough water, work that has traditionally required a crewed survey ship to stay at sea for weeks. Several companies now operate uncrewed or optionally crewed surface vessels for this purpose, supervised from shore. The UK company Ocean Infinity has built its business around a fleet of robotic vessels for survey and other offshore tasks, and smaller uncrewed survey craft are widely used by hydrographic offices and offshore contractors. These vessels do not carry cargo, but they are building the operational experience, remote operations practices and regulatory precedents on which cargo autonomy will draw. The third is the use of autonomy within crewed ships to reduce the burden of watchkeeping, especially at night and in open water. A ship whose system can maintain a reliable lookout and propose collision-avoidance manoeuvres, under the supervision of a single officer, is a step towards the reduced-crew ship, and many of the commercial products on the market are designed for exactly this use. Sea Machines, a US company, demonstrated a long autonomous voyage around Denmark with a tug in 2022 under remote supervision from the United States, and markets its systems both for uncrewed workboats and as a capability for crewed vessels. Deep-sea crewless operation of large cargo ships, the image most people have of autonomous shipping, is likely to come last, because it combines every difficulty at once: long periods far from help, the full complexity of machinery that must run unattended, heavy weather, and port approaches into the world's most congested waters at both ends of the voyage. It is not impossible, and the technologies described in this book are designed with it in mind. But the route to it runs through these earlier stages. The demonstrations reveal a consistent shape. Following a route is straightforward. Reliable perception, especially of small, low and unlit objects, is harder than it looks. Collision avoidance in open water against one or two predictable ships is tractable; collision avoidance in a crowded port approach full of fishing boats, pilot launches, ferries and yachts, some of them not transmitting any identification and some of them not following the rules, is the open frontier. Fuel-optimised routing is mature for crewed ships and becomes more interesting when no crew comfort constraints apply. And the link to shore, without which supervised autonomy is impossible, is both a lifeline and an attack surface. These are the topics of the chapters that follow. The next chapter begins where every ship's picture of the world begins: with radar and with AIS, two technologies that every commercial ship already carries, and whose strengths and weaknesses determine much of what an autonomous navigation system can and cannot do. Chapter 2: Radar and AIS, the Two Old Eyes Every merchant ship of any size already carries two systems that tell it about other vessels: marine radar, which has been standard on commercial bridges since the years after the Second World War, and the Automatic Identification System, made mandatory under the Safety of Life at Sea Convention in the early 2000s. An autonomous ship begins with the same two systems, and the difference between a competent and an incompetent design often comes down to how deeply its engineers understand what each one actually measures. They are not two versions of the same thing. Radar is an active sensor that observes the world. AIS is a radio network in which ships describe themselves. One sees whatever reflects; the other hears whatever chooses to speak. Most of what follows in this book depends on that distinction. How marine radar sees A navigation radar sends out short pulses of microwave energy from a rotating antenna and listens for the echoes. The time between transmission and echo gives the range to whatever reflected the pulse; the direction the antenna was pointing gives its bearing. Each rotation, typically every two to three seconds, paints a fresh picture of everything within range that reflects enough energy to stand out from the background. Commercial ships use two frequency bands. X-band radars operate around 9.4 gigahertz, a wavelength of about three centimetres. S-band radars operate around 3 gigahertz, a wavelength of about ten centimetres. The choice is a trade-off that every officer learns early. The shorter X-band wavelength allows a narrower beam from an antenna of a given size, and so gives sharper bearing resolution and better detection of small targets. But X-band energy is scattered and absorbed more strongly by rain, and it picks up more sea clutter, the echoes from waves themselves. S-band sees through heavy rain far better and suffers less from sea clutter, at the cost of a wider beam and a larger antenna. Large ships typically carry one of each, and the international carriage rules require ships of 3,000 gross tonnage and above to have two independent radars. For an autonomous system this redundancy is not a luxury. Two radars at different frequencies fail differently, and the comparison between them is itself information. Resolution matters more than is often appreciated, and it can be worked out from first principles. Range resolution depends on pulse length: two targets on the same bearing can be told apart only if their echoes do not overlap, so the minimum separation is half the distance light travels during one pulse. A short pulse of 0.05 microseconds, used on short range scales, gives a range resolution of about 7.5 metres. Bearing resolution depends on the horizontal width of the beam. A typical X-band antenna of about two metres has a beam roughly 1.2 degrees wide. At a range of six nautical miles, about 11,100 metres, that beam is roughly 230 metres across. Two small boats closer together than that, at the same range, will merge into a single echo. A human operator learns to suspect this when an echo looks elongated; an algorithm must be designed to suspect it too. Range itself is limited by geometry before it is limited by power. Radar energy travels in nearly straight lines, bent slightly by the atmosphere, so a target below the radar horizon cannot be seen at all. A standard approximation gives the radar horizon in nautical miles as about 2.2 times the square root of the antenna height in metres. A ship with its radar antenna 30 metres above the sea has a radar horizon of roughly 12 nautical miles; if the target's own reflecting surfaces stand 10 metres high, the maximum detection range rises to about 19 nautical miles, because the target's horizon adds to the radar's. A large container ship will be detected near that limit. A fibreglass yacht with its mast and hull a few metres high will be found much later, and a kayak or a semi-submerged container perhaps not until it is uncomfortably close, if at all. That last point leads to radar's most important weakness for autonomous navigation: small, low and weakly reflecting targets. A target's visibility to radar depends on its radar cross-section, which is a function of size, shape and material. Steel ships with flat surfaces and corners are excellent reflectors. Wooden and fibreglass boats are poor ones, which is why small craft are encouraged to carry radar reflectors. In a moderate sea, the echoes from waves near the ship form a bright, flickering field of clutter, and the radar's clutter-suppression controls, which reduce gain near the ship to make that field disappear, can just as easily make a small boat in it disappear too. Human radar observers manage this by adjusting the controls continually and by watching for an echo that persists across many rotations while the clutter changes. Automated systems must do the same thing statistically, and the setting of those thresholds is a deliberate trade between false alarms and missed targets that no design can avoid. Rain clutter, shadow sectors behind the ship's own masts and funnels, and false echoes add further complexity. False echoes can arise from side lobes of the antenna pattern near a large reflector, from energy bouncing between the ship's own structure and a target, or from a pulse reflecting off a nearby ship's side and returning along an indirect path. Bridges, cranes, and large vessels alongside a quay all produce such artefacts. In a busy port they are routine. Since the mid-2010s, solid-state radars have increasingly replaced the traditional magnetron design. Instead of a short, very powerful pulse, they transmit longer, lower-power coded pulses and use pulse compression to recover range resolution. Many also apply Doppler processing, measuring the frequency shift of returns to separate moving targets from the static clutter of waves or rain. For an autonomous system the improvement is significant: Doppler information gives a direct measurement of radial velocity that the classical radar could only infer by watching an echo move across several rotations. From echoes to tracks A raw radar picture is not yet useful for collision avoidance. What a ship needs is a set of tracks: objects with estimated position, course and speed, persisting over time. The process of turning echoes into tracks was automated long ago. The Automatic Radar Plotting Aid, ARPA, became a carriage requirement for larger ships from the 1980s, and today's radars include target-tracking functions governed by the IMO's performance standards for radar equipment, adopted in 2004 as resolution MSC.192(79). Tracking works by association and filtering. Each rotation produces a set of detected echoes. The tracker predicts where each existing track should be now, based on its last estimated course and speed, and looks for an echo near that prediction. If it finds one, it updates the track, blending the prediction and the measurement according to how much it trusts each. If it finds none, it coasts the track forward and lowers its confidence; after enough misses, the track is dropped. Echoes that match no track may be the start of a new one. From the tracks come the two numbers that dominate collision avoidance at sea: the closest point of approach, CPA, and the time to closest point of approach, TCPA. If both ships hold their present courses and speeds, how close will they come, and when? The calculation is simple geometry on relative motion. Suppose own ship is heading north at 15 knots and a target bears 045 degrees at 6 nautical miles, heading west at 12 knots. Subtracting own ship's velocity from the target's gives the target's motion relative to own ship: 12 knots west and 15 knots south, a relative speed of about 19.2 knots. Projecting the target's current relative position along that relative motion line shows how close it will pass and when. Here the target starts about 4.24 miles east and 4.24 miles north; moving relatively south-west at those rates, it will pass about two-thirds of a mile off at its nearest, a little under 19 minutes from now, crossing own ship's track about a mile astern. For two large ships in open water, a master's standing orders would typically treat that as far too close and require action. An autonomous system arrives at the same number in microseconds. The catch lies in the inputs. The IMO standards require that a tracking radar show a target's motion trend within about one minute of steady tracking and a full motion prediction within about three. Until then, the estimated course and speed are noisy, and the CPA derived from them can swing widely. When the target itself manoeuvres, the tracker takes further time to recognise the change, because a single echo that deviates from the prediction looks just like noise. Collision avoidance designs must therefore treat CPA not as a fixed number but as an estimate with an uncertainty that grows when a track is young or when a target has recently turned. Chapter 3 returns to this when it discusses sensor fusion. How AIS speaks AIS works on an entirely different principle. Every ship carrying a Class A transponder, which SOLAS requires of all ships of 300 gross tonnage and above on international voyages, cargo ships of 500 gross tonnage and above on domestic voyages, and all passenger ships regardless of size, broadcasts its own identity and state on two VHF maritime channels near 162 megahertz. Each transmission contains a nine-digit Maritime Mobile Service Identity, the ship's position from its own satellite positioning receiver, its course and speed over ground, its heading from the gyrocompass, its rate of turn and a navigational status such as "under way using engine" or "at anchor". Separately, every six minutes, the ship broadcasts static and voyage data: name, call sign, dimensions, ship type, draught and destination. The reporting rate scales with how quickly the situation can change. A Class A ship at anchor reports every three minutes. Under way below 14 knots it reports every ten seconds, rising to every three and a third seconds when changing course; between 14 and 23 knots, every six seconds or every two when turning; above 23 knots, every two seconds. The ships coordinate their transmissions through a scheme called self-organising time-division multiple access: each minute on each channel is divided into 2,250 slots, and each station announces which future slots it intends to use so that others avoid them. When traffic is very dense, stations far away are effectively crowded out first, so the system degrades by shrinking its range rather than by failing outright. Smaller vessels may carry Class B transponders, which transmit at lower power, report less frequently and take lower priority in slot allocation. Many small craft carry no AIS at all. AIS has transformed navigation since its introduction. It gives an identity to every echo that matches, allows ships to call each other by name on VHF, and reveals targets hidden behind headlands or other ships, because VHF radio diffracts around obstacles in a way that radar energy does not. Its course and speed come directly from the transmitting ship's own satellite receiver rather than from a minute or more of radar tracking, so a manoeuvre by the other ship shows up within seconds. Yet AIS has a structural weakness: it is self-reported. It tells you what the other ship's equipment says about itself, which is only as good as that equipment, its configuration and the honesty of whoever operates it. The IMO's guidelines on the operational use of AIS, most recently issued as resolution A.1106(29) in 2015, warn explicitly that not all ships carry AIS, that the information may be inaccurate, and that officers should not rely on AIS alone for collision avoidance. Each of these warnings maps directly onto a design requirement for autonomous systems. First, coverage is incomplete. Fishing vessels, yachts, small workboats, warships and many inland craft either need not carry AIS or may switch it off. Fishing fleets in some regions switch off to conceal their fishing grounds. Vessels involved in sanctions evasion, the so-called dark or shadow fleet, switch off or manipulate their transmissions to disguise their movements. An autonomous ship that trusted only AIS would be blind to a meaningful share of what it might hit. Second, the dynamic data inherit the errors of the transmitting ship's sensors. If that ship's satellite positioning is jammed or spoofed, as happened across the Gulf in June 2025 and has been reported repeatedly in the Baltic, Black Sea and eastern Mediterranean, its broadcast position will be wrong, sometimes by many miles. Heading depends on a correctly connected gyrocompass, and AIS units are occasionally installed with the heading missing or offset. Third, the static and voyage data are often wrong because they are entered by hand. Navigational status is frequently left at "under way using engine" while a ship is at anchor, or vice versa. Destinations are abbreviated inconsistently. Draughts are not updated after loading. For a human, this is a nuisance. For a machine that wants to infer, for example, whether a vessel is restricted in her ability to manoeuvre and therefore owed special consideration under the collision regulations, it is a hazard. Fourth, AIS can be falsified deliberately. Because the system was designed for cooperative use among ships, its messages are not authenticated. That is a cybersecurity issue to which Chapter 7 returns, from a defensive angle. For now the essential point is that AIS data are claims, not observations. Radar and AIS together The power of the two systems comes from their independence. Radar measures range and bearing relative to own ship, using nothing but its own transmitter and antenna. Its picture of the surrounding traffic does not depend on satellite positioning at all; if positioning fails, radar still shows where the other ships are relative to you. AIS, by contrast, reports absolute positions that depend on every ship's satellite receiver. When the two agree, an autonomous system can be confident: a radar track and an AIS report that coincide in position, course and speed are very probably the same ship, and the AIS report supplies its identity, dimensions and a fast-updating course while the radar track confirms that it physically exists where it says it does. When they disagree, the disagreement is the most valuable information the system has. An AIS report with no matching radar echo may be a ship hidden behind an obstacle, a ship beyond radar range, a spoofed message or a ship whose own positioning is wrong. A radar echo with no AIS match may be a small craft, a vessel that has switched off, a navigation buoy or a false echo. And if every AIS target in the vicinity appears offset from its radar echo by the same distance in the same direction, the likeliest explanation is not that all the other ships have moved. It is that own ship's positioning is wrong. That inference, which experienced navigators make intuitively when the radar overlay on the electronic chart drifts away from the coastline, is one an autonomous system must be built to make explicitly. It is one of the simplest and most effective defences against positioning interference available, and it costs nothing but careful software. Radar and AIS remain the foundations of maritime perception, but they share a blind spot. Neither reliably shows small, low, non-transmitting objects at short range, which is exactly where the highest-consequence decisions in congested waters are made. Filling that gap is the work of cameras, machine vision and the fusion of all these streams into a single picture, the subject of the next chapter. Chapter 3: Machine Eyes and the Fused Picture The collision regulations require every vessel to maintain a proper lookout "by sight and hearing as well as by all available means appropriate in the prevailing circumstances and conditions". Radar and AIS are among the available means. Sight is the oldest, and on a crewed ship it is still the one that settles arguments. The officer who sees the green sidelight of a vessel crossing from port knows something no radar echo could tell him. A ship without people needs something that does the job of the human eye, and then it needs a way to combine everything it senses into a single coherent picture that carries an honest estimate of its own reliability. This chapter covers both. Cameras and what they add The camera systems now fitted to autonomous and assisted ships typically combine visible-light cameras with thermal cameras, arranged to cover the full horizon, sometimes supplemented by a stabilised pan-tilt-zoom unit for close inspection of a particular target. Visible cameras offer high resolution and colour in daylight. Thermal cameras, working in the long-wave infrared band between roughly eight and fourteen micrometres, detect the heat contrast between objects and the sea, and work in darkness. A person in the water, a small boat's engine, a ship's funnel and exhausts all stand out in thermal imagery when they would be invisible in visible light at night. Cameras fill several specific gaps left by radar and AIS. The first is small and close objects: a fibreglass skiff, a kayak, a rigid inflatable boat or a buoy, which radar may lose in sea clutter at a few hundred metres and which carry no AIS. The second is classification. Radar can say that something is there; a camera can say that it is a sailing yacht, a tug pushing a barge, a fishing vessel trailing gear or a swimmer. That distinction matters because the collision regulations assign different responsibilities to different kinds of vessel, as the next chapter explains. The third is the interpretation of lights and shapes. At night, a vessel's navigation lights reveal its aspect: a masthead light visible over 225 degrees forward, a red sidelight to port and a green one to starboard each covering 112.5 degrees, and a white sternlight covering the remaining 135 degrees aft. Seeing red and white means looking at the vessel's port side. Additional light combinations signal special status: two all-round red lights in a vertical line indicate a vessel not under command; red over white indicates fishing other than trawling; green over white indicates trawling. By day, black shapes serve the same purpose, such as a single ball for a vessel at anchor or a cylinder for a vessel constrained by her draught. A system that can read these signals has access to information that AIS may report wrongly or not at all. The machine learning that does this reading has advanced rapidly. Object detectors based on convolutional neural networks, of the type popularised by families of models such as YOLO, can be trained to find and label vessels in images at video rates on modest onboard computers. Academic researchers have built public maritime datasets to train and test them, including the Singapore Maritime Dataset, published in 2017 with video from shore and from vessels in Singapore waters, the SeaShips dataset of 2018 with images of inland and coastal vessels, and later obstacle-detection benchmarks for small unmanned surface vehicles. Companies such as Orca AI, Sea Machines and Avikus train their own models on very large proprietary collections of footage gathered from fleets of ships fitted with their systems. The marine environment is hostile to optical sensing in ways that road-vehicle experience does not prepare engineers for. The sea surface is a moving mirror: sun glitter can blind a camera across a wide sector, and reflections of a ship's lights on calm water can double them. Spray, rain and salt crust accumulate on lenses, and the remedy, heated housings, wipers or air knives, has to work for months without a person to clean them. Fog and heavy rain reduce visible-light range to a few hundred metres or less, and also degrade thermal imaging. The horizon itself, which the detector relies on as a reference, may be obscured by haze or confused with coastline. The deepest limitation, however, is geometric. A single camera measures bearing very precisely but range only poorly. One common method estimates range from how steeply the camera must look down to see the waterline of an object, using the visible horizon as a reference for level. Ignoring the curvature of the Earth, from a camera 25 metres above the water, the waterline of an object one kilometre away sits about 1.4 degrees below the horizontal; one two kilometres away sits about 0.7 degrees below. Halving the angle doubles the range, which means that at long distance a tiny error in finding the waterline or the horizon, caused by a wave, the ship's pitch, or simply the pixel size of the image, produces a very large error in estimated distance. At ten kilometres the angle has shrunk to about 0.14 degrees, comparable to the correction needed for the Earth's curvature and far smaller than the ship's own pitching motion, so that an error of a few hundredths of a degree shifts the estimate by kilometres. Stereo camera pairs help at short range but need a wide baseline to be useful at sea. The practical conclusion is that cameras excel at short-range detection and at classification, and are best combined with radar for distance and speed. Lidar, the laser scanning that dominates road-vehicle research, has a narrower role at sea. It measures distance precisely, which makes it valuable for docking and for close-quarters manoeuvring alongside quays, as several automatic mooring and berthing systems have shown. But its useful range against small objects is typically a few hundred metres, laser light reflects poorly from water and is scattered by spray and fog, and the motion of the ship complicates its use. For open-water navigation it is a supplement, not a foundation. Hearing, the second element of the lookout rule, has received less attention. Sound signals still matter in restricted visibility, when vessels are required to sound prolonged blasts at intervals, and manoeuvring signals of short blasts announce course changes in sight of another vessel. Research groups and some developers have experimented with microphone arrays to detect and localise these signals, but sound at sea is distorted by wind and the ship's own noise, and this remains a less mature part of the field. A remote operations centre can, however, be given audio from the ship, and the MASS Code's emphasis on equivalent safety means that designers must show how the function of hearing is met, whether by sensing or by other means. The trade-offs among these sensors are summarised in Table 2, which gives typical characteristics as reported by manufacturers and in the technical literature; actual performance varies with installation and conditions. Table 2. Main perception sensors on an autonomous ship and their typical strengths and weaknesses. Sensor Measures Typical useful range Strengths Main weaknesses X-band radar Range, bearing, radar cross-section Out to radar horizon, about 12 to 20 nm for large ships Good resolution, independent of GNSS Rain and sea clutter, small boats hidden S-band radar Range, bearing Similar, better in rain Penetrates weather, less sea clutter Wider beam, poorer small-target detail AIS Identity, position, course, speed, status VHF range, often 20 to 30 nm Identity, fast updates, sees round obstacles Self-reported, absent on many craft, spoofable Visible camera Bearing, appearance, lights and shapes A few km in good visibility Classification, lights, small objects Glare, fog, night, poor range accuracy Thermal camera Heat contrast, bearing Up to a few km for vessels Works at night, finds people in water Lower resolution, weather, cost Lidar Precise 3D range Hundreds of metres Docking and close quarters Spray, fog, poor water returns Fusing the streams With several sensors observing the same world, the system must decide which observations belong to which object, and must combine them into estimates that are better than any single sensor could give. This is the problem of multi-sensor, multi-target tracking, and the tools used to solve it were developed largely for military air defence before being adapted to cars, aircraft and now ships. The workhorse is the Kalman filter, published by Rudolf Kálmán in 1960 and used in navigation ever since, including in the Apollo guidance computer. Its logic can be stated in words. For each tracked object the filter keeps two things: a best estimate of the object's state, such as its position and velocity, and a measure of how uncertain that estimate is, expressed as a covariance. At each step, it first predicts: given the last estimate and a model of how vessels move, where should the object be now? The uncertainty grows during prediction, because the model is imperfect and the object may have changed course. Then it corrects: when a measurement arrives, the filter compares it with the prediction and moves the estimate towards the measurement by an amount that depends on their relative uncertainties. A precise measurement pulls the estimate strongly; a noisy one pulls it only a little. The uncertainty shrinks after each correction. The elegance of the method for fusion is that sensors of very different character can feed the same filter. A radar measurement has a precise range and a less precise bearing. A camera measurement has a precise bearing and almost no useful range at long distance. An AIS report has a position whose accuracy depends on the other ship's receiver. Each is entered with its own uncertainty, and the filter weights it accordingly. When a radar plot and a camera bearing are combined, the radar dominates the estimate of distance while the camera sharpens the bearing. The fused track is better than either alone. Ships are slow but they do turn, and a filter that assumes constant velocity will lag behind a manoeuvre. A standard remedy is the interacting multiple model approach, in which several filters run in parallel, one assuming straight-line motion and others assuming turns at different rates, and the system continuously estimates which model best explains recent measurements. When a target begins to turn, the turning model's likelihood rises and the combined estimate follows the turn more quickly than a single filter could. The hardest part of tracking is not filtering but association: deciding which measurement belongs to which track. In open water with a few widely spaced ships this is trivial. In a crowded anchorage with tugs alongside larger vessels, or a harbour with ferries crossing in both directions, it is not. A simple method assigns each measurement to the nearest predicted track, which works until two tracks cross and swap identities. More sophisticated methods, such as joint probabilistic data association, weigh all plausible assignments by their probability, while multiple hypothesis tracking keeps several competing interpretations alive and resolves them as further evidence arrives. The choice involves real trade-offs in computation and in delay. Associating AIS reports with radar tracks raises a further complication: the AIS position refers to the location of the transmitting ship's positioning antenna, which may be far from the centre of a 400-metre ship, while the radar echo's centre depends on aspect. Good systems use the ship dimensions in the AIS static data to correct for this. Knowing where you are Everything so far concerns other objects. A navigation system also needs to know its own position, heading and motion, and it needs to know how much to trust them. Modern ships derive position overwhelmingly from global navigation satellite systems, primarily GPS but increasingly multi-constellation receivers using Galileo, GLONASS and BeiDou as well. Heading comes from gyrocompasses, and increasingly from satellite compasses using two antennas. Speed through the water comes from a log; speed over ground from the satellite receiver. An autonomous ship cannot afford to treat satellite position as an unquestioned fact. The interference documented in the Gulf, the Baltic and elsewhere means that a well-designed system must monitor the integrity of its own position continuously. The techniques are layered. Receivers can check consistency across constellations and frequencies, and detect the sudden jumps or implausible velocities that spoofing often produces. An inertial measurement unit, which senses accelerations and rotations, can carry a position forward for a limited time without any external input, with an error that grows steadily. The ship's own log and gyro provide dead reckoning. And radar offers an independent check against the coast: matching radar returns from charted coastlines and fixed structures to the electronic navigational chart gives a position that owes nothing to satellites. The comparison between AIS targets and radar echoes described in Chapter 2 adds another. When these sources disagree, the right response is not to pick one but to widen the uncertainty, flag the disagreement to the remote operations centre, and, if necessary, constrain the ship's behaviour until the disagreement is resolved. Electronic navigational charts complete the picture. They provide the static world, coastlines, depths, buoys, fixed structures, traffic separation schemes and restricted areas, against which moving targets are interpreted. Charts are produced to the International Hydrographic Organization's S-57 standard today, with the richer S-100 family, which supports more dynamic data such as under-keel clearance and water levels, being phased in over the coming years. A fused picture is not only a list of tracks; it is a list of tracks placed in the context of a chart. The outcome of all this work is a world model: a set of tracked objects, each with an estimated state, a classification, a source history and, critically, an uncertainty. That uncertainty is the most important output the perception system produces, and the place where weak designs most often go wrong. A system that reports every target with false confidence will plan manoeuvres that are too fine. A system that is honest about what it does not know can plan with margins, slow down when visibility deteriorates, and ask a human for help when its picture becomes unreliable. The deepest unsolved problem in maritime perception is the long tail of rare objects and conditions: a half-submerged shipping container lost overboard in a storm, a whale at the surface, a small boat drifting without lights in heavy rain at night. These are rare individually and, taken together, common enough to matter. No dataset contains enough of them, and no sensor sees them reliably. Developers manage this with redundancy across sensor types, with conservative speed and margin policies in conditions where detection is known to be weak, and with operational limits that keep autonomous operation within the envelope where the evidence supports it. That is the same principle seen from the perception side: a system that knows the limits of its own knowledge, and behaves accordingly. Perception produces a picture. The next question is what to do with it, and at sea that question is governed by a set of rules written half a century ago for human beings. Chapter 4: Rules of the Road for Machines Collision avoidance at sea is governed by the Convention on the International Regulations for Preventing Collisions at Sea, adopted in 1972 and in force since July 1977, known universally as the COLREGs. They are short, about forty rules and four annexes, and every deck officer in the world learns them nearly by heart. They are also, from the point of view of a software engineer, an unusual kind of specification. They do not say how close is too close. They do not define "ample time". They tell the mariner to act in accordance with "the ordinary practice of seamen" and, in the last resort, to depart from the rules themselves if that is what avoiding immediate danger requires. The COLREGs work because they are applied by people who have been trained in a shared culture of seamanship and who can read one another's intentions. An autonomous ship must operate inside that culture without having grown up in it. This chapter explains what that demands. How the rules are built The steering and sailing rules, Part B of the COLREGs, are organised in three sections. The first applies in any condition of visibility. Rule 5 requires a proper lookout. Rule 6 requires every vessel to proceed at a safe speed, defined not by a number but by a list of factors: visibility, traffic density, manoeuvrability, background lights at night, wind, sea and current, the draught relative to the available depth, and the characteristics and limitations of the radar. Rule 7 deals with determining whether risk of collision exists, and contains the most famous practical test in navigation: if the compass bearing of an approaching vessel does not appreciably change, risk of collision should be deemed to exist. It also warns against "assumptions made on the basis of scanty information, especially scanty radar information". Rule 8 governs the action taken to avoid collision. Any action must be taken in ample time, must be large enough to be readily apparent to another vessel observing visually or by radar, and a succession of small alterations should be avoided. Rules 9 and 10 cover narrow channels and traffic separation schemes, the lane systems that divide inbound and outbound traffic in busy waters such as the Dover Strait or the approaches to Singapore. The second section applies to vessels in sight of one another, and contains the rules that define who must act. When two power-driven vessels meet head-on or nearly so, Rule 14 requires each to alter course to starboard, so that each passes on the other's port side. When two power-driven vessels cross so as to involve risk of collision, Rule 15 requires the vessel that has the other on her own starboard side to keep out of the way and, if circumstances allow, to avoid crossing ahead of her. Under Rule 13, any vessel overtaking another, defined as coming up from a direction more than 22.5 degrees abaft the other's beam, must keep out of the way, whatever else the rules say. Rule 16 tells the give-way vessel to take early and substantial action. Rule 17 tells the stand-on vessel to maintain course and speed, but permits her to act once it becomes apparent that the give-way vessel is not taking appropriate action, and requires her to act when collision can no longer be avoided by the give-way vessel alone. Rule 18 establishes a hierarchy between categories of vessel: a power-driven vessel must keep out of the way of a vessel not under command, a vessel restricted in her ability to manoeuvre, a vessel engaged in fishing and a sailing vessel, and there are further gradations below. The third section, Rule 19, applies in restricted visibility, when vessels are not in sight of one another. Here there is no give-way or stand-on vessel. Every vessel that detects another by radar alone must determine whether a close-quarters situation is developing and, if so, take avoiding action in ample time. It must avoid altering course to port for a vessel forward of the beam, other than one being overtaken, and avoid altering course towards a vessel abeam or abaft the beam. Over all of this sits Rule 2. Nothing in the rules exonerates anyone from the consequences of neglecting any precaution required by the ordinary practice of seamen or by the special circumstances of the case, and in construing the rules due regard must be had to all dangers of navigation and collision and to any special circumstances, including the limitations of the vessels involved, which may make a departure from the rules necessary to avoid immediate danger. Rule 2 is the escape clause that makes the system work for humans. It is also the single largest problem for machines. Classifying an encounter Before an autonomous system can decide what the rules require, it must classify each encounter, and the classification depends on geometry. Mariners do this by looking at relative bearing and aspect: where the other ship is relative to own ship's heading, and which of the other ship's lights or sides they can see. The system must do the same with numbers. Table 3 shows the classification that many research and commercial systems use, with the sectors expressed in terms of the other vessel's relative bearing and heading. The boundaries are conventions drawn from the rules and from navigational practice, not fixed definitions; the head-on sector, in particular, is set differently by different developers. Table 3. Two-vessel encounter types between power-driven vessels in sight of one another, as typically encoded. Encounter Typical geometric test Own ship's role Required or expected action Head-on (Rule 14) Headings nearly reciprocal, target nearly ahead Both give way Alter to starboard, pass port to port Crossing, target to starboard (Rule 15) Target forward of own starboard beam, crossing Give way Keep clear, avoid crossing ahead, usually alter to starboard Crossing, target to port (Rule 15) Target forward of own port beam, crossing Stand on Hold course and speed, act if give-way vessel fails (Rule 17) Overtaking (Rule 13) Own ship approaching from more than 22.5° abaft target's beam Give way Keep clear of overtaken vessel until finally past Being overtaken (Rule 13) Target approaching from more than 22.5° abaft own beam Stand on Hold course and speed, act if necessary (Rule 17) Take a concrete case. Own ship, a container ship, is heading 000 degrees at 15 knots. A bulk carrier is detected bearing 050 degrees, five miles off, heading 290 degrees at 12 knots. Its bearing is steady over several minutes, so risk of collision exists under Rule 7. It is forward of own ship's starboard beam and the two are crossing, so this is a Rule 15 crossing with own ship as the give-way vessel. Rule 16 requires early and substantial action, and Rule 15 asks own ship to avoid crossing ahead. The orthodox response is a bold alteration to starboard, perhaps 40 to 60 degrees, turning own ship's bow towards the bulk carrier's stern so that it passes clearly astern, held until the bulk carrier is past and clear, followed by a return to the planned track. An alternative is to reduce speed substantially, which also lets the other vessel cross ahead, though a speed change is much harder for the other vessel to detect than a course change and large ships take a long time to slow down. Now reverse the geometry: the bulk carrier bears 310 degrees and is heading 070. It is on own ship's port bow, and own ship is the stand-on vessel. The rules now require own ship to hold course and speed. That is the instruction that most troubles both human navigators and algorithm designers, because holding course while another ship approaches on a collision course feels, and sometimes is, dangerous. Rule 17 offers a graduated release. Once it becomes apparent that the give-way vessel is not taking appropriate action, own ship may take action of her own, and if she does so she should, if circumstances allow, not alter to port for a vessel on her own port side. When the situation has deteriorated so far that collision cannot be avoided by the give-way vessel alone, own ship must act. The rules do not say when "apparent" becomes true, or how close is "cannot be avoided". Every system must decide. The vague words Most of the COLREGs' imprecision is deliberate. "Ample time", "readily apparent", "close-quarters situation", "safe speed" and "appreciably change" are left undefined because the right answer depends on the ships, the waters and the weather. A very large crude carrier at sea speed may need several miles to stop; a harbour tug can turn on its own length. A two-mile CPA that is comfortable for two large ships in the Pacific would be impossible in the Singapore Strait, where ships routinely pass within a few cables of one another. Autonomous systems handle this by making the implicit explicit, in parameters that can be inspected, tested and argued about. One family of methods uses the concept of a ship domain: the area around a ship that its navigator wants to keep clear of other vessels. The idea comes from empirical studies of real traffic. Yoshiaki Fujii and colleagues in Japan, observing traffic in Japanese waters in the early 1970s, found that the space mariners kept around their ships was roughly elliptical, longer ahead and astern than on the beam, and scaled with ship length. Elizabeth Goodwin, studying traffic in the southern North Sea a few years later, found the domain was asymmetric, larger to starboard than to port, which is what the rules would lead one to expect, since ships give way by passing astern of vessels on their starboard side. Modern systems use domains of this kind, sometimes adjusted for speed and manoeuvrability, as the threshold for "close quarters". A second approach sets minimum acceptable CPA and TCPA values by context: a larger CPA in open sea, a smaller one in coastal waters, smaller still in a port approach, and correspondingly different times at which action must begin. A ship's operator can then specify these in the same way that a master's standing orders on a crewed ship specify, for example, that the officer of the watch must call the master if any vessel is expected to pass within two miles. A third approach expresses the rules in formal logic, so that compliance can be checked mathematically against simulated or recorded trajectories. Researchers have used temporal logic and so-called rulebooks, ordered sets of rules with priorities, to capture the structure of COLREGs, including the principle that safety rules dominate procedural ones. These formalisms are valuable for testing and for making assumptions visible, though they do not by themselves settle what the numbers should be. Rule 8's requirement that action be "readily apparent" deserves particular attention, because it is about communication rather than geometry. A manoeuvre is a message. An alteration of five degrees may be enough mathematically to raise a CPA from zero to half a mile, but it is invisible to an officer watching from another ship and barely detectable by radar plotting for several minutes. That officer may then conclude that the give-way vessel is doing nothing and take action of her own, possibly in a direction that cancels out the first ship's small turn. A classic pattern of collision is two ships each making small, uncoordinated alterations that bring them together. For this reason, mariners are taught to make alterations large, commonly 30 degrees or more and sometimes much more, and autonomous systems are designed to do the same even when a smaller change would suffice on paper. Being predictable to humans is a safety requirement, not a courtesy. Where the rules run out The rules are written for encounters between two vessels. Many real situations involve several. Suppose own ship is give-way to a vessel on her starboard bow, but a third vessel is overtaking on her starboard quarter, and a fishing vessel lies ahead. An alteration to starboard to satisfy the first obligation may create a close-quarters situation with the overtaking ship; slowing may confuse the vessel astern. The COLREGs offer no algorithm for resolving such conflicts. Human officers resolve them by judgement, often by finding a manoeuvre that is clearly safe with respect to all vessels even if it does not perfectly honour each rule's ordinary expectation, and they rely on Rule 2. Machine systems must formalise that judgement, usually by searching across many possible manoeuvres and scoring each against the full set of vessels, which is the subject of the next chapter. A second problem concerns who knows what. The rules assume that both vessels can see each other and interpret each other's lights and behaviour. A vessel that is not following the rules, perhaps because its officer has not seen own ship, or has misjudged the situation, or is fishing and manoeuvring unpredictably, creates exactly the circumstance Rule 17 anticipates. An autonomous system must therefore model not only what the rules require of others but what others are likely to do, and keep monitoring whether they are doing it. In practice this means the system continuously compares a target's actual behaviour against the behaviour expected of it, and treats an unexpected lack of action, or an unexpected manoeuvre, as a trigger to re-plan. A third problem is the phrase "in sight of one another". For a human, being in sight means seeing the other vessel visually. For a ship whose eyes are cameras, does a detection by a thermal camera at night count? Does seeing a vessel on a remote operator's video screen count? The distinction matters because Rule 19, for restricted visibility, imposes quite different behaviour from the rules for vessels in sight, and an autonomous ship that switches between the two regimes at the wrong moment may act in a way the other ship does not expect. Developers generally resolve this by tying the regime to measured visibility and to whether the other vessel could plausibly see own ship, rather than to own ship's sensor capability alone. Finally, there is the question of whether the rules should change. The COLREGs text contains nothing about autonomous or remotely controlled ships, and assumes a watchkeeper with eyes and ears. The IMO's regulatory scoping exercise identified the COLREGs as one of the instruments where questions arise, particularly over lookout and over the meaning of "master" and "crew". The approach taken through the MASS Code has been to require that autonomous ships comply with the existing rules and achieve equivalent safety, rather than to rewrite the rules for a mixed fleet. That choice puts the burden on designers. An autonomous ship must behave in a way that a competent human seafarer on another vessel would recognise as seamanlike, because that seafarer will be applying the same rules and expecting the same conduct. On a crewed ship, when the geometry is ambiguous, officers often call each other on VHF radio to agree how they will pass. The practice is not without risk, because misidentified ships and misheard agreements have themselves contributed to collisions, but it is part of ordinary seamanship. An autonomous ship's equivalent is still developing. Remote operators can speak on VHF through the ship's radio. Some research systems generate and interpret simple radio messages automatically. More promisingly, several initiatives have explored exchanging planned routes between ships electronically, so that each can see the other's intended track for the next few minutes or hours. The European Sea Traffic Management projects tested route exchange between ships and shore in the late 2010s, and the International Hydrographic Organization's S-100 framework includes a product specification for route plans that would allow a standard format. If ships routinely shared intentions in this way, many of the ambiguities that the COLREGs leave to inference would disappear. Until they do, an autonomous ship must infer intentions from behaviour, as human mariners always have, and make its own intentions as clear as possible through the size and timing of what it does. That is the essential shift from human to machine. A human seafarer carries the rules as internalised judgement. A machine must carry them as explicit parameters, priorities and tests, and every one of those choices can be examined, simulated and challenged. That transparency is a weakness, because it exposes judgements that humans make silently. It is also the strongest argument for autonomy, because judgements that are written down can be improved. Chapter 5: The Algorithms of Avoidance With a fused picture of the surroundings and a machine-readable interpretation of the rules, an autonomous ship still has to decide what to do: which course, what speed, when to start turning, when to return to the planned track. This chapter looks inside the decision-making layer. It describes the main families of collision-avoidance algorithm, explains how each handles the slow and heavy dynamics of a large ship, and then turns to the hardest operating environment of all, the congested port approach, where many of the assumptions that make open-water avoidance tractable break down. A ship is not a car Collision avoidance at sea is shaped first by physics. A large merchant ship responds to its rudder and engine slowly, and it carries an enormous amount of momentum. The IMO's standards for ship manoeuvrability, adopted in 2002 as resolution MSC.137(76), set limits that give a sense of the scale. In a full-rudder turning test, a ship's advance, the distance it travels forward before its heading has changed by 90 degrees, should not exceed four and a half ship lengths, and its tactical diameter, the sideways distance to complete a half-circle, should not exceed five ship lengths. In a full astern stopping test from full ahead, the track reach should not exceed fifteen ship lengths, though administrations may accept more for very large ships. For a 300-metre ship, that means more than a kilometre to turn through a right angle and potentially several kilometres to stop. Two consequences follow. First, decisions must be made early, typically many minutes before the closest point of approach in open water, because the manoeuvre itself takes minutes to have effect. Second, the algorithm must reason about the ship's future motion using a model of how it responds, not simply command a new course and assume it will be achieved at once. Ship dynamicists have long used compact models for this purpose. The first-order model proposed by the Japanese engineer Kensaku Nomoto in the 1950s describes how a ship's rate of turn responds to rudder angle with two parameters, a gain and a time constant, and remains a common basis for autopilot and track-keeping design. More detailed models, including those standardised by the Japanese Manoeuvring Modeling Group, capture hull, propeller and rudder forces separately. Collision-avoidance algorithms typically use a simplified model to predict trajectories quickly, while lower-level controllers, autopilots or track pilots, take care of executing the chosen manoeuvre precisely. This produces a layered architecture found in almost every system. At the top, a voyage plan sets the route from port to port, the subject of the next chapter. Below that, a collision-avoidance layer modifies the plan locally to handle traffic and hazards over a horizon of minutes. Below that, a guidance and control layer converts the modified plan into rudder and propulsion commands every second or faster. Some designs add a fast reactive layer for emergencies, which can override the planner if a target appears suddenly at close range. The separation matters for safety as well as engineering. Each layer can be tested on its own, and each has a defined fallback if the layer above fails. Four ways to choose a manoeuvre The research literature on ship collision avoidance is large, and reviews have catalogued hundreds of distinct methods. Most fall into four families, summarised in Table 4. They are not mutually exclusive, and commercial systems frequently combine them. Table 4. Main families of collision-avoidance algorithm used for autonomous ships. Family Core idea Strengths Limitations Velocity obstacles Exclude velocities that lead to collision; choose best remaining Fast, intuitive, handles many targets Assumes targets keep course; dynamics simplified Graph and sampling search (A, RRT) Search a space of paths for lowest-cost route Handles static hazards and charts well Slower; moving targets need time dimension Model predictive control Simulate candidate manoeuvres over a horizon, score them, repeat Explicit dynamics, rules and risk in one cost Tuning-heavy; cost weights encode judgement Learning-based (e.g. reinforcement learning) Learn a policy from simulated or recorded experience Can capture complex behaviour Hard to verify, explain and certify Velocity obstacles take a geometric view. Imagine the set of all velocities, every combination of course and speed, that own ship could adopt right now. For each other vessel, some of those velocities would, if both held them, bring the two ships within an unsafe distance. That forbidden set, drawn in the space of velocities, forms a cone pointing towards the other vessel, and it is called the velocity obstacle. The idea was published for robotics by Paolo Fiorini and Zvi Shiller in 1998. Applied at sea, the algorithm computes a velocity obstacle for every tracked vessel, removes all those velocities from consideration, and then chooses, from what remains, the velocity closest to the one the voyage plan wants, subject to constraints such as the ship's maximum turn rate and speed range. To comply with the COLREGs, the method adds further exclusions. A give-way vessel in a crossing situation is barred from velocities that would take it across the other ship's bow or turn it to port. Researchers at NASA's Jet Propulsion Laboratory, working with unmanned surface vessels for the US Navy, published a well-known version of this COLREG-constrained approach in 2014 and tested it on water. Velocity obstacles are fast and easy to explain, which matters for certification, but in their simple form they assume that every target will keep its present course and speed, and they do not represent the time it takes a large ship to reach a new velocity. Graph and sampling search methods take the view of route planning. The sea area around the ship is represented as a grid or a graph, with cost assigned to cells near hazards, shallow water, other ships' predicted positions and so on, and an algorithm such as A*, which has been used in path planning since its publication in 1968, finds the lowest-cost path. Sampling methods such as rapidly-exploring random trees build up candidate paths by random exploration and are well suited to cluttered environments. These methods are natural for avoiding static hazards drawn from the chart, but to handle moving ships they must search in time as well as space, which multiplies the computation. Model predictive control has become the dominant approach in serious research and in many industrial systems, because it brings dynamics, rules and risk into a single framework. The idea is to simulate a set of candidate manoeuvres forward over a horizon of several minutes, predict where own ship and every other vessel will be, score each candidate by a cost function, apply the first part of the best one, and then repeat the whole process a few seconds later with fresh sensor data. One influential version, published by Tor Arne Johansen, Tristan Perez and Andrea Cristofaro of the Norwegian University of Science and Technology in 2016, keeps the candidate set small and explicit: a handful of course offsets, from hard to port through to hard to starboard, combined with a few speed options such as keep speed, slow down and stop. Each combination is simulated with a ship model, and its cost combines the predicted risk of collision with every target, penalties for violating the COLREGs, penalties for grounding, and a penalty for departing from the planned route or changing the manoeuvre from the previous cycle. A worked example shows how this produces seamanlike behaviour. Suppose own ship is the give-way vessel in a crossing with a target on the starboard bow, CPA currently 0.2 miles in 14 minutes. The planner evaluates, say, thirteen course offsets and three speed options. Holding course scores very badly on collision risk. A 10-degree alteration to starboard improves the CPA but leaves it under the threshold and is penalised for not being readily apparent. A 20-degree alteration to port might clear the target mathematically but is heavily penalised for violating Rule 15's expectation and for crossing ahead. A 45-degree alteration to starboard gives a comfortable CPA astern of the target, complies with the rules and has a moderate cost for leaving the route. It wins. A few seconds later the process runs again, and because changes in the chosen manoeuvre are penalised, the ship holds its 45-degree alteration steadily rather than wavering, until the target has passed and the cheapest option becomes returning to the route. That behaviour, decisive, early, large and stable, is what Rule 8 asks for, and it emerges from the cost function rather than being programmed step by step. The weakness of this approach lies in the cost function itself. The relative weights given to collision risk, rule compliance and route deviation encode judgements that a human navigator makes intuitively and differently in different circumstances. Tuning them is laborious, and a cost function that produces excellent behaviour in a thousand test scenarios can produce odd behaviour in the thousand-and-first. Much current work focuses on making those weights context-dependent, on representing the uncertainty of targets' future behaviour rather than assuming they hold course, and on ensuring that the planner always retains an escape option. Learning-based methods, particularly deep reinforcement learning, train a policy by letting a simulated ship experience millions of encounters and rewarding outcomes such as avoiding collision and following the rules. Published results are often impressive in simulation. The difficulty is assurance: a learned policy is hard to inspect, hard to explain to a regulator or a court, and may behave unpredictably in situations unlike its training. For now, learning is mostly used inside perception, for detecting and classifying objects, and as a source of predictions about other vessels' behaviour, rather than as the final authority on manoeuvres. The congested port Open-water collision avoidance against one or two large ships is, by the standards of this field, close to a solved problem. Port approaches are not. Consider a large container ship approaching a busy European or Asian port. In the space of an hour, it may need to leave a traffic separation scheme, cross the flow of through-traffic, pick up a pilot, enter a dredged channel with shallow water either side, pass ferries crossing on fixed schedules, encounter fishing boats and leisure craft that may carry no AIS and may not keep to the rules, slow to harbour speed while maintaining steerage way, and meet tugs that will come alongside and make fast. Every one of these elements changes the problem. The rules themselves change. Rule 9 governs narrow channels: a vessel proceeding along one must keep as near to the outer limit on its starboard side as is safe, and vessels of less than 20 metres in length and sailing vessels must not impede the passage of a vessel that can safely navigate only within the channel. Rule 10 governs traffic separation schemes: ships using a scheme must follow the lane direction, cross it as nearly as practicable at right angles, and join or leave at its ends where possible. Local regulations add speed limits, reporting points and sometimes one-way traffic. An autonomous system must load and apply these local rules, which are not always available in standardised machine-readable form. The physics changes too. In shallow, confined water, a ship sinks lower and trims as it moves, an effect called squat that increases roughly with the square of speed. Near a channel bank, pressure differences push the stern towards the bank and the bow away. Two ships passing close in a channel draw towards one another. Experienced pilots manage these effects by feel and by local knowledge. An autonomous system must model them, which requires better hydrodynamic models than open-water avoidance needs, and must rely on accurate charts of depth, which in some ports change with siltation faster than surveys are repeated. The traffic itself is the biggest change. Small, fast, unpredictable craft are the hardest targets for perception, as Chapter 3 explained, and they are also the hardest for planning, because their behaviour does not follow the patterns that the planner's predictions assume. A sailing dinghy may tack across the channel; a fishing boat may stop suddenly to haul gear; a pilot launch will deliberately approach to within a few metres. If the planner treats every one of these as a threat requiring a large manoeuvre, a large ship in a narrow channel will quickly find that no safe manoeuvre exists, and in the worst case could freeze or stop in a way that endangers itself and others. The practical answer, in nearly every serious project, is that the port approach is where autonomy is most constrained and human involvement greatest. The MEGURI2040 project's demonstrations in 2022 included autonomous operation of the container feeder Suzaku in Tokyo Bay, one of the busiest waterways in the world, and its certified services in 2026 operate within designated waters on domestic routes. Projects typically combine several measures: lower speed in congested waters, which buys time for perception and decision; heightened remote supervision, often with a remote operator in direct contact with the vessel traffic service, the shore-based authority that monitors and advises traffic in many ports; continued use of pilots, either aboard or, in some trial arrangements, remotely; and use of tugs for berthing. Automatic docking systems, which use precise positioning and short-range sensors such as lidar to bring a vessel alongside, have been demonstrated on ferries, including the Falco in 2018 and several Norwegian ferries since, but large cargo ships still rely on tugs and line handlers ashore. Testing a decision-maker How does anyone know that a collision-avoidance system is good enough? Sea trials alone cannot answer the question, because the dangerous encounters that matter most are rare. A ship could sail for years without meeting the particular three-vessel geometry that defeats its algorithm. The field has therefore converged on scenario-based testing in simulation, backed by limited real-world trials. The approach works in stages. Engineers first define a library of encounter scenarios: all the standard two-ship geometries at varying speeds and ranges, multi-ship situations, encounters with vessels that break the rules, encounters in restricted visibility, and encounters complicated by shallow water or sensor faults. They then generate many variations, sometimes many thousands or millions, by systematically or randomly varying the parameters, and run the system against each in simulation. Automated evaluators score each outcome on safety, measured by minimum separation, and on rule compliance, measured against the formal interpretations described in the previous chapter. Search algorithms can be pointed at the system deliberately to find the scenarios where it performs worst, which focuses engineering effort where it is most needed. Classification societies including DNV have published guidance on this kind of simulation-based assurance for autonomous and remotely operated ships. Real-world testing then validates the simulation. Norway designated the Trondheimsfjord as the world's first test area for autonomous ships in 2016, and Finland opened the Jaakonmeri test area in the Gulf of Bothnia, off its west coast, in 2017. These areas allow trials in real conditions under controlled rules. Recorded data from ships fitted with perception and advisory systems, operating in ordinary commercial service with crews still in command, adds a further source of evidence: the system can be run silently in parallel with the human officers, and its decisions compared with theirs. No testing campaign can prove that a collision-avoidance system will never fail. The aim is narrower and more achievable: to show, with evidence that a regulator can examine, that the system performs at least as well as competent human navigators across the range of situations it is permitted to face, and that when it meets a situation it cannot handle, it recognises the fact and falls back to a safe state. That second requirement, the subject of Chapter 8, is where the design of an autonomous ship either earns trust or loses it. Chapter 6: Routing for Fuel and Carbon Collision avoidance operates over minutes. Voyage optimisation operates over days and weeks, and it is where most of the money is. Fuel is typically the largest single operating cost of a deep-sea merchant ship, and international shipping as a whole accounted for just under three per cent of global anthropogenic greenhouse gas emissions in 2018, according to the IMO's Fourth Greenhouse Gas Study. The technologies that let a ship plan its own route and speed are not unique to autonomy, and many crewed ships already use them through commercial weather-routing services. But autonomy changes their character. A ship that is continuously sensing its own performance, continuously receiving forecasts and continuously able to adjust its own plan can optimise in a way that a crew, working watches and relying on a routing service's advice a few times a day, cannot. This chapter explains how the optimisation works, what limits it, and what changes when no one is aboard. Speed, weather and safe routes The most powerful lever on a ship's fuel consumption is speed, and the reason lies in a simple relationship. The power needed to push a displacement hull through calm water rises steeply with speed, roughly with its cube over the normal operating range, though the exact exponent varies with hull form and speed. Fuel burned per hour therefore rises roughly with the cube of speed, while the time needed to cover a given distance falls in proportion to speed. Multiply the two and the fuel burned over a voyage of fixed length rises roughly with the square of speed. A worked example makes the effect concrete. Suppose a container ship sailing at 20 knots burns 100 tonnes of fuel a day, and its voyage is 5,000 nautical miles. At 20 knots the voyage takes 250 hours, about 10.4 days, and burns around 1,040 tonnes. Slow the ship to 16 knots and, under the cube rule, daily consumption falls to about 51 tonnes, because 0.8 cubed is about 0.51. The voyage now takes 312.5 hours, about 13 days, and burns around 665 tonnes. A 20 per cent cut in speed has cut voyage fuel by about 36 per cent, at the cost of two and a half extra days at sea. The real numbers depend on the ship, its engine and its loading, and engines run less efficiently far below their design load, but the shape of the trade-off is robust. It is why the industry adopted slow steaming widely after the collapse in freight rates and the high fuel prices of 2008, and why speed remains the first variable any voyage optimiser considers. Weather and sea state add a second layer. Waves increase a ship's resistance, sometimes greatly, especially when coming from ahead. Wind adds resistance on the hull and on deck cargo; a laden container ship presents a large flat face to a headwind. Ocean currents add or subtract directly from speed over ground: the Gulf Stream and the Kuroshio can run at several knots in their cores, and a route that rides a favourable current, or avoids a contrary one, can save hours. Weather routing is the art of finding the path and speed profile through this moving field of wind, waves and current that best meets the voyage's objectives. The classical method for weather routing was published by R. W. James in 1957 and is called the isochrone method. Starting from the departure point, the ship's possible positions after, say, six hours are calculated for a fan of different headings, taking account of the forecast weather along each. The outer boundary of these positions is the isochrone, the line of points reachable in exactly six hours. From each point on that line the process is repeated, and the outermost envelope of the new set becomes the twelve-hour isochrone, and so on until the destination is reached. Tracing back through the isochrones gives the fastest route. Refinements developed later, including a modified isochrone method published by Hideki Hagiwara in 1989, made the approach more robust and better suited to computers. Modern optimisers more often use dynamic programming or related graph-search methods, in which the ocean is represented as a grid of possible positions at successive times, and the optimiser searches over both course and speed to minimise a chosen objective, typically fuel, subject to constraints such as arriving within a specified window. Making speed a decision variable, rather than fixing it, matters because the cheapest route is frequently not the fastest one, and because a ship may sensibly slow down through bad weather and speed up afterwards. The computation is substantial but well within the capacity of onboard or shore-based computers, and it can be repeated whenever a new forecast arrives, typically every six or twelve hours. The objective function is where commercial judgement enters. A charterer may care about arrival time, an owner about fuel, a regulator about emissions, and the cargo about motion. A refrigerated container ship, a car carrier and a crude tanker trade these differently. A good optimiser exposes the trade-off, showing for instance how much fuel each additional hour of arrival delay saves, so that the people who carry the commercial risk can decide. Weather routing began as a safety service rather than an economy measure, and the safety dimension has not gone away. Heavy weather can damage ships and cargo, and some forms of dangerous motion are specific to particular combinations of ship, wave and heading. Parametric rolling, in which waves of a certain length passing along a ship periodically change its stability and drive large, rapidly growing roll angles, is a particular hazard for large container ships with flared bows and flat sterns. In November 2020 the container ship ONE Apus lost more than 1,800 containers overboard in a Pacific storm, one of a series of large stack losses in that period that prompted renewed attention to heavy-weather operation. The IMO's guidance to masters on avoiding dangerous situations in adverse weather, issued in 2007 as MSC.1/Circ.1228, describes the wave conditions in which surf-riding, broaching and parametric rolling become likely, and how course and speed changes can avoid them. A voyage optimiser on an autonomous ship must treat such guidance as a hard constraint, not a cost to be traded against fuel. Forecast uncertainty complicates this. Weather forecasts lose skill with time, and a route optimised against a single forecast five days ahead may lead the ship into weather the forecast did not predict. Modern meteorological centres such as the European Centre for Medium-Range Weather Forecasts issue ensemble forecasts, running their models many times with slightly different starting conditions to give a range of possible outcomes. Routing against an ensemble rather than a single forecast allows the optimiser to prefer routes that are good across most plausible outcomes, and to avoid routes that are excellent if the forecast is right but dangerous if it is wrong. That is the same principle, carrying honest uncertainty through the system, that Chapter 3 applied to perception. Performance and punctuality A route optimiser is only as good as its model of the ship, and that model drifts. The single largest source of drift is the condition of the hull and propeller. Marine growth, from slime to barnacles, accumulates on the underwater hull whenever a ship sits in warm water, and it increases frictional resistance. A heavily fouled hull may need substantially more power to make the same speed than a clean one, and the effect builds up gradually between dry-dockings and hull cleanings. A damaged or fouled propeller loses efficiency in the same way. If the optimiser uses the ship's sea-trial performance curves without correcting for fouling, its fuel predictions will be wrong and its recommended speeds suboptimal. Measuring this is harder than it sounds, because a ship's fuel consumption on any given day depends on speed, draught, trim, wind, waves, current and water temperature, all at once. The international standard ISO 19030, published in 2016, sets out methods for measuring changes in hull and propeller performance from onboard data by correcting for these factors and comparing like with like over time. An autonomous ship, which is instrumented far more densely than a conventional one and which already logs its environment continuously for navigation, is well placed to apply such methods automatically. The same data allow it to estimate when hull cleaning or propeller polishing would pay for itself, and to feed that into scheduling decisions taken ashore. Trim is a second lever that is easy to overlook. For a given displacement, the angle at which a ship floats, slightly down by the head or by the stern, changes its resistance, sometimes by a few per cent, and the best trim varies with speed and draught. Crewed ships often adjust trim through the distribution of ballast and cargo using guidance from tank tests or computational studies. An autonomous ship can treat trim as one more variable in its optimisation, and, where its ballast system permits, adjust it during the voyage as speed and draught change with fuel burn. These measures are less dramatic than an optimised ocean route, but they compound. A voyage optimiser that knows the ship's true current performance, rather than its performance when new, will choose better speeds, better routes and better maintenance schedules, and the savings appear on every voyage rather than only when the weather is interesting. One of the largest inefficiencies in shipping has nothing to do with the ocean crossing itself. Ships commonly sail at full contracted speed to arrive at a port, only to wait at anchor for days because the berth is not ready. The custom has deep contractual roots: under many charter party terms a ship must proceed with "utmost despatch", and a vessel's laytime, the period the charterer has to load or discharge, often starts only once it has arrived and tendered notice of readiness. The result, known in the industry as "sail fast, then wait", burns fuel to gain nothing. Just-in-time arrival tries to fix this by sharing information between ports, terminals and ships so that each vessel adjusts its speed to arrive when its berth is actually available. The IMO, through its Global Industry Alliance to Support Low Carbon Shipping, published a guide to just-in-time arrival in 2020, alongside standard contract clauses that remove the incentive to race. The technical part is straightforward: an optimiser that knows the required arrival time can compute the lowest-fuel speed profile to meet it. The difficulty is organisational. Ports must publish reliable berth availability, and contracts must reward rather than penalise slower arrival. An autonomous ship, integrated digitally with a remote operations centre that is itself integrated with port systems, is a natural participant in such schemes, and several port authorities are developing digital port-call data exchanges that make it possible. The regulatory pressure Fuel saving has always mattered for cost. Since 2023 it has also mattered for compliance. Table 5 summarises the principal measures now in force or under negotiation that bear on a ship's fuel use and emissions, as published by the IMO and the European Union. Table 5. Principal regulations driving fuel and emissions optimisation in international shipping. Measure Body Status Effect on operation Energy Efficiency Existing Ship Index (EEXI) IMO In force from 1 January 2023 One-off technical efficiency standard, often met by engine power limits Carbon Intensity Indicator (CII) IMO In force from 1 January 2023 Annual operational rating A to E; tightening each year EU Emissions Trading System, maritime European Union Phased in from 2024 Allowances surrendered for emissions on voyages to, from and within the EU FuelEU Maritime European Union In force from 1 January 2025 Limits on greenhouse gas intensity of energy used on board IMO Net-Zero Framework IMO Approved April 2025; adoption adjourned October 2025 Proposed global fuel standard and pricing mechanism The Carbon Intensity Indicator matters most for day-to-day operation, because it rates each ship every year on the carbon dioxide it emits per unit of transport work, and the required level tightens year by year. A ship rated D for three consecutive years, or E in a single year, must submit a corrective plan. Speed, routing, waiting time and hull condition all feed directly into the rating. The European Union's measures add a direct cost per tonne of emissions on voyages touching EU ports. The IMO's revised greenhouse gas strategy, adopted in 2023, set a goal of net-zero emissions from international shipping by or around 2050, with indicative checkpoints for 2030 and 2040. The proposed Net-Zero Framework, intended to give those goals legal force through a global fuel standard and an emissions pricing mechanism, was approved by the Marine Environment Protection Committee in April 2025, but at an extraordinary session in October 2025 member states voted to adjourn its adoption for a year. Whatever its eventual form, the direction is clear: every tonne of fuel saved will be worth more than its purchase price. What autonomy changes Several things change when the ship is autonomous, and some of them are subtle. The first is continuity. A crewed ship following a weather-routing service typically receives recommended routes a few times a day and adjusts its course and speed when the master accepts them. An autonomous system can re-optimise continuously as new forecasts, measured weather, measured sea state from its own motion sensors and its own observed fuel consumption arrive. It can also maintain its own performance model, learning how its fuel consumption varies with speed, draught, trim, weather and hull fouling, and use that learned model rather than a generic one. Avikus reported that the Prism Courage improved its fuel efficiency by around seven per cent during its 2022 voyage through optimised routing and speed; figures in that range are commonly reported by vendors of both autonomous and advisory systems, though the baseline against which they are measured varies. The second is the removal of crew constraints. On a crewed ship, routing and speed decisions are shaped in part by the comfort and safety of the people on board, and by human working patterns: arrival times that suit a change of watch, avoidance of motion that makes sleep impossible. On an uncrewed ship those constraints disappear, though the cargo's constraints, the ship's structural limits and the heavy-weather guidance remain. More significantly, the ship itself can be designed differently. Removing the accommodation block reduces weight and wind resistance, and eliminates the hotel load of air conditioning, galley and fresh-water production. The economic studies of unmanned ships, from MUNIN onwards, have found that these design gains contribute meaningfully to the business case. The third is integration with new propulsion. Wind-assisted propulsion, using rotor sails, rigid wing sails or suction wings, has been fitted to a growing number of ships since the early 2020s. Its benefit depends strongly on wind direction and strength along the route, which means it can be fully exploited only by an optimiser that routes for wind as well as against waves. Battery-electric short-sea ships such as the Yara Birkeland add a different constraint: the energy budget for each leg, the timing of recharging, and the need to keep a reserve for contingencies. In both cases, routing becomes inseparable from energy management. The fourth change is less comfortable. An optimiser seeking the cheapest route may pull a ship away from the main shipping lanes, into waters with less traffic, fewer nearby vessels to assist in an emergency and weaker communications coverage. For a crewed ship that may be an acceptable risk. For an uncrewed one, which depends on a reliable link to shore and may need outside assistance for any mechanical failure, it can be a significant one. An optimiser for an autonomous ship must therefore include support constraints: the availability of communications, the distance to ports of refuge, and the presence of other traffic or rescue capability. The best route on paper is worthless if it takes the ship somewhere it cannot be helped. That last point leads directly to the link between ship and shore, without which supervised autonomy cannot exist, and which is the subject of the next chapter. Chapter 7: The Link Ashore and Its Defence A crewed ship is a self-contained world. It can cross an ocean with its radios switched off, as ships did for centuries, and the people aboard will still navigate, maintain the machinery and handle emergencies. A supervised autonomous ship is different. Its design assumes a remote operations centre ashore, with a master who retains overall responsibility and operators who monitor the ship's systems, approve or override its decisions and take direct control when needed. That arrangement turns the communications link between ship and shore into safety-critical infrastructure. It also turns the ship, the link and the operations centre into a single system that must be defended as a whole. This chapter examines the link and the defensive cybersecurity that now surrounds it, treating security in the way the industry's own standards do: as a matter of identifying threats, building resilience and ensuring that a ship under digital attack remains a safe ship. What flows between ship and shore The traffic on the link falls into a few classes with very different requirements. Telemetry, a stream of measurements describing the ship's position, heading, speed, engine and power plant status, alarms and the state of the autonomous system itself, is modest in volume but must be continuous and reliable. The fused world picture, the list of tracked targets with their positions, classifications and uncertainties, is also compact. Video from cameras, and radar imagery, are far heavier; a remote operator who wants to see what the ship sees needs megabits per second, and more for several high-definition cameras. Commands from shore, ranging from approval of a planned manoeuvre to direct steering inputs, are tiny but require the highest integrity and the lowest delay. Delay, or latency, matters differently for each class. For supervision over minutes, a delay of a second is irrelevant. For direct remote control in close quarters, it can matter a great deal, because the operator sees the past and steers into the future. Large ships respond slowly enough that a delay of a second or so is tolerable even for remote manoeuvring, but in a narrow channel with a small craft crossing, every second of lag erodes margin. This is one reason most designs give the onboard system responsibility for immediate collision avoidance, with the remote operator supervising rather than steering in real time. Close to shore, ships can use terrestrial mobile networks, and some ports and coastal states are building dedicated coverage for that purpose. Beyond a few tens of miles, the link is by satellite, and the options have changed rapidly. Geostationary satellites, orbiting about 36,000 kilometres above the equator, have provided most maritime broadband for two decades. Very small aperture terminal services in the Ku and Ka bands, such as Inmarsat's Global Xpress network, now part of Viasat, deliver broadband to stabilised antennas aboard ship. Because the signal must travel to the satellite and back, a round trip between ship and shore through a geostationary satellite takes at least about half a second, and in practice often longer once network processing is added. Coverage thins at high latitudes, where the satellites sit low on the horizon. L-band services, including Inmarsat's long-established safety services and Iridium's low-orbit network, offer lower bandwidth but strong resilience to weather and, in Iridium's case, coverage of the polar regions. The IMO recognised Iridium as a provider for the Global Maritime Distress and Safety System in 2018, ending Inmarsat's long monopoly in that role. The large change since about 2022 has been the arrival of low Earth orbit broadband constellations, chiefly SpaceX's Starlink and Eutelsat's OneWeb. Orbiting a few hundred to about 1,200 kilometres up, they offer latency in the tens of milliseconds and bandwidth that would have been unaffordable at sea a few years ago, through flat electronically steered antennas that need no mechanical stabilisation. Adoption across commercial fleets has been rapid. For autonomous shipping, the effect is to make continuous video supervision and low-latency remote control far more practical over the open ocean than they were. A resilient design never relies on one path. The guidance from classification societies and the requirements of the MASS Code for connectivity point towards diversity: at least two independent links, ideally using different orbits, different providers and different frequency bands, so that a single outage, weather fade or service failure cannot sever the ship from shore. The ship must monitor the quality of each link, route traffic by priority so that commands and critical telemetry survive when bandwidth shrinks, and know what to do when every link fails at once. That last question belongs to the next chapter, but the principle is set here: the ship's immediate safety must never depend on the link. The link enables supervision; it must not be a single point of failure. The threat picture The maritime sector's digital vulnerability became widely visible in June 2017, when the NotPetya malware, spreading from compromised Ukrainian accounting software, disabled systems across A.P. Moller-Maersk, the world's largest container shipping company at the time. The company had to reinstall thousands of servers and computers, and estimated the cost at up to 300 million US dollars. Its ships kept sailing, but its booking and terminal systems were badly disrupted. The episode taught the industry that shipping companies are targets, sometimes collateral ones, and that shore systems are part of the attack surface for ships. Further incidents followed. The US operations of COSCO Shipping Lines were hit by ransomware in 2018; CMA CGM suffered a ransomware attack in 2020, the same year the IMO's own website and services were disrupted by a cyberattack; and in November 2023 a cyber incident forced DP World Australia to suspend operations at its container terminals in several Australian ports for about three days. For autonomous ships the threats of greatest concern fall into a few categories, which can be described at the level that defensive design requires. The first is interference with positioning and identity signals. Satellite navigation signals are extremely weak by the time they reach the Earth, which makes them easy to overpower. Jamming, which drowns the signal in noise, and spoofing, which substitutes false signals that lead a receiver to compute a wrong position, have been reported at scale in the eastern Mediterranean, the Black Sea, the Baltic and, in June 2025, across the Gulf and the Strait of Hormuz. Much of this activity is associated with military conflict and the defence of sensitive sites, and merchant ships are affected as bystanders. AIS, as Chapter 2 explained, is unauthenticated, so false reports can be injected and genuine ones misrepresented. Both forms of interference attack the inputs to navigation rather than the ship's computers, which is why the defences lie largely in perception design: cross-checking satellite position against radar, inertial sensors and dead reckoning, and treating AIS as a claim to be verified rather than a fact. The second is compromise of the ship's own operational technology: the networks and computers that run navigation, propulsion, steering, power management and cargo systems. Historically these were isolated, proprietary and assumed safe because they were not connected to anything. Today they are connected, for remote monitoring, software updates, performance analytics and, on an autonomous ship, remote supervision. Entry points include the satellite link itself, remote access used by equipment vendors for maintenance, removable media such as USB drives used to update electronic charts, and crew or contractor devices. Security researchers and classification societies have documented weaknesses in maritime equipment over many years, including outdated operating systems, default passwords and unencrypted protocols. The third, and specific to autonomous operation, is the remote operations centre. A single centre may supervise many ships. If it were compromised, or simply taken offline by ransomware, the consequences would extend across a fleet at once. The operations centre is therefore a high-value asset whose protection must be at least as rigorous as that of any ship, and whose loss the ships must be designed to survive. The standards The regulatory response has developed in layers since 2017. Table 6 summarises the principal instruments, as published by the bodies concerned. Table 6. Principal cybersecurity instruments affecting ships and their operators. Instrument Issued by Scope Key requirement Resolution MSC.428(98) IMO, 2017 Shipping companies under the ISM Code Address cyber risk in safety management systems from 2021 MSC-FAL.1/Circ.3, revised since IMO Guidelines for all ships and companies Identify, protect, detect, respond, recover IACS UR E26 Classification societies New ships contracted from 1 July 2024 Cyber resilience of the ship as a whole, verified by class IACS UR E27 Classification societies Onboard systems and equipment on those ships Security capabilities of equipment, based on IEC 62443 USCG cybersecurity rule (33 CFR Part 101) US Coast Guard, 2025 US-flagged vessels, facilities and offshore installations Cybersecurity plans, officer, training, incident reporting The IMO's first step, resolution MSC.428(98) adopted in 2017, required shipping companies to address cyber risks within their safety management systems under the International Safety Management Code no later than the first annual verification of their Document of Compliance after 1 January 2021. Its accompanying guidelines, MSC-FAL.1/Circ.3, organise cyber risk management around five functions: identify, protect, detect, respond and recover, the same structure used by the US National Institute of Standards and Technology's Cybersecurity Framework. The shipping industry's own guidance, the Guidelines on Cyber Security Onboard Ships produced by BIMCO with other industry associations and revised several times since 2016, translates these principles into practical measures for owners and crews. These early instruments were goal-based and largely procedural. The step change came from the International Association of Classification Societies, which adopted two unified requirements that class societies apply in surveying new ships. UR E26, on the cyber resilience of ships, sets requirements for the ship as a whole: an inventory of the computer-based systems on board, a design that separates them into security zones with controlled conduits between them, protection against unauthorised access, detection of incidents, and the ability to respond to and recover from them, all demonstrated to the class society through documentation and survey across the ship's life. UR E27, on the cyber resilience of onboard systems and equipment, sets requirements for the equipment suppliers, drawing on the international IEC 62443 series of industrial cybersecurity standards: user authentication, access control, protection of data in transit, logging, secure software updates, and the ability to withstand denial-of-service conditions. The original versions applied from January 2024; revised versions, which clarified scope and differentiated requirements by ship type and size, apply to new ships contracted for construction on or after 1 July 2024. Ships built under these requirements are now entering service in significant numbers. National and regional rules add further layers. The US Coast Guard published a final rule on cybersecurity in the marine transportation system in January 2025, which took effect in July 2025 with requirements phased in over the following years for US-flagged vessels and regulated facilities, including cybersecurity plans, designated cybersecurity officers, training and reporting of incidents. In the European Union, the NIS2 Directive brings maritime transport operators, including shipping companies and port operators, within its regime of security and incident-reporting obligations. And the MASS Code adopted in 2026 makes cybersecurity an explicit element of the safety case for autonomous and remotely controlled ships. Designing for resilience Standards set the floor. The design principles that make an autonomous ship genuinely resilient can be stated plainly, and none of them is exotic. Segment ruthlessly. The zones-and-conduits model at the heart of UR E26 means that the navigation and propulsion control systems sit in their own protected zones, separated from business networks, crew internet access and entertainment systems, with only defined, monitored and minimal connections between them. A compromise of the crew Wi-Fi on a crewed ship, or of a passenger network on a ferry, should have no route to the steering gear. Authenticate and verify every command. Any instruction from shore that can change the ship's behaviour should be cryptographically authenticated, checked for integrity and freshness so that old messages cannot be replayed, and validated against the ship's own understanding of its situation. An onboard system that receives a command to steer towards a charted hazard should refuse it and alert the operations centre, whatever its origin. Keep safety functions independent. The functions that keep the ship safe in the short term, such as collision avoidance, grounding avoidance and the transition to a minimum risk condition, should run on board, should not depend on the link or on shore systems, and should be protected against modification except through controlled, verified update processes. This is the security counterpart of the principle already stated for connectivity: the link enables supervision but is never a single point of failure. Diversify navigation inputs. Defences against interference with satellite positioning rest on not depending on it alone. Multi-constellation, multi-frequency receivers with interference detection; inertial navigation that can bridge outages; radar-based positioning against charted features; and authenticated satellite signals, such as the Open Service Navigation Message Authentication that Europe's Galileo system brought into public service in 2025, which allows a receiver to verify that navigation data come from the genuine satellites, all contribute. Research and pilot projects in Europe have also explored terrestrial backups, such as ranging signals transmitted from existing maritime radio beacons and AIS stations. The aim is not a perfect source but a system that detects inconsistency quickly and degrades gracefully. Monitor, log and practise. Detection depends on knowing what normal looks like: which devices exist, what traffic they generate, and when something changes. Logs must be kept securely enough that an investigation after an incident can reconstruct what happened. And response plans, including the recovery of systems from clean backups and the operation of the ship and fleet without the operations centre, must be exercised, not just written. The NotPetya experience showed that recovery capability, not prevention, ultimately determined how quickly a company returned to normal. Govern the supply chain. Much of an autonomous ship's software comes from equipment vendors and is updated over its life. Requirements such as UR E27 push security capability into the equipment itself, but owners must also control how vendors access systems for maintenance, how updates are verified before installation, and how vulnerabilities discovered after delivery are handled. None of these measures makes an autonomous ship immune to attack, any more than locks make a house immune to burglary. What they do is change the question from whether a compromise can happen to what happens when it does. The answer an autonomous ship must be able to give is that it detects the problem, contains it, tells someone, and remains a safe ship while it does so. That brings us to the theme on which the whole of this book converges: how an autonomous ship fails safely. Chapter 8: Failing Safely Every chapter so far has ended at the same place. Perception must carry honest uncertainty because it will sometimes be wrong. Collision avoidance must be tested against scenarios designed to defeat it because it will sometimes meet one. Routing must respect the ship's need for support because things will break. The link to shore must never be a single point of failure because it will sometimes fail, by accident or by attack. This chapter takes the next step and asks the question directly: when an autonomous ship meets a situation it cannot handle, what should it do, and how can anyone be confident that it will do it? The answer is the core of the safety case for crewless shipping, and it involves engineering, human factors, assurance and law in roughly equal measure. Envelopes and fallbacks The starting point is to define where the autonomous system is permitted to operate at all. The automotive industry calls this the operational design domain; maritime projects often speak of an operational envelope. It is the set of conditions under which the system has been designed, tested and approved to operate: a geographical area or route, ranges of visibility, wind and sea state, traffic density, and the required health of sensors, machinery and communications. The MEGURI2040 ships are approved for designated waters. The Yara Birkeland sails a specific short route. A deep-sea autonomous ship might be approved for open-ocean passage within certain weather limits, but not for port approaches without a pilot. An envelope is useful only if the ship continuously monitors whether it is inside it. That means measuring visibility from cameras and from the reduction in radar detection range, sea state from the ship's own motion, traffic density from the track picture, link quality from the communications system and health from every sensor and machinery monitoring point. It also means predicting whether the ship will remain inside the envelope over the coming minutes and hours. A system that notices it has left its envelope only after the fact has failed. A system that sees fog forming ahead, or traffic thickening, or a link degrading, and alerts the remote operations centre with time to spare, has done its job. The amount of warning matters because the human in the loop needs time. Research on takeover of automated systems, across road vehicles, aviation and process control, consistently finds that a person who has been monitoring rather than actively controlling needs time to rebuild an understanding of the situation before acting well. The loss of Air France flight 447 over the Atlantic in 2009, in which the autopilot disconnected after airspeed sensors iced over and the crew failed to recover control of an aircraft that was flyable, is the best known illustration of how badly an abrupt handover can go. Ships move slowly enough that maritime takeovers can usually be given minutes rather than seconds, and well-designed systems exploit that. They raise requests early, present the operator with a concise summary of why help is needed and what the system proposes, and continue acting safely while the operator gets oriented. When the ship can no longer continue its mission safely, whether because it has left its envelope, lost a critical sensor, lost its link or detected a fault in its own decision-making, it must transition to what the field calls a minimum risk condition: a stable state in which the risk to the ship, to other vessels and to the environment is as low as circumstances allow, and which can be held until help arrives or the problem is resolved. What that state is depends entirely on context, and this is where maritime fallback differs most from its road equivalent. On a road, the minimum risk manoeuvre is usually to stop at the side. At sea, stopping can be dangerous. A ship that stops in the middle of a traffic separation scheme becomes an obstacle in a stream of traffic. A ship that stops and drifts near a lee shore will be driven aground. A ship that loses propulsion in heavy weather may fall broadside to the waves. The appropriate minimum risk condition might instead be to continue on the planned route at reduced speed while applying collision avoidance autonomously; to leave the traffic lane and proceed to a pre-planned safe area; to anchor in a designated anchorage; to hold position using dynamic positioning, where fitted; or, as a last resort, to stop and drift while broadcasting warnings. A well-designed system therefore carries not one fallback but a hierarchy of them, each assigned to particular phases of the voyage and particular kinds of failure, and each pre-computed so that it can be executed without the failed component. The voyage plan is annotated with these fallbacks before departure: for each segment, where would the ship go if it lost its link, if it lost a radar, if it lost propulsion on one engine? The plan is approved with its fallbacks, as a crewed ship's passage plan is approved with its contingencies, and the fallbacks are updated as the voyage proceeds. Consider a worked example. An uncrewed coastal container ship is on a night passage, approaching a traffic separation scheme. Its satellite links are healthy, visibility is good, and traffic is moderate. At 0210 the ship's systems detect that the satellite positions from its navigation receivers have begun drifting relative to radar fixes on charted headlands, a pattern consistent with spoofing. The system downgrades its confidence in satellite positioning, switches its primary positioning to a blend of radar fixes and inertial navigation, widens its safety margins, reduces speed and alerts the remote operations centre with a summary: suspected spoofing, position currently maintained by radar and inertial means, estimated uncertainty, recommended plan to continue at reduced speed to the next waypoint and then reassess. The operator reviews the evidence, agrees, and notifies the coastal authority. At 0240, the primary satellite link drops out. The secondary link carries telemetry but cannot sustain video. The ship's link-loss procedure for this segment says: continue autonomously on plan, at reduced speed, with full collision avoidance, for up to thirty minutes; if both links are lost for longer, leave the planned route before the separation scheme and proceed to a pre-designated holding area clear of traffic, where it can anchor. The secondary link holds, the primary is restored at 0255, and the ship continues. Nothing dramatic happened, which is the point. Each failure was detected, communicated and handled by a pre-planned, graduated response, and the ship remained predictable to other vessels throughout. Machinery and people The fallback logic for navigation is comparatively mature. The harder problem, as Chapter 1 noted, is machinery. On a crewed ship, engineers spend much of every day maintaining and repairing equipment, and many potential failures are caught as small problems before they become large ones: a leaking joint, a rising temperature, a vibration that sounds wrong. The Dali's blackout at Baltimore in 2024, traced by investigators to a single loose wire, shows how a minor defect can cascade into catastrophe, even with a crew on board. An uncrewed ship approaches this problem from several directions at once. It uses redundancy, duplicating critical systems such as propulsion, steering and power generation so that a single failure does not leave the ship helpless. It uses extensive condition monitoring, with sensors for temperature, pressure, vibration and oil quality feeding models that predict failures before they occur, and with the data sent ashore for engineering staff to review. It simplifies, preferring electric propulsion, fewer moving parts and equipment designed to run unattended for long periods. And it schedules maintenance in port, where shore teams can do the work a crew would have done at sea. Fire is a particular concern, since there is no crew to fight it, and designs rely on fixed detection and suppression systems throughout. The MASS Code includes requirements on fire safety for exactly this reason. These measures are well understood in principle and have long been applied in other industries, such as unattended machinery spaces on crewed ships and remote offshore installations. What remains uncertain is how far they can be pushed on long deep-sea voyages, where a ship may be days from the nearest port and from any assistance. That is one reason why the first commercial autonomous ships operate on short, coastal and domestic routes, and why the path to crewless deep-sea shipping is likely to run through long periods of reduced crewing, with fewer people aboard and much of the watchkeeping automated, before crews are removed entirely. Removing people from the ship does not remove them from the system. It moves them. The remote operations centre is staffed by people who supervise ships, interpret alerts, take decisions and sometimes take direct control. Their work has its own human factors, and these are just as consequential as the bridge-team factors that feature in accident reports today. The first is vigilance. Monitoring a system that almost always works is tiring and tends to erode attention, a pattern documented in human factors research since the Second World War. Operations centres must be designed so that operators are engaged at a sustainable level, with alerts that are meaningful rather than constant, and with rotations and workloads that keep them sharp. The second is the ratio of operators to ships. The economic promise of remote supervision is that one operator can watch several ships. The safety question is how many, under what conditions, and what happens when two ships need help at the same time. That ratio is not a fixed number. It depends on the phase of voyage, the traffic, the weather and the maturity of the onboard autonomy, and it must be set and adjusted on the basis of evidence. The third is skill. A remote operator who has never stood a watch at sea may lack the feel for a situation that an experienced officer brings. Most current projects staff their operations centres with qualified mariners for this reason. Over the longer term, the international training standards for seafarers, under the STCW Convention, which the IMO is now comprehensively reviewing, will need to address the qualifications of remote operators and remote masters. Recording what the machine was thinking Every ship of significant size already carries a voyage data recorder, the maritime counterpart of an aircraft's flight recorder. Required under SOLAS since the early 2000s, it stores bridge audio, VHF communications, radar images, position, heading, speed, engine orders and alarms, in a protected capsule designed to survive a sinking. Accident investigators rely on it to reconstruct what happened on the bridge in the final hours before a casualty. An autonomous ship needs more than this, because the bridge is no longer where decisions are made. When an investigator asks why the ship turned to starboard at 0347, the answer lies in the state of the autonomous system: which targets it was tracking and with what uncertainty, how it had classified the encounter, which manoeuvres it had considered and how it had scored them, and whether its operating envelope monitor had raised any warnings. It also lies in the operations centre: what the remote operator saw on the screen, what alerts were presented, what the operator did and when. Without those records, an investigation into an autonomous ship's accident would face the same problem as an investigation into a crewed ship without witnesses. The practical requirement is therefore that autonomous systems record their own decision-making in enough detail to reconstruct it, that operations centres record their own displays and actions, and that both are time-synchronised with the ship's conventional recorder. This is technically straightforward, since the data exist inside the systems already, but it must be designed in from the start, with formats that investigators can read and protections that prevent records being altered. It also has a value beyond accident investigation. Recorded decisions, including near-misses and handovers that ended well, are the raw material for the evidence-based widening of operating envelopes, and for improving the cost functions and parameters described in Chapter 5. The recorder becomes a learning system as well as a forensic one. There is a further benefit that matters to regulators and courts: explainability. A collision-avoidance system whose decisions can be reconstructed and explained in terms a mariner understands, such as "the target was classified as a crossing vessel on the starboard bow with a CPA of two cables, so the system altered 50 degrees to starboard to pass astern of it", is far easier to approve and to defend than one whose reasoning cannot be recovered. This is one reason why, as Chapter 5 noted, opaque learned policies remain largely confined to perception and prediction rather than final decision-making. Assurance, responsibility and the law How does a regulator or a class society decide that all of this is good enough? The emerging answer is the safety case: a structured argument, supported by evidence, that the ship achieves a level of safety at least equivalent to a conventional ship in its intended operations. That is the core test of the MASS Code. Classification societies have published guidance on how to build such a case, beginning with DNV's class guideline for autonomous and remotely operated ships in 2018, and others have followed with their own rules and guidelines. The evidence typically includes hazard analysis of the whole system, ship, link and operations centre together; simulation and scenario testing of the kind described in Chapter 5; sea trials; and operational data gathered over time. Methods developed for complex socio-technical systems, such as the systems-theoretic process analysis developed by Nancy Leveson at the Massachusetts Institute of Technology, which examines how control and feedback between components can fail even when no component itself fails, are increasingly applied, because the hazards of autonomous ships often lie in interactions between the onboard system, the remote operator and other vessels. Responsibility follows the principle the MASS Code sets: the master retains overall responsibility for the ship at all times, even when not on board. In practice this means a named, qualified person, usually in the operations centre, carries the authority and accountability that a master on board would carry. The shipowner remains responsible for providing a seaworthy ship, which under long-established maritime law means one that is properly equipped, manned and fit for the voyage; for an autonomous ship, "manned" is reinterpreted to include the operations centre and its staff, and "equipped" to include the autonomous systems, their software and their cybersecurity. Manufacturers of autonomous systems may also face liability where defects in their products cause harm, which is one reason they participate closely in class approval. Some obligations do not translate neatly. Every master has a duty under the SOLAS Convention to proceed with all speed to assist persons in distress at sea, when able to do so. An uncrewed ship cannot pull survivors from the water. It can, however, relay distress messages, stand by, illuminate the scene, provide a lee and deliver rescue equipment, and the MASS Code's requirements on search and rescue address how autonomous ships should contribute. The broader regulatory programme, including the experience-building phase that the IMO plans to run before the code becomes mandatory around 2032, exists precisely to discover such gaps through real operations rather than to guess at them. What emerges from all this is a picture of an autonomous ship not as a vessel that never needs help, but as one designed around the moment when it does. Its perception knows how uncertain it is. Its decision-making knows the limits of its envelope. Its link to shore is diverse and defended, but never essential for immediate safety. Its machinery is redundant and monitored. Its people ashore are given time, information and authority. And when something goes wrong, as it will, it degrades in a planned, predictable way to a state that harms no one. That is a more modest vision than some of the early publicity promised, and a far more credible one. Conclusion The argument of this book has been that the hard problem in autonomous shipping is not steering but judgement under uncertainty, and that the right response is to design ships that know the limits of their own knowledge and have rehearsed ways of becoming safe when they reach them. If that argument is correct, several consequences follow for the people who will build, regulate, insure, operate and share the sea with these ships. They are worth stating plainly, because some of them run against the way the subject is usually discussed. Capability is the wrong headline The first consequence is that the most important measure of an autonomous ship is not what it can do on a good day but how it behaves on a bad one. Public attention, and much of the industry's own marketing, focuses on demonstrations of capability: an ocean crossing, a busy bay transited, a hundred collisions avoided. These matter, but they say little about safety, because the dangerous situations are rare and the demonstrations select for success. The better questions are the ones Chapter 8 posed. What does the ship do when its positioning is spoofed at night near a traffic separation scheme? When it loses its primary link? When a camera ices over and a radar fails at the same time? When the remote operator is handling another ship? A shipowner evaluating a system, a class surveyor approving one or a port authority deciding whether to admit one should ask for the fallback hierarchy and the evidence behind it before asking about anything else. A vendor who answers those questions crisply has understood the problem. One who changes the subject back to capability has not. Envelopes should grow on evidence, not ambition The second consequence concerns pace. The credible projects described in this book, the Yara Birkeland, the MEGURI2040 ships, the Avikus systems, have all advanced by defining a narrow operating envelope, operating inside it under supervision, gathering evidence and then widening it. The Yara Birkeland went from five crew on the bridge to three over its first years of service. The Japanese programme moved from demonstrations in 2022 to certified commercial service on designated domestic routes in 2026. The IMO has chosen a similar path for regulation: a non-mandatory code in 2026, an experience-building phase, a mandatory code targeted for adoption around 2030 and entry into force in 2032. This incremental pattern is sometimes criticised as slow. It is better understood as the only path that produces the evidence a safety case needs. Each widening of the envelope, a longer route, a busier port, a reduction in crew, should be justified by data from the narrower one. That implies something practical about data: owners and developers who collect, share and analyse operational evidence systematically, including near-misses and handovers, will widen their envelopes faster and more safely than those who treat such data as proprietary embarrassment. An industry-wide arrangement for sharing anonymised autonomy incident data, comparable to the confidential reporting schemes that aviation built over decades, would do more to accelerate safe autonomy than any single technical advance. The crewed fleet is the main beneficiary, for now The third consequence is that the near-term value of everything in this book lies mostly with ships that still carry crews. Camera-based detection that finds the unlit fishing boat, fusion that flags a drifting positioning fix, collision-avoidance advice that proposes a bold and early alteration, routing that saves several per cent of fuel and improves a ship's carbon intensity rating: all of these are available today as aids to officers on watch. The global fleet is large and long-lived, and ships ordered this year will still be sailing in the 2050s. For decades, crewless ships will be a small share of the ships an autonomous navigation system meets, and a rising share of crewed ships will be using the same technologies to assist their officers. That has a further implication. Over time, many of the vessels an autonomous ship encounters will themselves be partially guided by algorithms. The designers of these systems, and the regulators who approve them, need to think about how different systems interact with one another, not only how each behaves against a conventional human-crewed ship. Seamanship has to be written down The fourth consequence is cultural. The COLREGs work because they rest on shared, largely unwritten seamanship, and Chapter 4 showed that autonomy forces that seamanship into explicit parameters: how close is close quarters, how big is readily apparent, how soon is ample time. This is uncomfortable for a profession that has always prized judgement over formula. It is also an opportunity. Explicit parameters can be examined, debated, tested against accident data and improved. The maritime community, through the IMO, class societies, accident investigators and professional bodies, is well placed to develop shared, published interpretations of the rules for machine application, so that every autonomous ship behaves in a way every other mariner can predict. Leaving each developer to invent its own interpretation would recreate, among machines, exactly the uncertainty that the rules exist to remove. Security is a property of the system, not the ship The fifth consequence concerns cybersecurity. Chapter 7 showed that the requirements of IACS UR E26 and E27, the IMO guidelines and national rules have moved maritime cybersecurity from good intentions to auditable obligations. But for autonomous ships, the unit that must be secured is not the ship. It is the ship, its links, its operations centre, its vendors and the positioning and identity signals it depends on, considered as one system. The most consequential single asset in that system may well be the operations centre, because it supervises many ships at once. And the most important defensive principle is not any particular control but the architectural one: no compromise of the link or the shore should be able to make a ship unsafe in the short term. Ships designed on that principle can tolerate the interference that is now routine in several of the world's most important waterways. Ships not designed on it should not be allowed to sail without crews. The people do not disappear The last consequence is about people. Autonomy at sea is often framed as the removal of seafarers. In practice, at least for the next two decades, it is a redistribution of their work: from bridges to operations centres, from routine watchkeeping to supervision and intervention, from the engine room at sea to maintenance teams in port. The skills that matter most in that redistributed system, situational awareness, judgement in ambiguous encounters, an engineer's feel for a machine about to fail, are the skills that experienced seafarers already have. The industry's shortage of qualified officers, which is one of the reasons for interest in autonomy, is also one of its constraints: it needs those same people to supervise, test and certify the systems that will eventually reduce the demand for them. Training standards, career paths and working conditions for remote operators are therefore not a side issue. They are part of the safety case. The crewless cargo ship is arriving, but not in the form of a vessel that sails alone, sees everything and never needs help. It is arriving as a carefully bounded system of ship, link and shore, operating within envelopes that grow with evidence, and designed above all to fail safely. That is less romantic than the early vision. It is also the only version that will earn the trust of the seafarers, port authorities, insurers and public whose consent it needs. Notes The following sources underpin the principal facts and figures in the text. Regulatory instruments are cited by their official designations so that readers can locate the current versions, which are periodically revised. Introduction On the Front Eagle and Adalynn collision of 17 June 2025 and the scale of satellite navigation interference in the Gulf, reporting by GPS World ("GNSS jamming widespread in Strait of Hormuz, ships collide", June 2025), citing analysis by Windward. On the adoption of the MASS Code at the 111th session of the Maritime Safety Committee (May 2026) and its timetable for a mandatory version, IMO press briefing "IMO adopts MASS Code", 2026, and summaries by DNV and ABS of MSC 111 outcomes. On the MEGURI2040 commercial services of the Genbu and Hokuren Maru No. 2, the Nippon Foundation, "Japanese consortia start commercial service of two fully autonomous ships under the MEGURI2040 project", 2026. Chapter 1 Seafarer population and officer shortfall figures: BIMCO and International Chamber of Shipping, Seafarer Workforce Report, 2021 edition. Share of world trade carried by sea: UNCTAD, Review of Maritime Transport, successive editions. The four degrees of autonomy: IMO, Outcome of the Regulatory Scoping Exercise for the use of Maritime Autonomous Surface Ships, MSC.1/Circ.1638, 2021. Yara Birkeland operation, crew numbers and road-transport savings: Kongsberg, "First electric autonomous ship", 2024. Prism Courage voyage, collision avoidance events and fuel and emissions figures: HD Hyundai and Avikus press release, June 2022. Dali allision with the Francis Scott Key Bridge and the cause of the blackout: US National Transportation Safety Board, final report on the Key Bridge collapse, November 2025. MUNIN project: Maritime Unmanned Navigation through Intelligence in Networks, EU Seventh Framework Programme, 2012 to 2015, final reports. Chapters 2 and 3 AIS carriage requirements: SOLAS Chapter V, Regulation 19. Reporting intervals and access scheme: ITU-R Recommendation M.1371. Operational guidance: IMO Resolution A.1106(29), 2015. Radar performance standards including target tracking: IMO Resolution MSC.192(79), 2004. Kalman, R. E., "A New Approach to Linear Filtering and Prediction Problems", Journal of Basic Engineering, 1960. Prasad, D. K., and others, "Video Processing from Electro-Optical Sensors for Object Detection and Tracking in a Maritime Environment: A Survey", IEEE Transactions on Intelligent Transportation Systems, 2017, which introduced the Singapore Maritime Dataset; Shao, Z., and others, "SeaShips: A Large-Scale Precisely Annotated Dataset for Ship Detection", IEEE Transactions on Multimedia, 2018. Chapters 4 and 5 Convention on the International Regulations for Preventing Collisions at Sea, 1972, as amended. Fujii, Y., and Tanaka, K., "Traffic Capacity", Journal of Navigation, 1971; Goodwin, E. M., "A Statistical Study of Ship Domains", Journal of Navigation, 1975. Standards for ship manoeuvrability: IMO Resolution MSC.137(76), 2002. Fiorini, P., and Shiller, Z., "Motion Planning in Dynamic Environments Using Velocity Obstacles", International Journal of Robotics Research, 1998; Kuwata, Y., and others, "Safe Maritime Autonomous Navigation with COLREGS, Using Velocity Obstacles", IEEE Journal of Oceanic Engineering, 2014. Johansen, T. A., Perez, T., and Cristofaro, A., "Ship Collision Avoidance and COLREGS Compliance Using Simulation-Based Control Behavior Selection with Predictive Hazard Assessment", IEEE Transactions on Intelligent Transportation Systems, 2016. Chapter 6 Shipping's share of global greenhouse gas emissions: IMO, Fourth IMO Greenhouse Gas Study, 2020. James, R. W., Application of Wave Forecasts to Marine Navigation, US Navy Hydrographic Office, 1957; Hagiwara, H., Weather Routing of (Sail-Assisted) Motor Vessels, doctoral thesis, Delft University of Technology, 1989. Heavy-weather guidance: IMO MSC.1/Circ.1228, 2007. Just-in-time arrival: IMO GloMEEP and Global Industry Alliance, Just In Time Arrival Guide, 2020. On the October 2025 adjournment of the Net-Zero Framework adoption: reporting by Seatrade Maritime and the Global Maritime Forum, October 2025. Chapter 7 Maersk's estimate of NotPetya costs: A.P. Moller-Maersk interim report, second quarter 2017. IMO Resolution MSC.428(98), 2017, and MSC-FAL.1/Circ.3 and its revisions. IACS Unified Requirements E26 and E27 and their revised application dates: IACS press release on UR E26 and E27. US Coast Guard final rule, "Cybersecurity in the Marine Transportation System", Federal Register, January 2025. Chapter 8 DNV, Class Guideline DNVGL-CG-0264, Autonomous and Remotely Operated Ships, 2018, and later editions. Leveson, N. G., Engineering a Safer World: Systems Thinking Applied to Safety, MIT Press, 2011. Bureau d'Enquêtes et d'Analyses, Final Report on the Accident on 1st June 2009 to the Airbus A330-203, Flight AF 447, 2012. Glossary AIS (Automatic Identification System). A VHF radio system in which ships broadcast their identity, position, course, speed and status; mandatory for most merchant ships under SOLAS. ARPA (Automatic Radar Plotting Aid). Radar functionality that automatically tracks echoes and calculates their course, speed, CPA and TCPA. COLREGs. The International Regulations for Preventing Collisions at Sea, 1972, which govern how vessels avoid collision. CPA and TCPA. Closest point of approach between two vessels if both hold course and speed, and the time until it occurs. CII (Carbon Intensity Indicator). The IMO's annual operational rating, from A to E, of a ship's carbon dioxide emissions per unit of transport work. GNSS. Global navigation satellite systems, including GPS, Galileo, GLONASS and BeiDou. MASS. Maritime Autonomous Surface Ship, the IMO's term for ships that can operate with varying degrees of independence from human interaction. Minimum risk condition. A stable state to which an autonomous ship transitions when it cannot continue safely, chosen to minimise risk in the circumstances. Operational envelope. The set of conditions, including area, weather, traffic and system health, within which an autonomous system is approved to operate. ROC (Remote Operations Centre). A shore facility from which autonomous or remotely controlled ships are supervised and, when necessary, controlled. Traffic separation scheme. A routing measure that separates opposing streams of traffic into lanes, governed by COLREGs Rule 10. UR E26 and E27. Unified requirements of the International Association of Classification Societies on the cyber resilience of ships and of onboard systems and equipment. Further Reading A. N. Cockcroft and J. N. F. Lameijer, A Guide to the Collision Avoidance Rules. The standard commentary on the COLREGs, rule by rule, with case law and interpretation. Essential for anyone encoding the rules in software. Thor I. Fossen, Handbook of Marine Craft Hydrodynamics and Motion Control (Wiley, second edition 2021).** The reference work on ship dynamics, guidance and control, including the models used inside collision-avoidance planners. Alan Bole, Alan Wall and Andy Norris, Radar and ARPA Manual (Butterworth-Heinemann, third edition 2014).** A thorough explanation of marine radar, target tracking and their limitations. Andy Norris, Integrated Bridge Systems, volumes on radar and AIS and on ECDIS and positioning (Nautical Institute). Clear, practical accounts of how bridge systems work and fail, from a long-standing authority on navigation technology. IMO, MSC.1/Circ.1638, Outcome of the Regulatory Scoping Exercise for the Use of Maritime Autonomous Surface Ships (2021).** The IMO's systematic review of which instruments autonomy affects and how; the foundation for the MASS Code. Nancy G. Leveson, Engineering a Safer World (MIT Press, 2011).** The case for systems-theoretic safety analysis, increasingly applied to autonomous ships and their operations centres. Tor Arne Johansen, Tristan Perez and Andrea Cristofaro, "Ship Collision Avoidance and COLREGS Compliance Using Simulation-Based Control Behavior Selection with Predictive Hazard Assessment" (IEEE Transactions on Intelligent Transportation Systems, 2016). A clear and influential statement of the model predictive approach to collision avoidance. BIMCO and others, The Guidelines on Cyber Security Onboard Ships (latest edition).** The shipping industry's practical guidance on cyber risk management, written for owners, operators and crews. IACS, Unified Requirements E26 and E27 (revised editions). The class requirements for cyber resilience of new ships and their equipment, freely available from the IACS website. IMO, Fourth IMO Greenhouse Gas Study (2020).** The authoritative assessment of shipping's emissions and of the operational and technical measures to reduce them. Hashtags: #AutonomousShipping #MaritimeAutonomy #AutonomousShips #MaritimeNavigation #NavigationAlgorithms #CollisionAvoidance #AutonomousNavigation #MaritimeSafety #MarineEngineering #UnmannedSurfaceVessels #MASS #COLREGs #MarineRadar #AutomaticIdentificationSystem #AIS #SensorFusion #MaritimeComputerVision #KalmanFiltering #VesselTracking #ClosestPointOfApproach #ModelPredictiveControl #VelocityObstacles #PathPlanning #MaritimeArtificialIntelligence #MachineLearning #WeatherRouting #VoyageOptimization #MaritimeCybersecurity #GNSSSpoofing #RemoteShipOperations