Download the Book (PDF): Introduction Walk through any campus library at eleven o'clock on the Tuesday of finals week and you can read the state of student health from the tables. Energy drink cans lined up like trophies. A half-eaten bag of pretzels that has been dinner. Someone asleep on a textbook, someone else with a hoodie pulled over their eyes and a leg bouncing under the table, a third person scrolling a group chat where everyone is announcing how little they have slept. None of this looks like a health crisis. It looks like effort. For most students it is simply what finals week is supposed to look like. This book starts from a different premise. The final exams at the end of a semester are a test of what you have learned, but they are also, whether anyone intends it or not, a test of how well you can run your body under load. Memory is made and retrieved by a physical organ that needs sleep to store what it has taken in, fuel to keep its attention steady, and a nervous system calm enough to let information come back out on demand. When students treat sleep, food, and stress as distractions to be managed after the exams, they quietly undercut the very machinery the exams are measuring. Three systems that pull on one another Most advice about finals week arrives in separate packages. The sleep advice comes from a health center flyer. The nutrition advice comes from a dining hall poster about blueberries. The stress advice comes from a wellness email suggesting a therapy dog event. Each piece is reasonable on its own. What gets lost is how tightly the three are linked, and that linkage is the reason this book exists. Consider how a typical exam week unravels. A student falls behind on a problem set and stays up late to finish it. The short night leaves them groggy, so they have a large coffee in the afternoon and another in the evening. The caffeine pushes their bedtime later still, and anxiety about the upcoming exam keeps them awake once they finally lie down. The next morning they skip breakfast because they overslept, run on vending-machine snacks, and feel jittery and unfocused by mid-afternoon, which they interpret as proof that they are not prepared. That feeling of unreadiness raises their stress, which makes sleep harder again. By Thursday they are considering an all-nighter, and someone down the hall has offered them a pill. Every link in that chain is ordinary, and every one is backed by research. Short sleep increases the appetite for quick, sugary food and makes emotional reactions stronger. Caffeine taken even six hours before bed measurably shortens sleep. Stress hormones and racing thoughts delay sleep onset. Hunger and dehydration make attention wobble, and a wobbly mind reads its own state as danger. The loop is self-reinforcing, which is why it feels so hard to escape once it has started, and why fixing only one part of it tends to disappoint. The encouraging corollary is that the loop runs in the other direction too. A reasonable night of sleep makes it easier to eat sensibly and to keep a level head. Regular meals and limited caffeine make sleep come more easily. A few minutes of controlled breathing before bed can make the difference between lying awake rehearsing disaster and actually falling asleep. You do not need to transform your life in a week. You need to pull a few levers that move the whole system at once. What this book offers The chapters that follow are organised around those levers. The first two deal with sleep: what it actually does for learning, what the research says about the all-nighter, and what to do when you are genuinely out of time. The third covers the study methods that give you the most learning per hour, because the fastest way to protect sleep is to stop wasting study time on techniques that feel productive and are not. The next three are about fuel: an honest look at what nutrition can and cannot do for an exam, a frank chapter on caffeine and on the misuse of prescription stimulants, and a practical chapter with sample eating plans for students who eat in a dining hall and for those living on a dorm budget with a microwave and a mini-fridge. The seventh chapter turns to stress itself, with specific techniques you can use in the minutes before an exam or in the middle of one, along with a clear account of test anxiety. The last chapter walks through exam day hour by hour and then addresses the part of the semester nobody plans for: the days after the last exam, when the body presents its bill. Throughout, the aim is to be honest about evidence. Some of what you will read rests on large, repeated studies: the benefits of practice testing and spaced review, the damage that sleep loss does to memory formation, the safe upper range of caffeine for healthy adults. Some rests on promising but limited work, such as recent trials of specific breathing techniques, and those are presented as what they are. Some popular claims, especially about foods that supposedly boost brainpower, rest on very little, and you will be told so plainly. A student who knows which advice is solid can spend their limited energy where it counts. Who this is for The primary reader is a college student facing a stretch of exams, papers, and projects, whether a first-year meeting finals for the first time or a senior who has survived several and would like the next round to feel different. The book is also written for the people around those students: parents who worry from a distance, resident advisers, coaches, academic advisers, and staff in student services who want to give advice that is grounded rather than generic. Where the advice changes depending on circumstance, for instance on whether you have a kitchen, a meal plan, or very little money for food, the book tries to meet you where you are. The scenarios in these pages describe composite students built from common situations. They are illustrations, not case studies of real people. The numbers, studies, and recommendations are real, and the Notes at the end identify the sources. A note on health and safety This book offers general information, not individual medical or psychological advice, and it does not replace care from a clinician who knows your history. If you have a diagnosed sleep disorder, diabetes, an eating disorder, a heart condition, ADHD, an anxiety disorder, or any condition that affects how you should sleep, eat, or use medication, the guidance of your own clinician takes precedence over anything here. Some situations call for help immediately rather than a better study plan. If you are having thoughts of harming yourself, or feel you cannot keep yourself safe, call or text 988 to reach the 988 Suicide and Crisis Lifeline in the United States, contact your campus counseling center's after-hours line, or call emergency services. Chest pain, a racing or irregular heartbeat that does not settle, fainting, or confusion after taking stimulants or large amounts of caffeine are medical emergencies. Most campuses have counseling, health, and basic-needs services that are busiest in the weeks before finals precisely because students wait too long to use them. Using them early is not a sign of weakness. It is part of the plan. The argument in one sentence If there is a single idea to carry through the chapters that follow, it is this: during finals week, taking care of your body is not a reward you earn after the studying is done, but one of the most effective forms of studying available to you. The student who sleeps seven hours, eats three unremarkable meals, and knows how to bring their heart rate down in the hallway before an exam is not being soft on themselves. They are protecting the organ that will sit the test. Chapter 1: Sleep Is Part of Studying Ask a room of students what sleep is for and most will describe it as rest: the body switching off so it can recharge. That description is not wrong, but it misses the part that matters most at the end of a semester. For the brain, sleep is not downtime. It is a period of intense, organised work in which the day's experiences are sorted, strengthened, linked to older knowledge, and in many cases prepared for recall days or weeks later. A student who cuts sleep to gain study hours is not simply trading rest for work. They are interrupting one kind of learning to do another. This chapter explains, in practical terms, what sleep contributes to memory, why it matters both before and after you study, why the benefits accumulate over weeks rather than being decided on a single night, and how to protect sleep when every instinct in finals week says to spend it on something else. The three jobs of memory, and where sleep fits Psychologists commonly divide memory into three stages. Encoding is the moment information first gets in: you read a paragraph about the Krebs cycle, and some representation of it forms in your brain. Consolidation is the slower process by which that fragile trace becomes stable, less vulnerable to interference, and connected to what you already know. Retrieval is the act of getting it back out when a question demands it. Exams test retrieval, but retrieval can only find what encoding put in and consolidation kept. Sleep touches all three stages, and it touches them in different ways. Encoding depends heavily on the hippocampus, a curved structure deep in each temporal lobe that acts as a temporary index for new facts and events. A well-known 2007 study by Seung-Schik Yoo, Matthew Walker, and colleagues, published in Nature Neuroscience, kept one group of young adults awake for a full night and let another group sleep normally. The next day both groups studied a set of pictures while their brains were scanned. The sleep-deprived group showed markedly reduced hippocampal activity while learning and remembered significantly fewer of the items when tested later. The authors' conclusion was blunt: a lack of prior sleep substantially compromises the brain's capacity to commit new experiences to memory. In practical terms, the night before a long study day is part of that study day. An all-nighter spent reviewing does not only cost you the next day's alertness. It dulls the machinery you would need to learn anything new tomorrow. Consolidation is where sleep has been studied most intensively. A major review by Björn Rasch and Jan Born in Physiological Reviews in 2013 brought together decades of evidence that sleep after learning benefits the retention of many kinds of memory, including the fact-and-concept memory that most exams test. During the deep, slow-wave stages of sleep, which are concentrated in the first half of the night, the brain appears to reactivate recently learned material, replaying patterns of activity from the day and gradually shifting memories from their temporary hippocampal index into more durable storage in the cortex. Rapid eye movement sleep, which is concentrated in the later part of the night, appears to contribute to other kinds of processing, including the integration of new information with existing knowledge and the regulation of emotional memory. Timing matters here as well. Steffen Gais, Brian Lucas, and Jan Born reported in 2006 that students who learned vocabulary and slept within a few hours retained more than students whose sleep came much later, even when the amount of time and the opportunity for interference were matched. Studying in the evening and then sleeping is therefore not a waste of the evening. It places the material right where the brain's overnight processing can reach it. Retrieval, the third stage, depends on attention, working memory, and the ability to inhibit irrelevant thoughts, all functions of the prefrontal cortex that are notably sensitive to sleep loss. A tired brain may still hold the information but struggle to search for it efficiently, to keep the question in mind while assembling an answer, or to notice that it has misread a question. This is why so many students describe the experience of walking out of an exam after a bad night and suddenly remembering the answer on the stairs. Sleep is a long game, and a hidden one If sleep matters this much, one might expect the night before an exam to be decisive. The best objective evidence we have from real classrooms suggests something more interesting. In 2019, Kana Okano, John Gabrieli, Jeffrey Grossman, and colleagues at the Massachusetts Institute of Technology gave wearable activity trackers to 100 students in an introductory chemistry course; 88 completed the study. The trackers recorded their sleep throughout the semester, and the researchers compared those records with the students' scores on quizzes and exams. Students who slept longer, slept better, and kept a more consistent schedule earned higher grades, and together the sleep measures accounted for nearly a quarter of the variation in academic performance. That is a large share for a single behaviour in a class where aptitude, preparation, and prior coursework all vary widely. The surprising finding was what did not predict grades. Sleep on the single night before a test showed no relationship to test performance. What mattered was sleep across the week and the month before. The interpretation that best fits the evidence is that sleep works on the learning as it happens: every night of good sleep during the semester consolidates the lectures, readings, and problem sets of the preceding days. By the time the exam arrives, most of the relevant consolidation has already occurred or failed to occur. This finding has two practical consequences, and both are worth taking seriously. The first is reassuring. If you have one bad night before an exam, perhaps because anxiety kept you awake despite doing everything right, you are not doomed. Your performance rests on a foundation built over weeks. Many students make a single poor night far worse by panicking about it, which is its own form of self-sabotage. The second is sobering. The idea that you can neglect sleep for most of the term and then catch up on learning in a heroic final week runs against the grain of how memory works. Finals week sleep still matters, especially for material you are learning or relearning that week, and it matters for your mood, judgment, and physical safety. But the strongest version of a sleep strategy for finals begins well before finals, and students who read this with several weeks to go should treat that as the most important sentence in the chapter. One reason sleep loss is so easy to accept is that it hides its own costs. In 1997 June Pilcher and Amy Walters randomly assigned college students either to sleep about eight hours or to stay awake for 24 hours before completing a critical-thinking test. The sleep-deprived students performed significantly worse. Yet when asked afterwards, they rated their own concentration and effort higher than the rested students did, and they estimated that they had performed better. The authors concluded that students are not aware of the extent to which sleep deprivation harms their cognitive work. This is the central trap of finals week. The tired brain is the very instrument you would need to notice that the tired brain is underperforming. Students who say they work fine on four hours of sleep are usually reporting how they feel, not how they perform, and those are different things. A small minority of people genuinely need less sleep than average, but they are rare, and they typically do not need caffeine to stay awake in the afternoon. Sleep loss also changes emotions in ways that feed directly into exam stress. In a companion study in Current Biology in 2007, Yoo, Walker, and colleagues showed participants upsetting images after a night of sleep deprivation or normal sleep. The amygdala, a region central to threat and emotional response, reacted roughly 60 percent more strongly in the sleep-deprived group, and its connection with the prefrontal regions that normally moderate it was weaker. A student running on little sleep is therefore more likely to experience an ambiguous practice exam score, a curt email from a professor, or a disagreement with a roommate as a disaster. Part of what feels like finals-week anxiety is, in plain physiological terms, finals-week sleep loss. The same research group later found, in a 2013 study in Nature Communications led by Stephanie Greer, that sleep deprivation reduced activity in frontal brain regions that evaluate food choices and increased the desire for high-calorie foods. This is one of the concrete mechanisms behind the cycle described in the introduction: poor sleep pushes toward poor fuel, and poor fuel makes the next stretch of studying harder. How much is enough The American Academy of Sleep Medicine and the Sleep Research Society issued a joint consensus statement in 2015, led by Nathaniel Watson, recommending that adults sleep seven or more hours per night on a regular basis for optimal health. The panel noted that young adults, and people recovering from sleep debt, may benefit from more than nine hours. For most college students, somewhere between seven and nine hours is the realistic target. Many students are nowhere near it. A large survey of 1,125 students at an urban Midwestern university, published by Hannah Lund and colleagues in 2010, found that more than 60 percent qualified as poor-quality sleepers on a standard questionnaire. Students overwhelmingly reported that emotional and academic stress disrupted their sleep, and tension and stress were the strongest predictors of poor sleep quality in the analysis. A 2014 review by Shelley Hershner and Ronald Chervin of sleepiness in college students described the same pattern across many campuses: irregular schedules, late bedtimes, heavy use of caffeine and other substances to manage alertness, and measurable costs to learning, mood, and safety. The implication is not that every student must hit eight hours every night during finals. It is that sleep is the resource most students are already short on when finals begin, and that the marginal hour of sleep is usually worth more to an exhausted student than the marginal hour of review. A finals-week sleep plan Knowing that sleep matters is easy. Getting it during a week when you have three exams and a paper is hard. The following practices come from the behavioural sleep medicine used to treat insomnia and from common-sense adaptation to student life. None of them requires perfection. Anchor your wake time. The single most useful rule is to get up at roughly the same time each day, including days with no morning exam. A fixed wake time stabilises your internal clock, builds the drive to sleep by evening, and makes it far easier to fall asleep at a reasonable hour. If your exams are at different times across the week, set your anchor for the earliest one and keep it all week rather than oscillating between 7 a.m. and noon. Get light in the morning and dim it at night. Bright light early in the day, ideally outdoors for a few minutes, helps set the body clock. In the last hour before bed, lower the lights. The evidence on screens specifically is more mixed than popular advice suggests, but the practical problem with phones at bedtime is less their light than their content: a message about someone else's study progress or a stressful group chat is a powerful alerting stimulus. Set a caffeine cutoff. The details are in the chapter on caffeine, but a controlled study by Christopher Drake and colleagues found that 400 milligrams of caffeine taken even six hours before bed still reduced sleep substantially. For most people, a cutoff around early afternoon is sensible. Protect a wind-down period. Stop studying twenty to thirty minutes before you intend to sleep. This is not wasted time. Moving directly from a problem set to bed leaves the mind running the problem set. Shower, prepare what you need for tomorrow, read something light, or do a few minutes of slow breathing. Write the worries down. A study by Michael Scullin and colleagues, published in 2018, asked healthy young adults to spend five minutes before bed writing either a to-do list of tasks they needed to complete over the next few days or a list of tasks they had already completed. In the sleep lab, the to-do list group fell asleep significantly faster, and the more specific their lists, the faster they fell asleep. For a student lying awake mentally rehearsing tomorrow's schedule, getting the list onto paper is a way of telling the brain that nothing will be forgotten if it switches off. Use the bed for sleep. If you study in bed all semester, your brain learns to associate the bed with alertness and effort. During finals, study at a desk, in the library, or anywhere that is not your pillow. If you have been lying awake for what feels like more than twenty minutes, get up, do something quiet and dull in low light, and return when you feel sleepy. This is a core element of cognitive behavioural therapy for insomnia, and it works better than lying there trying harder. Handle the environment. Residence halls in finals week are noisy at odd hours. Foam earplugs, a cheap sleep mask, and a frank conversation with a roommate about quiet hours are worth more than any supplement. If you share a room with someone on a very different schedule, agree on a few ground rules for the week: headphones, a desk lamp instead of the overhead light, and phones on silent after a set time. Be cautious with sleep aids. Alcohol may make you drowsy, but it fragments sleep in the second half of the night and reduces its quality; it is a poor sleep aid and a worse study aid. Over-the-counter antihistamine sleep aids often leave people groggy the next morning. Melatonin is sold as a supplement in the United States with variable actual doses; it is better understood as a signal that shifts the timing of the body clock than as a sedative. If you think you need medication to sleep, talk to a clinician at your health center rather than improvising during exam week. Ordinary finals-week sleep trouble fades when the stress fades. Some patterns deserve attention sooner. Loud snoring with gasping or pauses in breathing, overwhelming daytime sleepiness despite adequate time in bed, episodes of falling asleep in inappropriate situations, or insomnia that has persisted most nights for months are signs that something more than stress may be going on. Sleep disorders such as obstructive sleep apnea, narcolepsy, and chronic insomnia are treatable, and campus health services can refer you for assessment. Persistent sleep problems combined with low mood, hopelessness, or loss of interest in things you normally care about are a reason to contact counseling services, not something to wait out until the semester ends. The practical upshot of this chapter is a reframing. Most students treat sleep as a buffer: the flexible part of the schedule that absorbs overflow from everything else. The evidence suggests that, for anyone trying to learn, sleep belongs among the fixed commitments, alongside the exams themselves. When you schedule your finals week, put sleep in first and let the study blocks arrange themselves around it. The next chapter examines what happens when students do the opposite, and what to do if you find yourself at midnight with far more left to learn than hours left to learn it. Chapter 2: The All-Nighter, Examined Few rituals of college life are as romanticised as the all-nighter. It appears in films, in campus folklore, and in the stories students tell about the semester they nearly failed and then, fuelled by coffee and panic, pulled off a miracle. It carries a moral glow as well: staying up all night looks like commitment, and going to bed with work unfinished can feel like giving up. This chapter looks at the all-nighter without the glow. It asks how common it is, what actually happens to a brain awake for twenty-four hours, and, most practically, what a student should do at midnight when the material is not learned and the exam is at nine. How common, and what it predicts In a 2008 study published in Behavioral Sleep Medicine, Pamela Thacher surveyed 120 students at a four-year liberal arts college about what she called a single night of total sleep deprivation. Sixty percent reported having pulled at least one all-nighter since arriving at college. Using grade point averages obtained from the registrar rather than self-report, she found that students who pulled all-nighters had lower GPAs than those who did not. They also tended to have later bedtimes and a preference for evening activity, and there was a trend toward more symptoms of depression. A study like this cannot prove that the all-nighters caused the lower grades. Students who pull all-nighters may be more likely to procrastinate, to carry heavier workloads alongside jobs, or to be struggling in ways that affect both sleep and grades. But the finding sits alongside a large body of experimental work showing that sleep deprivation impairs memory formation, attention, and reasoning. There is no body of evidence showing that all-nighters improve outcomes. The folklore survives because people remember the time it worked and forget the times it did not, and because, as the last chapter showed, sleep-deprived people consistently overestimate their own performance. A night without sleep, hour by hour It helps to understand what the body is doing over the course of an all-nighter, because the experience is misleading in predictable ways. Two systems govern alertness. The first is sleep pressure, which builds steadily the longer you are awake, as the brain accumulates chemical signals of fatigue. The second is the circadian rhythm, the body's internal clock, which pushes alertness up and down on a roughly twenty-four-hour cycle regardless of how long you have been awake. For most young adults, the circadian drive for alertness is strong in the evening, which is one reason studying at 10 p.m. can feel productive. It falls sharply in the early morning hours, reaching its lowest point for many people somewhere around three to six in the morning. At that point sleep pressure is high and the clock has stopped fighting it. This is the stretch in which students find they have read the same page four times, and in which brief, involuntary lapses of a few seconds, known as microsleeps, become common. Most people do not notice microsleeps while they are happening. Then, as morning arrives, the circadian clock begins pushing alertness up again. Many students experience this as a second wind and take it as evidence that they have made it through. The second wind is real, but it is the clock talking, not recovery. Sleep pressure is still enormous, and it will reassert itself, often in the early afternoon, sometimes in the middle of an exam. The size of the impairment is not trivial. In a frequently cited experiment published in 2000 in Occupational and Environmental Medicine, Ann Williamson and Anne-Marie Feyer compared performance on a range of cognitive and motor tests during a period of up to twenty-eight hours without sleep with performance after measured doses of alcohol in the same participants. After seventeen to nineteen hours awake, performance on some tests was equivalent to or worse than performance at a blood alcohol concentration of 0.05 percent. After longer periods awake, performance on some measures reached levels equivalent to 0.1 percent, above the legal driving limit throughout the United States. A student who wakes at 8 a.m. on Wednesday and sits an exam at 9 a.m. on Thursday has been awake for twenty-five hours. Nobody would advise them to take the exam after several drinks. The comparison is imperfect, because alcohol and sleep loss affect the brain in different ways, but it gives a sense of scale. Students are not foolish for pulling all-nighters. The reasons are understandable, and naming them helps in resisting them. The first reason is simple arithmetic. If there are eight hours of material left and eight hours until the exam, the all-nighter looks like the only way to cover it. The flaw is that it treats an hour of study at 4 a.m. as equal to an hour at 4 p.m., and it treats covering material as equivalent to learning it. Both assumptions are false. Encoding at 4 a.m. after a long day is poor, and material skimmed without consolidation is unlikely to be available under exam conditions. The second reason is emotional. Going to bed with work undone can feel intolerable, a concession that you are not going to be ready. Staying up offers the comfort of still doing something. That comfort is real, but it is purchased at the price of the performance you are trying to protect. The third reason is social. On many campuses, surviving on little sleep during finals is a status marker, and students compare their sleep deficits the way athletes compare training loads. It helps to recognise this as a norm rather than a strategy. Nobody grading your exam will know or care how late you were up. The fourth reason is structural. Some students face genuine collisions: a paper due at midnight, an exam the next morning, and a shift at work in between. For them, the answer lies partly in planning several weeks ahead and partly in the triage approach described below, and partly in asking for help, including from professors, before the collision happens rather than after. Midnight decisions: triage, naps, and damage control Consider an illustrative scenario. It is midnight on Wednesday. A sophomore has an organic chemistry exam at nine on Thursday morning. She has reviewed about two-thirds of the material, and three chapters remain: one she understands reasonably well, one she has barely looked at, and one on a topic she knows the professor emphasised in lectures. She is tired, anxious, and has already had two coffees since dinner. She has several options, and Table 1 sets them out side by side. Table 1. Options for the night before an exam when preparation is incomplete. Option What it buys What it costs Best suited to Stay up all night Hours to skim remaining material Poor encoding, no consolidation, impaired recall and judgment at the exam Almost no one Study to about 2 a.m., sleep until exam morning Two focused hours plus a shortened night Some lost late-night sleep A student with specific gaps and a later exam Triage one hour, sleep a normal night, review briefly on waking Consolidation of what is known; clear head Some material left uncovered Most students most of the time Sleep now, wake early to study Full early-night deep sleep; fresh study session Early alarm; less time if oversleeping Morning types and those with afternoon exams The triage approach asks a different question from the arithmetic one. Instead of asking how to cover everything, it asks what will earn the most points given the brain she will have at nine tomorrow. Usually that means spending a final focused hour on the highest-yield material, which here is the emphasised topic and a quick retrieval check on the chapter she half-knows, and accepting that the unread chapter will be answered from partial knowledge and good reasoning. It means using practice questions rather than rereading, because a short burst of self-testing does more for recall than a long stretch of reading, as the next chapter explains. And then it means going to bed. There is a biological reason why even a shortened night is far better than none. Deep slow-wave sleep, the stage most closely linked to consolidation of fact-based memory, is concentrated in the first few hours of the night. A student who sleeps from 1 a.m. to 7 a.m. still gets much of the slow-wave sleep she would have had in a full night, along with a partial restoration of attention and emotional control. A student who does not sleep at all gets none of it. If she chooses the last option and sleeps first, she should set two alarms, put the phone out of reach so she has to get up to turn it off, and have her study materials laid out so the morning session starts immediately. That option suits people who genuinely work well early; it is less suited to anyone who struggles to wake. Naps are among the most underused tools in finals week. They can restore alertness during the afternoon slump, partially offset a short night, and give the brain an additional period of sleep that may help consolidate the morning's learning. Their main drawback is sleep inertia, the heavy grogginess that follows waking from deep sleep, which can last from several minutes to half an hour or more. A few practical rules make naps work well. A short nap of about ten to twenty minutes generally improves alertness without plunging you into deep sleep, so you wake relatively clear-headed. A longer nap of about ninety minutes allows a full sleep cycle and can be restorative after a very short night, but it risks more grogginess if you are woken from deep sleep, and it can make it harder to fall asleep at night. The awkward middle range, roughly forty to sixty minutes, is the most likely to leave you groggy. Naps are best taken in the early to mid-afternoon; a nap at 7 p.m. will compete with your night's sleep. One technique with some experimental support is the so-called caffeine nap. Caffeine takes roughly twenty minutes or more to take full effect. In a 1997 study of sleepy drivers in a simulator, Louise Reyner and James Horne found that a cup of coffee followed immediately by a short nap reduced sleepiness and driving incidents more effectively than either alone. The logic is that you fall asleep before the caffeine arrives and wake as it begins to work. It is a reasonable tool for a mid-afternoon slump, subject to the caffeine cutoff discussed later, and not a licence to replace a night's sleep. Sometimes the all-nighter has already happened. Perhaps the paper ran long, perhaps anxiety made sleep impossible. The goal then shifts to damage control. Eat a real breakfast with some protein and slow-release carbohydrate, and drink water. A moderate dose of caffeine, such as one cup of coffee, can help, but piling on more will add jitteriness without restoring the cognition that sleep would have. Get outside into daylight on the way to the exam. During the exam, read every question twice, underline key terms, and check your answers against the question actually asked, because sleep deprivation particularly increases careless errors and misreadings. If a question stumps you, move on and return to it, because a tired brain gets stuck more easily. Above all, do not drive. The comparisons with alcohol impairment are most serious on the road. The National Highway Traffic Safety Administration identifies drowsy driving as a significant cause of crashes, and young drivers and people who have missed sleep are among those at highest risk. The end of finals, when students often drive home after a string of short nights, is a particularly dangerous moment. If you are exhausted, get a ride, take public transport, or sleep first. After the exam, resist the temptation to sleep all afternoon. A nap of up to ninety minutes is fine. Then stay up until a reasonable bedtime and sleep a full night, so your schedule recovers rather than drifting into a pattern of afternoon sleep and night-time wakefulness that makes the next exam harder. Much of the research on sleep loss concerns memory, so students sometimes assume that writing a paper overnight is a different case. It is not, although the costs show up differently. Writing draws on the same prefrontal functions that sleep loss impairs: holding an argument in mind while choosing the next sentence, noticing when a paragraph has wandered, catching one's own errors. Students who write through the night often produce pages that felt brilliant at 4 a.m. and read as muddled at noon. The Pilcher and Walters finding applies here with particular force, because the paper will be judged by someone who is not tired. If a paper must be finished late, the most efficient division of labour is to draft while you still have some energy and leave editing for after sleep, even a short one. A draft finished at 1 a.m. and revised for an hour at 8 a.m. is usually better than a draft polished continuously until 6 a.m. For problem sets, a tired student is especially prone to the small errors in sign, units, or transcription that turn a correct method into a wrong answer. Checking work is worth more after sleep than more work before it. When the all-nighter becomes a pattern, and how to prevent it An occasional all-nighter is a lapse. A pattern of them, semester after semester, deserves a closer look, because it often points to something beyond poor planning. Some students have a strongly delayed body clock, a pattern sometimes diagnosed as delayed sleep-wake phase disorder, in which they genuinely cannot fall asleep until two or three in the morning and struggle to wake before late morning. For them, a 9 a.m. exam falls in what is effectively their biological night. Consistent morning light, a fixed wake time, and sometimes clinical treatment can shift the clock, and a sleep specialist or campus health service can help. Other students pull repeated all-nighters because chronic procrastination leaves them with no alternative. Procrastination is not simply laziness; it is often a way of avoiding the discomfort of a task that feels threatening or overwhelming, and it is common among people with anxiety, perfectionism, depression, or attention-deficit/hyperactivity disorder. If you recognise yourself in a cycle of avoiding work until panic makes it possible, that cycle is worth raising with a counselor or an academic coach. Many campuses offer structured help with time management and procrastination that is more effective than resolving, yet again, to start earlier next time. The most effective way to avoid the all-nighter is to recognise that it is usually the endpoint of decisions made days or weeks earlier. A student who writes out every exam, paper, and project deadline on a single calendar at the start of the last month of term can see collisions coming. A student who breaks each exam's material into daily portions can see early whether they are falling behind. A student who emails a professor a week before a deadline to explain a genuine conflict is far more likely to receive an extension than one who writes at 11 p.m. the night before. When the calendar shows an unavoidable crunch, the answer is to move work earlier, not later. That might mean finishing a paper draft during the weekend before exams, preparing summary sheets for each course before the first exam, or skipping some optional social events in the preceding weeks. None of this is glamorous. It rarely produces stories. It does produce students who arrive at exams having slept. The next chapter turns to the other half of this equation. If sleep is non-negotiable, the hours available for studying are finite, and the question becomes how to use them so that each one produces as much durable learning as possible. Chapter 3: Study Methods That Work With Your Biology If sleep is going to be protected, study time is finite, and the quality of each study hour becomes the whole game. This is where many students lose the most ground without realising it. They put in long hours with methods that feel productive and produce little durable learning, and then, sensing that the material is still not secure, they take the extra hours out of sleep. A better method is therefore not only an academic improvement. It is a health intervention, because it lets you reach the same level of preparation in less time and go to bed. This chapter is not a complete guide to learning science, which is a large field with excellent books of its own. It focuses on the handful of findings that matter most in the final weeks of a semester and on the specific ways those findings interact with sleep, stress, and fatigue. The illusion of fluency The most common study method among college students is rereading: going back over notes, slides, or textbook chapters, often with a highlighter in hand. It is easy to see why. Rereading feels good. On the second or third pass, the material looks familiar, the sentences flow, and the sense of recognition is easily mistaken for knowledge. Psychologists call this the illusion of fluency. The trouble is that exams rarely ask you to recognise a page. They ask you to produce an answer from memory, often in a form the page did not take. In 2013 a team led by John Dunlosky published an influential review in Psychological Science in the Public Interest that evaluated ten common learning techniques against the research literature. They rated each one as high, moderate, or low utility based on how well it worked across different learners, materials, and conditions. Table 2 summarises their ratings. Table 2. Utility of common study techniques as rated in a major review of the evidence. Technique What it involves Rated utility Practice testing Self-testing or practice questions on the material High Distributed practice Spreading study sessions out over time High Interleaved practice Mixing different types of problems within a session Moderate Elaborative interrogation Asking and answering why a fact is true Moderate Self-explanation Explaining how new information relates to what you know Moderate Summarization Writing summaries of material Low Highlighting and underlining Marking text while reading Low Rereading Reading material again Low Source: Dunlosky and colleagues, 2013. The review also rated keyword mnemonics and imagery for text as low utility. The low ratings do not mean that rereading or highlighting do nothing. A first read is necessary, and summarising can help some students with some material. They mean that, hour for hour, these techniques produce less learning than the alternatives, and they are particularly weak at producing the kind of flexible, durable recall that exams demand. Retrieval practice: studying by remembering The technique at the top of the list goes by several names: practice testing, the testing effect, retrieval practice. The core idea is that the act of pulling information out of memory is itself a powerful way of strengthening it. Every time you successfully recall something, you make it easier to recall again; every time you try and fail, you get precise feedback about what you do not yet know. A classic demonstration came in 2006, when Henry Roediger and Jeffrey Karpicke had students read short prose passages and then either reread them or take recall tests on them. When tested five minutes later, the rereading groups did slightly better. When tested two days or a week later, the pattern reversed, and the students who had practised retrieval remembered considerably more. Rereading produced a short-term boost that faded; retrieval produced learning that lasted. Just as importantly, students who had only reread predicted that they would do better than the students who had tested themselves. The feeling of fluency misled them. In practice, retrieval practice during finals can take many forms. Work old exams under timed conditions, without notes, and check afterward. Close the book at the end of a section and write down everything you can remember before looking again. Turn lecture headings into questions and answer them aloud. Use flashcards properly, which means attempting the answer before flipping the card, and keeping cards you miss in the deck until you get them right on a later pass. Explain a concept to a friend, or to an empty room, without looking at your notes. Draw a process or pathway from memory and compare it with the textbook version. Two findings make retrieval practice especially relevant to a book about health. The first concerns stress. Acute stress is well known to impair memory retrieval; this is part of why students blank on exams. In a 2016 study published in Science, Amy Smith, Victoria Floerke, and Ayanna Thomas had participants learn material either by restudying it or by practising retrieval. A day later, half of each group was put through a stressful procedure before being tested. The students who had learned by restudying showed the expected stress-related memory impairment. The students who had learned by retrieval practice did not; their memory held up under stress. The authors suggested that retrieval practice builds stronger memory representations that are less vulnerable to disruption. For a student who worries about freezing in the exam room, practising retrieval under mildly exam-like conditions is one of the best protections available. The second finding concerns anxiety itself. In a 2014 study of middle and high school students in classrooms that used frequent low-stakes quizzes, Pooja Agarwal and colleagues found that 72 percent of students reported that retrieval practice made them less nervous about tests and exams. The finding comes from younger students and from self-report, so it should not be over-read, but the mechanism is plausible: the unfamiliar is frightening, and a student who has already answered dozens of exam-style questions walks into the room having rehearsed the experience. Spacing, interleaving, and the rhythm of sleep The second high-utility technique is distributed practice, better known as spacing. Studying the same material across several separated sessions produces better long-term retention than the same amount of study massed into one session. A large meta-analysis by Nicholas Cepeda and colleagues in Psychological Bulletin in 2006 reviewed hundreds of comparisons and found the spacing effect to be robust. Later work by the same group suggested that the ideal gap between sessions grows with how long you need to remember the material: for an exam a week away, reviewing a topic on separate days works better than reviewing it several times on the same day. Spacing and sleep are natural partners. Every night between two study sessions is an opportunity for consolidation, so a topic studied on Monday and again on Wednesday has passed through two nights of sleep before the second session. The second session then functions partly as retrieval practice, because you are trying to recall material that has had time to fade a little. That slight difficulty is productive. The research literature often describes such challenges as desirable difficulties: conditions that make learning feel harder in the moment and produce better results later. Cramming is the opposite of spacing, and it has an unfortunate property. It really does work for a few hours, which is why it survives. A student who crams the night before can often recall a surprising amount the next morning. But crammed material fades quickly, which matters in any course with a cumulative final, in any sequence of courses where next semester builds on this one, and in any career that requires the knowledge. Cramming also collides directly with sleep, since it usually happens late at night. Interleaving is a related idea. Instead of practising one type of problem repeatedly before moving to the next, you mix problem types within a session. In a 2007 study by Doug Rohrer and Kelli Taylor, students who practised mixed sets of mathematics problems performed worse during practice but substantially better on a later test than students who practised in blocks. The reason is that exams do not tell you which technique a question calls for. Blocked practice lets you skip that step; interleaved practice forces you to identify the type of problem, which is exactly what the exam requires. As with retrieval practice, students typically believe blocked practice is more effective because it feels smoother. Building study days and weeks around your body Putting these principles together with the sleep guidance from the previous chapters produces a study day with a recognisable shape. Table 3 shows one version for a student in the middle of finals week with an exam two days away and another four days away. The times are illustrative; the structure is what matters. Table 3. An illustrative finals-week study day that protects sleep. Time Activity Why 7:30 a.m. Wake at anchor time, daylight, breakfast Stabilises body clock, fuels morning 8:30-10:30 Timed practice questions, nearer exam Hardest work while fresh; retrieval 10:30-11:00 Walk, snack, water Break restores attention 11:00-12:30 Interleaved problems, later exam Spacing across courses 12:30-1:30 Lunch away from desk Regular meal, mental rest 1:30-3:30 Review errors, explain concepts aloud Targeted repair of weak points 3:30-4:15 Exercise or short nap Offsets afternoon dip 4:15-6:00 Flashcards and mixed recall, both courses Spaced retrieval 6:00-7:00 Dinner, social contact Fuel and emotional reset 7:00-9:30 Lighter review, organise notes Lower-demand tasks for evening 9:30-10:30 Wind-down, to-do list Prepares for sleep 10:30 p.m. Lights out About eight hours in bed Several features of this day deserve comment. The hardest and most exam-like work comes first, when attention is strongest and before the day's fatigue accumulates. Sessions are broken by short breaks, which help sustain attention across a long day; many students find working in focused blocks of forty-five to ninety minutes with brief breaks between them more sustainable than marathon sessions. Meals are taken away from the desk, which gives the mind a genuine rest and makes it likelier that the student actually eats. And the day ends with lighter review rather than new, difficult material, so that the student is not trying to fall asleep in the middle of a problem. The afternoon slot for exercise or a nap is not a luxury. The early to mid-afternoon dip in alertness is a normal feature of the circadian rhythm, and fighting it with a third coffee tends to cost sleep later. A single session of moderate aerobic exercise has been shown in many studies to produce small, short-term improvements in attention and executive function; a 2012 meta-analysis by Yu-Kai Chang and colleagues in Brain Research found the overall effect of acute exercise on cognitive performance to be small but positive, with benefits visible after the exercise ended. Exercise also reduces tension and tends to improve sleep. A brisk twenty-minute walk counts. Students who cannot exercise can nap instead, following the rules in the previous chapter. The evening includes deliberate social contact. Finals week pushes many students into isolation, and isolation makes stress worse. A meal with friends, even a short one, provides perspective and a reminder that the exams are one part of life, not all of it. Retrieval practice, spacing, and interleaving apply to nearly every subject, but they look different depending on what the exam will ask. For problem-based exams in mathematics, physics, chemistry, economics, or engineering, the core activity should be solving problems without looking at worked solutions first. Many students study these courses by reading through solved examples and nodding along, which is rereading in disguise. The more demanding and more effective approach is to cover the solution, attempt the problem, get stuck, struggle for a few minutes, and only then check. Mixed problem sets drawn from across the course, rather than chapter by chapter, prepare you for the real task of recognising which method a question requires. For essay exams in history, literature, philosophy, or the social sciences, retrieval practice means writing. Predict likely questions from the syllabus and lecture themes, then outline answers from memory under time pressure. Practise stating an argument in two sentences and listing the three pieces of evidence you would use. Elaborative interrogation, which Dunlosky's review rated as moderately useful, fits these subjects especially well: for each key claim, ask why it is true, what would count against it, and how it connects to other parts of the course. For exams heavy with terminology and facts, as in introductory biology, anatomy, pharmacology, or language courses, flashcards used with genuine recall attempts and spaced over days are hard to beat. The mistake to avoid is making flashcards so elaborate that the making becomes the studying. Card creation is closer to summarising than to retrieval. For open-book exams, students often prepare less, reasoning that the answers will be at hand. This is usually a mistake. Open-book exams tend to test application rather than recall, and there is rarely time to look everything up. Know the material well enough to answer most questions without the book, and organise the book or notes so that the few things you do need to check can be found in seconds. Study groups can support all of these approaches when they are built around retrieval: members quizzing each other, explaining concepts without notes, and working problems together. They become unproductive when they drift into collective rereading or, more commonly, into collective anxiety. A group that spends an hour comparing how far behind everyone is has raised its stress without raising its preparation. A useful rule is to arrive having already studied alone, spend the session testing each other, and leave on time. One environmental factor deserves mention. Phones are powerful interrupters, and every notification that pulls your attention costs time to recover the thread of what you were doing. Research on whether the mere presence of a phone reduces cognitive capacity has produced mixed results, but there is no controversy about the cost of actually checking it. Putting the phone in another room or a bag during focused blocks, and checking it during breaks, is one of the simplest ways to increase the learning per hour that this whole chapter is about. Finally, it helps to plan the last two weeks with these principles in mind rather than improvising day by day. Begin by listing every exam, paper, and project with its date and weight. For each exam, estimate what needs to be covered and break it into portions. Then schedule each portion at least twice, on separate days, with the second pass built around retrieval rather than rereading. Put the earliest exam's material in the earliest slots, but include some spaced review of later exams from the start, so that nothing is being encountered for the first time the night before. Build in slack. Plans that fill every hour always fail, and when they fail the overflow usually comes out of sleep. Leave at least one or two unscheduled blocks every day, and one lighter day in the middle of the fortnight if you can. Use practice tests as a diagnostic. After a timed practice exam, do not simply note the score and move on. Sort your errors into those caused by not knowing the material, by misunderstanding the question, and by careless mistakes. Each type has a different remedy: the first needs study, the second needs practice reading questions carefully, and the third is often a sign of fatigue or rushing. Most importantly, measure progress by what you can produce from memory, not by pages covered or hours logged. A student who has done three timed practice exams and reviewed every error is better prepared than one who has reread the textbook twice, even if the second student has spent more hours at a desk. The second student is also far more likely to be tempted to spend the night before the exam trying to close a gap they cannot see clearly. The first student knows where they stand, and that knowledge, as much as any breathing technique, is what lets a person sleep before an exam. Chapter 4: Brain Food Without the Hype Every finals season brings a wave of advice about brain food. Blueberries for memory, walnuts because they look like brains, salmon for omega-3s, dark chocolate for focus, and a small pharmacy of supplements promising clarity, energy, and recall. Much of this advice is harmless. Some of it is expensive. Almost none of it will change your grade. This chapter takes a different approach. It starts from what the brain actually needs from food, separates the claims with reasonable evidence from those without, and arrives at a conclusion that is less exciting than a superfood list but considerably more useful: during exam week, the most important nutritional goals are to avoid being hungry, avoid being dehydrated, avoid large swings in energy, and avoid feeling physically unwell. Those goals are met by ordinary, regular, reasonably balanced eating and drinking. The rest is detail. What the brain needs from food The brain is an expensive organ. It makes up around 2 percent of adult body weight but accounts for roughly a fifth of the body's energy use at rest. Its preferred fuel is glucose, and it has very limited ability to store it, so it depends on a steady supply delivered by the blood. From this fact, a popular but misleading conclusion is often drawn: that the brain needs frequent doses of sugar to work well, and that a candy bar before an exam is a sensible performance aid. In reality, the body regulates blood glucose tightly. Between meals, the liver releases stored glucose to keep the supply steady, and a healthy person who has eaten reasonably in the previous several hours is not running the brain on empty. What does disrupt attention is genuine hunger, which is distracting in its own right, and the rise and fall in energy and mood that some people experience after large amounts of rapidly digested carbohydrate on an empty stomach. The evidence on the so-called sugar rush is instructive. In 2019, Konstantinos Mantantzis and colleagues published a meta-analysis in Neuroscience and Biobehavioral Reviews that combined 31 studies with 1,259 participants examining how carbohydrate consumption affects mood. They found no positive effect of carbohydrates on any aspect of mood at any time point. Within the first hour after consumption, carbohydrates were associated with more fatigue and less alertness compared with placebo. The sugar rush, in other words, is largely a myth, and the sugar crash, or at least a modest dip in alertness, has somewhat better support. Nobody needs to fear a cookie. But the idea that sugar is a study aid does not hold up. Beyond glucose, the brain depends on a wide range of nutrients, including essential fatty acids, iron, B vitamins, and others, for its long-term structure and function. Deficiencies matter. Iron deficiency, for instance, is relatively common in young women and can cause fatigue and poor concentration; anyone with persistent exhaustion that does not improve with sleep should ask a clinician about checking for it. But the relevant timescale for building a well-nourished brain is months and years, not the four days before an exam. A student who eats a varied diet most of the time already has what they need. A student who does not will not fix it with a bag of walnuts during finals. Hydration and regular meals If there is one nutritional habit with a clear short-term payoff during exam week, it is staying adequately hydrated. A 2012 review by Ana Adan in the Journal of the American College of Nutrition concluded that dehydration equivalent to about 2 percent of body mass can impair performance on tasks requiring attention, psychomotor skill, and immediate memory, and can worsen mood. Studies of milder dehydration produce less consistent results, and the effects vary between individuals. But finals week creates easy conditions for falling behind on fluids: long hours in heated libraries, a diet heavy in coffee, and a tendency to forget to drink when absorbed in work. The practical rules are simple. Keep a water bottle on the desk and drink from it regularly. Have a glass of water with each meal and each coffee. Use thirst and urine colour as guides; pale yellow is the target. There is no need to force large volumes; more is not better once you are adequately hydrated, and drinking excessive amounts of water can itself be dangerous. Most exam rooms permit a clear water bottle; check the rules and bring one. Regular meals matter for a related reason. Hunger is a powerful competitor for attention, and irritability when hungry is familiar enough to have earned its own slang. Students who skip breakfast, work through lunch, and eat a large meal late at night are setting up exactly the pattern that makes concentration hardest during the day and sleep hardest at night. The research on breakfast specifically is more nuanced than its reputation. Studies in children and adolescents suggest that eating breakfast is associated with better attention and memory later in the morning, though the evidence is not uniform. In adults, the findings are more mixed, and there is no strong basis for insisting that everyone must eat first thing on waking. What matters more is not facing a morning exam on an empty stomach after a long overnight fast. If you are not a breakfast eater, a small, easily digested meal an hour or two before a morning exam, such as yoghurt with fruit, toast with peanut butter, or a banana and a glass of milk, is a reasonable compromise. The composition of meals matters mainly through two practical effects. The first is satiety. A meal that includes protein, some fat, and fibre-rich carbohydrate keeps you full for longer than a meal of refined carbohydrate alone, which means fewer mid-afternoon distractions and less reliance on vending machines. The second is comfort. Very large, heavy meals are commonly followed by a period of sluggishness, and very greasy or unfamiliar foods can cause digestive trouble at exactly the wrong moment. The night before and the morning of an exam are not the time to experiment with a new spicy dish or a giant fried meal. A useful default is the plate model promoted in different forms by the United States Department of Agriculture's MyPlate guidance and by the Harvard T.H. Chan School of Public Health's Healthy Eating Plate: roughly half the plate vegetables and fruit, a quarter whole grains or other starchy foods, and a quarter protein, with water to drink. It is not a precise prescription. It is a picture that can be approximated in almost any dining hall and, with some adaptation, in a dorm room, as the chapter on meal plans will show. Checking the claims, and snacking sensibly Students encounter a steady stream of claims that particular foods or supplements will sharpen the mind. Table 4 summarises what the evidence broadly supports for the most common ones. The verdicts are necessarily brief, and the underlying literature in several cases is large and contested; the aim is to indicate where your money and attention are best spent during an exam period. Table 4. Common brain-food claims and what the evidence supports for exam week. Claim What the evidence broadly shows Practical verdict Sugar gives an energy boost Meta-analysis found no mood benefit and more fatigue within an hour Not a study aid Fish oil sharpens memory Diet matters long term; short-term benefit in healthy young adults not established Eat fish if you like it; skip pills for finals Blueberries boost recall Small studies of berry flavonoids show inconsistent short-term effects Good food; not a performance tool Dark chocolate improves focus Cocoa flavanol studies small and mixed; contains a little caffeine Fine as a treat Nootropic supplements Largely untested blends; quality and contents not verified before sale Avoid Water improves performance Dehydration of about 2 percent of body mass impairs attention and mood Worth doing Regular balanced meals Prevent hunger, energy dips, and distraction Worth doing Two items on the list deserve extra comment. The first is omega-3 fatty acids, particularly those found in oily fish. They are genuinely important components of brain tissue, and diets that include fish regularly are associated with good long-term health. It does not follow that a capsule taken during finals week will improve exam performance in a healthy young adult, and trials of supplementation in healthy young people have generally not produced reliable short-term cognitive benefits. If you like fish, eat it. There is no reason to buy supplements for an exam. The second is the category often marketed as nootropics: capsules and powders combining ingredients such as herbal extracts, amino acids, vitamins, and often caffeine, with names that promise focus or limitless potential. In the United States, dietary supplements do not require approval from the Food and Drug Administration before they are sold, and manufacturers are responsible for their own claims of safety and labelling. The FDA has repeatedly warned about supplements that contain undeclared drugs or unsafe amounts of stimulants. A product that makes you feel sharper may be doing so because it contains a substantial dose of caffeine, or something less benign, that is not clearly listed. For a student during finals, the combination of unknown stimulants, existing caffeine intake, sleep loss, and anxiety is a poor bet. Long study days usually involve snacking, and there is nothing wrong with that. A snack between meals can keep hunger from breaking concentration, especially when meals are five or six hours apart. The question is what kind of snack does that job well. The most useful snacks combine something that satisfies quickly with something that lasts. An apple with a handful of peanuts, a banana with a cheese stick, whole-grain crackers with hummus, a small pot of yoghurt with a spoonful of granola, a hard-boiled egg and a piece of fruit, or a handful of trail mix all fit the pattern. They are portable, cheap when bought in bulk, and require no kitchen. The least useful snacks are the ones eaten continuously and without attention: a family-sized bag of chips or a sleeve of cookies consumed one handful at a time over a two-hour study session. The problem is not moral. It is that grazing without noticing tends to leave people both overfull and unsatisfied, and it turns eating into a form of procrastination. A simple habit helps. When you want a snack, put a portion on a plate or in a bowl, step away from the desk to eat it, and then return to work. The small ritual turns the snack into a genuine break, which improves attention on return, and it makes it far less likely that a whole package disappears unnoticed. Stress eating deserves a sympathetic word. Many people eat more under stress, particularly foods high in sugar and fat, and many others lose their appetite entirely. Both are normal responses. Neither is a failure of character. The stress eater gains little from guilt, which adds to the stress, and more from making sure that regular meals are in place, so that eating is less likely to become the main way of coping. It also helps to have other short, reliable ways of changing state, such as a five-minute walk, a call to a friend, or the breathing techniques described in Chapter 7, available as alternatives when the urge to eat is really an urge to feel better. The student who loses their appetite under stress faces the opposite problem. Skipping meals because nothing seems appealing can lead to lightheadedness, poor concentration, and a shaky feeling that is easily mistaken for anxiety and then makes the anxiety worse. For these students, small, frequent, bland meals are often easier than large ones: toast, soup, rice, bananas, yoghurt, smoothies, or a glass of milk. Eating on a schedule, rather than waiting until hunger announces itself, prevents the problem from building. Special circumstances, and a word on alcohol General advice has to bend around individual circumstances, and some students need to plan more carefully than others. Students with diabetes should be especially attentive during exam periods. Stress, disrupted sleep, and irregular meals can all affect blood glucose, and both high and low readings can impair concentration. Plan meals and snacks around exam times, carry fast-acting glucose into the exam, and contact your campus disability or accessibility office well in advance if you may need accommodations such as permission to eat or check glucose during the exam. Students with a history of disordered eating face a particular risk. The pressure, loss of routine, and sense of lost control that come with finals can intensify restrictive eating, bingeing, or compensatory behaviours. If you notice yourself skipping meals as a way of coping, or eating in ways that feel out of control, it is worth reaching out to your counselor, your treatment team, or campus counseling services before exams begin rather than after. Regular, planned meals are part of treatment for many people, and exam week is a time to protect that structure. Students observing a religious fast during an exam period, as can happen when Ramadan falls during spring examinations, can plan around it: prioritising hydration and a substantial, slow-digesting pre-dawn meal, scheduling the most demanding study for the hours when they feel most alert, and, if needed, discussing exam timing with their institution. Many universities have established procedures for this. Student athletes, and anyone doing heavy training during exams, have higher energy needs than their sedentary peers, and under-fuelling combined with sleep loss is a common route to both poor performance and injury. Athletic departments often have sports dietitians available who can help plan exam-week eating. It is worth stating plainly that alcohol has no place in an exam-week plan. It impairs the formation of new memories, fragments sleep in the second half of the night, reduces sleep quality, and can leave a lingering effect on attention the next day even after it has left the bloodstream. A drink to unwind in the evening may feel calming, but it undermines the very sleep that consolidates the day's studying. The weeks before and during exams are, for most students, the time of the semester when drinking is most clearly working against them. The subject returns in the final chapter, because the end of exams is often the time of heaviest drinking, with its own set of risks. Brain food, understood honestly, is not a list of special ingredients. It is the unremarkable practice of eating regularly, drinking enough water, not running on sugar, avoiding the gamble of untested supplements, and choosing meals that keep you full and comfortable. That practice will not raise your intelligence. It will prevent hunger, dehydration, and discomfort from lowering it, which during exam week is what matters. The next chapter turns to the substance that most students use far more deliberately than any food: caffeine, along with its much riskier relatives. Chapter 5: Caffeine, Energy Drinks, and Study Drugs Caffeine is the most widely used psychoactive substance in the world, and during finals week it becomes, for many students, a kind of scaffolding for the whole day. Used thoughtfully, it is a genuinely useful tool: it increases alertness, reduces the perception of fatigue, and can improve performance on simple attention tasks, especially in people who are tired. Used carelessly, it becomes one of the main forces driving the cycle this book describes, trading tonight's sleep for this afternoon's focus and adding physical symptoms of anxiety to the real thing. This chapter explains how caffeine works, how much is reasonable, how to time it, and why energy drinks and caffeine products deserve more caution than coffee. It then addresses a harder subject: the use of prescription stimulants without a prescription, a practice common enough on many campuses that most students will be offered pills at some point, and one whose promised benefits are largely illusory while its risks are real. How caffeine works and how long it lasts Throughout the waking day, a chemical called adenosine builds up in the brain. Adenosine is one of the main signals of sleep pressure: the more of it accumulates, the sleepier you feel. Caffeine works mainly by blocking adenosine receptors, so the brain temporarily stops registering the sleepiness signal. That is why caffeine feels like energy. It does not provide any. It masks fatigue that is still there, and when the caffeine wears off, the accumulated sleep pressure reasserts itself. Caffeine is absorbed within about an hour of drinking it, and its effects typically build over twenty to sixty minutes. It is cleared slowly. In healthy adults, the time it takes for the body to eliminate half of a dose, known as the half-life, is commonly several hours, and it varies widely between people depending on genetics, hormonal contraceptive use, pregnancy, smoking, liver function, and some medications. A practical consequence is that a large coffee at four in the afternoon may still leave a meaningful amount of caffeine in circulation at midnight. The clearest demonstration of what that means for sleep comes from the study by Christopher Drake, Timothy Roehrs, John Shambroom, and Thomas Roth, published in 2013 in the Journal of Clinical Sleep Medicine. Participants took 400 milligrams of caffeine, or a placebo, at bedtime, three hours before bedtime, or six hours before bedtime, and their sleep was measured at home with both diaries and a portable sleep monitor. Caffeine disrupted sleep at all three time points. Even when taken six hours before bed, it reduced objectively measured sleep by more than an hour, and participants did not necessarily perceive the full extent of the disruption. The authors concluded that the data support avoiding substantial caffeine for at least six hours before bedtime. For a student aiming to sleep at 11 p.m., that means finishing the last significant dose by about 5 p.m., and earlier is better for people who are sensitive. How much is reasonable, and how to use it The United States Food and Drug Administration has cited 400 milligrams a day, roughly four or five cups of brewed coffee, as an amount not generally associated with dangerous, negative effects in healthy adults. This is a general guide, not a target, and individual sensitivity varies enormously. People who are pregnant are advised to limit caffeine to lower amounts, commonly 200 milligrams a day. People with heart rhythm problems, high blood pressure, anxiety disorders, or panic attacks, and people taking certain medications, including stimulant medications for ADHD, may need to keep intake well below 400 milligrams and should ask their clinician. The difficulty for students is that caffeine content is often hidden or surprising. Table 5 gives approximate amounts for common sources. The figures are typical values, and actual content varies considerably by brand, brewing method, and serving size. Table 5. Approximate caffeine content of common drinks and products. Source Typical serving Approximate caffeine (mg) Brewed coffee 8 oz (240 ml) 95 Large coffeehouse brewed coffee 16 oz (475 ml) 200-330 Espresso 1 oz (30 ml) shot 65 Black tea 8 oz (240 ml) 45 Green tea 8 oz (240 ml) 30 Cola 12 oz (355 ml) 30-45 Energy drink, standard can 8.4 oz (250 ml) 80 Energy drink, large can 16 oz (475 ml) 150-300 Caffeine tablet 1 tablet 100-200 Sources: Mayo Clinic and FDA consumer guidance; product labels. Values are approximate. The table reveals how easily a student can pass 400 milligrams without feeling that they have overdone it. A large coffee in the morning, a large energy drink in the afternoon, and a cola at dinner could together approach 600 milligrams. Add a pre-workout powder or a "focus" supplement with undeclared or unclear caffeine content, and the total becomes a guess. Too much caffeine produces a recognisable set of symptoms: jitteriness, restlessness, rapid heartbeat or palpitations, stomach upset, headache, irritability, and insomnia. In people prone to anxiety, it can produce sensations nearly indistinguishable from an anxiety attack, including a pounding heart, shakiness, and a feeling of dread. A student who drinks a large energy drink before an exam and then feels their heart racing may reasonably conclude that they are panicking about the exam. Often, at least part of what they are feeling is the drink. For students who use caffeine, the goal during finals is to get the benefit without the costs. A few principles help. Keep your usual pattern steady. Regular caffeine users develop tolerance, and stopping abruptly produces withdrawal symptoms, including headache, fatigue, low mood, and difficulty concentrating, that typically begin within a day and can last several days. Finals week is not the time to quit. It is also not the time to double your intake, because tolerance means extra doses bring diminishing returns and more side effects. Dose deliberately rather than continuously. A cup of coffee before the hardest study block of the morning, and perhaps one in the early afternoon, provides benefit at the moments you need it. Sipping from a bottomless cup all day, or keeping an energy drink open on the desk for hours, spreads the caffeine thinly across the day and pushes it into the evening. Respect the cutoff. Given the Drake findings, set a time after which you switch to water or decaffeinated drinks. For most students, mid-afternoon is a sensible default. On exam day, stick to normal. An exam is a situation in which you will already be physiologically aroused, and adding a larger-than-usual dose of caffeine increases the risk of jitteriness, a racing heart, and a need for the bathroom in the middle of the exam. Have your usual amount, at your usual time, with food. Remember what caffeine cannot do. It can help a moderately tired person stay alert. It cannot restore the memory consolidation that sleep provides, and in someone who is severely sleep-deprived, it improves alertness less than it seems to while doing nothing for the quality of reasoning. Energy drinks and concentrated caffeine Energy drinks warrant particular caution for several reasons. Their caffeine content is often high and variable, serving sizes encourage drinking large amounts quickly, and many contain additional ingredients, such as guarana, which is itself a source of caffeine, along with large amounts of sugar. They are also very easy to drink fast, especially cold, which delivers a large dose in a short time. Federal data suggest the risks are not hypothetical. A 2013 report from the Substance Abuse and Mental Health Services Administration's Drug Abuse Warning Network estimated that emergency department visits involving energy drinks roughly doubled between 2007 and 2011, to more than 20,000 in 2011. Many of those visits involved energy drinks combined with alcohol or other drugs. Mixing caffeine with alcohol deserves specific warning. Caffeine can mask the sedative cues of alcohol, so people feel less intoxicated than they are and may drink more or make riskier decisions, such as driving, while remaining just as impaired. In 2010 the FDA took action against manufacturers of premixed caffeinated alcoholic beverages for this reason. The combination is especially common at end-of-semester celebrations. Concentrated caffeine products carry the greatest risk of all. Pure powdered caffeine and highly concentrated liquid caffeine can contain a dangerous dose in a very small volume. The FDA has warned that a single teaspoon of pure powdered caffeine can contain roughly as much caffeine as 28 cups of coffee, an amount that can be toxic, and deaths have occurred. The agency has taken regulatory action against bulk sales of these products to consumers. A student who sees such a product advertised as a cheap study aid should avoid it entirely. Warning signs of caffeine toxicity include a rapid or irregular heartbeat, chest pain, severe agitation or confusion, vomiting, tremors, and seizures. These are reasons to seek emergency medical care, not to wait it out. Study drugs: the promise, the evidence, and the risks On many campuses, finals week brings a quieter market alongside the energy drinks: prescription stimulants such as mixed amphetamine salts, sold under the brand name Adderall, and methylphenidate, sold as Ritalin and other brands. These medications are prescribed for attention-deficit/hyperactivity disorder, and for people with ADHD they can be effective treatments when used under medical supervision. Taken without a prescription, they are often called study drugs, and they are widely believed to improve focus, stamina, and grades. How common is this? In a 2015 meta-analysis of studies of college students, Kari Benson and colleagues estimated the rate of stimulant misuse at about 17 percent. They found that students misuse these drugs primarily for academic reasons and most often obtain them from peers who have prescriptions. Estimates vary considerably between studies and campuses, but the overall picture is clear: in a typical dormitory, some students are taking them, and most will know someone who could supply them. The evidence on whether the drugs deliver what students expect is considerably less encouraging than the rumour mill suggests. In 2013, Irena Ilieva, Joseph Boland, and Martha Farah at the University of Pennsylvania published a carefully designed, double-blind, placebo-controlled trial in which healthy young adults took either mixed amphetamine salts or a placebo and then completed a battery of thirteen measures of cognitive ability, including memory, reasoning, creativity, and tests resembling academic work. The drug did not enhance any of these abilities for participants in general. Yet participants believed that their performance was more enhanced when they had taken the active drug than when they had taken the placebo. The drug made them feel sharper without making them perform better. Two years later, Ilieva, Cayce Hook, and Farah published a meta-analysis in the Journal of Cognitive Neuroscience that combined 48 studies with more than 1,400 participants. They found small effects of stimulants on some aspects of cognition in healthy people, including inhibitory control and short-term memory, and evidence of publication bias that likely inflated the effects on some measures. Their overall conclusion was that the effect of these drugs on healthy cognition is probably modest, and that healthy users may be relying on them more for energy and motivation than for any genuine cognitive boost. The most directly relevant evidence for students comes from a study of grades. In 2017, Amelia Arria and colleagues followed 898 undergraduates without an ADHD diagnosis over two years and compared the changes in their grade point averages according to whether they had used prescription stimulants nonmedically. Students who abstained showed a small improvement in GPA over the period. Those who used the drugs showed no significant gains and no detectable advantage over their peers. The authors concluded that the promise of academic benefit from these drugs is likely illusory. There is an obvious irony here. The feeling that a stimulant is helping is strong enough to keep students using them, and that feeling is exactly what the controlled research suggests is misleading. It resembles the Pilcher and Walters finding about sleep deprivation: a subjective sense of performance that does not match the objective one. Against modest or absent benefits stand risks that are real and, in some cases, severe. The medical risks of prescription stimulants taken without supervision include insomnia, loss of appetite, anxiety and agitation, elevated heart rate and blood pressure, and, in susceptible people or at high doses, heart rhythm problems and, rarely, psychotic symptoms. Someone with an undiagnosed heart condition does not know they have one; this is one of the reasons these drugs are prescribed only after a medical assessment. The sleep disruption alone undermines the purpose for which students take them, since a stimulant taken to fuel an evening of study will usually cost that night's sleep and therefore the consolidation of whatever was studied. Combining stimulants with large amounts of caffeine or with alcohol compounds the risks. Stimulants also carry a potential for misuse and dependence, which is why they are tightly controlled. The legal risks are also significant. In the United States, these medications are Schedule II controlled substances. Possessing them without a prescription, and especially giving or selling them to others, is a criminal offence, and students who share their own prescriptions can face both legal and university disciplinary consequences. The most serious and most recent risk concerns what is actually in the pills. The Drug Enforcement Administration has repeatedly warned that counterfeit pills made to look like legitimate prescription medications, including Adderall, are widespread in the illicit market, and that these fake pills may contain methamphetamine or fentanyl. A pill obtained from someone who got it from someone else cannot be assumed to be what it appears to be. Students have died after taking pills they believed were prescription stimulants. If you or someone you know has taken a pill of uncertain origin and shows signs of overdose, such as extreme drowsiness or unresponsiveness, slow or stopped breathing, or blue lips, call emergency services immediately and give naloxone if it is available. Many campuses now make naloxone freely available, and many states have laws offering some legal protection to people who call for help during an overdose. Students who have been prescribed stimulant medication for ADHD have a different set of concerns during finals. Take your medication as prescribed, and do not increase the dose or add extra doses to cover late-night studying without talking to your prescriber, since this can disrupt sleep and increase side effects. Be cautious with caffeine, which adds to the stimulant effect. Keep your medication secure; finals week is when requests from friends are most common, and saying no protects both you and them. It can help to have a simple, rehearsed answer ready, such as saying that you need every dose and your prescriber counts them. If you suspect you have undiagnosed ADHD because you have struggled with attention and organisation for years, the right route is an assessment through your campus health or counseling service or a clinician, not a borrowed pill. If you are diagnosed, treatment may include medication, but also accommodations such as extended exam time or a reduced-distraction testing room, which campus disability offices can arrange when documentation is in place. Those arrangements take time, so the best moment to start is well before the next finals period. The common thread through this chapter is that the substances students reach for during finals week promise to buy time and focus, and they deliver less than they promise while charging more than they advertise. A moderate, well-timed dose of caffeine is a legitimate tool. Beyond that, the returns fall quickly and the costs, first to sleep and then to health, rise. What lies at the other end of that trade is not a better exam performance but a more wired, more anxious, and more exhausted student, which is a poor way to arrive at the chapters on stress that follow. Chapter 6: Eating Well From a Dining Hall or a Dorm Room The previous two chapters argued that the nutritional goals for exam week are modest: eat regularly, stay hydrated, keep caffeine in bounds, and choose meals that keep you full and comfortable. This chapter turns those goals into food. It is written for the two situations most college students are actually in: those who eat mainly in a campus dining hall on a meal plan, and those who are feeding themselves on a limited budget with a microwave, a mini-fridge, and perhaps occasional access to a shared kitchen. It ends with a frank discussion of what to do when money for food is genuinely short, because for a substantial number of students it is. The meal plans here are illustrations, not prescriptions. Adjust quantities to your appetite and activity, substitute foods you like and can afford, and follow any dietary requirements set by your clinician, religion, allergies, or ethics. The point is to show that eating well during finals week does not require special foods, much money, or much time. Working the dining hall A dining hall is, in principle, an excellent place to eat during exams. Someone else shops, cooks, and cleans. There is usually a salad bar, a protein station, grains, fruit, and something warm. The difficulty is not availability but choice and timing. Faced with a pizza station, a dessert bar, and a soft-serve machine at the end of a long study day, many students eat the same few comfort foods at every meal. Others stop going altogether during finals because the walk and the social noise feel like a distraction, and they end up living on snacks. A few habits make the dining hall work for you. Use the plate model as a quick check. Before sitting down, look at the plate. Is roughly half of it vegetables and fruit? Is there a clear source of protein, such as chicken, fish, eggs, beans, tofu, lentils, yoghurt, or cheese? Is there a starch that will keep you full, ideally a whole grain or potatoes rather than only white bread? This takes five seconds and corrects most imbalances without requiring you to give up anything you enjoy. Build from the salad bar and grill rather than only the entrée line. Salad bars in many dining halls hold far more than lettuce: beans, chickpeas, hard-boiled eggs, grains, seeds, cheese, and roasted vegetables. They are a fast way to add protein and fibre to a meal whose main dish is light on both. Take something for later. Many dining halls allow students to take a piece of fruit or a small item when they leave. Check your campus's policy. A banana or apple in your bag at the library is the difference between a planned snack and a vending machine. Eat at regular times. Dining hall hours sometimes change during exam periods, with some halls extending late-night service and others reducing hours. Check the schedule at the start of the week and plan your meals around it, so you are not discovering at 8:30 p.m. that the nearest hall closed at eight. Treat meals as breaks. Leave the laptop in your bag. Eat with a friend if you can. Twenty or thirty minutes away from the material is a genuine rest for attention, and brief social contact is one of the more reliable buffers against stress. Watch the late-night trap. Late-night dining options and delivery apps tend to offer the heaviest foods at the worst time. A very large, greasy meal close to bedtime can interfere with sleep through indigestion and reflux. If you are genuinely hungry late at night, a modest snack such as yoghurt, cereal with milk, toast with peanut butter, or a piece of fruit with cheese is kinder to sleep. Feeding yourself on a dorm budget Students without a meal plan, or with a limited one, face a different challenge: they have to plan, shop, and prepare food during the busiest weeks of the year, often without a proper kitchen. The good news is that a very serviceable exam-week diet can be built from inexpensive, shelf-stable and freezer-friendly foods, prepared with a kettle, a microwave, and a bowl. A useful approach is to shop once, in the days before exams begin, from a short list of staples that combine into many meals. The following list is a starting point: · Breakfast and grains: rolled oats, whole-grain bread, microwaveable rice pouches or instant brown rice, whole-grain crackers, pasta if you have kitchen access. · Protein: eggs, canned tuna or salmon, canned beans and chickpeas, peanut butter, plain yoghurt, milk or fortified soy milk, cheese, hummus, tofu if you have a fridge. · Vegetables and fruit: bags of frozen mixed vegetables and frozen spinach, baby carrots, bananas, apples, oranges, frozen berries, canned tomatoes. · Extras: nuts or trail mix, instant soup, spices or a jar of salsa, and a few treats you genuinely enjoy. Frozen vegetables and fruit deserve special mention. They are typically frozen soon after harvest, keep for months, microwave in minutes, and are often cheaper than fresh produce. They are the single easiest way to get vegetables into a dorm-room diet. Several basic meals can be assembled from this list in under ten minutes. Oats cooked in the microwave with milk and topped with a banana and a spoonful of peanut butter make a filling breakfast. Eggs can be scrambled in a mug in the microwave, stirring once or twice and cooking in short bursts until set all the way through, and served with toast. A rice pouch, a can of black beans rinsed and heated, frozen vegetables, and salsa make a burrito bowl. Canned tuna mixed with a little yoghurt or mayonnaise on whole-grain bread with baby carrots on the side makes lunch. Instant noodles become a reasonable meal with a handful of frozen vegetables and an egg stirred in while hot, or with leftover beans or tofu. A baked potato cooked in the microwave and topped with beans, cheese, or yoghurt is filling, cheap, and warm. If you have access to a kitchen, even a shared one, an hour of batch cooking on the weekend before exams can cover many meals. A large pot of chilli, lentil soup, pasta with vegetables, or rice with beans and vegetables can be portioned into containers and refrigerated for a few days or frozen for longer. The time spent is repaid many times over during the week, when each meal becomes a matter of reheating. A word on food safety, because it is easy to neglect when tired. Mini-fridges are small and often overfull, and their temperature can drift. Keep perishable foods such as cooked leftovers, dairy, and eggs cold, use leftovers within a few days, reheat them until steaming hot, and when in doubt, throw them out. A bout of food poisoning during exam week is a disproportionate price for saving a portion of leftover rice. Sample days, common mistakes, and different diets Table 6 lays out one illustrative exam day for each situation. Both follow the same shape: a breakfast with protein and slow-release carbohydrate, a mid-morning snack, a balanced lunch, an afternoon snack, a dinner that is satisfying but not heavy, and an optional light snack if hunger returns before bed. Caffeine is included, with a cutoff, because most students use it. Table 6. Two illustrative exam-week eating days. Meal Dining hall Dorm budget Purpose Breakfast Eggs, whole-grain toast, fruit, coffee Microwave oats with milk, banana, peanut butter; coffee Fuel for morning study Mid-morning Yoghurt and fruit taken from the hall Apple and a handful of nuts Prevents hunger dip Lunch Salad bar with beans or chicken, grains, roasted vegetables Tuna sandwich on whole-grain bread, carrots, water Balanced, not heavy Afternoon Tea or last coffee before cutoff; fruit Crackers and hummus; tea Bridges to dinner Dinner Protein, vegetables, rice or potatoes; small dessert Rice pouch, black beans, frozen vegetables, salsa, cheese Satisfying, easy to digest Evening, if hungry Cereal with milk Yoghurt with frozen berries Light, sleep-friendly The table is not meant to be followed item by item. Its point is the rhythm: food every three to five hours, protein in most meals, vegetables or fruit in most meals, caffeine early, and nothing enormous late at night. A student who follows the rhythm with entirely different foods is doing it right. Across both situations, the same handful of mistakes account for most exam-week eating problems. Each has a simple fix. The first is skipping meals to save time. It feels efficient, but a missed lunch usually costs more than the thirty minutes it saves, in the form of a sluggish, distracted afternoon and a large, late dinner that interferes with sleep. The correction is to put meals on the study schedule as fixed blocks, exactly as Table 3 in Chapter 3 does, so that eating is part of the plan rather than an interruption of it. The second is living on delivery. Food delivery apps are convenient during finals, but they are expensive, the portions tend to be large and heavy, and the timing is unpredictable, so a meal ordered at seven may arrive at eight-thirty when hunger has already turned into snacking. An occasional delivered meal is fine. A week of them strains both budget and sleep. The correction is to keep a few ten-minute meals from the staples list available as the default and to treat delivery as the exception. The third is eating everything at the desk. Students who eat while studying rarely notice what or how much they eat, get no real break, and often stretch a meal into an hour of half-attention to both food and work. The correction is to step away, even to a bench outside the library, for the length of the meal. The fourth is replacing meals with bars and shakes. Protein bars, meal-replacement drinks, and smoothies have their place, especially for the student who loses their appetite under stress or needs something to carry into a long afternoon. As a steady diet, they tend to leave people hungry, and many bars are closer to confectionery than their packaging suggests. The correction is to use them as a snack or an emergency meal, not as the plan. The fifth is forgetting to drink. Coffee is not a substitute for water, and many students realise only in the evening, with a headache, that they have had nothing else all day. The correction is to fill a water bottle every morning and refill it at every meal. The staples and sample days above adapt easily to most dietary patterns. Vegetarian and vegan students can build protein from beans, lentils, chickpeas, tofu, tempeh, peanut butter and other nut butters, nuts and seeds, and, for vegetarians, eggs, yoghurt, milk, and cheese. Canned beans and lentils are among the cheapest protein sources available and need only rinsing and heating. Fortified soy milk provides protein along with calcium and, often, vitamin B12, which vegans need from fortified foods or supplements. In dining halls, the salad bar and any dedicated plant-based station are the places to start. Students who keep halal or kosher can find that dining hall options are limited, although many campuses now offer certified stations or clearly labelled items. Canned fish, eggs, dairy where permitted, beans, and vegetarian staples provide flexible, affordable options for dorm-room meals, and campus religious organisations often know where certified foods can be found locally. Students with food allergies or coeliac disease should be cautious about shared toasters, microwaves, and serving utensils in dining halls and shared kitchens, and should keep a reliable supply of safe foods in their own room during exam week, when there is little time for a reaction or for searching for something safe to eat. Dining services at most institutions have staff, sometimes a registered dietitian, who can advise on safe choices and arrangements; it is worth contacting them before exams rather than during them. Anyone with a severe allergy should carry their prescribed emergency medication, including into the exam room. When money for food is short Any honest guide to student eating has to acknowledge that for many students the problem is not choosing among foods but affording them. Food insecurity, defined as limited or uncertain access to adequate food, is common among college students in the United States. In a 2020 scoping review in Advances in Nutrition, Cassandra Nikolaus and colleagues analysed 51 study samples and found estimates ranging from 10 percent to 75 percent, depending heavily on how and over what period it was measured. Surveys using a twelve-month reference period produced an average estimate of about 31 percent. Whatever the precise figure, the review makes clear that food insecurity is a widespread problem on campuses and not a rare exception. Food insecurity collides with finals in obvious ways. Money often runs lowest at the end of the semester, meal plan balances may be exhausted, and part-time work hours may be cut or, conversely, may compete with study time. Hunger undermines concentration and sleep and adds to stress, and the shame many students feel about it keeps them from seeking help. If this describes your situation, several resources are worth checking, and using them is exactly what they are for: · Most colleges and universities now have a campus food pantry or basic-needs center, often with extended hours or special distributions around finals. Many do not require proof of need. · Many campuses run meal-swipe donation programs, in which students with unused swipes donate them to students who need them. Swipe Out Hunger is one national organisation that helps campuses set these up. · Dean of students offices and financial aid offices frequently manage emergency grants or short-term loans for students facing unexpected hardship, and can sometimes adjust aid packages. · Some students are eligible for the Supplemental Nutrition Assistance Program, although student eligibility rules are more restrictive than for the general population and include specific exemptions. A campus basic-needs office can help determine whether you qualify. · Community food banks and pantries off campus are also open to students. For parents and supporters, one of the most practical forms of exam-week help is food. A care package of shelf-stable staples, such as oats, nut butter, crackers, canned fish, instant soup, dried fruit, and trail mix, or a grocery delivery arranged before exams begin, removes a planning task at the moment when students have the least time. It is also a form of support that does not require the student to ask. The plan for exam-week eating fits in a sentence: decide in advance what and when you will eat, so that when the week arrives you do not have to decide. Whether that means knowing the dining hall hours and carrying a banana, or stocking a shelf with oats, beans, and frozen vegetables, the goal is to take food off the list of things you have to think about. Finals week offers enough decisions without adding one at every meal. Chapter 7: Calming the Body Fast: Acute Stress and Test Anxiety Picture the hallway outside a large lecture hall ten minutes before a final. Some students are flicking through flashcards with a kind of frantic speed. Some are quizzing each other in loud voices. One is pacing. Another is sitting on the floor with eyes closed. Almost everyone's heart is beating faster than usual, and more than a few are aware of it, along with sweaty palms, a tight stomach, and a feeling that their mind has gone blank on material they knew perfectly well the night before. This chapter is about that moment and the many smaller versions of it that occur through finals week: the spike of alarm on opening a practice exam, the racing heart at 1 a.m., the freeze in the middle of an exam when a question looks unfamiliar. It explains what the body is doing, why the most common response, trying hard to calm down, often misfires, and which specific techniques have evidence behind them. It then addresses test anxiety as a pattern, and the point at which stress becomes something that deserves professional help. What the stress response is for When the brain perceives a situation as demanding and important, it activates the sympathetic nervous system and the hormonal stress system. Heart rate and blood pressure rise, breathing quickens, blood flow shifts toward the large muscles, stored energy is released into the blood, and attention narrows onto the perceived threat. The hormone cortisol rises more slowly and helps sustain the response. This package evolved to prepare the body for action, and it is not inherently harmful. In moderate amounts, arousal sharpens attention and speeds responses. Athletes, musicians, and public speakers all perform with elevated heart rates, and many would perform worse without them. The trouble during exams comes from two directions. The first is intensity: when arousal climbs very high, the body's signals become distracting in their own right, and the narrowed attention that suits a physical threat suits abstract reasoning poorly. The second, and often more important, is interpretation. A pounding heart and a tight stomach can be read as signs of readiness or as signs of impending failure. The second reading adds a layer of worry, and worry is costly for thinking. Research on test anxiety distinguishes two components. Emotionality refers to the bodily symptoms: racing heart, sweating, nausea, trembling. Worry refers to the thoughts: I am going to fail, I always blank, everyone else understands this. Worry is the component most closely linked to poorer performance, and one explanation is that anxious thoughts compete for working memory, the limited mental workspace needed to hold a problem in mind and manipulate it. A student who is solving an equation while also rehearsing the consequences of failing it is, in effect, trying to do two tasks with the resources for one. This helps explain the scale of the problem. In a 2018 meta-analysis in the Journal of Affective Disorders covering 238 studies conducted over three decades, Nathaniel von der Embse and colleagues found that test anxiety was consistently and negatively related to a wide range of educational outcomes, including standardised tests, university entrance exams, and grade point average. The relationships were in the small to moderate range: meaningful, but far from deterministic. Higher test anxiety was associated with lower self-esteem and with tests perceived as difficult or high-stakes, which describes most final exams. The practical goal, then, is not to eliminate arousal, which is neither possible nor desirable, but to keep it in a useful range and to change its meaning, so that it serves performance rather than competing with it. The techniques below approach that goal from two sides: those that act on the body directly, through breathing, and those that act on interpretation and worry. Breathing: the fastest lever you control Breathing is unusual among bodily functions. It runs automatically, but it can also be taken over deliberately at any moment, and changes in breathing produce rapid changes in heart rate and in the balance between the arousing and calming branches of the autonomic nervous system. Each inhalation slightly speeds the heart and each exhalation slightly slows it, a normal phenomenon called respiratory sinus arrhythmia. Breathing patterns that lengthen the exhalation relative to the inhalation therefore tilt the system, breath by breath, toward calm. The physiological sigh and cyclic sighing. The physiological sigh is a pattern that the body produces spontaneously, for example during sleep or after crying: a double inhalation followed by a long exhalation. To do it deliberately, breathe in through the nose until the lungs feel nearly full, then take a second, shorter sip of air through the nose to top them up, and then exhale slowly and completely through the mouth, making the exhalation noticeably longer than the two inhalations together. Repeated for several minutes, the pattern is called cyclic sighing. The best-known study of this technique was published in 2023 in Cell Reports Medicine by Melis Yilmaz Balban, David Spiegel, Andrew Huberman, and colleagues at Stanford University. It is worth describing carefully, both because it is often cited in exaggerated terms and because it is genuinely useful. The researchers randomly assigned 108 adults, most of them recruited from an undergraduate psychology class, to one of four daily five-minute practices for a month: cyclic sighing, box breathing with equal inhales, holds, and exhales, a form of cyclic hyperventilation with breath retention, or mindfulness meditation. Participants rated their mood and anxiety before and after each daily session and wore a device that tracked physiological measures. All four groups showed improvements after their daily sessions, with reduced anxiety and negative mood and increased positive mood. The breathing groups combined showed a larger increase in positive mood than the meditation group, and cyclic sighing showed the largest effect: a significantly greater daily increase in positive mood than mindfulness meditation, and a greater reduction in resting respiratory rate over the month. The benefit of cyclic sighing on positive mood also grew with the number of days participants had practised. The groups did not differ in their reductions in state anxiety, and there were no measurable changes in sleep. The authors themselves noted the study's limits. It was conducted remotely during the COVID-19 pandemic, was designed as an exploratory study, was registered only retrospectively, and had relatively small group sizes, with thirty participants in the cyclic sighing group. It tested a daily five-minute practice over a month, not a single sigh in an exam hallway. The fair summary is that cyclic sighing is a simple, safe, free technique with promising evidence for improving mood and reducing physiological arousal with regular practice, and a plausible physiological basis for a quick calming effect in the moment. That is a good deal more than most stress advice can claim, and a good deal less than headlines sometimes suggest. In practice, cyclic sighing can be used in two ways during finals. As a daily practice, five minutes at the start of a study day or before bed builds the skill and may help mood across the week. As an acute tool, two or three sighs in the hallway before an exam, or quietly at your desk when you notice your heart racing in the middle of one, provide an immediate way to slow the breath and interrupt a spiral. The long exhalation can be done silently through pursed lips, so no one else in the room will notice. Slow paced breathing. A broader body of research concerns breathing slowly, at around five to seven breaths per minute, compared with the typical resting rate of twelve to twenty. A 2018 systematic review by Andrea Zaccaro and colleagues in Frontiers in Human Neuroscience examined studies of slow breathing below ten breaths per minute in healthy people and found that it was associated with increased heart rate variability, a marker of flexible autonomic regulation, and with psychological reports of greater relaxation, comfort, and alertness and reduced anxiety, anger, and confusion. The review was based on a small number of eligible studies, but its findings are consistent with a longer tradition of clinical work using heart rate variability biofeedback, in which people learn to breathe at their own resonant rate, typically near six breaths per minute. A simple version is to breathe in through the nose for a count of four and out for a count of six, which gives six breaths per minute, and continue for two to five minutes. If a count of four and six feels strained, shorten both while keeping the exhalation longer. The goal is gentle and smooth, not deep and forced. Overbreathing, taking very large breaths rapidly, can cause lightheadedness and tingling, which can themselves feel alarming; if that happens, make each breath smaller and slower. Box breathing. The pattern of equal counts for inhaling, holding, exhaling, and holding, often four seconds each, is widely taught in the military and elsewhere. In the Stanford trial it performed reasonably but did not stand out. Some people find the counting structure helpful for occupying a racing mind. It is a reasonable alternative for those who find it more natural than sighing. Whichever technique you choose, the most important advice is to practise it before you need it. A skill rehearsed daily during a calm week is far more available under pressure than one you are trying to recall from a video while your heart pounds. Five minutes a day during the fortnight before exams is enough. Changing the meaning of arousal, and emptying the worry Breathing acts on the body. A second family of techniques acts on the interpretation of the body's signals, and here the evidence is surprisingly strong for such simple interventions. The work most associated with this idea comes from the psychologist Jeremy Jamieson and colleagues. In a study published in 2010 in the Journal of Experimental Social Psychology, participants who were preparing to take the Graduate Record Examination came to a laboratory to sit a practice test. Half were given a short explanation before starting: that people often feel anxious during standardised tests, that this arousal does not hurt performance and can actually help it, and that if they felt anxious, they should remind themselves that it could be helping them. The other half received no such information. The reappraisal group showed a physiological sign of increased sympathetic activation, which is to say their bodies were, if anything, more aroused, and they outperformed the control group on the math section. When the participants later reported their scores on the real GRE, taken one to three months afterwards, the reappraisal group again scored higher on the math section. In a 2016 study in Social Psychological and Personality Science, Jamieson and colleagues took the idea into community college mathematics classrooms, where many students carry considerable anxiety about the subject. Students who received brief reappraisal materials before exams performed better and reported less evaluation anxiety than those who did not. A related line of research by Alison Wood Brooks, published in 2014 in the Journal of Experimental Psychology: General, found that people who reappraised pre-performance anxiety as excitement, sometimes simply by saying "I am excited" aloud, performed better at singing, public speaking, and mathematics than those who tried to calm down. Brooks also found that most people believe trying to calm down is the best strategy, which may explain why it is the one most often attempted. The logic is that anxiety and excitement are physiologically similar high-arousal states, so moving from one to the other is easier than moving from high arousal to calm. Trying to force calm when the body is clearly aroused often produces a sense of failure: I am supposed to be relaxed and I am not, which becomes one more thing to worry about. Relabelling the arousal as useful lets it stay while changing what it means. In practice, reappraisal can be as short as a sentence you say to yourself. Before an exam, when you notice your heart racing, you might think: my body is getting ready; this is energy I can use. When a hard question makes your stomach drop: this is the feeling of a challenge, not a catastrophe. The words matter less than the stance, which treats the body's response as an ally. Reappraisal and breathing are not in competition. A useful sequence in the hallway is to take two or three slow sighs to bring the arousal down from its peak, then consciously relabel what remains as readiness. The third technique targets worry directly. In 2011, Gerardo Ramirez and Sian Beilock published a set of studies in Science testing a simple intervention: having students spend about ten minutes immediately before an exam writing freely about their thoughts and feelings about the upcoming test. In laboratory experiments and in a randomised field study in a high school biology class, this brief expressive writing improved exam scores, and the benefit was concentrated among students who were habitually anxious about tests. The explanation offered was that writing out worries reduces the extent to which they occupy working memory during the exam itself, freeing that capacity for the questions. A student can adapt this easily. Arrive at the exam location a little early, or use the time before leaving your room, and write for several minutes, as honestly as possible, about what you are worried about: the questions you fear, what failing would mean, the chapter you did not review enough. Do not try to argue with the worries or solve them. The act of putting them on paper is the intervention. Then close the notebook and turn your attention to the exam. This technique pairs naturally with the bedtime to-do list from Chapter 1. Both work by externalising thoughts that would otherwise circle, one to allow sleep and the other to allow thinking. Even well-prepared students sometimes freeze during an exam: a question looks unfamiliar, the mind goes blank, the heart jumps, and a flood of thoughts about failing arrives. A short, practised recovery routine can turn this from a spiral into a brief pause. First, notice what is happening and name it to yourself: this is a stress spike. Naming a feeling tends to reduce its grip. Second, put the pen down, drop your shoulders, and take two or three slow exhalation-weighted breaths, which can be done silently. Third, relabel: this is my body preparing me; I have handled hard questions before. Fourth, move. Skip the question, mark it, and go to one you can answer. Early success restores working memory and confidence, and on returning to the hard question, material often surfaces that was inaccessible under the first surge of alarm. This routine takes perhaps thirty seconds. It is worth rehearsing during practice exams so that it becomes automatic. When stress is more than stress Most finals-week anxiety, however unpleasant, is a normal response to a demanding situation, and it subsides when the situation ends. Some students experience something more severe, and it is important to recognise the difference. A panic attack is a sudden surge of intense fear with physical symptoms that can include a pounding heart, chest tightness, shortness of breath, dizziness, trembling, tingling, and a sense of unreality or impending doom, typically peaking within minutes. Panic attacks are frightening but not dangerous in themselves, and slow breathing with a long exhalation is often helpful. However, some of these symptoms overlap with medical emergencies. Chest pain, severe shortness of breath, fainting, or symptoms following stimulant or high caffeine use warrant urgent medical evaluation, particularly if this is the first time you have experienced them. Signs that stress has become something that deserves professional support include anxiety that is present most days and interferes with sleeping, eating, or functioning; recurrent panic attacks; avoiding classes or exams because of anxiety; persistent low mood, hopelessness, or loss of interest in things you usually enjoy; heavy reliance on alcohol or other substances to cope; and any thoughts of self-harm or suicide. Campus counseling centers see exactly these problems, and many offer same-day or crisis appointments. If you are in crisis or thinking about ending your life, call or text 988 in the United States to reach the 988 Suicide and Crisis Lifeline, or call emergency services. Students with diagnosed anxiety disorders, and those whose test anxiety is severe and long-standing, may also be eligible for academic accommodations through the campus disability or accessibility office, such as extended time or a quieter testing environment. As with ADHD, these arrangements take time to put in place, so the conversation is best started early in the semester. Effective treatments exist. Cognitive behavioural therapy, including approaches designed specifically for test and performance anxiety, has a strong evidence base, and a few sessions before finals can provide tools that last well beyond one semester. Needing that help is common and says nothing about ability. Many of the students in that hallway who look entirely composed are using exactly the techniques in this chapter, and some of them learned them from a counselor. Chapter 8: Exam Day and the Days After Everything in the preceding chapters converges on a few hours in an exam room. This chapter walks through those hours and the hours around them, from the evening before an exam to the week after the last one. It is deliberately practical. By exam day, the learning is largely done, and the aim is to arrive with that learning intact and accessible, to perform steadily under pressure, and then to recover properly so that one exam does not damage the next. The chapter closes with the part of finals that almost nobody plans for: the days after the semester ends, when the body's accumulated costs come due and when some of the riskiest behaviour of the whole period takes place. The evening and morning before The evening before an exam should look different from the rest of the week. The goal is to consolidate, not to cover new ground. Plan to finish serious study at a reasonable time, perhaps two hours before bed. Use the last session for a light retrieval pass over the material you know is most likely to appear: answering a few practice questions, running through key flashcards, or explaining the main concepts aloud. Avoid opening an entirely new topic, which is more likely to produce anxiety than learning at this stage. Then prepare for the morning, so that there is nothing left to decide. Pack your bag with everything the exam requires: student identification, pens and pencils, an approved calculator with working batteries, a clear water bottle, a watch if phones are not allowed, and any permitted materials. Check the exam room and time one more time, since rooms for final exams are often different from regular classrooms. Plan your route and allow for delays. Set two alarms. Lay out breakfast, or know exactly where you will get it. About half an hour before bed, write your to-do list for the next day and, if worries are circling, a few lines about them. Do some slow breathing. Then go to bed at your usual time rather than much earlier, since trying to sleep hours ahead of your body clock often just produces more time lying awake. If sleep does not come easily, remember two things. First, as Chapter 1 described, the best classroom evidence suggests that the single night before a test matters less for performance than the sleep of the preceding weeks. A restless night is unpleasant but rarely decisive. Second, lying quietly with your eyes closed is more restful than getting up to study in a panic. If you have been awake for a long time and feel frustrated, get up briefly, do something quiet in dim light, and return to bed when drowsy. Wake at your anchor time, or early enough to have at least an hour and a half before the exam begins. Get some daylight, even briefly, and drink a glass of water. Eat something. The ideal pre-exam breakfast is familiar, moderately sized, and includes some protein and slow-release carbohydrate: eggs and toast, oats with milk and fruit, yoghurt with granola, or peanut butter on whole-grain bread with a banana. If nerves make eating difficult, choose something small and bland, such as toast or a banana, and eat it slowly. Avoid a very large, heavy, or unfamiliar meal. Have your usual amount of caffeine at your usual time, not more. As Chapter 5 explained, extra caffeine on a high-arousal morning adds jitteriness and a racing heart without adding performance. Drink some water, but not so much that you spend the exam thinking about the bathroom. Use the time before leaving for a short warm-up of perhaps twenty to thirty minutes: a few practice problems or quick self-testing on key points. The aim is to switch your brain into retrieval mode and remind yourself that you know the material. It is not the time to discover gaps you cannot fix. Plan to arrive a little early, perhaps ten to fifteen minutes before the start, but not so early that you spend half an hour in a crowded hallway absorbing other students' panic. Conversations along the lines of "Did you study the chapter on X? It's definitely going to be on it" help no one. If you find the pre-exam crowd stressful, wait somewhere quieter nearby, use the bathroom, and then do your breathing and expressive writing. Two or three physiological sighs followed by a reappraisal statement, as described in Chapter 7, make a good last step before entering the room. In the exam room and between exams Once the exam begins, a few habits consistently help. Spend the first minute or two looking over the whole paper. Note how many questions there are, how the marks are distributed, and roughly how much time each section deserves. This protects against the most common and most avoidable disaster: running out of time with high-value questions untouched. Start with a question you can answer confidently. Early success calms the stress response and restores working memory. There is no rule that questions must be answered in order. Read each question slowly and completely, and underline the key terms and instructions: compare, explain, calculate, list three. Many lost marks come not from ignorance but from answering a question that was not asked. For essay and long-answer questions, spend a minute or two jotting a brief outline before writing. It saves time overall and produces a more organised answer. If you freeze or panic, use the recovery routine from Chapter 7: notice it, breathe with a long exhalation, relabel the arousal as readiness, and move to another question. Keep track of time with a watch or the room clock, and check it at planned points rather than constantly. If you are running short, write outlines or key steps for the remaining questions, since partial credit is often available. Leave time at the end to check your answers, especially for calculations and for questions you answered quickly. Check that you have answered every part of every question and that your name and identification are on everything. After the exam, resist the post-mortem. Standing outside comparing answers with classmates tends to raise anxiety, especially when someone confidently reports an answer different from yours, and it cannot change your score. If you have another exam coming, the most useful thing to do with the one you have just finished is to put it down. Finals schedules frequently cluster exams together, sometimes two on the same day or several in consecutive days. The period between exams is where recovery happens or does not. After an exam, take a genuine break before turning to the next subject. Eat a proper meal. Walk outside. If you slept poorly, take a short nap in the early afternoon, following the rules in Chapter 2. Then switch deliberately to the next course, beginning with retrieval practice on its key material. On days with two exams, treat the gap between them as a pit stop, not a study session. Eat, drink water, move a little, use the bathroom, and do a short, calm review of the second subject's main points. Trying to cram an entire course into a two-hour gap after an exhausting morning is more likely to raise anxiety than scores. Across a multi-day exam period, protect the routine: anchor wake time, regular meals, caffeine cutoff, wind-down, and sleep. The routine matters more as the week goes on, because fatigue accumulates and each exam slightly erodes the reserves the next one needs. Other formats, and a word for supporters Not every final is a timed paper exam in a hall. Take-home exams, online exams with a time window, oral examinations, presentations, and final projects each bring their own health traps. Take-home exams and exams with a long submission window often seem gentler, but they invite a particular kind of overwork. With no clock enforcing an end, students may work late into the night polishing answers, sacrificing sleep in the middle of a week that also contains other exams. The remedy is to impose the limits the format does not: decide in advance how many hours the exam deserves, schedule them in daytime blocks, stop at a set time each evening, and submit when the planned time is used rather than when the answers feel perfect. Perfection is rarely what separates grades, and the sleep lost pursuing it will be felt in the next exam. Online exams taken in a dorm room carry their own risks: unreliable internet, a roommate returning halfway through, notifications, and the temptation to check a phone. Test your connection and any required software the day before, find out what to do if something fails, tell roommates when you need quiet, and put the phone in another room if the exam rules permit. Treat the morning exactly as you would for an in-person exam, with breakfast, daylight, and a warm-up, rather than rolling out of bed to the laptop. Oral exams and presentations produce the most visible stress symptoms: a shaky voice, a dry mouth, a pounding heart. These are the situations in which reappraisal has been most directly tested, in Brooks's studies of public speaking and in related work, and in which breathing techniques are most useful because they can be used moments before speaking. Rehearse aloud several times, ideally in front of a friend, so that retrieval under observation is familiar. Keep water at hand. Slow down deliberately in the first minute, since arousal speeds speech. Final projects and portfolios due during exam week should be treated as exams with a very long window. The central risk is that they are left until the last days and then collide with timed exams, producing precisely the midnight dilemma described in Chapter 2. Breaking them into dated pieces several weeks ahead is the single most effective prevention. Parents, partners, roommates, and staff often want to help during exam week and are unsure how. The most useful help tends to be practical and low-pressure. Food, as Chapter 6 suggested, removes a task. A short message of encouragement that does not ask for a progress report is often more welcome than a phone call about grades. Offering a ride home instead of letting a student drive after their last exam can prevent a genuinely dangerous trip. And noticing signs that stress has become something more, such as withdrawal, hopelessness, or talk of not coping, and gently encouraging contact with counseling services, can matter a great deal. Supporters do not need to become study coaches. They need to make the healthy choices a little easier and the unhealthy ones a little less necessary. After the last exam The final exam of the semester ends with a burst of relief, and the days that follow are often treated as outside the concerns of a book about finals. They should not be, for several reasons. Sleep recovery takes time. After a period of short sleep, many people sleep very long the first night and then drift into late, irregular hours. Recovery sleep is valuable, but a single long night does not fully reverse the effects of accumulated sleep loss. A 2019 study in Current Biology by Christopher Depner, Kenneth Wright, and colleagues found that weekend recovery sleep did not prevent the metabolic effects of repeated short sleep, such as reduced insulin sensitivity and increased evening eating, once short sleep resumed. The practical lesson for the end of term is to allow extra sleep for a few nights, but to re-establish a regular wake time within a day or two rather than letting the schedule drift for weeks, and to recognise that the best protection is not to build up so much debt in the first place. Celebration carries real risks. The end of exams is traditionally marked with parties, and for many students it brings the heaviest drinking of the semester. That drinking often happens on top of accumulated sleep deprivation, which amplifies alcohol's effects on judgment and coordination, and sometimes in combination with energy drinks, which can mask how intoxicated a person has become. The National Institute on Alcohol Abuse and Alcoholism defines binge drinking as a pattern that brings blood alcohol to 0.08 percent, typically four or more drinks for women or five or more for men in about two hours. Students who plan to celebrate can reduce the risks by sleeping first, eating before and while drinking, alternating alcoholic drinks with water, avoiding mixing alcohol with energy drinks or other substances, and staying with friends who will look out for them. Signs of alcohol poisoning, such as confusion, vomiting while semi-conscious, slow or irregular breathing, seizures, or being impossible to wake, call for emergency help immediately. Do not leave someone to sleep it off. The drive home is a hazard. Many students travel home immediately after their last exam, often by car and often after a week of short nights. This is exactly the situation in which drowsy driving is most likely. The comparison between sleep loss and alcohol impairment described in Chapter 2 applies directly: a driver who has been awake since early morning and slept badly all week may be significantly impaired while feeling merely tired. Sleep before a long drive, share the driving, take breaks every couple of hours, and pull over to nap if you notice yourself yawning repeatedly, drifting from your lane, or failing to remember the last few miles. Caffeine and open windows are not substitutes for sleep. The let-down is normal. Many students describe a flat, tired, or low period after exams end, and some find that they get sick soon after, often with a cold. Stress and short sleep over several weeks affect both mood and immune function, and a drop in adrenaline-fuelled focus can leave people feeling surprisingly drained. The response is to rest, eat regularly, move gently, and spend time with people you enjoy. If low mood persists for more than a couple of weeks, or is accompanied by hopelessness or thoughts of self-harm, reach out to a clinician, a counseling service, or the 988 Suicide and Crisis Lifeline. Final grades usually arrive days or weeks after the last exam, and waiting for them can be its own source of anxiety. It helps to decide in advance when you will check, rather than refreshing the portal every hour, and to remember that no single grade determines a future. If a grade is disappointing, there are usually concrete steps available. Many professors will review an exam with a student who asks respectfully, which is valuable for learning even if the grade does not change. Academic advisers can explain options such as retake policies, withdrawals, or grade appeals where there has been a genuine error, and can help plan the next semester. A poor result during a semester that included illness, family difficulty, or a mental health crisis may also qualify for retroactive consideration at some institutions; the dean of students office can advise. Whatever the grades, the week after finals is the best time to do something most students never do: a short review of how the semester went, written while it is fresh. It need not be long. What worked? Where did the plan break down? When did sleep start slipping, and why? Which study methods produced learning, and which only produced hours? What resources would have helped, and when should they have been contacted? A student who writes half a page of honest answers in December or May is far better placed to make the next finals period different, and that difference is, in the end, the purpose of this book. Conclusion The chapters of this book have treated sleep, food, study methods, caffeine, and stress one at a time, because that is how evidence is gathered and how advice is easiest to follow. But the student at midnight in the library does not face these as separate subjects. They face a single question, repeated in many forms across the week: something has to give, so what should it be? Taken together, the evidence gives a clearer answer to that question than any single chapter could on its own. It suggests an order of sacrifice, a ranking of what to protect and what to let go when time runs short, and that ranking is almost exactly the reverse of what most students instinctively do. What to give up first The instinctive order runs like this. Sleep goes first, because it looks like empty time. Meals go next, because they take time away from the desk. Stress management is never started, because it seems optional. Caffeine rises to fill the gaps. Study hours, and particularly the familiar methods of rereading and highlighting, are the last thing to be cut, because they feel like the work itself. The evidence suggests reversing almost every step. The first thing to give up is low-yield study. Rereading, highlighting, recopying notes, and passive review of solved examples all feel productive and produce comparatively little durable learning, as Chapter 3 showed. Replacing them with retrieval practice and spaced review usually frees hours while improving results. For most students, this is the largest reserve of time available, and it costs nothing but the discomfort of testing themselves. The second thing to give up is marginal caffeine. The first cup has real value for a tired person. Beyond a moderate dose, extra caffeine adds jitteriness and physical symptoms of anxiety, and any caffeine taken late in the day takes sleep away, as the Drake study showed, even from people who do not notice it. Cutting the afternoon energy drink costs little and returns sleep. The third thing to give up, reluctantly and only partly, is breadth. When there is genuinely more material than time, triage is better than skimming everything badly. Covering the highest-yield topics well, and accepting that some questions will be answered from partial knowledge, beats arriving exhausted with a thin familiarity with everything. What should be given up last is sleep. This is the book's central claim, and it rests on converging evidence rather than any single study: that sleep before learning prepares the brain to encode, that sleep after learning consolidates, that students who sleep well across the semester earn better grades, that sleep-deprived students overestimate their own performance, and that sleep loss magnifies emotional reactivity and hunger for quick food. No other single behaviour touches so many of the systems exams depend on. Close behind sleep come regular meals and water, which cost little time and protect attention through the day. And the few minutes spent on breathing, reappraisal, or writing out worries are among the cheapest investments in the book, with evidence of benefit disproportionate to their cost. Put simply: when something has to give, give up the methods that feel like work but are not, then the extra stimulants, then some breadth. Give up sleep last, and give it up in small amounts rather than whole nights. Why the whole is different from the parts There is a further conclusion that only becomes visible when the chapters are read together. Each of the three systems this book describes can be pushed in isolation for a short time. A person can go without sleep for a night, skip meals for a day, or run on stress hormones through a difficult morning, and the body will cover the gap. What it covers poorly is all three at once, because each depends on the others to recover. Sleep loss increases the appetite for poor food and the intensity of stress. Stress delays sleep. Poor food and excess caffeine make both stress and sleep worse. A finals week that strains all three simultaneously is not three times as hard as one that strains one; it is harder than that, because the usual routes to recovery are blocked. The same interdependence explains why modest changes can have outsized effects. A student who sets a caffeine cutoff sleeps better, which makes them less reactive, which makes them less likely to eat badly and less likely to lie awake worrying, which improves the next night's sleep. The most important practical lesson of the book may be that no one needs to optimise everything. Two or three well-chosen changes, applied consistently, move the whole system. For most students those changes are a fixed wake time, a caffeine cutoff, retrieval practice in place of rereading, and a planned meal rhythm. Everything else is refinement. Finals week begins in the first week The last conclusion concerns timing, and it is uncomfortable. The best evidence on sleep and grades points to the weeks before exams rather than the night before. The spacing effect rewards reviewing material across many days. The strongest defence against the all-nighter is a calendar drawn up a month earlier. Accommodations for ADHD, anxiety, or health conditions take weeks to put in place. The breathing techniques that work under pressure work best after practice. Batch-cooked meals and a stocked shelf require a weekend that is not already consumed by exams. In other words, a holistic approach to finals week is really a way of running the whole semester with finals in mind. The student who reads this book with three days to go will still benefit from much of it, particularly the guidance on the night before, exam-day routines, caffeine, and stress in the moment. The student who reads it with six weeks to go can change the outcome far more. That is why the book ends not with a final tip but with an invitation to plan. At the start of the next term, write the exam dates into a calendar. Decide on a wake time and keep it. Build retrieval practice into the weekly routine from the first lecture. Learn one breathing technique while nothing is at stake. Find out where the food pantry, the counseling center, and the disability office are before you need them. None of these acts is dramatic, and none will feel as satisfying as a heroic final push. Together they make the heroic push unnecessary. What finals week really measures Final exams are designed to measure what students have learned in their courses. They also measure, less officially, something that no transcript records: how well a person can look after themselves while under sustained pressure. That second skill is not a distraction from education. It is one of the most transferable things a college education can build, and the habits formed around exams tend to reappear in the professional deadlines, family crises, and long demanding stretches that come later in life. A student who learns to protect sleep during finals, to eat on a schedule when busy, to use caffeine as a tool rather than a crutch, to decline the pill that promises focus, and to calm their body in a hallway before a test is not only more likely to do well on the exam in front of them. They are practising a way of working that will outlast every course they take. That is the deepest sense in which caring for the body during finals week is part of studying: it is study in the skill that will matter longest. Notes Sources are listed by chapter in the order in which they are first drawn on in the text. Introduction 1. Drake C, Roehrs T, Shambroom J, Roth T. Caffeine effects on sleep taken 0, 3, or 6 hours before going to bed. Journal of Clinical Sleep Medicine. 2013;9(11):1195-1200. https://doi.org/10.5664/jcsm.3170 1. 988 Suicide and Crisis Lifeline. https://988lifeline.org Chapter 1: Sleep Is Part of Studying 1. Yoo SS, Hu PT, Gujar N, Jolesz FA, Walker MP. A deficit in the ability to form new human memories without sleep. Nature Neuroscience. 2007;10(3):385-392. https://doi.org/10.1038/nn1851 2. Rasch B, Born J. About sleep's role in memory. Physiological Reviews. 2013;93(2):681-766. https://doi.org/10.1152/physrev.00032.2012 3. Gais S, Lucas B, Born J. Sleep after learning aids memory recall. Learning and Memory. 2006;13(3):259-262. https://doi.org/10.1101/lm.132106 4. Okano K, Kaczmarzyk JR, Dave N, Gabrieli JDE, Grossman JC. Sleep quality, duration, and consistency are associated with better academic performance in college students. npj Science of Learning. 2019;4:16. https://doi.org/10.1038/s41539-019-0055-z 5. Pilcher JJ, Walters AS. How sleep deprivation affects psychological variables related to college students' cognitive performance. Journal of American College Health. 1997;46(3):121-126. https://doi.org/10.1080/07448489709595597 6. Yoo SS, Gujar N, Hu P, Jolesz FA, Walker MP. The human emotional brain without sleep: a prefrontal amygdala disconnect. Current Biology. 2007;17(20):R877-R878. https://doi.org/10.1016/j.cub.2007.08.007 7. Greer SM, Goldstein AN, Walker MP. The impact of sleep deprivation on food desire in the human brain. Nature Communications. 2013;4:2259. https://doi.org/10.1038/ncomms3259 8. Watson NF, Badr MS, Belenky G, et al. Recommended amount of sleep for a healthy adult: a joint consensus statement of the American Academy of Sleep Medicine and Sleep Research Society. Sleep. 2015;38(6):843-844. 9. Lund HG, Reider BD, Whiting AB, Prichard JR. Sleep patterns and predictors of disturbed sleep in a large population of college students. Journal of Adolescent Health. 2010;46(2):124-132. https://doi.org/10.1016/j.jadohealth.2009.06.016 10. Hershner SD, Chervin RD. Causes and consequences of sleepiness among college students. Nature and Science of Sleep. 2014;6:73-84. 11. Scullin MK, Krueger ML, Ballard HK, Pruett N, Bliwise DL. The effects of bedtime writing on difficulty falling asleep: a polysomnographic study comparing to-do lists and completed activity lists. Journal of Experimental Psychology: General. 2018;147(1):139-146. https://doi.org/10.1037/xge0000374 Chapter 2: The All-Nighter, Examined 1. Thacher PV. University students and "the all nighter": correlates and patterns of students' engagement in a single night of total sleep deprivation. Behavioral Sleep Medicine. 2008;6(1):16-31. https://doi.org/10.1080/15402000701796114 2. Williamson AM, Feyer AM. Moderate sleep deprivation produces impairments in cognitive and motor performance equivalent to legally prescribed levels of alcohol intoxication. Occupational and Environmental Medicine. 2000;57(10):649-655. https://doi.org/10.1136/oem.57.10.649 3. Reyner LA, Horne JA. Suppression of sleepiness in drivers: combination of caffeine with a short nap. Psychophysiology. 1997;34(6):721-725. 4. National Highway Traffic Safety Administration. Drowsy driving. https://www.nhtsa.gov/risky-driving/drowsy-driving Chapter 3: Study Methods That Work With Your Biology 1. Dunlosky J, Rawson KA, Marsh EJ, Nathan MJ, Willingham DT. Improving students' learning with effective learning techniques: promising directions from cognitive and educational psychology. Psychological Science in the Public Interest. 2013;14(1):4-58. https://doi.org/10.1177/1529100612453266 2. Roediger HL, Karpicke JD. Test-enhanced learning: taking memory tests improves long-term retention. Psychological Science. 2006;17(3):249-255. 3. Smith AM, Floerke VA, Thomas AK. Retrieval practice protects memory against acute stress. Science. 2016;354(6315):1046-1048. https://doi.org/10.1126/science.aah5067 4. Agarwal PK, D'Antonio L, Roediger HL, McDermott KB, McDaniel MA. Classroom-based programs of retrieval practice reduce middle school and high school students' test anxiety. Journal of Applied Research in Memory and Cognition. 2014;3(3):131-139. 5. Cepeda NJ, Pashler H, Vul E, Wixted JT, Rohrer D. Distributed practice in verbal recall tasks: a review and quantitative synthesis. Psychological Bulletin. 2006;132(3):354-380. 6. Cepeda NJ, Vul E, Rohrer D, Wixted JT, Pashler H. Spacing effects in learning: a temporal ridgeline of optimal retention. Psychological Science. 2008;19(11):1095-1102. 7. Rohrer D, Taylor K. The shuffling of mathematics problems improves learning. Instructional Science. 2007;35:481-498. 8. Chang YK, Labban JD, Gapin JI, Etnier JL. The effects of acute exercise on cognitive performance: a meta-analysis. Brain Research. 2012;1453:87-101. https://doi.org/10.1016/j.brainres.2012.02.068 Chapter 4: Brain Food Without the Hype 1. Mantantzis K, Schlaghecken F, Sünram-Lea SI, Maylor EA. Sugar rush or sugar crash? A meta-analysis of carbohydrate effects on mood. Neuroscience and Biobehavioral Reviews. 2019;101:45-67. https://doi.org/10.1016/j.neubiorev.2019.03.016 2. Adan A. Cognitive performance and dehydration. Journal of the American College of Nutrition. 2012;31(2):71-78. 3. United States Department of Agriculture. MyPlate. https://www.myplate.gov 4. Harvard T.H. Chan School of Public Health. Healthy Eating Plate. https://nutritionsource.hsph.harvard.edu/healthy-eating-plate/ 5. United States Food and Drug Administration. Dietary supplements: information for consumers. https://www.fda.gov/food/dietary-supplements Chapter 5: Caffeine, Energy Drinks, and Study Drugs 1. Drake C, Roehrs T, Shambroom J, Roth T. Caffeine effects on sleep taken 0, 3, or 6 hours before going to bed. Journal of Clinical Sleep Medicine. 2013;9(11):1195-1200. 2. United States Food and Drug Administration. Spilling the beans: how much caffeine is too much? Consumer update. https://www.fda.gov/consumers/consumer-updates/spilling-beans-how-much-caffeine-too-much 3. Mayo Clinic. Caffeine content for coffee, tea, soda and more. https://www.mayoclinic.org/healthy-lifestyle/nutrition-and-healthy-eating/in-depth/caffeine/art-20049372 4. Substance Abuse and Mental Health Services Administration. The DAWN Report: Update on emergency department visits involving energy drinks: a continuing public health concern. 2013. https://www.samhsa.gov/data/sites/default/files/DAWN126/DAWN126/sr126-energy-drinks-use.htm 5. United States Food and Drug Administration. Pure and highly concentrated caffeine. Consumer guidance, fda.gov. 6. Benson K, Flory K, Humphreys KL, Lee SS. Misuse of stimulant medication among college students: a comprehensive review and meta-analysis. Clinical Child and Family Psychology Review. 2015;18(1):50-76. https://doi.org/10.1007/s10567-014-0177-z 7. Ilieva I, Boland J, Farah MJ. Objective and subjective cognitive enhancing effects of mixed amphetamine salts in healthy people. Neuropharmacology. 2013;64:496-505. https://doi.org/10.1016/j.neuropharm.2012.07.021 8. Ilieva IP, Hook CJ, Farah MJ. Prescription stimulants' effects on healthy inhibitory control, working memory, and episodic memory: a meta-analysis. Journal of Cognitive Neuroscience. 2015;27(6):1069-1089. https://doi.org/10.1162/jocna00776 9. Arria AM, Caldeira KM, Vincent KB, et al. Do college students improve their grades by using prescription stimulants nonmedically? Addictive Behaviors. 2017;65:245-249. https://doi.org/10.1016/j.addbeh.2016.07.016 10. Drug Enforcement Administration. Public safety alert: sharp increase in fake prescription pills containing fentanyl and meth. 2021. https://www.dea.gov/alert/sharp-increase-fake-prescription-pills-containing-fentanyl-and-meth Chapter 6: Eating Well From a Dining Hall or a Dorm Room 1. Nikolaus CJ, An R, Ellison B, Nickols-Richardson SM. Food insecurity among college students in the United States: a scoping review. Advances in Nutrition. 2020;11(2):327-348. https://doi.org/10.1093/advances/nmz111 2. Swipe Out Hunger. https://www.swipehunger.org 3. United States Department of Agriculture, Food and Nutrition Service. SNAP eligibility for students. https://www.fns.usda.gov/snap/students Chapter 7: Calming the Body Fast 1. von der Embse N, Jester D, Roy D, Post J. Test anxiety effects, predictors, and correlates: a 30-year meta-analytic review. Journal of Affective Disorders. 2018;227:483-493. https://doi.org/10.1016/j.jad.2017.11.048 2. Balban MY, Neri E, Kogon MM, et al. Brief structured respiration practices enhance mood and reduce physiological arousal. Cell Reports Medicine. 2023;4(1):100895. https://doi.org/10.1016/j.xcrm.2022.100895 3. Zaccaro A, Piarulli A, Laurino M, et al. How breath-control can change your life: a systematic review on psycho-physiological correlates of slow breathing. Frontiers in Human Neuroscience. 2018;12:353. https://doi.org/10.3389/fnhum.2018.00353 4. Lehrer PM, Gevirtz R. Heart rate variability biofeedback: how and why does it work? Frontiers in Psychology. 2014;5:756. 5. Jamieson JP, Mendes WB, Blackstock E, Schmader T. Turning the knots in your stomach into bows: reappraising arousal improves performance on the GRE. Journal of Experimental Social Psychology. 2010;46(1):208-212. https://doi.org/10.1016/j.jesp.2009.08.015 6. Jamieson JP, Peters BJ, Greenwood EJ, Altose AJ. Reappraising stress arousal improves performance and reduces evaluation anxiety in classroom exam situations. Social Psychological and Personality Science. 2016;7(6):579-587. https://doi.org/10.1177/1948550616644656 7. Brooks AW. Get excited: reappraising pre-performance anxiety as excitement. Journal of Experimental Psychology: General. 2014;143(3):1144-1158. https://doi.org/10.1037/a0035325 8. Ramirez G, Beilock SL. Writing about testing worries boosts exam performance in the classroom. Science. 2011;331(6014):211-213. https://doi.org/10.1126/science.1199427 Chapter 8: Exam Day and the Days After 1. Depner CM, Melanson EL, Eckel RH, et al. Ad libitum weekend recovery sleep fails to prevent metabolic dysregulation during a repeating pattern of insufficient sleep and weekend recovery sleep. Current Biology. 2019;29(6):957-967. https://doi.org/10.1016/j.cub.2019.01.069 2. National Institute on Alcohol Abuse and Alcoholism. Drinking levels defined. https://www.niaaa.nih.gov/alcohol-health/overview-alcohol-consumption/moderate-binge-drinking 3. National Highway Traffic Safety Administration. Drowsy driving. https://www.nhtsa.gov/risky-driving/drowsy-driving Further Reading On learning and memory Peter C. Brown, Henry L. Roediger III, and Mark A. McDaniel, Make It Stick: The Science of Successful Learning (Harvard University Press, 2014). The most accessible book-length account of retrieval practice, spacing, and interleaving, written by two of the researchers whose work underpins Chapter 3. Benedict Carey, How We Learn: The Surprising Truth About When, Where, and Why It Happens (Random House, 2014). A science journalist's lively tour of learning research, including useful chapters on sleep and on why cramming misleads. Retrieval Practice (retrievalpractice.org) and The Learning Scientists (learningscientists.org). Two free websites run by cognitive scientists, with short guides and downloadable materials on evidence-based study strategies. On stress, performance, and anxiety Sian Beilock, Choke: What the Secrets of the Brain Reveal About Getting It Right When You Have To (Free Press, 2010). An account of why people underperform under pressure, from one of the authors of the expressive-writing study in Chapter 7. Kelly McGonigal, The Upside of Stress: Why Stress Is Good for You, and How to Get Good at It (Avery, 2015). A readable introduction to research on stress mindsets and reappraisal, best read alongside the more cautious framing in this book. On sleep Colleen E. Carney and Rachel Manber, Quiet Your Mind and Get to Sleep (New Harbinger, 2009). A self-help workbook based on cognitive behavioural therapy for insomnia, useful for students whose sleep problems persist beyond exam periods. American Academy of Sleep Medicine, Sleep Education (sleepeducation.org). Reliable, plain-language information on healthy sleep and on sleep disorders, from the professional society for sleep medicine. On caffeine, supplements, and substances United States Food and Drug Administration consumer updates on caffeine and on dietary supplements (fda.gov). Short, authoritative pages on safe caffeine intake, concentrated caffeine products, and the regulation of supplements. Help and support 988 Suicide and Crisis Lifeline (988lifeline.org). Call or text 988 in the United States for free, confidential support at any time. Swipe Out Hunger (swipehunger.org). A national nonprofit working on college student food insecurity, with information on campus meal-sharing programs. Hashtags: #FinalsWeek #FinalExamPreparation #StudentWellness #HolisticStudentHealth #ExamSuccess #HealthyStudyHabits #SleepForLearning #SleepHygiene #MemoryConsolidation #BrainFood #BrainHealthyNutrition #StudentNutrition #HealthyEatingForStudents #HydrationForFocus #StressReduction #ExamStressManagement #TestAnxietyRelief #MindfulnessForStudents #BreathingExercises #AcademicPerformance #EffectiveStudyTechniques #ActiveRecall #RetrievalPractice #SpacedRepetition #TimeManagementForStudents #AvoidAllNighters #CaffeineAwareness #MentalWellbeing #ExamDayPreparation #PostExamRecovery

Download the Book (PDF): Introduction On 21 April 2015, on a test guideway cut through the mountains of Yamanashi Prefecture west of Tokyo, a seven-car train owned by the Central Japan Railway Company reached 603 kilometres per hour. Nobody on board felt a wheel turn, because no wheel was touching anything. The train was floating about ten centimetres above the floor of a concrete trough, held there by magnetic fields from coils cooled to within a few degrees of absolute zero, and pushed forward by a travelling magnetic wave that the guideway itself generated. The run was the fastest any passenger rail vehicle has ever gone. It was also, in a sense, unremarkable to the engineers who arranged it. They had been running trains above 500 km/h on that line for years. The record was a demonstration; the real work was the decades of test running that preceded and followed it. Eight years earlier, on a stretch of the new eastern French high-speed line near the Meuse, a specially prepared TGV had run at 574.8 km/h on steel wheels. That record was a different kind of achievement. To make it possible, the French engineers raised the voltage in the overhead wire, increased the tension in the contact wire well beyond what everyday service uses, fitted larger wheels, and prepared a single stretch of track with exceptional care. The train did what it was asked to do for a few minutes, and then everyone went home. Nobody has ever proposed running a scheduled wheeled train anywhere near that speed. The gap between those two stories is what this book is about. Ground transport at 500 km/h is no longer science fiction; it has been demonstrated on wheels and on magnets, and one line built for it, the Chuo Shinkansen between Tokyo and Nagoya, is under construction. Yet the practical ceiling for scheduled service on conventional high-speed railways has sat at roughly 300 to 350 km/h for more than a decade, and the most ambitious current wheeled train, China's CR450, is designed to run at about 400 km/h. Why is the step from 350 to 500 so much harder than the step from 200 to 350 was? And what exactly has to change to make it? The argument of this book is that the answer lies almost entirely at the boundaries of the train. At 500 km/h, the difficult problems are not inside the vehicle. Motors, power electronics, brakes and passenger comfort systems are mature and scale reasonably well. The hard problems sit at the three places where a moving train must touch its surroundings: where it is held up, where it pushes through the air, and where it draws its electricity. Each of those interfaces is governed by physics that worsens sharply with speed, often with the square, the cube, or an even higher power of velocity. And each has a natural limit set by some characteristic speed of the system itself: the speed of a travelling wave in the contact wire, the speed at which a wheel stops gripping the rail reliably, the speed at which a pressure pulse in a tunnel steepens into something like a sonic boom. Engineering progress at the top end has come less from brute force than from changing which physical regime the interface operates in, so that the punishing scaling law is replaced by a kinder one. That idea shapes the structure of the chapters. The first sets out the scaling laws themselves: how resistance, power, noise and pressure grow with speed, and what that means for the cost of every extra kilometre per hour. The second asks how far the wheel can go, and why the answer is close to 400 km/h for regular service even though wheels have been pushed far beyond that for a few minutes at a time. The third and fourth turn to magnetic levitation, first in its attractive, actively controlled form, as used on the German Transrapid and the Shanghai airport line, and then in the superconducting, repulsive form that Japan has developed for the Chuo Shinkansen. These chapters spend time on mechanism, because the reasons maglev behaves as it does are physical and can be understood without equations. The fifth and sixth chapters deal with air. In the open, the problem is drag and noise, and the dominant noise sources at high speed are not where most people would guess. In tunnels, the problem becomes something stranger: a train entering a bore acts like a piston, launching a pressure wave that runs ahead at the speed of sound and, if conditions are wrong, emerges from the far portal as a loud bang that rattles windows in nearby houses. Japan discovered this phenomenon on its own network in the 1970s, and the long, sculpted noses of modern Japanese trains are largely a response to it. For a maglev line that runs mostly underground, tunnel aerodynamics is not a side issue; it shapes the vehicle, the tunnel bore, and the cost of the whole project. The seventh and eighth chapters deal with electricity, and specifically with the oddly delicate business of collecting tens of megawatts from a copper wire through a sliding contact at hundreds of kilometres per hour. The overhead wire is not a rigid rail; it is a tensioned string that carries waves, and the pantograph pressing up against it is a small mass-spring system riding on that string. When the train approaches the speed of the waves it creates, the wire behaves in ways that make steady contact impossible. This is one of the least appreciated limits on wheeled high-speed rail, and one of the reasons superconducting maglev, which delivers propulsion power through the guideway instead, sidesteps the problem entirely. A few boundaries of scope are worth stating. This is a book about physics and engineering, not about transport policy, although the conclusion draws out what the physics implies for decisions about whether and where to build. It concentrates on the systems that have actually been built and tested at very high speed: the Japanese Shinkansen and SCMaglev, the French TGV, the German ICE and Transrapid, and the Chinese high-speed network and its recent maglev and wheeled test programmes. It treats vacuum-tube concepts briefly and sceptically, because their central difficulties lie outside the three interfaces discussed here and because their record so far has been one of announcements rather than service. It avoids mathematics beyond a few proportionalities and worked numerical examples, which are labelled where they use illustrative round numbers rather than published figures. The reader who finishes the book should be able to look at a high-speed train and read its design: why the nose is the length it is, why the pantograph has the shape it does and sits where it sits, why the tunnel portals on some lines have strange perforated hoods, why the Chuo Shinkansen guideway is a U-shaped trough rather than a pair of rails, and why a train that is perfectly happy at 320 km/h can be a very poor neighbour at 400. More importantly, that reader should be able to judge claims about the next generation of fast ground transport by asking the right questions. When someone promises a faster train, the useful response is not to ask how powerful its motors are. It is to ask what is holding it up, what the air is doing around it, and how it gets its power. Chapter 1: The Arithmetic of Speed Every argument about very fast trains eventually comes back to a handful of proportionalities. They are simple enough to state in a sentence each, and between them they explain why a railway that runs comfortably at 300 km/h does not become a 500 km/h railway by fitting bigger motors. The purpose of this chapter is to lay those relationships out plainly, attach real numbers to them where real numbers exist, and show how they combine. Later chapters examine each interface in detail; here the aim is to see the whole shape of the problem at once. Resistance, power and the tyranny of the square A train moving along a level track has to overcome resistance from three broad sources. Some resistance does not depend on speed at all: the rolling friction of wheels on rails, the losses in bearings, the small amount of energy dissipated by deforming the contact patch between steel and steel. Some grows in proportion to speed: flange contact, some mechanical losses, and the momentum given to air drawn into the train for cooling and ventilation. And some grows with the square of speed: aerodynamic drag. Railway engineers have summarised this for over a century with what is called the Davis equation, after the American engineer W. J. Davis, who proposed it in 1926. It writes the resistance of a train as a constant term, plus a term proportional to speed, plus a term proportional to speed squared. The coefficients differ from train to train and have to be found by testing, usually by letting a train coast from high speed and measuring how fast it slows. The form of the equation, though, is the same for a freight train and a Shinkansen. At low speed the constant term dominates. At high speed the squared term takes over completely. For a modern high-speed train at 300 km/h, aerodynamic drag typically accounts for somewhere around three quarters or more of total resistance, and the share keeps rising as speed increases. By 400 km/h the train is, to a good approximation, an object whose resistance is set by the air. The square law for force is bad; the law for power is worse. Power is force multiplied by speed. If drag force rises with the square of speed, the power needed to overcome it rises with the cube. Doubling speed requires roughly eight times the power just to hold that speed against the air. A worked example makes the consequences concrete. The numbers here are illustrative round figures of the right order for a modern eight-car high-speed train, not the measured data of any particular model. Suppose that at 300 km/h, or 83 metres per second, the train meets a total resistance of 50 kilonewtons, of which 40 kilonewtons is aerodynamic and 10 kilonewtons is mechanical. The power needed at the wheel is force times speed: 50,000 newtons times 83 metres per second, or about 4.2 megawatts. Now ask the same train to cruise at 500 km/h, about 139 metres per second. The speed ratio is five thirds. The aerodynamic force rises by the square of that, a factor of about 2.8, to roughly 111 kilonewtons. Let the mechanical part rise modestly to, say, 12 kilonewtons. Total resistance is now about 123 kilonewtons, and the power required is 123,000 times 139, or about 17 megawatts. A two-thirds increase in speed has quadrupled the continuous power demand. Energy per trip tells a slightly gentler story, because a faster train spends less time on the journey. The energy to cover a fixed distance against a resistive force is force times distance, so energy per kilometre rises with the square of speed rather than the cube. In the example above, the energy needed per kilometre of cruising rises from about 50 megajoules to about 123, a factor of roughly 2.5. That is the honest figure to use when comparing the running cost of a 300 km/h service with a 500 km/h one on the same alignment: around two and a half times the traction energy for the cruising portion, before counting the extra energy spent accelerating to the higher speed. This is why the most effective high-speed engineering of the last forty years has gone into reducing the drag coefficient rather than increasing power. Every percentage point removed from aerodynamic drag at the top end is worth more than the same percentage added to installed power, because the power has to be bought, carried, cooled, and supplied through the wire, while the drag reduction is free once the shape is made. When JR Central introduced its improved L0 maglev prototype in 2020, the headline claim was not more thrust but a front end reshaped to cut air resistance by about 13 percent compared with the earlier design. Every effect has its own exponent Drag is only the most familiar of the speed-dependent effects. Almost every phenomenon that matters at the top end of the speed range has its own exponent, and a few have exponents high enough to change the character of the problem completely. Table 1 collects the main ones. The exponents are the standard approximate scaling relationships found in the engineering literature on train aerodynamics, railway noise and pantograph–catenary interaction; the real dependence is always messier, but the approximate power law is what governs design decisions. Table 1. How key effects grow with train speed (approximate scaling). Effect Approximate scaling with speed Doubling speed multiplies it by Why it matters Aerodynamic drag force v² 4 Sets traction force and brake loads Power to overcome drag v³ 8 Sets installed power and wire current Energy per km against drag v² 4 Sets running cost Kinetic energy to absorb in braking v² 4 Sets brake capacity and stopping distance Aerodynamic noise (dipole sources) about v⁶ in sound power 64 (about 18 dB) Dominates noise above roughly 300 km/h Tunnel micro-pressure wave at portal about v³ 8 Can produce audible booms at tunnel exits The noise row deserves special attention. Aerodynamic noise from unsteady flow around bluff bodies, such as a pantograph or the edges of a bogie cavity, comes mostly from what acousticians call dipole sources. The sound power radiated by such sources rises roughly with the sixth power of flow speed. In decibel terms, doubling speed adds about 18 dB to aerodynamic noise, a change that the human ear perceives as several times louder. Rolling noise from wheels and rails, by contrast, rises far more slowly, typically by around 9 dB per doubling of speed under the common 30 log v rule. The two curves cross. Below about 300 km/h on a well-maintained line, wheel-rail rolling noise is usually the largest source a trackside listener hears. Above it, aerodynamic noise from the pantograph, the nose, the bogie areas and the gaps between cars takes over and climbs steeply. The crossover point varies with the train and the track, but its existence is the reason that environmental noise, rather than any mechanical limit, has been what actually stopped Japanese operators from running faster on existing lines. The tunnel row is also unusual. When a train enters a tunnel it compresses the air ahead of it, sending a pressure wave down the bore at the speed of sound. The steepness of that wave front, and with it the strength of the pulse that radiates out of the far portal, rises with roughly the cube of the train's entry speed. In a long tunnel with smooth concrete slab track, the front can steepen further as it travels, and what emerges is an impulsive bang. A train running 25 percent faster produces a portal pulse almost twice as strong. Chapter 6 deals with this phenomenon in detail; for now the point is that it is one more cost that rises faster than speed itself. Characteristic speeds: when the scaling law breaks Power laws describe how things get gradually worse. The deeper limits on high-speed rail arise where something does not merely get worse but changes its behaviour, because the train's speed approaches some characteristic speed of the system it is interacting with. There are several such speeds, and much of the rest of this book is organised around them. The first is the speed of transverse waves in the overhead contact wire. A tensioned wire carries waves at a speed set by the tension divided by the wire's mass per metre, under a square root. For the tensions and wire sizes used on high-speed lines this works out at roughly 500 to 600 km/h. A pantograph pressing upward against the wire creates a bump that travels away from it in both directions at that wave speed. As the train approaches the wave speed, the bump ahead of the pantograph cannot escape fast enough, the wire's response bunches up in front of the contact point, and steady contact becomes impossible. This is closely analogous to an aircraft approaching the speed of sound. The rule adopted in practice is to keep operating speed comfortably below the wave speed, usually at no more than about 70 percent of it. Raising the line speed of a catenary therefore means raising its wave speed, which means raising the tension or lightening the wire, and both have limits set by the strength of copper alloys. Chapters 7 and 8 take this up. The second characteristic speed belongs to the wheelset itself. A railway wheelset with coned or profiled treads steers itself along the track, but above a certain speed that self-steering motion becomes unstable and the wheelset begins to oscillate from side to side in a sustained motion called hunting. The critical speed depends on the wheel profile, the stiffness of the suspension that ties the wheelset to the bogie frame, and the dampers fitted between bogie and car body. Good design pushes the critical speed comfortably above the maximum operating speed. Every increase in operating speed demands more stiffness or more damping, and stiffer suspensions make the wheel steer less well in curves and wear itself and the rail more. Chapter 2 discusses the trade. The third is the speed of sound itself, which matters in tunnels. A train at 500 km/h is moving at about 0.4 of the speed of sound in air. That is not a supersonic problem, but it is far enough up the scale that the compressibility of air is no longer negligible inside a confined bore. The pressure waves generated at tunnel entry, the waves reflected from portals, and the waves generated when trains pass each other underground all add together in ways that depend on train speed, tunnel length and the timing of entry. Chapter 6 shows that this sets a real constraint on tunnel diameter and vehicle shape at maglev speeds. There is a fourth characteristic speed that applies to some track structures: the speed of waves travelling through the ground beneath the railway. On very soft soils, the speed of the Rayleigh waves that run along the ground surface can be as low as a couple of hundred kilometres per hour. A train approaching that speed drives a large amplification of track deflection, an effect first observed on a soft-soil section of the West Coast line in Sweden in the late 1990s when X2000 trains running at around 200 km/h produced alarming track movement. It is solved by stiffening the ground, usually with lime-cement columns or by building on viaducts, and is a construction cost rather than a fundamental limit. Many new high-speed lines, particularly in China, sidestep it by carrying most of the route on long viaducts with piled foundations. The integrated picture Put these effects together and a pattern emerges. Between 200 and roughly 320 km/h, the cost of going faster is real but incremental. Drag and power rise steeply, but the motors and the supply network can deliver them; noise rises, but mainly from rolling sources that can be managed with better track and wheel maintenance; tunnel waves exist but can be tamed by moderate nose lengths and sealed car bodies; and the catenary operates well below its wave speed. Between 320 and 400 km/h, every one of those curves starts to bite at once. Aerodynamic noise overtakes rolling noise and begins its 18 dB-per-doubling climb. The catenary must be re-tensioned and the pantograph redesigned to stay well below the wave speed. Tunnel portals begin to need hoods, or trains need longer noses that cost seating capacity. Wheel and rail wear increases. Japan's ALFA-X test train ran at up to 400 km/h from 2019, but the production train being developed from it, the E10 series due to enter service on the Tohoku Shinkansen around 2030, is planned for 320 km/h rather than the 360 km/h its developers once hoped for. China's CR450 has been tested at speeds above 450 km/h but is designed for commercial operation at about 400 km/h, with rollout planned for selected sections of its busiest corridors. Above 400 km/h, at least two of the interfaces break down on a conventional railway. The catenary's wave speed is too close, and the noise and tunnel pressure pulses become very hard to keep within environmental limits without tunnelling or covering large parts of the route. That is the zone where the case for changing the physics, rather than fighting it, becomes strong. A superconducting maglev changes the support interface, abolishing wheel-rail contact and hunting. It changes the power interface, abolishing the pantograph and the catenary wave limit. What it cannot change is the air. That is why the most important engineering on the Chuo Shinkansen concerns aerodynamics and tunnels, and why the maglev's advantage over the wheel is real but narrower than its enthusiasts often suggest. One more quantity needs to be added to the picture before going further: time saved. The benefit of higher speed is measured in minutes, and minutes saved per kilometre fall as speed rises. Covering 100 km at 300 km/h takes 20 minutes; at 400 km/h, 15 minutes; at 500 km/h, 12 minutes. Each 100 km/h step saves less time than the one before, while costing more energy, more noise mitigation and more structure. On a 300 km route with intermediate stops, acceleration and braking also eat into the gain, because a train limited by adhesion or passenger comfort spends a long time reaching its top speed. A train accelerating at a comfortable 0.5 metres per second squared needs nearly five minutes and almost 20 km just to reach 500 km/h. Very high speeds pay off only on long, straight, sparsely stopped routes. That arithmetic is what explains the choices the world's railways have actually made. It explains why Japan built a maglev for its single densest intercity corridor, where a few minutes multiplied by tens of millions of passengers a year justify extraordinary expense, rather than across its whole network. It explains why France, which held the wheeled world record for decades, settled at 320 km/h for service. And it explains why the Shanghai airport maglev, a 30 km line, cut its top service speed from 431 km/h to 300 km/h in 2021: on a route that short, the extra speed bought less than a minute, since the scheduled trip at 300 km/h still takes just over eight minutes, while costing energy and wear every trip. The chapters that follow look at each interface in turn, starting with the oldest and most familiar one: the steel wheel on the steel rail. Chapter 2: The Limits of the Wheel The steel wheel on a steel rail is one of the most efficient ways ever devised to move a heavy load over land. The contact patch between a wheel and a rail is about the size of a small coin, yet it carries loads of several tonnes with very little rolling loss, because hard steel deforms so little. The same property that makes the wheel efficient makes it unforgiving at high speed: there is almost no compliance in the contact itself, so every irregularity in the rail, every imbalance in the wheel, every lateral motion of the wheelset is transmitted directly into forces. This chapter asks how far that system can be pushed, and where the ceiling for regular passenger service actually lies. A century of records Electric trains passed 200 km/h remarkably early. In 1903, experimental three-phase railcars built by Siemens and AEG reached about 210 km/h on a military railway between Marienfelde and Zossen, south of Berlin. That speed was not exceeded for decades, partly because nobody had a commercial reason to go that fast and partly because the track of the time could not sustain it. The modern story begins in France in March 1955, when two electric locomotives, CC 7107 and BB 9004, each reached about 331 km/h on a straight section of line in the Landes. The runs were famous at the time, and among engineers they are remembered for what they did to the overhead line and the pantographs: the contact strips were badly damaged by arcing, the catenary was wrecked, and the track was left distorted. They showed that the train could be propelled to such speeds but that the interfaces could not survive them. The sequence of records since then is, in effect, a history of fixing those interfaces one by one. Table 2 lists the main record-setting runs on steel wheels and the conditions under which they were made. Table 2. Selected speed records and test runs on steel wheels. Year Train Speed (km/h) Notable conditions 1955 SNCF CC 7107 and BB 9004 about 331 Severe pantograph and catenary damage 1981 TGV Sud-Est set 16 380 Shortened set on the new Paris–Lyon line 1988 ICE-V (Germany) 406.9 First wheeled train above 400 km/h 1990 TGV Atlantique set 325 515.3 29.5 kV supply, 32 kN contact wire tension 1996 JR Central 955 (300X) 443.0 Japanese test train on the Tokaido line 2007 V150 (TGV POS power cars, Duplex cars) 574.8 31 kV supply, 40 kN contact wire tension 2010 CRH380A 486.1 Test run on the Beijing–Shanghai line 2023 CR450 development test train 453 Single-train test on the Fuzhou–Xiamen line The French figures are drawn from SNCF and Alstom accounts of the record programmes; the Chinese figures are those reported by China State Railway Group at the time. The pattern is worth noticing. Each record run was a special event with special preparations: modified trains, shortened consists, raised voltages, re-tensioned wires, extra-large wheels, and a carefully chosen and maintained section of track. The 2007 V150 run used two power cars from the TGV POS fleet and three modified double-deck cars, with motors under the trailers as well, for a total output of about 19.6 megawatts, nearly twice that of a standard TGV. The wheels were enlarged to 1,092 millimetres to keep motor speeds within limits. The overhead line voltage was raised from the standard 25 kilovolts to 31, and the contact wire tension from 25 to 40 kilonewtons. None of those measures is suitable for daily service. Raised voltage stresses insulation throughout the line and the train. A contact wire at 40 kilonewtons is pulled close to the point where repeated bending and wear would make it prone to fatigue failure. A shortened train is useless for carrying passengers. The records tell us what is physically possible for a few minutes; they say very little about what is sustainable for twenty years and many millions of kilometres. Adhesion, hunting and the bogie Two mechanical limits matter most for the wheel at very high speed: the grip available between wheel and rail, and the stability of the wheelset. Grip, or adhesion, is what lets a train accelerate and brake through its wheels. It is usually expressed as a coefficient: the ratio of the largest longitudinal force the wheel can transmit to the vertical load it carries. On clean, dry rail at low speed, the coefficient can exceed 0.3. On wet rail, contaminated with leaves or fine debris, it can fall below 0.1. More awkwardly, adhesion falls as speed rises, because at high speed water and contaminants have less time to be squeezed out of the contact patch, and the contact itself behaves differently. Railway designers therefore assume much lower adhesion at high speed than at low. Two consequences follow. The first concerns traction. At 300 km/h a train needs relatively little extra force to accelerate, but at 500 km/h the tractive force required merely to hold speed against drag is large, and the force available from the wheels is limited by adhesion times the weight on the driven axles. A high-speed train with distributed traction, such as a Shinkansen or a modern Chinese CR-series train, spreads its motors along most of its axles, so that the adhesion of many wheels is used. That is one reason virtually all very fast trains now use distributed rather than concentrated power: the V150 record train needed powered axles under its trailers to get enough grip. The second consequence concerns braking. A train braking from 500 km/h must dissipate four times the kinetic energy of one braking from 250 km/h, and if adhesion is low it cannot rely on the wheels alone. Non-adhesion brakes, which act directly between the train and the rail or the air, become important. ALFA-X carried both eddy-current brakes, which induce currents in the rail to create a retarding force without contact, and aerodynamic brake panels on the roof that open like small wings to add drag in an emergency, a feature that also helps in an earthquake stop. Hunting is the other limit. A railway wheelset is not a simple pair of wheels on an axle; the treads are profiled so that the rolling radius is larger near the flange and smaller towards the outside. When the wheelset drifts to one side, the wheel on that side rolls on a larger radius than the other, travels further per revolution, and steers the axle back towards the centre. This elegant mechanism is what lets a train follow the track without the flanges grinding constantly against the rails. But it is a feedback system, and like many feedback systems it can oscillate. Above a critical speed, the self-steering overshoots, the wheelset swings to the other side, overshoots again, and settles into a sustained side-to-side motion that grows until the flanges strike the rails. At its worst, hunting can damage track and, in extreme cases, contribute to derailment. The critical speed is controlled by the stiffness with which the wheelset is held in the bogie frame, the profile of the wheel and rail, and, crucially, the yaw dampers fitted between the bogie and the car body. These dampers resist rapid twisting of the bogie relative to the body and are the main tool for raising the critical speed of modern high-speed trains. But stiffer wheelset guidance makes the wheelset steer less freely on curves, increasing wheel and rail wear and the forces exerted on the track. Wheel profiles also wear in service, changing the effective conicity and with it the critical speed. A high-speed operator therefore lives with a tight maintenance discipline: wheels must be reprofiled on lathes at regular intervals to keep the geometry within the band where the train is stable at full speed. The faster the line, the narrower that band becomes. The track as a precision instrument At 500 km/h a train covers 139 metres every second. A dip in the rail one millimetre deep and ten metres long, a defect too small to see, becomes a vertical disturbance at a frequency of about 14 hertz, close to the frequencies at which the unsprung masses of a bogie and the track itself resonate. Every geometric imperfection in the track is converted by speed into vibration, and the frequencies climb into ranges that are harder to isolate from passengers and more damaging to rail fastenings and ballast. Ballasted track has a specific problem at very high speed. The aerodynamic suction under a fast train can lift individual stones from the ballast bed, especially in winter when ice falls from the underside of the train and strikes the ground. A stone thrown up at 300 km/h can damage the underside of the train, the rail, or a passing train. The phenomenon, known as ballast flight, was a significant concern during French and Belgian high-speed trials and on some Spanish lines, and the usual mitigation is to keep the ballast surface lower than the sleeper tops, or to replace ballast with concrete slab track. Japan and China have built most of their high-speed lines on slab track, which also holds its geometry with far less maintenance. Slab track has its own cost and, as later chapters explain, makes tunnel pressure waves steepen more than ballasted track does. The structure beneath the track matters too. Bridges and viaducts have natural frequencies, and a train whose axles cross a span at regular intervals excites the bridge at a frequency set by train speed divided by the axle spacing. If that frequency coincides with a natural frequency of the bridge, the deck can resonate, with accelerations large enough to destabilise ballast. European design rules limit deck acceleration on ballasted bridges for this reason, and dynamic analysis of every bridge at speeds well above the line speed is part of the design of any new high-speed railway. A line designed for 350 km/h cannot simply be run at 450 without checking every structure along it. Track alignment imposes the last and most expensive constraint. Lateral acceleration in a curve rises with the square of speed divided by curve radius. To keep passengers comfortable at 500 km/h with practical amounts of cant, or banking, curves need radii of the order of 8 kilometres. The Chuo Shinkansen, built for 500 km/h, has a minimum curve radius of 8,000 metres on its main line, according to JR Central's published route plans. A line designed for 300 km/h can get by with radii around 4 to 5 kilometres. In mountainous or densely built terrain, the extra straightness demanded by higher speed translates directly into extra tunnels and viaducts, which is why very fast lines are so expensive to build. Earthquakes and the wheel In Japan, and in other seismically active countries such as Taiwan, Turkey and parts of China, one more hazard bears directly on how fast a wheeled train can safely run. A strong earthquake shakes the track sideways, sometimes by tens of centimetres within a second or two. A wheel held to the rail only by a flange a few centimetres high can be lifted off. For decades the Shinkansen avoided any in-service derailment. That record ended on 23 October 2004, when the Chuetsu earthquake struck Niigata Prefecture while a Joetsu Shinkansen train, Toki 325, was running at about 200 km/h. Eight of its ten cars left the rails. Nobody was killed or seriously injured, partly because the train stayed broadly aligned with the track and partly because it did not strike a structure or another train. The investigation that followed prompted two kinds of response. The first was to keep derailed trains close to the track: JR East fitted L-shaped guides beneath its bogies, designed to catch on the rail and stop a derailed car from sliding far sideways, and other operators adopted guard rails or similar devices on vulnerable sections. The second was to stop trains sooner. Japan's railways had used seismic detection since the 1990s, but the systems were extended and accelerated, so that sensors along the coast and inland detect the first, faster but weaker primary waves of a large earthquake and cut power to trains, triggering emergency braking, before the stronger secondary waves arrive. The physics of that race is unforgiving at higher speeds. Primary waves travel through the ground at several kilometres per second, secondary waves at roughly half that, so the warning interval for a nearby earthquake is a few seconds to a few tens of seconds. A train braking from 320 km/h needs several kilometres and well over a minute to stop. Every second of warning removes some speed before the shaking arrives, but a train at 400 km/h has a third more kinetic energy to shed than one at 350 km/h and covers more ground in the warning interval. A second derailment, of a Tohoku Shinkansen train during the March 2022 Fukushima earthquake, again caused no deaths but showed that the problem had not gone away. The ALFA-X test programme included vibration dampers and other measures aimed specifically at keeping the train on the rails during earthquakes, and its roof-mounted aerodynamic brakes were developed partly to shorten emergency stopping distances when adhesion alone cannot. This is one reason the large clearance of a superconducting maglev, discussed in Chapter 4, carries so much weight in Japanese engineering judgement. A vehicle contained in a U-shaped trough with ten centimetres of magnetic clearance on each side and passive guidance that needs no power cannot leave its guideway in the way a wheel leaves a rail. Where the practical ceiling lies Bring these limits together and a reasonably clear picture emerges of where steel-wheel service tops out. The mechanical problems of adhesion, hunting and track quality can be managed at 400 km/h with today's knowledge, at a cost in maintenance. China's CR450 programme is the most serious current attempt to establish 400 km/h as a commercial speed. The prototypes, CR450AF and CR450BF, were unveiled in late 2024, and by August 2026 had completed some 600,000 kilometres of operational testing on the Chongqing–Xiamen high-speed railway, with further trials planned for the Xiong'an–Shangqiu line before certification. They are intended for corridors including Beijing–Shanghai, initially on selected sections. China's own statements describe the programme as reducing weight, running resistance and noise relative to the existing CR400 Fuxing fleet, which already operates at 350 km/h; that combination is exactly what the arithmetic of Chapter 1 demands. Japan, with its stricter environmental noise standards and denser lineside population, has drawn the ceiling lower. The ALFA-X test programme on the Tohoku Shinkansen explored running up to 400 km/h and studied noise, pantograph behaviour, earthquake stability, snow accumulation and braking, but the production E10 series that follows from it is planned for 320 km/h. Europe has largely settled at 300 to 320 km/h for new trains and lines, judging that the time savings above that speed do not justify the extra energy, maintenance and environmental cost across a network of mixed distances. The wheel, then, can go a long way. It is not the mechanics of the wheel itself that set the ceiling; with enough care, wheels can be made stable and grippy enough for 400 km/h service and, for minutes at a time, nearly 575 km/h. What sets the ceiling is the combination of everything else: the overhead line's wave speed, the noise from the pantograph and running gear, the tunnel pressure waves, the bridge dynamics, the curve radii, and the escalating cost of maintaining a track to sub-millimetre accuracy. Each can be pushed back, but each costs more as speed rises, and around 400 km/h they converge. If the goal is 500 km/h for daily service, there are two options. One is to keep the wheel and solve each of those problems separately and expensively. The other is to remove the wheel entirely and replace the contact between train and track with a magnetic field. That second option removes adhesion limits, hunting, rolling noise, and, depending on the design, the overhead wire. It also introduces new problems of its own. The next two chapters examine the two ways of doing it. Chapter 3: Floating by Attraction There are two fundamentally different ways to hold a vehicle off the ground with magnets. One pulls the vehicle upward towards a steel rail, using electromagnets whose strength is adjusted thousands of times a second to keep a small gap constant. The other pushes the vehicle upward away from the guideway, using currents induced by the motion of very strong magnets past conducting coils. The first is called electromagnetic suspension, or EMS, and is the basis of the German Transrapid, the Shanghai airport line and most of China's maglev programme. The second is called electrodynamic suspension, or EDS, and is the basis of Japan's superconducting maglev. They differ in almost every respect that matters: how the vehicle behaves at rest, how large the gap is, what the guideway must look like, how the system fails, and how it scales to high speed. This chapter deals with attraction; the next with repulsion. Why a magnet cannot simply hold a train It seems natural to imagine levitation as a matter of placing magnets of the right polarity under a vehicle and letting them float. It does not work, and the reason is a nineteenth-century result in physics known as Earnshaw's theorem. In plain terms, the theorem says that no arrangement of fixed magnets or static charges can hold an object in stable equilibrium by those forces alone. There is always some direction in which a small displacement makes the force push the object further away from its balance point. A bar magnet floating above another bar magnet will flip or slide off. A steel plate drawn up towards an electromagnet will either fall away or slam into it. The attractive case is especially stark. The pull between an electromagnet and a steel rail grows rapidly as the gap shrinks, roughly with the inverse square of the gap for a given current. Imagine a vehicle hanging beneath a steel rail by electromagnets. If the gap is slightly too large, the pull weakens and the vehicle drops further. If the gap is slightly too small, the pull strengthens and the vehicle rises until it strikes the rail. There is a balance point, but it is unstable, like a pencil balanced on its tip. The solution is active control. A gap sensor measures the distance between the magnet and the rail many times a second. A controller compares it with the desired value and adjusts the current in the magnet: more current if the gap is opening, less if it is closing. Done fast enough and with enough authority, this turns an unstable system into a stable one, in the same way that a person can balance a broomstick on a fingertip by constantly moving the hand. The idea was patented by the German engineer Hermann Kemper in 1934, but it could not be implemented practically until fast power electronics and reliable sensors arrived in the 1960s and 1970s. The consequences of this design choice run through everything else about an EMS system. Because the force is attractive, the magnets must pull the vehicle up towards the guideway from below. That means the vehicle wraps around the guideway beam, with its levitation magnets tucked underneath a steel rail on the underside of the beam's overhanging edges. Because the controller's job is easier when the gap is small, and because the magnet's force falls off quickly with distance, the gap is small: about 10 millimetres on the Transrapid. Because the system levitates through active control rather than through motion, it levitates at any speed, including standing still. And because everything depends on the controller, the system's safety case rests on redundancy in sensors, controllers, magnets and power supplies, so that the failure of any one does not drop the vehicle. The Transrapid design The Transrapid, developed in West Germany from the late 1960s by a consortium that eventually became a joint venture of Siemens and ThyssenKrupp, is the most fully worked-out EMS system for high speed. Its design shows how the levitation choice shapes the rest of the machine. The guideway is an elevated beam, often of concrete with steel components, whose top surface carries the vehicle's emergency skids and whose sides and underside carry the functional parts. Along the underside of each overhanging edge runs a laminated steel stator pack containing three-phase windings. The vehicle's levitation magnets are arranged beneath these stator packs and pull upward towards them. A separate set of guidance magnets on the vehicle's sides pulls sideways against steel rails on the edges of the beam, keeping the vehicle centred. Each set is controlled independently, and the vehicle is divided into many small magnet modules, each with its own gap control, so that losing a few modules does not compromise levitation. The ingenious part is that the levitation magnets do double duty. The stator packs on the guideway carry a three-phase winding, and when that winding is energised it creates a travelling magnetic field that moves along the guideway. The vehicle's levitation magnets lock onto that travelling field and are dragged along with it. This is a linear synchronous motor, which is essentially an ordinary electric motor whose stator has been cut open and laid flat along the track. The speed of the vehicle is set by the frequency of the current fed into the guideway windings; the thrust is set by the current's magnitude and phase. Because the motor's active, powered part is in the guideway rather than on the vehicle, this arrangement is called a long-stator motor. The long stator has a striking consequence for the power interface: the vehicle needs no pantograph to propel itself. Traction power is delivered to the guideway from substations along the route, which energise only the section of winding the vehicle is currently passing over. The vehicle still needs electricity for its levitation magnets, control systems, lighting and air conditioning. The Transrapid obtained this through linear generators: windings built into the levitation magnets that pick up power from the harmonic ripple in the travelling field as the vehicle moves. Above roughly 100 km/h these generators supplied the onboard needs, with batteries carrying the vehicle through low-speed running and stops, assisted where needed by contact power rails in stations. The design also determines how the guideway must be built. A 10-millimetre gap at high speed leaves very little room for error. Long guideway beams sag and deflect under their own weight, under temperature changes, and under the vehicle's passage; their supports settle with time. The guideway must be built and maintained to tolerances of a few millimetres over long distances, with the stator packs aligned to fractions of a millimetre. The magnet controllers can follow slow, smooth changes in guideway position, but sharp steps or misalignments at beam joints become forcing inputs that the system must reject. This precision makes the guideway expensive and is the main reason Transrapid lines cost far more per kilometre than conventional railways in comparable settings. Emsland, Shanghai and the lessons of operation Germany built a 31.5-kilometre Transrapid test track in Emsland, in the north-west of the country, which opened in the 1980s and was used for development and demonstration runs for over two decades. Trains on that track reached speeds above 450 km/h. Plans for commercial Transrapid lines in Germany, first between Hamburg and Berlin and later between Munich and its airport, were each abandoned over cost. The test track is also the site of the worst maglev accident to date. On 22 September 2006, a Transrapid vehicle running at well over 150 km/h with passengers aboard struck a maintenance vehicle that had been left on the guideway. Twenty-three people were killed. The cause was procedural: the maintenance vehicle's presence on the guideway was not secured against by the operating rules in force, and the dispatcher cleared the train to run. The accident said nothing about the physics of magnetic levitation, but it marked the effective end of German ambitions for the technology. It also made a point that applies to every guided transport system at very high speed: an elevated guideway that a vehicle wraps around cannot be left by anything on it, and the interlocking between operations and maintenance must be as robust as the levitation itself. The one high-speed Transrapid line in commercial service is in Shanghai, connecting Longyang Road metro station with Pudong International Airport. Construction began in March 2001 and revenue service started in 2003. The line is about 30 kilometres long, with only two stations. For most of its life it ran at up to 431 km/h, covering the distance in a little over seven minutes; in November 2003 a test run reached 501 km/h. In 2021 the operator cut the top speed to 300 km/h, lengthening the trip to just over eight minutes. Shanghai is in many ways an engineering success and a transport puzzle. The system has run for more than two decades, carrying tens of millions of passengers with a strong reliability record, and its trains, guideway and long-stator propulsion work as designed. But a 30-kilometre airport shuttle that stops short of the city centre is the wrong application for a 431 km/h vehicle. Most of each journey is spent accelerating and decelerating, and the high top speed buys very little. Ridership was well below capacity in the early years and fell further once Shanghai Metro Line 2 was extended to the airport, giving travellers a cheaper through route into the city. The speed reduction of 2021 was a rational response to the arithmetic of Chapter 1: on such a short route, the energy and wear cost of the last 130 km/h was not worth the minute or so it saved. China's high-speed EMS programme China has taken EMS further than anyone since Germany. It has built low- and medium-speed maglev lines for urban and airport service, such as the Changsha airport link opened in 2016 and Beijing's S1 line opened in 2017, running at about 100 km/h and below. These use short-stator linear induction motors on the vehicle and power rails along the guideway, a simpler and cheaper arrangement suited to modest speeds. For high speed, CRRC, China's state-owned rolling stock manufacturer, unveiled a 600 km/h EMS maglev in Qingdao in July 2021. Its design follows the Transrapid's essentials: attractive levitation with a gap of about 10 millimetres, a wraparound vehicle, and long-stator propulsion in the guideway. The CRRC programme has emphasised a lighter carbon-fibre body, a modified aerodynamic shape, and power electronics able to supply the guideway motor at the higher frequencies needed for 600 km/h. As of late 2026, however, the vehicle has been tested only on short test sections and no long commercial line for it has been approved. Claims in popular media that China already operates a 600 km/h maglev service are not correct. Several Chinese research groups have also pursued very rapid acceleration tests on short tracks, which are best read as experiments in electromagnetic propulsion rather than as demonstrations of passenger systems. In mid-2025 a team at the Donghu laboratory in Hubei reported accelerating a test vehicle of about 1.1 tonnes to 650 km/h within about seven seconds over a 600-metre section of a 1-kilometre track. Late in 2025 researchers at the National University of Defence Technology reported propelling a one-tonne vehicle to 700 km/h in about two seconds on a 400-metre superconducting electric maglev test track. And the state aerospace company CASIC has built a two-kilometre low-vacuum tube test line at Datong in Shanxi, reporting speeds above 620 km/h in early 2024. These are genuine engineering results about drives, sensors and short-duration levitation, but none of them addresses the problems that dominate a real line: aerodynamics in long tunnels, guideway cost per kilometre, switching between tracks, energy consumption over hundreds of kilometres, and safety cases for vehicles carrying hundreds of passengers. What attraction buys and what it costs The EMS approach has real strengths. It levitates at standstill, so the vehicle needs no wheels for low-speed running, and there is no speed range in which the vehicle must transition from rolling to floating. Its magnets are conventional, at room temperature, without the cryogenic plant that superconducting systems require. Its magnetic fields are relatively contained, since the field lines close through the steel rail across a narrow gap, so stray field in the passenger compartment is modest. And the long-stator motor removes the overhead wire. Its weaknesses all follow from the narrow gap and the active control. The guideway must be made and kept extremely precise, which is expensive. Earthquakes, foundation settlement and thermal movements all matter more when the clearance is 10 millimetres. Every vehicle depends on a large number of fast control loops, all of which must keep working; the redundancy and fail-safe behaviour needed to certify that are achievable but demanding. And the switches, where the vehicle must move from one guideway to another, are massive steel beams that bend or slide as a unit, slower to operate than railway points. A rough calculation shows how demanding the control task is. If the pull of a magnet varies with the inverse square of the gap, then closing a 10-millimetre gap by a single millimetre raises the force by the square of ten ninths, about 23 percent; opening it by a millimetre lowers the force by about 17 percent. A vehicle magnet carrying a share of a car's weight therefore sees its force change by a fifth for a one-millimetre error, and the controller must correct that error before it grows. At 430 km/h the vehicle travels about 12 centimetres every millisecond, so a guideway step at a beam joint arrives without warning and must be absorbed within a few milliseconds. This is why EMS gap controllers run at high rates, why the magnet coils and power electronics are designed to change current quickly, and why the guideway beams are manufactured and aligned as precision components rather than as ordinary civil structures. The switching problem deserves a word of its own, because it shapes how an EMS network could be operated. A railway turnout moves two short lengths of rail by a few centimetres. An EMS switch must move an entire section of guideway beam, often tens of metres long and weighing many tonnes, because the vehicle wraps around the beam and cannot be steered off it by any small moving part. Transrapid switches are long steel beams that bend elastically under hydraulic or electric drives to align with one track or the other; the high-speed versions are long enough to be passed at around 200 km/h in the diverging direction. They work reliably, but they take seconds rather than fractions of a second to change, and they are expensive. A dense network with many junctions, of the kind that conventional high-speed railways rely on to serve many destinations from a shared trunk, is much harder to build with such switches. EMS maglev is naturally suited to point-to-point lines. The narrow gap matters especially in a country like Japan. A guideway in a seismically active region will move during earthquakes, and a vehicle with a 10-millimetre clearance has little margin for that movement. That consideration, among others, pushed Japanese engineers decades ago towards a design with a gap ten times larger. Achieving that gap meant abandoning attraction and active control altogether, and finding a way for the motion of the vehicle itself to generate the levitating force. The way they found involves superconductivity, and it is the subject of the next chapter. Chapter 4: Superconducting Levitation The Japanese superconducting maglev, now officially called SCMaglev, rests on a principle that every student of physics meets early and few think about again: when a magnet moves past a loop of wire, it induces a current in the loop, and that current creates its own magnetic field which opposes the change that produced it. This is Lenz's law. A strong enough magnet moving fast enough past a row of coils will induce currents whose fields push back hard enough to lift a train. No sensors, no controllers, no fast feedback loops are needed for levitation itself. The motion does the work. Making that principle into a passenger railway took Japan more than half a century, and the result is the only transport system yet designed from the start for regular service at 500 km/h. From a laboratory idea to a test line The idea of levitating vehicles by superconducting magnets moving over conducting coils was developed in the United States in the 1960s by James Powell and Gordon Danby at Brookhaven National Laboratory, who proposed a particular coil arrangement that reduced the drag such a system would otherwise suffer. Japanese National Railways had begun research into linear-motor propulsion in 1962, and adopted the superconducting approach in the following decade. In 1979 its ML-500 test vehicle reached 517 km/h on a test track in Miyazaki Prefecture on the southern island of Kyushu, an unmanned run but a striking demonstration of feasibility. The Miyazaki track was too short and too simple for a serious development programme, and in the 1990s work moved to a new test line in Yamanashi Prefecture, built to the alignment intended for an eventual commercial route. The first 18.4-kilometre section opened in 1997. MLX01 test trains on that line reached 552 km/h with people aboard in 1999 and 581 km/h in 2003. The line was extended to 42.8 kilometres and reconstructed to commercial standards in 2013, and a new generation of vehicles, the L0 series, began running on it. On 21 April 2015 a seven-car L0 train reached 603 km/h, the current world speed record for a rail vehicle. More importantly, the Yamanashi line has accumulated very large distances of test running under conditions intended to match commercial service, including long runs at 500 km/h, repeated passing of trains, and running through the tunnels that make up much of the alignment. An improved version of the L0 began testing in August 2020 with a reshaped nose and other modifications. How the guideway lifts the train The SCMaglev guideway is a U-shaped concrete trough. The vehicle rides inside it, its sides close to the trough's sidewalls. On each side of each bogie, the vehicle carries a set of powerful superconducting magnets, facing outward towards the walls. Mounted on the inside faces of the sidewalls are coils of ordinary conducting wire, arranged in a precise and rather clever pattern. Each levitation coil on the sidewall is wound in the shape of a figure eight, with an upper loop and a lower loop connected so that current circulates in opposite directions in the two. This is the null-flux arrangement descended from Powell and Danby's work. Imagine a vehicle magnet passing the coil exactly at its centre height, so that half its field threads the upper loop and half the lower loop. The voltages induced in the two loops are equal and opposite and cancel; no current flows; no force is produced; and, importantly, no energy is lost to resistance in the coil. Now imagine the magnet passing a little below the centre. More of its field threads the lower loop than the upper, the cancellation is incomplete, and a net current flows around the figure eight. In the lower loop that current opposes the magnet and pushes it down and away; in the upper loop it attracts the magnet upward. Both effects lift the magnet back towards the centre. The result is a self-restoring force that depends on how far the vehicle sits below the coil centres. When the train is heavy or slow, it rides lower and the restoring force increases until it balances the weight. At cruising speed the vehicle floats about 100 millimetres above the floor of the guideway, according to JR Central's published descriptions of the system: roughly ten times the gap of an EMS maglev. The levitation is passively stable in the vertical direction. Earnshaw's theorem, which forbids stable levitation by static fields, does not apply, because these forces depend on the motion of the magnets and on currents that change with time. Lateral guidance uses the same coils. The figure-eight coils on the left and right sidewalls are cross-connected beneath the guideway. When the vehicle sits exactly in the middle, the voltages induced in the coils on opposite sides balance and no guidance current flows. If the vehicle drifts towards one wall, the imbalance drives a current through the cross-connection that pushes the vehicle back towards the centre. Again, no sensor or controller is required. The cost of this elegance is that the levitating force depends on speed. A magnet moving slowly past a coil induces little current, and therefore little lift. Below a certain speed the induced forces cannot carry the vehicle's weight. The SCMaglev lifts off at around 150 km/h, and below that speed it rolls on retractable rubber-tyred wheels, both vertical support wheels and lateral guide wheels. A journey therefore begins and ends on wheels: the train accelerates on its tyres, rises smoothly onto its magnetic cushion as speed builds, and settles back onto its tyres as it slows into a station. Speed-dependent levitation also produces speed-dependent drag. Currents flowing in the guideway coils encounter electrical resistance, and the energy dissipated there appears as a braking force on the vehicle called magnetic drag. For an EDS system the magnetic drag rises steeply from zero at standstill, peaks at a relatively low speed, and then falls away as speed increases further, while the lift approaches a steady value. The ratio of lift to magnetic drag therefore improves with speed. At 500 km/h, magnetic drag is a modest part of the total resistance, which is dominated, as for any fast vehicle, by the air. At low speed, magnetic drag is a significant penalty, which is one more reason the system relies on wheels for the slow parts of every trip. The superconducting magnets Lift by induced currents requires a magnetic field strong enough to induce large currents across a gap of around ten centimetres. Conventional copper electromagnets could produce such fields only with enormous power consumption and weight. Superconductors make it practical. A superconducting coil has zero electrical resistance below a critical temperature, so once a current has been established it circulates indefinitely without any power supply. JR Central's magnets have been charged once and then left in this persistent mode, maintaining their field for long periods with no electrical input to the coils themselves. The conventional superconducting magnets used on the Yamanashi line for most of its history are wound from niobium-titanium alloy, the same material used in hospital MRI scanners and in the magnets of particle accelerators. Niobium-titanium becomes superconducting only below about 9 kelvin, so the coils are kept at around 4 kelvin, immersed in liquid helium, inside a cryostat: a vessel that surrounds the cold coil with vacuum insulation and layers of radiation shielding to reduce the heat leaking in from the surroundings. Refrigerators on the vehicle recondense the helium that evaporates. Keeping a magnet at 4 kelvin on a vehicle at 500 km/h is much harder than keeping one cold in a hospital basement. The vehicle vibrates; the guideway coils are discrete rather than continuous, so the magnets see a rippling field as they pass each coil; and that ripple induces eddy currents in the metallic shields around the cold coil, generating heat. Any heat that reaches the superconductor locally can push a small region above its critical temperature. That region suddenly has resistance, the large current flowing through it dissipates heat, the normal region spreads, and the whole magnet loses its superconductivity in a fraction of a second. This event is called a quench. Japanese engineers spent much of the 1980s and 1990s on the problem, redesigning shields, supports and coil windings to minimise the heat deposited by vibration and field ripple, and quenches became rare. A quench does not drop the train: the remaining magnets continue to provide levitation and guidance, and the vehicle can settle onto its wheels if needed. But it takes a magnet out of service, and the reliability of the cryogenic system was for many years the system's most important development challenge. The next step has been high-temperature superconductors. These are ceramic materials that superconduct at much higher temperatures than niobium-titanium, which means the magnet can be cooled by electrically powered refrigerators instead of liquid helium. JR Central has developed and tested high-temperature superconducting magnets on the Yamanashi line, and describes them as operating at around minus 255 degrees Celsius, cooled directly by refrigerators without liquid helium or liquid nitrogen. Higher operating temperatures give a much larger margin against quench, because the heat capacity of materials rises steeply with temperature and a given amount of deposited heat raises the temperature less. Removing liquid helium also simplifies maintenance and supply, an important consideration given the fluctuating global availability of helium. JR Central has indicated that the high-temperature magnets are intended for commercial use on the Chuo Shinkansen. Propulsion, power and stopping The same vehicle magnets that provide levitation also provide propulsion. Behind the levitation coils on the guideway sidewalls lies a second layer of coils, the propulsion coils, which are fed from substations along the line with three-phase current. Energised in sequence, they produce a magnetic field that travels along the guideway. The vehicle's superconducting magnets lock onto this travelling field, as in the Transrapid's long-stator motor, and the train moves at a speed set by the frequency of the supply. Like the Transrapid, SCMaglev therefore has a linear synchronous motor in the guideway, and its traction power never passes through a sliding contact or an overhead wire. Because the motor's active part is in the guideway, the guideway is divided into sections, each switched on only while a train is passing through it. The power converters in the substations control the train's speed and position directly. In practice the train is driven from a control centre that commands the guideway, rather than by a driver applying power on board. This is a significant departure from railway practice, and it simplifies some safety questions: two trains cannot be propelled by the same section at the same time, and the system knows exactly where each train is, because it is feeding current to the coils that are moving it. The vehicle still needs power for lighting, air conditioning, control electronics and the magnets' refrigerators. Early test vehicles carried gas-turbine generators for this. Later designs used linear generators that harvested energy from the ripple in the guideway field, much as the Transrapid did. The improved L0 vehicles tested from 2020 receive onboard power by induction, from coils laid along the guideway, which transfer power across the gap through magnetic fields without any physical contact. This was a meaningful milestone: it means the train needs no onboard fuel and no contact at all with any power supply at any speed. Braking takes several forms. In normal operation the linear motor works in reverse, slowing the train and returning energy to the supply. If that fails, the train carries aerodynamic brakes that deploy from its body to add drag at high speed, and disc brakes on its rubber-tyred wheels for low speed after it has settled back onto them. Even with complete loss of guideway power, the magnetic drag from the levitation coils, which requires no power, contributes to slowing the train, and the passive guidance continues to keep it centred in the trough throughout. The difference between the two families of maglev, and between maglev and the fastest wheels, is summarised in Table 3. The values for the Transrapid are those of the Shanghai line; for CRRC's 600 km/h EMS system they are those released by CRRC; for SCMaglev, those published by JR Central; for the CR450, those released by China State Railway Group. Table 3. Comparison of very high-speed ground transport systems. System Support method Running clearance Propulsion and power Design top speed (km/h) Transrapid (Shanghai) Attractive EMS, active control About 10 mm Long-stator motor in guideway 431 (300 in service since 2021) CRRC 600 km/h maglev Attractive EMS, active control About 10 mm Long-stator motor in guideway 600 (not in service) SCMaglev (L0) Repulsive EDS, passive, wheels below about 150 km/h About 100 mm Guideway motor, inductive onboard power 500 in service plan; 603 record CR450 Steel wheel on rail Wheel-rail contact Pantograph from overhead catenary About 400 in service plan The Chuo Shinkansen The commercial application of all this is the Chuo Shinkansen, a new line from Shinagawa in Tokyo to Nagoya, and eventually on to Osaka. Its first section is 286 kilometres long and is planned to take about 40 minutes at a maximum speed of 500 km/h, compared with around 90 minutes on today's fastest Tokaido Shinkansen trains. The line runs almost directly between the two cities, which means crossing the Japanese Alps rather than following the coast. As a result, roughly 85 percent of the route is in tunnel, including deep tunnels under the Tokyo metropolitan area and the Southern Alps Tunnel, about 25 kilometres long and in places more than a kilometre below the surface. Construction began in 2014, with opening originally targeted for 2027. The schedule has been derailed by the Southern Alps Tunnel, specifically by the 8.9-kilometre section that passes under Shizuoka Prefecture. The prefecture's governor from 2009 to 2024, Heita Kawakatsu, refused permission to start tunnelling there from 2017 onward, arguing that the work could drain water from the headwaters of the Oi River, on which downstream communities and agriculture depend. In March 2024 JR Central abandoned the 2027 target and said the line would open in 2034 or later. In October 2025 the company announced that its construction cost estimate had risen to about 11 trillion yen, roughly double the original figure, citing materials and labour inflation, difficult ground conditions in mountain tunnels, and additional earthquake measures. In July 2026, after a prefectural expert panel accepted JR Central's 28 environmental protection measures, Shizuoka's new governor, Yasutomo Suzuki, approved construction in the prefecture. The Shizuoka section is expected to take at least ten years to tunnel, and most current estimates put the opening of the Tokyo–Nagoya line around 2036. The engineering choices of the line follow directly from the physics described in this chapter and the last. The large magnetic gap gives tolerance for guideway movement in a country where earthquakes are routine, and passive levitation and guidance continue to function if power fails. The guideway motor and inductive power supply remove the catenary and its wave-speed ceiling entirely. What the line could not escape is the air, and with 85 percent of its length in tunnels, the aerodynamic problems are acute. The vehicle's long nose, the tunnel cross-sections, the provision for pressure relief, and the noise control at the short open-air sections are all governed by aerodynamics rather than by levitation. Those are the subject of the next two chapters. Chapter 5: The Air in the Open At 500 km/h, air stops being something a train moves through and becomes something it has to fight. A sixteen-car train is a body some 400 metres long and around 3.5 metres wide and tall, running a few tens of centimetres above the ground. Aerodynamically it is a very strange object: far longer and slimmer than any aircraft fuselage, operating permanently in ground effect, flanked by structures, platforms and other trains, and periodically plunging into tunnels. Its drag, its noise, its stability in crosswinds and its effect on everything near the track are all set by the air flowing around it. This chapter deals with the train in the open; the next deals with tunnels, where the problems become sharper. Where the drag comes from When people think of a fast train's aerodynamics, they think of the nose. The nose matters, but on a long train it is far from the whole story. The drag on a high-speed train can be divided into several parts, and their relative sizes surprise most people. First there is pressure drag at the front and rear. The air piles up against the nose, creating high pressure on the front surfaces, and fails to close smoothly behind the tail, leaving a region of low pressure and turbulent wake. A well-shaped nose and tail reduce both effects. On a modern high-speed train, however, they account for a relatively modest fraction of the total, because both the nose and tail have been refined for decades and the train's length dwarfs its frontal area. Second, and for long trains often the biggest single part, there is skin friction along the sides and roof. The air in contact with the train's surface moves with it, and the layer of air dragged along by the train, called the boundary layer, grows thicker towards the rear. Near the tail of a long train the boundary layer can be well over a metre thick. The shearing of air within this layer absorbs energy continuously along the whole train. Because this component grows with train length, a sixteen-car train has considerably more aerodynamic drag than an eight-car train, though much less than twice as much. Third, there is the underbody. The underside of a train is a cluttered landscape of bogies, wheels, brake discs, motors, transformers, cooling equipment and cable runs, all in close proximity to a fixed ground with sleepers or slab track. Air flowing through this gap is violently disturbed. Studies of high-speed train aerodynamics, including the extensive Japanese, German and French work summarised in the research literature, consistently find that the bogie regions and the underbody contribute a large share of total drag. Covering the bogies with side skirts and smoothing the underside with fairings are among the most effective drag reduction measures available. Fourth, there are the protrusions on the roof, chiefly the pantographs and the insulators and equipment around them. A pantograph is a lattice of struts and a collector head standing in the free airflow above the roof, and it generates drag far out of proportion to its size. On some trains the pantographs alone have been estimated to account for a noticeable percentage of total drag. Finally, the gaps between cars, where air can enter and swirl, add drag of their own; most modern high-speed trains close these gaps with full-width flexible fairings that make the train's outer skin continuous. The order of these items explains the evolution of high-speed train design. Early high-speed trains concentrated effort on the nose, which was visible and dramatic. Later generations, particularly the Shinkansen from the 1990s onward, gave as much attention to smooth sides with flush windows and doors, covered bogies, closed inter-car gaps, and reduced roof equipment. The improved L0 maglev's 13 percent drag reduction came primarily from reshaping the front end, but SCMaglev has the advantage of no bogie clutter in the conventional sense: its bogies sit inside the guideway trough and are faired into the body. Drag at speed also depends strongly on the ground. A train runs in a narrow gap above the track, with the ground stationary relative to the track but moving relative to the train. Wind tunnel testing of trains is notoriously difficult for this reason: a stationary model in a wind tunnel sees air and ground moving together, which is wrong. Researchers use moving-belt floors, or test models fired along tracks on rocket sleds or catapults, or simply instrument real trains. Each method has its uncertainties, and the published drag numbers for different trains are often not directly comparable. Noise: the problem that sets the speed limit For railways in densely populated countries, aerodynamic noise is not a secondary concern but often the decisive one. Japan's environmental quality standard for Shinkansen noise, set in 1975, limits the maximum noise level at trackside residential areas to 70 dB(A), and to 75 dB(A) in mixed commercial and industrial areas. These limits apply to the loudest moment of each train passage, measured at points near the line, and are strict by international standards. The Shinkansen operators have spent half a century working to stay within them while raising speeds from 210 km/h to 320 km/h. That effort is the best guide to how noise behaves at high speed. The scaling discussed in Chapter 1 is the key. Rolling noise, generated by the vibration of wheels and rails excited by roughness in their surfaces, rises by about 9 dB for each doubling of speed. Aerodynamic noise rises by about 18 dB per doubling, because the sound power of dipole sources scales with roughly the sixth power of flow speed. At 200 km/h rolling noise dominates. By 300 km/h, aerodynamic noise is comparable. Above 300 to 350 km/h, aerodynamic sources dominate, and every further increase in speed is paid for in their steep curve. The aerodynamic sources are distributed along the train, but four stand out. The first is the pantograph. When air flows past a cylinder, such as the struts of a pantograph, it sheds vortices alternately from either side, creating a periodic force on the cylinder and a tonal noise at a frequency set by the flow speed divided by the cylinder's diameter. The same mechanism makes telephone wires sing in wind; in acoustics it is called an aeolian tone. A pantograph's struts, head and horns create a strong cluster of such tones. Because the pantograph sits on the roof, well above any trackside noise barrier, its noise also radiates directly to neighbours. The second source is the bogie region, where turbulent flow around wheels and equipment generates broadband noise near the ground. The third is the nose and the first few metres of the train, where the flow accelerates around the front. The fourth comprises the inter-car gaps and any steps or cavities in the body surface. A small calculation shows why the pantograph's tones fall right in the range where the ear is most sensitive. For a cylinder in a steady flow, the frequency of vortex shedding is about one fifth of the flow speed divided by the cylinder's diameter; that ratio of one fifth, the Strouhal number, is one of the most robust results in fluid mechanics. Take a pantograph strut 50 millimetres across on a train at 350 km/h, about 97 metres per second. The shedding frequency is 0.2 times 97 divided by 0.05, or roughly 390 hertz. At 500 km/h, the same strut sheds vortices at about 560 hertz. Both frequencies sit squarely in the band where human hearing is acute and where A-weighted noise measurements count fully. A strut of different diameter sings at a different note, and a pantograph has many members of different sizes, so it radiates a chord of tones whose pitch rises with speed and whose loudness rises much faster. Designers can lower the loudness by shaping the cross-sections so that vortices shed less coherently along the length of each member, which smears the tones into weaker broadband noise; they can shield the members from the airflow; or they can remove members altogether. The Japanese response to pantograph noise is a case study in engineering at the margin. In the 1990s, JR West's 500 series trains, intended to run at 300 km/h on the Sanyo Shinkansen, adopted a radically new pantograph: a single vertical post supporting a T-shaped wing-like collector, in place of the traditional diamond or single-arm lattice. The post was given a cross-section designed to reduce vortex shedding, and its edges carried small serrations inspired by the leading-edge combs on owl wing feathers, which break up the coherent vortices that produce tonal noise. Eiji Nakatsu, the engineer responsible for the 500 series' development and a keen birdwatcher, has described in public talks how studies of owls and kingfishers influenced the design. Later generations of Shinkansen took different approaches: single-arm pantographs with carefully shaped arms and heads, partially enclosed by shrouds or by roof fairings that hide the base of the pantograph and the insulators from the airflow. Equally important has been the reduction in the number of raised pantographs. Early Shinkansen trains raised a pantograph on every second car. Modern sixteen-car trains raise two, connected by a high-voltage roof bus line so that if one loses contact momentarily the other maintains supply, which also suppresses the electric arcing that was once a significant noise source in its own right. Trackside measures complete the picture. Noise barriers along the line block the lower sources from wheels, rails and bogies; on many lines they have been raised and fitted with absorptive or diffracting tops. They cannot block the pantograph, which is why the pantograph has received so much attention on the vehicle itself. Bogie skirts and underbody fairings reduce lower-body noise. Smooth, well-maintained rails and wheels, ground regularly to remove roughness, keep rolling noise low. How does maglev change this? The SCMaglev has no wheel-rail rolling noise at speed, and no pantograph. Its noise is almost entirely aerodynamic, which sounds like an advantage but becomes less of one as speed rises: at 500 km/h, aerodynamic noise from the body itself is substantial, and the guideway sidewalls only partly contain it. JR Central has planned noise-insulating hoods, essentially lightweight enclosing structures, over open-air sections of the Chuo Shinkansen that pass near homes. That is a telling decision. At 500 km/h, even a train with no wheels on rails and no pantograph is too loud to run in the open near dense housing within Japanese standards without enclosing the line. Slipstream, crosswinds and passing trains A fast train drags air along with it. The boundary layer described above moves at a substantial fraction of the train's speed near its surface, and the turbulent wake behind the tail carries air along in gusts. People standing on a platform as a high-speed train passes without stopping feel this as a sudden blast, strong enough to unbalance a pushchair or sweep loose objects onto the track. Trackside workers are exposed to the same effect. European interoperability standards set limits on slipstream velocities at defined distances from the track and require new trains to be tested against them; operators also set safety distances on platforms and restrict passing speeds at stations without platform screen doors. At 500 km/h, slipstream would make open platforms unworkable, and SCMaglev stations are fully enclosed, with the guideway separated from passengers. When two trains pass each other at a combined speed approaching 1,000 km/h, the high-pressure region ahead of each nose and the low-pressure region behind it sweep along the side of the other train within a fraction of a second. The pressure pulses push the side walls in and out, load the windows, and can be felt by passengers as a jolt. The pulse strength depends on the square of the train speed and falls off with the distance between track centres. That is one reason why lines designed for higher speeds use wider spacing between tracks, which in turn makes viaducts and tunnels wider and more expensive. Crosswinds pose a different hazard. A train is a long, high-sided body. A strong wind blowing across the track exerts a lateral force and an overturning moment that rise with the square of the wind speed relative to the train, and the effective crosswind angle depends on the combination of the wind and the train's own speed. The leading car is usually the most vulnerable, because the flow around its nose produces large lateral forces. In Japan, where typhoons are common, the railways maintain networks of anemometers along exposed sections and impose automatic speed restrictions or stops when the wind exceeds set thresholds. Wind fences shield exposed viaducts and embankments. Lightweight trains, which are attractive for energy efficiency and track wear, are more susceptible to overturning, so weight reduction and crosswind safety pull in opposite directions; this is a tension that the CR450 programme, with its emphasis on reduced weight, has had to resolve. For a maglev, crosswind takes a somewhat different form. The vehicle cannot overturn in the sense that a wheeled train can, because it is contained within its guideway trough, and its magnetic guidance resists lateral displacement. But aerodynamic side forces still load the guidance system and induce motion that passengers can feel, and the open trough can produce awkward flow patterns around the vehicle. The guideway's walls provide some shelter, and the Chuo line's short open-air sections are largely covered by noise hoods that also act as wind shields. Shape as compromise A high-speed train's external shape is the outcome of many competing demands. A long, slender nose reduces pressure drag, softens the pressure pulses when trains pass, and, as the next chapter explains, is the single most effective vehicle-based measure against tunnel micro-pressure waves. But every metre of nose is a metre of train that carries no passengers, so long noses cost capacity, and the length of a train is limited by platform lengths. A smaller cross-section reduces drag and tunnel pressure effects but makes the interior cramped. Flush surfaces and closed gaps reduce drag and noise but make maintenance access harder. A low, rounded roof reduces drag but leaves less room for equipment, pushing it underneath the floor where it increases underbody drag. Nowhere is this compromise more visible than at the front of the most recent Japanese trains. The E5 series on the Tohoku Shinkansen, running at 320 km/h, has a nose about 15 metres long. ALFA-X, the test train built to explore 360 to 400 km/h, carried two different noses: 16 metres at one end and 22 metres at the other. On a ten-car train of around 250 metres, a 22-metre nose at each end would occupy nearly a fifth of the length. JR East's decision to plan the E10 series for 320 km/h, not 360, reflects in part how expensive the necessary aerodynamic measures for higher speed become in seats, energy and noise mitigation. That pressure on nose length comes mostly from tunnels, where the air has nowhere to go and the train's shape governs the strength of the pressure waves it launches. The tunnel is where the aerodynamics of very fast trains is at its most demanding, and it is where the Chuo Shinkansen will spend most of its running time. Chapter 6: Tunnels and Pressure Waves In March 1975, Japanese National Railways extended the Sanyo Shinkansen westward from Okayama to Hakata on the island of Kyushu. The new section ran through a series of long tunnels, laid with concrete slab track rather than ballast, and its trains ran at up to 210 km/h, the same speed as on the original Tokaido line. Soon after opening, residents living near some tunnel exits began to complain of loud explosive noises, accompanied by rattling windows and doors, a fraction of a second before trains emerged. Nothing comparable had been heard on the older line. The phenomenon, which came to be known as the tunnel micro-pressure wave and is popularly called a tunnel boom, turned out to be one of the defining problems of high-speed rail aerodynamics. Every high-speed train that has been designed since has been shaped partly by it, and at maglev speeds it becomes a central design driver. The train as a piston When a train enters a tunnel, it pushes air in front of it. In the open, that air can escape sideways and upward. In a tunnel, it can go only forward or backward through the gap between train and tunnel wall. The train acts as a leaky piston being driven into a tube. The immediate effect is a compression wave. As the nose enters the portal, the air pressure ahead of it rises, and this pressure rise travels down the tunnel at the speed of sound, about 340 metres per second, far faster than the train itself. The amplitude of the compression wave, meaning the size of the pressure rise, depends roughly on the square of the train speed and on the blockage ratio, which is the ratio of the train's cross-sectional area to the tunnel's. A large train in a small tunnel at high speed produces a large compression wave. For a high-speed train entering a typical double-track tunnel at around 300 km/h, the wave amounts to a few kilopascals, a few percent of atmospheric pressure. What matters for the tunnel boom, however, is not the size of the wave but its steepness: how quickly the pressure rises as the wave front passes a given point. The steepness depends on how fast the nose enters, and on how quickly the train's cross-sectional area builds from zero at the nose tip to its full value. A blunt nose builds the cross-section rapidly and produces a steep wave front; a long, gradually tapered nose builds it slowly and produces a gentler one. The steepness of the wave front scales approximately with the cube of the train's speed, one power from the wave's amplitude, one more from its time scale, and the result is that small increases in speed can produce large increases in steepness. As the compression wave travels down the tunnel, two things happen to it. Friction with the tunnel walls and the track tends to smooth it out. At the same time, a nonlinear effect makes it steepen: the higher-pressure part of the wave is slightly warmer and denser and travels slightly faster than the lower-pressure part ahead of it, so the back of the wave front gradually catches up with the front. Which effect wins depends on the tunnel. On ballasted track, the porous ballast absorbs some of the wave's energy and the smoothing tends to prevail. On smooth concrete slab track, there is less damping, and in a long tunnel the steepening can win, sharpening the front into something approaching a shock wave by the time it reaches the far end. That is why the problem appeared on the Sanyo line and not on the Tokaido: the new line combined long tunnels with slab track. When the steepened front reaches the far portal, most of its energy reflects back into the tunnel as an expansion wave, because the open end behaves acoustically almost like a free surface. A small part radiates out of the portal as a pressure pulse. The strength of that radiated pulse is proportional to the steepness of the arriving wave front, not to its amplitude. A steep front produces an impulsive bang; a gradual one produces nothing audible. The radiated pulse spreads outward and can rattle windows, shake doors and startle people at considerable distances, all before the train itself arrives at the portal. The timing of these events is worth working through, because it explains why residents near tunnel exits heard the bang before they saw the train. Take a tunnel 5 kilometres long and a train entering at 300 km/h, about 83 metres per second. The compression wave launched at entry travels the length of the tunnel at about 340 metres per second and arrives at the far portal roughly 15 seconds later. The train itself needs about 60 seconds to cover the same distance. The boom therefore arrives some 45 seconds before the train emerges, which is why the phenomenon was at first so puzzling to the people who heard it. At 500 km/h, the train takes 36 seconds and the warning shrinks to about 21 seconds, but the pulse itself is several times stronger, because the steepness of the original wave front has risen with roughly the cube of speed: by a factor of about (5/3)³, or 4.6, compared with entry at 300 km/h, before any additional steepening along the tunnel. Meanwhile, the reflected expansion wave from the far portal heads back towards the train, and when it meets the train the pressure around the train drops sharply. When the tail enters the tunnel, it generates an expansion wave of its own. Over the next tens of seconds, these waves bounce between the portals, passing the train repeatedly, and every passage is felt as a pressure change by the train's body and, if it is not sealed, by its passengers. In a tunnel with two tracks, a second train entering from the far end adds its own family of waves. Taming the boom The countermeasures that Japanese engineers developed after 1975, and that are now standard practice internationally, attack the problem at each stage. At the entrance portal, the aim is to make the compression wave rise more slowly. The most effective measure is a hood: an extension of the tunnel built outward from the portal, with a slightly larger cross-section than the tunnel, and with openings or slots along its length. As the train enters the hood, some of the air pushed ahead of it escapes through the openings, so the pressure in front of the train builds gradually rather than suddenly. A well-designed hood lengthens the rise time of the compression wave and therefore reduces its steepness. Hoods are visible at the entrances to many Shinkansen tunnels, and the Yamanashi maglev test line has them at its portals as well. Their design is a matter of careful testing and simulation, since the size, spacing and shape of the openings determine how effectively the wave is softened. Within the tunnel, the aim is to stop the wave from steepening. Ballasted track helps, though it is rarely chosen in long tunnels because of maintenance and other costs. Side branches, such as ventilation shafts or cross passages, can absorb or scatter part of the wave. Researchers have also investigated acoustic resonators built into the tunnel walls, designed to damp the steepening at particular frequencies. The most important measures, however, are on the train. A train's nose determines how quickly its cross-sectional area grows as it enters the tunnel, and therefore the steepness of the compression wave it launches. Analysis and experiment have shown that, for a given nose length, the best results come from a nose whose cross-sectional area increases as steadily as possible along its length. That is why the noses of modern Japanese trains look the way they do: not the conical point of an aircraft, but a long, flattened, sometimes asymmetric shape whose cross-section grows smoothly from tip to cab. The famous duck-bill noses of the 700 series and the long, flowing noses of the E5 and the L0 are refined variations of this principle, calculated with computational fluid dynamics and checked by firing scale models into model tunnels. Nose length is the dominant factor. To a first approximation, the steepness of the compression wave is inversely related to the time the nose takes to enter the tunnel, which is nose length divided by train speed. To keep the steepness constant while increasing speed, the nose must grow at least in proportion. Because steepness scales with roughly the cube of speed, in practice the nose has to grow faster than that. Table 4 traces this in the Japanese fleet. The nose lengths are those published by the operating railways and manufacturers. Table 4. Nose length and speed in Japanese high-speed vehicles. Train Entered testing or service Maximum running speed (km/h) Approximate nose length 500 series (JR West) 1997 300 15 m N700 series (JR Central, JR West) 2007 300 10.7 m E5 series (JR East) 2011 320 15 m ALFA-X (JR East, test train) 2019 400 in testing 16 m and 22 m L0 series (JR Central, maglev) 2013 500 planned; 603 in testing 15 m The N700 is the instructive exception. Its 10.7-metre nose is shorter than the 500 series', yet it runs at the same top speed. The N700 was optimised with computational methods to reach similar tunnel performance with a shorter nose, gaining seating capacity, and its operation also benefited from improved tunnel measures on its lines. It shows that a sophisticated shape can partly substitute for length. But the principle still holds: ALFA-X needed noses of 16 and 22 metres to explore 360 to 400 km/h, and those lengths would have consumed a sizeable fraction of a production train. The maglev has an additional option, which is to build tunnels with larger cross-sections, reducing the blockage ratio. Because the Chuo Shinkansen was designed from the start for 500 km/h, its tunnel bore and guideway were sized with tunnel aerodynamics in mind, and its tunnel portals are equipped with hoods. The L0's 15-metre nose, working with these measures, keeps the micro-pressure wave within limits at 500 km/h. That combination illustrates a general rule: at very high speed, the tunnel and the train must be designed together as a single aerodynamic system. Pressure inside the train The pressure waves that cause tunnel booms outside also matter inside the train. As a train enters a tunnel, passes through the compression and expansion waves reflected back and forth along it, and meets other trains, the air pressure outside its body changes repeatedly, sometimes by several kilopascals in a few seconds. Passengers' ears are sensitive to rapid pressure change. A drop or rise of even one kilopascal in a couple of seconds can cause discomfort, and larger changes can be painful, especially for people with colds or blocked ears. A train whose body lets outside air in freely passes every one of these changes straight to its passengers. For that reason, Japanese and later European high-speed trains have been built with pressure-tight car bodies. Doors seal against inflatable gaskets; windows are fixed; and the ventilation system either closes its intakes when a pressure transient is detected or uses high-pressure fans that resist the external pressure swings. The car body becomes, in effect, a pressure vessel, and it must be designed for the fatigue of millions of pressure cycles over its life. The European technical specifications for interoperability set an absolute ceiling on the pressure variation a train may experience while passing through a tunnel, as a health limit for occupants whose cars might lose sealing: the variation must not exceed 10 kilopascals. This number constrains tunnel design. For a given train, the pressure variation in a tunnel depends on the blockage ratio, the train speed and the tunnel length; it is largest at certain critical tunnel lengths, where the compression and expansion waves reflected from the portals and generated by the train's nose and tail entry superpose most unfavourably. A designer working to the 10-kilopascal limit at a given speed is driven towards a larger tunnel cross-section. That is why European tunnels built for 300 km/h and above have much larger bores than those of older lines, often in the region of 90 to 100 square metres or more for twin-track tunnels. The problem grows sharply with speed. Since pressure wave amplitudes scale roughly with the square of speed, a train at 500 km/h generates waves almost three times stronger than one at 300 km/h in the same tunnel. To hold pressure changes within the same limits, the blockage ratio must fall significantly, so the tunnel must be larger, or the train smaller. Tunnel costs rise steeply with cross-section: excavation volume, lining, support, and spoil disposal all increase. On a line that is 85 percent tunnel, this is a dominant cost factor, and it partly explains why the Chuo Shinkansen's cost has risen as its detailed tunnel construction has progressed. Resistance in a tube, and the vacuum dream There is one more tunnel effect worth understanding, because it lies behind a recurring idea in fast transport. A train in a tunnel experiences more aerodynamic drag than the same train in the open. The air displaced by the train must flow backward through the narrow annular gap between train and wall, and that flow is faster and more turbulent than the flow around a train in open air. Tunnel drag can be well above open-air drag, depending on the blockage ratio and tunnel length. For a line that runs mostly in tunnel, this increases energy consumption, and the heat it generates warms the tunnel air, which must be managed with ventilation. The obvious response, if most of a line is enclosed anyway, is to pump out the air. A train in a partially evacuated tube would experience less drag, less noise and smaller pressure waves. This is the essence of the vacuum-tube or hyperloop concept, proposed in various forms for more than a century and revived after 2013 with a series of high-profile start-ups. The physics, however, is less forgiving than the promotional material suggested. A vehicle in a tube must still let the air it encounters pass around it; if it moves too fast relative to the residual air in a tight tube, the flow in the gap chokes, an effect known in aerodynamics as the Kantrowitz limit, and the vehicle pushes a column of air ahead of it. Maintaining even a modest vacuum over hundreds of kilometres of tube, with stations, switches and expansion joints, and with leaks that inevitably develop, is a large and continuous energy and maintenance burden. Evacuating passengers from a sealed low-pressure tube in an emergency is far harder than evacuating them from a railway tunnel. The record so far supports the scepticism. Hyperloop One, the best-funded of the American companies, shut down at the end of 2023 without building a passenger line. Chinese state-backed projects, notably the low-vacuum maglev test line at Datong, have produced genuine engineering results on short test tracks but no route plan with a credible path to service. The problems of the tube are not those of levitation or power, which are solved, but of air, structure and safety at scale. That is the same reason the Chuo Shinkansen's hardest engineering has been in its tunnels. The lesson of this chapter, then, is that in a tunnel, a train cannot outrun its own air. Every gain in speed must be paid for in nose length, tunnel diameter, pressure sealing, or portal hoods, and the cost of each rises steeply. For a maglev line built mostly underground, this cost is the true price of 500 km/h. The next two chapters turn from air to electricity, and to the last of the three interfaces: the contact between the train and its power supply. Chapter 7: The Wire That Carries Waves A high-speed train running at full power on a 25-kilovolt line draws several hundred amperes, and the largest trains draw considerably more. All of that current passes through a handful of carbon strips, each a few centimetres wide, pressed against the underside of a copper wire by a force of little more than the weight of a large bag of groceries. The strips slide along the wire at the train's speed. If they lift off the wire for even a few milliseconds, the current jumps the gap as an electric arc, eroding both strip and wire, generating electromagnetic interference and, at worst, interrupting traction. Keeping this sliding contact steady at 300 km/h is a solved problem, but only just. Keeping it steady at 400 km/h is an active engineering frontier. At 500 km/h, on any overhead line that has been built, it is effectively impossible for regular service. This chapter explains why, beginning with the wire. Anatomy of a catenary The overhead line on a high-speed railway is not a single wire but a suspended structure. At its top is the messenger wire, also called the catenary wire, which hangs between support masts in the natural sagging curve a cable takes under its own weight. That curve is mathematically called a catenary, which is how the whole structure got its name. Beneath the messenger hangs the contact wire, the one the pantograph actually touches, suspended by thin vertical wires called droppers at intervals of several metres. The droppers are cut to different lengths so that the contact wire hangs nearly level, or in a deliberately shaped profile, even though the messenger above it sags. The masts that carry this structure stand at intervals of roughly 50 to 65 metres on high-speed lines. At each mast, a registration arm or steady arm holds the contact wire laterally. The wire is not hung directly above the centre of the track but is offset alternately to the left and right at successive masts, typically by a couple of hundred millimetres, so that in plan view it zigzags gently along the line. This stagger sweeps the contact point back and forth across the width of the pantograph's carbon strips and spreads the wear evenly; a wire hung dead straight would wear a groove in a single spot on every strip. Both wires are kept under tension. The tension is applied at the ends of each tension length, a section of overhead line typically around 1,200 to 1,500 metres long, by weights hanging on pulleys or by spring tensioners that keep the tension constant as the wires expand and contract with temperature. Adjacent tension lengths overlap so that a passing pantograph slides smoothly from one contact wire to the next. Several variants of this basic layout are in use on high-speed lines. Germany's Re330 design, built for 330 km/h, adds a short stitch wire at each support, a secondary loop between the messenger and the contact wire that softens the hard spot the mast would otherwise create. Japan's Shinkansen lines have traditionally used compound catenaries with an auxiliary wire between the messenger and contact wire, giving a structure with more mass and stiffness, well suited to the multiple pantographs of early Shinkansen trains and to heavy current loads. French high-speed lines have used simple catenaries with high tension. Chinese 350 km/h lines have drawn on both German and Japanese practice, generally using simple or stitched catenaries with copper-magnesium or copper-tin contact wires at high tension. The span problem Look at a catenary from the side and it seems a simple, nearly straight line. To a pantograph pushing up on it, it is anything but uniform. Near the masts, the contact wire is held firmly by the registration and the droppers are short; the wire is stiff. At mid-span, it is held only by long droppers and its own tension; it is soft. The upward stiffness of the contact wire, the force needed to lift it by a given amount, varies by a significant factor between the mast and the middle of the span. A pantograph pushing up with a roughly constant force therefore lifts the wire more at mid-span than at the masts. As the train moves, the pantograph rides up and down once every span. At 300 km/h, with 60-metre spans, the train passes a mast about every 0.7 seconds, so the pantograph is driven at a frequency of about 1.4 hertz. That is uncomfortably close to some of the low natural frequencies of the pantograph and the catenary, and at certain speeds the response can grow. Designers counter this in several ways. The contact wire is often hung with a deliberate presag, a slight downward curve at mid-span, typically around a thousandth of the span length, so that the pantograph's uplift at the soft middle produces a more nearly level contact path. Stitch wires soften the hard spots at the supports. The droppers near each support are made as long as practical. All these measures make the stiffness more uniform along the span, which reduces the excitation of the pantograph and evens out contact force. These are refinements, though. The fundamental limit is not stiffness variation, but waves. Waves on a string A tensioned wire carries transverse waves, like a guitar string or a skipping rope given a flick. The speed of those waves is set by two quantities only: the tension in the wire and the wire's mass per unit length. Specifically, it is the square root of tension divided by mass per metre. Higher tension makes the waves faster; heavier wire makes them slower. The contact wire on a high-speed line is a solid copper alloy conductor with a cross-section of 100 to 150 square millimetres, weighing roughly one to one and a third kilograms per metre. A useful example is the copper-magnesium wire with a 120 square millimetre cross-section used on Germany's Re330 lines, which weighs about 1.07 kilograms per metre and is tensioned to 27 kilonewtons. Divide 27,000 by 1.07, take the square root, and the wave speed comes out at about 159 metres per second, or roughly 570 km/h. This treats the contact wire alone and ignores the effect of the messenger and droppers, which modify the real figure somewhat, but it is the standard first estimate. Now picture a pantograph pressing upward on the wire. It creates a local upward bend that propagates away along the wire in both directions at the wave speed. If the pantograph is stationary, the bend spreads symmetrically. If the pantograph is moving, the wave running ahead of it moves away from it at only the difference between the wave speed and the train speed, while the wave running behind moves away at the sum. As the train approaches the wave speed, the forward wave can barely outrun the pantograph. Its energy piles up in a short stretch of wire just ahead of the contact point. The wire ahead of the pantograph is lifted more and more steeply, and the contact force fluctuates wildly. Engineers capture this with a quantity called the Doppler factor, defined as the wave speed minus the train speed, divided by the wave speed plus the train speed. At standstill it equals one. As the train approaches the wave speed, it falls towards zero. Its practical significance is that wave reflections, which occur wherever the wire's properties change, at every dropper, every steady arm, every clamp, are amplified as the Doppler factor falls. A wave reflected from a dropper ahead of the pantograph travels back towards it, is concentrated by the approaching motion, and arrives as a sharper kick. In the literature on contact line design, the amplification of such reflections is often expressed as the reflection coefficient divided by the Doppler factor; as the Doppler factor approaches zero, the amplification grows without limit. That is why there is a practical rule of thumb that the operating speed should not exceed about 70 percent of the contact wire wave speed, which corresponds to a Doppler factor of a little under 0.2. At higher ratios, contact quality deteriorates rapidly regardless of how good the pantograph is. The rule explains a great deal about the history of high-speed electrification. Table 5 shows how wave speed, the 70 percent operating limit and the Doppler factor at 350 km/h vary with tension and wire size. The values are calculated from the simple string formula, using the mass per metre of standard copper-alloy contact wires of 120 and 150 square millimetres; they are illustrations of the relationship rather than the certified ratings of particular systems. Table 5. Contact wire wave speed and operating margin for illustrative tensions. Contact wire Tension Wave speed 70% of wave speed Doppler factor at 350 km/h 120 mm², about 1.07 kg/m 20 kN about 490 km/h about 345 km/h 0.17 120 mm², about 1.07 kg/m 27 kN about 570 km/h about 400 km/h 0.24 150 mm², about 1.34 kg/m 30 kN about 540 km/h about 380 km/h 0.21 150 mm², about 1.34 kg/m 40 kN about 625 km/h about 435 km/h 0.28 The table shows several things. A 20-kilonewton contact wire of modest section is limited to about 345 km/h by the 70 percent rule; it is adequate for 300 km/h, which is roughly where older European high-speed lines were rated. A 27-kilonewton wire, as in the Re330 design, gives margin up to about 400 km/h. Heavier wire, chosen to carry more current and to wear longer, needs more tension to achieve the same wave speed, because its extra mass slows the waves. The 40-kilonewton row corresponds to the tension used for the 2007 V150 record. Using the 150 square millimetre figure for illustration, the record run at 574.8 km/h took place at about 92 percent of the wave speed, with a Doppler factor of about 0.04, far beyond any limit acceptable for service. That the record was possible at all reflects a very carefully adjusted line, a short run, and the acceptance of contact quality that would be intolerable day after day. How far can the tension go? If raising tension raises the wave speed, why not simply pull the wire harder? The answer lies in the strength and life of copper. A contact wire must conduct large currents efficiently, so it must be mostly copper. Pure copper is relatively weak, especially once warmed by current. Alloying with small amounts of silver, tin or magnesium increases strength substantially while reducing conductivity somewhat; the copper-magnesium alloys used for the highest-tension high-speed wires have conductivities noticeably below that of pure copper. Even so, the working tension must be kept well below the wire's breaking load, because the wire also experiences bending at each clamp and dropper, vibration from every passing pantograph, thermal cycling from day to night and from current flow, and wear. Wear is crucial: as the pantograph strips slide along the wire, they gradually abrade its underside, and operators typically allow the cross-section to wear down by a substantial fraction before replacement. A wire that is safe at its nominal tension when new must still be safe when worn, because the tension, applied by weights or springs, does not reduce as the wire thins. The stress in the remaining metal rises as it wears. These limits put a practical ceiling on contact wire tension at around 30 to 40 kilonewtons with the alloys available today. That corresponds to wave speeds in the range of roughly 550 to 650 km/h, and therefore to an operating ceiling, under the 70 percent rule, of perhaps 400 to 450 km/h. China's CR450 programme, aiming at about 400 km/h, sits right at this edge, and the Chinese railway industry has reported work on higher-strength contact wires and redesigned catenaries to support it. Mechanical waves are not the only limitation. A train at 400 km/h needs more power than one at 350 km/h; under the cube law, about half as much again for the aerodynamic part alone, and that power must arrive through the same contact. The high-speed network's 25-kilovolt supply, usually arranged as a two-by-25-kilovolt autotransformer system to reduce voltage drop and interference, can deliver it, but the current in the contact wire rises, increasing heating. Feeding points must be closely spaced, and neutral sections, the short insulated gaps that separate supplies from different phases of the national grid, must be passed with power briefly cut, which becomes more delicate at high speed. A worked example shows how the electrical and mechanical sides combine. Take the illustrative train of Chapter 1, which needed roughly 4 megawatts at the wheel to hold 300 km/h. Allowing for losses in the transformer, converters and motors, and for onboard auxiliary loads, it might draw around 5 megawatts from the wire. At 25 kilovolts that is a current of around 200 amperes. At 400 km/h, the same train's aerodynamic power demand rises by the cube of four thirds, about 2.4 times, and its total draw from the wire might approach 10 to 11 megawatts, or more than 400 amperes. If two such trains are coupled and run as one, with two pantographs, each pantograph carries that current, and the trailing one does so on a wire still oscillating from the passage of the leading one. Meanwhile, the span-passing frequency at 400 km/h on 60-metre spans is about 1.9 hertz, and the dropper-passing frequency, with droppers every 6 to 10 metres, is between about 11 and 19 hertz. Each of these frequencies, and their multiples, sweeps through the natural frequencies of the pantograph and catenary as the train accelerates, and the line must be tuned so that no strong resonance falls at the speeds where the train spends most of its time. The wire, in other words, is a mechanical and electrical system with its own characteristic speed. Below that speed it can be made to work extremely well. Approaching it, its behaviour changes in kind, and the best engineering only delays the problem. But the wire is only half of the interface. The other half is the pantograph, a fast-moving machine with its own dynamics, its own aerodynamics and its own failure modes. Getting the two halves to work together is the subject of the next chapter. Chapter 8: The Pantograph at the Edge The pantograph is easy to overlook. It folds flat against the roof when not in use, rises on its arms when the train is powered, and looks like a simple piece of linkage. It is in fact one of the most finely tuned pieces of machinery on a high-speed train, and at the top of the speed range it is the component most likely to set the limit. It must press against the wire hard enough to keep contact continuously, but not so hard that it lifts the wire excessively, wears it quickly, or excites the waves described in the previous chapter. It must follow the wire's height, which varies with span, temperature and wear, and it must do so while standing in an airstream of several hundred kilometres per hour that is pushing on it with forces comparable to the contact force itself. It must carry megawatts of power through a sliding contact without arcing. And, as Chapter 5 explained, it is one of the loudest sources of noise on the train. This chapter looks at how pantographs work, what limits them, and why superconducting maglev abandoned them entirely. A machine made of two masses A modern high-speed pantograph has a hinged frame, almost always now of the single-arm type: a lower arm pivoting on a base frame on the roof, an upper arm hinged to it at a knee, and a collector head at the top. The head carries two parallel contact strips, typically made of carbon, sometimes impregnated with metal to improve conductivity, mounted transversely so that the zigzagging contact wire sweeps across them. Curved horns at each end of the head guide the wire back onto the strips if it strays. The frame is raised and held up by an air-pressure actuator or a spring, and the force it applies is regulated to set the static uplift, the upward force at standstill. Dynamically, the pantograph is best understood as two masses connected by springs. The frame is the larger mass, and it responds slowly; it follows the long-wavelength variation in wire height across each span and between spans. The head is the smaller mass, suspended on the top of the frame by its own short springs, and it responds quickly, following the shorter irregularities in the wire: the kinks at droppers, the steps at clamps, the minor variations in height caused by wear or installation. The design trick is to keep the head as light as possible, because a lighter head can follow fast changes in wire height without the contact force varying much. Every gram of head mass is a gram that must be accelerated up and down many times a second at high speed. Pantograph designers spend a great deal of effort on light materials for the head and its strips, and on head suspensions that isolate the head from the inertia of the frame. The pantograph and the catenary are, in combination, a coupled dynamic system. The catenary has its own natural frequencies, which depend on span length, tension and mass, and so does the pantograph. The train's speed sets the frequencies at which the pantograph is driven: the span-passing frequency, the dropper-passing frequency, and their harmonics. Some speeds put these driving frequencies close to natural frequencies of the system, and contact quality at those speeds can be markedly worse. A good high-speed catenary and pantograph are designed together, and their combination is tested and certified as a pair across the full speed range. Forces from the air As speed rises, the airflow over the pantograph exerts forces that can rival or exceed the static uplift. Air flowing over the arms and head generates aerodynamic lift and drag. Because the single-arm pantograph is asymmetric, the aerodynamic forces depend on which direction the train is travelling: with the knee leading and with the knee trailing, the flow over the arms is different and so is the lift. At 300 km/h the aerodynamic contribution to contact force can be large, and it grows with the square of speed. Unchecked, this would mean a pantograph pushing ever harder against the wire as speed increased, increasing wire uplift, wear and noise. Designers therefore tune the aerodynamics of the head and arms deliberately. Small spoilers, fins and shaped profiles on the head are adjusted so that the aerodynamic uplift at operating speed adds the right amount of force, and so that the force is similar for both directions of travel. Wind tunnel tests and line tests are used to calibrate these elements for each pantograph and train combination. On some trains the pantograph's air supply is adjusted with speed to compensate. The tuning is delicate: a change of a few degrees in a spoiler can alter the contact force at top speed by tens of newtons. European standards specify the target. For alternating-current lines, the standard for pantograph and overhead line interaction, EN 50367, and the European technical specifications for energy subsystems set an upper bound for the mean contact force as a function of speed, given by 0.00097 times the square of speed in kilometres per hour, plus 70 newtons. At 300 km/h that gives about 157 newtons; at 350 km/h, about 189. The same rules require the statistical standard deviation of the contact force to stay below 30 percent of the mean, so that the force rarely drops anywhere near zero. Lower bounds on the force are also specified, to keep contact secure. The reason for the upper bound is that more force does not mean better contact. Higher force lifts the contact wire more, increasing its motion and the waves on it. It increases wear on both wire and strips. It can lift the wire at the steady arms enough to strike the registration hardware. The reason for the variance limit is arcing: if the force fluctuates widely, its low troughs will occasionally reach zero, the strips will leave the wire, and the current will arc. Interoperability rules in Europe limit the percentage of running time in which arcing occurs, measured with optical sensors that detect the ultraviolet light from arcs; for high-speed lines, this limit is a fraction of one percent. Testing a new pantograph and catenary combination is therefore a matter of measuring contact force and arcing under real running. Instrumented pantographs carry force sensors in the head and accelerometers to correct for the head's inertia, and transmit their measurements from the high-voltage roof to the train's interior through optical links. Simulation has become a partner to testing. Engineers model the catenary as a long finite-element structure and the pantograph as a set of masses and springs, and simulate their interaction at speed. In 2015, an international benchmark exercise organised by a group of research teams including Politecnico di Milano compared ten independent pantograph-catenary simulation codes on the same test case and found broad agreement among them on the main statistics of contact force. Simulations must be validated against measurements under the EN 50318 standard before being used to support approval of a new design. Two pantographs, one wire A long train often needs more than one pantograph, either because it is formed of two coupled units each with its own, or because it needs to draw more current than one pantograph can carry comfortably, or because a single pantograph is a single point of failure. Multiple pantographs introduce a new problem. The leading pantograph sets the wire in motion, and the waves it generates are still travelling along the wire when the trailing pantograph arrives. The trailing pantograph meets a wire that is already oscillating, often with large amplitude. Its contact force varies much more, and it arcs more often. Railways deal with this in different ways. The spacing between raised pantographs is chosen so that the trailing one meets the wire at a favourable phase of the motion, or simply far enough behind for the motion to have decayed. Line ratings for multiple-pantograph operation are often lower than for single-pantograph operation. French TGV practice on 25-kilovolt lines has been to raise only one pantograph per trainset and carry power along a roof cable to the other power car, avoiding the problem altogether. Japanese Shinkansen trains raise two pantographs per sixteen-car train, connected by a high-voltage bus line along the roof so that if one briefly loses contact, the other keeps the circuit closed and no arc forms. The bus line was originally introduced to reduce arcing noise; it also improves electrical continuity, and it allowed operators to reduce the number of raised pantographs from the large numbers of the early Shinkansen to two. Wind, ice and the weather on the wire A catenary is an outdoor structure hundreds of kilometres long, and the weather acts on it continuously. Strong crosswinds push the contact wire sideways. Because the wire's stagger already places it a couple of hundred millimetres off the track centre at each mast, and because the pantograph head is of finite width, a wind that blows the wire far enough sideways at mid-span can carry it beyond the end of the carbon strips and onto the horns, or past them altogether. If the wire slips off the end of the head, the pantograph can rise above it and then snag on the next support, tearing down the overhead line and damaging the train. This event, called a dewirement, is one of the most disruptive failures on an electrified railway, because repairing the line can take many hours. Designers limit it by restricting span lengths on exposed sections, by choosing stagger values with wind in mind, and by setting wind-speed thresholds above which trains slow down. Pantograph head widths are standardised in part to ensure that trains and lines from different builders are compatible in this respect. Wind can also make the wires gallop. When a thin layer of ice forms on a wire, or when the wind meets the wire at certain angles, the wire's cross-section can become aerodynamically unstable, and the wire begins to oscillate vertically with large amplitude at a low frequency. Galloping wires disrupt contact and can damage fittings. Ice also coats the contact surface directly, insulating it from the strips; the first train of a frosty morning may arc heavily as its pantograph scrapes ice from the wire. Some railways run trains or special pantographs through the night in icing conditions to keep the wire clear, and others apply anti-icing treatments or briefly raise the line current to warm the wire. Heat matters as well. On hot days the wires expand; the tensioning weights or springs take up the extra length and keep the tension constant, but only within the travel the tensioning device allows. On lines with large seasonal temperature ranges, the tension lengths, the travel of the weights and the movement of the registration arms all have to be designed for the full range. At high speed, a tensioning device that has reached the end of its travel, leaving the wire slack, changes the wave speed and the contact quality immediately. Every one of these effects is small in normal service. Each one becomes more significant as the margin between operating speed and wave speed shrinks. Wear, current and the limit of the sliding contact A pantograph's carbon strips are consumable, and so is the contact wire. Each passing train removes a little material from both. The rate of wear depends on contact force, current, speed, the materials involved and, critically, arcing: an arc can erode more material in milliseconds than many kilometres of steady sliding. On busy high-speed lines, contact wires are replaced after a set amount of wear, and the cost of that replacement is part of the price of speed. Strips are replaced far more often, sometimes after a few tens of thousands of kilometres. Current also matters when the train is stationary. A train standing in a station drawing power for heating, lighting and air conditioning passes current continuously through the same small patch of contact wire, which can heat that patch noticeably. Pantograph designers take this into account in the choice of strip materials and contact area. Monitoring has become as important as design. Japan's famous yellow inspection trains, nicknamed Doctor Yellow, used to run at line speed measuring track geometry and contact wire height, stagger and wear. JR Central retired its Doctor Yellow set in January 2025, moving the inspection role to measuring equipment fitted to ordinary N700S trains in passenger service, a shift that gives more frequent data on the condition of the catenary. Similar monitoring systems, using cameras, lasers and instrumented pantographs on service trains, are spreading across European and Chinese high-speed networks. All of this works, at 320 km/h and, with greater difficulty, at 350. At 400 km/h the margins shrink: the 70 percent rule leaves little room even with high-tension wires; the aerodynamic forces on the pantograph rise by about 30 percent compared with 350 km/h; pantograph noise rises by several decibels; the trailing pantograph's problems worsen; and wear accelerates. The CR450 programme's ambition of commercial operation at around 400 km/h represents, in effect, a bet that the pantograph-catenary interface can be held at the edge of its envelope for daily service. If that bet succeeds, the wheel will have reached its practical power-collection limit. Taking the wire away Every difficulty in this chapter and the last arises from the need to collect high power through a sliding contact on a wire that carries waves. Maglev systems with long-stator propulsion remove that need entirely. In the Transrapid and in Japan's SCMaglev, the traction power is supplied to the guideway, not to the vehicle; the vehicle's magnets are the moving part of a motor whose stator is the track. The power needed on the vehicle for lighting, ventilation, control and refrigeration is a small fraction of traction power, and SCMaglev obtains it by inductive transfer from the guideway without contact. That removes the catenary's wave-speed limit, eliminates the pantograph's noise and drag from the roof, and abolishes arcing, strip wear and wire replacement. It also changes where the system's cost lies. A long-stator motor needs propulsion coils along the entire length of the guideway, connected to power converters in substations, with switching to energise only the sections a train occupies. That is a far larger investment per kilometre than an overhead line. The trade, then, is between a cheap, simple, limited interface and an expensive, complex, unlimited one. At 300 km/h, the overhead line wins on cost by a wide margin. At 500 km/h, it is not a contender. That conclusion completes the picture built across the preceding chapters. Of the three interfaces where a fast train touches its surroundings, the wheel can be pushed to around 400 km/h; the overhead wire can be pushed to around the same figure; and the air can be managed at 500 km/h only at great expense in nose length, tunnel diameter and enclosure. Maglev removes the first two limits. It leaves the third in place. What that means for the future of very fast ground transport is the subject of the conclusion. Conclusion The physics of very fast ground transport comes down to a simple asymmetry. Two of the three interfaces that limit a train's speed can be engineered away; the third cannot. The steel wheel's limits of adhesion, hunting and track precision disappear when the train floats on a magnetic field. The overhead wire's wave-speed limit and the pantograph's struggle for steady contact disappear when the motor is built into the guideway and onboard power arrives by induction. But nothing removes the air. A train at 500 km/h must push its way through the atmosphere, and the forces, noise and pressure waves that result grow with speed far faster than speed itself. Everything that follows about the future of fast trains flows from that asymmetry. The first consequence is that 500 km/h is a system, not a vehicle. One cannot reach it by building a faster train and running it on an existing railway. The wheel-rail and pantograph-catenary limits converge around 400 km/h, and even reaching that figure requires a line designed for it, with high-tension catenary, wide curves, large tunnels, strong bridges and a demanding maintenance regime. Going beyond it requires abandoning the wheel and the wire together, which means building a new kind of guideway along the entire route. That guideway must be designed around the vehicle's aerodynamics from the outset: tunnel cross-sections and portal hoods sized for the train's pressure waves, enclosures over open sections sized for its noise, enclosed stations sized for its slipstream. The Chuo Shinkansen is the only project to have taken this on in full, and its cost, roughly 11 trillion yen for the Tokyo-Nagoya section alone, is an honest measure of what the physics demands on a mountainous, heavily tunnelled route. The second consequence is that the economic case for 500 km/h is narrow and specific. The arithmetic of the first chapter showed that traction energy per kilometre rises roughly with the square of speed, while time saved per kilometre falls as speed rises. A route benefits from 500 km/h only if it is long enough for the train to spend most of its time at top speed, straight enough to allow curves of around eight kilometres radius, and busy enough that the minutes saved, multiplied by the number of travellers, justify the capital cost. Tokyo to Osaka meets those conditions as well as any corridor on earth. It is a 500-kilometre axis between two of the world's largest metropolitan economies, with an existing high-speed line that carries hundreds of thousands of passengers on a typical day and is operating close to its capacity. JR Central's argument has always been as much about adding capacity and resilience, a second line through the interior in case the coastal Tokaido line is disrupted by earthquake, as about speed. Few other corridors combine such demand with the financial strength of an operator able to fund much of the work itself. The third consequence is that, for most countries, the useful frontier lies at the upper end of the wheeled range, not at maglev speeds. China's CR450, aimed at around 400 km/h on its busiest corridors, is a test of whether the wheel and the wire can be held at the edge of their envelopes for daily service at acceptable cost. Japan's E10 series, planned for 320 km/h, reflects a judgement that the noise and tunnel costs of higher speeds on its existing alignments are not worth paying. Europe's settled choice of 300 to 320 km/h reflects the same arithmetic applied to shorter distances and denser settlement. These are not failures of ambition. They are reasonable answers to the question of where the marginal minute stops being worth its marginal cost, and the answer depends on the route, the population around it and the price of energy, not on what the technology can do in a record run. The fourth consequence concerns how to read claims about the next leap. Announcements of trains capable of 600, 800 or 1,000 km/h appear regularly. Some describe real achievements in propulsion, sensing or levitation on short test tracks. None of them addresses the problem that dominates at these speeds, which is aerodynamic. The vacuum tube is the logical answer to the air problem, and it is the right answer to examine seriously; but it exchanges the air problem for problems of structure, leakage, thermal expansion, emergency evacuation and energy consumption over hundreds of kilometres of sealed tube, none of which has yet been solved at scale. The useful questions to ask of any such claim are the ones that run through this book. What holds the vehicle up, and with what clearance? How does it collect its power? What does the air do around it, in the open and in a tunnel, and at what cost in noise and structure? A proposal that cannot answer those questions with real engineering is not yet a transport system. The fifth consequence is that the most consequential engineering for fast trains is often the least visible. A reader leaving this book might remember the superconducting magnets and the record speeds. The more important lessons lie in less glamorous places: the 15-metre noses whose cross-section grows smoothly from tip to cab; the perforated hoods on tunnel portals; the presag in a contact wire; the owl-feather serrations on a pantograph; the high-voltage bus line that lets a train run with two pantographs instead of eight; the decision to place a motor in a guideway rather than on a vehicle. Each is a response to a scaling law, and each moves a limit a little further away. Progress at the top end of the speed range has come mostly from such responses, which change the physical regime of an interface, rather than from adding power. There is a final point, which is about patience. The superconducting maglev that will run between Tokyo and Nagoya began as a research programme in 1962. It reached 517 km/h in 1979 and set its 603 km/h record in 2015. If current estimates hold, passengers will ride it in the mid-2030s, some seventy years after the first experiments. That timeline is not a sign of failure. It reflects how long it takes to understand, test and certify a new system at the boundaries of physics, and to build the long, expensive, carefully designed infrastructure it needs. A 500 km/h railway is not a faster train. It is a new relationship between a vehicle and its ground, its air and its power, and building that relationship well is the work of generations. Glossary Blockage ratio. The cross-sectional area of a train divided by the cross-sectional area of the tunnel it is in. Higher ratios produce stronger pressure waves and more drag. Catenary. The overhead line structure on an electrified railway, consisting of a sagging messenger wire, droppers and a nearly level contact wire that the pantograph touches. Doppler factor. For a contact wire, the wave speed minus the train speed, divided by their sum. It falls towards zero as the train approaches the wave speed, and reflections on the wire are amplified as it does. Electrodynamic suspension (EDS). Levitation by repulsion, produced when strong magnets move past conducting coils and induce currents that oppose them. Used in Japan's SCMaglev; it works only above a threshold speed. Electromagnetic suspension (EMS). Levitation by attraction, in which actively controlled electromagnets on the vehicle pull upward towards a steel rail on the guideway. Used in the Transrapid and Chinese maglev systems. Hunting. A self-sustaining side-to-side oscillation of a railway wheelset that occurs above a critical speed, controlled by suspension stiffness, dampers and wheel profile. Linear synchronous motor. An electric motor unrolled along a track. On long-stator maglev systems, its powered windings are in the guideway and the vehicle's magnets are carried along by the travelling field. Micro-pressure wave. The pressure pulse radiated from a tunnel exit when a steep compression wave, launched by a train entering the other end, reaches the portal. Strong pulses are heard as tunnel booms. Null-flux coil. A figure-eight coil whose two loops cancel each other when a magnet passes exactly at its centre, so that current, force and losses arise only when the magnet is displaced. Used for levitation and guidance in SCMaglev. Pantograph. The hinged current collector on a train's roof that presses carbon contact strips against the overhead contact wire. Quench. The sudden loss of superconductivity in a magnet when part of the coil warms above its critical temperature. Stagger. The deliberate side-to-side zigzag of the contact wire relative to the track centre, which spreads wear across the pantograph's contact strips. Further Reading Baker, Chris, Terry Johnson, Dominic Flynn, Hassan Hemida, Andrew Quinn, David Soper and Mark Sterling. Train Aerodynamics: Fundamentals and Applications. Butterworth-Heinemann, 2019. The most complete modern treatment of the subject, covering drag, slipstream, crosswinds, pressure waves in tunnels and the test methods used to measure them, with substantial attention to European standards. Raghunathan, Raghu S., H.-D. Kim and T. Setoguchi. "Aerodynamics of high-speed railway trains." Progress in Aerospace Sciences 38 (2002): 469–514.** A long review article that remains a good entry point to the research literature on train aerodynamics, including tunnel compression waves and micro-pressure waves. Baker, Chris. "The flow around high speed trains." Journal of Wind Engineering and Industrial Aerodynamics 98 (2010): 277–298.** A concise, physically clear description of the flow regions around a moving train, from nose to wake, written by one of the field's leading researchers. Thompson, David. Railway Noise and Vibration: Mechanisms, Modelling and Means of Control. Elsevier, 2009. The standard reference on rolling noise, aerodynamic noise and their control, with an excellent explanation of why different noise sources scale differently with speed. Kiessling, Friedrich, Rainer Puschmann, Axel Schmieder and Egid Schneider. Contact Lines for Electric Railways: Planning, Design, Implementation, Maintenance. Third edition. Publicis, 2018. The engineer's handbook of overhead line design, including the wave propagation and Doppler factor concepts that govern high-speed catenaries. Bruni, Stefano, Jorge Ambrosio, Alberto Carnicero and others. "The results of the pantograph–catenary interaction benchmark." Vehicle System Dynamics 53, no. 3 (2015): 412–435.** A comparison of ten independent simulation codes for pantograph-catenary dynamics, useful for understanding how the interface is modelled and validated. Knothe, Klaus, and Sebastian Stichel. Rail Vehicle Dynamics. Springer, 2017. A rigorous textbook on wheel-rail contact, wheelset stability and hunting, for readers comfortable with mathematics. Lee, Hyung-Woo, Ki-Chan Kim and Ju Lee. "Review of maglev train technologies." IEEE Transactions on Magnetics 42, no. 7 (2006): 1917–1925.** A clear survey of electromagnetic and electrodynamic suspension systems and their propulsion methods, comparing the German, Japanese and other approaches. Moon, Francis C. Superconducting Levitation: Applications to Bearings and Magnetic Transportation. Wiley, 1994. An accessible scientific account of how superconductors make levitation possible, including the physics of electrodynamic suspension and null-flux coils. Iwasa, Yukikazu. Case Studies in Superconducting Magnets: Design and Operational Issues. Second edition. Springer, 2009. A technical but readable treatment of superconducting magnet engineering, including stability, cooling and quench protection. Hashtags: #HighSpeedRail #HighSpeedRailTechnology #RailwayEngineering #RailTechnology #Maglev #MagneticLevitation #SuperconductingMaglev #ElectromagneticSuspension #ElectrodynamicSuspension #TrainAerodynamics #AerodynamicDrag #AerodynamicNoise #PantographDynamics #PantographCatenaryInteraction #OverheadCatenarySystems #RailwayElectrification #LinearSynchronousMotors #Superconductivity #RailVehicleDynamics #WheelRailInteraction #HighSpeedTrainDesign #TunnelAerodynamics #TunnelPressureWaves #RailwayNoiseControl #Shinkansen #TGV #Transrapid #SCMaglev #RailwayInnovation #FutureOfRailTransport

Download the Book (PDF): Introduction At the 2022 World Cup in Qatar, a computer decided, in effect, where a footballer's shoulder was. Twelve cameras mounted beneath the roof of each stadium followed every player, estimating up to 29 points on each body fifty times a second, while a sensor suspended inside the match ball reported its motion five hundred times a second. When an attacker received the ball from a position that might have been offside, the system located the instant of the pass, drew the line through the relevant limb of the second-last defender, and alerted the video officials, who checked the result and passed it to the referee. A few minutes later the crowd saw a three-dimensional animation of the moment, rendered from the same data. By the 2026 tournament in North America, every player had been scanned in three dimensions so that the system could reason about real body shapes rather than generic stick figures, and clear positional offsides were being sent straight to the assistant referees on the touchline. That episode compresses the subject of this book. A decade ago, the position of a player's body was something only a human eye could judge, and the human eye was the standard against which any machine was measured. Today, in the best-equipped stadiums, the machine is the standard and the human checks its work. The same shift has happened, more quietly, in training grounds and medical rooms. A strength coach who once judged fatigue by watching a player's gait and asking how he felt now receives a morning report built from GPS units, accelerometers, heart-rate straps and force plates. An analyst who once tagged passes by hand now receives the coordinates of all twenty-two players and the ball for every frame of every match, and can ask questions that would have taken a season of manual coding to answer. The promise that comes with these systems is easy to state and easy to oversell. Track everything, the story goes, and you can see the game as it really is; feed the tracking into machine learning, and you can predict who will win, which tactic will work, and which player is about to tear a hamstring. Some of that promise has been kept. Much of it has not, and the reasons it has not are more interesting, and more useful to anyone working in sport, than the successes. The argument of this book This book argues one thing. Computer vision has largely solved the problem of measurement in elite sport: the positions and, increasingly, the postures of athletes can now be recovered from video with an accuracy that would have seemed implausible in 2010. That success has moved the hard problem somewhere else. The difficult part is no longer seeing the athlete. It is turning what is seen into decisions that are better than the ones experienced people would have made without it, under conditions of small samples, adaptive opponents, rare outcomes and human bodies that do not behave like the machines that measure them. Seen this way, the three topics in the title of this book are not three separate applications of one technology. They form a gradient of difficulty. Extracting skeletal poses from broadcast video is an engineering problem with clear ground truth: a knee is either where the model says it is or it is not, and the error can be measured. Optimizing formations and tactics is harder, because there is no ground truth for the best tactic, only outcomes that are noisy, scarce and shaped by an opponent who is optimizing against you. Predicting injury is hardest of all, because injuries are rare, their causes are tangled, and the act of predicting one changes the conditions that would have produced it. The further a problem sits along this gradient, the more the value of artificial intelligence depends not on the model but on the people and processes around it. That claim has practical consequences, and the chapters that follow try to draw them out. A club buying a tracking system should ask different questions from a club buying an injury-risk model. A sports scientist reading a paper that reports an impressive accuracy figure for injury prediction should know which number actually matters and why it is usually not the one in the abstract. A coach being shown a tactical recommendation generated by a neural network should know what the network was trained to imitate, and therefore what it cannot know. What this book covers and what it leaves out The book is concerned with the use of artificial intelligence, and especially computer vision and machine learning, in the analysis of elite team sport and the management of athletes' bodies. Most of the examples are drawn from association football, basketball, baseball and American football, because those are the sports in which the data infrastructure is most developed and the public record is richest. The ideas transfer readily to rugby, hockey, cricket and others, and examples from those sports appear where they illuminate something. It does not cover sports betting as an industry, although betting appears where it shapes how tracking data is collected and sold. It does not attempt a survey of every commercial product on the market, which would be out of date within a year. It does not teach programming, although it describes algorithms in enough detail that a technically minded reader could find the original papers and follow them. And it does not deal with consumer fitness technology except where the same methods are used by professional teams. The book assumes no specialist background. Terms from computer vision, statistics and sports science are explained when they first appear. Readers with technical training will find the explanations brief; readers without it will find that nothing essential depends on mathematics. How the book is organised The first chapter describes how sport came to be measured, from hand-written match notes to optical tracking, and what kind of data each generation of technology produces. The second explains how computer vision recovers the positions and skeletal poses of athletes from ordinary video, step by step, from detecting a person in a frame to estimating the three-dimensional shape of a body. The third examines where that pipeline fails and how much its errors matter, because every downstream analysis inherits them. The fourth chapter turns from measurement to meaning, describing the models analysts use to put a value on space, possession and individual actions. The fifth applies those ideas to formations and tactics, including the attempts to use graph neural networks and generative models to recommend set pieces and positional structures, and explains why tactical optimization differs from optimization in almost any other field. The sixth and seventh chapters move from the game to the body. The sixth describes how training load is measured and what physiological models sit behind the numbers. The seventh asks what machine learning can and cannot do to predict and prevent injury, and why the honest answer is less dramatic and more useful than the marketing. The eighth considers who owns the data generated by athletes' bodies, how it is governed, and how an organization can build an analytics function that improves decisions rather than merely producing reports. The conclusion draws these threads together and argues for a particular way of thinking about the role of machine intelligence in sport: as an instrument that sharpens human judgment where the problem is well defined, and as a disciplined partner in reasoning, rather than an oracle, where it is not. Chapter 1: From Notebooks to Optical Tracking Every analytical method in sport is limited by what it can measure. The sophistication of a model counts for little if the data underneath it records only that a pass was made, not where the receiver was standing, how fast the nearest defender was closing, or which way the passer's hips were facing. The history of sports analytics is therefore, to a large extent, a history of measurement: of successive technologies that expanded what could be recorded, each of which changed the questions analysts were able to ask. Understanding that history matters for a practical reason. The data a club or league holds today is usually a layered inheritance of several generations of technology, and each layer carries the assumptions and blind spots of the system that produced it. An analyst who does not know where a number came from cannot know what it is silently leaving out. Event data: the game as a list of things that happened The oldest form of sports data is the event log. Baseball's box score dates to the nineteenth century, and its descendants still structure how the sport is reported. In football, the best-known pioneer was Charles Reep, a former Royal Air Force wing commander who from the early 1950s recorded matches by hand in his own notation, logging passes, shots and the sequences that led to them. Reep's analysis, published with Bernard Benjamin in the Journal of the Royal Statistical Society in 1968, observed that most goals came from moves of three passes or fewer, and he drew from this the conclusion that teams should move the ball forward quickly and directly. Later analysts pointed out that most moves of any kind are short, so short moves producing most goals says little about whether short moves are more efficient per attempt. The episode is still a useful warning: event data invites conclusions about causes when it describes only frequencies. Modern event data is collected industrially. Companies such as Opta (now part of Stats Perform), StatsBomb (now Hudl StatsBomb) and Wyscout employ trained operators, often assisted by software, to log every on-ball action in a match: passes, shots, tackles, interceptions, clearances, fouls. Each event carries a timestamp, the player responsible, a location on the pitch and a set of qualifiers such as body part, pass height or outcome. A typical professional football match generates somewhere in the region of two to three thousand events. Some providers have added richer context. StatsBomb, for example, began recording the positions of visible players at the moment of each shot, and later at every event, which partially bridges the gap to tracking data. Event data has two great strengths. It is cheap enough to collect for thousands of matches across dozens of leagues, which makes it the backbone of recruitment analysis; and it is semantically rich, because a human has already decided what each action was. Its fundamental weakness is that it describes only the player with the ball. In football, a player is in possession for perhaps two or three minutes of a ninety-minute match. Everything else, the running, positioning, pressing and covering that occupies the vast majority of every player's time, is invisible. Tracking data: the game as a field of moving points Tracking data records the position of every player, and usually the ball, many times per second for the whole of a match. It turns a match from a list of discrete events into a continuous spatiotemporal record: twenty-two trajectories plus a ball, sampled at ten to twenty-five frames per second for football in most commercial systems, producing well over a million coordinates per match. The first widely used football tracking system was developed by Prozone, a British company founded in the 1990s, which installed multiple fixed cameras around stadiums and used a combination of software and human operators to follow each player. Early Prozone data was used mainly for physical reports: distance covered, distance at high speed, number of sprints. Those reports transformed conditioning in the Premier League in the 2000s, because for the first time a fitness coach could see exactly what demands a match had placed on each player, and compare them across positions and seasons. Basketball followed a similar path. SportVU, a system originally developed in Israel for tracking in football and later owned by STATS LLC, used six cameras mounted in the catwalks of an arena to record player and ball positions twenty-five times per second. A handful of NBA teams installed it in 2010 and 2011, and from the 2013–14 season the league installed it in every arena. That decision created one of the first complete, league-wide tracking datasets in any sport, and much of the academic work on spatiotemporal sports analytics in the following five years was built on it. The NBA later moved its official tracking to Second Spectrum, from the 2017–18 season, and then to Sony's Hawk-Eye Innovations, whose partnership with the league began with the 2023–24 season and added three-dimensional capture of player movement. American football took a different route. The NFL's Next Gen Stats system, rolled out across the league in the mid-2010s, relied primarily on radio-frequency identification chips embedded in each player's shoulder pads, read by receivers placed around the stadium. Radio tracking is less vulnerable to occlusion than optical tracking, which matters in a sport where twenty-two large bodies collide at the line of scrimmage, but it measures a single point on each player rather than the shape of the body. Baseball's Statcast system launched in three ballparks in 2014 and expanded to all thirty in 2015, initially combining a radar system for ball flight with optical cameras for players. From the 2020 season Major League Baseball replaced it with a Hawk-Eye system of twelve optical cameras per ballpark, some running at high frame rates, capable of tracking both the ball and the skeletal pose of players. In May 2024 the league published bat-tracking metrics, such as bat speed and swing length, measured by the same camera network, five of whose cameras record at three hundred frames per second. Skeletal data: the game as moving bodies The third generation of data does not reduce a player to a single point. It estimates the positions of many anatomical landmarks on each body, such as head, shoulders, elbows, wrists, hips, knees, ankles and feet, and so captures posture, orientation and limb motion. The MLB Hawk-Eye system tracks eighteen points on each player's body. FIFA's semi-automated offside technology at the 2022 World Cup used up to twenty-nine points per player at fifty frames per second, and at the 2026 tournament it was supplemented by three-dimensional scans of every player, so that the virtual offside line could be drawn against a real body shape rather than a generic skeleton. The difference between a point and a skeleton is not a refinement; it is a different kind of information. A point tells you where a defender is. A skeleton tells you which way he is facing, whether his weight is on his front foot, whether he has turned his hips to cover a run, and whether a striker's shoulder was beyond the last defender's toe at the moment of a pass. Many of the questions that coaches have always asked, about body shape when receiving, about scanning before receiving the ball, about the mechanics of a pitching delivery, are questions about posture, and only skeletal data can answer them at scale. The three generations can be compared directly. Table 1 summarises what each kind of data records and what it typically supports. Table 1. Three generations of sports data. Data type What is recorded Typical rate Main uses Event data On-ball actions with location and qualifiers Per action Recruitment, match reports, shot models Positional tracking x, y position of all players and ball 10–25 Hz Physical load, team shape, space models Skeletal tracking Many body landmarks per player in 3D 30–50 Hz or more Officiating, biomechanics, body orientation Wearables: measuring from the inside out Alongside optical systems, a parallel stream of data comes from devices worn by athletes. The most common in team sport is the GPS unit, a small device worn in a vest between the shoulder blades, which records position from satellite signals, typically at ten hertz, and combines it with an accelerometer, gyroscope and magnetometer sampling much faster. Companies such as Catapult and STATSports supply most professional football and rugby clubs. In indoor sports, where satellite signals do not penetrate, local positioning systems using radio beacons fill the same role. Wearables measure things cameras cannot. The accelerometer captures impacts, jumps and the micro-movements of contact sport that do not change a player's position much but load the body heavily. Heart-rate straps measure internal physiological response. Some teams add instrumented insoles, force plates for jump testing, sleep trackers and saliva assays. These streams feed directly into the load-management systems examined later in this book. Wearables also raise sharper governance questions than cameras, because they are attached to the body and can measure physiological states that athletes may regard as private. In the NBA, the 2017 collective bargaining agreement allowed teams to ask players to wear devices but made their use voluntary, and barred teams from using the resulting data in contract negotiations. That arrangement anticipated many of the ownership debates that have since spread across professional sport. Ball tracking and the officiating precedent A parallel lineage, older than player tracking in its public visibility, concerns the ball. Hawk-Eye, developed in Britain by Paul Hawkins and colleagues in the early 2000s, used several high-speed cameras around a cricket ground or tennis court to triangulate the ball's position and reconstruct its trajectory. Television first used it in cricket in 2001 as a broadcast graphic, showing the predicted path of a ball that struck a batter's pad. Tennis adopted it for line-call challenges from 2006, and cricket integrated ball tracking into its Decision Review System for leg-before-wicket appeals later that decade. The officiating use of ball tracking set an important precedent for everything that followed. For the first time, a computer vision system's output was treated as authoritative in competition, overruling a human official. That required something that pure analytics never had to provide: public, audited accuracy. Tennis authorities tested the systems and published tolerances; cricket's rules explicitly built a margin of uncertainty into the review system, so that a marginal projection would revert to the umpire's original decision. The idea that a machine's measurement should carry a stated uncertainty, and that decisions near the boundary of that uncertainty should be handled differently, is one that analytics departments would do well to borrow. By the mid-2020s, tennis had gone further. Several tournaments replaced line judges entirely with automated electronic line calling, and Wimbledon did so from its 2025 championships, ending a tradition that had lasted well over a century. Football followed a different path, with goal-line technology approved from 2012 and used at the 2014 World Cup, and then the semi-automated offside system described in the Introduction. In each case the pattern was the same: a narrow, well-defined question with a clear geometric answer, a camera system engineered specifically for it, a testing regime, and only then the transfer of authority from human to machine. Why the history matters The pattern across sports is consistent. Each new layer of measurement first gets used to answer the questions people were already asking, more precisely: how far did he run, how fast did he throw, was he offside. Only later, often years later, do analysts discover the questions the new data makes newly possible: how much space a team controls, how valuable a run without the ball was, how a pitcher's arm path changes as he tires. The second consistent pattern is that the data never arrives clean. Each system has its own coordinate conventions, its own frame rate, its own way of handling a player who leaves the camera's view, its own definitions of a sprint or an acceleration. A club that buys event data from one provider, tracking data from another and GPS data from a third has three representations of the same ninety minutes that do not line up exactly. Synchronizing them, which means matching the event log's timestamps to the tracking frames and the GPS clock, is unglamorous work that absorbs a surprising fraction of any analytics department's time, and errors in it propagate into every model built on top. Most importantly, the progression from events to points to skeletons tracks the progression this book is about. With each step, the raw measurement has become more complete and more objective. The question of what the measurements mean has not become correspondingly easier. If anything, the richer the data, the larger the space of possible interpretations and the easier it becomes to find patterns that are not really there. Broadcast video as the universal source One final development changes the economics of everything described above. Installed tracking systems are expensive and exist only in stadiums whose owners have paid for them. That covers the top divisions of the wealthiest leagues but excludes most of the world's football, most youth and women's competitions, most historical matches, and most opponents a scout might want to study in a lower league abroad. Broadcast video, by contrast, exists for almost every professional match played anywhere. If positions and poses can be extracted reliably from the broadcast feed itself, from a single moving camera that zooms, pans and cuts to replays, then tracking data becomes available for the whole football world, retrospectively and without installing anything. Several commercial providers, including SkillCorner and Hudl's StatsBomb, now sell tracking data derived from broadcast footage, and the quality has improved sharply as computer vision has advanced. Broadcast tracking is a harder problem than fixed-camera tracking. Players leave the frame constantly. The camera's position and zoom change every second. Replays and close-ups interrupt the wide view. Different broadcasters use different camera positions. Recovering a consistent, metric representation of the pitch and everyone on it from such footage requires the full pipeline of modern computer vision: detection, tracking, identification, camera calibration and pose estimation. That pipeline, and what each stage actually does, is the subject of the next chapter. Chapter 2: From Pixels to Skeletons A frame of broadcast football is, to a computer, a grid of roughly two million coloured pixels. Somewhere in that grid are twenty or so players, a referee, a ball, white lines painted on grass, advertising boards and a crowd. The task of a tracking pipeline is to turn a sequence of such grids, fifty of them a second, into something an analyst can use: the identity, pitch position and body posture of every player at every instant, expressed in metres on a fixed coordinate system rather than in pixels on a moving screen. No single algorithm does this. The problem is decomposed into stages, each solving a narrower problem and passing its output to the next. Understanding the stages matters for anyone who uses tracking data, because each introduces its own characteristic errors, and the errors compound. This chapter walks through the pipeline in the order the data flows: finding people, following them over time, knowing who they are, placing them on the pitch, and finally recovering the shape of their bodies. Finding people: detection The first stage is object detection: for each frame, draw a bounding box around every person and the ball. Modern detectors are deep convolutional or transformer-based neural networks trained on large labelled image collections. The YOLO family of models, first published by Joseph Redmon and colleagues in 2016 and developed through many later versions by several groups, became popular in sport because it is fast enough to process video in real time. Other detectors trade speed for accuracy. A general-purpose detector trained on everyday photographs will find most players in a wide broadcast shot, but sport stresses it in specific ways. Players in the far half of the pitch may be only twenty or thirty pixels tall. Players cluster in tight groups at set pieces, where one body hides most of another. Kits of similar colours blur together under floodlights. The ball is tiny, fast and frequently hidden behind a foot. For these reasons, providers fine-tune detectors on footage from the sport itself, with thousands of hand-labelled frames, and they treat the ball as a separate problem, often using specialised small-object detectors combined with physical constraints on how a ball can move between frames. The output of detection is, for each frame, a list of boxes with confidence scores. It knows nothing about time. A player detected in frame 100 and the same player in frame 101 are, to the detector, unrelated boxes. Following people: multi-object tracking The second stage links detections across frames into continuous trajectories, called tracklets. The classical approach, still the core of most systems, has two ingredients. A motion model, often a Kalman filter, predicts where each existing track should appear in the next frame given its recent velocity. An association step then matches the new detections to the predicted positions, typically by solving an assignment problem that minimises some combined cost of distance and appearance dissimilarity. The Hungarian algorithm, dating from the 1950s, is commonly used for this matching. Appearance matters because motion alone fails when two players cross. If one player runs behind another, their boxes merge, and when they separate the tracker must decide which is which. Systems such as DeepSORT, published in 2017, added a learned appearance embedding: a neural network maps each detected person to a vector, trained so that crops of the same individual produce nearby vectors and crops of different individuals produce distant ones. ByteTrack, published in 2022, showed that much could be gained simply by also associating low-confidence detections, which are often partially occluded players, rather than discarding them. Team sports are unusually hard for these methods. Players on the same team wear identical kits, so appearance embeddings struggle to tell them apart. Movements are abrupt: a footballer can change direction in a fraction of a second in a way a pedestrian rarely does. And in broadcast footage, players constantly leave and re-enter the frame as the camera pans. The SoccerNet benchmarks, a set of public datasets and challenges for football video understanding begun in 2018, have made the size of these problems visible: tracking performance that looks excellent on pedestrian benchmarks drops sharply on football footage. Knowing who they are: identification A tracklet is an anonymous trajectory. To be useful it must be attached to a named player. Identification combines several weak signals. Team assignment is usually solved by clustering the colours of each player's kit. Shirt numbers can be read by a text-recognition model when the player's back faces the camera at sufficient resolution, which in broadcast footage happens only intermittently. Positional priors help: a player who has been at left-back for twenty minutes is probably still the left-back. The team sheet constrains the possibilities. In practice, commercial providers combine automated identification with human review. An operator corrects identity swaps, joins tracklets that the algorithm broke, and checks the output against the official event log. The amount of human correction is one of the main differences between data products, and it is rarely disclosed in detail. A buyer who wants to know how reliable broadcast tracking is should ask not only how good the model is but how much manual quality control sits behind the delivered file. Placing them on the pitch: camera calibration Detection and tracking happen in pixel coordinates. Analysis needs pitch coordinates in metres. The bridge between them is camera calibration: estimating, for every frame, the mathematical transformation that maps points on the pitch surface to points in the image. Because the pitch is approximately flat, this transformation for points on the ground is a homography, a projective mapping between two planes described by a three-by-three matrix with eight free parameters. If four or more points can be identified both in the image and on a model of the pitch, the homography can be solved. Pitch markings provide those points: the corners of the penalty area, the intersections of the halfway line with the touchlines, the centre circle. A calibration network learns to detect these landmarks, or to segment the painted lines directly, and an optimisation step finds the homography that best aligns the detected lines with the known template. Each player's position on the pitch is then obtained by taking the bottom-centre of their bounding box, roughly where their feet touch the ground, and mapping it through the homography. This works well when the player is upright and the box is accurate. It fails in instructive ways: a player who is jumping appears to be farther away than he is, because his feet are off the ground; a player whose feet are hidden by another body has no reliable ground point at all. Broadcast calibration has to cope with a moving camera. The main broadcast camera pans and zooms continuously, so the homography changes every frame. When the camera zooms in tightly on a duel near the touchline, few pitch markings may be visible, and the calibration becomes uncertain. Systems handle this by tracking the camera's motion between frames, by enforcing smoothness over time, and by exploiting the fact that the broadcast camera usually sits at a fixed physical location even as it rotates and zooms. When the broadcast cuts to a replay, a close-up or a different camera, the pipeline must detect the cut and either discard those frames or recognise the new view. Installed multi-camera systems face the same geometry with far better conditions. Their cameras are fixed and calibrated in advance, often using surveyed reference points in the stadium. Because several cameras see each player from different angles, the system can triangulate positions in three dimensions rather than assuming the player is on the ground. This is why installed systems can track the ball in the air and why their accuracy is consistently higher than broadcast-derived data. Recovering the body: pose estimation The final stage estimates the skeleton. Two-dimensional human pose estimation predicts the pixel locations of a fixed set of body keypoints. The most widely used academic convention, from the COCO dataset released by Microsoft and academic partners in 2014, defines seventeen keypoints: nose, eyes, ears, shoulders, elbows, wrists, hips, knees and ankles. Sports systems often add points for the feet, heels and toes, which matter for offside decisions and for gait analysis. There are two broad strategies. Bottom-up methods find all keypoints in the image first and then group them into individuals. OpenPose, published by Zhe Cao and colleagues at Carnegie Mellon University in 2017, made this approach famous; it learned not only keypoint locations but fields that indicate which limbs connect which keypoints, so that it could assemble skeletons in crowded scenes. Top-down methods first detect each person and then estimate the pose within each bounding box. HRNet, published in 2019, kept high-resolution feature maps throughout the network and became a standard top-down backbone; later transformer-based models such as ViTPose pushed accuracy further. Top-down methods are generally more accurate when people are well separated and the detector is good; bottom-up methods degrade more gracefully in crowds. Sport has both conditions, sometimes within the same second. Most pose networks do not output a single coordinate per joint. They output a heatmap, a small image for each keypoint in which brightness indicates the likelihood that the joint lies there. The peak of the heatmap gives the estimate, and its spread is a rough indication of uncertainty. That uncertainty is valuable downstream and is often thrown away. Lifting to three dimensions A two-dimensional skeleton tells you where joints appear in the image, not where they are in space. A forward lean and a backward lean can project to nearly identical images from certain angles. For biomechanics, offside decisions and many tactical questions, the third dimension matters. With multiple calibrated cameras, three-dimensional pose is recovered by triangulation: the same keypoint, seen from two or more known viewpoints, defines rays that intersect at the joint's true position. This is the principle behind installed systems in baseball, tennis and top-level football officiating. Its accuracy depends on the number of cameras that see each joint at each instant, the precision of the calibration, and the synchronisation of the cameras' clocks. With a single camera, which is the broadcast case, the problem is mathematically underdetermined, and learning must fill the gap. Monocular lifting models take a sequence of two-dimensional poses and predict the three-dimensional configuration that most plausibly produced them, drawing on what they have learned about human body proportions and movement from motion-capture datasets. Using a sequence rather than a single frame helps considerably, because the way a projection changes over time constrains the underlying motion; temporal convolutional models such as the one published by Dario Pavllo and colleagues in 2019 exploited this. Parametric body models go further. The SMPL model, published by Matthew Loper and colleagues in 2015, represents a human body as a mesh controlled by a small number of shape and pose parameters, so that fitting it to an image produces not just a skeleton but a full body surface with plausible limb lengths. FIFA's decision to scan every player at the 2026 World Cup can be understood as replacing the generic body model with an individual one for each athlete. Spotting events without a human logger Tracking produces trajectories; event data, as Chapter 1 described, is still largely logged by people. A natural extension of the vision pipeline is to recognise events automatically: to detect that a pass, shot, tackle or foul occurred, and when. The SoccerNet project framed this as "action spotting", the task of finding the precise instant of each event type in a long broadcast video, and published benchmarks on which many research groups have competed. Models for action spotting typically combine features extracted from video frames by a pretrained network with temporal models that look at a window of several seconds around each candidate moment. Tracking data makes many events easier to detect than raw video does. A pass, for instance, can be inferred when the ball leaves the vicinity of one player and arrives near another, with a characteristic change in velocity. A shot shows a rapid acceleration of the ball toward the goal. These rules break down in hard cases, such as deflections, miscontrols and balls that pass through a crowd, which is where learned models add value. In practice, providers increasingly use automated detection to propose events that a human operator then confirms or corrects, which speeds up logging and improves consistency without removing the human check. The combination matters because the most useful analyses join the two data types. Knowing that a pass was played, and also where all twenty-two players were at the instant it was played, is what allows the valuation models of Chapter 4 to work. When both event and tracking data are derived from the same video by the same pipeline, they are synchronised by construction, which removes one of the most stubborn sources of error in combining separately supplied datasets. A single moment, end to end It helps to follow one moment through the pipeline. Suppose a broadcast shows a midfielder receiving the ball near the centre circle, turning and playing a through-ball for a striker running behind the defence. The detector finds roughly fifteen people in the frame: most of the outfield players visible in a wide shot, plus the referee, and the ball as a small blob near the midfielder's feet. The tracker links these boxes to the trajectories it has been following, and when the striker briefly disappears behind a defender it keeps his track alive on the basis of his predicted motion, reconnecting it when he reappears. Kit colour assigns each player to a team; the striker's number was read a few seconds earlier when he turned his back to the camera, and the tracker carries that identity forward. Calibration identifies the halfway line and the arc of the centre circle, solves the homography, and maps every player's foot position onto pitch coordinates. Pose estimation adds that the midfielder's hips were open toward the attacking half as he received, and that the last defender's weight was on his heels. The output is a few hundred numbers for this one frame, and many thousands across the two seconds of the move. None of them says that the through-ball was good. That judgment requires a model of what the numbers mean, which is the job of the valuation methods described later in the book. But every judgment those models make rests on this chain of estimates, and its reliability is the subject of the next chapter. The pipeline as a chain of dependencies The stages are often described separately, but in deployed systems they are entangled. Pose estimation needs good detection boxes. Identification benefits from pose, because the orientation of the torso indicates whether a shirt number should be visible. Calibration benefits from pose, because a player's feet give ground-contact points even when pitch lines are hidden. Tracking benefits from all of them. Modern systems increasingly train several stages jointly, or feed later outputs back to correct earlier ones in a second pass. The practical consequence for anyone using the data is that errors are correlated, not independent. The moments when detection struggles, such as a crowded penalty box at a corner, are the same moments when tracking swaps identities, calibration loses lines behind bodies, and pose estimation confuses whose knee belongs to whom. And those crowded, contested moments are disproportionately the ones that decide matches. The next chapter examines this problem directly: how good these systems really are, where they fail, and what their failures do to the analyses built on top of them. Chapter 3: Where Vision Fails and Why It Matters The output of a tracking system looks authoritative. A file of coordinates given to two decimal places of a metre, sampled twenty-five times a second, has the visual texture of precision. It is natural to treat it as a record of what happened. It is better treated as a set of measurements, each with an error, and the errors are not random noise scattered evenly over the match. They cluster in particular situations, they grow when quantities are differentiated, and they differ from one system to another in ways that can quietly change the conclusions of an analysis. This chapter is about those errors. It is the least glamorous chapter of the book and one of the most practically important, because almost every application discussed later, from valuing a pass to deciding whether a player's training load is too high, inherits the properties of the measurement beneath it. What accuracy means for a tracking system Asking how accurate a tracking system is invites a single number, and vendors are happy to supply one. But accuracy has several distinct components, and a system can be excellent on one and poor on another. Positional accuracy is the distance between the reported and true location of a player at a given instant. It is usually summarised as a mean or root-mean-square error in centimetres, measured against a reference system such as a high-end motion-capture installation or surveyed markers. Completeness is the fraction of the time each player is actually tracked; gaps can be filled by interpolation, but filled values are estimates, not measurements. Identity accuracy is the fraction of the time each trajectory is attached to the correct player; an identity swap may leave every individual position accurate while assigning ten seconds of a centre-back's movement to the striker. Temporal accuracy concerns synchronisation, whether the timestamp on a frame matches the timestamp on the event log, the ball sensor, or another camera. And derived-quantity accuracy concerns the speeds, accelerations and distances computed from the positions, which, as discussed below, can be much worse than the positions themselves. Independent validation of sports tracking has improved over the past decade. Researchers at the Technical University of Munich, notably Daniel Linke, Daniel Link and Martin Lames, published a series of studies from 2018 onward comparing optical, radio and GPS systems in football against a reference motion-capture system under match-like conditions. Their general findings have held up: position estimates from good systems are accurate to within a fraction of a metre, but errors grow at high speeds and during rapid changes of direction, and derived measures such as accelerations differ markedly between systems. FIFA established a quality programme for electronic performance and tracking systems, testing commercial products against a reference system and certifying those that meet set standards, which has given clubs a common baseline for comparison. Broadcast-derived tracking is harder to validate, because the true positions of players in a televised match are usually unknown. Providers validate against installed systems where both are available, and publish summary figures; independent evaluations are scarcer. The broad picture is that broadcast tracking is now good enough for many tactical and recruitment purposes but less reliable than installed systems for physical metrics, and markedly less complete, because players off screen are not observed at all and must be estimated or omitted. The derivative problem The most important technical point in this chapter is also the simplest. Speed is the rate of change of position. Acceleration is the rate of change of speed. Each differentiation amplifies noise. Suppose a system reports a player's position every tenth of a second with a random error of ten centimetres. Computing speed from two successive positions divides the difference in positions by a tenth of a second. A random error of ten centimetres in each position can produce an apparent movement of a few tenths of a metre, which, divided by a tenth of a second, is an apparent speed error of several metres per second. Acceleration, computed by differencing two such noisy speeds, is worse again. Raw, unsmoothed acceleration traces from tracking data are close to meaningless. Every system therefore smooths. Positions are filtered, often with a low-pass filter or a Kalman smoother, before speeds and accelerations are computed. But smoothing involves a choice, and the choice changes the answer. Heavy smoothing suppresses noise but also suppresses genuine rapid changes, so that a sharp deceleration as a player plants to change direction is spread over a longer interval and reported as gentler than it was. Light smoothing preserves rapid events but lets noise through, so that the count of high-intensity accelerations rises spuriously. Studies of GPS data in team sports have shown that the number of accelerations above a fixed threshold recorded for the same activity can differ substantially depending on the device's filtering settings and the minimum duration required for an effort to count. This matters enormously for load management, discussed later in the book. The number of high-intensity accelerations and decelerations a player performs is one of the measures most often tied to soft-tissue injury risk. If that number depends heavily on a filter setting chosen by an engineer at the device manufacturer, then comparisons across devices, across seasons when firmware was updated, and across clubs using different systems are unreliable. A practitioner who sees a jump in a player's acceleration count should first ask whether anything changed in the measurement chain. A simple hypothetical shows how large the effect can be. Take a single change-of-direction drill in which a player decelerates hard from a sprint, plants and turns. A tracking system with lightly filtered output may record two distinct high-intensity decelerations, because the noisy speed trace dips below the threshold, rebounds slightly and dips again. A system with heavier filtering may record one deceleration of lower peak magnitude. A third system, requiring that an effort last at least half a second to count, may record none, because the smoothed deceleration crosses the threshold only briefly. All three are reporting the same movement honestly according to their own rules. Multiply that discrepancy across hundreds of efforts in a week and the totals can differ by a large proportion, enough to move a player across any threshold a monitoring system uses to flag risk. Where the errors cluster Tracking errors are not uniformly distributed across a match. Table 2 summarises the main failure modes, when they occur, and which downstream analyses they distort most. Table 2. Common tracking failure modes and their downstream effects. Failure mode When it occurs Most affected analyses Occlusion Set pieces, crowded boxes, tackles Pressure, marking, pose at key moments Identity swap Players crossing, same-kit clusters Individual load, player-level value models Off-screen gaps Broadcast footage, wide play Team shape, pressing, distance covered Calibration drift Tight zoom, few visible lines Positions near touchlines, offside-type judgments Derivative noise Sprints, sharp turns Accelerations, high-intensity counts Ball loss Aerial balls, ball hidden by feet Possession timing, pass speed, event sync The pattern is unwelcome. Occlusion and identity confusion peak during set pieces and goalmouth scrambles, which are disproportionately where goals are scored and conceded. Derivative noise peaks during sprints and sharp changes of direction, which are disproportionately where muscle injuries occur. The ball is hardest to track when it is in the air or among feet, which is when possession is contested. The moments that matter most are the moments measured worst. This is not a reason to distrust tracking data wholesale. It is a reason to treat the data as having a reliability that varies with context, and to be particularly cautious about analyses that focus on the high-leverage moments. A tactical model of corner kicks, for instance, rests on positions captured during exactly the situation in which tracking is weakest. Pose estimation's particular weaknesses Skeletal tracking inherits all the weaknesses of positional tracking and adds its own. Joint estimates are least reliable for the extremities, wrists and ankles, which move fastest and are smallest in the image, and most reliable for the large central joints, shoulders and hips. Self-occlusion is pervasive: a player seen side-on hides one arm and one leg behind his own torso, and the network must infer where they are. Left-right confusion is a classic pose-estimation error, in which the model correctly locates both knees but swaps their labels; for gait analysis or injury biomechanics, that error can invert the conclusion. Monocular three-dimensional lifting adds a further layer of uncertainty. Depth, the distance from the camera, is the least constrained dimension, and errors in depth are typically much larger than errors across the image. A broadcast-derived three-dimensional skeleton may be adequate to tell whether a player's body is open or closed to the ball, and quite inadequate to measure the angle of knee flexion at foot strike with the precision a biomechanist would need to assess injury risk. This is why biomechanical assessment for injury purposes still relies largely on controlled settings: laboratory motion capture with reflective markers, or markerless systems with many calibrated cameras in a fixed space. The quantity of interest, a joint angle or a load on a ligament, is sensitive to errors of a few degrees, and match-video pose estimation is not yet reliably that precise. The gap is closing, and markerless systems validated against marker-based capture are increasingly used in clinics and training centres, but the claim that broadcast video can be mined for injury biomechanics should be met with a request for validation data. Bias as well as noise Some errors are not random but systematic, and systematic errors are more dangerous because they do not average out. A detection model trained mostly on men's top-division football, filmed by high-end broadcasters in well-lit stadiums, may perform worse on lower divisions with lower camera positions, on youth matches with smaller players, on women's football if the training data under-represents it, or on footage from weaker lighting conditions. If a recruitment department uses broadcast tracking to compare a player in a top league with a player in a smaller league, part of the difference in their metrics may reflect the difference in measurement quality rather than in the players. The same concern applies within a match. If a system systematically underestimates the speed of players far from the camera, because they occupy fewer pixels, then full-backs on the far touchline will consistently appear slower than those on the near side. These are not hypothetical worries; they are the kinds of effects any serious validation study looks for. They also reinforce a general rule: comparisons are safest within a single system, under similar conditions, and become progressively riskier as the conditions diverge. Living with imperfect measurement The response to imperfect measurement is not to wait for perfect measurement. It is to design analyses that are robust to the errors that exist. Several practices help. The first is to propagate uncertainty rather than discard it. Pose networks produce confidence scores; trackers know when they have interpolated across a gap; calibration routines know when they are working from too few lines. If those signals are passed downstream, a model can down-weight unreliable frames rather than treat them as equally trustworthy. Most commercial data products do not expose this information in detail, and asking for it is a reasonable demand from a sophisticated buyer. The second is to aggregate appropriately. A single acceleration count from a single session is noisy; a rolling average over weeks is much more stable. A single pass value is uncertain; a player's average over a season is much less so. Many of the most robust uses of tracking data operate at aggregate levels where errors partially cancel. The third is to keep the measurement chain fixed. A club that changes GPS supplier, updates firmware or moves from installed to broadcast tracking should expect discontinuities in every derived metric and should run old and new systems in parallel for a period to calibrate one against the other. Historical comparisons across such changes should be treated with caution. The fourth, and most important, is to remember what the data is for. A tracking system used to decide offside must be accurate at a single critical instant to within a few centimetres, which is why FIFA invests in dedicated cameras, three-dimensional body scans and human review. A tracking system used to describe how a team's defensive line typically shifts when the ball goes wide needs only to be roughly right most of the time. Asking whether a system is accurate enough is meaningless without asking: accurate enough for what? The rest of this book is, in part, an answer to that question for the uses that matter most. Chapter 4: Putting a Value on Space and Action Tracking data tells you where everyone was. It does not tell you whether that was good. The step from measurement to meaning requires a model that assigns value to situations and actions: how much more likely is a goal now than a second ago, how much of the pitch does this team control, how much did that pass, or that run without the ball, change the expected outcome of the possession? These valuation models are the intellectual core of modern sports analytics. Everything that follows in the tactical chapter, and much of what clubs do in recruitment, depends on them. The models share a common logic, which is worth stating plainly before describing any one of them. A sporting contest can be treated as a sequence of states. Each state has a value, defined as the expected eventual outcome from that state, such as the probability of scoring in the current possession minus the probability of conceding. Each action moves the game from one state to another. The value of an action is the change in state value it produces. Once you can estimate state values, you can credit actions, and once you can credit actions, you can credit players. The difficulty lies entirely in estimating state values well, and that depends on how much of the state you can observe. Expected goals: the first state-value model The most familiar valuation model is expected goals, or xG. An xG model estimates the probability that a given shot becomes a goal, based on features such as its distance and angle to goal, the body part used, the type of assist and whether it followed a set piece. It is trained on tens or hundreds of thousands of historical shots, typically with logistic regression or gradient-boosted trees, and its outputs are calibrated probabilities: among all shots assigned an xG of 0.1, roughly one in ten should have been scored. Expected goals is a state-value model with a very narrow state: the moment of the shot. Its power comes from replacing a noisy, binary outcome, goal or no goal, with a smoother estimate of chance quality. Over a handful of matches, goals are dominated by luck; over a season, a team's xG difference is a better predictor of its future results than its actual goal difference. That single finding did more than anything else to persuade football clubs that analytics had something to offer. Expected goals also illustrates the limits of event data. A standard xG model knows where the shot was taken but not, unless tracking is added, where the goalkeeper and defenders were. A shot from twelve metres with the goalkeeper stranded and no defender in the way is a very different chance from a shot from the same spot through a crowd. Providers who capture the positions of players at the moment of the shot, whether from tracking or from manual annotation of freeze-frames, produce noticeably better models. The improvement is a small preview of what tracking adds to every valuation problem. Extending value to every action If shots can be valued, so can the actions that lead to them. Two influential approaches extended valuation from the shot to the whole possession using event data. Expected threat, popularised by the analyst Karun Singh in 2018, divides the pitch into a grid and estimates, for each cell, the probability that possession of the ball in that cell will lead to a goal within the next few actions. It does so with a simple Markov model: from each cell, the ball is shot with some probability, moved to another cell with some probability, or lost. Solving the resulting equations gives a value for each cell. A pass or carry is then valued by the difference in expected threat between where the ball started and where it ended. The method is transparent, easy to compute and surprisingly informative; its weakness is that it ignores everything except the ball's location. VAEP, short for Valuing Actions by Estimating Probabilities, published by Tom Decroos, Lotte Bransen, Jan Van Haaren and Jesse Davis at the KDD conference in 2019, took a more flexible approach. It trains classifiers to estimate, for the game state after each action, the probability of scoring and the probability of conceding within the next ten actions, using features of the last few actions: their types, locations, outcomes and timing. An action's value is the change in scoring probability minus the change in conceding probability. Because it accounts for defensive risk and uses richer context, VAEP can credit actions such as a tackle that stops a dangerous attack, which expected threat struggles to value. The authors also defined SPADL, a standard language for representing on-ball actions from different data providers, which made it easier for others to build on the work. Both methods remain fundamentally on-ball. They can tell you how much a pass increased the team's chances. They cannot tell you about the run that created the space for the pass, or the defensive positioning that made the alternative passes impossible. Pitch control: who owns the space Tracking data makes it possible to model the off-ball game, and the central concept for doing so is pitch control. A pitch-control model estimates, for every location on the pitch at a given instant, the probability that each team would gain possession if the ball were sent there. The best-known formulation was published by William Spearman, then at Hudl, in a 2018 paper for the MIT Sloan Sports Analytics Conference titled "Beyond Expected Goals". Spearman's model is physically motivated. For each location, it computes how long each player would take to reach it, given their current position and velocity and assumptions about maximum speed and acceleration. It also computes how long the ball would take to travel there. Players who can arrive before or soon after the ball have a chance of controlling it, accumulated over time according to a simple probabilistic rule. Summing over players gives each team's probability of control at that point. Repeated over a fine grid, this produces a map of the pitch in which every location is shaded by which team owns it. At the same conference in 2018, Javier Fernández and Luke Bornn presented a related model, "Wide Open Spaces", which used a statistical rather than a physics-based approach to define each player's area of influence as a function of position, velocity and distance to the ball. The two papers, together with others in the same period, established space control as a measurable quantity. Pitch control turns tactical language into numbers. "He dragged the centre-back out of position" can be expressed as a change in the control surface: the run reduced the defending team's control of a dangerous region. "They squeezed the space between the lines" becomes a measurable reduction in the attacking team's control of a band in front of the defence. Combined with a value surface, an estimate of how valuable possession at each location would be, pitch control produces a measure of how much valuable space each team owns at each moment, and how actions on and off the ball change it. Expected possession value: valuing everything at once The most ambitious valuation models try to estimate state value using the full tracking state: every player's position and velocity, the ball's position, and the context of the possession. This idea, called expected possession value or EPV, was introduced in basketball by Dan Cervone, Alexander D'Amour, Luke Bornn and Kirk Goldsberry, using SportVU tracking data; their full statistical treatment appeared in the Journal of the American Statistical Association in 2016. Their model treated a basketball possession as a stochastic process that moves between coarse states, such as a particular player holding the ball in a particular region, and fine-grained states given by the full tracking data, and it estimated at every moment the expected number of points the possession would produce. Javier Fernández, Luke Bornn and Dan Cervone later adapted EPV to football, publishing a framework that decomposes the value of a possession into the probabilities and values of the main options available to the player on the ball: pass, shoot or carry. Each component is estimated by a separate model, many of them deep neural networks that take as input a stack of spatial maps encoding the positions and velocities of players and the ball. The decomposition matters because it makes the model's output inspectable: an analyst can see not only that a situation was valuable, but which option made it so and which passes were available. With EPV, the off-ball game becomes valuable in its own right. A run that pulls a defender away and raises the EPV of the ball-carrier's situation, even if the ball never goes to the runner, can be credited. A defender whose positioning consistently lowers the opponent's EPV, by cutting off the most valuable passing lanes before they open, can be distinguished from one who merely makes many tackles because he is often out of position. A worked example: valuing a run that receives nothing Consider a concrete situation. A team is building an attack down the right. The right-back has the ball near the halfway line. The right winger is wide and level with him; the striker is on the shoulder of the opposing centre-backs; a central midfielder is twenty metres infield. The winger now makes a curved run inside and forward, into the channel between the opposing left-back and left centre-back. The left-back follows him. The ball never goes to the winger. Instead, the right-back plays a pass down the line into the space the left-back has vacated, where the central midfielder, who has drifted wide, collects it and crosses. An event-data model sees two actions: the right-back's pass and the midfielder's cross. It credits the pass for moving the ball into a more dangerous area. It knows nothing about the winger, who appears in the event log nowhere at all. A pitch-control model sees more. Before the winger's run, the area down the right touchline in the attacking third was controlled by the defending team, because the left-back was close to it. As the winger ran and the left-back followed, control of that area shifted to the attacking team; the midfielder, moving wide, could now reach it before any defender. A possession-value model goes a step further: it estimates that, before the run, the most valuable option available to the right-back was a short, safe pass; after the run, a pass into the vacated channel became available and carried a substantially higher expected value. The difference between those two values, attributable to the change in the defending team's shape that the winger's run caused, can be credited at least partly to the winger. How much should be credited to him, as opposed to the right-back who saw the pass or the midfielder who moved into the space, is a modelling decision rather than a fact. Different credit-assignment schemes give different answers, and none is uniquely correct. But even a rough answer is a large advance over the event-data view, in which the most important action of the move does not exist. Comparing the approaches The valuation models differ in what data they need, what they can credit, and how easily their outputs can be explained. Table 3 sets them side by side. Table 3. Main families of valuation model in football. Model Data needed What it values Main limitation Expected goals Event data, optionally freeze-frames Shots Ignores everything before the shot Expected threat Event data Ball movement by location Ignores player positions VAEP Event data All on-ball actions, incl. defending No off-ball movement Pitch control Tracking data Space owned by each team Needs a separate value surface Possession value (EPV) Tracking data Situations, options and off-ball effects Complex, data-hungry, harder to validate Ghosts and counterfactuals A valuation model tells you what a situation was worth. Coaches often want to know something different: what should have happened. If a defender had positioned himself two metres further infield, would the goal have been prevented? A line of work begun around 2017 tried to answer that question by training models to imitate typical behaviour. Hoang Le, Yisong Yue, Peter Carr and Patrick Lucey, working with tracking data from a top European league, trained deep imitation-learning models to predict where defenders would normally move in a given situation, a technique the authors called "data-driven ghosting". The model's prediction can be drawn as a ghost alongside the real player, and the gap between them indicates where a player departed from the league's typical behaviour. Teams could even condition the model on a particular opponent's habits to simulate how that team would have defended a given attack. Ghosting is powerful and easily misread. The ghost represents what an average defender in the training data would have done, not what an optimal defender would have done. A player who departs from the ghost may be making a mistake, or may be doing something better than average. The model has no way to tell the difference unless it is combined with a valuation model that scores both the ghost's and the real player's positioning, and even then the counterfactual rests on the assumption that the attacking players would have done exactly what they actually did, which in reality would have changed in response. What valuation models can and cannot do Valuation models are the bridge between tracking and decision-making, and three general points about them carry through the rest of this book. First, every value is an expectation conditioned on the data the model can see. A model that does not see body orientation cannot know that a passing lane was unavailable because the passer was facing the wrong way. A model trained on one league's patterns encodes that league's typical tactics as its baseline. The values are not facts about football; they are facts about the model. Second, valuation models are descriptive. They estimate how valuable situations were given how the game has typically been played. They do not, on their own, say what would happen if a team played differently, because a team that changed its behaviour would face opponents who changed theirs in response. This limitation becomes central when the same models are used to optimise tactics, the subject of the next chapter. Third, validation is hard and often neglected. An xG model can be checked against goals. An EPV model can be checked against eventual outcomes of possessions, but the outcomes are rare and noisy, and two models with quite different credit assignments can fit the outcomes equally well. When a valuation model says that a midfielder's off-ball runs were the most valuable in the league, the claim is a combination of measurement, modelling choices and assumptions, and should be treated as a strong hint for human scouts to investigate rather than a verdict. Chapter 5: Formations, Tactics and the Limits of Optimization Coaches have always thought in shapes. A formation written as 4-3-3 or 3-5-2 is a compressed description of where players are expected to stand, how responsibilities are divided, and how the structure should change between attack and defence. Tracking data allows those shapes to be observed rather than declared, and machine learning offers to go further: to identify which structures work best against which opponents, and to recommend adjustments. This chapter describes what has been achieved in detecting and evaluating tactical structure, the most serious attempt so far to generate tactical recommendations with artificial intelligence, and the reasons why tactical optimization is a fundamentally different problem from optimization in most other domains. Formations are not what the team sheet says The first contribution of tracking data to tactics was descriptive. A team announced as playing 4-3-3 may defend in a 4-5-1 when the wingers drop back, attack in a 2-3-5 when the full-backs push high and one midfielder drops between the centre-backs, and shift between these shapes several times a minute. The nominal formation is a starting point, not a description. Detecting the real structure from tracking data is harder than it sounds, because players constantly swap positions. A winger and a full-back may exchange places on an overlapping run; two central midfielders may rotate who drops and who advances. If each player's average position is computed over a match, these rotations blur the picture into an unhelpful cloud. Alan Bialkowski, Patrick Lucey and colleagues addressed this in work published in 2014 by assigning players to roles rather than tracking individuals: at each moment, players are matched to a set of role positions in a way that minimises total displacement, so that whichever player is currently in the left-back position is treated as the left-back. Averaging over roles rather than people recovers a crisp formation even when individuals rotate freely. Laurie Shaw and Mark Glickman extended the idea in 2019, clustering the shapes a team adopts in and out of possession over short intervals and showing how teams moved between a small number of characteristic configurations through a match. These methods have practical value immediately. Opposition analysts can describe how an upcoming opponent builds up from the back, how its shape changes when the ball goes wide, and where gaps habitually appear during transitions. Coaches can check whether their own team adopts the shapes they have trained. Recruitment departments can identify players who have played in a particular role in a particular structure, which is often more informative than the positional label in a database. From describing to evaluating Evaluating a tactical structure requires connecting shapes to outcomes, which brings in the valuation models of the previous chapter. One approach is to measure, for each phase of play, how much valuable space each team controls under different structures. A team that builds up with three players behind the ball rather than two may concede less space to the press but control less space higher up. Pitch-control and possession-value models allow that trade-off to be quantified for a particular team over a season. Another approach examines specific tactical behaviours. Pressing is the clearest example. Tracking data allows analysts to identify when a team is pressing, which players initiate it, how quickly the rest follow, and whether the press forces the ball into the areas the team intended. Measures such as the number of opponent passes allowed per defensive action, first popularised as a crude event-data proxy for pressing intensity, have been superseded by tracking-based measures of pressure on the ball-carrier and the passing options available to him. The evaluation question becomes concrete: when this team presses in this way, how often does it win the ball in a dangerous area, and how often does it get bypassed and exposed? These evaluations are useful, but they are retrospective. They describe how well a team's structure has worked against the opponents it actually faced, with the players it actually had. The step from there to recommending a better structure is where the difficulties begin. TacticAI and corner kicks The most prominent attempt to generate tactical recommendations with modern machine learning is TacticAI, developed by Google DeepMind in collaboration with Liverpool Football Club and published in Nature Communications in March 2024 by Zhe Wang, Petar Veličković and colleagues. It deliberately chose a narrow problem: corner kicks. Corners are well suited to the purpose. They begin from a fixed restart, so every example starts from a comparable situation. They are frequent enough to provide data, around ten per Premier League match, and the outcome, a shot or no shot, a first touch by attacker or defender, arrives within seconds. And they are a recognised source of goals in which preparation pays off, so coaches already spend time designing and rehearsing them. TacticAI represents each corner as a graph. Each player is a node carrying features such as position, velocity, height and weight; edges connect players to one another. A graph neural network passes messages along these edges, so that each player's representation is updated by information about the others, and after several rounds the network can make predictions about the situation as a whole. The model was trained on 7,176 corners from Premier League matches. It performs three tasks: predicting which player is most likely to receive the ball, predicting whether a shot will result, and suggesting adjustments to the positions and movements of players that would make a desired outcome, such as an attacking team getting a shot or a defending team preventing one, more likely. A notable design choice was to build in the geometry of the problem. A corner taken from the left is, tactically, a mirror image of one taken from the right. Rather than hope the network would learn this from data, the authors constrained it to treat reflections of the pitch consistently, an application of the broader approach known as geometric deep learning. This made more efficient use of a dataset that, by the standards of modern machine learning, is small. The evaluation is as interesting as the model. Five experts from Liverpool, three data scientists, a video analyst and a coaching assistant, were shown real corner setups alongside setups generated or adjusted by TacticAI, without being told which was which. They frequently found the generated setups hard to distinguish from real ones, and when asked to choose between TacticAI's suggested adjustments and the original positioning, they favoured the suggestions in about ninety per cent of cases. What TacticAI shows, and what it does not TacticAI is a genuine advance and a useful case study precisely because its authors were careful about its scope. It shows that graph-based models can capture the structure of a set piece well enough to predict receivers and shot likelihood, and to propose adjustments that experienced practitioners find credible and useful. It shows that building in known symmetries improves learning from limited data. And it presents its output as an assistant for coaches to review rather than a replacement for their judgment. What it does not show is that the suggested adjustments win more matches. The expert evaluation measures whether practitioners find the suggestions plausible and preferable, which is valuable but is not the same as measuring their effect on the pitch. That gap is not a criticism of the work; closing it is extremely difficult, for reasons that apply to all tactical optimization. Why tactics resist optimization In many fields, machine learning optimizes against a fixed environment. A model that recommends a better delivery route does not cause the roads to rearrange themselves in response. Sport is different in at least four ways, and each places a limit on what data-driven tactical recommendation can achieve. The opponent adapts. A tactic's value depends on what the opponent does, and opponents watch film. If a team discovers through analysis that a particular corner routine produces good chances, and uses it repeatedly, opposing analysts will identify it and defend against it. The historical data on which the routine was evaluated describes opponents who were not expecting it. This is the logic of a game in the mathematical sense: the best strategy depends on the other side's strategy, and a model trained on past behaviour estimates the value of actions against a population of opponents that will change once the action becomes known. Some of the value of a novel set piece lies precisely in its novelty, which by definition is not represented in historical data. The data is off-policy. Models learn from what teams actually did. If no team in the dataset ever placed four players on the six-yard line at a corner, the model has no evidence about what happens when one does. Recommendations that move far from the observed range of behaviour are extrapolations, and the model's confidence in them is not trustworthy. Generative models that produce realistic-looking setups are, by design, staying close to what has been seen; that makes their suggestions plausible but limits how radical they can be. Outcomes are rare and noisy. Goals are scarce in football. Even shots, the intermediate outcome many models target, are affected by execution: the quality of the delivery, a slip, a deflection. Distinguishing a genuinely better tactic from a lucky run requires far more repetitions than any single team generates in a season. This is why set pieces, with their repeated, comparable starting conditions, are the natural first target, and why open-play tactics are much harder. Players are not interchangeable. A tactic that works with one set of players may fail with another. A high defensive line requires fast centre-backs; a build-up through a deep-lying playmaker requires a player who can receive under pressure. Models trained on league-wide data learn average relationships, and the coach's question is almost always specific: what should my players do against this opponent? Data for that specific question is always thin. Simulation and its limits One response to the scarcity of real data is simulation. If a sufficiently realistic model of a sport existed, tactics could be tested in millions of simulated matches. Reinforcement learning agents have been trained in simulated football environments, notably the Google Research Football environment released in 2019, and they learn coordinated behaviours such as passing combinations and offside traps. The difficulty is the gap between simulation and reality. Simulated players have simplified physics, perfect information or artificially constrained information, and no fatigue, fear or miscommunication. Tactics that exploit quirks of a simulator do not transfer. More promising is the use of learned models of real behaviour, such as the ghosting models described in the previous chapter, as simulators: if a model can predict how real defenders typically respond to an attacking movement, it can be used to evaluate attacking variations against realistic defensive responses. But those responses are themselves learned from historical play, and inherit the off-policy problem. The NFL offers a different and instructive use of simulation. When the league redesigned the kickoff for the 2024 season to reduce high-speed collisions, it drew on tracking data and modelling, in its work with Amazon Web Services, to simulate the effects of rule options on player speeds and collision types. Rule design is a more tractable problem than tactical optimization, because the league is changing the environment for everyone at once and is interested in aggregate safety outcomes rather than in beating a particular opponent. Scouting opponents at scale The most routine and perhaps most valuable tactical use of artificial intelligence is not recommending tactics but describing opponents efficiently. Before every match, opposition analysts watch hours of the next opponent's recent games, tagging patterns: how they build up, which side they favour, how they defend set pieces, who presses and who does not. This work is labour-intensive, and its quality depends on how many matches an analyst can watch in the days available. Tracking and event data, combined with pattern-detection methods, accelerate it. Possessions can be clustered by their trajectories, so that an analyst sees at a glance that an opponent's attacks fall into a handful of recurring types, perhaps a long diagonal to a target winger, a slow build-up through a midfielder dropping between the centre-backs, and quick combinations down the left, each with its frequency and typical outcome. Defensive set-piece structures can be classified automatically: zonal, man-to-man or hybrid, with the positions of the zonal markers and the assignments of the man-markers. Search tools allow an analyst to ask for every instance in the last ten matches where the opponent lost the ball in its own half and conceded a shot within ten seconds, and receive the video clips directly. None of this replaces the analyst's judgment about what matters. Its effect is to shift time from finding clips to interpreting them. The analyst who once spent most of the week assembling footage can spend more of it deciding which two or three patterns are worth the players' limited attention. That, too, is translation: from a comprehensive statistical description to a short list of things a squad can absorb in a meeting and act on in a match. FIFA's own tournament tools point in the same direction. At the 2026 World Cup, the governing body provided every one of the forty-eight teams with a generative-AI knowledge assistant to support pre-match and post-match analysis, so that smaller federations without large analytics staffs had access to the same baseline of information as the best-resourced teams. Whether such tools narrow or merely reshape the gap between rich and poor teams depends on what each staff does with the output, which returns the argument to the people in the loop. The analyst as translator Given these limits, the most effective tactical uses of artificial intelligence look less like optimization and more like structured exploration. A model identifies patterns in an opponent's defending that an analyst might not have noticed, such as a full-back who reliably steps out when the ball goes to a particular zone. Valuation models quantify how often a team's own structure leaves dangerous space and in which phases. A generative model like TacticAI proposes adjustments that a coach can review, discard or adapt. The coaching staff decide which ideas to test in training, watch how the players execute them, and judge whether they suit the squad. In this loop, the decisive skill is translation: turning a model's output into a coaching point a player can act on, and turning a coach's intuition into a question a model can test. Clubs that have built strong analytics functions tend to place analysts close to coaching staff, with shared language and frequent contact, rather than producing reports from a separate department. The model can see patterns across thousands of situations that no human can hold in mind. The coach knows the players, the opponent's manager, the state of the dressing room and the weather forecast. The best tactical decisions combine both, and the history of tactical analytics so far suggests that neither alone is sufficient. Chapter 6: Measuring the Load on the Body The second half of this book turns from the game to the athlete's body. Here the questions are no longer about who controls space or which corner routine works, but about how much stress a player has absorbed, whether the body has adapted to it or is accumulating damage, and when a player is fit to train or play. These questions have always been central to coaching and sports medicine. What has changed is the volume and resolution of the data available to answer them, and the arrival of machine-learning methods that promise to turn the data into forecasts. Before examining those forecasts, it is necessary to understand what is being measured and what physiological thinking lies behind the measurements. Most of the confusion in this field, including much of the overselling of injury prediction, comes from treating load numbers as if they were direct readings of the body's state, when they are proxies built on simplified models of how the body adapts. External and internal load Sports scientists distinguish two kinds of load. External load is the work an athlete performs, measured independently of how the body responds: distance run, distance at high speed, number of sprints, number of accelerations and decelerations, number of jumps, number of pitches thrown, total tonnage lifted in the gym. Tracking systems and wearable devices measure external load. Internal load is the physiological and psychological stress that work imposes on the individual: heart-rate response, blood lactate, perceived effort, hormonal changes. Two players who run the same distance at the same speeds may experience very different internal loads depending on their fitness, fatigue, illness, heat, and sleep. The distinction matters because adaptation and injury are driven by internal load acting on specific tissues, while the most easily measured quantities are external. A centre-back and a winger may record similar total distances in a match, but the winger's distance contains many more maximal sprints, which load the hamstrings heavily, while the centre-back's contains more aerial duels, which load the landing mechanics of the knee and ankle. Neither total distance nor heart rate captures these tissue-specific demands well. The simplest and most widely used internal-load measure is also the cheapest. Session rating of perceived exertion, popularised by Carl Foster and colleagues in a 2001 paper, asks athletes about thirty minutes after a session to rate how hard it was on a ten-point scale, and multiplies the rating by the session's duration in minutes. The resulting number, in arbitrary units, correlates reasonably well with heart-rate-based measures across many sports and is used by clubs at every level. Its persistence alongside expensive tracking technology says something important: an athlete's own report of how a session felt contains information that no sensor captures, and well-run load-monitoring systems treat it as data rather than as a soft supplement to the real numbers. The fitness-fatigue model Behind most load monitoring sits a simple model of adaptation proposed by Eric Banister and colleagues in the 1970s. In Banister's impulse-response model, each training session produces two effects: a fitness effect that improves performance and decays slowly, and a fatigue effect that impairs performance and decays quickly. Performance at any moment is a baseline plus the accumulated fitness minus the accumulated fatigue. Shortly after hard training, fatigue dominates and performance dips; after a period of reduced training, fatigue dissipates faster than fitness and performance peaks. This is the logic of tapering before a competition. The model is elegant and captures something real, but it is a caricature of physiology. It treats fitness and fatigue as single quantities when in reality different systems, cardiovascular, neuromuscular, metabolic, connective tissue, adapt and fatigue on different timescales. It requires parameters fitted for each individual, and those parameters are hard to estimate reliably from the limited and noisy data of a real training programme. In applied practice, its main legacy is conceptual: the idea that the effect of load depends on what preceded it, that recent load and longer-term load should be compared, and that the body's capacity is built by the same stresses that temporarily reduce it. The acute:chronic workload ratio and its fall The most influential load metric of the past decade, and the most contested, grew directly from this thinking. The acute:chronic workload ratio compares an athlete's recent load, typically the past week, with the load to which the athlete has become accustomed, typically the average over the past four weeks. A ratio near one indicates that recent training matches what the body is prepared for. A ratio well above one indicates a spike: the athlete is doing much more than recently accustomed to. Work by Tim Gabbett, Billy Hulin and colleagues in cricket, rugby league and Australian football, summarised in Gabbett's widely cited 2016 paper in the British Journal of Sports Medicine on the "training–injury prevention paradox", argued that injury risk was elevated when the ratio rose above roughly 1.5, and lowest in a range of roughly 0.8 to 1.3. The paper also made a broader argument that was valuable and remains so: high chronic training loads are protective rather than dangerous, because a body accustomed to hard work tolerates hard work, and it is rapid increases, not high loads as such, that carry the most risk. The ratio's simplicity and its intuitive "sweet spot" made it extremely popular. Monitoring software began displaying it in traffic-light colours, and many clubs adopted the thresholds as rules. Criticism followed, and it has been substantial. Researchers pointed out that the most common way of computing the ratio, with the acute week included inside the chronic four-week average, creates a mathematical coupling between numerator and denominator that can produce spurious associations; Lolli and colleagues set out this problem in the British Journal of Sports Medicine in 2019. Williams and colleagues had already argued in 2017 that exponentially weighted moving averages were a better way of representing decaying load than simple rolling averages, and that the choice of method changed the results. Franco Impellizzeri, Aaron Coutts and colleagues, in a 2020 paper in the International Journal of Sports Physiology and Performance, argued that the ratio had fundamental conceptual problems: ratios behave poorly as statistical variables, the underlying evidence came largely from observational studies with methodological weaknesses, and the thresholds had been treated as established when they were not. Systematic reviews found inconsistent associations between the ratio and injury across studies. The debate has not ended in a simple verdict, and many practitioners still find the comparison of recent and accustomed load useful as a way of flagging sudden changes for discussion. But the episode carries a lesson that applies with even more force to machine-learning models. A metric that is easy to compute, easy to display and plausibly connected to a real physiological idea can be adopted across an entire industry before its statistical properties have been examined. Once adopted, it shapes decisions: players are rested, sessions are cut short, return-to-play schedules are planned around it. A colour-coded dashboard confers a kind of authority that the underlying evidence may not support. What wearables and tracking add Modern tracking and wearable systems have multiplied the number of load variables available. A football club using GPS vests may record dozens of variables per session: total distance, distance in several speed bands, number of sprints, peak speed, accelerations and decelerations above several thresholds, a composite accelerometer-based load figure, metabolic power estimates and more. Basketball and other indoor sports add jump counts and landing forces. Baseball counts pitches and, increasingly, measures the biomechanics of each delivery from Hawk-Eye skeletal tracking or dedicated motion capture. Three developments in this area are especially relevant to artificial intelligence. First, individualised baselines: rather than comparing an athlete to fixed thresholds, systems learn each athlete's typical response to typical loads and flag deviations. A player whose heart rate during a standard warm-up drill is unusually high, or whose peak speed in a standard sprint is down, may be fatigued or unwell, even if their external load looks normal. Second, neuromuscular readiness testing, typically a countermovement jump on a force plate before training, offers a direct functional measure of fatigue, and machine-learning models can decompose the force trace into features that are more sensitive than jump height alone. Third, the fusion of multiple streams, external load, internal load, readiness tests, sleep, wellness questionnaires, into a single picture of each athlete is a natural task for statistical models, because no human can track dozens of variables for thirty players every day. A match week in the data The way these measures are used becomes clearer when followed through a typical week. Many football clubs organise training around the match day, labelling each day by its distance from the game: the day after a match is MD+1, the day before is MD-1. In a week with a single Saturday match, the pattern often runs roughly as follows. On MD+1, players who played the full match do recovery work, and their monitoring focuses on readiness: wellness questionnaires, perhaps a jump test, and a check of the match's external load. Players who did not play, or played only briefly, often do a compensatory session designed to replace the high-speed running and sprinting they missed, because without it they would fall behind the starters in the very qualities that protect against injury. Tracking data from the match tells staff exactly how much each substitute needs. In the middle of the week, at MD-4 and MD-3, come the most demanding sessions: one emphasising strength and short, intense efforts in small-sided games with many accelerations and decelerations, another emphasising larger spaces, longer runs and exposure to high speeds. Here external-load targets are planned in advance and checked against what the GPS recorded. If a session's sprint exposure falls short because a drill was shorter than intended, staff may add a brief top-up. Session RPE is collected afterwards to see how the work was perceived. Toward the match, MD-2 and MD-1 taper: shorter sessions, lower volume, sharp but brief efforts to keep players primed without accumulating fatigue. Readiness measures on these days guide individual adjustments. A player whose jump test is unusually poor, or who reports poor sleep and soreness, may do a reduced session. At every stage the data supports a decision someone was going to make anyway: how hard to train, who needs extra work, who needs less. It does not replace the plan, which comes from coaching principles about how to prepare a team for competition. In weeks with two matches, the pattern compresses and the data's role shifts toward recovery and rotation decisions, which is often where the most consequential load-management choices are made. Biomechanical load from video Skeletal tracking promises to add a different kind of load measure: not how much an athlete moved, but how. Two athletes may perform the same number of decelerations, but one may brake with the knee collapsing inward, a movement pattern associated in laboratory studies with higher stress on the anterior cruciate ligament, while the other brakes with good alignment. If pose estimation could identify such patterns during normal training and matches, it would offer a measure of tissue-specific load that conventional tracking cannot. Baseball is furthest along this path, because the pitching motion is repetitive, high-stakes and filmed from fixed, high-quality camera positions. Teams have long used marker-based motion capture to analyse pitching mechanics, and markerless systems that estimate joint angles from multiple cameras are now common in training facilities. The pressure is acute. In December 2024 Major League Baseball released a report, based on interviews with more than two hundred people including players, coaches, surgeons and biomechanists, which identified rising pitch velocity, the pursuit of maximal spin and movement, and consistent maximum effort as the likely primary drivers of a rise in pitcher arm injuries. The same data-driven methods that helped pitchers throw harder, by measuring and optimising spin and velocity, were thus implicated in the injuries; the report called for further research into workload, fatigue and biomechanics. In other sports, using match-video pose estimation for biomechanical load faces the accuracy limits described in Chapter 3. The joint angles that matter for ligament loading are sensitive to errors of a few degrees, and the moments of interest, sharp cuts and landings in traffic, are often partially occluded. The more realistic near-term use is screening in controlled settings: standardised tests of landing, cutting or running, filmed with a calibrated multi-camera system, with pose estimation providing joint angles that would previously have required a laboratory. The load-monitoring system as a whole It helps to think of load monitoring not as a set of metrics but as a system with several layers. Table 4 lays out the main layers, what they measure and the kind of decision each supports. Table 4. Layers of a typical elite load-monitoring system. Layer Example measures Typical decision supported External load Distances, speed bands, accelerations, pitch counts Session design, weekly planning Internal load Session RPE, heart-rate response Individual adjustment of volume Readiness Jump tests, wellness questionnaires, sleep Same-day modification of training Movement quality Pose-based joint angles, landing tests Targeted conditioning, technique work Medical context Injury history, screening results Return-to-play planning Each layer has different noise characteristics and different relationships to the outcomes coaches care about. A good system uses them in combination and keeps humans, the strength coach, the physiotherapist and the doctor, in the loop to interpret them. The measures exist to support conversations about individual athletes, not to replace them. The obvious next question is whether all this data, fed into machine-learning models, can predict who will be injured. That is the question many vendors claim to answer, and it is the subject of the next chapter. Chapter 7: Predicting Injury and Why It Is So Hard No application of artificial intelligence in sport is marketed more aggressively, or misunderstood more often, than injury prediction. The pitch is compelling. Elite clubs lose large sums to injury, in wages paid to players who cannot play and in matches lost without them. A system that could identify which players were about to be injured, days in advance, would be worth almost any price. Vendors offer risk scores, colour-coded dashboards and reports of high accuracy. Academic papers report models with impressive-sounding performance figures. The reality is more modest, and understanding why is more useful than either uncritical enthusiasm or blanket dismissal. This chapter explains the statistical structure of the injury-prediction problem, why headline accuracy figures usually mislead, what the better studies have actually found, and where machine learning genuinely helps. The argument is that injury prediction, as usually conceived, asks the wrong question; that the right question is about modifiable risk and its management; and that once the question is reframed, data and models become valuable in a more modest but more dependable way. Rare events and misleading accuracy Begin with the arithmetic. Suppose a squad of twenty-five players trains five days a week, and over a season of forty weeks the squad suffers thirty non-contact injuries that cause time loss. That is about five thousand player-days of exposure and thirty injuries: an injury occurs on roughly one player-day in a hundred and seventy. Even if the definition is broadened to injuries within the next week, the great majority of player-weeks contain no injury. In this setting a model that always predicts "no injury" is correct more than ninety-nine per cent of the time, measured day by day. Any accuracy figure must therefore be compared with that trivial baseline, and the plain percentage of correct predictions is almost meaningless. Researchers instead report measures such as the area under the receiver operating characteristic curve, often abbreviated AUC, which summarises how well a model ranks injured cases above uninjured ones across all possible thresholds; an AUC of 0.5 corresponds to random ranking and 1.0 to perfect separation. Even AUC can flatter. A model with a respectable AUC may still be nearly useless in practice, because of what happens when it is turned into a decision. Consider a model that flags players as high risk. Because injuries are rare, even a model with good sensitivity and specificity will produce many more false alarms than true warnings. If the model correctly flags seventy per cent of players who go on to be injured, and wrongly flags only ten per cent of those who do not, then, in a population where one player-week in fifty contains an injury, roughly seven out of every eight flags will be false. A coach told every week that several key players are at high risk, most of whom will be fine, will either rest players unnecessarily, at a cost in performance and in training adaptation, or learn to ignore the warnings. The relevant performance measures for a deployed system are the positive predictive value, the fraction of flags that are followed by injury, and the net benefit of acting on the flags compared with not acting, and these are rarely the numbers reported in promotional material. Roald Bahr, in a widely cited 2016 review in the British Journal of Sports Medicine, made this argument about pre-season screening tests: even a test strongly associated with injury risk at the group level will, for these statistical reasons, perform poorly at identifying which individuals will be injured. Association is not prediction. A factor can double the risk of injury and still be almost useless for picking out the individuals who will be hurt, because most people with the factor are never injured and many injuries occur in people without it. What the studies have found Machine-learning studies of injury prediction have multiplied since the late 2010s. A representative example is the 2018 study by Alessio Rossi, Luca Pappalardo, Paolo Cintia and colleagues in PLOS ONE, which used GPS training data from a professional Italian football club over a season to forecast non-contact injuries with decision-tree models. The authors reported that their model could identify a substantial fraction of injuries with a manageable number of false alarms, and they argued that machine learning outperformed approaches based on the acute:chronic workload ratio on their data. The study was careful and influential. It was also based on one club, one season and a small number of injuries, which is typical of the field and severely limits what can be concluded. Systematic reviews have repeatedly drawn the same picture. Reviews of machine learning for sports injury prediction published in the early 2020s, including one by Van Eetvelde and colleagues in the Journal of Experimental Orthopaedics in 2021, found that studies were numerous but heterogeneous: different sports, different injury definitions, different input variables, different validation methods. Many had small numbers of injuries relative to the number of candidate predictors, a recipe for overfitting. Many validated their models on data from the same athletes and seasons used for training, sometimes with random splits that allowed information from the future to leak into training. External validation, testing a model built at one club on data from another, was rare. Reported performance varied widely, and the strongest results tended to come from the weakest validation designs. None of this means the signal is absent. Load, previous injury, age and some measures of fitness and movement are associated with injury risk in well-conducted studies. The problem is that the signal is weak relative to the noise, the outcome is rare, and the data available to any one organization is small. Under those conditions, more flexible models, which is to say more sophisticated machine learning, are more prone to finding patterns that do not generalise, not less. Why injuries resist prediction Beyond the statistics, there are structural reasons why injury prediction is harder than most prediction problems machine learning has solved. Injuries have many interacting causes. A hamstring strain may involve fatigue, a previous strain, strength deficits, sprint exposure, the timing of the sprint within a session, sleep, travel, and the specific mechanics of the moment. Natália Bittencourt and colleagues argued in the British Journal of Sports Medicine in 2016 that sports injury should be understood as an emergent outcome of a complex system, in which the same risk factor can matter a great deal in one context and not at all in another, rather than as the sum of independent risk factors. Machine learning can in principle model interactions, but learning them reliably requires far more injuries than any club experiences. Much of the cause is unmeasured. A model can learn only from the variables it is given. Tracking and wearables measure what athletes do; they do not measure the state of a specific tendon or the micro-damage accumulating in a specific muscle. Most of the proximate causes of injury are inside the body and invisible to current monitoring. Prediction changes the outcome. In a well-run club, a player who shows signs of high risk is managed: his training is modified, he is rested, he receives treatment. If the intervention works, he is not injured, and the data records a high-risk player who stayed healthy. A model trained on such data learns that the warning signs are less dangerous than they are, because the outcomes it observes have already been changed by the medical staff's response. This is a form of confounding by intervention, and it means that historical data from good clubs systematically understates the risk the warning signs actually carry. The game is not a laboratory. Some injuries are partly matters of chance: a slip on wet turf, an awkward landing after a challenge, a collision. Even a perfect model of an athlete's internal state could not predict where the opponent's studs will land. From prediction to risk management The most productive reframing in the field shifts the goal from predicting individual injuries to managing modifiable risk. The questions become: which factors that raise injury risk can be changed, which interventions change them, and how can monitoring be used to apply those interventions where they are most needed? Here the evidence is considerably stronger than for prediction. Structured warm-up programmes have been tested in randomised trials. The FIFA 11+ programme, a set of running, strength, plyometric and balance exercises, was shown by Torbjørn Soligard and colleagues in a cluster-randomised trial of youth female footballers published in the BMJ in 2008 to reduce injury rates. The Nordic hamstring exercise, an eccentric strengthening movement, has been shown in randomised trials, including studies by Jesper Petersen and colleagues in 2011 and Nick van der Horst and colleagues in 2015, both in the American Journal of Sports Medicine, to reduce hamstring injuries in football. These are not machine-learning results. They are results from controlled interventions, the kind of evidence that the observational data of a single club can never supply. Data and models fit into this picture as tools for targeting and adherence. Monitoring can show which players are not completing their preventive programmes, which players have strength asymmetries that a programme should address, and which training weeks contain sudden spikes in sprint exposure that a coach might smooth out. A model that ranks players by relative risk, even if it is poor at predicting who will actually be injured, can still help prioritise limited physiotherapy time, provided its users understand what the ranking means. The value lies not in a prophecy but in a structured prompt for attention. Return to play: where data is most useful If predicting the first injury is the hardest problem in this field, managing the return from an injury is one of the most tractable, and it is where monitoring data arguably earns its keep most clearly. Previous injury is among the most consistent risk factors for future injury, and the period immediately after a return to competition carries elevated risk of recurrence. The decision about when a player is ready, and how quickly to reintroduce full demands, is therefore one with high stakes and a clear structure. Data helps in three ways. First, tracking from previous matches defines the target. A winger's typical match involves a known volume of high-speed running and a known number of sprints at close to his maximum speed. Rehabilitation can be designed to rebuild toward those figures progressively, so that by the time the player returns to team training he has already been exposed, in controlled settings, to the demands he will face. Without match tracking, those demands would be guesses. Second, objective measures track recovery of capacity. Strength tests on dynamometers, force-plate jump tests and sprint times can be compared with the player's own pre-injury values, where they exist, and with the uninjured limb. A player who reports feeling ready but whose hamstring strength remains well below his previous level, or whose top speed in training has not returned, is a candidate for further work. Third, monitoring after the return detects whether the player is coping with the reintroduced load. A sharp change in readiness measures, or a return of asymmetry, can prompt a conversation before a minor setback becomes a recurrence. Machine learning has a modest role here; the logic is largely that of careful individual comparison. But the case illustrates a broader point. The data is most useful when the question is specific, the individual's own history provides a baseline, and the decision is clear: is this player ready for the next step? Those conditions are absent in the general problem of predicting which healthy player will next be injured. League-scale data: the NFL's approach The most ambitious attempt to overcome the small-data problem is to pool data across an entire league. The NFL, working with Amazon Web Services, built a platform called the Digital Athlete that combines data from all thirty-two teams, including sensors in shoulder pads, video and optical tracking from practices and games. Reporting in 2025 described it as generating some five hundred million data points a week and as drawing on information from more than fifteen hundred players to establish league-wide reference points for workload measures such as accelerations, decelerations and changes of direction. League-scale data changes what is possible. With thousands of athletes and many more injuries, some patterns that are invisible at club level become detectable, and models can be validated on teams they were not trained on. Equally important, a league can act on aggregate findings in ways a club cannot. The NFL has used its injury data and simulations to inform rule changes, including the redesigned kickoff introduced in 2024 and a ban on the hip-drop tackle, and equipment decisions such as helmet design and protective padding. The league has reported falling injury and concussion numbers in recent seasons, while acknowledging that rule, equipment and medical changes have all contributed and that attributing the improvement to any one tool is difficult. This points to a general conclusion. The most reliable gains from injury analytics so far have come not from predicting which individual will be injured on Tuesday, but from identifying the situations, rules, equipment and training patterns that raise risk across a population, and changing them. Those are decisions about systems rather than individuals, and they suit the statistical strengths of the data. How to evaluate an injury-risk model For a club considering an injury-risk product, or a practitioner reading a study, a handful of questions separate credible claims from inflated ones. First, how many injuries were in the data, and how many predictors? A model fitted with fifty variables to thirty injuries is almost certainly overfitted. Second, how was the model validated? Validation on later seasons, or on different teams, is far more convincing than random cross-validation within one dataset. Third, which performance measure is reported? An AUC or accuracy figure without the positive predictive value at a usable threshold says little about practical value. Fourth, what injuries are being predicted? Lumping all injuries together, including contact injuries that no load metric could foresee, dilutes the signal; specific injuries with plausible mechanisms, such as hamstring strains in sprint-heavy sports, are more tractable. Fifth, what would be done differently in response to a warning, and is there evidence that doing it reduces injury? A prediction with no effective intervention attached is of little value. Sixth, how will the model's errors be noticed? A system deployed without ongoing monitoring of its false alarms and missed cases will drift out of usefulness unnoticed. Asked honestly, these questions usually lead to a more modest product than the one advertised: a monitoring and decision-support system that helps staff notice changes, prioritise attention and document their reasoning, rather than an oracle. That is not a disappointment. It is what the data can actually support, and a well-designed system of that kind is worth having. Chapter 8: Whose Data, Whose Decisions The previous chapters treated data as a technical resource: something to be measured accurately, modelled carefully and validated honestly. But the data in sports analytics is unusual. It is generated by people's bodies, often continuously, sometimes from devices strapped to their skin, and it can reveal things about their health, fatigue and decline that they may not wish to share with an employer, an opponent, a future employer or a betting market. It is also commercially valuable, sold to broadcasters, gambling companies and fantasy-sports platforms. And it is used inside organizations where power is unevenly distributed between those who generate the data and those who decide what to do with it. This chapter addresses two connected questions. The first is about rights: who owns the data, who may use it and for what, and what protections athletes have. The second is about organizations: how a club or federation can build an analytics function that actually improves decisions rather than producing an expensive stream of unread reports. The two are linked, because an analytics function that athletes do not trust will receive worse data, and one that coaches do not trust will change nothing. Data from bodies is different Most performance data falls into a grey area between public and private. The fact that a footballer ran eleven kilometres in a televised match, or that a pitcher's fastball averaged ninety-six miles per hour, is observable by anyone in the stadium and has long been reported. A centre-back's sprint speed in the eighty-fifth minute, extracted from tracking data, is a more precise version of the same kind of fact. But a player's heart-rate variability on waking, his sleep duration, his hormone levels, the force asymmetry between his legs in a morning jump test, and his place on an injury-risk ranking are different in kind. They are health information, collected by an employer, and they can reveal a condition or a decline before the player himself has chosen to acknowledge it. European data-protection law reflects this distinction. Under the General Data Protection Regulation, personal data includes any information relating to an identifiable person, and data concerning health is a special category whose processing is permitted only under specific conditions. Much of the data generated in load monitoring and medical screening plausibly falls into that category. Clubs operating in Europe must therefore have a lawful basis for collecting it, limit its use to defined purposes, secure it appropriately and respect players' rights to access it. Consent, the most intuitive basis, is problematic in employment, because consent given to an employer who decides your selection and contract is not freely given in any meaningful sense, and clubs often rely instead on other grounds, such as the necessity of processing for occupational health purposes, with the constraints those carry. In North American leagues, where privacy law is more fragmented, the main protections come from collective bargaining. The NBA's 2017 agreement with the players' union, as noted earlier in the book, made wearable use voluntary, barred the use of wearable data in contract negotiations, and created a joint committee to review and approve devices. Other leagues have negotiated their own provisions. These agreements recognise that the most sensitive risk is not public exposure but use against the player inside the employment relationship: a club that knows from monitoring data that a player's physical capacities are declining has an informational advantage in negotiating his next contract. Who profits from performance data A separate set of disputes concerns commercial use. Event and tracking data are sold to broadcasters, media companies, fantasy platforms and, above all, betting operators, for whom fast, accurate in-game data is essential to offering live markets. Leagues license official data feeds to specialist companies, which sell them on. The players whose actions generate the data have traditionally received nothing directly from these sales, beyond what flows to them indirectly through league revenues. That arrangement has been challenged. In the United Kingdom, a group led by the former Cardiff City manager Russell Slade and the data technologist Jason Dunlop, under the name Project Red Card, announced in 2021 that it was acting for some 850 current and former footballers in a claim against betting, gaming and data-collection companies, arguing that the use of players' personal performance data without consent or compensation breached their rights under the GDPR. Letters before action were sent to more than a dozen companies. The legal questions are genuinely difficult: how far data about an athlete's public performance is personal data, whether the commercial interests of leagues and data companies provide a lawful basis for processing, and how any value would be shared. Whatever the outcome of particular claims, the episode signals that the commercial value of performance data has become large enough for athletes to contest its distribution. Integrity adds another dimension. Detailed, fast data is valuable to people betting on matches, and anyone with access to non-public information, such as an injury-risk flag, a fitness test result or the knowledge that a player will be rested, has information that could be misused in betting markets. Clubs and leagues therefore have reasons beyond privacy to control who can see internal monitoring data and to treat it with the same discipline as other price-sensitive information. Fairness and the reach of the camera The spread of broadcast-derived tracking raises a quieter set of questions. When tracking required installed cameras, it covered only the wealthiest competitions, and players elsewhere were largely invisible to data-driven recruitment. Broadcast tracking extends coverage to many more leagues, which opens opportunities for players in smaller markets to be noticed. It also means that players in those leagues are now measured, evaluated and ranked by organizations with whom they have no relationship and to which they have given no consent, using systems whose accuracy in their league, as Chapter 3 noted, may be lower than in the leagues for which the systems were designed. Youth sport raises the sharpest version of these concerns. Academies monitor children's growth, maturation, physical output and, increasingly, movement patterns from video. Such monitoring can protect young athletes, for example by identifying growth spurts during which training loads should be reduced, and it can disadvantage them, for example by encouraging early selection decisions on the basis of physical measures that mostly reflect when a child happens to have matured. Relative age effects, the over-representation in youth squads of children born early in the selection year, are a long-documented example of physical maturity being mistaken for talent, and more precise physical measurement can entrench rather than correct that bias if it is used naively. Data on minors deserves the strictest governance and the most cautious interpretation. Women's sport presents a further issue. Much of the published evidence on load, injury and biomechanics comes from male athletes, and models built on male data may transfer poorly. Female footballers suffer anterior cruciate ligament injuries at substantially higher rates than male players, and the causes are still debated, with anatomical, hormonal, biomechanical and environmental explanations, including equipment and facilities designed around men, all under investigation. Analytics programmes that simply apply male thresholds and models to women's teams risk being both inaccurate and complacent. Building an analytics function that changes decisions The second half of this chapter turns from rights to practice. Many clubs now employ data scientists, buy multiple data feeds and maintain dashboards. Far fewer can point to specific decisions that were made better because of them. The gap is mostly organizational rather than technical, and the clubs that have closed it share several characteristics. They start from decisions, not data. The useful question is not "what can we do with tracking data?" but "which recurring decisions do we make, how are they made now, and where would better information change them?" Recruitment shortlists, weekly training plans, return-to-play timelines, opposition preparation and set-piece design are recurring decisions with identifiable owners. An analytics project attached to one of them, with the decision-maker involved from the start, has a much better chance of mattering than one that produces insights in search of an audience. They put analysts close to decision-makers. Clubs that have integrated analytics successfully tend to place analysts within the coaching, medical and recruitment departments, attending meetings and training, learning the language of the people they support. Physical proximity matters because much of the value lies in informal exchanges: a coach asking whether a particular pattern is real, an analyst noticing that a dashboard is being misread. They report uncertainty honestly. Chapter 3 described the errors in measurement and Chapter 7 the limits of prediction. An analytics function that hides these behind confident dashboards will eventually be caught out, by a high-profile false alarm or a missed injury, and lose credibility it may not regain. One that presents its outputs as probabilities and ranges, explains what a model can and cannot see, and acknowledges when the honest answer is "we don't know", builds the kind of trust that lets its advice be heard when it is confident. They measure their own impact. Few clubs systematically check whether their analytics changed outcomes. A recruitment model's recommendations can be tracked against the subsequent performance of players signed and not signed. A load-management protocol can be compared against injury rates before and after, with appropriate caution about everything else that changed. A set-piece programme can be evaluated over a season. Without such evaluation, the function cannot learn which of its products are valuable. They involve athletes. Players who understand what is measured, why, and how it benefits them provide better data: they wear devices properly, answer wellness questionnaires honestly, and report early symptoms rather than hiding them. Players who suspect that data is being used to justify dropping them, or will surface in contract talks, game the questionnaires and resent the devices. Transparency about purpose, access for players to their own data, and visible use of data to help rather than judge are therefore not only ethical commitments but practical requirements for good monitoring. Buying technology without buying a story A practical issue sits between governance and organizational design: procurement. Clubs buy tracking systems, wearables, data feeds and analytics platforms from vendors whose sales material understandably emphasises capability over limitation. The questions raised throughout this book translate into a buyer's checklist that is worth stating plainly. For any measurement system, ask for independent validation against a reference standard, under conditions resembling the club's own use, and for the specific metrics the club intends to rely on, not just headline positional accuracy. Ask how the system filters and derives speeds and accelerations, and whether those settings can be fixed so that metrics remain comparable over time. For broadcast-derived data, ask how much is observed and how much is interpolated, and how much human quality control is applied. For any model that produces scores or predictions, ask what it was trained on, how it was validated, and on whose data. Ask for performance figures at decision-relevant thresholds, not only summary measures. Ask what happens to the club's own data: whether it will be pooled with other clients' data, used to train the vendor's models, or sold on, and under what terms. For any platform that will hold athletes' health information, ask where the data is stored, who at the vendor can access it, how it is secured, and what happens to it if the contract ends. Data-protection obligations do not transfer to the vendor merely because the vendor holds the data. These questions are not adversarial. Good vendors welcome them, because they distinguish serious products from superficial ones. A vendor unable to answer them is telling the buyer something important. The limits of the dashboard A recurring failure in sports analytics is the dashboard that becomes an end in itself. Monitoring platforms make it easy to collect and display hundreds of variables with automatic thresholds and colour coding. The appearance of comprehensive oversight can substitute for the harder work of understanding what the variables mean for a particular athlete. A physiotherapist who sees a red flag beside a player's name is pushed toward action regardless of whether the flag reflects a genuine change or a measurement artefact; a coach who sees green may relax when a conversation with the player would have revealed a problem the sensors could not detect. The alternative is to design monitoring around a small number of measures with known reliability and clear links to decisions, reviewed by people who understand them, supplemented by direct conversation. That is less impressive in a presentation to the board than a wall of screens, but more likely to keep players healthy and teams winning. The general principle is the one that has run through this book: the measurement problem has largely been solved, and what remains is a problem of judgment, of deciding what the measurements mean and what to do about them. Organizations that recognise this invest in people who can exercise that judgment and in the processes that support it, not only in more data. Conclusion The arrival of artificial intelligence in elite sport is often told as a story of replacement: the scout replaced by the algorithm, the referee by the camera, the physio's hunch by the risk score. The evidence of the preceding chapters supports a different story. Where machines have genuinely replaced human judgment, it is in problems with clear ground truth and abundant data: locating a ball, a body, a shoulder at a precise instant. Where problems lack ground truth, as with the best tactic, or are dominated by rare and tangled outcomes, as with injury, the machine has become a powerful instrument but not a replacement, and the quality of its contribution depends on the people who use it. This gradient, from measurement to tactics to injury, is the main thing to carry away. It suggests a practical way of judging any claim made for artificial intelligence in sport. Ask first what the ground truth is: is there a correct answer, and can the system's output be checked against it? Ask second how much relevant data exists for the specific decision at hand, not for the sport in general. Ask third whether the environment responds to the decision, as opponents respond to tactics and medical staff respond to risk flags. The more clearly defined the answer, the larger the dataset and the more passive the environment, the more a model can be trusted to act on its own. The reverse conditions call for a model that informs a human who decides. What has changed for good Some changes are durable. Tracking, positional and increasingly skeletal, is becoming universal at the top of professional sport and is spreading downward through broadcast-derived systems. Within a few years it will be normal for a club in a modest league to hold tracking data on thousands of matches across dozens of competitions. Officiating will continue to move from human perception to machine measurement checked by humans, as offside decisions already have in elite football and line calls have in professional tennis. Valuation models that credit off-ball movement and positional discipline have given analysts a vocabulary for parts of the game that were previously discussed only in the impressionistic language of coaching, and that vocabulary will be refined rather than abandoned. The training ground has changed as well. Load monitoring, for all its overselling, has made the management of athletes' bodies more deliberate. A strength coach in 2026 knows far more about what each player did last week than a strength coach in 2006, and the broader lessons of the load-monitoring era, that sudden spikes are riskier than consistently high work, that recovery is as much a part of training as effort, that individual responses differ, have become common sense in elite sport. What remains unsettled Other questions are open. Whether generative tactical models such as TacticAI will move from plausible suggestions to demonstrable improvements in results is unknown, and will be hard to establish because of the adaptive, low-scoring, small-sample nature of the problem. Whether pooled, league-scale injury data will allow genuinely predictive individual models, rather than better system-level decisions, remains to be seen; the NFL's experience suggests the main gains may continue to lie in rules, equipment and training patterns rather than in forecasting individual injuries. Whether broadcast pose estimation will become accurate enough for biomechanical assessment during matches depends on advances in monocular three-dimensional reconstruction that are under way but not complete. The governance questions are equally unsettled. The ownership and commercial use of performance data, the protection of health information collected by employers, the treatment of data on young athletes, and the fairness of evaluating players with systems that perform unevenly across leagues and populations are being addressed piecemeal, through collective bargaining, data-protection law and occasional litigation. A more coherent settlement will probably be needed as the data grows more detailed and more valuable. What practitioners should do differently For those working in sport, the argument of this book points to several concrete practices. Treat every metric as a measurement with an error, and know where the error comes from. Before acting on a change in a player's numbers, ask whether anything changed in the measurement chain. Prefer aggregated, stable measures for decisions about individuals, and reserve fine-grained, high-frequency measures for questions where their noise can be tolerated or averaged away. Judge predictive models by the numbers that matter for decisions, the fraction of warnings that prove correct, the cost of false alarms, the evidence that acting on a warning helps, rather than by headline accuracy. Use tactical models to widen the set of ideas coaches consider, not to narrow it to whatever the model scores highest; and test promising ideas in training before trusting them in matches. Attach every analytics product to a recurring decision with a named owner, and check afterwards whether it changed that decision for the better. Earn the trust of athletes by being transparent about what is measured and why, by giving them access to their own data, and by ensuring that data collected to protect them is not used against them. The judgment that remains Sport has always been a contest of judgment under uncertainty: the scout deciding whether a teenager will grow into his talent, the coach choosing whether to press high against a quicker opponent, the physio deciding whether a player with a tight hamstring should train today. Artificial intelligence has not removed that uncertainty. In some places it has reduced it, sharply; in others, it has made the uncertainty more visible by measuring everything around it with precision. That visibility is itself a gain. A coach who knows how much of a decision rests on evidence and how much on judgment is in a better position than one who mistakes a confident dashboard for knowledge. The machines now see the game more completely than any human observer can. Deciding what to do about what they see remains, for the foreseeable future, a human responsibility, and the organizations that thrive will be those that equip people to exercise it well. Further Reading Books Michael Lewis, Moneyball: The Art of Winning an Unfair Game (2003).** The account of the Oakland Athletics' use of statistical analysis that brought sports analytics to a general audience. It predates tracking data but remains the best description of the organizational resistance that new evidence meets. Chris Anderson and David Sally, The Numbers Game: Why Everything You Know About Football Is Wrong (2013).** An accessible early statement of what event data could reveal about football, including a discussion of Charles Reep and the direct-play debate. Christoph Biermann, Football Hackers: The Science and Art of a Data Revolution (2019).** A journalist's survey of how clubs and analysts in Europe adopted data, with good coverage of space-control and off-ball models. Kirk Goldsberry, SprawlBall: A Visual Tour of the New Era of the NBA (2019).** How tracking and shot data reshaped basketball strategy, by one of the researchers behind the expected possession value work. Ben Lindbergh and Travis Sawchik, The MVP Machine: How Baseball's New Nonconformists Are Using Data to Build Better Players (2019).** The shift in baseball from using data to evaluate players to using it, with high-speed video and biomechanics, to change how they move. Rory Smith, Expected Goals: The Story of How Data Conquered Football and Changed the Game Forever (2022).** A narrative history of football analytics and the people who built it. Papers Zhe Cao, Tomas Simon, Shih-En Wei and Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields", CVPR (2017). The OpenPose paper, a landmark in bottom-up pose estimation. Silvio Giancola and colleagues, "SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos", CVPR Workshops (2018). The first of the SoccerNet datasets, which have since grown into public benchmarks for tracking, calibration and re-identification in football video. William Spearman, "Beyond Expected Goals", MIT Sloan Sports Analytics Conference (2018). The physics-based pitch-control model and its use to value off-ball positioning. Tom Decroos, Lotte Bransen, Jan Van Haaren and Jesse Davis, "Actions Speak Louder than Goals: Valuing Player Actions in Soccer", KDD (2019). The VAEP framework and the SPADL action language. Daniel Cervone, Alexander D'Amour, Luke Bornn and Kirk Goldsberry, "A Multiresolution Stochastic Process Model for Predicting Basketball Possession Outcomes", Journal of the American Statistical Association (2016).** The foundational treatment of expected possession value. Zhe Wang, Petar Veličković and colleagues, "TacticAI: An AI Assistant for Football Tactics", Nature Communications (2024).** The graph-neural-network approach to corner kicks developed with Liverpool FC. Tim Gabbett, "The Training–Injury Prevention Paradox: Should Athletes Be Training Smarter and Harder?", British Journal of Sports Medicine (2016).** The influential case for the acute:chronic workload ratio and for high chronic loads as protective. Franco Impellizzeri and colleagues, "Acute:Chronic Workload Ratio: Conceptual Issues and Fundamental Pitfalls", International Journal of Sports Physiology and Performance (2020).** The most thorough critique of the ratio. Roald Bahr, "Why Screening Tests to Predict Injury Do Not Work—and Probably Never Will...: A Critical Review", British Journal of Sports Medicine (2016).** A clear explanation of why association with injury does not imply useful individual prediction. Hashtags: #AIInSports #SportsAnalytics #ArtificialIntelligence #SportsTechnology #MachineLearning #SportsDataScience #PlayerTracking #AthleteTracking #ComputerVision #HumanPoseEstimation #MotionTracking #PredictiveAnalytics #MatchOutcomePrediction #SportsPerformanceAnalysis #TacticalAnalysis #TacticalOptimization #SportsBiomechanics #InjuryPrediction #InjuryPrevention #AthleteMonitoring #TrainingLoadManagement #ExpectedGoals #ExpectedThreat #PitchControl #ExpectedPossessionValue #GraphNeuralNetworks #SportsStrategy #PerformanceOptimization #SportsDataVisualization #DataDrivenCoaching

Download the Book (PDF): Introduction In a kitchen somewhere, a woman who has been blind since birth holds her phone over a can she has just taken from the back of a cupboard. A few seconds later a synthetic voice tells her that it is chopped tomatoes, that the best-before date on the lid has passed by four months, and that the can appears to have a small dent near the rim. Ten years ago she would have shaken the can, guessed from its weight, and either opened it or waited for a sighted friend to visit. Twenty years ago she might have kept a drawer of rubber bands and Braille labels and hoped nobody had moved anything. In a hospital room in California, a man whose amyotrophic lateral sclerosis has taken away his ability to speak intelligibly watches a screen as words he is attempting to say appear on it, decoded from the electrical activity of a few hundred neurons in his motor cortex. A synthetic voice, trained on recordings of his own voice from before the illness, reads them aloud. The first time it worked, he wept, and so did his family. In a classroom, a Deaf teenager reads live captions of her teacher's lecture on a tablet. They are mostly right. When the teacher says a chemistry term the system has never encountered, the caption turns into nonsense for a sentence, and she looks across the room at the interpreter, who is still the one person in the building she can rely on completely. These three scenes contain most of what this book is about. Each describes something that would have sounded like science fiction within living memory and is now ordinary, or close to ordinary, for some people. Each involves a form of artificial intelligence: a vision-language model that can describe an arbitrary photograph, a neural decoder that maps brain activity onto intended words, a speech recognition system that turns sound into text in real time. And each contains a quieter detail that matters as much as the headline: the woman had to decide whether to trust the date the machine read; the man spent weeks training the decoder and depends on a research team to keep it running; the teenager knows exactly when the captions fail and has a human backup for that moment. The argument of this book The controlling idea of this book is simple to state and harder to live by. Artificial intelligence is genuinely transforming what disabled people can do, but its value is measured in agency rather than accuracy. A tool that is right ninety-five percent of the time and lets the user know when it is unsure, correct it, and choose when to use it can be life-changing. A tool that is right ninety-nine percent of the time but conceals its errors, takes decisions out of the user's hands, or exists only as long as a company finds it profitable can be worse than nothing. The technical frontier, which is advancing faster than at any point in the history of assistive technology, is no longer the main constraint. The constraints are trust, reliability in messy real conditions, the data on which systems are trained, who pays, who maintains the device after the trial ends, and who gets to define the problem in the first place. That argument is not an attempt to temper excitement. Some of what follows is among the most remarkable work in modern science. In 2021 a man paralysed from the neck down wrote sentences at ninety characters a minute by imagining handwriting. In 2025 a research team reported a neuroprosthesis that turned a man's attempted speech into synthesised voice with a delay of about a fortieth of a second, letting him interject, sing simple melodies, and change intonation to ask a question. General-purpose vision models now describe photographs with a fluency that would have been unimaginable to the engineers who built the first reading machines. None of this should be minimised. But the history of assistive technology is also a history of devices that dazzled at demonstration and were abandoned in daily life, of products designed for disabled people by those who did not ask them what they needed, and of companies that went bankrupt and left the people relying on their implants without support. Artificial intelligence does not escape that history. In some ways it intensifies it, because the systems are more powerful, more opaque, and more dependent on the continued goodwill of the organisations that run them. What the book covers The book proceeds from the oldest and most widely used applications to the newest and most invasive, then turns to the questions that cut across all of them. The first chapter places AI within the long lineage of assistive technology, from the reading machines of the 1970s to the screen readers built into every smartphone, and sets out the framework the rest of the book relies on: the distinction between treating disability as a defect to be fixed and treating it as a mismatch between a person and an environment that can be redesigned. Chapters 2 and 3 deal with blindness and low vision: first, machines that describe images, text and scenes; then the harder problem of moving safely through the physical world. Chapters 4 and 5 turn to deafness, first to automatic captioning, which has quietly become one of the most widely used AI accessibility tools in the world, and then to sign language translation, which is far harder than most engineers expect and has been the site of some of the field's most instructive failures. Chapter 6 addresses speech: recognition systems that can understand atypical voices, voice banking and cloning for people losing their speech, and augmentative communication accelerated by language models. Chapters 7 and 8 are devoted to brain-computer interfaces. The first explains how they work and traces the extraordinary run of results in decoding movement, handwriting and speech. The second follows them out of the laboratory into the world of companies, regulators, surgeons and insurers, where questions of durability, data and long-term support decide whether the technology helps anyone outside a trial. Chapter 9 considers the disabilities that receive less attention from technologists, including motor impairments that affect device use, and cognitive, learning and neurological differences. Chapter 10 confronts the problems no single product can solve: biased training data, the fraud of automated accessibility fixes, the law, and the principle captured in the disability movement's slogan, "Nothing about us without us." The conclusion draws out what all of this implies for the people who build, buy, regulate and use these systems. A note on language and scope Disabled people do not agree on vocabulary, and this book does not pretend otherwise. Many people in the United Kingdom and in disability studies prefer "disabled people," on the grounds that people are disabled by their environments; many people in the United States prefer "people with disabilities." Many Deaf people who use sign language identify as members of a linguistic and cultural minority, marked with a capital D, rather than as people with a hearing impairment. Some autistic people prefer identity-first language and some do not. The book mostly uses "disabled people," uses "Deaf" with a capital where cultural identity is meant, and tries elsewhere to follow how people describe themselves. The scope is deliberately selective. Artificial intelligence touches nearly every area of disability, from prosthetic limbs controlled by muscle signals to algorithms that schedule home care. The focus here is on systems that restore or substitute for perception, communication and control, because these are where the technology has advanced fastest and where the stakes for individual autonomy are clearest. The book does not cover medical diagnosis or genetic screening, which raise their own debates and deserve their own treatment. According to the World Health Organization, around 1.3 billion people, roughly one in six of the world's population, experience significant disability, and the proportion is rising as populations age. Most of them live in low- and middle-income countries, and most will never have access to a brain implant or a pair of AI glasses. Any honest account of AI for accessibility must keep that fact in view. The most consequential technologies are often not the most spectacular, but the ones that run on a cheap phone, work without an internet connection, and remain under the control of the person using them. Chapter 1: The Long Lineage of Assistive Technology Artificial intelligence did not invent assistive technology. It arrived in a field with a long history, a vocal community of users, a body of hard-won lessons, and a set of recurring mistakes. Understanding that inheritance is the best protection against repeating it, and it explains why disabled people often greet announcements of new AI breakthroughs with a mixture of interest and weary scepticism. Reading machines and talking computers The modern story begins with reading. In 1976 the inventor Ray Kurzweil unveiled the Kurzweil Reading Machine, a device about the size of a washing machine that could scan a printed page, recognise the characters on it regardless of typeface, and read the text aloud in synthetic speech. It combined three technologies that were each at the edge of feasibility: a flatbed scanner, omni-font optical character recognition, and text-to-speech synthesis. Early units cost tens of thousands of dollars and were installed mainly in libraries and institutions for the blind. The musician Stevie Wonder bought one of the first, and the story of his excitement became part of the machine's publicity. The Kurzweil machine is worth remembering not only as a first but because its components are, in refined form, still at the heart of how blind people encounter print. Optical character recognition, which in the 1970s relied on hand-crafted rules for recognising letter shapes, became a statistical and then a neural-network technology. Speech synthesis passed through formant synthesis, the robotic-sounding voices that many blind computer users still prefer for their speed and clarity, then concatenative synthesis built from fragments of recorded human speech, and finally the neural voices of the late 2010s that are difficult to distinguish from a person. Each step was a form of machine learning, even if nobody called it artificial intelligence at the time. The more consequential development for daily life was the screen reader: software that reads aloud whatever is on a computer screen and lets the user navigate it with the keyboard. Early screen readers worked on text-based systems such as DOS, where the screen contents were simply characters in memory. The arrival of graphical interfaces in the late 1980s and 1990s threatened to shut blind users out entirely, because a window full of icons and buttons is not a stream of text. The solution was the accessibility application programming interface, a layer through which applications describe their own structure, saying in effect "this is a button, its label is Save, it is currently disabled." Screen readers such as JAWS, first released for Windows in 1995, and later the free NVDA, built their usefulness on these interfaces. The decisive moment for mobile devices came in 2009, when Apple shipped VoiceOver on the iPhone 3GS. Before then, touchscreens were widely assumed to be unusable by blind people, since there were no physical buttons to find. VoiceOver showed otherwise: the user drags a finger across the screen, hears what is under it, and double-taps to activate. Google followed with TalkBack on Android. Within a few years, the smartphone had become the single most important assistive device for many blind people, because it combined a screen reader, a camera, GPS, and a network connection in a mainstream product that cost a fraction of the specialised devices it replaced. That last point is central to everything that follows. Most of the AI accessibility tools discussed in this book run on smartphones or on platforms built by large technology companies. Their reach comes from riding on mainstream hardware; their vulnerability comes from depending on companies whose priorities are not set by disabled users. Captions, telephones, and the shape of access for Deaf people A parallel history unfolded for Deaf and hard-of-hearing people. The teletypewriter, adapted for telephone use in the 1960s by the Deaf physicist Robert Weitbrecht, let Deaf people type conversations over phone lines for the first time. Closed captioning for television began in the United States in the early 1980s and became mandatory for most programming over the following decades. Relay services, in which an operator reads typed text aloud to a hearing caller and types back the reply, were required nationally by the Americans with Disabilities Act of 1990. Video relay services, in which a sign language interpreter mediates a call by video, followed in the 2000s. Each of these systems carried a lesson. Captioning showed that a feature built for a small group could become ubiquitous once it was cheap enough: captions are now used by vast numbers of hearing people watching videos on mute. Relay services showed that the quality of access depends heavily on the human in the loop, on interpreters' skill and on how much they can be trusted with private information. And the long fight over each of these measures showed that access was won through law and activism as much as through invention. Two ways of thinking about disability Behind these histories lies a conceptual shift that every serious discussion of accessibility now takes as its starting point. For much of the twentieth century, disability was understood primarily through what is often called the medical model: a disability is a deficit located in an individual's body or mind, and the goal of intervention is to cure, correct or compensate for it. On this view the problem with being blind is that one cannot see, and the ideal technology would restore sight. From the 1970s onward, disabled activists and scholars, particularly in the United Kingdom, articulated what became known as the social model. On this view, impairment is a fact about a body, but disability is created by the interaction between that body and a world designed without it in mind. A wheelchair user is disabled by steps, not by her legs; a Deaf person is disabled by the absence of interpreters and captions, not by his ears. The goal is not primarily to fix people but to remove barriers. Neither model captures everything, and few people now hold either in its pure form. Chronic pain, fatigue and progressive illnesses involve suffering that no ramp removes, and many disabled people want medical treatment as well as social change. But the social model has had an enormous practical influence. It underlies accessibility law, which requires buildings, websites and services to accommodate people rather than requiring people to adapt. And it supplies the most useful question to ask of any AI accessibility tool: does it put the burden of adaptation on the disabled person, or does it change the environment so that the burden disappears? The difference is not academic. A camera app that reads a restaurant menu aloud is helpful, but it leaves the restaurant's inaccessible menu untouched and puts the work on the blind diner. A restaurant that publishes its menu in an accessible format removes the barrier for everyone. A brain implant that lets a paralysed person type is extraordinary; a world in which software is designed to be controlled by any input method makes the implant far more useful. AI tools are frequently framed as individual fixes, which can quietly let institutions off the hook. Holding both perspectives at once, valuing the tool while asking what barrier it is compensating for, is one of the habits this book tries to model. Nothing about us without us The disability rights movement gave the world a slogan that has become a design principle: "Nothing about us without us." It was popularised in the 1990s, most notably as the title of James Charlton's 1998 book on disability oppression and empowerment, and expresses the demand that disabled people be full participants in decisions that affect them, rather than objects of charity, research or policy designed by others. In technology, the principle has a sharp edge, because the history of assistive devices is littered with products designed by people who imagined what disabled people needed rather than asking them. The classic example, which will recur in Chapter 5, is the sign language glove: a device covered in sensors that purports to translate signing into speech. Such gloves have been invented and celebrated in the press many times. Deaf people have consistently explained that they do not want them, that they misunderstand what sign language is, and that they place the burden of communication on the Deaf person rather than on the hearing world. The gloves keep being invented anyway, because the inventors are often hearing engineering students who find the problem intuitively appealing and never speak to a Deaf person before building one. A related phenomenon is what disability writers sometimes call "inspiration porn": the presentation of disabled people as heart-warming stories whose purpose is to make non-disabled audiences feel good. Technology demonstrations are especially prone to this. A video of a paralysed person moving a cursor with their mind, or a blind person "seeing" through an app, invites a particular kind of emotional response that can obscure basic questions. How well does it work outside the demonstration? Who chose this problem to solve? What happens when the funding ends? Abandonment and the gap between demonstration and life The most sobering finding in the assistive technology literature concerns abandonment. Studies stretching back to the early 1990s have found that a substantial fraction of assistive devices are abandoned by their users. A widely cited 1993 survey by Betsy Phillips and Hongxin Zhao found that nearly a third of all devices obtained by the people they studied had been abandoned, many within the first year. Later studies have produced varying figures depending on the device and population, but the pattern is robust: a device that seems useful on paper often fails in daily life. The reasons are instructive. Devices are abandoned when they do not match what the user actually needs, when they are uncomfortable, unreliable or difficult to maintain, when the user's condition changes, when they mark the user as different in social settings, and, crucially, when the user was not involved in choosing them. The last factor appears again and again: devices selected with the user's genuine participation are kept, and devices selected for them are abandoned. These findings predate artificial intelligence, but they apply to it with full force. An AI system adds new failure modes to the old ones. It can be wrong in ways that are hard to detect. Its behaviour can change after a software update. It may require a network connection that is not always available, or a subscription that is not always affordable. It may collect data about the user that the user would prefer to keep private. And because the most capable AI systems are built and operated by a small number of large companies, a user's access can disappear because of a business decision made in another country. The economics of specialist devices There is one more piece of inheritance that shapes how AI tools are received: price. For most of its history, assistive technology has been a small, specialised market. A refreshable Braille display, which raises and lowers tiny pins to render a line of text in Braille, has typically cost well over a thousand dollars and often several thousand, because the mechanism is intricate and the number of buyers small. Dedicated communication devices for people who cannot speak, sold as medical equipment, have long cost thousands of dollars more than a mainstream tablet running similar software. Specialist screen readers for desktop computers were, for many years, priced at levels that only employers, schools or government programmes could realistically pay. Disabled people sometimes describe this as a disability tax. Because the market is small, prices are high; because prices are high, most purchases are funded by third parties such as insurers, health services, schools and vocational rehabilitation agencies; and because third parties pay, the product is often chosen to satisfy the funder's rules rather than the user's preferences. A person who needs a communication device may wait months for an assessment and approval, receive a device chosen from an approved list, and be unable to replace it when it breaks or when their needs change. That arrangement is part of why abandonment rates are high. The arrival of the smartphone disrupted this market. When a tablet costing a few hundred dollars, running an app costing perhaps a few hundred more, could do much of what a dedicated communication device did, families and clinicians had a choice. When screen readers came free with every phone and every Mac, and a capable free screen reader appeared for Windows, the price of entry to digital life for blind people fell sharply. Mainstream technology did not merely add features; it changed who controlled the purchase. A disabled person who buys a phone off the shelf owns it outright, can choose what to install on it, and does not need anyone's permission to try something new. AI tools mostly sit on the mainstream side of this divide. Many are free, or bundled with products people already own. That is a large part of their appeal, and a large part of why their adoption has been so fast. But "free" in the technology industry usually means that the costs are paid in other ways: in data, in dependence on a platform, or in the risk that a feature subsidised today will be withdrawn tomorrow. The old specialist market was expensive and slow, but its products were at least sold to and supported for disabled customers. The new mainstream market is cheap and fast, but disabled people are a small share of its customers, and their needs compete with everyone else's. The curb-cut effect and its limits Accessibility advocates often invoke the curb-cut effect. Curb cuts, the small ramps at the edges of pavements, were won by wheelchair users through activism in the 1970s, most famously in Berkeley, California. Once installed, they turned out to benefit parents with pushchairs, travellers with luggage, delivery workers and cyclists. The general lesson is that designing for disabled people often produces better design for everyone. Many AI accessibility features illustrate the effect. Captions designed for Deaf viewers are used by hearing people in noisy places. Voice assistants that began partly as accessibility tools are now everyday conveniences. Text-to-speech, once a specialised technology for blind readers, is used by commuters listening to articles. Image description, now built into major platforms, helps search engines as well as screen reader users. But the curb-cut story can also be misused. When accessibility is justified mainly by its benefits to non-disabled people, the needs that are specific to disabled people, which have no mass-market spillover, risk being neglected. Sign language translation, communication systems for people with severe motor impairments, and tools for people with intellectual disabilities do not obviously benefit the mainstream. They need to be justified on their own terms, as matters of equal participation, not as by-products of features that happen to sell. What artificial intelligence changes Against this background, what is new about the current wave of AI? Three things stand out. The first is generality. Earlier assistive technologies solved narrow problems: recognising printed text, detecting a colour, reading a barcode. Modern AI, particularly large models trained on vast and varied data, can address open-ended tasks. A vision-language model can describe a photograph of anything, answer follow-up questions about it, and read handwriting, signs and screens. A language model can turn a few keywords into a fluent sentence. This generality is what makes the current moment feel different, and it is genuinely different. The second is the ability to learn from individuals. A speech recognition system can be adapted to a person whose speech is affected by cerebral palsy; a brain-computer interface decoder can be trained on the specific neural patterns of a single participant. Personalisation was always possible in principle, but machine learning makes it practical. The third is that the most capable systems are now built for general markets by a handful of very large companies. This brings resources and reach that specialised assistive technology firms could never match. It also means that disabled people are once again relying on products whose design, pricing and continued existence they do not control. Each of these features cuts both ways. Generality brings flexibility but also unpredictable errors. Personalisation brings performance but requires personal data. Mainstream development brings scale but also dependence. The chapters that follow examine how those tensions play out across specific disabilities and technologies, beginning with the oldest and in many ways most successful domain: helping blind people to know what is in front of them. Chapter 2: Machines That Describe For a blind person, a large share of daily friction comes from small pieces of visual information that sighted people absorb without noticing. Which of these identical boxes is the cereal? Is this shirt navy or black? What does the letter from the bank say? Has the display on the washing machine changed? Is there a stain on this tablecloth? Did the photograph my friend just posted show her new baby or her new dog? None of these is a dramatic problem. Together they amount to a constant tax on time, independence and privacy, and for most of history the only reliable way to pay it was to ask a sighted person. Computer vision for blind and low-vision people is, at bottom, a long effort to offer an alternative to asking. Its arc runs from narrow recognisers that did one thing well, through a period in which human volunteers filled the gaps machines could not, to the present moment, in which general-purpose vision-language models can describe almost anything. Each stage solved real problems, and each revealed new ones. From single tasks to the open world The first generation of camera-based tools did one thing at a time. Optical character recognition apps read printed text; the KNFB Reader, a descendant of Kurzweil's work released as a smartphone app in 2014, became known among blind users for its accuracy on documents. Colour identifiers announced colours. Currency readers recognised banknotes. Barcode scanners looked products up in databases. These tools were reliable within their narrow domains precisely because the domains were narrow: a banknote recogniser only has to tell apart a few dozen designs. What they could not do was answer an arbitrary question about an arbitrary scene. A blind researcher's insight in 2010 was that if machines could not yet do this, people could. The VizWiz project, led by Jeffrey Bigham and colleagues and presented that year, let blind users take a photograph, record a spoken question, and receive answers from paid crowd workers within a minute or so. The project's real legacy was data: thousands of real photographs taken by blind people, with real questions attached. Those photographs looked nothing like the clean, well-framed images on which computer vision systems were then trained. They were blurred, badly lit, partially framed, often pointing at the wrong thing entirely, because a person who cannot see the viewfinder cannot easily aim a camera. Later work by Danna Gurari and colleagues turned these images into the VizWiz dataset, published in 2018, which became a standard benchmark precisely because it exposed how poorly mainstream models handled the kinds of images blind people actually take. In 2015 a Danish furniture craftsman named Hans Jørgen Wiberg, who was himself losing his sight, launched Be My Eyes. The app connected blind users by live video to sighted volunteers who had signed up to help. The idea was disarmingly simple, and it scaled remarkably: millions of volunteers registered, far outnumbering the blind users, so that calls were typically answered within seconds. Volunteers helped people read expiry dates, match clothes, navigate unfamiliar rooms and sort post. A professional alternative, Aira, founded in the same period, offered trained agents who could provide longer and more skilled assistance, including help with navigation, for a subscription. These human services remain important, and for good reason. A person can understand context, ask clarifying questions, notice what the user did not think to ask about, and take responsibility for what they say. But human assistance carries costs of its own. It requires inviting a stranger, however well-meaning, into one's home, one's mail, one's medicine cabinet. Many blind users report hesitating to call volunteers for trivial or private matters, or feeling that they are using up goodwill. Human help is also not always available in the moment, and professional help costs money. Meanwhile, the machines were catching up. Microsoft released Seeing AI in 2017, an app that bundled several recognisers into "channels": short text read aloud the moment it appeared in the camera's view, documents, products, people, currency, colours, light levels, and a "scene" channel that offered a one-sentence description of a photograph. Google's Lookout app followed in 2019 with a similar design. In 2016 Facebook introduced automatic alternative text, which generated short descriptions of photographs in users' feeds for screen reader users, of the form "Image may contain: two people, smiling, outdoor." These descriptions were crude, but they were the first time many blind users had any idea what was in the endless stream of images on social media. The arrival of general vision-language models The step change came in 2023. Large language models had become capable of fluent conversation; now they were combined with vision encoders that could interpret images. In March of that year, alongside the announcement of OpenAI's GPT-4, Be My Eyes revealed that it had been testing a feature, later called Be My AI, that let users send a photograph to the model and receive a detailed description, then ask follow-up questions in natural language. Over the following year it was rolled out to the app's users, and other tools, including Seeing AI, began incorporating similar models. The difference in experience was immediate and, for many users, profound. Instead of "a person standing in a room," the model might describe the person's clothing, posture and expression, the objects on the shelves behind them, the text on a poster on the wall, and the light coming through a window. A user could ask "what does the poster say?" or "is there a power socket near the bed?" and get an answer. People used it to read restaurant menus and work out which items were vegetarian, to understand charts in work documents, to identify plants in their gardens, to navigate the settings screens of devices whose accessibility had been neglected, and to enjoy family photographs in a way that had previously depended on someone else's patience. It is worth being precise about why these models work so much better. Earlier systems were trained to produce labels or short captions for a fixed set of categories. A vision-language model learns a joint representation of images and text from enormous numbers of image-text pairs, and then inherits the language model's capacity to compose, explain and answer questions. It can therefore handle objects and situations it was never explicitly trained to recognise, and it can adjust the level of detail to the question. It can also read text in images far more flexibly than traditional character recognition, including handwriting, stylised fonts and text at odd angles. Describing is not the same as knowing The same properties that make these systems useful make their failures harder to manage. A vision-language model generates a description by predicting plausible text conditioned on the image. Usually the plausible text is also the true text. Sometimes it is not. The model may report text that is not there, misread a number, confidently identify a medication from a blurry label, describe an object that a similar scene would typically contain but this one does not, or miss something important because it was not salient in the training data. In the language of the field, the model hallucinates. For a sighted user, a hallucination in an image description is an amusing error that can be checked at a glance. For a blind user, there is usually no glance to check with. That asymmetry is the central problem of AI visual description. The user is relying on the system precisely because they cannot verify its output, which means that the cost of an error depends heavily on whether the user can tell when the system is likely to be wrong. Consider three cases. Reading the colour of a shirt: an error is mildly embarrassing. Reading the dose on a medication bottle: an error could be dangerous. Describing whether the road is clear: an error could be fatal, which is why responsible tools explicitly tell users not to rely on them for crossing roads. The same underlying model performs all three tasks, with roughly similar fluency and apparent confidence. Nothing in the tone of the description distinguishes a correct answer from an incorrect one. Experienced users develop strategies. They ask the same question twice, or from a different angle, and compare answers. They ask the model to read text verbatim rather than summarise it. They use the AI for a first pass and call a human volunteer when the stakes are high. They learn which kinds of question the model handles well and which it tends to fabricate. In effect they become expert auditors of a system they cannot directly inspect, which is a cognitive burden, but one many prefer to the social burden of asking for help. Designers can reduce the burden. Systems can be built to express uncertainty when the image is blurred or the text is partially occluded, to say "I cannot read the last digit" rather than guessing. They can distinguish between what is directly visible and what is inferred. They can offer a quick route to a human when the user wants confirmation. Be My Eyes, for example, has kept its volunteer network running alongside the AI, so that users can escalate to a person. These are design choices, not technical inevitabilities, and they matter as much as the model's raw accuracy. Four ways of knowing what is in front of you By the mid-2020s, a blind person with a smartphone had several quite different ways of getting visual information, each with its own profile of strengths and weaknesses. The differences matter because users switch between them depending on the task, and because the choice of method is itself an expression of autonomy. Table 1 compares the four main approaches across the dimensions that users most often weigh. Table 1. Four approaches to visual assistance for blind and low-vision users. Approach Speed Typical failure Privacy Cost to user Volunteer by video (Be My Eyes volunteers) Seconds to connect Variable skill, social reluctance A stranger sees the scene Free Trained professional agent (Aira) Seconds to connect Limited by plan minutes Professional, bound by policies Subscription, some free access Task-specific recognition (short-text and currency modes in Seeing AI, Lookout) Instant, often on-device Narrow scope Can run without the cloud Free General vision-language model (Be My AI, AI features in Seeing AI) A few seconds Fluent hallucination Image sent to a company's servers Free or bundled The table shows that no single approach dominates. Task-specific recognisers are fast and private but limited. General models are flexible but can fabricate. Humans are the most trustworthy for high-stakes judgements but carry social and privacy costs. A good ecosystem preserves all four, and a worrying development would be one in which the cheapest option, the general model, crowded out the others because funders and companies assumed it had made them obsolete. Privacy and the photographs blind people take The privacy dimension deserves a closer look. Blind people's photographs often contain things that most people would never photograph and send to a company: the contents of their post, their bank statements, their prescription labels, their bodies, their children, the screens of their computers showing confidential work. A sighted person reading a letter does not create a record. A blind person using an AI service to read the same letter usually transmits an image of it to a remote server, where it may be stored and, depending on the provider's policies, used to improve the system. This is not a hypothetical concern. The VizWiz researchers found that a meaningful share of the images blind people submitted contained private information, often unintentionally, because the photographer could not see what else was in the frame. Later research has explored ways to detect and obscure private content automatically before images leave the device, and companies offering AI description have published privacy policies of varying strength. For a user, the practical question is often whether a given task can be done on-device, a capability that improves as phones gain more powerful processors. Describing people The hardest descriptions are of people. When a sighted person looks at a photograph of a group of friends, they take in, almost instantly, who is there, their approximate ages, what they are wearing, their expressions, perhaps their apparent ethnicity or gender, whether they look tired or happy. Blind people want access to much of this information too, for the same reasons sighted people use it: to recognise friends in photographs, to understand the mood of a picture, to know who is in the room. But describing people raises questions that describing a tin of tomatoes does not. In 2023, when Be My AI was first being tested, the service obscured people's faces in photographs before the model analysed them, a restriction linked to concerns at OpenAI about facial recognition and about the model making inferences about people's identities or characteristics. Blind testers objected publicly. They pointed out that sighted people see faces all the time, that a description of a family photograph that omits everyone's expression is a poor description, and that the restriction treated blind people as a privacy risk rather than as users. The episode was reported in the mainstream press, and the restrictions were later loosened for some uses, but it exposed a genuine dilemma. The same capability that lets a blind person know that her colleague looks upset could let a stranger identify people in a crowd. There is also the question of how to describe. Should a description mention skin colour, apparent gender, apparent disability, weight, attractiveness? Sighted viewers notice these things, but a model that announces them may be wrong, may impose categories that the people described would reject, and may reproduce stereotypes absorbed from its training data. Research by the accessibility scholar Cynthia Bennett and colleagues, published in 2021, interviewed blind people and others about descriptions of race, gender and disability in images, and found that the answers were complicated: many blind people wanted this information, particularly when it was relevant to the meaning of the image, but were wary of descriptions that guessed at identity from appearance. A photograph illustrating a news story about racial discrimination needs a description that says who is pictured; a holiday snapshot may not. Human describers handle this through judgement about context and purpose. Models can be instructed to do something similar, describing visible attributes such as clothing and hair rather than inferring identity, and leaving the user free to ask for more. But there is no neutral setting. Every choice about how a system describes people is an editorial decision, and the people best placed to make it, blind users and the people being described, have rarely been asked. Alternative text and the burden of description There is one more dimension to the story of machine description, and it returns to the social model. Much of the visual content that blind people struggle with is digital: photographs on websites and social media, charts in reports, images in emails. For decades, the accessibility standard for such content has been alternative text, a written description attached to the image by its author. When authors write good alt text, the problem is solved at its source, and the blind reader receives a description shaped by someone who knows why the image is there. In practice, most images on the web lack useful alternative text. Automated audits of the most popular websites, such as the annual WebAIM Million study, have repeatedly found missing alternative text among the most common accessibility failures. Automatic description generated by AI can fill the gap, and major platforms now offer it. But there is a risk that automation becomes an excuse. A generated description of a chart can say that it is a line graph with an upward trend; only the author knows which point the chart was meant to make. A generated description of a news photograph can identify the setting; only the editor knows why that photograph was chosen. If organisations stop writing alt text because "the AI will handle it," blind readers will receive descriptions that are fluent, generic and occasionally wrong, in place of descriptions that are purposeful and right. The most defensible position is that automatic description is a valuable fallback and a powerful tool for content that will never be described by a human, such as the photographs in a friend's message, and a poor substitute for description by authors who have the knowledge and the obligation to provide it. The same principle will recur throughout this book. AI is at its best when it extends a person's reach into a world that has not been made accessible, and at its worst when it is used to justify not making that world accessible in the first place. Reading a menu or a letter is one thing. Moving safely through a busy street, a railway station or an unfamiliar building is a different and harder problem, because errors are measured not in embarrassment but in injury, and because the world does not wait while the user asks a follow-up question. Chapter 3: Moving Through the World Ask blind people what technology could most improve their lives and many will not mention reading or describing images at all. They will talk about getting around: finding the right bus stop, locating the entrance to an unfamiliar building, crossing a complicated junction, finding a particular shop in a shopping centre, navigating an airport, or simply walking to a friend's house in a neighbourhood they do not know. Independent travel is where the gap between what technology promises and what it reliably delivers is widest, and where the cost of a mistake is highest. The cane, the dog, and the long failure of gadgets The baseline against which every navigation technology must be judged is formidable. The long white cane, refined into a systematic mobility technique in the 1940s by Richard Hoover and others working with blinded veterans, is cheap, durable, needs no batteries, and gives continuous, unambiguous information about the ground immediately ahead: steps, kerbs, obstacles, changes in surface. A trained guide dog adds the ability to avoid obstacles at body height, to find doors and crossings, and to exercise what trainers call intelligent disobedience, refusing a command that would lead into danger. Orientation and mobility instructors teach techniques that let skilled travellers move confidently through complex environments using sound, texture, wind and memory. For decades engineers have tried to improve on these tools, and for decades most of their inventions have gone unused. Electronic travel aids that used ultrasound or infrared to detect obstacles and signal them through vibration or sound were developed from the 1960s onward. Some found devoted users; most did not achieve wide adoption. The reasons echo the abandonment research from Chapter 1. The devices conveyed information that skilled cane users could already get; they produced streams of beeps or vibrations that were tiring to interpret; they occupied the hand or ears that the traveller needed for other things; they were expensive; and they failed in rain, in crowds or in bright sunlight. A tool that adds information but also adds noise, and that breaks in the conditions where it is most needed, is not an improvement. The lesson is that navigation is not a single problem. It has at least three layers. The first is obstacle avoidance and safety at the scale of a few metres, where the cane and the dog are already excellent. The second is orientation and wayfinding at the scale of a street or building: knowing where one is, which way one is facing, and how to reach a destination. The third is what might be called situational understanding: knowing what is going on around one, which shop is which, whether the queue is for the ticket office or the toilets, whether the bus that has just pulled in is the right one. Most of the real progress from AI has come in the second and third layers, not the first. Satellite navigation and its blind spots Satellite positioning transformed outdoor wayfinding for blind people, as it did for everyone else. But mainstream navigation apps are designed for drivers and sighted pedestrians who can glance at a map; their instructions assume a visual frame of reference and are often too coarse. Specialist apps emerged to fill the gap. BlindSquare, launched in 2012, combined GPS with location data from crowd-sourced databases to announce nearby places, intersections and points of interest as the user walked. Microsoft's Soundscape, released in 2018, used three-dimensional audio to place sounds in space, so that a "beacon" for the user's destination seemed to come from the direction of the destination itself, and announcements of nearby features came from where those features were. Soundscape's history is instructive. It was widely admired by blind users, and it explored a genuinely original idea: rather than telling people where to turn, it gave them an awareness of the space around them and let them decide. In 2023 Microsoft discontinued it. The company released the code as open source, and volunteers and other organisations have since built versions of their own, but for many users the episode reinforced a familiar anxiety. A tool they had built into their daily routines disappeared because a company changed its priorities. Satellite positioning also has hard limits. In urban canyons between tall buildings its accuracy can degrade to tens of metres, which is the difference between the right side of the street and the wrong one. Indoors it barely works at all. And the "last fifty metres" problem, finding the actual door of a destination once one has arrived roughly in front of it, remains notoriously difficult. A sighted person sees the door; a blind person may walk past it repeatedly. Computer vision on the move Artificial intelligence enters navigation mainly through computer vision, which can recognise and locate features of the environment that satellite systems cannot. Several approaches have emerged, each tackling a different part of the problem. One approach adds machine-readable markers to the environment. NaviLens, developed in Spain, uses brightly coloured square codes that a phone camera can detect from a much greater distance than a conventional QR code and at wide angles, without the user needing to aim precisely. When detected, the code tells the app what it is attached to, and the app guides the user towards it with audio cues. Transit systems in Barcelona and elsewhere have deployed these codes on signs and platforms, and New York's transit authority has piloted them. This is an interesting hybrid of the social and individual approaches: the environment is modified, cheaply, to make it legible to a tool the user carries. A second approach uses the sensors built into mainstream phones. When Apple added lidar depth sensors to some iPhones in 2020, it also added a People Detection feature to its Magnifier app, which tells the user how far away the nearest person is, useful in queues and for social distancing during the pandemic. A later Door Detection feature identified doors, read signs on them, and described how they opened. These features are modest, but they show the value of running vision on-device, without network delay, using hardware the user already owns. A third approach is the wearable camera. Smart glasses free the user's hands and point the camera where the head is pointing, which is more natural than holding a phone up in front of oneself while also holding a cane. Envision built its assistive software on Google Glass Enterprise Edition, and found an enthusiastic user base, until Google discontinued the hardware in 2023, forcing the company to look for other platforms. Meta's camera-equipped glasses, made with Ray-Ban, became a mass-market alternative; in late 2024 Be My Eyes integrated its volunteer service with them, so that a user could say a voice command and connect to a sighted volunteer who would see through the glasses' camera, and Meta's own assistant could describe scenes and read text. By 2025 the two companies were promoting further accessibility features for the glasses. For many blind users this combination of hands-free camera, conversational AI, and on-demand human help is the closest thing yet to a general-purpose visual assistant that fits into daily movement. A fourth approach is physical guidance. Researchers have long experimented with robotic guides, from wheeled devices to legged robots modelled on guide dogs. One commercial attempt, the Glide from the American start-up Glidance, takes the form of a handle attached to a small wheeled base that uses cameras and sensors to steer gently around obstacles and towards destinations, with the user walking behind it. Smart canes such as the WeWALK, developed in Turkey, add ultrasonic obstacle detection for head-height hazards and connect to navigation apps. Whether such devices achieve the wide adoption that earlier electronic travel aids did not remains to be seen; they will face the same test of whether they add more than they cost in weight, attention and reliability. Why reliability is not a single number The tension at the heart of AI-assisted navigation is that the tolerance for error depends dramatically on what the system is being asked to do. A system that identifies the right shopfront nine times out of ten is useful: on the tenth occasion the user discovers the mistake on entering and tries again. A system that correctly reports whether the road is clear nine times out of ten is lethal. This is not merely a matter of improving accuracy until it is "good enough." The errors of learned vision systems are not distributed randomly. They cluster in conditions that are underrepresented in training data: unusual lighting, heavy rain, snow, unfamiliar architecture, cities in countries where little data was collected. They also cluster in the edge cases that matter most for safety: an electric scooter left across the pavement, a temporary construction barrier, a cyclist on the footpath, a quiet electric vehicle. A system that has been evaluated on average performance may hide catastrophic weaknesses in precisely these conditions. Responsible navigation tools therefore draw a clear line. They present themselves as supplements to the cane or the dog, not replacements; they warn users not to rely on them for crossing roads; and they tend to focus on information that is useful but not safety-critical, such as reading signs, finding entrances and identifying bus numbers. Experienced travellers generally agree with this division of labour. The cane tells them the ground is safe; the AI tells them where they are and what is around them. There is a second, subtler reliability problem: attention. A traveller who is listening to a stream of descriptions from an AI system has less attention available for the sounds of traffic and the feedback from the cane. Orientation and mobility specialists worry that verbose systems can make people less safe, not more, by flooding the auditory channel. The best-designed tools are restrained, speaking only when spoken to or when something important changes. Restraint is rarely a selling point in a product demonstration, but it may matter more than any capability. Autonomous vehicles as accessibility The most ambitious mobility promise of artificial intelligence concerns not walking but driving. For blind people, people with epilepsy, many people with intellectual disabilities and others who cannot hold a licence, the inability to drive is one of the most significant barriers to employment, social life and independence, especially outside cities with good public transport. A fully autonomous vehicle would in principle remove it. The symbolism has not been lost on the industry. In 2012 Google released a widely shared video of Steve Mahan, a blind man, being driven by one of its prototype self-driving cars to a fast-food drive-through and a dry cleaner. In the years since, robotaxi services operating without safety drivers in several American cities have been used by blind riders, and disability organisations have lobbied for these services to be designed accessibly from the start: with audio guidance to locate the car, non-visual controls inside it, and support for wheelchair users. But the promise has repeatedly arrived later than predicted, and its accessibility is not automatic. A blind passenger needs to find the vehicle at the kerb, which is the same last-fifty-metres problem in a new form. A wheelchair user needs a vehicle that can take a wheelchair, which most autonomous fleets have not prioritised. And regulators in several countries still require driver's licences or human supervisors in ways that would exclude the people who stand to gain most. The autonomous vehicle illustrates the theme of this book at the scale of a whole industry: the technical achievement is necessary but not sufficient, and accessibility depends on decisions that are made, or not made, by people who are mostly not disabled. The indoor frontier If outdoor navigation is improving, indoor navigation remains one of the least solved problems. Large buildings such as hospitals, airports, universities and shopping centres are complex, change frequently, and lack satellite positioning. Several approaches have been tried: Bluetooth beacons installed throughout a building, Wi-Fi positioning, magnetic field mapping, and visual positioning, in which the phone's camera recognises its location by matching what it sees against a previously captured three-dimensional model. Each requires someone to install infrastructure or build a map, which means that accessibility depends once again on the building's owner choosing to invest. A journey in layers It helps to follow a single journey to see how these technologies fit together, and where they do not. Imagine a blind woman starting a new job in an office building she has never visited, in a part of the city she does not know. The evening before, she uses a navigation app to explore the route virtually, listening to the sequence of streets and junctions and the points of interest along the way. Some apps let users "walk" a route in advance in this way, building a mental map before they set foot outside. In the morning she walks to the station with her cane, using the same app, which announces street names as she passes junctions. Her cane, not the app, tells her about the kerb, the uneven paving stone and the sign board left on the pavement. At the station she needs the right platform. If the station has machine-readable signs, her phone guides her towards them and reads the platform information aloud. If it does not, she asks a member of staff, or uses an AI assistant on her glasses to read a departure screen, knowing that the assistant may misread a time or a platform number on a screen glimpsed from an angle. On the train she listens to announcements, if they are audible. Arriving at her destination station, she needs to find the right exit, which is often the moment when navigation apps are least helpful, since station interiors are poorly mapped. Out on the street again, the app brings her to within a few metres of the office building. Now she faces the last fifty metres. Is the entrance on this face of the building or around the corner? Is it a revolving door or a sliding one? She asks her glasses to describe what is ahead, and the assistant describes a glass frontage with a door in the centre and a sign above it. She walks to it and confirms with her hand. Inside, the reception desk is somewhere in a large, echoing lobby. A volunteer on a video call through her glasses guides her to it in a few seconds. Every layer of this journey used different tools, and at every layer the human element, her own skill with the cane, the station staff, the volunteer, remained essential. The AI contributed most at the points where she needed to read or interpret something visual and where an error was recoverable. It contributed least at the points where safety was at stake. And the journey would have been simpler for everyone if the station had been well mapped, the signs readable by machines, and the office building's entrance described on its website. The most promising developments combine methods. A traveller arriving at an airport might use satellite navigation to reach the terminal, a visual marker at the entrance to calibrate their position, a building map maintained by the airport, and an AI assistant on smart glasses to read departure boards and find the correct gate, with a human agent on call for the moments when everything else fails. No single technology carries the whole journey. What makes the system work is the combination and, above all, the user's ability to choose which tool to rely on at each moment. That theme of layered tools under the user's control recurs in a different form in the next chapter, which turns from sight to hearing and to a technology that has quietly become one of the most widely used AI accessibility tools in the world. Chapter 4: Listening Machines Of all the forms of artificial intelligence that serve disabled people, automatic speech recognition may be the one that has changed the most lives while attracting the least attention. It runs on nearly every smartphone sold today. It produces the captions on videos, video calls and live broadcasts. It lets a hard-of-hearing grandmother follow the conversation at a family dinner, a Deaf student read a lecture as it happens, and a worker with hearing loss take part in a meeting without asking colleagues to repeat themselves. It is also imperfect in ways that matter, and its imperfections fall unevenly on different speakers and different listeners. Who the listeners are It is important to be clear about who benefits from captioning, because the population is larger and more varied than is often assumed. The World Health Organization's World Report on Hearing, published in 2021, estimated that more than 1.5 billion people live with some degree of hearing loss, of whom around 430 million have hearing loss that is disabling. The vast majority of these people are not culturally Deaf sign language users. They are people who lost some or all of their hearing in adulthood, often through age or noise exposure, who speak and read the spoken language of their country and who may use hearing aids or cochlear implants. For them, captions are often the most natural form of access. Culturally Deaf people, whose first or preferred language is a signed language, are a smaller but distinct group. For many of them, the written form of the surrounding spoken language is a second language, sometimes learned with difficulty because of the educational policies that historically suppressed signing. Captions help, but they are not equivalent to access in their own language. That distinction matters for the next chapter, and it is worth holding on to here: captioning is excellent access for many people with hearing loss and partial access for many Deaf signers. How machines learned to transcribe Speech recognition has a long history of disappointing its promoters. Systems in the 1990s and 2000s required users to train them on their own voices, speak slowly and distinctly, and tolerate frequent errors. Dictation software found a niche, including among people with motor impairments who could not type, but it was a long way from transcribing natural conversation. The transformation came with deep learning in the 2010s. Neural networks trained on many thousands of hours of transcribed speech reduced error rates dramatically, and the improvements accelerated as datasets grew and architectures evolved. By the end of the decade, major technology companies were reporting error rates on standard benchmarks of conversational telephone speech that approached those of professional human transcribers. Those benchmark results overstated real-world performance, as benchmark results usually do, but the practical improvement was undeniable. In 2022 OpenAI released Whisper, a speech recognition model trained on around 680,000 hours of audio collected from the web, covering many languages, accents, recording conditions and background noises. Whisper was released openly, so that anyone could run it, and it proved notably robust to the messy audio of real life. It became the basis for a large number of applications, including captioning tools developed by small companies and independent developers. The capability that had once belonged to a handful of companies was now available to everyone. Captions in everyday life The accessibility applications followed the technology closely. YouTube introduced automatic captions in 2009; for years they were notoriously unreliable, and Deaf creators campaigned against poor automatic captions with the hashtag #NoMoreCraptions, popularised by the deaf YouTuber Rikki Poynter. The campaign's point was not that automation was bad but that platforms and creators were treating unchecked automatic captions as if they were adequate, when they frequently garbled names, technical terms and anything said in an accent the system had rarely encountered. In 2019 Google released Live Transcribe, an Android app developed in collaboration with Gallaudet University, the American university for Deaf and hard-of-hearing students. It displayed a running transcript of nearby speech on the phone's screen, letting a hard-of-hearing person follow face-to-face conversation. One of its leaders was Dimitri Kanevsky, a Google research scientist who is himself deaf and who had spent decades working on speech recognition. The same year, Google introduced Live Caption for Android, which captioned any audio playing on the device, including videos, podcasts and calls. Apple added Live Captions to its devices in 2022. Video conferencing platforms such as Zoom, Microsoft Teams and Google Meet built in automatic captions, which during the pandemic became the main route by which many people with hearing loss followed work meetings and classes. The sheer reach of these tools is hard to overstate. A capability that once required booking a professional stenographer, at substantial cost, is now available instantly and for free to anyone with a recent phone. For a hard-of-hearing person who might once have avoided group conversations, a doctor's appointment or a parents' evening, this is a profound change in daily life. The quality gap Yet the quality of automatic captions varies in ways that the user experiences directly. Several factors drive the variation. The first is the speaker. Speech recognition systems perform best on speech that resembles their training data, which has historically been dominated by certain accents and speaking styles. A widely cited study published in 2020 in the Proceedings of the National Academy of Sciences by Allison Koenecke and colleagues tested five commercial speech recognition systems from major technology companies and found that their average word error rate for Black American speakers was roughly twice that for white American speakers. Similar disparities have been found for regional accents, non-native speakers and, as Chapter 6 discusses, people whose speech is affected by disability. A Deaf student whose lecturer has a strong accent may receive far worse captions than a classmate in the next room. The second is the content. Specialised vocabulary, such as medical terms, legal language and the names of people and places, is where automatic captioning most often fails, and it is often exactly where accuracy matters most. A transcription that renders a drug name as a similar-sounding common word, or turns a lecturer's key technical term into nonsense, can make the most important sentence of the hour unintelligible. The third is the setting. Background noise, overlapping speakers, distance from the microphone and poor audio connections all degrade performance. Group conversations, in which people interrupt each other and the topic shifts rapidly, are among the hardest cases, and they are also among the situations where hard-of-hearing people most need help. Most consumer captioning tools also struggle to indicate who is speaking, so that a transcript of a meeting becomes an undifferentiated stream of words. The fourth is the absence of everything that is not words. Professional captioners convey sound effects, music, tone and the identity of speakers; a good human caption tells the viewer that a line was said sarcastically or that a door slammed off-screen. Automatic systems mostly transcribe words alone, and silently drop words they cannot recognise, which can change meaning without the reader ever knowing. Measuring what matters in a caption Part of the difficulty in judging automatic captions is that the standard metric, word error rate, does not measure what matters to the reader. Word error rate counts the proportion of words that are wrongly substituted, deleted or inserted compared with a reference transcript. It treats every word equally. Dropping "the" counts the same as changing "not" to "now", even though the first error is harmless and the second reverses the meaning of a sentence. A transcript can have a respectable word error rate and still mislead the reader on the one point that mattered. Researchers in subtitling and media accessibility have developed alternative measures. One, known as the NER model after the components it counts, was developed by the subtitling researcher Pablo Romero-Fresco and the subtitler Juan Martínez. Rather than counting raw word errors, it assesses each error by its effect on meaning, distinguishing minor errors that the reader can easily recover from, standard errors that cause confusion, and serious errors that introduce false information. It also gives credit for acceptable editing, since good live captioners often condense speech without losing meaning. The British broadcasting regulator Ofcom used this approach to assess the quality of live subtitling on television. The idea behind it is applicable well beyond broadcasting: the quality of a caption is a property of what the reader understands, not of how closely the text matches the audio. This matters for decisions about automation. An institution choosing between human and automatic captioning is likely to be shown accuracy figures based on word error rate, under test conditions. A Deaf or hard-of-hearing user experiences something different: whether, at the end of a lecture or a consultation, they understood what was said, and whether anything they understood was wrong. Evaluations that ask that question directly, ideally with the people who will rely on the captions, are rarer and more expensive than benchmark tests, but they are the evaluations that tell the truth. A related problem is that automatic systems do not flag their own uncertainty. A human captioner who misses a word often indicates it; an automatic system usually substitutes a plausible guess and moves on. Some research systems have experimented with marking low-confidence words, for instance by displaying them in a different style, so that the reader knows where to be sceptical. This is the captioning equivalent of the uncertainty signals discussed for image description in Chapter 2, and it rests on the same principle: a tool that tells the user when it might be wrong is more useful than one that is slightly more accurate but silent about its errors. Human captioners and the danger of substitution Before automatic captioning, the standard for live access was human real-time captioning, known in the United States as Communication Access Realtime Translation, or CART. Trained stenographers type at the speed of speech using a chorded keyboard and a personal dictionary, producing near-verbatim transcripts with speaker identification and corrections. Their accuracy in difficult conditions remains well above that of automatic systems, and they can prepare in advance by learning the vocabulary of a course or a conference. Human captioning is expensive, and it is here that the economic pressure becomes visible. Universities, employers, courts and broadcasters facing budgets have every incentive to replace human captioners with automatic captions, which cost almost nothing at the margin. For many routine uses, automation is a reasonable choice. But the substitution is often made on the basis of average accuracy figures, without asking whether the captions are good enough for the specific user in the specific setting, and without giving the disabled person a voice in the decision. A Deaf law student who needs accurate captions of a seminar on contract law is poorly served by a tool that performs well on casual conversation. The emerging best practice is not to choose between human and machine but to combine them. Some services now offer automatic captions corrected in real time by a human editor, which is cheaper than full human captioning and more accurate than automation alone. For high-stakes settings, such as medical consultations, legal proceedings and examinations, the guidance from Deaf and hard-of-hearing organisations is generally that the person who needs access should decide what form it takes. Beyond transcription: sound awareness and hearing aids Speech is not the only sound that matters. Doorbells, smoke alarms, crying babies, running water, a dog barking, a car horn approaching: hearing people respond to these constantly without thinking. Machine learning models trained to classify environmental sounds now run on mainstream phones and watches. Apple introduced Sound Recognition in 2020, and Android offers a comparable Sound Notifications feature. These tools send a visual or vibrating alert when they detect sounds the user has chosen. They are not reliable enough to replace a dedicated, certified smoke alarm system with flashing lights, but for everyday awareness they fill a real gap. Artificial intelligence has also moved inside hearing aids themselves. The most persistent complaint of hearing aid users is that, while the devices amplify sound, they amplify everything, so that in a noisy restaurant the conversation across the table is drowned out by the clatter of dishes and the voices at the next table. Hearing aid makers have long used signal processing to address this, and recent devices incorporate neural networks trained to separate speech from noise. In 2024 the Swiss manufacturer Sonova introduced a hearing aid with a dedicated chip for running such a network in real time. The same year, the United States Food and Drug Administration authorised a hearing aid feature for Apple's AirPods Pro 2, following an earlier change in American rules that allowed hearing aids for mild to moderate loss to be sold over the counter. Consumer earbuds that function as hearing aids could bring help to millions of people who have never used one, partly because they avoid the stigma that many people associate with traditional hearing aids. Captions in the field of view One of the more striking recent developments puts captions directly in front of the user's eyes. Several companies have released smart glasses or software for existing glasses that display live captions on a small transparent display, so that a hard-of-hearing person can look at the speaker's face while reading what they are saying. Big technology companies have demonstrated similar prototypes, sometimes combined with live translation between spoken languages. The appeal is obvious: reading captions on a phone means looking down, losing the speaker's facial expressions and lip movements, which are themselves important cues, and signalling to everyone in the conversation that something unusual is going on. Glasses keep the user's attention on the person. The practical obstacles are the familiar ones: battery life, weight, the quality of microphones in noisy environments, the cost of the hardware, and the question of whether the people being transcribed have agreed to have their words processed by a remote server. That last question applies to all automatic captioning and is often overlooked. When a hard-of-hearing person uses a phone app to transcribe a conversation, everyone in that conversation is being recorded and processed, sometimes by a cloud service. In many settings this is uncontroversial. In a medical consultation, a confidential business meeting or a private family argument, it may not be. The increasing availability of on-device recognition, where no audio leaves the phone, eases the problem, and it is another example of how the location of computation, on the device or in the cloud, is not merely an engineering detail but a question of autonomy and trust. Automatic captioning shows what AI accessibility looks like when it matures: cheap, ubiquitous, genuinely useful, quietly imperfect, and at risk of being used to justify the removal of better but costlier alternatives. Sign language translation, the subject of the next chapter, shows what it looks like when the technology is still far from maturity and the people it is meant to serve have strong and well-founded views about what they want from it. Chapter 5: Sign Language and the Limits of Translation Every few years a news story appears with a familiar shape. A team of young engineers has built a device, often a glove studded with sensors, sometimes a camera system or a smartphone app, that "translates sign language into speech." There is a photograph of a smiling inventor, a quotation about breaking down barriers, perhaps an award. Deaf people read these stories with a sigh. Sign language translation is one of the most frequently promised and least delivered applications of AI in accessibility, and the reasons why are among the most instructive in the whole field. What sign languages are The first reason is that most of the people who set out to translate sign language do not understand what a sign language is. It is worth spelling this out, because the misunderstanding is so common and so consequential. Sign languages are natural human languages, with their own grammar, vocabulary and history. They are not gestural codes for spoken languages, nor are they universal. American Sign Language is more closely related to French Sign Language, from which it partly descends, than to British Sign Language, even though the United States and Britain share a spoken language. The World Federation of the Deaf estimates that there are more than seventy million deaf people worldwide and more than three hundred sign languages. The linguistic study of sign languages is usually dated to 1960, when the American linguist William Stokoe published an analysis showing that the signs of ASL could be decomposed into a small set of contrasting elements, in the same way that spoken words decompose into sounds. This work helped establish that sign languages were languages in every scientific sense, a claim that was controversial at the time and that changed Deaf education. Linguists now describe signs in terms of several parameters: the shape of the hand, its location relative to the body, its movement, the orientation of the palm, and what are called non-manual markers, such as facial expressions, head movements, eye gaze and mouth shapes. Non-manual markers are not decoration. In ASL, for example, raised eyebrows can mark a yes-or-no question, and particular mouth shapes modify the meaning of a sign to indicate size or intensity. A system that tracks only the hands is missing part of the grammar. Sign languages also make extensive use of space. A signer can establish a person or object at a point in the space in front of them and then refer back to it by pointing or by directing a verb towards that point. Verbs can move between locations to show who did what to whom. Depicting constructions, often called classifiers, use handshapes representing categories of objects, a vehicle, a person, a flat surface, to show how things move and relate to one another, in a way that has no direct equivalent in spoken language. Several pieces of information are expressed simultaneously, through the two hands, the face and the body, rather than one after another. Finally, most sign languages have no widely used written form. This has a direct consequence for machine learning. Spoken language translation systems learned from enormous quantities of text, much of it already translated between languages. There is no comparable corpus for sign languages. Training data must be video, which is far more expensive to collect, annotate and store, and annotating it requires fluent signers, of whom there are relatively few with the relevant technical skills. The trouble with gloves Against this background, the persistent appeal of sign language gloves looks less like a technical project and more like a symptom. In 2016 two undergraduate students at the University of Washington won a prestigious student invention prize for a pair of gloves called SignAloud, which used sensors to detect hand movements and convert them to speech. The project attracted wide media coverage. It also attracted a public letter from faculty in the university's own ASL programme and from members of the Deaf community, who pointed out that the gloves could recognise only a limited set of hand configurations, ignored facial grammar and the use of space entirely, and were developed without meaningful involvement from Deaf people. A widely read essay in The Atlantic the following year explained to a general audience why Deaf people found such devices frustrating. The objection was not only that the gloves did not work well. It was that their premise was backwards. A glove asks the Deaf person to wear special equipment so that hearing people need not learn to communicate. It treats sign language as a problem to be converted rather than a language to be respected. And by presenting a crude device as a solution, it risks persuading decision-makers that interpreters and sign language education are less necessary than they are. The gloves embody the medical model in technological form, and the reaction to them is the social model in action. That episode has become a teaching case in human-computer interaction courses, and more careful researchers have absorbed its lessons. A landmark paper published in 2019 by Danielle Bragg and colleagues, an interdisciplinary team that included Deaf researchers, set out what a responsible research agenda for sign language technology would require: Deaf involvement at every stage, much larger and more diverse datasets collected with consent, attention to the full linguistic structure of signing, and clarity about which applications Deaf people actually want. What machines can and cannot yet do Modern sign language technology uses computer vision rather than gloves. Pose estimation models track the positions of the hands, arms, body and face in ordinary video; neural networks then learn to map sequences of poses, or raw video, onto signs, glosses or written sentences. The task is divided into several sub-problems that differ greatly in difficulty and maturity, as Table 2 summarises. Table 2. The main tasks in sign language technology and their state of development. Task What it involves Maturity Main difficulty Realistic near-term use Isolated sign recognition Identifying single signs from a fixed vocabulary Good in controlled settings Signer variation, limited vocabulary Learning apps, dictionary lookup Fingerspelling recognition Reading letter-by-letter spelling Improving quickly Speed and co-articulation Names, addresses, search input Continuous recognition and translation Converting fluent signing into written sentences Early Grammar of space and face, scarce data Limited, supervised domains Sign generation by avatar Producing signing from text Usable for scripted content Natural facial grammar, fluency Fixed announcements, simple notices Isolated sign recognition, identifying which of a set of known signs a person has produced, is the easiest and the most advanced. It works reasonably well in controlled conditions with a known vocabulary, and it has genuine uses. PopSign, developed by researchers at the Georgia Institute of Technology and the Rochester Institute of Technology with support from Google, is a smartphone game that helps hearing parents of deaf children learn ASL vocabulary, using sign recognition to check their attempts. This is a good example of an application that serves the Deaf community's priorities: most deaf children are born to hearing parents, and early exposure to sign language matters greatly for language development. Fingerspelling recognition, reading the hand alphabet used for names and words without established signs, has advanced through public competitions and larger datasets. Continuous sign language translation, turning natural, fluent signing into accurate written sentences, is far harder. For years the field's main benchmark was a German dataset drawn from weather forecasts on public television that were accompanied by sign language interpretation. Weather forecasts are a narrow domain with repetitive vocabulary, and systems that performed well on it did not generalise. Newer datasets, such as How2Sign, released in 2021 with around eighty hours of ASL video, and YouTube-ASL, assembled by Google researchers in 2023 from about a thousand hours of ASL videos with captions, have enlarged the available data. Even so, these are tiny by the standards of spoken and written language, and they come disproportionately from particular groups of signers. In 2025 Google announced SignGemma, a model intended to translate ASL into English text, and invited Deaf people and researchers to give feedback before release. That framing, involving the community before launch rather than announcing a solution, reflects a real change in how large companies approach the subject, though the measure of success will be whether Deaf people find the model useful in daily life and whether they retain influence over how it is deployed. Avatars and the other direction Translation can also run the other way, from spoken or written language into sign. Here the technology takes the form of a signing avatar, a computer-generated figure, or, in newer systems, a photorealistic synthetic video of a signer. The appeal for organisations is clear: an avatar can sign announcements, website content or information leaflets at a fraction of the cost of filming a human interpreter, and it can be updated instantly. In Britain, the company Signapse has developed systems that generate British Sign Language videos of railway announcements, deployed at some stations, as well as signed versions of website content. Such deployments address a real gap: railway announcements are often inaccessible to Deaf passengers, and a signed version of a fixed message such as a delay or a platform change is clearly better than nothing. Deaf organisations have nonetheless been cautious. In 2018 the World Federation of the Deaf and the World Association of Sign Language Interpreters issued a joint statement on signing avatars. They accepted that avatars might be suitable for short, pre-scripted, one-way information, such as announcements, but argued that they were not an acceptable substitute for human interpreters in live interaction, in complex or important information, or in any situation that requires the flexibility, cultural knowledge and judgement of a skilled interpreter. They also stressed that the quality of avatar signing had to be judged by Deaf people, because hearing developers are rarely in a position to tell whether signing is fluent, intelligible and natural. The concern about quality is not pedantic. Avatar signing has historically been stiff and robotic, with limited facial expression, which means it omits part of the grammar. A Deaf viewer may be able to decode it, much as a hearing listener can decode a crude speech synthesiser, but the effort is greater and the meaning less certain. Newer generative systems produce more natural-looking signers, but naturalness of appearance is not the same as linguistic accuracy. A photorealistic signer who signs ungrammatically may be more misleading, not less, because it looks authoritative. Interpreters, automation, and who decides Behind the technical debate lies an economic one. Qualified sign language interpreters are scarce and their services are costly, which means that Deaf people in many settings, including hospitals, courts, schools and workplaces, routinely go without the interpretation to which they may be legally entitled. Any technology that offers a cheaper alternative will be attractive to the institutions that pay. That creates the same danger seen with captioning: automated systems being deployed to replace interpreters in settings where they are not good enough, on the strength of demonstrations and average accuracy figures, without the Deaf person having any say. Deaf scholars and advocates have repeatedly made the point that the most effective way to improve access is not necessarily to automate interpretation but to make sign language more widely available: to teach it to deaf children and their families early, to train more interpreters, including Deaf interpreters, and to make sign language recognised and supported in public life. Technology can contribute to all of these goals. Learning apps can help hearing families acquire sign language; remote video interpreting can bring human interpreters to places they could not otherwise reach; recognition tools can help Deaf people search video dictionaries using signs rather than written words. These are less dramatic than a device that "translates sign language," but they work with the language community rather than around it. A recent precedent shows how easily cost-saving technology can degrade access. Video remote interpreting, in which a sign language interpreter joins a conversation by video link rather than in person, spread rapidly through American hospitals in the 2000s and 2010s. In principle it was a good idea: interpreters could reach rural hospitals and emergency departments at any hour, without travel. In practice, Deaf patients and their organisations reported frozen screens, poor connections, cameras that could not capture the patient's signing when the patient was lying in bed, and interpreters who could not see what was happening in the room. Some patients in pain, under medication or with limited vision could not use a screen at all. Hospitals that had replaced in-person interpreters with video units sometimes insisted on using them in exactly these circumstances, and a series of lawsuits under the Americans with Disabilities Act followed. The National Association of the Deaf and other organisations published guidance on when remote interpreting is and is not appropriate. Automated sign language translation, if deployed in the same spirit, would repeat the pattern with a technology that is, for now, far less capable than a remote human interpreter. The lesson of video remote interpreting is not that remote technology is bad; many Deaf people use and value it. It is that a technology chosen by an institution to reduce costs, and used without regard to the Deaf person's judgement about whether it is working, will fail precisely in the high-stakes situations where access matters most. The safeguard is simple to state: the Deaf person must be able to say "this is not working" and have that decision respected. There is also a question of data and ownership. Training sign language models requires video of real signers, which is personal and identifiable in a way that text is not: a signer's face is part of the language. Collecting this data responsibly requires informed consent, fair compensation, and clarity about how it will be used and who will profit from the resulting models. Deaf communities are small, and many have a long memory of research done on them rather than with them. Projects that build trust, involve Deaf researchers as leaders and share benefits with the community are likely to obtain better data and produce more useful systems than projects that scrape whatever video they can find online. The trajectory of sign language technology suggests a broader lesson for this book. Progress has been slow not because the engineers are unskilled, but because the problem is harder than outsiders assume, the data is scarce, and the people with the knowledge to solve it were long excluded from the work. Where Deaf people have taken leading roles, the questions asked have changed, and so have the answers. The next chapter turns to people whose difficulty is not understanding speech but producing it, and to technologies that, at their best, give people back their own voices. Chapter 6: Restoring Voice Speech is so central to human social life that its loss can be among the most isolating experiences a person can have. People who cannot speak, or whose speech is hard for others to understand, report being talked over, ignored, addressed through their companions, assumed to be cognitively impaired, and excluded from conversations that move too fast for them to join. The causes are many: amyotrophic lateral sclerosis, cerebral palsy, Parkinson's disease, multiple sclerosis, stroke, traumatic brain injury, head and neck cancer, autism for some people, and a range of developmental conditions. The technologies that address these difficulties fall into three families, and artificial intelligence has transformed each of them: recognition systems that can understand atypical speech, voice synthesis that can preserve or recreate a person's own voice, and communication aids that help people compose what they want to say more quickly. Being understood by machines Chapter 4 described how speech recognition became good enough to caption ordinary conversation. For people whose speech is affected by a motor disorder, a condition known as dysarthria, the same systems often fail badly. Dysarthric speech may be slow, slurred, breathy or irregular in rhythm, and mainstream recognisers, trained overwhelmingly on typical speech, may get most words wrong. This is a double exclusion. The voice assistants and dictation tools that could most help people with motor impairments, who often find typing difficult as well, are precisely the tools that cannot understand them. The fix, in principle, is straightforward: train systems on atypical speech. In practice, such data barely existed. In 2019 Google announced Project Euphonia, a research effort to collect recordings from people with speech impairments and use them to build better recognition. Participants recorded hundreds of phrases each. The research showed that models personalised to an individual speaker, fine-tuned on that person's recordings, could reach accuracy far above that of general models, often good enough for practical use even for people whose speech was hard for unfamiliar listeners to understand. The work led to Project Relate, an Android app released for testing in 2021 that let people with atypical speech train a personal model and use it for transcription, for repeating their words in a clear synthetic voice, and for controlling a voice assistant. Personalisation has a cost: each user has to record a substantial amount of speech, which can be tiring for people with progressive conditions, and the resulting model is useful only to that person. The longer-term goal is general models that understand a wide range of atypical speech out of the box. That requires large, shared datasets, which is why the Speech Accessibility Project, launched in 2022 at the University of Illinois Urbana-Champaign's Beckman Institute, was notable. It was backed jointly by several of the largest technology companies, including Amazon, Apple, Google, Meta and Microsoft, which are normally fierce competitors, and it set out to recruit people with conditions including Parkinson's disease, ALS, cerebral palsy, Down syndrome and the after-effects of stroke to record speech samples, with the resulting data made available to researchers and companies under licence. Participants were paid for their time. The project has since reported that models trained on its data substantially improved recognition for speakers with Parkinson's disease, and Microsoft has said that the data contributed to improvements in its commercial speech service. The Speech Accessibility Project is a useful model of how data for disability should be gathered: with consent, payment, and the participation of disability organisations in recruitment, and in a form that benefits many products rather than one. It also illustrates a cost that is easy to overlook. People with speech disabilities are being asked to contribute their time and voices to train systems that should have been built to understand them in the first place. That they are paid and thanked does not change the fact that the burden of fixing a biased technology falls, once again, on the people it excluded. Smaller companies have also built products specifically for atypical speech. Voiceitt, founded in Israel, offers an app that learns an individual's speech patterns and lets them dictate text, communicate with others through a clear synthetic voice, and control smart home devices. Its existence points to a recurring tension. Specialist firms can focus on users that large companies neglect, but they are vulnerable to the economics of serving a small market, and their users may find themselves stranded if the business fails. Stuttering and the patience of machines Atypical speech is not only a matter of articulation. People who stutter speak words clearly but with repetitions, prolongations and blocks, moments in which the word does not come. Voice assistants have been particularly frustrating for them, for a reason that has nothing to do with recognising sounds: the systems decide that the speaker has finished when they hear a pause, and a block looks like a pause. A person who stutters may be cut off mid-request again and again, and some give up on voice interfaces entirely. This is a design problem as much as a recognition problem, and it has begun to be addressed. In 2021 researchers at Apple released a dataset called SEP-28k, containing tens of thousands of short audio clips from podcasts featuring people who stutter, labelled for different types of disfluency, to support research on recognition of stuttered speech. Apple later added a setting that lets users extend the time its voice assistant waits before deciding they have finished speaking, and a further option for the assistant to listen for atypical speech. Google researchers have published work on making recognition more robust to disfluencies. None of this is technically glamorous. It is a matter of noticing that a design choice built around typical speakers, namely that a short silence means the end of a sentence, excludes a large group of people, and of giving them control over that choice. The lesson generalises. Many accessibility failures in AI systems are not failures of capability but of assumption: about how long people take to respond, how they phrase things, how quickly they move, what counts as a normal pattern. A system that lets users adjust those assumptions to fit themselves is often more valuable than a marginally more accurate system that does not. Keeping one's own voice For people who know they are going to lose their speech, most commonly people with ALS or those facing surgery for throat or mouth cancer, a different question arises: when I can no longer speak, what will I sound like? For decades the answer was a generic synthetic voice. The physicist Stephen Hawking's voice, produced by a speech synthesiser from the 1980s, became so associated with him that he declined to change it even when more natural voices became available. For most people, though, being given a stranger's voice, often of the wrong age, accent or even gender, is an additional loss on top of the loss of speech. Voices carry identity: regional origins, personality, the way a person laughs or emphasises a word. Children who hear a parent's new synthetic voice may not recognise it as their parent. Voice banking addresses this. A person records themselves while they can still speak, and the recordings are used to build a synthetic voice that resembles their own. Early services, such as ModelTalker in the United States, required recording well over a thousand sentences and produced voices that captured some of the speaker's character but sounded clearly artificial. The company VocaliD, founded by the speech scientist Rupal Patel, explored blending the remaining vocal characteristics of a person who could make only limited sounds with recordings from a donor of similar age and background, so that people who had never had typical speech could have a voice that was distinctively theirs. A related practice, message banking, involves recording specific phrases, jokes and expressions in one's natural voice to be played back later; it captures the voice perfectly but only for what was recorded. Neural speech synthesis changed the economics dramatically. Modern voice cloning can produce a convincing likeness from minutes, rather than hours, of recording, and sometimes from old home videos or voicemails if the person has already lost their voice. In 2023 Apple introduced Personal Voice, a feature that lets users record a set of phrases, around fifteen minutes of reading, and generates a synthetic voice on the device itself, so that the recordings need not be sent to Apple. The voice can then be used with the device's text-to-speech features to speak typed words in calls and conversations. Other companies, notably ElevenLabs, have offered free access to their voice cloning tools for people with ALS and other conditions through partnerships with disability organisations. One public example illustrates what this can mean. In July 2024 Jennifer Wexton, a member of the United States House of Representatives who had been diagnosed with progressive supranuclear palsy, a neurodegenerative condition that had severely affected her speech, delivered a speech on the House floor using an AI-generated version of her own voice, built from recordings made before her illness. It was reported to be the first time a member of Congress had addressed the chamber in this way. Beyond its symbolism, it showed a practical point: a voice that sounds like the speaker changes how listeners respond, restoring a measure of the speaker's presence that a generic voice does not. The same technology that gives people their voices back also creates risks. Voice cloning is used for fraud, impersonation and disinformation, and companies that offer it have introduced safeguards such as verification of consent and detection of synthetic speech. For people who rely on a cloned voice, these safeguards can become barriers. Some banks use voice recognition for authentication, which a synthetic voice may defeat or fail. A person whose voice is a clone may also worry about who else can use it and what happens to the voice model if the company that hosts it changes its terms or closes. Ownership and control of a person's synthetic voice is an unsettled area of law and practice, and for this group of users it is not a theoretical concern but a matter of identity. Speaking faster: augmentative and alternative communication For many people who cannot speak, the main communication tool is augmentative and alternative communication, usually called AAC. AAC ranges from low-tech boards of letters, pictures and symbols that the user points to, through dedicated electronic devices, to apps on tablets that speak the words the user selects. People who cannot use their hands may access AAC through switches operated by a head movement or a single finger, or through eye-tracking cameras that let them select letters by looking at them. The fundamental problem of AAC is speed. Typical spoken conversation proceeds at somewhere between about 120 and 180 words per minute. Many AAC users, particularly those who rely on switches or eye gaze, communicate at a small fraction of that rate, often well under twenty words per minute and sometimes far less. Hawking, in his later years, could manage only a word or so per minute using a sensor triggered by a cheek muscle. At such speeds, conversation in the ordinary sense becomes almost impossible. By the time an AAC user has composed a response, the conversation has moved on. Listeners finish sentences for them, often incorrectly, or lose patience. Word prediction has been part of AAC for decades: the system suggests likely next words based on what has been typed so far. Hawking's system, developed with Intel, incorporated prediction software from the company SwiftKey. Large language models dramatically extend what prediction can do. Rather than predicting the next word from the last few, a language model can predict whole phrases from a few letters or keywords, take account of the conversation so far, and adapt to the user's style. Google researchers working with people with ALS who used eye gaze explored a system called SpeakFaster, published in 2024, in which users typed only the initial letters of the words they intended, and a language model expanded these abbreviations into full sentences for the user to accept or edit. In tests, it substantially reduced the number of selections users needed to make. Whose words are these? This is where the question of agency becomes acute. If a language model proposes a whole sentence and the user accepts it, are those the user's words? In many cases the answer is clearly yes: the user had a meaning in mind, the model expressed it efficiently, and the user approved. But the pressure of conversation creates a pull towards accepting a suggestion that is close enough rather than spending a minute composing exactly what one meant. Over time, a user's communication may drift towards the bland, generic phrasing that language models favour, and away from their own idiosyncratic voice, humour and opinions. AAC users and researchers have raised exactly these concerns. They worry about losing the texture of their personalities, about the system putting words in their mouths, about listeners not knowing whether a sentence came from the person or the machine, and about the system listening to every conversation in order to provide context-aware suggestions. There is also a power dimension. AAC users have often had their communication controlled by others: by professionals who decided which words were programmed into their devices, by family members who spoke for them. A system that subtly shapes what they say reproduces that control in a new form. Good design can mitigate these risks. Suggestions can be offered as options rather than defaults. Systems can learn from the user's own past utterances, so that suggestions reflect the user's style. Users can control what the system listens to and when. And users can choose to signal to listeners when they are using rapid AI-assisted expression, so that a quick reply is not mistaken for a considered statement. Above all, the user must remain in control of the final message. A communication aid exists to carry a person's meaning, and its success should be measured by how faithfully it does that, not merely by how quickly it produces text. The tools described in this chapter all depend on the user retaining some movement: of the mouth, the hands, the head or the eyes. For people with the most severe paralysis, including those with advanced ALS or a brainstem stroke, even eye movement may be lost or unreliable. For them, the remaining route to communication runs directly from the brain, which is the subject of the next two chapters. Chapter 7: Reading the Brain There is a condition, most often caused by a stroke in the brainstem, in which a person is fully conscious, able to think, feel, see and hear, but almost completely paralysed, often able to move only their eyes. It is called locked-in syndrome. In its most extreme form, which can occur at the end stage of ALS, even eye movements are lost. For people in these states, every assistive technology described so far fails, because each requires some voluntary movement to operate. The brain-computer interface, or BCI, begins from a different premise: if the brain still forms the intention to move or speak, perhaps the intention can be read directly, bypassing the broken pathway between brain and body. That idea is old. The term "brain-computer interface" was coined in the early 1970s by the computer scientist Jacques Vidal at the University of California, Los Angeles. What has changed in the past two decades, and especially in the past five years, is that the idea has started to work well enough to change individual lives. Artificial intelligence, in the form of machine learning decoders, has been central to that progress. How the brain can be read To read intentions from the brain, a system must first record its activity, and the method of recording shapes everything else. There are three broad approaches, which differ in how close the sensors are to the neurons. The least invasive is electroencephalography, or EEG, which records electrical activity through electrodes placed on the scalp. EEG is safe, cheap and widely available, and EEG-based BCIs have been used for decades. But the skull and scalp blur and weaken the signals, so that EEG captures the summed activity of very large populations of neurons. EEG systems can detect broad states and certain kinds of evoked response, such as the brain's characteristic reaction when a person sees a letter they have been waiting for, and they can support slow spelling. They cannot read fine-grained intentions such as the intended movement of individual fingers or the sounds of intended speech. The next level of invasiveness is electrocorticography, or ECoG, in which a thin sheet of electrodes is placed on the surface of the brain, beneath the skull. ECoG has long been used in hospitals to locate the source of seizures in people with epilepsy before surgery, and much early BCI research took advantage of these patients, who had electrodes implanted for clinical reasons and volunteered for experiments during their hospital stay. ECoG signals are much clearer than EEG, and they cover larger areas of the cortex than penetrating electrodes, which is useful for decoding speech, since speech involves coordinated activity across a broad region. The most invasive approach uses microelectrodes that penetrate a millimetre or two into the cortex and record the electrical spikes of individual neurons or small groups of them. The best-known device of this kind is the Utah array, developed by Richard Normann at the University of Utah: a silicon chip about four millimetres square, bristling with around a hundred needle-like electrodes. Utah arrays provide the most detailed information available about what the brain is doing, but they record from a tiny patch of tissue, require open brain surgery, and their signal quality tends to decline over years as the brain's tissue reacts to the foreign object. Whatever the recording method, the central computational problem is the same. The system records patterns of neural activity while the person attempts, or imagines, a range of actions, and it learns a mapping from activity to intention. This mapping is the decoder. Early decoders used relatively simple statistical methods, which worked well for controlling a cursor in two dimensions. More complex tasks, such as decoding handwriting or speech, have depended on modern machine learning: recurrent and transformer neural networks that model the evolving sequence of neural activity, combined with language models that use knowledge of which sequences of words are likely to correct errors. From cursors to robotic arms The modern era of intracortical BCIs began with the BrainGate research programme, a collaboration that has involved, over the years, Brown University, Massachusetts General Hospital, Stanford University and other institutions. In 2006 the journal Nature published a paper by Leigh Hochberg and colleagues describing Matthew Nagle, a young man paralysed below the neck after being stabbed, who had a Utah array implanted in his motor cortex. By imagining moving his hand, he could move a cursor on a screen, open simulated email, change television channels, and open and close a prosthetic hand. It was crude by later standards but unmistakably a proof that intentions could be read from the cortex of a paralysed person and turned into action. The following years extended the principle. In 2012 a BrainGate paper described a woman who had been paralysed by a brainstem stroke fifteen years earlier using a robotic arm, controlled by her neural activity, to lift a bottle and drink coffee through a straw, the first time she had served herself a drink since her stroke. At the University of Pittsburgh, Jan Scheuermann, paralysed by a degenerative disease, learned to control a sophisticated robotic arm with remarkable dexterity, feeding herself chocolate. Later work at Pittsburgh added stimulation of the sensory cortex, so that a participant could feel pressure on the fingers of a robotic hand, which improved his ability to grasp and manipulate objects. These were striking demonstrations, but robotic arms are expensive, bulky and need supervision. For most people with severe paralysis, the more immediately useful goal is to operate a computer, because a computer gives access to communication, information, entertainment and control of the home environment. Typing performance improved steadily through the 2010s, but even the best results with point-and-click typing remained well below the speed of ordinary typing on a phone. Handwriting and speech In 2021 the field took a leap. A team at Stanford led by Francis Willett, Krishna Shenoy and Jaimie Henderson reported in Nature on a participant, a man in his sixties paralysed from the neck down by a spinal cord injury, who was asked to imagine handwriting letters as if holding a pen. A recurrent neural network decoded the neural activity associated with each attempted letter. He reached about ninety characters per minute, with raw accuracy above ninety-four percent, rising above ninety-nine percent when an autocorrect language model was applied. That rate was roughly comparable to typing on a smartphone for people of his age, and more than double the previous record for BCI typing. The insight behind it was that complex, distinctive movements such as letters are easier to tell apart in neural activity than the simple straight-line movements of a cursor. The bigger prize was speech. Speech is the fastest natural means of human communication, and people with ALS or brainstem stroke have lost precisely this. Decoding it is harder than decoding hand movement, because speech involves rapid, coordinated movements of the lips, tongue, jaw and larynx, and because the space of possible utterances is enormous. The first convincing demonstration came in 2021 from Edward Chang's laboratory at the University of California, San Francisco, published in the New England Journal of Medicine. A man who had lost the ability to speak after a brainstem stroke more than fifteen years earlier had an ECoG array placed over the speech motor cortex. When he attempted to say words, the system decoded them from a vocabulary of fifty words and assembled them into sentences at around fifteen words per minute. The error rate was high by the standards of everyday speech, but the result showed that intended speech could be decoded from the cortex of someone who had been unable to speak for years. Two papers published in Nature in August 2023 moved the field from proof of concept to something approaching usefulness. At Stanford, Willett and colleagues worked with Pat Bennett, a woman with ALS who could still make some sounds but could not speak intelligibly. Using four Utah arrays implanted in speech-related areas of her cortex, the system decoded attempted speech at sixty-two words per minute. With a fifty-word vocabulary, the word error rate was about nine percent; with a vocabulary of about 125,000 words, which approximates open conversation, it was about twenty-four percent. At UCSF, Chang's team worked with Ann Johnson, who had been left unable to speak by a brainstem stroke in 2005. A high-density ECoG array of 253 electrodes allowed decoding at seventy-eight words per minute, with a synthesised voice personalised using a recording of her speaking at her wedding, and a digital avatar whose face moved in time with the decoded speech. In 2024 a team at the University of California, Davis, led by Sergey Stavisky and David Brandman, reported in the New England Journal of Medicine on Casey Harrell, a man in his forties with ALS whose speech had become very difficult to understand. Four arrays were implanted in his left precentral gyrus. On the first day of use, after about thirty minutes of training, the system decoded his attempted speech from a fifty-word vocabulary with more than ninety-nine percent accuracy. Over the following months, with a vocabulary of 125,000 words, it reached accuracy of around ninety-seven percent, and he used it for hours each day to talk with his family and colleagues, in a voice synthesised from recordings made before his illness. The same team published a further advance in Nature in 2025. Rather than decoding words and then speaking them, the new system translated neural activity directly into sound, producing synthesised voice with a delay of about a fortieth of a second, comparable to the delay with which people hear their own voices. This let Harrell interrupt, emphasise words, change intonation to ask a question, and even sing short, simple melodies. It also allowed him to produce sounds that were not words in any dictionary, such as interjections. Listeners could understand a majority of the words the system produced, against very few when they listened to his natural speech. The distinction matters: a text-based decoder, however accurate, turns conversation into a sequence of composed messages, whereas real-time voice restores something closer to the rhythm of talking. Table 3 sets out some of these milestones, drawing on the published papers, to show how quickly performance has advanced. Table 3. Selected milestones in decoding communication from the brains of paralysed people. Year and team Participant Recording method Approach Reported performance 2021, Stanford (Willett et al.) Man with spinal cord injury Two Utah arrays Imagined handwriting About 90 characters per minute, over 99% accuracy with autocorrect 2021, UCSF (Moses et al.) Man with brainstem stroke ECoG Attempted speech, 50 words About 15 words per minute 2023, Stanford (Willett et al.) Pat Bennett, ALS Four Utah arrays Attempted speech 62 words per minute; about 24% error, large vocabulary 2023, UCSF (Metzger et al.) Ann Johnson, brainstem stroke 253-electrode ECoG Speech, voice and avatar 78 words per minute 2024, UC Davis (Card et al.) Casey Harrell, ALS Four Utah arrays Attempted speech About 97% accuracy, 125,000-word vocabulary 2025, UC Davis (Wairagkar et al.) Casey Harrell, ALS Four Utah arrays Direct voice synthesis Delay of about 25 milliseconds Source: the peer-reviewed papers listed in the Notes for this chapter. What the decoders are actually doing It is worth being clear about what these systems do and do not read. They do not read thoughts in any general sense. They record from the motor areas of the cortex that would control the muscles of the hand or the vocal tract, and they decode the person's attempt to move those muscles. The participant has to try to speak or write; idle thoughts do not appear on the screen. Most of the speech systems decode sequences of phonemes, the basic sound units of language, and then use a language model to assemble the most likely words and sentences. The language model is an important part of their accuracy, which raises the same question as in the previous chapter: when the system's knowledge of probable sentences corrects a noisy neural signal, how much of the output is the person and how much is the model? The researchers are well aware of this, and the move to direct voice synthesis is partly a move towards outputs that are shaped less by a language model's expectations. Recent research has also complicated the distinction between attempted and imagined speech. A study published in 2025 by a Stanford-led team found that inner speech, words a person imagines saying without attempting to move, could be decoded to some extent from the same motor areas, though less accurately than attempted speech. The researchers explored safeguards, including a system that would decode only after the user imagined a specific password phrase. This work has obvious benefits for people who find attempting to speak tiring, and obvious implications for mental privacy, which the next chapter takes up. Decoders also face a practical problem: neural signals change from day to day, as electrodes shift slightly and the brain adapts. Early systems needed frequent recalibration, often daily, with the participant repeating training tasks. Newer methods adapt continuously, learning from the user's ordinary use, and the UC Davis system was notable for maintaining performance with little recalibration. For a system to be used at home without a research team, this kind of stability is essential. The non-invasive route Given the risks of brain surgery, it is natural to ask whether similar results could be achieved without it. Research on non-invasive decoding has also advanced, and it is worth understanding where it stands. EEG-based spelling systems have been used by people with ALS for many years. The most established design, known as the P300 speller, flashes rows and columns of letters on a screen; when the letter the user is attending to flashes, the brain produces a characteristic electrical response about three hundred milliseconds later, which the system detects. By accumulating responses over several flashes, the system works out which letter the user wants. It is reliable for some users and needs no surgery, but it is slow, typically a few characters per minute, demands sustained visual attention, and is tiring. For people in a completely locked-in state who have lost even eye movement, even this may not work. More ambitious non-invasive work has used functional magnetic resonance imaging, which measures changes in blood flow across the whole brain. In 2023 a team at the University of Texas at Austin, led by Alexander Huth, reported in Nature Neuroscience a decoder that could reconstruct the gist of stories that participants listened to or imagined telling, from fMRI recordings, using a language model to generate candidate word sequences and choose those that best matched the recorded activity. The result attracted wide attention, partly because of its implications for mental privacy. But its practical limits are severe: fMRI scanners are enormous and expensive, the participant must lie still inside one, the decoder required many hours of individual training data, and it captured the general meaning of language rather than exact words. The researchers also showed that the decoder did not work without the participant's cooperation, since a person could defeat it by thinking about something else. Other groups, including researchers at Meta, have explored decoding typed or intended sentences from magnetoencephalography, which measures the brain's magnetic fields with large, shielded instruments, and from EEG. These studies have shown progress in the laboratory, but none yet approaches the speed and accuracy of implanted systems in a form that a person could use at home. The honest summary is that, for the foreseeable future, there is a trade-off that cannot be engineered away. The closer the sensors are to the neurons, the richer the information and the better the performance; the further away, the safer and more accessible the system. People with severe paralysis, their families and their clinicians will have to weigh that trade-off for themselves, and their choices will differ. What the demonstrations do not show These results are extraordinary, and the people who took part in them deserve recognition as pioneers who accepted brain surgery and years of demanding sessions largely for the benefit of others. But it is important to understand their limits. The numbers of participants are tiny: often one, sometimes a handful. Results from one person's brain, with one placement of electrodes, may not generalise. The systems have operated largely under the supervision of research teams, with researchers present or on call, using equipment connected through ports in the skull. Most of the intracortical studies used devices with wires passing through the scalp, which carry infection risk and are not suitable for long-term life at home. And the published papers report performance during carefully designed sessions, which may differ from performance during the unstructured, tiring, interrupted business of daily life. None of this diminishes the science. It defines the next problem. Turning a laboratory demonstration into a medical device that can be implanted safely by ordinary neurosurgeons, used without technical support, maintained for decades and paid for by health systems is a separate challenge, and in some ways a harder one. It is the challenge that a new generation of companies has now taken on, and the subject of the next chapter. Chapter 8: Brain Interfaces Leave the Laboratory For most of their history, implanted brain-computer interfaces were academic research tools. The devices were made by a small number of specialist manufacturers, used under research protocols with one or two participants at a time, and connected to racks of computers through cables plugged into ports on the participant's head. That has changed. By the mid-2020s several well-funded companies were running human trials of devices designed from the outset to be medical products: fully implanted or minimally invasive, wireless, and meant to be used at home without a scientist in the room. The arrival of venture capital, celebrity founders and regulatory programmes for breakthrough devices has transformed the field's pace and visibility. It has also raised, in sharper form, questions that the academic work could defer: how safe is it, how long will it last, who will maintain it, who owns the data, and who will pay? Five approaches to the same problem The companies differ in their engineering philosophies, and the differences reflect the trade-offs described in the previous chapter between invasiveness and signal quality. Table 4 summarises the main approaches being pursued in human trials, using publicly reported specifications. Table 4. Main implanted brain-computer interface approaches in human studies by 2026. Device and developer Placement Surgery Channels Status by 2026 Utah array (Blackrock Neurotech) Penetrates cortex Open craniotomy About 96 per array Workhorse of academic research for two decades Stentrode (Synchron) Inside a blood vessel next to motor cortex Catheter through jugular vein 16 Early feasibility complete; pivotal trial planned N1 (Neuralink) Flexible threads in cortex Robot-assisted, small craniotomy About 1,024 Feasibility studies in several countries Layer 7 (Precision Neuroscience) Thin film on cortical surface Narrow slit in skull 1,024 per module Cleared for temporary use up to 30 days Connexus (Paradromics) Penetrates cortex Craniotomy About 420 First patients in FDA-approved study from 2026 Source: company announcements, regulatory filings and published papers. Synchron, founded in Australia and later based in New York, takes the least invasive route of the implanted devices. Its Stentrode is a mesh stent carrying electrodes, delivered by catheter through the jugular vein into a large blood vessel that runs alongside the motor cortex, much as cardiologists place stents in the heart. No opening in the skull is required. The trade-off is signal quality: recording through the wall of a blood vessel yields far less detailed information than electrodes in the cortex, so the Stentrode supports a small number of reliable commands rather than high-speed decoding. Its users, people with ALS and other forms of severe paralysis, have used it to send messages, shop online and manage finances, typically selecting items with the help of eye tracking and using the brain signal as a click. After early studies in Australia, Synchron implanted its first patient in the United States in 2022, and reported in 2024 that its early feasibility study had met its primary safety endpoint. In 2025 Apple announced a new protocol for brain-computer interfaces as input devices within its accessibility framework, and Synchron demonstrated a participant controlling an iPad directly through the device. Late that year the company raised substantial new funding to prepare a larger pivotal trial, the kind needed to seek full market approval. Neuralink, founded by Elon Musk, took the opposite approach, aiming for very high channel counts and full implantation. Its device, about the size of a large coin, sits in an opening in the skull, flush with the bone, and connects to 64 flexible polymer threads carrying about a thousand electrodes, which a purpose-built surgical robot inserts into the cortex while avoiding blood vessels. The device is charged wirelessly and transmits data by radio. In January 2024 the company implanted its first participant, Noland Arbaugh, who had been paralysed from the shoulders down in a diving accident. He became a vivid advocate, describing playing chess and the strategy game Civilization late into the night by moving a cursor with his thoughts. Within weeks, however, many of the threads had retracted from his brain, and the device's performance fell sharply. The company recovered much of the lost performance by changing its decoding software, and adjusted its procedure for later participants. By early 2026, according to reports, Neuralink had implanted around twenty people across studies in the United States, Canada, the United Kingdom and elsewhere, and had announced plans for speech decoding and for an implant intended to stimulate the visual cortex of blind people. The company has been criticised by scientists for releasing results mainly through social media and live-streamed presentations rather than peer-reviewed publications, which makes independent assessment difficult. Precision Neuroscience, co-founded by a neurosurgeon who had earlier been a co-founder of Neuralink, uses an ultra-thin flexible film of electrodes that rests on the brain's surface, inserted through a narrow slit in the skull rather than a full craniotomy. In 2025 it received clearance from the United States Food and Drug Administration to use its electrode array for up to thirty days, a limited approval aimed at uses such as mapping brain function during surgery, but one that gave it a foothold in clinical practice. Paradromics, based in Texas, has built a high-channel-count device with penetrating microwires, designed primarily for speech restoration. It received FDA approval in late 2025 for a clinical study, and the first implantation in that study took place at the University of Michigan in 2026. Blackrock Neurotech, the manufacturer of the Utah array, continues to supply most academic research, and Chinese research institutes and companies have begun their own human trials, with government support. Durability, maintenance and the obsolescence problem The most sobering case in the history of neural implants is not a brain-computer interface but a visual one. Second Sight Medical Products developed the Argus II, a retinal implant that gave some people blinded by retinitis pigmentosa a rudimentary form of artificial vision: they could perceive light, edges and large shapes. It received approval in Europe in 2011 and in the United States in 2013, and a few hundred people had it implanted. Then the company ran into financial difficulties, stopped producing the device, and largely ceased supporting it. A 2022 investigation by IEEE Spectrum described implant recipients who found themselves with obsolete hardware in their bodies, unable to get repairs or replacement parts, and in some cases told by doctors that their devices might complicate medical scans or would need surgery to remove. The company later merged with another firm and shifted its focus. For the people who had trusted it with their eyes, there was no clear path forward. The Argus II story has become a cautionary tale throughout the neurotechnology field, and it should be. A brain implant is not a phone that can be replaced when the manufacturer stops issuing updates. It is in the person's head. Its external components, software and decoders require maintenance for as long as the person lives, which might be decades. If the company that made it is acquired, changes strategy or goes bankrupt, the person may lose not only a capability on which they have built their life but also access to the expertise needed to manage the hardware safely. Several remedies have been proposed. Regulators could require companies to set aside funds or make arrangements for long-term support before approval. Device specifications, software and decoders could be placed in escrow, to be released to a third party if the company fails. Industry groups could agree on common standards so that implanted hardware could be supported by other manufacturers' systems. None of these is yet standard practice. Meanwhile participants in trials face a further difficulty: when a study ends, what happens to their implant and their access to the system? Some early BCI participants have had devices removed, sometimes against their wishes, when funding or study protocols ended. Ethicists have argued that researchers owe trial participants post-trial obligations, including continued access where the device has become part of their lives, but the details remain unsettled. The physical durability of the hardware is itself an open question. Utah arrays have functioned in some people for many years, but with gradual decline in the number of electrodes recording useful signals. The newer flexible devices are designed to cause less tissue reaction, but there is not yet long-term evidence about how they will perform over a decade or more. The ideal brain implant would last a lifetime or be easily replaceable. None of the current devices has been shown to meet that standard. What potential users say they want The people who stand to benefit from brain-computer interfaces have views about them, and researchers have asked. Surveys of people with spinal cord injury, such as a widely cited study by Kim Anderson published in 2004, found that for people with tetraplegia, regaining arm and hand function was the highest priority, above walking. Surveys of people with ALS, including one led by Jane Huggins and published in 2011, explored what potential users would want from a BCI and found that many were interested, but that they had clear expectations about accuracy and speed below which a system would not be worth the effort, and that attitudes to surgery varied widely. Later research with potential users has reinforced a consistent message: they want systems that work reliably every day, that can be set up without expert help, that do not require long calibration sessions, and that allow them to do the things they value, such as talking with family, working, and controlling their environment, rather than impressive tasks chosen by researchers. The trial participants themselves occupy an unusual position. Many are people with progressive or catastrophic conditions who have been told that no treatment will restore what they have lost. They may have strong reasons to volunteer, including hope of personal benefit, a wish to contribute to science, and a wish to make meaning out of their situation. Ethicists have long worried about the "therapeutic misconception," in which research participants believe that a study is designed primarily to benefit them, when its main aim is to generate knowledge. In BCI research, the misconception is complicated by the fact that participants often do benefit, sometimes dramatically, during the study. Informed consent in these circumstances must be unusually thorough about the risks, the uncertainty, the time commitment, and especially about what will happen when the study ends. Participants have also become public figures in a way that earlier generations of research volunteers did not. Several have spoken at length to journalists, posted on social media, and appeared in company presentations. This gives them a voice in how the technology is portrayed, which is welcome. It also places them in the middle of commercial and reputational interests, as the human face of companies seeking investors and regulatory approval. Neural data and mental privacy Brain-computer interfaces generate a type of data that has no precedent: continuous recordings of a person's neural activity. Today's devices record from motor areas and are used to decode intended movements and speech, which is far from reading thoughts in the popular sense. But as the inner speech research described in the previous chapter shows, the boundary between what a person attempts and what they merely think is not perfectly sharp, and neural data may reveal information about a person's health, emotional states, attention and cognitive decline that they did not intend to share. As consumer neurotechnology spreads, including headbands and earbuds with EEG sensors sold for meditation, sleep and focus, the volume of neural data held by companies is growing well beyond medical contexts. Law has begun to respond. In 2021 Chile amended its constitution to protect mental integrity and to require that the law regulate neurotechnologies, the first country to do so, and in 2023 its Supreme Court ordered a consumer neurotechnology company to delete a user's brain data. In 2024 the American states of Colorado and California amended their privacy laws to classify neural data as sensitive personal information, subject to stronger protections, and other states followed. In 2025 UNESCO's member states adopted a recommendation on the ethics of neurotechnology. These measures are early and uneven, and they mostly address consumer devices rather than medical implants, which are governed by health privacy rules. For BCI users, the most practical privacy questions are concrete. Who can access the recordings from my implant? Are they used to train the company's next product? Can my insurer or employer obtain them? If I decide to leave the study or stop using the device, can I have my data deleted? Can a decoder be prevented from decoding things I did not intend to communicate? Safeguards such as the password phrase explored in inner-speech research, or decoders that operate only when explicitly activated by the user, are design choices that put control in the user's hands, and they should become standard. Who gets an implant, and who pays? Even if every technical, safety and privacy problem were solved, a question would remain about access. Implanted brain-computer interfaces require neurosurgery, specialist follow-up, technical support and expensive hardware. The populations who stand to benefit most, people with ALS, high-level spinal cord injury and brainstem stroke, are relatively small, and many are poor, because severe disability tends to reduce income. Whether the devices become available beyond trials depends on whether national health systems and insurers will pay for them, and at what price. The history of other neural implants is mixed. Cochlear implants, which stimulate the auditory nerve to provide a sense of hearing, are now routinely funded in many wealthy countries, after decades of evidence. Deep brain stimulation for Parkinson's disease is covered for appropriate patients in many health systems. Both took many years to move from experimental to routine. Brain-computer interfaces will need evidence not only of safety and performance but of benefit in terms that payers recognise, such as quality of life, reduced need for care, and the ability to work. Measuring those outcomes, rather than words per minute in a laboratory, is the work of the coming decade. There is also a question about which uses deserve priority. Some technology leaders have described brain-computer interfaces as the first step towards enhancing healthy human cognition, or towards a symbiosis between humans and artificial intelligence. Whatever the merits of those visions, disabled people and their advocates have pointed out that they risk distracting attention and investment from the people for whom BCIs are a medical necessity now. A device designed for medical users can be optimised for reliability, durability and supportability. A device designed as a stepping stone to a consumer product may be optimised for other things. Many of these concerns recall the abandonment research from Chapter 1. The most impressive brain-computer interface is useless if it requires a team of engineers to operate, decays after three years, or leaves its user stranded when the company changes direction. The measure of success is not the record speed at a conference but whether, ten years from now, people are living with these devices at home, using them for the things they care about, and confident that someone will still support them. The technologies of the last two chapters serve people with the most severe paralysis. The next chapter turns to a much larger and less publicised group: people whose disabilities affect how they use their hands, process information, read, focus or plan, and for whom some of the most useful applications of AI involve no implant at all. Chapter 9: Hands, Minds and Attention Public discussion of AI for accessibility gravitates towards the dramatic: the blind person who "sees," the paralysed person who types by thought. Yet the largest groups of disabled people are not blind, Deaf or locked in. They are people with arthritis, tremor, repetitive strain injury, cerebral palsy, multiple sclerosis or muscular dystrophy who find keyboards, mice and touchscreens difficult; people with dyslexia and other learning disabilities; people with intellectual disabilities; autistic people and people with attention deficit hyperactivity disorder; people living with the effects of brain injury, stroke or dementia. Many have disabilities that are invisible to others. Some of the most consequential AI accessibility tools for them involve no specialised hardware at all, and some of the most important questions concern not whether the tools work but whose idea of normal they are built around. Control without hands For people with motor impairments, the basic problem is input: how to tell a computer or phone what to do when using one's hands is slow, painful, tiring or impossible. Traditional solutions include alternative keyboards, trackballs, head-mounted pointers, mouth sticks, switches operated by any reliable movement, and scanning interfaces in which the computer highlights options in sequence until the user activates a switch. All of these work, and many people rely on them, but they are often slow and fatiguing. Artificial intelligence has made two alternatives far more capable: voice control and camera-based control. Voice control, which depends on the speech recognition described in Chapter 4, lets users operate an entire device by speaking commands, dictating text, and saying the names or numbers of items on the screen in order to select them. Apple added a comprehensive Voice Control feature to its devices in 2019, and Microsoft added a similar Voice Access feature to Windows in 2022. For a person with a spinal cord injury or severe arthritis, being able to use mainstream devices fully by voice, without specialised software, is a large change. The limitation, as Chapter 6 discussed, is that voice control works only as well as recognition of the user's speech, and many people with motor impairments also have speech affected by the same condition. Camera-based control uses computer vision to track the user's head, face or eyes. Dedicated eye-tracking systems, which use infrared cameras to determine where a person is looking, have long been used by people with ALS and other severe motor impairments to type, communicate and control computers, but they cost thousands of dollars. Machine learning has made it possible to approximate eye and head tracking with the ordinary front-facing camera of a phone, tablet or laptop. In 2024 Apple introduced an eye-tracking feature on its phones and tablets that used the front camera and on-device machine learning, allowing users to navigate by looking at items and dwelling on them to select. Google released Project Gameface in 2023, an open-source tool developed with Lance Carr, a gamer with muscular dystrophy, which lets users move a cursor with head movements and trigger actions with facial gestures such as raising the eyebrows or opening the mouth, and later brought the approach to Android. Google's earlier Look to Speak app, released in 2020, lets people select pre-written phrases by looking left, right or up, and speaks them aloud. Camera-based tracking with ordinary hardware is generally less precise than dedicated eye trackers, and it can struggle in poor light or with involuntary movements. But its cost is close to zero for anyone who already owns a recent device, which matters a great deal for people in countries and circumstances where a dedicated eye tracker is out of reach. Releasing tools as open source also allows other developers to adapt them, which partly answers the vulnerability, discussed in earlier chapters, of depending on a single company's product decisions. Reading, writing and dyslexia Dyslexia, a specific difficulty with reading and spelling that is not explained by general intelligence or education, is among the most common learning disabilities. Estimates of its prevalence vary with the definition used, but it affects a significant minority of people in every language studied. Assistive technology for dyslexia predates modern AI: text-to-speech to listen to written material, speech-to-text to write without spelling, spell checkers that cope with phonetic misspellings, and reading tools that adjust spacing, fonts and colours. AI has improved each of these. Neural text-to-speech voices are far more pleasant to listen to for long periods than older synthetic voices. Speech recognition is accurate enough for extended dictation. Spelling and grammar correction based on language models can work out what a writer meant from severely misspelled text, where older spell checkers offered unhelpful suggestions. Reading tools such as Microsoft's Immersive Reader, built into its products, combine these functions with features such as highlighting each word as it is read aloud and breaking words into syllables. Large language models add a new dimension. They can summarise long documents, explain difficult passages in simpler language, rewrite a rough draft into clear prose, and answer questions about a text so that a reader can check their understanding. For a student with dyslexia facing a two-hundred-page reading list, or an employee with dyslexia facing a long policy document, these capabilities can make the difference between keeping up and falling behind. Many people with dyslexia report that language models have changed their working lives. The tensions are different from those in other areas, and they are less about reliability than about legitimacy. In education, the same capabilities that make AI an assistive technology for a dyslexic student make it a potential means of academic dishonesty for anyone. Schools and universities that ban or restrict generative AI to protect the integrity of assessment may inadvertently remove a tool that functions as a reasonable adjustment for disabled students, and students may face suspicion for using tools they need. The emerging answer in many institutions is to treat AI use as something that can be formally agreed as an accommodation, but this places a familiar burden on disabled students to disclose their disability, obtain documentation and negotiate with institutions in order to use what is, for others, a forbidden shortcut. Plain language and intellectual disability People with intellectual disabilities are among the most excluded from information. Official letters, medical information, legal documents, consent forms, news and the terms of service of almost every digital product are written in language that many of them cannot understand. The disability movement has long promoted "easy read" formats: short sentences, common words, one idea per sentence, supporting images, and, crucially, testing with people with intellectual disabilities to check that the result is actually understood. Organisations such as Inclusion Europe have published standards for easy-to-read information. Language models are obviously capable of rewriting complex text in simpler words, and they can do so instantly and cheaply, which opens the possibility of making vast amounts of information accessible that would never be manually converted. Researchers and disability organisations have begun to explore this, and some public bodies have experimented with automated plain-language versions of documents. Two problems arise. The first is accuracy. Simplification involves choosing what to leave out, and a model may omit exactly the qualification that matters, for instance the condition under which a benefit is paid, the side effect that requires urgent attention, or the deadline for an appeal. A simplified text that is wrong is worse than a complex text that is right, especially for readers who may not be able to check it against the original. The second is that easy read has never been only about simple words. Its standards insist on testing by people with intellectual disabilities, because experts and writers are poor judges of what is easy to understand for someone else. Automated simplification without that testing risks producing text that is simpler in the eyes of its producers but not in the experience of its readers. The defensible role for AI is as a drafting tool that speeds up the work of human easy-read writers and testers, not as a replacement for them. Neurodivergence and the question of normal Autistic people and people with ADHD are often described collectively, along with others, as neurodivergent, a term that reflects the neurodiversity movement's view that these are natural variations in human cognition rather than diseases to be cured. Many neurodivergent people face difficulties with executive function, the cluster of skills involved in planning, starting tasks, organising time, switching between activities and managing working memory, and with the unwritten rules of social communication in workplaces and institutions designed around neurotypical expectations. Some of the most enthusiastically adopted AI tools among neurodivergent people are modest. One widely cited example is Goblin Tools, a free set of small utilities created by a Belgian developer, Bram De Buyser, which uses a language model to break a daunting task into smaller steps, estimate how long things might take, rewrite a message to sound more or less formal, or suggest how a message might be interpreted by its recipient. Its popularity spread through online neurodivergent communities by word of mouth. General-purpose chatbots are used in similar ways: to draft emails that a person has been avoiding, to plan a day, to rehearse a difficult conversation, to decode an ambiguous message from a manager. What these uses have in common is that the person chooses the tool, uses it for their own goals, and decides what to do with its output. A very different strand of technology development has targeted autistic people with the aim of making them behave more like non-autistic people: apps and devices that read facial expressions and tell the user what emotion another person is feeling, or that train autistic children to make eye contact and suppress behaviours such as hand-flapping. Many autistic adults and researchers have criticised these approaches. They point out that automatic emotion recognition from facial expressions rests on weak scientific foundations, since facial movements do not map reliably onto internal emotional states across people and cultures. They draw on the idea, developed by the autistic scholar Damian Milton, of a "double empathy problem": communication difficulties between autistic and non-autistic people run in both directions, so that framing them as a deficit solely of the autistic person is mistaken. And they argue that training autistic people to mask their natural behaviour is associated with exhaustion and poor mental health. The contrast between these two strands is the contrast between the social and medical models discussed in Chapter 1, played out in software. A tool that helps a person do what they want, on their own terms, is accessibility. A tool designed by others to make a person appear more normal, whether or not they want to, is something else, and calling it accessibility does not make it so. AI as a workplace adjustment Many of the tools described in this chapter are general-purpose AI assistants, the same products that workplaces are adopting for productivity. That overlap creates an opportunity and a risk for disabled employees. The opportunity is that tools which function as reasonable adjustments, such as transcription of meetings, summarisation of long documents, drafting help, task breakdown and reminders, may already be available on the employee's work computer, without the need to request specialist software through a lengthy procurement process. For an employee with dyslexia, ADHD, a brain injury or a chronic illness that causes fatigue and difficulty concentrating, this can remove barriers without requiring disclosure, which many disabled people prefer to avoid because of fear of stigma. Several governments and large employers that trialled AI assistants among their staff have reported that some of the most enthusiastic users were employees who identified as neurodivergent or disabled, though such trials are rarely designed as rigorous evaluations. The risk is that the existence of general AI tools is used to deny more specific accommodations. An employer faced with a request for a human note-taker, a sign language interpreter, a quieter workspace, flexible hours or a specialist piece of software might respond that the employee can use the AI assistant instead. Sometimes that will be a good solution. Sometimes it will not, because the AI tool is unreliable for the employee's particular needs, or because the barrier is not one that software can address. Organisations such as the Job Accommodation Network in the United States emphasise that accommodations should be determined through an interactive process with the employee, and that principle applies with full force to AI. The question is not whether an AI tool could in theory help, but whether it actually removes the barrier that this person faces, as judged by the person. There is a further subtlety. If AI assistants are monitored by employers, as many workplace tools are, then an employee using one as a disability adjustment may be revealing information about their disability through their usage patterns, for instance by frequently asking for summaries or for help drafting routine messages. Policies on workplace AI rarely consider this. An employee's use of an accessibility tool should be as private as their medical information, and that principle needs to be designed into how workplace AI is deployed. Memory, cognition and the risk of surveillance Brain injury, stroke and dementia can affect memory, attention, orientation and the ability to manage daily tasks. AI-enabled tools offer reminders, prompts, step-by-step guidance through tasks such as cooking, conversational support, and the ability for family members or carers to monitor safety remotely. For a person with early dementia who wishes to continue living independently, a system that reminds them to take medication, helps them find their way home and alerts a family member if something goes wrong can extend independence by years. The risk is that support becomes surveillance. Systems that monitor movement around the home, track location, analyse speech for signs of decline or flag unusual behaviour to carers can protect people, but they can also strip them of privacy and autonomy, often without meaningful consent, since cognitive impairment complicates consent itself. Much of the market for such systems is driven by families, care providers and insurers rather than by the people being monitored. The question to ask of any such system is the same as for every tool in this book: who is it for, who controls it, and what happens when the user's wishes conflict with those of the people paying for it? Across all of these areas, a pattern emerges. The most valued tools tend to be general-purpose, cheap, chosen by the user and used on the user's terms. The most contested tend to be designed by others, aimed at making the disabled person conform, or deployed by institutions in ways the person cannot refuse. That pattern points to the problems that the final chapter takes up: problems that no single product can solve, because they lie in the data systems are built from, the incentives of those who build and buy them, and the rules that govern them. Chapter 10: Data, Law and Who Decides The chapters so far have moved through specific disabilities and specific technologies. Certain problems have kept reappearing in all of them: systems trained on data that underrepresents disabled people, tools deployed by institutions to save money rather than to serve users, companies that discontinue products on which people depend, and the persistent question of who gets to decide what disabled people need. These problems cannot be solved by better models alone. They are structural, and they are shaped by data practices, market incentives and law. This chapter examines them directly. Disabled people as outliers The Canadian inclusive design researcher Jutta Treviranus, who founded the Inclusive Design Research Centre at OCAD University in Toronto, has offered one of the clearest explanations of why machine learning tends to work poorly for disabled people. Machine learning systems learn patterns from data and optimise their performance across the data as a whole. People who are statistically typical are well represented, and the system learns to serve them well. People who differ from the majority, in their speech, movement, appearance, behaviour or circumstances, appear in the data rarely, and each of them differs from the majority in a different way. The system treats them as noise, or performs poorly on them because it has never learned their patterns. Disability is, in this sense, the ultimate long tail. There is no single "disabled" pattern to learn. A person with cerebral palsy, a person with a facial difference, a wheelchair user, a person who uses a cane, an autistic person, a person with a stammer: each is an outlier in a different direction. Even within a single condition the variation is enormous. Gathering enough data to represent every kind of difference is impossible in principle, which is why Treviranus and others argue that fairness for disabled people cannot be achieved simply by adding more data on disabled people to existing systems. It requires designing systems that cope gracefully with the unexpected, that let users correct and personalise them, and that do not make high-stakes decisions about individuals on the basis of statistical resemblance to a norm. This is not an abstract concern. Earlier chapters described speech recognition systems that fail for people with dysarthria, vision models that struggle with photographs taken by blind people, and sign language datasets dominated by a few groups of signers. Similar failures appear in systems that were never intended for disabled people but that affect them anyway. Pedestrian detection systems in automated vehicles may be less reliable at recognising wheelchair users or people with unusual gaits if they were trained mostly on typical pedestrians. Face recognition used for identity verification may fail for people with facial differences or conditions affecting facial movement, locking them out of services. Proctoring software that monitors students during online examinations may flag disabled students as suspicious because their eye movements, body movements or use of assistive technology deviate from the norm. When AI makes decisions about disabled people The tools in this book are mostly ones that disabled people choose to use. A growing share of AI systems, however, make or support decisions about disabled people, and these raise different concerns. Employment is one example. Automated tools for screening job applicants, analysing video interviews, administering game-based personality assessments or ranking candidates may disadvantage disabled applicants in ways that are hard to detect: a speech pattern, a facial expression, a gap in employment history, slow responses on a timed test or an unusual answer style may all lower a candidate's score for reasons connected to their disability rather than their ability to do the job. In 2022 the United States Equal Employment Opportunity Commission and Department of Justice issued guidance warning employers that the use of such tools could violate the Americans with Disabilities Act, for instance if a tool screened out disabled applicants or if employers failed to provide reasonable accommodations in an automated process. One prominent video interviewing company had already dropped facial analysis from its assessments in 2021 after sustained criticism. Public services are another. Governments increasingly use algorithms to detect fraud, allocate benefits, assess eligibility for care and prioritise cases. Disabled people are heavy users of these services and particularly vulnerable to their errors. In the United Kingdom, for example, reporting in 2024 based on internal documents described a fairness analysis of a machine learning system used to flag universal credit advance claims for fraud investigation, which had found disparities in how the system treated people according to characteristics including age, disability and nationality. In the United States, the replacement of human assessments with algorithmic tools for allocating home care hours has, in several states, led to sudden cuts in disabled people's care and to legal challenges. When a person whose daily life depends on care hours finds them cut by a system whose logic they cannot see or challenge, the problem is not only accuracy but due process. When AI itself is inaccessible A less discussed problem is that AI products themselves are often inaccessible. Chat interfaces that stream text word by word can overwhelm screen readers, which may announce every fragment or nothing at all. Image generation tools that produce pictures without descriptions exclude blind users from a whole category of creative work. Voice interfaces that require rapid, fluent speech exclude people with speech disabilities. Systems that verify that a user is human, such as visual puzzles that ask the user to identify traffic lights or distorted text, have long been a notorious barrier for blind people and people with cognitive disabilities; the World Wide Web Consortium has published guidance documenting the problem, and audio alternatives are often hard to use as well. As more services place AI agents and verification steps in front of customers, these barriers multiply. The irony is sharp: technologies that are promoted as transforming accessibility are frequently released without basic accessibility testing. It reflects the broader pattern in which accessibility is treated as a feature to be added later rather than a requirement from the start. Disabled people's organisations have argued that AI developers should be held to the same accessibility standards as any other software provider, and the European rules described below begin to do this. Evaluation is a related gap. AI systems are typically judged on benchmarks, standardised tests that compare one model against another. Very few widely used benchmarks measure performance for disabled users, and those that exist, such as the VizWiz dataset for visual questions from blind people or the datasets of atypical speech described in Chapter 6, are small compared with mainstream benchmarks. What is not measured is not optimised. A model that improves its score on standard benchmarks may get worse for disabled users without anyone noticing, until users report it. Building and maintaining disability-relevant benchmarks, with the involvement of disabled people, is unglamorous infrastructure work, and it is among the most effective ways to make the industry pay attention. The overlay problem Perhaps the clearest example of AI used to evade rather than deliver accessibility concerns websites. Most of the web is inaccessible in ways that are well understood: images without alternative text, buttons without labels, forms that cannot be completed with a keyboard, poor colour contrast, content that does not work with screen readers. Fixing these problems properly requires changes to a website's code and content, guided by the Web Content Accessibility Guidelines, the international standard maintained by the World Wide Web Consortium. From around 2019, a number of companies began marketing "accessibility overlays": a single line of code that, once added to a website, would supposedly use artificial intelligence to detect and fix accessibility problems automatically, making the site compliant with standards and protecting its owner from lawsuits. The pitch was attractive to businesses facing legal pressure, and overlays were installed on many thousands of sites. Blind users and accessibility professionals reported a different experience. The overlays often failed to fix the underlying problems, sometimes interfered with the screen readers that blind people already used, and in some cases made sites harder to use. In 2021 a public statement called the Overlay Fact Sheet, signed by hundreds of accessibility practitioners, many of them disabled, explained why overlays could not deliver what they promised and urged organisations not to use them. The National Federation of the Blind, one of the largest organisations of blind people in the United States, publicly criticised one of the most prominent vendors. Websites using overlays continued to be sued for inaccessibility. In January 2025 the United States Federal Trade Commission announced an order requiring accessiBe, one of the best-known overlay vendors, to pay one million dollars to settle allegations that it had misrepresented its AI-powered product's ability to make any website compliant with accessibility guidelines, and that it had presented paid reviews as if they were independent. The case was a rare instance of a regulator directly addressing exaggerated claims about AI accessibility. Its broader lesson applies well beyond websites: when a product promises to make accessibility automatic, cheap and effortless, and the people who need accessibility say that it does not work, the people who need accessibility are usually right. The law Law has been the main engine of accessibility for decades, and it is increasingly being applied to AI. Table 5 sets out several of the frameworks that matter most, though the details vary widely across countries and the list is far from complete. Table 5. Selected legal frameworks relevant to AI and accessibility. Framework Jurisdiction Year Relevance to AI and accessibility Convention on the Rights of Persons with Disabilities United Nations, ratified by most states 2006 Duty to ensure accessible information and technology, and to promote universally designed technology Americans with Disabilities Act United States 1990 Applied to websites, apps and automated employment tools through regulation, guidance and courts Department of Justice web accessibility rule United States, state and local government 2024 Sets a specific technical standard for public bodies' websites and apps European Accessibility Act European Union Applied from 2025 Accessibility requirements for many products and services, including e-commerce, banking and e-books Artificial Intelligence Act European Union 2024, phased in Bans AI exploiting vulnerabilities linked to disability; providers of high-risk systems must meet accessibility requirements The United Nations Convention on the Rights of Persons with Disabilities, adopted in 2006, is the foundational international instrument. It commits states to ensure that disabled people have access to information and communications technologies on an equal basis with others, and to promote the research, development and availability of new technologies, giving priority to affordable ones. It was negotiated with unprecedented participation from disabled people's organisations, and it embodies the principle of "nothing about us without us." In the United States, the Americans with Disabilities Act of 1990 predates the web but has been applied to it through a long series of court cases and settlements. In 2024 the Department of Justice adopted a rule setting a specific technical standard, based on the Web Content Accessibility Guidelines, for the websites and mobile apps of state and local governments. In the European Union, the European Accessibility Act began to apply in June 2025, requiring a wide range of products and services, including computers, smartphones, e-commerce, banking services, e-books and transport ticketing, to meet accessibility requirements. The EU's Artificial Intelligence Act, adopted in 2024 and coming into force in stages, prohibits AI practices that exploit the vulnerabilities of people due to their disability in ways that cause significant harm, and requires providers of high-risk AI systems to comply with accessibility requirements. These laws matter in two ways. They create obligations that make institutions take accessibility seriously, which is the social model in legal form. And they create a market: once accessibility is required, companies that can deliver it have customers. The risk, illustrated by the overlay story, is that legal pressure also creates a market for products that offer the appearance of compliance without the substance. Enforcement, and the involvement of disabled people in judging what counts as accessible, are what distinguish the two. Who decides Running through every chapter of this book is a single question: who decides what disabled people need, and how their needs are met? The history of assistive technology suggests a clear answer about what works. Tools designed with the genuine participation of disabled people are more likely to be useful and to be kept. Be My Eyes was founded by a man who was losing his sight. Project Gameface was developed with a disabled gamer. The Speech Accessibility Project recruited participants through disability organisations and paid them. The most respected sign language research involves Deaf researchers as leaders, not merely as subjects. Conversely, the sign language gloves, the overlays and many of the "social skills" apps for autistic people were designed by people who did not share the experience they were addressing and did not listen when the people who did told them it was wrong. Participation means more than user testing at the end of a project. It means disabled people in research teams, product teams and leadership, in the organisations that fund research and the bodies that set standards. It means paying disabled people for their expertise rather than treating their time as free. It means building systems that disabled users can customise, correct and override, because no team, however inclusive, can anticipate every user's needs. And it means that when a product affects the lives of disabled people, they have a voice in decisions about whether it is changed, sold or discontinued. There is also a global dimension that is easy to forget in a discussion dominated by technologies developed in a few wealthy countries. The World Health Organization and UNICEF's Global Report on Assistive Technology, published in 2022, estimated that more than 2.5 billion people need one or more assistive products, a number expected to rise to 3.5 billion by 2050 as populations age, and that access to the products people need ranges from as little as three percent in some low-income countries to ninety percent in some high-income ones. Most AI accessibility tools work best in English and a handful of other major languages, require recent smartphones and reliable internet connections, and are tested mainly on users in wealthy countries. A blind person in rural Africa, a Deaf person who uses a local sign language with no digital presence, or a person with a speech impairment who speaks a language with little training data is not served by the advances this book has described. Closing that gap requires deliberate investment in languages, devices and contexts that commercial incentives alone will not reach. The final question is what all of this implies, taken together, for how AI for accessibility should be built, judged and governed. That is the subject of the conclusion. Conclusion Set side by side, the technologies in this book look at first like a story of steady triumph. A blind woman reads a can's expiry date in seconds. A hard-of-hearing man follows a family dinner through captions on his phone. A woman with ALS speaks at sixty-two words per minute through electrodes in her cortex; a man with ALS sings a simple melody through a voice synthesised directly from his neural activity. A person who cannot use their hands navigates a tablet with their eyes, using nothing but the camera already built into it. A decade ago most of this was impossible. The pace of change is real, and nothing in the preceding chapters should be read as diminishing it. But the chapters also contain another pattern, less often told, and it supports a conclusion that could not have been drawn from any single technology on its own. Across vision, hearing, sign, speech, brain interfaces and cognition, the systems that succeeded in people's lives were not simply the most accurate. They were the ones whose users could tell when they were wrong, could correct or override them, could choose when to use them, and could rely on them still existing next year. The systems that failed, or did harm, were often impressive in demonstration, but they concealed their errors, took decisions out of users' hands, were imposed by institutions to save money, or disappeared when their makers changed course. In accessibility, agency is not a soft value to be balanced against performance. It is a component of performance, because a tool that a person cannot trust, control or keep is a tool they will eventually stop using, and the abandonment research is unambiguous on that point. Five questions for any AI accessibility tool If agency is the measure, it can be made concrete. The chapters suggest five questions that anyone building, funding, buying or regulating such a system should be able to answer, and that disabled users are already, in effect, asking. The first is whether the user can tell when the system is likely to be wrong. A vision model that describes a medication label with the same fluency whether it is right or wrong shifts all the risk to the person least able to check. A caption system that silently drops words it cannot recognise does the same. Systems that express uncertainty, distinguish observation from inference, and make it easy to escalate to a human respect the user's judgement instead of replacing it. The second is whether the user can correct, personalise and override it. The most successful speech recognition for atypical voices was personalised; the most successful brain-computer interfaces adapt to their individual users. A communication aid powered by a language model must leave the final word with the person speaking. Treviranus's argument about outliers implies that no system will ever be trained on enough data to serve every disabled person well by default. Adaptability is the only general answer to that problem. The third is who holds the data and what is done with it. Blind people's photographs, Deaf signers' faces, the voices of people with ALS and the neural recordings of BCI users are among the most intimate data any technology collects. Where computation can be done on the device, it should be. Where it cannot, users should know what is stored, for how long and for what purpose, and should be able to have it deleted. The fourth is what happens if the provider stops. Soundscape, Google Glass and the Argus II retinal implant each left users stranded. For an app, that is an inconvenience. For an implant, it is a crisis. Any system on which disabled people are likely to build their lives should come with a credible answer: open-sourced code, escrowed specifications, funded long-term support, interoperable standards or a clear path to removal. Regulators and health systems are in a position to demand those answers before approval and funding, and they should. The fifth is who chose the problem. The sign language glove, the accessibility overlay and the emotion-recognition app for autistic children all solved problems that their designers imagined rather than problems that disabled people identified. The products that were kept, from Be My Eyes to Project Gameface to the Speech Accessibility Project's datasets, involved disabled people from the beginning. "Nothing about us without us" is not a courtesy; it is the most reliable predictor of usefulness the field has. Pointing the technology upstream There is a further conclusion that follows from putting the social model beside the technologies. Almost every tool in this book compensates, in one way or another, for a world that was not designed with disabled people in mind. The camera app reads the menu that was not published accessibly; the overlay attempts to patch the website that was built without labels; the caption tool stands in for a captioner nobody booked; the avatar signs the announcement because no interpreter was funded. This is valuable work, and for the foreseeable future people will depend on it. But if AI is used only to equip individuals to cope with inaccessible environments, it risks entrenching those environments by making their inaccessibility seem less urgent. The same capabilities can be pointed upstream. Language models can help authors write alternative text that explains why an image is there, rather than leaving it to a stranger's model to guess. They can help developers find and fix accessibility problems in their own code, the opposite of an overlay. They can help easy-read writers produce first drafts that are then tested by people with intellectual disabilities. They can help institutions schedule interpreters, prepare captioners with specialised vocabularies and translate documents into accessible formats at scale. Each of these uses changes the environment, not just the person, and each leaves the disabled user with fewer barriers to work around in the first place. A serious accessibility strategy for any organisation should spend at least as much effort on AI that makes its own products and services accessible as on AI tools that it hands to disabled users. What remains unsettled Some questions in this book have no settled answer, and it would be false to pretend otherwise. How much of a language model's assistance can a person accept before their words stop being their own? Where exactly should the line fall between attempted speech, which a person means to share, and inner speech, which they do not? Who should own a synthetic voice cloned from someone who has died? Should brain-computer interfaces be developed primarily as medical devices for people who need them now, or as stepping stones to consumer products for everyone, and can they be both? How can the benefits of these technologies reach the billions of disabled people who live in countries and speak languages that commercial developers largely ignore? These are not purely technical questions, and they will not be answered by engineers alone. They will be answered, well or badly, by the combination of disabled people's organisations, researchers, companies, regulators, health systems and courts that the preceding chapters have described. What can be said with confidence is this. Artificial intelligence has opened possibilities for disabled people that would have seemed fantastical within living memory, and the pace is not slowing. Whether those possibilities become lasting gains in freedom and participation depends less on the next breakthrough in accuracy than on decisions about trust, control, data, continuity and voice. Those decisions are being made now. Disabled people have been saying for decades what good decisions look like. The most important thing the builders of these technologies can do is to keep listening to them. Notes Introduction 1. World Health Organization, Global Report on Health Equity for Persons with Disabilities (Geneva: WHO, 2022), source of the estimate of about 1.3 billion people, or one in six, experiencing significant disability. Chapter 1 1. James I. Charlton, Nothing About Us Without Us: Disability Oppression and Empowerment (Berkeley: University of California Press, 1998). 2. Betsy Phillips and Hongxin Zhao, "Predictors of Assistive Technology Abandonment," Assistive Technology 5, no. 1 (1993): 36–45. 3. Michael Oliver, The Politics of Disablement (London: Macmillan, 1990), an influential statement of the social model of disability. Chapter 2 1. Jeffrey P. Bigham et al., "VizWiz: Nearly Real-Time Answers to Visual Questions," Proceedings of the ACM Symposium on User Interface Software and Technology (2010). 2. Danna Gurari et al., "VizWiz Grand Challenge: Answering Visual Questions from Blind People," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018). 3. Danna Gurari et al., "VizWiz-Priv: A Dataset for Recognizing the Presence and Purpose of Private Visual Information in Images Taken by Blind People," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019). 4. Cynthia L. Bennett et al., "It's Complicated: Negotiating Accessibility and (Mis)Representation in Image Descriptions of Race, Gender, and Disability," Proceedings of the CHI Conference on Human Factors in Computing Systems (2021). 5. Kashmir Hill, reporting in The New York Times (July 2023) on the obscuring of faces in early tests of Be My AI. 6. WebAIM, The WebAIM Million, annual accessibility analysis of the top one million home pages, published since 2019. Chapter 4 1. World Health Organization, World Report on Hearing (Geneva: WHO, 2021). 2. Alec Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision" (OpenAI, 2022), the paper describing the Whisper model. 3. Pablo Romero-Fresco and Juan Martínez Pérez, "Accuracy Rate in Live Subtitling: The NER Model," in Audiovisual Translation in a Global Context, ed. Rocío Baños Piñero and Jorge Díaz Cintas (Palgrave Macmillan, 2015). 4. Allison Koenecke et al., "Racial Disparities in Automated Speech Recognition," Proceedings of the National Academy of Sciences 117, no. 14 (2020): 7684–7689. Chapter 5 1. William C. Stokoe, Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf (Buffalo: University of Buffalo, 1960). 2. Michael Erard, "Why Sign-Language Gloves Don't Help Deaf People," The Atlantic, November 2017. 3. Danielle Bragg et al., "Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective," Proceedings of the ACM SIGACCESS Conference on Computers and Accessibility (2019). 4. Amanda Duarte et al., "How2Sign: A Large-Scale Multimodal Dataset for Continuous American Sign Language," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021). 5. David Uthus, Garrett Tanzer and Manfred Georg, "YouTube-ASL: A Large-Scale, Open-Domain American Sign Language–English Parallel Corpus," Advances in Neural Information Processing Systems (2023). 6. World Federation of the Deaf and World Association of Sign Language Interpreters, "WFD and WASLI Statement on Use of Signing Avatars" (2018). Chapter 6 1. Speech Accessibility Project, Beckman Institute for Advanced Science and Technology, University of Illinois Urbana-Champaign, project announcements and reports, 2022–2025. 2. Colin Lea et al., "SEP-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter," Proceedings of ICASSP (2021). 3. Shanqing Cai et al., "Using Large Language Models to Accelerate Communication for Eye Gaze Typing Users with ALS," Nature Communications 15 (2024). Chapter 7 1. Leigh R. Hochberg et al., "Neuronal Ensemble Control of Prosthetic Devices by a Human with Tetraplegia," Nature 442 (2006): 164–171. 2. Leigh R. Hochberg et al., "Reach and Grasp by People with Tetraplegia Using a Neurally Controlled Robotic Arm," Nature 485 (2012): 372–375. 3. Jennifer L. Collinger et al., "High-Performance Neuroprosthetic Control by an Individual with Tetraplegia," The Lancet 381 (2013): 557–564. 4. Francis R. Willett et al., "High-Performance Brain-to-Text Communication via Handwriting," Nature 593 (2021): 249–254. 5. David A. Moses et al., "Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria," New England Journal of Medicine 385 (2021): 217–227. 6. Francis R. Willett et al., "A High-Performance Speech Neuroprosthesis," Nature 620 (2023): 1031–1036. 7. Sean L. Metzger et al., "A High-Performance Neuroprosthesis for Speech Decoding and Avatar Control," Nature 620 (2023): 1037–1046. 8. Nicholas S. Card et al., "An Accurate and Rapidly Calibrating Speech Neuroprosthesis," New England Journal of Medicine 391 (2024): 609–618. 9. Maitreyee Wairagkar et al., "An Instantaneous Voice-Synthesis Neuroprosthesis," Nature 644 (2025). 10. Erin M. Kunz et al., "Inner Speech in Motor Cortex and Implications for Speech Neuroprostheses," Cell (2025). 11. Jerry Tang, Amanda LeBel, Shailee Jain and Alexander G. Huth, "Semantic Reconstruction of Continuous Language from Non-Invasive Brain Recordings," Nature Neuroscience 26 (2023): 858–866. Chapter 8 1. Eliza Strickland and Mark Harris, "Their Bionic Eyes Are Now Obsolete and Unsupported," IEEE Spectrum, February 2022. 2. Regulatory and clinical milestones for Synchron, Neuralink, Precision Neuroscience and Paradromics are drawn from company announcements and reporting on US Food and Drug Administration decisions, 2022–2026. 3. Kim D. Anderson, "Targeting Recovery: Priorities of the Spinal Cord-Injured Population," Journal of Neurotrauma 21, no. 10 (2004): 1371–1383. 4. Jane E. Huggins, Patricia A. Wren and Kirsten L. Gruis, "What Would Brain-Computer Interface Users Want? Opinions and Priorities of Potential Users with Amyotrophic Lateral Sclerosis," Amyotrophic Lateral Sclerosis 12, no. 5 (2011): 318–324. 5. Colorado House Bill 24-1058 (2024) and California Senate Bill 1223 (2024), amending state privacy law to cover neural data. 6. UNESCO, Recommendation on the Ethics of Neurotechnology (2025). Chapter 10 1. US Equal Employment Opportunity Commission, "The Americans with Disabilities Act and the Use of Software, Algorithms, and Artificial Intelligence to Assess Job Applicants and Employees" (May 2022). 2. Overlay Fact Sheet, overlayfactsheet.com (2021). 3. US Federal Trade Commission, order concerning accessiBe, announced January 2025. 4. World Health Organization and UNICEF, Global Report on Assistive Technology (Geneva, 2022). 5. United Nations, Convention on the Rights of Persons with Disabilities (2006). Further Reading Charlton, James I. Nothing About Us Without Us: Disability Oppression and Empowerment (University of California Press, 1998).** The book that popularised the disability movement's central slogan, and still the best short account of why disabled people's control over decisions affecting them matters. Hendren, Sara. What Can a Body Do? How We Meet the Built World (Riverhead, 2020).** A designer and engineer's reflections on disability, adaptation and the built environment; an excellent companion to the social model as it applies to technology. Williamson, Bess. Accessible America: A History of Disability and Design (NYU Press, 2019).** A history of how American design, from curb cuts to consumer products, has been shaped by disability activism. Bragg, Danielle, et al. "Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective" (ASSETS, 2019). The clearest single statement of what responsible sign language technology research requires, written by a team that included Deaf scholars. World Health Organization and UNICEF. Global Report on Assistive Technology (2022).** The authoritative global picture of who needs assistive technology and who has access to it. Freely available online. Willett, Francis R., et al. "A High-Performance Speech Neuroprosthesis" (Nature, 2023), and Card, Nicholas S., et al. "An Accurate and Rapidly Calibrating Speech Neuroprosthesis" (New England Journal of Medicine, 2024). Two landmark papers in speech decoding, readable by non-specialists in their introductions and discussions. Treviranus, Jutta. Essays and talks on inclusive design and the statistical exclusion of outliers, published through the Inclusive Design Research Centre at OCAD University. A sharp account of why data-driven systems tend to fail people at the margins and what inclusive design offers instead. Whittaker, Meredith, et al. Disability, Bias, and AI (AI Now Institute, 2019).** A short report that maps how AI systems can encode and amplify discrimination against disabled people. Web Content Accessibility Guidelines (WCAG), World Wide Web Consortium. The international standard for accessible digital content, and the benchmark against which claims of automated accessibility should be judged. Hashtags: #AIForAccessibility #ArtificialIntelligence #AssistiveTechnology #DisabilityInclusion #AccessibleTechnology #InclusiveInnovation #DigitalAccessibility #DisabilityEmpowerment #AIForGood #AccessibilityInnovation #InclusiveDesign #UniversalDesign #HumanCenteredAI #DisabilityRights #AccessibleAI #ComputerVision #VisionLanguageModels #VisualAssistance #ScreenReaders #AutomaticCaptioning #SpeechRecognition #SignLanguageTechnology #AugmentativeCommunication #VoiceBanking #BrainComputerInterfaces #Neuroprosthetics #AccessibleNavigation #AdaptiveTechnology #ResponsibleAI #DigitalInclusion

Download the Book (PDF): Introduction Open a news site on a train as it enters a tunnel. Tap a headline. In most cases the browser shows a spinner for twenty or thirty seconds and then a grey page with a little dinosaur or a cloud icon, telling you that you are not connected to the internet. Now open a native news app on the same train. It shows the articles it downloaded this morning, lets you read them, and quietly queues the bookmark you tapped until the signal returns. The difference between those two experiences is not a matter of screen size, programming language, or app store. It is a matter of who controls the network request. In the browser, by default, nobody does: the page asks, the network fails, and the browser gives up. In the native app, the developer wrote code that sits between the interface and the network and decides what to do when the network is slow, absent, or lying. A Progressive Web App is a website whose developer has taken back that control. The mechanism is the service worker: a script that the browser installs alongside a site, runs on its own thread outside any page, and consults before it sends any network request the site makes. The service worker can answer a request from a local cache, forward it to the network, combine the two, or construct a response from scratch. It can wake up when a push message arrives from a server and show a notification. It can be asked to retry a failed upload when connectivity comes back, even after the user has closed the tab. Pair that script with a small JSON file called a web app manifest, which tells the operating system the app's name, icons, colours and how it should launch, and serve everything over HTTPS, and you have the ingredients the industry has called a Progressive Web App since 2015, when Alex Russell and Frances Berriman gave the pattern its name. The argument of this booklet The usual pitch for PWAs is that they let one codebase reach every device without app store friction. That is true and it is not the interesting part. The interesting part is that a service worker turns the network from an assumption into a dependency that you design around, the way a backend engineer designs around a database that might be slow or a payment provider that might time out. Once you hold that view, most PWA decisions become engineering decisions with clear tradeoffs rather than a checklist to tick for an installability badge. The controlling claim of this booklet is therefore simple: a Progressive Web App is a reliability contract written in code, and the work is to decide, request by request, what the user is promised when the network fails, then to keep that promise within the limits each browser actually enforces. Every chapter tests that claim against a different part of the system. Caching strategies are the vocabulary for stating the promise. Storage quotas and eviction policies are the limits on how long you can keep it. Background sync and push are the mechanisms for keeping it when the page is not open. Web performance metrics are how you find out whether you kept it for real users on real phones. And the platform politics, particularly Apple's, determine which parts of the promise you are even allowed to make on a given device. That second clause matters. PWA advocacy has a long history of describing the web as it will be rather than as it is. A developer who reads that "PWAs support background sync" and builds an offline-first field-inspection app around it will discover, on the first iPhone in the field, that Safari has never implemented the Background Sync API. A team that reads that "storage is persistent" will discover that Safari, in its default tracking-prevention mode, deletes script-written storage for origins the user has not interacted with in seven days. This booklet is written for the developer who would rather know those facts before shipping. Where support varies, the text says so and says what to do about it. What has changed recently Several shifts in the last few years change how a PWA should be built in late 2026, and it is worth naming them at the outset because older tutorials predate them. Apple shipped Web Push for Home Screen web apps in iOS and iPadOS 16.4 in March 2023, ending the single largest functional gap between PWAs and native apps on the iPhone. In iOS 18.4 it added Declarative Web Push, a format in which the push message itself describes the notification, so that no JavaScript has to run to display it. In iOS 26 it went further: every site added to the Home Screen now opens as a web app by default, whether or not it ships a manifest. Those gains came alongside a scare. In February 2024, in the betas of iOS 17.4, Apple removed standalone Home Screen web apps for users in the European Union, citing the work required to comply with the Digital Markets Act's requirement to allow other browser engines. Installed web apps opened as ordinary Safari tabs, lost their notifications, and lost access to their stored data. After weeks of protest from developers and questions from regulators, Apple reversed the decision on 1 March 2024 and restored the feature, still built only on WebKit. The episode is the clearest demonstration available that the parts of a PWA that depend on operating system integration exist at the platform owner's discretion, and it shapes the advice in the final chapter. On the performance side, Google replaced First Input Delay with Interaction to Next Paint as a Core Web Vital in March 2024. INP measures the latency of every interaction across a visit, not just the first one, and it is far harder to pass on low-end Android phones. A PWA that caches aggressively but runs heavy JavaScript on every tap will load instantly and still feel sluggish, and INP will say so. On the tooling side, Workbox, the most widely used library for writing service workers, moved from Chrome's developer relations team to Chrome's Aurora team, and its release cadence has slowed to maintenance updates; version 7.4.0, released in November 2024, was described as critical dependency updates. It remains a sound foundation, but teams should understand what it does under the hood, because the libraries around it are now more stable than evolving. Chromium also shipped the Service Worker Static Routing API in Chrome 123, which lets a service worker tell the browser to skip running JavaScript for certain requests, addressing one of the oldest performance complaints about service workers. And Firefox, long the holdout on installation, added the ability to pin websites to the Windows taskbar as standalone windows in Firefox 143 in September 2025. How to read this booklet The chapters build on one another but each can be read alone by someone who knows basic JavaScript and HTTP. Chapter 1 defines what a PWA is in technical terms, what the manifest and service worker each contribute, and what installability means on each major platform. Chapter 2 covers the service worker lifecycle in detail, because nearly every production bug in a PWA traces back to a misunderstanding of when a new service worker takes control. Chapter 3 sets out the caching strategies, with stale-while-revalidate given the most attention because it is the most useful and the most misused. Chapter 4 moves from caching individual requests to designing an offline experience: app shells, navigation fallbacks, and data stored in IndexedDB. Chapter 5 deals with storage quotas, eviction, and cache versioning, the unglamorous constraints that decide whether an offline promise survives a week. Chapter 6 covers writes made while offline, the outbox pattern, and the Background Sync API with its fallbacks. Chapter 7 covers push notifications end to end, including VAPID, payload encryption, iOS requirements, and permission design. Chapter 8 turns to performance and the Core Web Vitals, including the costs a service worker can add as well as the ones it removes. Chapter 9 covers operating a PWA in production: tooling, testing, updates, kill switches, and the platform politics. The conclusion draws out what follows from all of this for teams deciding how far to invest. Code snippets are short and use plain service worker APIs unless a library is named. They are written to teach the mechanism, so they omit some error handling a production version would need; where the omission matters, the surrounding text says so. One note on vocabulary. "Offline" in this booklet means any condition in which the network cannot be relied upon, which includes the far more common cases of a connection that is technically up but delivering a few kilobytes per second, or a captive portal at a hotel that returns a login page for every request. Designing only for the clean case where navigator.onLine is false misses most of the real failures users experience. The service worker's value lies precisely in handling the murky middle, and the strategies in the chapters that follow are built for it. Chapter 1: What a Progressive Web App Actually Is The phrase "Progressive Web App" is used loosely enough that two engineers can argue about whether a site is one without either being wrong. Some use it to mean any site that can be installed to a home screen. Others mean any site with a service worker. Marketing material often means "a website that feels like an app", which is not a technical property at all. Before building one, it helps to pin down the parts, what each part does, and which parts the operating system cares about. The three ingredients A PWA in the technical sense has three ingredients: it is served over HTTPS, it ships a web app manifest, and it registers a service worker. Each solves a different problem. HTTPS is a precondition rather than a feature. Service workers can intercept and rewrite every request a site makes, which is exactly the power a network attacker would want. Browsers therefore refuse to register a service worker on any origin that is not secure. The one exception is localhost, which browsers treat as secure so that developers can work without certificates. In practice this requirement is no longer a hurdle, since free certificates and automatic HTTPS at most hosting providers have made plain HTTP rare, but it does mean that an internal tool served over HTTP on a corporate network cannot become a PWA until someone gives it a certificate. The web app manifest is a JSON file linked from the page's HTML. It tells the browser and operating system how the app should present itself when installed. A minimal useful manifest looks like this: { "name": "Field Notes", "short_name": "Notes", "start_url": "/?source=pwa", "scope": "/", "display": "standalone", "background_color": "#ffffff", "theme_color": "#1f4e79", "icons": [ { "src": "/icons/192.png", "sizes": "192x192", "type": "image/png" }, { "src": "/icons/512.png", "sizes": "512x512", "type": "image/png" }, { "src": "/icons/maskable-512.png", "sizes": "512x512", "type": "image/png", "purpose": "maskable" } ] } It is linked with . The display value is the field with the most visible effect. standalone launches the app in its own window without browser address bar or tabs, so it looks and behaves like a native app in the task switcher. minimal-ui keeps a small set of navigation controls. browser opens a normal tab, which defeats the purpose. fullscreen hides even the status bar and is mainly for games. The start_url with a query parameter is a common trick for distinguishing launches from the installed app in analytics. The maskable icon has extra padding so that Android can crop it into a circle, a squircle, or whatever shape the launcher uses without cutting off the logo. The manifest has grown many optional members over the years: shortcuts for long-press menus on the launcher icon, share_target so the app appears in the operating system's share sheet, file_handlers so it can open certain file types on desktop, protocol_handlers, screenshots for a richer install dialog, and id, which gives the app a stable identity independent of its start URL. Support for each varies, and almost all of them are Chromium-only or Chromium-first. The core members above are what matter everywhere. The service worker is the third ingredient and the one that does real work. It is a JavaScript file, registered from a page, that the browser installs and then runs in a separate context whenever there is an event for it to handle. Registration is a single call: if ('serviceWorker' in navigator) { window.addEventListener('load', () => { navigator.serviceWorker.register('/sw.js'); }); } Waiting for the load event is a deliberate choice. Registering a service worker on first visit triggers its installation, which may download and cache a set of files. Doing that while the page itself is still loading competes for bandwidth with the resources the user is waiting for. Deferring registration until after load costs nothing on the first visit, because the service worker cannot help that visit anyway, and protects the first impression. What the service worker can and cannot do A service worker has no access to the DOM. It cannot read or change the page, cannot touch window, and cannot use synchronous APIs like localStorage. It communicates with pages by posting messages, and it stores data using the asynchronous Cache Storage API and IndexedDB. It runs only when there is an event to handle and the browser terminates it when idle, often after about thirty seconds, so it cannot keep state in global variables between events and expect them to survive. Every service worker author learns this the hard way at least once: a counter or an in-memory map that works perfectly in development, where DevTools keeps the worker alive, and resets at random in production. The events a service worker can handle define what a PWA can do. The install and activate events belong to its lifecycle and are covered in Chapter 2. The fetch event fires for every network request from pages the service worker controls, including the navigation request for the page itself, and is the basis of everything in Chapters 3 through 5. The push event fires when a push message arrives from a server, and notificationclick fires when the user taps a notification; both are the subject of Chapter 7. The sync event, where supported, fires when the browser judges connectivity to have returned after a page asked to be told; Chapter 6 covers it and its limits. The message event lets pages send instructions, such as "skip waiting and take over now". A service worker controls pages within its scope, which by default is the directory containing the script. A worker at /sw.js controls the whole origin; one at /app/sw.js controls only URLs under /app/. This is why service worker files are usually served from the site root. A server can widen the allowed scope with a Service-Worker-Allowed response header, but serving the file from the root is simpler and avoids surprises. The most important fact about a service worker is that it does not run on the first page load. The page loads normally from the network, registers the worker, and the worker installs in the background. Only on the next navigation, or after the worker explicitly claims existing pages, does it start intercepting requests. Users do not experience a PWA's offline behaviour until their second visit at the earliest. Teams that test only by loading a page once and then turning off the network are testing something users will never see. Installability across platforms "Installable" means the browser will offer to add the site to the home screen, dock, or app launcher as a standalone app. The rules differ by browser, and they have loosened over time. Chrome historically required a manifest with the right fields and a service worker with a fetch handler before it would offer installation. That requirement produced an industry of empty fetch handlers written purely to satisfy the check. Google acknowledged the problem and removed the fetch-handler requirement for installing from the browser menu in Chrome 108 on mobile and Chrome 112 on desktop, substituting a default offline page for sites that do not provide their own. A manifest with a name, icons, a start URL and a display mode remains the expectation. Chrome's automatic install prompt, the beforeinstallprompt event that sites can intercept to show their own install button, still uses a richer set of signals, including user engagement with the site. Microsoft Edge follows Chromium's behaviour and also lets sites be packaged into the Microsoft Store. Samsung Internet, a large share of Android browsing, supports installation and the manifest well. Safari on iOS and iPadOS has never shown an automatic install prompt and does not fire beforeinstallprompt. Users install through the Share menu with "Add to Home Screen". Until iOS 26, a site added this way opened as a standalone web app only if it declared a standalone display mode in its manifest or used Apple's older apple-mobile-web-app-capable meta tag; otherwise it opened as a Safari bookmark. In iOS 26, Apple changed the default: every site added to the Home Screen opens as a web app, and users can switch off "Open as Web App" if they prefer a bookmark. A manifest is no longer required for standalone launch on Apple's platforms, though it is still how a site controls its name, icon, colours and scope. On macOS, Safari 17 added "Add to Dock", which turns any site into a standalone app window. Firefox on Android has long supported adding sites to the home screen. On desktop, Firefox removed its experimental site-specific browser feature in 2021 and offered nothing comparable until Firefox 143, released in September 2025, added the ability on Windows to pin a site to the taskbar so that it opens in a simplified, separate window. The upshot is that installation is not a property you can rely on as a gate. A PWA must work as an ordinary website in a browser tab, because most visits will be exactly that, and should treat installation as an upgrade that unlocks a better presentation and, on iOS, the right to request push notifications. Knowing how the app was launched Because the same code runs as a tab and as an installed app, a PWA often needs to know which it is. An install button makes no sense inside an installed app; instructions for adding to the Home Screen make sense only in iOS Safari; analytics are more useful when they distinguish the two. Several mechanisms provide the answer. The display-mode media query reports how the page is presented. It can be used in CSS, for instance to add a back button in standalone mode where the browser's own controls are absent, and in JavaScript: const standalone = window.matchMedia('(display-mode: standalone)').matches || window.navigator.standalone === true; // older iOS property let deferredPrompt = null; window.addEventListener('beforeinstallprompt', (e) => { e.preventDefault(); // suppress the browser's own mini-infobar deferredPrompt = e; showInstallButton(); // reveal our own, contextual control }); installButton.addEventListener('click', async () => { if (!deferredPrompt) return; deferredPrompt.prompt(); const { outcome } = await deferredPrompt.userChoice; track('install_prompt', outcome); deferredPrompt = null; }); window.addEventListener('appinstalled', () => track('installed')); The beforeinstallprompt event exists only in Chromium-based browsers. Capturing it lets a site defer the prompt until a moment when installation makes sense, such as after the user has completed a task, and show it from its own button. Each captured event can be used once. The appinstalled event fires when installation completes by any route, including the browser menu, which makes it the right place to record installs. On iOS there is no event and no programmatic prompt. The only option is to detect that the page is running in Safari on an iPhone or iPad and not in standalone mode, then show brief instructions pointing at the Share menu. Doing this on every visit is irritating; doing it after the user has shown commitment, such as on their third visit or after they have saved something, and remembering a dismissal, is the pattern that works. Analytics deserve the same care. Tagging the start_url with a query parameter, as in the manifest above, distinguishes launches from the home screen icon. Recording the display mode on every page view distinguishes sessions in the installed app from sessions in a tab, which matters because the two populations behave differently and, as later chapters show, the browser treats their storage and notifications differently too. Where the line with native apps sits It is worth being specific about what a PWA still cannot do compared with a native app, because overclaiming here leads to bad architectural decisions. On Android and desktop Chromium, the gap is narrow for most business and content apps. Installed PWAs get their own windows and task-switcher entries, push notifications, badges on the launcher icon, share targets, file handling on desktop, and access through various Chromium-specific APIs to Bluetooth, USB, serial ports and the file system. Google's Trusted Web Activity mechanism lets a PWA be wrapped and published in the Play Store with minimal native code. What remains missing is deep integration: widgets, background execution on a schedule the developer controls, contact and calendar access, and the reliability of push delivery that native apps get from tighter platform integration. On iOS, the gap is wider. All browsers on iOS outside the EU must use Apple's WebKit engine, so the capabilities of a PWA on an iPhone are exactly the capabilities Apple chooses to give WebKit's Home Screen web apps. Those now include push notifications, badges, and full-screen standalone display, but not Background Sync, Periodic Background Sync, Background Fetch, Web Bluetooth, or Web USB. Storage is subject to the rules described in Chapter 5. Within the EU, the Digital Markets Act permits alternative browser engines on iOS, but as the 2024 episode showed, Apple has not extended Home Screen web app support to them, so even an EU user running a Chromium-based browser on an iPhone gets a WebKit-powered web app when they install one. None of this makes PWAs a poor choice. It makes them a specific choice. They are excellent for content sites, commerce, internal business tools, dashboards, booking flows, and anything else where reach and update speed matter more than hardware access. They are a weak choice for apps whose core value depends on background execution, such as a fitness tracker that must record location continuously, or on platform integrations that only native code can reach. The organizations that have had the most success with PWAs, from Twitter's "Twitter Lite" in 2017 to the many retailers that followed, were generally serving users on low-end Android devices and slow networks, where the alternative was a large native download that many users would never install. A working definition Pulling these threads together gives a working definition that the rest of this booklet uses. A Progressive Web App is a website that: 1. Is served over HTTPS and works in any modern browser as an ordinary site. 2. Registers a service worker that defines, for each category of request, what happens when the network is slow or unavailable. 3. Provides a manifest so that, where the platform supports it, users can install it and launch it as a standalone app. 4. Uses additional capabilities such as push and background sync where they exist, and degrades to a sensible experience where they do not. The word "progressive" in the name carries the fourth point, and it is the one most often neglected. The original 2015 framing was progressive enhancement: start with something that works everywhere, then layer on capabilities that improve the experience where the platform allows. That discipline is not an aesthetic preference. It is what keeps a PWA usable on the iPhone that lacks background sync, the corporate laptop with service workers disabled by policy, and the browser in private mode that refuses persistent storage. Every capability discussed in this booklet should be wrapped in feature detection, and every design should answer the question of what the user sees when the capability is missing. The second point carries the controlling idea. A service worker's real job is not to make a site installable or to earn a score in an audit tool. It is to state, in code, what the user is promised when the network does not cooperate. The next chapter explains the machinery that decides when that code runs, and why getting it wrong produces the most frustrating class of PWA bugs: users stuck on an old version of the site with no obvious way to get the new one. Chapter 2: The Service Worker Lifecycle Most of the painful bugs in production PWAs are not caching bugs. They are lifecycle bugs: a user who keeps seeing last month's version of the site, a deploy where half the open tabs run new code and half run old code against a changed API, a cache that was deleted while a page still needed files from it. All of them come from misunderstanding one design decision the authors of the service worker specification made deliberately: a new version of a service worker does not take control of pages that are already open. This chapter explains that decision, the states a worker passes through, and the patterns for shipping updates that neither strand users nor break them. Why the lifecycle is conservative Consider what would happen if a new service worker took over immediately. A user has the app open in a tab. That page loaded its HTML, CSS and JavaScript under version 1 of the service worker, which served them from a cache named for version 1. You deploy version 2, which renames assets, changes the cache name, and deletes the old cache during activation. If version 2 took over the open tab immediately, the next lazy-loaded JavaScript chunk the version 1 page requested would not exist in the new cache and might not exist on the server either, because the build renamed it. The page would break in the middle of the user's session. The specification avoids this by ensuring, by default, that only one version of a service worker controls a page for that page's whole life, and that a new version waits until every page controlled by the old version has closed. The cost of this safety is that updates are slow to land. A user who keeps a single tab open for days, which is common for installed PWAs and for web mail and dashboards, will run the old version for days. Every update strategy in this chapter is a way of choosing a different point on the line between safety and freshness, deliberately rather than by accident. The states A service worker passes through a fixed sequence of states, and the browser exposes them through the ServiceWorkerRegistration object as installing, waiting, and active. Parsed and installing. When a page calls register(), the browser downloads the script, parses it, and fires the install event. This is the place to precache the files the app needs to work offline. The worker extends the event with a promise using event.waitUntil(), and installation succeeds only if that promise resolves. If any file fails to download, the promise rejects, installation fails, and the browser discards the new worker. That all-or-nothing behaviour is a feature: it guarantees that an installed worker has a complete set of the files it expects. const VERSION = 'v42'; const PRECACHE = `precache-${VERSION}`; const ASSETS = ['/', '/app.3f9a1c.js', '/app.8b20d4.css', '/offline.html']; self.addEventListener('install', (event) => { event.waitUntil( caches.open(PRECACHE).then((cache) => cache.addAll(ASSETS)) ); }); Installed and waiting. If there is no existing active worker, the new one proceeds directly to activation. If there is one, the new worker enters the waiting state and stays there until every client controlled by the old worker is gone. "Client" here means any window, tab, or worker the service worker controls. Reloading a page does not release it, because during a reload the old page is still alive when the new one starts; the browser keeps the old worker in control across the reload. This is the source of the familiar confusion where a developer refreshes repeatedly and never sees the new version. Closing every tab, or navigating them all away from the origin, is what lets the waiting worker proceed. Activating and activated. When the waiting worker is allowed to proceed, it receives the activate event. This is the moment to clean up after the old version, typically by deleting caches the new version no longer uses. It is safe to do here and not during install, because by the time activation happens no page is still relying on the old worker's caches. self.addEventListener('activate', (event) => { event.waitUntil( caches.keys().then((keys) => Promise.all( keys .filter((k) => k.startsWith('precache-') && k !== PRECACHE) .map((k) => caches.delete(k)) ) ) ); }); After activation the worker is active and handles fetch, push and other functional events for pages it controls. A further subtlety is that an active worker does not automatically control pages that were opened before it existed. On a user's very first visit, the page that registered the worker loaded without one, and stays uncontrolled until it navigates or reloads. The worker can change that by calling self.clients.claim() during activation, which makes it the controller of all in-scope clients immediately. Redundant. A worker that fails to install, or that is replaced by a newer one, becomes redundant and is discarded. How updates are detected The browser checks for a new version of the service worker script on navigations to in-scope pages, on functional events such as push and sync if a check has not happened in the last 24 hours, and whenever a page calls registration.update() explicitly. It fetches the script and compares it byte for byte with the installed one. Any difference, even a single changed comment, counts as a new version and starts installation of a new worker. Modern engines extend the comparison to scripts loaded through importScripts() as well, so a change in an imported helper also counts. The browser's HTTP cache used to complicate this. If the server sent sw.js with a long cache lifetime, the browser could keep using its cached copy and never see the update. Browsers have since changed the default so that update checks for the service worker script bypass the HTTP cache, governed by a registration option called updateViaCache whose default value is 'imports', meaning only imported scripts may come from the HTTP cache. It remains sensible practice to serve sw.js with Cache-Control: no-cache or a short max-age, because some intermediary caches and CDNs ignore browser semantics, and a CDN edge holding a stale sw.js for a day is a real and recurring incident. Because any byte change triggers an update, the content of the script should change exactly when the app changes. Build tools handle this by injecting the list of precached files, with content hashes in their names or a revision field alongside, into the service worker at build time. A new build with different assets produces a different service worker, and a build with identical assets produces an identical one. Hand-maintaining a VERSION constant works for small projects but invites the error of forgetting to bump it, after which users never receive the new assets. Choosing an update strategy There are three common strategies for getting a new version in front of users, and each makes a different tradeoff. Wait for natural turnover. Do nothing special. The new worker installs in the background and waits; the next time the user has closed all tabs and opens the app again, they get the new version. This is the safest option and the right default for content sites where users rarely keep tabs open for long. Its weakness is installed PWAs on mobile, where the operating system may keep the app's page suspended in memory for a long time, so "all tabs closed" happens less often than developers expect. Skip waiting immediately. Call self.skipWaiting() in the install handler. The new worker activates as soon as it installs, and combined with clients.claim() it takes control of every open page at once. This gets updates to users fastest and reintroduces precisely the mixed-version hazard the lifecycle was designed to prevent: open pages now run old JavaScript while their requests are handled by new service worker logic and new caches. Whether that is acceptable depends on your app. If the service worker's routing is simple, assets are content-hashed, old assets remain on the server for a while after deploy, and the API is backward compatible, skipping waiting is often fine. If any of those conditions fails, it produces intermittent breakage that is very hard to reproduce. Prompt the user. Let the new worker wait, detect that it is waiting from the page, and show a small message: "A new version is available. Reload." When the user accepts, the page posts a message to the waiting worker telling it to skip waiting, listens for the controllerchange event that fires when the new worker takes over, and then reloads. This gives users control and ensures the reload happens at a moment of their choosing, with every page loaded fresh under the new worker. // In the page const reg = await navigator.serviceWorker.register('/sw.js'); function offerUpdate(worker) { showBanner('New version available', () => worker.postMessage('SKIP_WAITING')); } if (reg.waiting) offerUpdate(reg.waiting); reg.addEventListener('updatefound', () => { const nw = reg.installing; nw.addEventListener('statechange', () => { if (nw.state === 'installed' && navigator.serviceWorker.controller) offerUpdate(nw); }); }); let reloaded = false; navigator.serviceWorker.addEventListener('controllerchange', () => { if (!reloaded) { reloaded = true; location.reload(); } }); // In sw.js self.addEventListener('message', (e) => { if (e.data === 'SKIP_WAITING') self.skipWaiting(); }); The check for navigator.serviceWorker.controller distinguishes an update from the first install: on the first visit there is no controller, so there is nothing to update from and no banner should appear. The reloaded guard prevents a reload loop in the case where DevTools' "update on reload" option is on. For long-lived sessions, pair the prompt with a periodic call to registration.update(), for example once an hour while the page is visible, so that the browser checks for a new version even if the user never navigates. Many teams also force the issue at natural break points: if a waiting worker exists when the user navigates between major sections of a single-page app, perform a full page load instead of a client-side route change, letting the new worker take over invisibly. A deploy, step by step Abstract state diagrams hide how these rules play out for real users. Following one deploy through three users makes the consequences concrete. Suppose a team runs a booking PWA with the prompt-to-reload strategy. Version 41 of the service worker is live. At 10:00 the team deploys version 42, which changes the booking form's JavaScript and adds a new field to the booking API request. The server's API accepts requests both with and without the new field, and the build keeps version 41's assets on the CDN for a week. The first user, Amira, has the installed app open on her phone, suspended in the background since breakfast. At 10:30 she switches back to it. Switching back is not a navigation, so no update check happens immediately. Her page is still version 41 under worker version 41, and everything works, because both halves match. The page's hourly registration.update() timer fires at 10:45; the browser fetches sw.js, sees new bytes, and installs version 42 in the background, downloading only the files whose revision hashes changed. Version 42 enters the waiting state. The page's updatefound listener notices and shows a banner. Amira finishes her booking on version 41, taps "Reload", and the page posts SKIP_WAITING. Version 42 activates, deletes the version 41 precache, takes control, and the page reloads on version 42. Her session was never mixed. The second user, Ben, opens the site in a desktop browser tab at 11:00 for the first time in a month. The navigation triggers an update check. Version 42 installs and waits, but his page is controlled by version 41, which serves the version 41 shell from its precache. Ben sees the banner a few seconds after load. He ignores it and books on version 41, which works because the server still accepts the old request shape. When he closes the tab, version 42 activates; next time, he is on the new version without ever having reloaded. The third user, Chen, visits for the very first time at 12:00. There is no existing worker, so version 42 installs and activates immediately, with no waiting. Her page loaded from the network and is not controlled until the next navigation, unless the worker calls clients.claim(). Because no controller existed when the new worker installed, the banner logic correctly shows nothing. Two design choices made all three paths safe: the API accepted both versions of the request, and the old assets remained available. Remove either and Ben's session breaks, not Amira's. The users at risk in a service worker deploy are those who ignore the update, and the system must be built to tolerate them for days, not minutes. Lifecycle failures in production A few failure patterns recur often enough to be worth naming. The first is the broken precache. One file in the precache list returns a 404, perhaps because a build step renamed it, and installation fails silently. Users stay on the old version indefinitely, and nothing in the page's console shows an error, because the failure happens in the service worker's context. The defence is to report install failures: wrap the addAll promise, catch the rejection, send it to your error reporting endpoint, and rethrow so installation still fails. A sharp rise in install failures after a deploy is a signal worth alerting on. The second is the stale shell after deploy. The service worker serves the app's HTML from cache, and the HTML references content-hashed assets. After a deploy, the cached HTML still references the old hashes. If the server deleted the old assets during deploy, users whose old HTML loads before the new worker activates request files that no longer exist. The defence is to keep at least the previous release's assets available on the server for a period after each deploy, a practice that also helps users on the network who loaded the old HTML moments before the deploy. The third is the zombie worker: a service worker deployed in error, perhaps one that caches the HTML of every page forever, and that keeps serving stale content even after the site's code is fixed, because the faulty worker answers the navigation request before the server ever sees it. The browser will still check for an updated sw.js on navigation, so the fix is always to deploy a corrected worker at the same URL. In the worst case, that corrected worker is a kill switch, described in Chapter 9, that deletes all caches and unregisters itself. The Clear-Site-Data response header with the value "storage" is a heavier remedy that clears everything for the origin, including service worker registrations, and is useful when the site's own JavaScript cannot be trusted to run. The fourth is reliance on in-memory state. Because the browser stops idle workers, a worker that stores, say, an authentication token in a global variable will lose it unpredictably. State that must persist between events belongs in IndexedDB or Cache Storage, or must be re-derivable from the request. The lifecycle as a contract The lifecycle forces a question that traditional web deployments never asked: when you ship a new version, who gets it, and when? On a conventional website, the answer is everyone, on their next request. With a service worker, the answer is a policy you choose. That choice is part of the reliability contract this booklet is about. A worker that updates too eagerly breaks sessions in progress; one that updates too lazily strands users on old code and, worse, on old bugs. The practical guidance is to start with the prompt-to-reload pattern for anything with long-lived sessions, use natural turnover for simple content sites, reserve immediate skip-waiting for workers whose logic is simple and whose assets and APIs are backward compatible, and in every case monitor install and activation in production. With the machinery of control understood, the next chapter turns to what the worker actually does once it has control: deciding, for each request, where the response should come from. Chapter 3: Caching Strategies and the Case for Stale-While-Revalidate Once a service worker controls a page, every request the page makes passes through its fetch handler. The handler can respond in any way it likes, which is both its power and its danger. A handler that answers everything from cache produces a site that loads instantly and never changes. One that answers everything from the network produces a site that is no more reliable than it was before, plus the overhead of running a script for every request. The useful middle ground is a small set of named strategies, each of which makes a specific promise about freshness and availability. Choosing among them, request type by request type, is the core design activity in a PWA. The handler and the Cache Storage API The tools are simple. A fetch event exposes the outgoing Request and a method, respondWith(), that takes a promise for a Response. If the handler does not call respondWith(), the browser handles the request as if no service worker existed. The Cache Storage API, available as caches in both pages and workers, stores pairs of Request and Response objects in named caches. caches.match(request) searches all caches; cache.put(request, response) stores one. Two details trip up nearly everyone. First, a Response body is a stream that can be read only once. To both return a network response to the page and store it in a cache, you must call response.clone() and use one copy for each purpose. Second, matching is by URL and, by default, by the Vary header and HTTP method. Query strings are part of the URL, so /api/items?page=2 and /api/items?page=3 are separate entries; caches.match() accepts an ignoreSearch option when you want them treated as one. A third detail matters for anything served from another origin, such as images from a CDN or fonts from a font service. If the request is made without CORS, the response is "opaque": the service worker cannot read its status code or body. An opaque response for a 404 looks identical to one for a 200. Caching opaque responses therefore risks caching errors, and browsers also pad the quota accounting for each opaque response heavily, by several megabytes in Chromium, to avoid leaking the true size of cross-origin resources. A handful of cached opaque responses can consume a surprising share of an origin's quota. Request cross-origin resources with CORS where the server permits it, and be selective about caching the rest. The five basic strategies The industry has converged on five strategies, named in Google's offline cookbook from around 2014 and later codified in Workbox. Table 1 summarizes them before each is examined. Table 1. The five basic caching strategies and when each fits. Strategy Responds from Freshness Offline Typical use Cache only Cache Fixed until next deploy Yes Precached app shell Network only Network Always fresh No Payments, analytics, admin writes Cache first Cache, else network Stale until evicted Yes, once cached Hashed assets, fonts, images Network first Network, else cache Fresh when online Yes, last seen copy HTML, critical API data Stale-while-revalidate Cache, then refresh One visit behind Yes, once cached Avatars, feeds, config, non-critical API Cache only answers from the cache and never touches the network. It is appropriate for files precached during install, where the service worker's own lifecycle guarantees they are present and current for that version. Used for anything else it produces a page that silently fails, because a cache miss has no fallback. Network only forwards the request and does nothing else. It is the right choice for requests that must not be served stale or must not be replayed: payment submissions, analytics beacons (which have their own retry mechanisms), and any request where a cached answer would be wrong rather than merely old. In practice the simplest way to implement network only is to not call respondWith() at all. Cache first checks the cache, returns a hit immediately, and on a miss fetches from the network and stores the result. It is ideal for resources that never change at a given URL, which in a modern build pipeline includes every JavaScript and CSS file with a content hash in its name. A file called app.3f9a1c.js will always have the same content, so serving it from cache forever is correct. It is also reasonable for fonts and for images where staleness is harmless. It is dangerous for anything whose content changes at a stable URL, because the first version cached is the one users will see until the cache is cleared. Network first tries the network and falls back to the cache if the request fails. It suits HTML pages and API responses where users should see the latest content whenever they can, and where seeing yesterday's content offline is better than seeing nothing. Its weakness is the definition of "fails". A request on a terrible connection may take thirty seconds before failing, and the user stares at a blank page throughout. A network-first strategy without a timeout is a promise to be fresh at any cost, which is rarely the promise you want. The fix is to race the network against a timer and fall back to cache if the timer wins: async function networkFirst(request, cacheName, timeoutMs = 3000) { const cache = await caches.open(cacheName); const network = fetch(request).then((res) => { if (res.ok) cache.put(request, res.clone()); return res; }); const timeout = new Promise((resolve) => setTimeout(() => resolve(cache.match(request)), timeoutMs)); const winner = await Promise.race([network, timeout]); return winner || network; // timed out with no cached copy: keep waiting } The last line matters: if the timer fires and the cache has nothing, the strategy should not give up but keep waiting for the network, since there is nothing better to offer. And because the network request keeps running even after the timer wins, the cache is still updated when it eventually completes, so the next visit gets fresher content. Stale-while-revalidate answers immediately from the cache if it can and, in parallel, fetches a fresh copy from the network to update the cache for next time. If the cache is empty, it waits for the network. It is the subject of the rest of this chapter. Stale-while-revalidate in depth The name comes from an HTTP Cache-Control extension, defined in RFC 5861 in 2010, that lets a server tell caches they may serve a stale response for a set period while revalidating it in the background. The service worker version applies the same idea under the developer's control. async function staleWhileRevalidate(event, cacheName) { const cache = await caches.open(cacheName); const cached = await cache.match(event.request); const refresh = fetch(event.request).then((res) => { if (res.ok) cache.put(event.request, res.clone()); return res; }).catch(() => cached); event.waitUntil(refresh.then(() => {})); return cached || refresh; } self.addEventListener('fetch', (event) => { const url = new URL(event.request.url); if (url.pathname.startsWith('/api/feed')) { event.respondWith(staleWhileRevalidate(event, 'feed')); } }); The event.waitUntil() call is essential and is the most common omission in hand-written versions. Without it, the browser may terminate the service worker as soon as the response has been delivered, killing the background fetch before it updates the cache. With it, the worker stays alive until the refresh completes. The strategy's appeal is obvious: it combines the speed of cache first with a guarantee that content is never more than one visit out of date. For resources where "one visit behind" is acceptable, it is close to ideal. A user's avatar, the list of categories in a shop, a configuration file of feature flags, a news feed where the user will pull to refresh anyway: in each case, showing the previous version instantly and updating quietly is better than either showing nothing until the network responds or showing a version that never updates. The appeal also explains why stale-while-revalidate is the most misused strategy. Its promise is subtle, and teams apply it to resources where "one visit behind" is wrong. The first misuse is prices, stock levels, and anything transactional. If a product page shows a cached price and the refresh brings a different one, the user saw a price that was wrong at the moment they saw it. If they add to cart on the basis of it, the discrepancy surfaces at checkout. Transactional data should be network first with a timeout, and when served from cache it should be visibly marked as possibly out of date. The second misuse is HTML that references versioned assets. If the HTML for the app shell is served stale-while-revalidate and a deploy changes the asset hashes, the first load after the deploy serves old HTML referencing old assets. If those assets are still in cache, this is merely one visit behind. If they have been cleaned up, the page breaks. HTML should be either precached as part of the worker's version, so that HTML and assets change together, or served network first. The third misuse is the infinite staleness of infrequent visitors. "One visit behind" means something very different for a user who visits daily and one who visits every three months. The second user sees content three months old on every visit, then waits for the next visit to see the update. For data that ages badly, add a maximum age: store the fetch time alongside the response, for example in a custom header on the cached copy or in IndexedDB, and fall back to network first if the cached copy is older than a threshold. The fourth misuse is forgetting to tell the user. When the refresh brings content that differs from what is on screen, the page is now showing stale data while fresher data sits in the cache. For a feed, the right behaviour is often to tell the page, through postMessage or the BroadcastChannel API, that new content is available, and let it show a "New posts" pill rather than either swapping the content under the user's thumb or leaving it stale until the next visit. Workbox ships a plugin, BroadcastCacheUpdate, for exactly this. Composing strategies into a routing table A real service worker is a routing table that maps request patterns to strategies. A typical content or commerce PWA might use precache with cache only for the app shell and its hashed assets; cache first with an expiration limit for images; stale-while-revalidate for fonts' CSS, user avatars, and non-critical API responses such as category lists; network first with a three-second timeout for HTML navigations and product details; and network only for the checkout API, authentication endpoints, and analytics. Two rules keep such a table sane. First, match the most specific routes first and make the default explicit. A request that matches no route should fall through to the network, not to a catch-all cache-first rule that will one day cache something it should not. Second, never cache responses to requests that carry user-specific authorization unless the cache key reflects the user. A service worker cache is per origin, not per user; if two people share a device and log in one after the other, a cached /api/me response will show the first user's data to the second. Clearing user-specific caches on logout is the minimum; using network first or network only for personal data is often simpler. The non-GET methods deserve a word. The Cache Storage API only stores GET requests. POST, PUT and DELETE requests cannot be answered from cache, and a service worker that tries will throw. A worker that wants to do something useful with writes made while offline needs a different mechanism entirely, which is the subject of Chapter 6. The same table in Workbox Hand-written strategies are the best way to understand the mechanics, but a production routing table is usually expressed through a library. In Workbox, the routing table described above becomes a handful of declarations, each pairing a match condition with a strategy object and optional plugins: import { precacheAndRoute } from 'workbox-precaching'; import { registerRoute } from 'workbox-routing'; import { CacheFirst, NetworkFirst, StaleWhileRevalidate, NetworkOnly } from 'workbox-strategies'; import { ExpirationPlugin } from 'workbox-expiration'; precacheAndRoute(self.__WB_MANIFEST); // injected at build time registerRoute(({ url }) => url.pathname.startsWith('/api/checkout'), new NetworkOnly()); registerRoute(({ request }) => request.mode === 'navigate', new NetworkFirst({ cacheName: 'pages', networkTimeoutSeconds: 3 })); registerRoute(({ request }) => request.destination === 'image', new CacheFirst({ cacheName: 'images', plugins: [ new ExpirationPlugin({ maxEntries: 60, maxAgeSeconds: 30 * 86400 }) ] })); registerRoute(({ url }) => url.pathname.startsWith('/api/categories'), new StaleWhileRevalidate({ cacheName: 'api-swr' })); The self.__WB_MANIFEST placeholder is replaced by the build tool with a list of every file in the build output and a revision hash for each, which is how the service worker's bytes change exactly when the app changes. The networkTimeoutSeconds option implements the race against a timer shown earlier. The routes are checked in registration order, which is why the network-only checkout route comes first. Requests that match no route are not intercepted at all, which gives the explicit network default recommended above. The library does not change the design questions. It makes the answers shorter to write and harder to get subtly wrong, particularly the response cloning, the waitUntil calls, and the expiration bookkeeping. A team that cannot explain what each line would do if written by hand should spend an afternoon writing the hand-rolled versions before relying on the abstraction. Requests that do not fit the strategies Some request types defeat the standard strategies and need special handling. Audio and video. Media elements fetch with Range headers, asking for a byte range of the file rather than the whole thing, so that playback can start before the download finishes and the user can seek. A cached full response does not satisfy a range request unless the service worker slices it, constructing a 206 Partial Content response with the requested bytes. Workbox has a RangeRequestsPlugin for exactly this. Without it, cached media either fails to play offline or, on some browsers, plays only from the beginning. For large media, explicit download into the cache under user control, with progress shown, is usually better than opportunistic caching. Server-sent events, WebSockets and long polling. These are streams, not request-response pairs, and caching them makes no sense. WebSocket connections do not pass through the service worker at all. Server-sent event requests do, and should be left alone by the routing table. Requests with credentials to third parties. Authentication flows that redirect through an identity provider, such as OAuth sign-in, pass through navigation requests and redirects. A navigation handler that caches or rewrites these, or that serves the app shell for the callback URL before the server has set a session cookie, breaks sign-in in ways that are hard to diagnose. Exclude authentication paths from navigation handling explicitly. Very large API responses. A search endpoint that returns megabytes of results per query, cached under stale-while-revalidate with the query string as part of the key, fills the cache with entries that will rarely be requested again. Either exclude such endpoints or cap them tightly with expiration rules. Strategy as promise Each strategy is a promise to the user stated in a few lines of code. Cache first promises speed and accepts that the content may be old. Network first promises freshness and accepts that the user may wait. Stale-while-revalidate promises speed now and freshness next time. Network only promises correctness and accepts failure when offline. Writing a routing table is the act of deciding, for each kind of thing the app shows, which promise is appropriate, and the misuse cases above all come from making a promise the data cannot support. These per-request promises are necessary but not sufficient. A user who opens the app offline does not care whether each request individually had a sensible fallback; they care whether the app as a whole does something coherent. The next chapter moves from strategies for requests to the design of an offline experience. Chapter 4: Designing the Offline Experience A routing table of sensible caching strategies does not by itself produce a good offline experience. Picture a user opening a recipe app on a plane. The app shell loads from cache, so the header and navigation appear instantly. The home screen fetches a list of recipes with network first; it times out and falls back to a cached list from last week. The user taps a recipe that was never cached. The request fails. The page shows a spinner forever, or a JavaScript error, or a blank card. Every request had a strategy, yet the experience is broken, because nobody decided what the app should do for a resource it has never seen while the network is gone. Designing offline is the work of making those decisions for the app as a whole. It has three parts: deciding what must work offline, structuring the app so that the essential parts load without the network, and storing the data those parts need in a form the app can query. Deciding what must work offline The first decision is not technical. It is a product decision about which tasks the user should be able to complete without a connection, and the honest answer for most apps is "some of them". Trying to make everything work offline is expensive and often impossible: a search across a million products cannot be served from a phone's storage, and a live chat cannot send messages to someone who will not receive them until later without the user understanding that. A useful way to make the decision is to sort the app's features into four tiers. 1. Must work fully offline. The core loop that justifies the app. For a field-inspection tool, filling in an inspection form and attaching photos. For a reading app, reading saved articles. For a note-taking app, creating and editing notes. 2. Must work with recently seen data. Viewing things the user has already looked at, such as order history, recent messages, or the last loaded dashboard, with a clear indication that the data may be out of date. 3. Must fail gracefully. Features that genuinely require the network, such as search, checkout, or live data, should show a clear, specific message rather than a generic error, and should never lose what the user typed. 4. Can fail. Non-essential extras, such as recommendations, ads, or social widgets, can simply be absent when offline. The tiers translate directly into engineering. Tier 1 requires precaching, local data storage, and an outbox for writes. Tier 2 requires runtime caching of responses or of data in IndexedDB. Tier 3 requires error states designed in advance and form state preserved locally. Tier 4 requires only that the app does not break when those requests fail, which is usually a matter of not letting a failed fetch propagate as an unhandled exception. The app shell and the navigation problem The most influential structural pattern for PWAs is the app shell: the minimal HTML, CSS and JavaScript needed to draw the app's user interface frame, precached at install time and served from cache on every launch, with content loaded into it afterwards. It suits single-page applications particularly well. The shell loads instantly from cache regardless of the network, and the app then fetches data and renders it. Offline, the shell still loads, and the app can render cached data or a meaningful empty state. For a single-page application, the service worker implements the shell by answering every navigation request with the same cached HTML file: self.addEventListener('fetch', (event) => { if (event.request.mode === 'navigate') { event.respondWith( caches.match('/index.html', { cacheName: PRECACHE }) .then((res) => res || fetch(event.request)) ); } }); The request.mode === 'navigate' check identifies requests for top-level documents, as opposed to images, scripts, or API calls. Answering them all with the same shell means that a user who opens a deep link like /recipes/481 gets the shell instantly and the client-side router renders the right view. Exclusions matter here: navigations to server-rendered routes that are not part of the single-page app, such as /admin or /auth/callback, must be allowed through to the network, or the shell will intercept a login redirect and break it. The shell pattern has a cost that took the industry a few years to recognize. A shell renders a frame, and then the app must fetch data and render content, which means the content the user wants arrives only after JavaScript has downloaded, parsed, and run, and after at least one network round trip for data. On a fast device that is unnoticeable. On a low-end phone it pushes Largest Contentful Paint well back, as Chapter 8 discusses. The shell made repeat visits instant to the first pixel and sometimes slower to the first useful pixel. Server-rendered and multi-page sites take a different approach. Each navigation returns full HTML with content already in it, which is good for first paint and for search engines, but means there is no single shell to precache. For these sites the usual strategy is network first with a timeout for navigations, caching each page as the user visits it, plus an offline fallback page for navigations that fail with nothing cached: async function handleNavigation(event) { try { const res = await fetch(event.request); const cache = await caches.open('pages'); cache.put(event.request, res.clone()); return res; } catch { return (await caches.match(event.request)) || (await caches.match('/offline.html')); } } The offline fallback page is the minimum viable offline experience and every PWA should have one, precached at install. It should look like the site, explain plainly that the page is not available offline, and, where possible, list pages that are available, which the page can discover by reading the pages cache's keys. A news site might list the articles already cached; a store might offer the user's saved list. The difference between a branded page saying "You are offline; here are the five articles you can read" and the browser's default error is the difference between an app and a website that happens to have a service worker. A third approach, used by some content sites, is streaming composition: the service worker stitches together a cached header and footer with a network-fetched content fragment, using the Streams API to send the cached header to the browser immediately while the content is still arriving. This combines the instant first paint of the shell model with server-rendered content. It is powerful and relatively complex, and most teams will get more from navigation preload, covered in Chapter 8. Structured data belongs in IndexedDB Cache Storage stores HTTP responses. It is the right place for files and for API responses you intend to replay verbatim. It is the wrong place for data the app needs to query, filter, update, or merge. If the app shows a list of a user's 500 notes and the user edits one offline, you do not want to rewrite a cached JSON blob of all 500; you want to update one record. IndexedDB is the browser's transactional object database, available in both pages and service workers. It stores structured JavaScript objects, supports indexes for efficient lookup, and supports transactions so that a set of related writes either all happen or none do. Its native API is event-based and notoriously awkward, so most applications use a thin promise wrapper; Jake Archibald's idb library, at around a kilobyte, is the common choice, and Dexie offers a richer query layer. import { openDB } from 'idb'; const db = await openDB('fieldnotes', 2, { upgrade(db, oldVersion) { if (oldVersion < 1) { const notes = db.createObjectStore('notes', { keyPath: 'id' }); notes.createIndex('byUpdated', 'updatedAt'); } if (oldVersion < 2) { db.createObjectStore('outbox', { keyPath: 'opId' }); } }, }); await db.put('notes', { id: crypto.randomUUID(), body: 'Pump 3 leaking', updatedAt: Date.now(), synced: false }); const recent = await db.getAllFromIndex('notes', 'byUpdated'); The upgrade callback is IndexedDB's schema migration mechanism. Every change to object stores or indexes requires bumping the database version and handling the upgrade from each older version, because a user might arrive with any older version on disk. The cascading if (oldVersion < n) pattern handles that. The upgrade has a lifecycle hazard of its own that mirrors the service worker's. If one tab has the database open at version 1 and another tab tries to open it at version 2, the upgrade is blocked until the first tab closes its connection. Pages should listen for the versionchange event on their open connection and close it, then prompt the user to reload, or the new version will hang. Teams that deploy new database versions without handling this discover it when users report that the app "stopped loading" after an update. Local-first thinking Once an app stores its working data in IndexedDB, a different architecture becomes available: read and write locally first, always, and synchronize with the server in the background. The interface never waits for the network to show or save data. The network becomes a replication channel rather than the source of every answer. This local-first model makes the app fast and resilient by construction, because the common path never touches the network. It also introduces problems that request-response apps never had to solve. The most important are identity, ordering, and conflict. Identity: records created offline need identifiers before the server sees them. Generate them on the client with crypto.randomUUID() rather than relying on server-assigned auto-increment IDs, so that a note created offline can be referenced, edited, and linked before it syncs. Ordering: if a user creates a record and then edits it twice while offline, the server must receive those operations in order or reconcile them. Storing writes as an ordered log of operations, rather than only the final state, makes ordering explicit. Conflict: if two devices edit the same record while one of them is offline, someone must decide what wins. The simplest policy, last write wins by timestamp, is easy to implement and silently loses data, because device clocks drift and the "later" edit may not be the one the user wanted. Field-level merging, where edits to different fields of the same record both survive, is a substantial improvement for form-like data. For collaborative text, conflict-free replicated data types (CRDTs), implemented in libraries such as Yjs and Automerge, guarantee that all replicas converge to the same state without a central arbiter, at the cost of storage overhead and a more complex data model. The choice depends on how likely conflicts are and how costly they are. A single-user note app on one phone rarely conflicts and can use last write wins. A shared inventory tool used by a team across devices must handle conflicts explicitly and should show the user when an edit was overridden. Whatever the policy, it should be decided deliberately and documented, because it is part of what the app promises the user about their data. Worked example: a field inspection app It helps to apply these ideas to one concrete case. Consider a PWA used by building inspectors who visit plant rooms, basements, and rooftops, where mobile signal is often absent. Each inspection consists of a checklist of about fifty items, free-text notes, and photographs, and must be submitted to the office system by the end of the day. The tiering is straightforward. Tier 1, fully offline, covers opening the day's assigned inspections, completing checklists, writing notes, and taking photos. Tier 2, recently seen data, covers previous inspections of the same site for reference. Tier 3, graceful failure, covers searching the full archive and reassigning jobs, which require the server. Tier 4 covers the news feed on the home screen. The service worker precaches the app shell, including the checklist rendering code and the template definitions for every inspection type, so a new kind of inspection can be started offline. Each morning, when the inspector opens the app on the office Wi-Fi, the app fetches the day's assignments and the last two inspections for each site and writes them to IndexedDB. The interface reads exclusively from IndexedDB; it never waits on the network to display a checklist. Every checklist answer is written to IndexedDB as the inspector taps it, not when they press "Save", so a crash or a closed tab loses nothing. Photos are the hard part. A modern phone camera produces images of several megabytes, and fifty inspections with ten photos each would fill a meaningful share of storage and take a long time to upload on a weak connection. The app therefore resizes each photo on the device, using a canvas or createImageBitmap in a Web Worker, to a resolution adequate for the report, stores the result as a Blob directly in IndexedDB, which accepts binary data, and records it in the outbox as a separate upload operation that depends on the inspection record existing on the server. When the inspector presses "Submit", the inspection is marked complete locally and its operations are queued. The interface shows the inspection as "Waiting to send" with a count of pending photos. The outbox, described in Chapter 6, sends the record first, then the photos one at a time, so that a weak connection can make progress in small increments rather than failing a single large request repeatedly. Only when the server confirms every part does the inspection show as "Sent", and only then are the local photos eligible for deletion. Conflicts are rare but real: an office manager might edit an inspection's metadata while the inspector is offline. The app uses field-level merging for metadata and treats checklist answers as owned by the inspector, so the inspector's answers always win, and any overridden metadata change is shown to the manager. That policy was a business decision, written down before the code. This design does not depend on any capability missing from Safari, which matters because many such teams issue iPhones. It uses precaching, IndexedDB, and in-page outbox draining, and it is honest with the inspector about what has and has not reached the office. Telling the user the truth The final element of offline design is communication. An app that works offline but does not say so leaves users unsure whether their work has been saved. An app that shows a large red "You are offline" banner every time a single request fails, including on a working connection, trains users to ignore it. Several practices help. Show connection state subtly and accurately, based on actual request failures rather than only on navigator.onLine, which reports whether the device has any network interface and says nothing about whether the server is reachable. Mark data served from cache with its age where age matters, such as "Updated 3 hours ago" on a dashboard. Show pending writes as pending, with a small indicator on each unsynced item and a count somewhere global, and confirm when they have been sent. Disable, rather than hide, actions that require the network, with a short explanation, so the user learns what will be possible when they reconnect. And never discard user input because a request failed: persist form state locally as the user types, so that a failed submission can be retried without retyping. These are interface details, but they are where the reliability contract becomes visible to users. A PWA that silently keeps its promises but never tells users what those promises are earns less trust than one that is honest about what works and what is waiting. All of this, however, depends on the data still being there when the user returns, and the browser offers fewer guarantees on that front than most developers assume. Storage limits and eviction are the subject of the next chapter. Chapter 5: Storage Quotas, Eviction, and Cache Hygiene An offline promise is only as good as the storage behind it. A native app writes to its sandbox and the data stays until the user deletes the app or the app deletes it. Web storage is different in kind. The browser owns it, decides how much each site may use, and may delete it when the device runs short of space or when a privacy policy says it should. A developer who designs an offline-first app without understanding those rules is making promises on behalf of a third party who has not agreed to them. This chapter covers how much a PWA can store, when the browser will delete it, how to ask for stronger guarantees, and how to keep a service worker's caches from growing without bound. What counts and how much is allowed Browsers group most storage under a single quota per origin. Cache Storage, IndexedDB, the Origin Private File System, and service worker registrations all count against it. The older Web Storage API, localStorage and sessionStorage, sits outside that system with its own small, fixed limit, around 5 to 10 megabytes depending on the browser, and is synchronous, so it blocks the main thread and is unavailable in service workers. It is suitable for small preferences and nothing else. The quotas for the main storage system have grown generous in recent years and, for most PWAs, size is no longer the binding constraint. Table 2 summarizes the current limits as documented by MDN. Table 2. Per-origin storage limits in major browser engines (source: MDN, "Storage quotas and eviction criteria"). Engine Best-effort quota Persistent quota Notes Chromium (Chrome, Edge) Up to 60% of total disk Up to 60% of total disk Browser-wide cap of 80% of disk Firefox Smaller of 10% of disk or 10 GiB Up to 50% of disk, capped at 8 TiB Best-effort is also subject to a group limit per site Safari (macOS 14+, iOS 17+) About 60% of disk for browser apps Same About 15% in other WebKit apps; cross-origin frames get about a tenth of the parent's quota Safari's numbers reflect a significant change WebKit announced in 2023, shipped in Safari 17. Before it, Safari capped origins at roughly a gigabyte, after which it would prompt the user, and many developers still quote older, stricter limits. The newer policy brought Safari broadly into line with other engines on size. Home Screen web apps on iOS get the browser-level quota. A PWA can ask the browser how much it is using and how much it may use: const { usage, quota } = await navigator.storage.estimate(); console.log(`Using ${(usage / 1e6).toFixed(1)} MB of ${(quota / 1e6).toFixed(0)} MB`); The figures are estimates, deliberately imprecise to avoid giving sites a fingerprinting signal about disk size, and they include padding for opaque responses. Chromium in particular pads each opaque cross-origin response by several megabytes in its quota accounting regardless of real size, so a PWA that caches a few hundred opaque images can report gigabytes of usage while consuming far less disk. This is one more reason, beyond the inability to check their status codes, to avoid caching opaque responses. When a write would exceed the quota, it fails with a QuotaExceededError. Cache Storage put() rejects; IndexedDB transactions abort. Code that writes to storage should handle that error, typically by evicting the oldest runtime cache entries and retrying, or by telling the user that offline storage is full. When browsers delete data Quotas govern how much a site may write. Eviction governs when the browser removes what was written, and it is the part that breaks offline promises. Storage in every major browser is by default best-effort. Best-effort data may be deleted by the browser without asking the user, under two main conditions. The first is storage pressure. When the device runs low on disk space, or when total browser storage exceeds the browser's overall cap, the browser evicts origins in least-recently-used order. Eviction removes all of an origin's data at once, not individual entries: caches, IndexedDB databases, and service worker registrations go together. This is deliberate, because partial eviction could leave an app in an inconsistent state, with an IndexedDB record pointing at a cached file that no longer exists. For developers it means that an offline app on a nearly full phone may lose everything the user stored in it, including unsynced work, if the user has not opened the app recently. The second applies only to Safari and is more surprising. When Safari's cross-site tracking prevention is enabled, which is the default, Safari deletes all script-writable storage for an origin, including IndexedDB, Cache Storage, service worker registrations and localStorage, if the user has not interacted with that origin for seven days of Safari use. Interaction means a click or tap on the site as a first party. The policy, introduced in 2020 with Safari 13.1 as part of Intelligent Tracking Prevention, targets trackers that use first-party storage to persist identifiers. It catches legitimate web apps in the same net. A user who installs a PWA from Safari in a browser tab, uses it intensively, and then does not open it for a week of browsing may return to find it empty. The important exception is that WebKit applies this seven-day clock differently to Home Screen web apps. When WebKit introduced the policy, it stated that web apps added to the Home Screen are not part of Safari and have their own counter of days of use, so their storage is cleared only if the web app itself goes unused for seven days of its own use; a Home Screen app that is not opened accumulates no days and so is not cleared on that basis. In practice, installed PWAs on iOS are far better protected than the same site in a Safari tab, which is one strong argument for encouraging iOS users to install. The practical consequence is that any data that exists only on the client is at risk, and the risk is highest for exactly the users an offline app is meant to serve: those who use it intermittently. Unsynced writes should be synchronized at the first opportunity, not batched for convenience. An app that holds a week of field data locally, waiting for the user to press "upload", is designed to lose it. Asking for persistence The Storage API lets a site ask for its storage to be made persistent, which exempts it from automatic eviction under storage pressure. Persistent data is removed only when the user explicitly clears it. if (navigator.storage && navigator.storage.persist) { const already = await navigator.storage.persisted(); if (!already) { const granted = await navigator.storage.persist(); reportPersistence(granted); } } How browsers decide whether to grant the request differs. Firefox shows the user a permission prompt. Chromium-based browsers and Safari decide automatically, without prompting, based on signals such as whether the site is installed, bookmarked, or frequently used, or has been granted notification permission. A request from a site the user has just visited for the first time is usually denied; the same request from an installed PWA is usually granted. Two pieces of advice follow. First, call persist() at a meaningful moment, such as after installation or after the user has created their first piece of local data, rather than on first page load, where it will usually fail and, in Firefox, show a prompt the user has no context for. Second, record whether persistence was granted and treat it as an input to how aggressively you sync. Persistence protects against pressure-based eviction; it does not protect against the user clearing site data, against Safari's tracking-prevention rules for non-installed sites, or against the device being lost. Cache hygiene Even with generous quotas, a service worker's caches need active management. Without it, runtime caches grow with every page visited and every image loaded, and old precaches pile up across versions. Large caches slow down caches.match(), consume the user's disk, and increase the chance of hitting the quota at an awkward moment. Version the precache and delete old versions at activation. Chapter 2 showed the pattern: include a version or hash in the precache's name, and in the activate handler delete any cache with the same prefix and a different name. Build-time tools do this automatically and, better, use per-file revision hashes so that a new version downloads only the files that actually changed. Bound runtime caches by count and age. A runtime cache for images should hold, say, the most recent 60 images and nothing older than 30 days. The Cache Storage API has no built-in expiry, so you track insertion time yourself, typically in IndexedDB keyed by URL, and prune when adding entries. Workbox's ExpirationPlugin implements exactly this, and its purgeOnQuotaError option deletes the cache automatically if a quota error occurs. Writing it by hand is straightforward but fiddly, and is one of the stronger arguments for using a library. Partition by purpose. Keep separate caches for precached shell assets, pages, images, and API responses. Separate caches let you apply different limits, clear one category without the others, and debug more easily, since the browser's developer tools list caches by name. Clear user data on logout. On a shared device, caches holding personal API responses and IndexedDB databases holding personal records must be cleared when the user signs out, or the next user may see them. Doing this in the page's logout handler and also in the service worker, in response to a message, covers both the open page and any future requests. Do not cache errors. Check response.ok before writing a network response to cache. A transient 500 from the server, cached under a cache-first strategy, becomes a permanent error for that user. The same applies to redirects for navigations in some browsers, and to the opaque responses discussed above. Worked example: sizing a reading app Quotas expressed as percentages of disk are hard to reason about until they are turned into an app's own numbers. Consider a news-reading PWA that wants subscribers to be able to read the day's edition on the underground. The precached shell, with JavaScript, CSS, fonts and icons, comes to about 1.5 megabytes after compression on the wire, somewhat more on disk because caches store decompressed bodies in some engines. A day's edition is around eighty articles. Article text as JSON averages perhaps 15 kilobytes; with one lead image each, resized for phone screens to around 100 kilobytes, an edition is roughly 9 megabytes. Keeping a week of editions for readers who fall behind brings the total to about 65 megabytes, plus a bounded runtime cache for images from browsing, capped at 200 entries, perhaps another 20 megabytes. Under any current quota on any device with a few gigabytes free, 85 megabytes is comfortably within limits. Size is not the problem. The design questions are elsewhere. Should the edition download automatically every morning, which on Chromium could use Periodic Background Sync for installed users but on iOS can happen only when the user opens the app? Should images download on cellular data, given that many users pay by the megabyte? What happens to the cache if the browser evicts the origin under storage pressure on a nearly full phone; does the app notice on next launch, show an empty state, and refetch, or does it assume the edition is present and break? And how is the week-old edition removed, by an expiration rule run on each launch or by a cleanup when the new edition is stored? Each of those is a promise decision. The default answers that tend to work are: download the edition when the app is opened on Wi-Fi or when the user explicitly asks, let users choose whether to download on cellular, always check for the presence of expected data rather than assuming it, and delete old editions in the same transaction that stores the new one, so storage never holds more than the declared budget. Detecting that data has gone Because eviction removes an origin's data wholesale and silently, an app cannot rely on being told. It has to notice. The simplest technique is a sentinel: when the app first stores data, it writes a small marker record to IndexedDB with an installation identifier and a timestamp, and it sends that identifier to the server along with the user's account. On every launch, the app checks for the marker. If the user is signed in, according to a cookie that survived, but the marker is missing, the local data has been cleared since the last session, whether by eviction, by the user clearing site data, or by a Safari privacy rule. What happens next is a design decision with real consequences. If the server holds everything, the app simply resynchronizes and the user notices at most a slower first load. If the server knows that the previous installation had unsynced operations, because it tracks the last acknowledged operation per installation and the client had reported a higher pending count, it can tell the user plainly that some work from a given date may not have been received, rather than leaving them to discover the gap. Reporting these events to analytics, segmented by browser and installed state, turns eviction from an invisible failure into a measured one, and often provides the evidence needed to justify encouraging installation or syncing more aggressively. Beyond the Cache API: the Origin Private File System For apps that need to store large binary files and manipulate them efficiently, such as photo editors, audio tools, or apps that run a database compiled to WebAssembly, the Origin Private File System offers a file-like storage area private to the origin. It is available in all major engines through navigator.storage.getDirectory(), counts against the same quota, and is subject to the same eviction rules. In dedicated Web Workers it offers synchronous access handles with high performance, which is what makes running SQLite in the browser practical. Several projects use this to give PWAs a real SQL database locally, an attractive option for apps with complex queries over local data. The persistence caveats in this chapter apply to it in full. The real offline budget The limits in this chapter suggest a way of thinking about an app's offline budget that is more useful than "how many megabytes may I store". The real budget has three dimensions. There is space: how much the browser will let you keep. For most apps it is now ample, and the constraint is courtesy to the user's device more than quota. There is time: how long the browser will let you keep it. Best-effort data can vanish under storage pressure, and on Safari, data for a non-installed site can vanish after a week of non-use. Persistence and installation extend this, but nothing makes it unconditional. And there is trust: how much of the user's work you are willing to hold on the client without a server copy. The answer should be as little as possible for as short a time as possible. Local storage is a buffer, not a vault. That third dimension is where the next chapter begins. If client storage is a buffer, then getting writes off the device and onto the server reliably, ideally even when the user has closed the app, becomes the critical path for every offline-capable PWA. Chapter 6: Offline Writes, the Outbox, and Background Sync Reading offline is the easy half of offline support. Caching strategies decide what the user sees; if they are wrong, the user sees something old. Writing offline is the hard half. When a user submits a form, sends a message, or saves an edit without a connection, the app has accepted responsibility for data that exists nowhere else. If the write is lost, the user's work is lost, and they will not know until much later, if ever. This chapter covers how to accept offline writes safely, how to deliver them reliably, and what the Background Sync API adds and where it is missing. The outbox pattern The foundation of reliable offline writes is the same on every platform, native or web: write the user's intent to durable local storage first, update the interface as if the operation succeeded, and send it to the server separately, retrying until the server acknowledges it. The local queue of pending operations is usually called an outbox. In a PWA, the outbox lives in IndexedDB. Each entry records an operation, not just a state: an identifier, the kind of operation, its payload, a timestamp, and a count of attempts. async function enqueue(db, op) { const entry = { opId: crypto.randomUUID(), createdAt: Date.now(), attempts: 0, ...op }; await db.put('outbox', entry); return entry.opId; } await enqueue(db, { type: 'createNote', path: '/api/notes', method: 'POST', body: { id: noteId, text } }); A separate function drains the outbox: it reads entries in creation order, sends each to the server, deletes it on success, and on failure increments the attempt count and stops, leaving remaining entries for the next attempt. async function drainOutbox(db) { const entries = await db.getAll('outbox'); entries.sort((a, b) => a.createdAt - b.createdAt); for (const e of entries) { const res = await fetch(e.path, { method: e.method, headers: { 'Content-Type': 'application/json', 'Idempotency-Key': e.opId }, body: JSON.stringify(e.body), }).catch(() => null); if (res && res.ok) { await db.delete('outbox', e.opId); continue; } if (res && res.status >= 400 && res.status < 500) { await quarantine(db, e, res.status); continue; // will never succeed by retrying } await db.put('outbox', { ...e, attempts: e.attempts + 1 }); throw new Error('retry later'); } } Three details in this sketch carry most of the reliability. The idempotency key is the operation's identifier sent as a header. Networks fail in the worst possible place: after the server has processed a request but before the client receives the response. The client sees a failure and retries; without an idempotency key the server creates a duplicate note, sends a second message, or charges twice. With one, the server records the keys it has processed and returns the original result for a repeat. This must be implemented on the server; the client can only supply the key. Payment providers such as Stripe popularized the pattern, and any API that accepts writes from an offline-capable client needs it. The distinction between retryable and permanent failures matters. A network error or a 5xx response may succeed later. A 400 Bad Request or a 422 validation error will fail forever; retrying it indefinitely blocks every entry behind it if the outbox is processed in order. Such entries should be moved aside to a quarantine store and surfaced to the user, who may need to correct something. Authentication failures, a 401, are a special case: they usually mean the session has expired, and the right response is to pause the outbox, prompt the user to sign in again, and then resume, not to discard the work. Ordering is preserved by processing entries in creation order and stopping at the first retryable failure. If operations are independent, parallel sending is faster, but most real outboxes contain dependent operations, such as creating a record and then attaching a photo to it, that must arrive in sequence. When to drain the outbox The outbox makes writes durable. The remaining question is when to try sending them. Without any special API, a page can try at several moments: immediately after enqueueing, when the browser fires the online event, when the page becomes visible again through the visibilitychange event, on app start, and on a timer with exponential backoff while entries remain. That covers the common cases well, and it works in every browser. It has one gap. All of these triggers require the page to be open. If the user submits a form on a train, sees it marked as pending, and closes the app before the train leaves the tunnel, nothing will send the write until they open the app again, which might be days later, by which time the data may matter less or, on Safari without installation, may have been evicted. The Background Sync API The Background Sync API closes that gap, in the browsers that support it. A page registers a sync request with a tag. The browser promises to fire a sync event at the service worker when it believes connectivity is available, immediately if it already is, and to do so even if the page has since closed. // In the page, after enqueueing const reg = await navigator.serviceWorker.ready; if ('sync' in reg) { await reg.sync.register('outbox'); } else { drainOutbox(db).catch(() => scheduleRetry()); } // In sw.js self.addEventListener('sync', (event) => { if (event.tag === 'outbox') { event.waitUntil(openDb().then(drainOutbox)); } }); If the promise passed to waitUntil() rejects, the browser treats the sync as failed and schedules a retry with backoff. Chromium retries only a small number of times, and on the final attempt the event's lastChance property is true. The handler can use that to record that background delivery failed, so the page can surface the pending work next time it opens. Registering the same tag again while one is pending does not create a second sync, which makes it safe to call register('outbox') after every enqueue. The API has real limits even where it is supported. The browser decides when to fire the event, and it does not promise immediacy. The service worker still has a bounded amount of time to complete its work before the browser terminates it, on the order of minutes, so it suits small writes rather than large uploads. And the sync fires once connectivity returns, which on an unstable connection may mean the connection drops again mid-request. Browser support is the decisive limit. Background Sync is implemented in Chromium-based browsers, including Chrome, Edge, Opera and Samsung Internet, and has been since Chrome 49 in 2016. Firefox has not shipped it. Safari has not implemented it on any platform, including iOS, and the specification remains a draft in the Web Platform Incubator Community Group rather than a W3C standard. As of late 2026, any PWA that must work on iPhones cannot rely on Background Sync, which means the in-page triggers described above are not a fallback but the primary mechanism, with Background Sync as an enhancement for Chromium users. Periodic Background Sync and Background Fetch Two related Chromium APIs are often confused with Background Sync. Periodic Background Sync lets an installed PWA ask to be woken at intervals to refresh content, for example to download the morning's news before the user opens the app on the train. The site requests a minimum interval, and the browser decides the actual frequency based on how engaged the user is with the site. In Chrome it requires the PWA to be installed, fires only on networks the device has used before, and fires rarely or not at all for sites the user seldom opens. It shipped in Chrome 80 in 2020, and no other engine has implemented it. It is useful for content freshness on Android, but it cannot be used for anything that must happen on a schedule. Background Fetch lets a page hand a large download, or a set of downloads, to the browser, which performs it with a visible progress notification, continues even if the page closes, and wakes the service worker with the result. It is designed for content such as podcasts, videos or offline map tiles. It too is Chromium-only. Table 3 sets out the three background mechanisms alongside the in-page approach. Table 3. Ways to do work when the network returns or the page closes. Mechanism Runs when page closed Trigger Engines Fit In-page retries No online, visibility, timer, app start All Baseline for every outbox Background Sync Yes Connectivity returns Chromium only Small queued writes Periodic Background Sync Yes Browser-chosen interval, installed apps only Chromium only Prefetching content Background Fetch Yes Explicit large download Chromium only Media and large files Queueing requests versus queueing operations There are two ways to build an outbox, and the choice affects everything downstream. The first is to queue requests. The service worker intercepts failed POST requests in its fetch handler, serializes them, stores them in IndexedDB, and replays them later, returning a synthetic response to the page in the meantime. This is what Workbox's BackgroundSyncPlugin does: attach it to a network-only route for an API path, and any request that fails is queued and retried on the next sync event, or, in browsers without Background Sync, each time the service worker starts up. It is attractive because the application code does not change; the page makes ordinary fetch calls and the worker makes them durable. import { registerRoute } from 'workbox-routing'; import { NetworkOnly } from 'workbox-strategies'; import { BackgroundSyncPlugin } from 'workbox-background-sync'; registerRoute(({ url }) => url.pathname.startsWith('/api/notes'), new NetworkOnly({ plugins: [ new BackgroundSyncPlugin('notes-queue', { maxRetentionTime: 24 * 60 }) ] }), 'POST'); The maxRetentionTime option, in minutes, sets how long a queued request is kept before it is discarded, which is itself a policy decision about how long the app will keep trying on the user's behalf. The second is to queue operations. The application writes a description of what the user did, such as "create note with this ID and this text", to the outbox, and a sync routine translates it into requests. This is the approach shown earlier in this chapter. Request queueing is quick to add and works well for fire-and-forget writes, such as logging an event, submitting feedback, or adding an item to a list where the order and the result do not matter to the interface. It has three weaknesses for anything more complex. The page does not know what is queued, because the queue lives in the worker and holds opaque requests, so the interface cannot show pending items accurately without extra messaging. Requests captured with an authentication token may be replayed after the token has expired. And requests are frozen at the moment of capture, so if the user edits a pending note twice, the queue holds three requests that must all succeed in order, rather than one operation describing the final state. Operation queueing requires the application to be designed around the outbox, but it gives the interface full visibility, lets the sync routine attach a fresh token at send time, and allows pending operations to be coalesced, so that a create followed by two edits can be sent as a single create with the final content. For any app where offline writing is a core feature rather than an edge case, operation queueing is the better foundation. What the server must do Offline writes change what the server sees, and an API designed only for online clients often mishandles them. Writes arrive late. A note created at 09:12 may reach the server at 16:40. If the server stamps records with the time it received them, reports and sorting will be wrong. The client should send the time the user acted, and the server should store both that and the receipt time. Writes arrive in bursts. An inspector who reconnects after a day underground may send hundreds of operations in a minute. Rate limits tuned for interactive use can reject them, and the client will back off and retry, slowing the drain further. Either raise limits for authenticated sync traffic or provide a batch endpoint that accepts many operations in one request. Writes conflict. An update may refer to a record that another user has since changed or deleted. The server must respond with something the client can act on: a 409 Conflict with the current server version, so the client can apply its merge policy, or a 410 Gone for a deleted record, which the client should surface to the user rather than silently drop. An API that responds to every problem with a generic 500 forces the client to retry forever. Writes repeat. As described above, every write endpoint used by an offline client must honour idempotency keys, because retries after ambiguous failures are a normal part of operation rather than an exceptional case. Designing for the iPhone without background sync Because Safari lacks all three background mechanisms, teams building PWAs that need reliable offline writes on iOS should plan around their absence rather than hope for them. The first technique is sending eagerly. Attempt delivery immediately on every enqueue and on every opportunity the page has. Most writes made "offline" are made during brief connectivity drops and will succeed within seconds if the app keeps trying while it is open. The second is making pending work visible. If the user can see that three items have not been sent, they are far more likely to open the app later, or to keep it open until the counter reaches zero. Field workers using a well-designed offline tool learn to check for a green indicator before closing it, in the same way they learn to save a document. The third is using push to prompt. If the server knows the user has been working, for example because it received a partial sync or because the user's schedule says an inspection was due, it can send a push notification reminding them to open the app and complete the sync. Opening the app via the notification gives the page a chance to drain the outbox. This turns push, which iOS does support for installed web apps, into a substitute for the background sync it does not. The fourth is flushing at page hide. When the page is about to be hidden or closed, the visibilitychange event fires with document.visibilityState set to hidden, and a page can use fetch() with the keepalive option, or navigator.sendBeacon(), to send small final requests that the browser will complete after the page is gone. Both have payload limits, roughly 64 kilobytes in total for keepalive requests in flight, and no way to report success back to the page, so they cannot replace the outbox: the entries must stay queued until a later confirmed sync removes them. They do, however, get many final writes to the server in the cases where the connection is up as the user leaves. The write path as the heart of the contract The previous chapters described promises about what users see. The write path carries promises about what users did, and those are weightier. A user who sees stale data is mildly inconvenienced. A user whose inspection report, order, or message vanishes has been let down in a way they will remember. The architecture that keeps that promise is the same everywhere: durable local outbox, idempotent server endpoints, careful separation of retryable and permanent failures, and every available trigger for delivery, with Background Sync as one trigger among several rather than the foundation. Built that way, the app behaves well on Chrome for Android, where the background mechanisms fill in the gaps, and still behaves honestly on an iPhone, where the user can see what is waiting and the app takes every chance it gets to send it. One of those chances comes from a capability iOS gained only recently and which now works across every major platform: the ability for a server to wake the app with a message. Push notifications are the subject of the next chapter. Chapter 7: Push Notifications That Users Keep Push notifications are the capability that most changed the case for PWAs. For years the standard objection to building a web app instead of a native one was "but we need push", and on the iPhone that objection was decisive. Since Apple shipped Web Push for Home Screen web apps in iOS and iPadOS 16.4 in March 2023, every major platform supports it. The technical machinery is now well understood and broadly interoperable. What remains hard is not sending a notification but earning the permission to send one and keeping it, because browsers have spent years building defences against sites that abuse it. How web push works Web push combines two specifications: the Push API, which lets a service worker receive messages from a server, and the Notifications API, which lets it show them. Between your server and the user's browser sits a push service operated by the browser vendor: Google's Firebase Cloud Messaging infrastructure for Chrome, Mozilla's push service for Firefox, Apple's push service for Safari, and so on. Your server never talks to the browser directly. It sends a message to the push service, addressed by an endpoint URL unique to that browser installation and site, and the push service delivers it when the device is reachable. The flow has four steps. 1. Subscribe. The page, after the user grants notification permission, asks the service worker registration's pushManager to subscribe. The browser contacts its push service and returns a PushSubscription containing an endpoint URL and two keys used for encryption. 2. Store. The page sends the subscription to your server, which stores it against the user. 3. Send. When there is something to say, your server encrypts the payload with the subscription's keys, signs a JSON Web Token with its own private key to prove its identity, and POSTs to the endpoint. 4. Receive. The push service wakes the browser, the browser wakes the service worker, and the worker's push handler shows a notification. // In the page, inside a click handler const permission = await Notification.requestPermission(); if (permission === 'granted') { const reg = await navigator.serviceWorker.ready; const sub = await reg.pushManager.subscribe({ userVisibleOnly: true, applicationServerKey: VAPID_PUBLIC_KEY, // base64url-decoded Uint8Array }); await fetch('/api/push/subscribe', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(sub) }); } // In sw.js self.addEventListener('push', (event) => { const data = event.data ? event.data.json() : {}; event.waitUntil(self.registration.showNotification(data.title || 'Update', { body: data.body, icon: '/icons/192.png', badge: '/icons/badge-72.png', tag: data.tag, data: { url: data.url || '/' }, })); }); self.addEventListener('notificationclick', (event) => { event.notification.close(); event.waitUntil(clients.openWindow(event.notification.data.url)); }); The server identity mechanism is called VAPID, Voluntary Application Server Identification, specified in RFC 8292. The server generates an elliptic-curve key pair once, gives the public key to browsers as the applicationServerKey, and signs each request with the private key. A subscription created with one VAPID public key will accept messages only from a server holding the matching private key, so a leaked endpoint cannot be used by someone else to spam the user. Losing the private key, or rotating it carelessly, invalidates every existing subscription, which makes it a secret to manage with the same care as a database credential. Payload encryption, specified in RFC 8291, ensures that the push service, which relays the message, cannot read it. The scheme is fiddly enough that almost nobody implements it by hand; mature libraries exist for every major server language, such as web-push for Node.js and pywebpush for Python. Payloads are limited to around 4 kilobytes, which is enough for a title, a body, and a URL but not for content. The idiomatic pattern is to send a small payload with an identifier and let the service worker fetch details if needed, bearing in mind that it must still show a notification promptly. The request to the push service also carries headers that are easy to overlook and worth setting deliberately. TTL says how long, in seconds, the push service should hold the message if the device is offline; a breaking news alert might have a TTL of an hour, a chat message a day. Urgency lets the push service decide whether to wake a device in a low-power state. Topic lets a newer message replace an undelivered older one with the same topic, so a user who has been offline does not receive ten stale score updates on reconnecting. The userVisibleOnly rule The userVisibleOnly: true option in the subscription call is not optional in practice. Chrome requires it and Safari enforces the same rule. It is a promise that every push message will result in a visible notification. Browsers enforce it because silent pushes would let a site wake its service worker in the background at will, to track the user or drain the battery, without the user's knowledge. The enforcement is real. If a service worker's push handler does not show a notification, Chrome shows a generic one on the site's behalf saying that the site has been updated in the background, which looks broken to users. Safari goes further: developers have documented iOS revoking a site's push subscription after as few as three push events that did not produce a notification. Every push handler must therefore call showNotification() inside event.waitUntil(), on every code path, including error paths. A handler that tries to fetch details, fails, and returns without showing anything is a subscription-killing bug on iOS. This rule also rules out a pattern many teams want: using push to trigger silent background data refreshes. On the web, push is for notifying the user, and every message must be worth a notification. The closest thing to a background refresh is Periodic Background Sync, which, as Chapter 6 explained, exists only in Chromium and only for installed apps. Push on iOS and iPadOS Apple's implementation, introduced in iOS 16.4, uses the same standard APIs as other browsers, so code written for Chrome works with no Apple-specific branches. Apple's push service endpoints simply appear as subscription endpoints, and standard libraries handle them. There are, however, conditions that shape product design. Installation is required. Web push on iOS and iPadOS is available only to web apps added to the Home Screen. A site in a Safari tab cannot request notification permission at all. This means the path to push for an iPhone user is longer: visit the site, choose Share and then Add to Home Screen, open the installed app, and then grant permission. A PWA targeting iOS needs an in-app explanation of how to install, shown only to iOS Safari users who are not already in standalone mode, which the page can detect with the display-mode: standalone media query or the older navigator.standalone property. Permission requires a user gesture. The permission prompt can be triggered only in response to a direct user action, such as a tap on a button. Calling requestPermission() on page load does nothing on iOS. This is good practice everywhere and mandatory on Apple's platforms. Notifications behave as native ones. On iOS, web app notifications appear in Notification Center and on the lock screen, respect Focus modes, and can be managed per app in Settings, just like notifications from App Store apps. The Badging API, which sets a number on the app icon, is also supported for Home Screen web apps. In iOS 18.4, Apple added Declarative Web Push. In this format the push message itself is a JSON document, marked with "web_push": 8030 in reference to the Web Push RFC, that contains the notification's title, body, and a navigate URL to open on tap. The browser can display such a message without running any JavaScript at all. If a service worker exists, it receives the push event with the proposed notification and may replace it; if it fails to do so in time, the declarative notification is shown as a fallback. This removes the silent-push penalty as a failure mode, because there is always a notification to show, and it is more energy efficient. As of late 2026 it is a WebKit feature, also available in Safari on macOS from version 15.5, and a server can send the declarative format to Apple endpoints and the classic format elsewhere, or send the declarative format everywhere with a service worker that handles it for browsers that do not interpret it natively. Earning and keeping permission The technical side of push is a solved problem. The human side is where most implementations fail. Browsers have watched sites abuse notification prompts for a decade, and they have responded. Chrome introduced a quieter permission interface in 2020 that replaces the prompt with a small icon in the address bar for sites with low acceptance rates, or for users who routinely deny permission. Sites that show prompts on page load, before the user has done anything, are exactly the sites whose acceptance rates trigger it. In October 2025, Google announced that Chrome on desktop and Android would automatically revoke notification permission from sites that send many notifications and receive little engagement, notifying the user when it does so. Installed web apps are exempt from that revocation, which is a concrete advantage of installation on Chromium as well as on iOS. The design principles that follow are not new, but they are now enforced. Ask in context, after value. Never prompt on first page load. Offer notifications at the moment the user would obviously want one: after they place an order ("Notify me when it ships"), follow a topic, or start a conversation. Show your own interface element first, explaining what notifications will contain and how often, and call requestPermission() only when the user taps it. A user who dismisses your own pre-prompt can be asked again later; a user who denies the browser's prompt usually cannot, because the browser remembers the denial. Send what you promised. If the pre-prompt says "only when your order ships", send only that. The fastest way to lose permission is to treat it as a marketing channel after obtaining it for transactional reasons. Use tags and replacement. The tag option on a notification makes a newer notification with the same tag replace an older one instead of stacking. A chat app should collapse multiple messages from the same conversation into one updated notification. The renotify option controls whether the replacement alerts the user again. Handle clicks well. The notificationclick handler should focus an existing window showing the relevant content if there is one, rather than opening a new window each time. It can find existing windows with clients.matchAll({ type: 'window' }) and call focus() and navigate() on the best match. Landing on the home page rather than the specific item the notification was about is a common and avoidable frustration. Clean up dead subscriptions. When a push service responds to a send with a 404 or 410 status, the subscription is gone: the user revoked permission, uninstalled the app, or cleared data. The server should delete it immediately. Continuing to send to dead endpoints wastes resources and, at scale, can lead push services to treat the sender as poorly behaved. The browser may also replace a subscription and fire a pushsubscriptionchange event in the service worker; support for that event is uneven, so the page should also re-check the current subscription on each app start and send it to the server if it has changed. Badges, actions, and quieter signals Not every update deserves an interruption. The Badging API offers a quieter channel: navigator.setAppBadge(count) sets a number or dot on the installed app's icon, and navigator.clearAppBadge() removes it. It works for installed PWAs on desktop Chromium, on macOS, and on iOS and iPadOS for Home Screen web apps since 16.4, and it can be called from the service worker as well as the page, so a push handler can update the badge while showing a notification. On iOS, setting a badge requires notification permission. A messaging app might show a notification for direct messages but only update the badge count for group activity, giving users a sense of what is waiting without an alert every few minutes. Notification actions, buttons such as "Reply" or "Archive" shown on the notification itself, are supported in Chromium on Android and desktop through the actions option, with the chosen action reported in notificationclick as event.action. Safari does not display them. Design every notification so that tapping its body does something sensible, and treat actions as shortcuts for platforms that show them. Finally, measure. Record when notifications are shown, clicked, and closed without a click, by tag or category, and watch for categories whose click-through rate is near zero. Those are the notifications that train users to revoke permission, and on Chromium they are the pattern Google's auto-revocation targets. Cutting a low-value category usually raises the value of everything else the app sends. The sending side The server half of push is where operational problems accumulate, and it is often built hastily as an afterthought to the client work. A user does not have one subscription. They have one per browser, per device, per installed app: a phone's installed PWA, the same site in a desktop browser, and a tablet each produce separate endpoints. The server should store subscriptions in their own table, keyed by endpoint, with a reference to the user, the time of creation, the user agent at subscription time, and the time of the last successful delivery. Sending to a user means sending to all their live subscriptions, and the notification's tag keeps a user who reads on one device from seeing a stale duplicate on another for long, though cross-device dismissal is not something web push provides. import webpush from 'web-push'; webpush.setVapidDetails('mailto:ops@example.com', VAPID_PUBLIC, VAPID_PRIVATE); async function notifyUser(userId, payload) { const subs = await db.subscriptions.forUser(userId); await Promise.all(subs.map(async (s) => { try { await webpush.sendNotification(s.subscription, JSON.stringify(payload), { TTL: 3600, urgency: 'normal', topic: payload.tag }); await db.subscriptions.markDelivered(s.endpoint); } catch (err) { if (err.statusCode === 404 || err.statusCode === 410) { await db.subscriptions.delete(s.endpoint); } else { throw err; } } })); } The mailto: contact in the VAPID details gives push service operators a way to reach the sender if its traffic causes problems. It should be a monitored address. For anything beyond small volumes, sending belongs in a background job queue rather than in the request that triggered it. A breaking-news alert to a million subscribers is a million encrypted HTTP requests to several push services, each of which may rate-limit or fail transiently. A queue with retries, concurrency limits per push service, and backoff on 429 Too Many Requests responses keeps a spike from overwhelming either your servers or the push services. Encryption is computationally non-trivial at that scale, which is one reason commercial push providers exist; for most applications a well-built queue with a standard library is enough. Finally, treat delivery as best-effort. Push services may delay or drop messages, devices may be off, and the user may have muted notifications at the operating system level, which the server cannot see. For anything that genuinely must reach the user, such as a security alert or a time-critical booking change, push should be one channel among several, with email or SMS as the durable record. Push in the reliability contract Push belongs in a booklet about offline support for two reasons. First, it is the one background capability that works across every major platform, including the iPhone. That makes it the mechanism of last resort for the problem Chapter 6 left open: when the app holds unsynced work and the user has closed it, a push notification is the only cross-platform way to bring them back. Second, every push is itself a promise, that the message is worth the interruption, and the browsers now keep score. Sites that break that promise lose the channel; installed apps that keep it gain protections that ordinary websites do not have. The next chapter turns from capabilities to speed. A service worker can make an app dramatically faster, and it can also make it measurably slower, and the Core Web Vitals are how you find out which. Chapter 8: Performance, Core Web Vitals, and the Cost of the Service Worker The pitch for PWAs has always included speed: pages that load instantly from cache, no matter the network. That pitch is true for repeat visits to well-built PWAs, and it hides two awkward facts. A service worker adds its own startup cost to every request it handles, which can make a site slower when the cache is not helping. And loading fast is only one part of feeling fast; an app that renders its first frame instantly from cache and then freezes for half a second on every tap does not feel like a native app, whatever the audit score says. This chapter covers how to measure performance on real phones, where service workers help and where they hurt, and how to fix the responsiveness problems that caching cannot touch. The Core Web Vitals as of 2026 Google's Core Web Vitals are three field metrics intended to capture loading, responsiveness, and visual stability as users experience them. They are measured in real browsers, mainly through the Chrome User Experience Report (CrUX), which aggregates anonymized data from Chrome users who have opted in, and they feed into Google Search ranking as one signal among many. A page passes a metric when at least 75 percent of page loads meet the "good" threshold, so the target is the experience of your slower users, not the median. The set changed on 12 March 2024, when Interaction to Next Paint (INP) replaced First Input Delay (FID) as the responsiveness metric. FID measured only the delay before the browser began processing the first interaction on a page, and nearly every site passed it, which made it a poor signal. INP observes every click, tap, and key press during a visit, measures the full time from input until the next frame is painted, including the time spent running event handlers and rendering, and reports a value close to the worst interaction. It is a much harder test, and it falls hardest on JavaScript-heavy single-page apps running on mid-range and low-end Android phones, which are exactly the devices PWAs often target. Table 4 lists the current thresholds. Table 4. Core Web Vitals thresholds, assessed at the 75th percentile of page loads (source: Google, web.dev). Metric Measures Good Needs improvement Poor Largest Contentful Paint (LCP) Loading of main content 2.5 s or less Up to 4.0 s Over 4.0 s Interaction to Next Paint (INP) Responsiveness to input 200 ms or less Up to 500 ms Over 500 ms Cumulative Layout Shift (CLS) Visual stability 0.1 or less Up to 0.25 Over 0.25 Two measurement realities matter for PWAs. First, field data and lab data disagree, often sharply. A Lighthouse run on a developer's laptop with a fast connection says little about a three-year-old Android phone on a congested mobile network. Collect field data from your own users with the web-vitals JavaScript library, which reports each metric along with attribution data identifying the element or interaction responsible, and send it to your analytics. Second, CrUX and most tooling measure page loads. In a single-page app, the user may navigate a dozen times within one page load, and those client-side "soft navigations" are not measured as separate loads for LCP; their interactions do count toward the page's INP, which tends to rise the longer a session lasts. Chrome has been experimenting with a Soft Navigations API to measure them, but as of late 2026 your own instrumentation is the reliable way to see them. Where a service worker helps loading On a repeat visit to a well-built PWA, the service worker's contribution to LCP is large. The HTML, CSS, JavaScript, fonts, and often the hero image come from local storage instead of the network. Time to first byte for a precached shell can drop from hundreds of milliseconds on a mobile connection to a few milliseconds, and on a slow or flaky connection the difference can be seconds. How much of that translates into LCP depends on architecture. For a server-rendered page served network first, the service worker helps only with subresources; the HTML still comes from the network. For an app shell, the shell arrives instantly, but the LCP element is usually content rendered after a data fetch, so LCP is bounded by the data request plus JavaScript execution. Precaching the shell while leaving the content on a slow network path can produce a site that paints a header instantly and its main content no sooner than before. The fix is to make the content path fast as well: cache the most likely first-screen data with stale-while-revalidate so the app can render real content immediately, then update it, and make sure the LCP image is discoverable early rather than injected by JavaScript after a fetch. Where a service worker hurts A service worker is a JavaScript program, and the browser must start it before it can handle a request. If the worker is not already running, the browser has to spin up a thread, load the script, and execute it before the fetch handler can even begin. Google's measurements of this startup have ranged from tens of milliseconds on desktop to hundreds of milliseconds on slow mobile devices. For a request the worker answers from cache, that cost is usually worth paying. For a request the worker simply forwards to the network, as with network first or network only strategies for navigations, it is pure overhead: the navigation waits for the worker to start, and only then does the network request begin. This is the main reason some sites have measured worse navigation performance after adding a service worker, and it has three remedies. Navigation preload. With navigation preload enabled, the browser starts the network request for a navigation in parallel with booting the service worker, and the worker's fetch handler can use the in-flight response via event.preloadResponse instead of issuing its own request. The startup cost and the network round trip overlap rather than add. It is supported in Chromium, Firefox, and Safari 15.4 and later. self.addEventListener('activate', (event) => { event.waitUntil(self.registration.navigationPreload?.enable()); }); self.addEventListener('fetch', (event) => { if (event.request.mode !== 'navigate') return; event.respondWith((async () => { try { return (await event.preloadResponse) || (await fetch(event.request)); } catch { return caches.match('/offline.html'); } })()); }); Every service worker that handles navigations with a network-based strategy should enable navigation preload. The server receives a Service-Worker-Navigation-Preload header on preload requests and can use it to return a lighter response, such as content without the page frame, if the worker composes pages. Static routing. The Service Worker Static Routing API, shipped in Chrome 123, lets a worker declare at install time that certain requests should go straight to the network or straight to a named cache without running the fetch handler at all. The browser applies these rules without starting the worker. This eliminates startup overhead for requests the worker would not have done anything interesting with, such as API calls that are always network only, and it supports a race-network-and-fetch-handler source that races the network against the worker for cases where either could win. As of late 2026 it is Chromium only; other browsers ignore it, so the fetch handler must still implement the same routes. self.addEventListener('install', (event) => { if (!event.addRoutes) return; event.addRoutes([ { condition: { urlPattern: '/api/*' }, source: 'network' }, { condition: { urlPattern: '/assets/*' }, source: { cacheName: PRECACHE } }, ]); }); Not handling what you do not need. The simplest remedy is to return early from the fetch handler, without calling respondWith(), for requests the worker does not need to intercept. That does not avoid startup entirely, since the worker must run to decide, but it avoids adding work after startup. And a site that has no offline functionality to offer should not register a service worker with a fetch handler at all; since Chrome 108 on mobile and 112 on desktop, an empty fetch handler is no longer needed for installability, and Chromium may skip starting a worker whose fetch handler it can detect is a no-op, but the cleanest choice is not to have one. The back/forward cache One of the largest loading wins on mobile has nothing to do with the service worker. The back/forward cache, or bfcache, keeps a complete snapshot of a page in memory when the user navigates away, including its JavaScript state, and restores it instantly if they press back. Back and forward navigations are a large share of mobile navigations, and a bfcache restore is effectively instant, which makes it one of the cheapest improvements available. Pages can make themselves ineligible without realizing it. The most common causes are unload event handlers, which browsers treat as incompatible with bfcache (use pagehide or visibilitychange instead); open connections such as WebSockets or IndexedDB transactions left active when the page is hidden; and, historically in Chrome, responses sent with Cache-Control: no-store. Chrome's DevTools include a bfcache tester that lists the reasons a page was not restored, and the NotRestoredReasons API reports them from the field. A PWA should listen for the pageshow event with event.persisted true to refresh data that may have changed while the page was frozen, the bfcache equivalent of the stale-while-revalidate update message. Responsiveness is a main-thread problem INP problems are almost never caused by the network, and no caching strategy fixes them. They come from the main thread being busy when the user interacts, or from the work an interaction triggers taking too long before the next frame can paint. INP breaks each interaction into three parts, and attribution data from the web-vitals library reports all three. Input delay is time spent waiting for the main thread to become free, usually because some other long task, such as hydration of a server-rendered page, a large JSON parse, or a third-party script, was running when the user tapped. Processing time is time spent in your event handlers. Presentation delay is time spent on style calculation, layout, and painting after the handlers finish, which balloons with large DOM trees and with layout-triggering reads interleaved with writes. The fixes follow the parts. Break long tasks into chunks under 50 milliseconds so input can be handled between them; scheduler.yield(), available in Chromium since version 129, yields to the browser and resumes with priority, and setTimeout(resolve, 0) inside a promise is the portable fallback. In event handlers, do the minimum needed to paint visual feedback, such as showing a pressed state or a spinner, and defer the rest until after the next frame. Keep DOM size in check, and use CSS content-visibility: auto to skip rendering off-screen sections of long pages. Move expensive computation such as search indexing, image processing, or diffing large data sets to a Web Worker, which runs on its own thread and is distinct from the service worker. Audit third-party scripts, which are often the largest source of long tasks on commercial sites, and load them after the page is interactive or not at all. PWAs face a specific temptation here. Because the service worker makes JavaScript bundles cheap to load on repeat visits, teams stop worrying about bundle size. But download cost was never the only cost of JavaScript. The parse, compile, and execution costs are paid on every launch, on the user's CPU, and on a low-end phone they are several times higher than on a developer's laptop. A 2-megabyte bundle served instantly from cache still takes the same time to execute. Caching hides one cost and leaves the larger one in place. Worked diagnosis: a slow filter A concrete case shows how the INP breakdown guides a fix. Imagine a product-listing PWA whose field data shows a 75th-percentile INP of 420 milliseconds on mobile, in the "needs improvement" band. Attribution from the web-vitals library points to one interaction above all others: tapping a category filter chip on the listing page, on mid-range Android devices. The breakdown for that interaction shows an input delay of 30 milliseconds, processing time of 290 milliseconds, and presentation delay of 100 milliseconds. Input delay is small, so the main thread was usually free when the user tapped; the problem is what happens after. Profiling the interaction in Chrome DevTools, with CPU throttling set to approximate a mid-range phone, shows that the click handler synchronously filters a list of 2,000 products held in memory, sorts the result, updates the application's state store, and lets the framework re-render the entire grid, including products below the fold. It also writes the filter to the URL and to IndexedDB, and fires an analytics event that serializes the whole state object. The fixes follow the breakdown. First, the handler immediately toggles the chip's visual state and yields, so that the user sees feedback in the next frame; that alone bounds the perceived delay. Second, the analytics call and the IndexedDB write are moved after the yield and made asynchronous, since neither affects what is painted. Third, the grid is virtualized so that only the products in and near the viewport are rendered, which cuts both processing and presentation time. Fourth, the filtering and sorting move to a precomputed index built once when the data loads, rather than recomputed on every tap. After these changes, field INP on the same device class falls under 200 milliseconds, and the filter interaction no longer appears at the top of the attribution reports. None of it involved the service worker. The caching layer made the listing load quickly; making it respond quickly was a separate piece of work, and the metric is what made the difference visible. Layout stability CLS problems in PWAs are usually the familiar ones: images without dimensions, late-loading fonts that change text metrics, ads and embeds that push content down, and banners injected at the top of the page. PWAs add one of their own: content that renders from cache and then visibly jumps when fresher data arrives from a stale-while-revalidate refresh. If the refreshed list has a different number of items or a new banner, the layout shifts under the user. The fix is the one Chapter 3 recommended for other reasons: do not swap fresh content in automatically; signal that it is available and let the user choose to see it. Reserving space for elements whose size is known, such as images and notification banners, removes most of the rest. Measuring the contract Performance belongs in the reliability contract because "fast" is itself a promise, and users experience it on their own hardware. A PWA that loads instantly on the developer's phone and responds sluggishly on a customer's is breaking a promise that nobody on the team can see without field measurement. The minimum instrumentation for a production PWA is the web-vitals library reporting LCP, INP, and CLS with attribution, segmented by device class, connection type, and whether the page was controlled by a service worker, which navigator.serviceWorker.controller reveals. That last dimension is the only reliable way to know whether your service worker is helping, and teams that add it are sometimes surprised by the answer. The final chapter turns from building a PWA to running one: the tools, tests, and operational habits that keep these promises intact across deploys, and the platform realities that decide which promises can be made at all. Chapter 9: Operating a PWA in Production A service worker is unusual production code. It runs on the user's device, persists between visits, intercepts every request, and can outlive the version of the site that installed it. A bug in it does not go away when you deploy a fix; it goes away when every affected browser has fetched the fix and activated it, which, as Chapter 2 showed, can take days. That makes operating a PWA a distinct discipline from operating a website. This chapter covers the tooling choices, the testing practices, the emergency procedures, and the platform risks that a team running a PWA in production should plan for. Choosing tools Most production service workers are not written entirely by hand. The precache manifest must be generated from the build output, runtime caches need expiration, and routing logic benefits from a tested abstraction. The question is which layer to build on. Workbox is the most widely used library for writing service workers. It was created by Chrome's developer relations team and provides modules for precaching with per-file revisioning, routing, the standard caching strategies, cache expiration, background sync with a fallback queue, broadcast updates, and navigation preload, along with build plugins for webpack and a command-line tool. Its status has changed in a way teams should understand. Ownership moved from Chrome's developer relations team to Chrome's Aurora team, which works on framework integrations, and releases since have been maintenance releases: version 7.4.0, released in November 2024, was described as critical dependency updates. The library is stable, widely deployed, and continues to receive security updates, but it is no longer evolving quickly, and new platform features such as static routing are not necessarily wrapped by it. Using Workbox remains a sound choice for its precaching and expiration modules, which are fiddly to get right by hand, provided someone on the team understands the underlying APIs well enough to work outside it. Framework integrations wrap Workbox or similar logic for specific build tools. vite-plugin-pwa is the most popular for Vite-based projects and generates both the manifest and a Workbox-based service worker, with options for prompt-to-reload update flows. Angular ships its own @angular/service-worker package with a configuration-driven model. Next.js and other meta-frameworks have community packages of varying maintenance quality; check the commit history before adopting one, since an abandoned service worker plugin is a liability that runs on every user's device. Hand-written workers are reasonable for small apps with simple needs, such as an offline page and a precached shell, and they have the advantage that everything in them is visible. The risk is reimplementing expiration, revisioning, and error handling with subtle bugs. A common middle path is to use Workbox's precaching module, or a build step that injects a precache list, and hand-write the routing. Whichever tool you choose, keep the service worker itself small. Every byte is fetched on every update check and parsed on every worker start, and a large worker slows the startup cost Chapter 8 described. Testing service workers Service workers are harder to test than ordinary front-end code because their interesting behaviour spans page loads, states, and network conditions. A few practices make it tractable. Use the browser's developer tools deliberately. Chrome's Application panel shows registered workers and their states, lets you trigger update, skip waiting, and unregister, simulates offline mode, lists Cache Storage contents, and can fire test push and sync events. Its "Update on reload" checkbox, which forces a new worker to install and activate on every reload, is useful during development and hides the real lifecycle; turn it off when testing update flows. Firefox and Safari have equivalent, less extensive tools. The "Bypass for network" option skips the worker entirely, which is useful for isolating whether a bug is in the worker or the page. Automate the lifecycle. End-to-end testing frameworks such as Playwright can test service worker behaviour in Chromium, including toggling the browser context offline, intercepting requests at the network level, and inspecting the worker. A useful baseline suite visits the site, waits for the service worker to activate, goes offline, and checks that the shell loads, the offline page appears for an uncached route, and a form submission is queued; then goes online and checks that the queue drains. A second suite deploys version N, loads the app, deploys version N+1, and verifies that the update prompt appears and that accepting it leads to a working page on the new version. That second test catches the class of bugs that otherwise appears only in production. Test on real devices. Emulated offline mode in desktop developer tools does not reproduce iOS storage rules, the absence of Background Sync in Safari, the throttling of service worker startup on low-end Android devices, or the behaviour of installed apps when the operating system suspends them. Keep at least one recent iPhone and one inexpensive Android phone in the test rotation, and test the installed app, not just the browser tab. Test bad networks, not just no network. Most real offline failures are slow, lossy, or captive connections. Throttle to a slow 3G profile, add packet loss with an operating system level tool, and test behind a captive portal, which answers every request with a redirect to a login page. A network-first strategy without a timeout, or a worker that caches redirects, fails visibly under these conditions and invisibly under clean offline simulation. Monitoring Because service worker code runs on users' devices, its failures do not appear in server logs. Instrument it explicitly. Report installation failures and activation errors from the worker by sending them to an endpoint, including the worker's version. Record, from the page, which service worker version controls it, so that you can see how fast a new version reaches users after a deploy; a version that never reaches more than half your traffic suggests a failing install. Report outbox depth and age from the page, so that you can see whether queued writes are draining. Report storage persistence status and quota usage occasionally. And segment the Core Web Vitals by whether a service worker controlled the page, as Chapter 8 recommended. Server-side, watch the push sending pipeline: delivery success rates by push service, the rate of 404 and 410 responses indicating dead subscriptions, and subscription growth against unsubscribes. A sudden rise in dead subscriptions from Apple's push service, for example, may indicate a push handler that failed to show a notification and triggered revocation. The kill switch Every team that runs a service worker should have a kill switch ready before it is needed. A faulty worker can serve a broken or stale version of the site to every returning user, and normal fixes may not reach them quickly if the faulty worker caches its own HTML aggressively. The kill switch is a replacement sw.js, deployed at the same URL, that removes the worker and its effects: self.addEventListener('install', () => self.skipWaiting()); self.addEventListener('activate', (event) => { event.waitUntil((async () => { const keys = await caches.keys(); await Promise.all(keys.map((k) => caches.delete(k))); await self.registration.unregister(); const windows = await self.clients.matchAll({ type: 'window' }); windows.forEach((w) => w.navigate(w.url)); })()); }); It works because the browser still checks for a new sw.js on each navigation, bypassing the HTTP cache for the script, even when the faulty worker is serving everything else. The new worker installs, skips waiting, deletes every cache, unregisters itself, and reloads open windows so that they fetch fresh content from the network. As a heavier backstop, the server can send the Clear-Site-Data: "storage" header on a response, which instructs the browser to clear all storage for the origin, including service worker registrations, though it also erases IndexedDB data, including any unsynced writes, so it should be a last resort. Two preparations make the kill switch usable under pressure. Keep it in the repository, tested, with a documented deploy procedure. And never change the URL of your service worker script: a worker registered at /sw.js can only be replaced by a new script at /sw.js, and a deploy that moves it to /service-worker.js leaves the old one running forever for every user who had it. Security considerations A service worker raises the stakes of some ordinary web vulnerabilities, because anything it does persists. The most serious is cross-site scripting. If an attacker can run script on your origin, they can register a service worker of their own, provided they can also get a script file served from your origin at a path whose scope covers the pages they want, or replace yours if your build pipeline or CDN is compromised. A malicious worker can intercept every subsequent request, rewrite responses, and persist after the original vulnerability is fixed, until a legitimate worker replaces it. Defences are the usual ones applied more strictly: a Content Security Policy with a restrictive worker-src directive, no user-uploaded content served from the same origin as the application at paths that could act as a service worker script, and strict control over who can deploy to the path where sw.js lives. Browsers require the service worker script to be served with a JavaScript MIME type, which blocks the crudest attacks using uploaded files, but not all of them. The second concern is caching personal data, raised in Chapter 3. Responses containing a user's information, cached under a key that does not include the user's identity, can leak to the next person who uses the device. Clear personal caches and IndexedDB stores on logout, avoid caching responses to authenticated API requests unless necessary, and never cache responses that carry Cache-Control: private or no-store without understanding why the server sent them. The third is token handling in the worker. A service worker that needs to make authenticated requests, for instance to drain an outbox in a sync event, needs credentials when no page is open. Cookies sent with credentials: 'include' or same-origin requests work naturally and are the simplest choice. Bearer tokens stored in IndexedDB are readable by any script on the origin, which makes cross-site scripting more damaging, and should be short-lived with refresh handled carefully. Never keep tokens only in the worker's global variables, both for the reliability reasons discussed in Chapter 2 and because it encourages sloppy assumptions about where secrets live. Platform risk and what the EU episode taught The capabilities of a PWA are not fixed by standards. They are granted by browser and operating system vendors, and they can be withdrawn. The clearest demonstration came in early 2024. The Digital Markets Act required Apple to allow alternative browser engines on iOS in the European Union from March 2024. In February 2024, in the betas of iOS 17.4, users in the EU found that Home Screen web apps no longer opened as standalone apps; tapping one produced a prompt to open it in the default browser, as an ordinary tab. Apple confirmed in its developer documentation that it was removing standalone web apps in the EU, arguing that supporting them on alternative engines would require building an entirely new integration architecture with security and privacy implications, which it considered impractical given the Act's other demands. The consequences for affected apps were severe: no standalone window, no push notifications, and no access to data stored by the installed app, which was separate from the browser's storage. Developers and advocacy groups protested, and press reports said the European Commission had begun asking Apple and developers about the change. On 1 March 2024, Apple reversed the decision, saying Home Screen web apps would continue in the EU, built on WebKit as before. iOS 17.4 shipped with them intact. The episode ended well for PWA developers, but the lessons outlast it. First, OS-integrated capabilities such as standalone display, push, and the storage protections of installed apps exist at the platform owner's discretion and can be removed in a point release. Second, the EU outcome preserved the status quo rather than opening it: Home Screen web apps on iOS still run only on WebKit, even in the EU, so a PWA's capabilities on an iPhone remain whatever Apple gives WebKit. Third, the reversal came after organized pressure from developers and regulators, which suggests the web platform's capabilities on mobile are partly a political outcome as well as a technical one. The trend since has been positive. Apple added Declarative Web Push in iOS 18.4 and made every Home Screen site open as a web app by default in iOS 26. Firefox added taskbar web apps on Windows in version 143. Chrome made installation easier and exempted installed web apps from notification auto-revocation. None of this changes the engineering conclusion. Build so that the app remains useful if any OS-level capability disappears: data synced to the server rather than living only on the device, a version that works in a browser tab, and an alternative channel, such as email, for anything critical that currently depends on push. App stores as a distribution channel Many teams want their PWA in app stores for discoverability or because their customers expect it. On Android, Trusted Web Activity lets a PWA be packaged as a Play Store app with a thin native wrapper that opens the site full screen in the user's Chrome-based browser, sharing its storage and service worker; Google's Bubblewrap tool and Microsoft's PWABuilder generate the package. The site must pass Digital Asset Links verification proving the app and site belong to the same owner. The Microsoft Store accepts PWAs directly. Apple's App Store does not accept PWAs as such; a web app wrapped in a native shell that adds no native functionality risks rejection under Apple's guideline against apps that are simply repackaged websites, and wrapping it in a WKWebView loses the Home Screen web app features, such as web push, that make a PWA worthwhile on iOS in the first place. Store distribution changes operational practice. Store listings require review, privacy declarations, and versioned releases, even though the content updates through the website. Treat the wrapper as a separate deliverable with its own release process, and keep its native code as close to zero as possible so that the web remains the place where the app actually changes. Operations as part of the promise The previous chapters described what a PWA promises users. This chapter has been about keeping those promises over time: across deploys, across devices, across browser changes, and across the occasional decision by a platform owner. The habits are not exotic. Choose tools you understand. Test the lifecycle, not just the page. Monitor what runs on users' devices. Keep a kill switch. And design so that the app survives the loss of any capability you do not control. With those in place, the remaining question is not how to build a PWA but whether, and how far, to invest in one, which is where the conclusion takes up the argument. Conclusion The case for Progressive Web Apps is usually framed as a contest with native apps: can the web do what an app store app can do? After the chapters of this booklet, that framing looks like the wrong question. On Android and desktop Chromium, the web can do most of what most business and content apps need. On the iPhone, it can now do more than at any point in its history, including push notifications, but still lacks every background execution mechanism that Chromium offers. The contest has no single answer, and asking it leads teams to make capability checklists rather than design decisions. The more useful question is the one this booklet has pressed throughout: what does the app promise the user when the network fails, and can it keep that promise within the limits each browser enforces? Asked that way, several conclusions follow that are not obvious at the outset. The promise is mostly about writes, and the iPhone sets the terms Most PWA material concentrates on reading offline: precaching shells, caching strategies, stale-while-revalidate. Those techniques matter and they are the easier half. The harder and more consequential half is writing offline. Chapter 6 showed that the reliable architecture is the same everywhere, a durable outbox in IndexedDB with idempotent server endpoints, and that the background mechanisms that would deliver it while the app is closed exist only in Chromium. Chapter 5 showed that local data is best-effort storage that the browser may delete under pressure and that Safari deletes from non-installed sites after a week of non-use. Put those together and the binding constraint on an offline-capable PWA is not storage size or caching sophistication. It is how quickly user-generated data can be moved off the device on an iPhone, where nothing can send it while the app is closed. Teams that design for Chrome first and treat Safari as a degraded case get this backwards. Design the write path for Safari's constraints, eager sending, visible pending state, push as a nudge to reopen, and encourage installation, and then let Background Sync on Chromium improve the experience. An app built the other way around works beautifully in the demo and loses data in the field. Installation is now a reliability feature, not a vanity metric For years, installation was treated as the PWA's marketing moment: an icon on the home screen, a standalone window, an "app-like" feel. The evidence in this booklet points to a more practical value. On iOS, installation is the only route to push notifications, and installed web apps keep their own storage clock, so they are protected from the seven-day deletion that applies to sites in Safari tabs. On Chromium, installation makes navigator.storage.persist() far more likely to be granted, enables Periodic Background Sync, and exempts the app from the notification auto-revocation Google announced in 2025. In each case, installation buys durability. That changes how install prompts should be designed. The argument to put to the user is not "get our app" but "keep your work safe and get notified when it matters". For apps whose value depends on offline data or notifications, a well-timed install explanation, especially the manual Add to Home Screen instructions iOS still requires, is part of the reliability design, not a growth experiment. Speed is two problems, and the service worker solves only one Chapter 8 separated loading from responsiveness. A service worker can make repeat loads very fast, and with navigation preload and static routing it can avoid slowing down the requests it does not help. It does nothing for Interaction to Next Paint, which depends on how much JavaScript runs on the main thread when the user taps. Since INP replaced FID in March 2024, the gap between a PWA that loads like an app and one that feels like an app is measured, and it is measured on the slower devices where PWAs promise the most. The practical implication is a reallocation of effort. Teams that have built a competent service worker usually get more from cutting main-thread work than from refining their caching further. The app-like feel the PWA movement has promised since 2015 depends more on JavaScript discipline than on any service worker technique. What remains unsettled Several questions are open as of late 2026, and they will shape the next few years of PWA work. The first is whether Safari will ever implement Background Sync or an equivalent. WebKit's choice to add Declarative Web Push, which shows a notification without running code, suggests a philosophy of granting capabilities that do not require background JavaScript execution. A declarative equivalent for queued writes, in which the browser replays a stored request when connectivity returns without waking a service worker, would fit that philosophy and solve the biggest remaining gap. Nothing announced so far points to it. The second is the regulatory path. The 2024 episode left Home Screen web apps on iOS tied to WebKit even in the EU. Whether regulators eventually require Apple to let alternative engines power installed web apps, and whether those engines would bring Chromium's background capabilities with them, is unresolved. A PWA strategy that depends on that outcome is a bet, not a plan. The third is the maturity of the tooling. Workbox is stable and in maintenance mode; the frameworks increasingly own their service worker integrations; and new platform features such as static routing are arriving faster than libraries wrap them. Teams will need more first-hand understanding of the underlying APIs, not less, which is a reversal of the trend of the late 2010s, when a library could be expected to hide them. What to do differently For a team deciding how far to go with a PWA, the argument of this booklet reduces to a few changes of practice. Write down the promise before writing the service worker. For each kind of data the app shows, choose a strategy and state in plain language what the user sees offline. For each kind of action the user takes, state what happens if the network fails at that moment. That document is the specification for the routing table, the outbox, and the interface states, and it is the thing to review when someone proposes stale-while-revalidate for prices. Treat client storage as a buffer. Sync early, show pending work, request persistence at a meaningful moment, and never let the only copy of user work sit on the device for longer than it must. Make the update policy explicit. Choose between natural turnover, prompt-to-reload, and immediate takeover deliberately, monitor version adoption, and keep a tested kill switch. Measure on the devices your users have, segmented by whether the service worker was in control, and spend effort where the measurements point, which will often be the main thread rather than the cache. And design for the loss of what you do not control. Push, installation, and storage protections are granted by platform owners and have been withdrawn before, if briefly. A PWA that keeps working as a plain website when they are gone is not a lesser app. It is the one whose promises hold. Glossary App shell. The minimal HTML, CSS and JavaScript needed to draw an app's interface frame, precached by the service worker and served on every launch, with content loaded into it afterwards. Background Sync. A browser API, implemented only in Chromium-based browsers, that fires a sync event in the service worker when connectivity returns, even if the page that requested it has closed. Best-effort storage. The default storage mode in all major browsers, in which the browser may delete an origin's data without asking the user when storage is under pressure. Cache Storage. The asynchronous API, available as caches in pages and service workers, that stores pairs of HTTP requests and responses in named caches. Idempotency key. A unique identifier sent with a write request so that the server can recognize and ignore a repeat of an operation it has already processed. IndexedDB. The browser's transactional object database, used by PWAs for structured data and offline write queues. Interaction to Next Paint (INP). The Core Web Vital, since March 2024, that measures the time from a user interaction to the next frame painted, reported close to the worst interaction in a visit. Navigation preload. A service worker feature that starts the network request for a page navigation in parallel with starting the service worker, removing worker startup time from the critical path. Outbox. A durable local queue of pending write operations, sent to the server and removed only once the server acknowledges them. Service worker. A script that the browser installs for an origin and runs separately from pages, intercepting their network requests and handling events such as push and sync. Stale-while-revalidate. A caching strategy that responds immediately from cache and simultaneously fetches a fresh copy to update the cache for the next request. VAPID. Voluntary Application Server Identification, the scheme defined in RFC 8292 by which a server proves its identity to a browser's push service using a signed token. Further Reading MDN Web Docs, "Progressive web apps" and "Service Worker API." developer.mozilla.org/en-US/docs/Web/Progressive_web_apps. The most complete vendor-neutral reference for the manifest, service worker lifecycle, Cache Storage, Push, and Background Sync, with browser compatibility tables for each API. MDN Web Docs, "Storage quotas and eviction criteria." developer.mozilla.org/en-US/docs/Web/API/Storage_API/Storage_quotas_and_eviction_criteria. The source for the per-engine quota figures in Chapter 5, with clear explanations of best-effort versus persistent storage and Safari's proactive eviction. W3C, Service Workers specification. w3c.github.io/ServiceWorker. The normative description of registration, update, the lifecycle states, and the fetch event. Dense, but the final word when browser behaviour seems surprising. Jake Archibald, "The offline cookbook" and "The service worker lifecycle." web.dev. Two essays by one of the specification's editors that established the vocabulary of caching strategies and remain the clearest account of why the lifecycle works as it does. Chrome for Developers, Workbox documentation. developer.chrome.com/docs/workbox. Reference for Workbox's modules, with explanations of precaching, expiration, and routing that are useful even for teams not using the library. The GitHub repository at github.com/GoogleChrome/workbox records the project's ownership change and release history. WebKit blog, "Web Push for Web Apps on iOS and iPadOS" (2023), "Meet Declarative Web Push" (2025), and "Updates to Storage Policy" (2023). webkit.org/blog. Apple's own descriptions of the Home Screen push requirements, the declarative push format, and the storage quota changes in Safari 17. WebKit blog, "Full Third-Party Cookie Blocking and More" (March 2020). webkit.org/blog/10218. The announcement of the seven-day cap on script-writable storage, including the statement that Home Screen web apps keep their own counter of days of use. web.dev, "Interaction to Next Paint" and "Optimize INP." web.dev/articles/inp. Google's definition of INP, its thresholds, and practical guidance on input delay, processing time, and presentation delay. Chrome for Developers, "Use the Service Worker Static Routing API" and "Speed up service worker with navigation preloads." developer.chrome.com/blog. The explanations of the two main remedies for service worker startup cost discussed in Chapter 8. IETF RFC 8030, RFC 8291 and RFC 8292. The Generic Event Delivery Using HTTP Push protocol, Message Encryption for Web Push, and VAPID. Useful when debugging a push server or evaluating a push library. Martin Kleppmann and colleagues, "Local-first software: You own your data, in spite of the cloud" (Ink and Switch, 2019). inkandswitch.com/local-first. The essay that named the local-first approach and set out the role of CRDTs, relevant to the conflict-resolution choices in Chapter 4. Hashtags: #ProgressiveWebApps #PWA #PWADevelopment #WebDevelopment #OfflineFirst #OfflineSupport #ServiceWorkers #ServiceWorkerAPI #WebAppManifest #CacheStorageAPI #CachingStrategies #StaleWhileRevalidate #NetworkFirstCaching #CacheFirstStrategy #OfflineCaching #IndexedDB #BackgroundSync #WebPushNotifications #PushAPI #Workbox #AppShellArchitecture #LocalFirstDevelopment #OfflineDataSynchronization #StorageManagement #ProgressiveEnhancement #CoreWebVitals #WebPerformanceOptimization #PWASecurity #CrossPlatformWebApps #ReliableWebApplications

Download the Book (PDF): Introduction A child does not learn to see by being handed a labelled dataset. Long before anyone tells her that the round thing is a ball and the furry thing is a dog, she has learned that objects persist when they move behind a sofa, that a cup seen from above and a cup seen from the side are the same cup, that the edge where the table meets the floor is a boundary and not a stripe painted on one surface. She has extracted a structured model of the visual world from the world itself, using nothing but the regularities of what she sees and the consequences of what she does. Labels, when they arrive, attach to concepts that are already there. For most of the history of modern computer vision, machines learned the other way round. The deep learning revolution in vision, which is usually dated to the 2012 ImageNet competition, was a revolution in supervised learning. A convolutional network was shown more than a million photographs, each tagged by a human annotator with one of a thousand categories, and trained to reproduce those tags. The features the network learned along the way turned out to be astonishingly useful. They could be transferred to detection, segmentation, medical imaging, satellite analysis and dozens of other problems, usually by keeping the network and retraining only its last layer or two. For about eight years, "pretrained on ImageNet" was the default starting point for almost any visual task. The trouble is that a representation trained to predict a thousand human labels learns, above all, what is needed to predict those thousand labels. If two breeds of terrier are separate classes, the network will work hard to tell them apart. If the colour of the sky or the arrangement of objects on a table is irrelevant to the label, the network has every incentive to discard that information. Labels are also expensive. ImageNet took years of crowdsourced work to assemble; a dense segmentation dataset, where every pixel must be assigned to a class, costs far more per image; and in fields such as pathology or radiology, each label may require the time of a specialist physician. The supply of labelled images grows slowly. The supply of unlabelled images grows by billions every day. Self-supervised learning is the attempt to close that gap: to train visual representations from images alone, by constructing a learning problem whose answers are already contained in the data. Hide part of an image and ask the network to fill it in. Show it two distorted crops of the same photograph and ask it to recognise that they belong together. Ask it to predict the abstract features of a region it cannot see from the regions it can. None of these tasks requires a human annotator, and all of them, if designed well, force the network to learn something about how the visual world is put together. That is the subject of this booklet: how machines have learned to extract useful visual features without manual labels, what the competing families of methods actually do, why some of them work far better than others, and how their representations are used for classification and segmentation, including in the zero-shot setting where the model is given no task-specific training at all. The argument The field can look like a bewildering sequence of acronyms. SimCLR, MoCo, BYOL, SwAV, SimSiam, Barlow Twins, VICReg, DINO, iBOT, BEiT, MAE, SimMIM, data2vec, I-JEPA, CLIP, SigLIP, DINOv2, DINOv3: each arrived with a paper claiming a new state of the art, and each is easy to mistake for a fresh idea. The argument of this booklet is that they are best understood as successive answers to a single design question, and that once the question is clear, the history becomes legible and the current state of the art becomes predictable. The question is this: what should the network be asked to predict, and in what space should that prediction be made? Every self-supervised method defines a target that the network must produce from some partial or transformed view of an image. The choice of target is not a technical detail. It is the whole of the method's prior about what matters in images. A network asked to reconstruct raw pixels will spend capacity on texture, lighting and noise, because those things are in the pixels. A network asked to give the same output for two heavily augmented crops will learn to ignore whatever the augmentations change, which is good when the augmentations remove nuisance variation and harmful when they remove information a downstream task needs. A network asked to predict abstract features of a hidden region, rather than its pixels, can learn to model what is predictable about the scene while discarding what is not. What the target includes, the representation learns; what the target excludes, the representation is free to throw away. Seen this way, the development of the field over the past decade follows a clear line. Early "pretext tasks" such as predicting rotations or solving jigsaw puzzles chose targets that were easy to construct but only loosely related to semantics. Contrastive learning replaced them with a target defined by invariance: two views of the same image should map close together, views of different images should map apart. That worked remarkably well, and it brought self-supervised features level with supervised ones for the first time, but it bound the representation to whatever invariances the hand-designed augmentations encoded, and it learned mostly global, image-level information. Masked image modeling, arriving with the vision transformer, took the opposite approach and asked the network to reconstruct what had been hidden, which preserved spatial detail and scaled beautifully but produced features that needed fine-tuning before they were useful. The strongest current systems, exemplified by DINOv2 and its successors, combine the two ideas and move the prediction into the network's own learned feature space: a global invariance objective for semantics, a masked prediction objective for local structure, both computed against targets produced by a slowly evolving copy of the network itself. The second half of the argument concerns what these representations are for. Self-supervised features have become valuable not mainly because they match supervised features on ImageNet classification, which they now do, but because they carry information that labels never asked for. The attention maps of a self-supervised vision transformer outline objects without ever having been told what an object is. Its patch features cluster into semantic parts. A simple linear layer on top of frozen features can produce a usable segmentation map or depth estimate. These dense, transferable properties are what make modern self-supervised encoders the backbone of systems for segmentation, depth estimation, robotics, medical imaging and remote sensing, and they follow directly from the choice of target. How the booklet proceeds The first chapter describes the problem self-supervision set out to solve and the early pretext tasks, which are worth studying because their failures exposed the design question clearly. The second explains the vision transformer, the architecture that most current self-supervised methods are built on, and why it changed what was possible. The third and fourth treat contrastive learning and its successors that learn without negative examples, including the central problem of representational collapse. The fifth covers self-distillation and the DINO family, where the most striking emergent properties of self-supervised vision first appeared. The sixth turns to masked image modeling and the masked autoencoder. The seventh examines methods that predict in latent space rather than pixel space, from data2vec to the joint-embedding predictive architectures. The eighth considers language as a source of supervision, through CLIP and its descendants, and the case for and against learning vision without words. The ninth explains how these representations are evaluated and used for classification and segmentation, including zero-shot and unsupervised segmentation. The tenth looks at scale, data curation and the practical problems that appear when self-supervised models grow large, and the conclusion draws out what follows for anyone building or choosing a visual representation today. A word on what the booklet assumes. The reader does not need to have trained a neural network, but should be comfortable with the idea that a network is a function with millions or billions of adjustable parameters, tuned by gradient descent to reduce a loss. Mathematical notation is kept to a minimum; where a loss function matters, it is described in words. Numbers are quoted where they come from published results and where they help, and they should be read as indicators of relative performance at the time of publication rather than as permanent rankings, because benchmarks in this field move every few months. The booklet is also deliberately focused on still images and on representation learning: the question of how to produce a general-purpose encoder whose features serve many downstream tasks. Generative image models, which also learn from unlabelled data, appear only where their ideas feed back into representation learning. Video is treated where it extends the argument, particularly in the discussion of predictive architectures. Within that scope, the aim is to give the reader a working understanding of why self-supervised vision works, when it fails, and how to think about the choices that remain open. Chapter 1. The Label Bottleneck and the First Pretext Tasks What supervised pretraining actually bought To understand why self-supervised learning matters, it helps to be precise about what supervised pretraining on ImageNet delivered and what it did not. ImageNet, assembled by Fei-Fei Li and colleagues and released in 2009, became the basis of the ImageNet Large Scale Visual Recognition Challenge. The version used for the challenge, now usually called ImageNet-1K, contains about 1.28 million training images in 1,000 categories. When Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton won the 2012 challenge with a deep convolutional network, now known as AlexNet, the result was striking enough to redirect the field. Within a few years, a sequence of deeper architectures, VGG, GoogLeNet and ResNet, had pushed classification error on the benchmark below the error rate reported for a trained human annotator in one widely cited comparison. The more consequential discovery was about transfer. Researchers found that the intermediate layers of a network trained on ImageNet classification contained general-purpose visual features. Early layers detected edges and colour blobs, middle layers responded to textures and parts, and later layers responded to whole objects. A network trained for one task could be reused for another: object detection systems such as Faster R-CNN, semantic segmentation networks, pose estimators and medical image classifiers were all routinely initialised from ImageNet weights. This practice, pretraining on a large labelled dataset and then fine-tuning on a smaller target dataset, became the standard recipe of applied computer vision. That recipe had three weaknesses, and self-supervised learning was a response to all three. The first was cost. Labels must be produced by people, and the more detailed the label, the more expensive it is. An image-level class tag might take a crowd worker a few seconds. A bounding box for every object takes longer. A segmentation mask, where every pixel is assigned to an object or class, can take minutes per image, and the COCO dataset's per-instance polygon masks and the Cityscapes dataset's fine pixel-level annotations represent a very large investment of human time. In specialised domains, cost is not the only problem: a histopathology slide or a retinal scan may have to be read by a trained clinician, and there may simply not be enough clinicians willing to annotate. The second weakness was scale. Because labels are expensive, labelled datasets grow slowly, while the volume of unlabelled imagery available on the web, in hospital archives, from satellites and from cameras on vehicles grows enormously. Companies with access to vast internal datasets could partially escape the constraint: Google trained models on JFT-300M, an internal dataset of around 300 million images with noisy, automatically generated labels, and Facebook trained on billions of Instagram images using their hashtags as weak labels. These efforts showed that more data helped, but they relied on label signals that were noisy, proprietary or both. The third weakness was the most interesting scientifically. A network trained to predict labels learns what the labels reward. ImageNet's categories include more than a hundred dog breeds but no category for "a scene with three objects on a table". A classifier has no incentive to encode how many objects are present, where they are relative to each other, what the lighting is, or what lies in the background, unless those things happen to help distinguish the thousand classes. Several studies found that ImageNet-trained convolutional networks rely heavily on texture rather than shape; Robert Geirhos and colleagues showed in 2019 that such networks would classify an image with the silhouette of a cat filled in with elephant-skin texture as an elephant, whereas humans would say cat. Supervised features are good, but they are shaped by the labelling scheme, and the labelling scheme is a narrow and somewhat arbitrary slice of what is visually meaningful. Self-supervised learning promised to escape all three weaknesses at once. If the training signal could be extracted from images themselves, the cost of labels would disappear, the training set could be as large as the available imagery, and the representation would be shaped by the structure of the visual world rather than by a particular taxonomy. The idea of a pretext task The basic move in self-supervised learning is to invent a task whose labels can be generated automatically from unlabelled data. This task is called a pretext task, because nobody actually cares about solving it; it is a pretext for making the network learn useful features. After training on the pretext task, the network is evaluated by how well its features transfer to tasks people do care about, such as classification or segmentation. The idea is older than deep learning. Autoencoders, which compress an input into a small code and then reconstruct it, are a form of self-supervision in which the target is the input itself. Denoising autoencoders, introduced by Pascal Vincent and colleagues in 2008, corrupted the input and asked the network to recover the clean version, which forced it to learn something about the structure of the data rather than simply copying. In natural language processing, the word2vec models of 2013 learned word embeddings by predicting a word from its neighbours, and the success of that approach, and later of BERT's masked language modeling, would strongly influence vision. Between roughly 2015 and 2019, vision researchers produced a remarkable variety of pretext tasks. Each was clever, and each told the field something about what self-supervision could and could not do. Context prediction. Carl Doersch, Abhinav Gupta and Alexei Efros proposed in 2015 to sample two patches from an image, one at the centre of a three-by-three grid and one from one of the eight surrounding positions, and ask a network to predict which position the second patch came from. To solve this, the network must know something about how parts of objects relate: that a cat's ear goes above its eye, that the sky goes above the horizon. The authors found that a naive network could cheat by exploiting low-level cues, such as the way lenses cause colour channels to shift slightly towards the image edges (chromatic aberration), which revealed a patch's absolute position without any understanding of content. They had to add gaps between patches, jitter their positions and randomly drop colour channels to block these shortcuts. Jigsaw puzzles. Mehdi Noroozi and Paolo Favaro extended the idea in 2016 by cutting an image into a three-by-three grid of tiles, shuffling them according to one of a fixed set of permutations, and asking the network to identify which permutation had been applied. This required reasoning about all nine pieces jointly. Again, shortcuts had to be blocked by leaving gaps between tiles and by normalising each tile's statistics independently. Colourisation. Richard Zhang, Phillip Isola and Alexei Efros, and independently Gustav Larsson and colleagues, trained networks in 2016 to predict the colour channels of an image from its greyscale version. To colour a scene correctly, a network must recognise that grass is green, that skies are blue and that a stop sign is red, so semantic understanding should help. Zhang and colleagues treated colour prediction as classification over quantised colour bins rather than regression, because regression towards the average produced washed-out, sepia-toned results. Inpainting. Deepak Pathak and colleagues introduced context encoders in 2016, which removed a large region from the centre of an image and trained a network to fill it in. A reconstruction loss alone produced blurry predictions, so they added an adversarial loss to encourage realistic output. In retrospect this was the clearest precursor of masked image modeling, which would become dominant five years later. The context encoder's features transferred reasonably but not impressively, and part of the reason, as the later chapters explain, was that a convolutional network asked to fill a single large hole is solving a different and less useful problem than a transformer asked to reconstruct many small scattered patches. Rotation prediction. Spyros Gidaris, Praveer Singh and Nikos Komodakis showed in 2018 that a startlingly simple task worked well: rotate an image by 0, 90, 180 or 270 degrees, and ask the network to predict the rotation. To know that an image is upside down, one must know what the right way up looks like, which requires recognising objects and their canonical orientation. RotNet, as it was called, was cheap to implement and gave some of the best transfer results of its generation. Clustering. Mathilde Caron and colleagues at Facebook AI Research proposed DeepCluster in 2018, which alternated between clustering the network's features with k-means and training the network to predict the cluster assignments as if they were labels. This approach had no hand-designed geometric task at all; the targets came from the network's own evolving representation. It was an early example of a theme that would become central: the network bootstrapping its own targets. Table 1 sets these early tasks side by side, with the knowledge each was meant to induce and the shortcut each was vulnerable to. Table 1. Representative pretext tasks from the pre-contrastive era. Task Target the network predicts Intended knowledge Typical shortcut or failure Context prediction (2015) Relative position of a second patch Spatial layout of object parts Chromatic aberration and edge continuity reveal position Jigsaw (2016) Index of the tile permutation Part-whole configuration Low-level boundary matching between tiles Colourisation (2016) Colour channels from greyscale Object and material identity Many objects have no single correct colour Inpainting (2016) Pixels of a removed central region Scene completion Blurry averages; emphasis on texture Rotation (2018) One of four rotation angles Canonical object orientation Fails on rotation-symmetric or orientation-free imagery DeepCluster (2018) Cluster index from k-means on own features Grouping of similar images Degenerate clusters without careful balancing What the pretext tasks taught the field By the standards of the time, these methods worked. Features learned this way were far better than random initialisation and could be fine-tuned to respectable performance on detection and classification. But they consistently fell well short of supervised ImageNet pretraining. When evaluated by the standard "linear probe" protocol, which freezes the network and trains only a linear classifier on top of its features using ImageNet labels, the best pretext-task methods of 2018 reached top-1 accuracies in the range of the mid-forties to around fifty-five percent, depending on architecture and on which layer was probed, against well over seventy percent for a supervised ResNet-50. A large-scale benchmarking study by Priya Goyal, Dhruv Mahajan, Abhinav Gupta and Ishan Misra in 2019 confirmed that scaling up the data and the model helped these methods, but not enough to close the gap. Three lessons emerged from this period, and each one points to the design question that organises this booklet. The first lesson was that networks will find the cheapest route to solving a task, and that route is often not semantic. Context prediction could be solved with chromatic aberration; jigsaw puzzles with boundary pixel statistics; rotation with the position of watermarks or black borders in some images. Every pretext task opened its own set of shortcuts, and closing them required a growing list of ad hoc tricks. The underlying problem was that the target of the prediction was a geometric or photometric fact about the image, and many low-level features carry that fact without any need for understanding. The second lesson was that the layer at which features are most useful depends on how far the pretext task is from the downstream task. In pretext-trained networks, the best features for classification were often found in intermediate layers rather than at the top. The final layers had specialised to the pretext task: they had become very good at predicting rotation angles, which is not the same as being good at recognising objects. The network learned general features on the way up and then threw some of them away to focus on the artificial target. This is exactly what one would predict if the representation is shaped by its target, and the target is only loosely related to semantics. The third lesson was that the target should ideally be defined by the data's own structure rather than by a single hand-chosen transformation. DeepCluster pointed in this direction by deriving targets from the network's own feature space. Rotation and jigsaw each picked one kind of regularity and hoped it would be enough. What was needed was a target that captured the general idea that different views of the same scene share content, without committing to a specific geometric puzzle. The turn to instance discrimination The decisive step came from reframing the problem. Instead of inventing a puzzle, one could treat every image in the dataset as its own class. Alexey Dosovitskiy and colleagues had tried a version of this as early as 2014 with "Exemplar CNNs", which created a surrogate class for each of a few thousand image patches and trained the network to recognise augmented versions of each patch as belonging to its class. The idea did not scale directly, because a classifier with one output per image becomes unmanageable once there are a million images. In 2018, Zhirong Wu, Yuanjun Xiong, Stella Yu and Dahua Lin made it scale. Their method, called instance discrimination, stored a feature vector for every training image in a "memory bank" and trained the network so that the feature of an image was close to its own stored vector and far from the stored vectors of all other images. Distances were measured in a normalised embedding space, and the loss was a softmax over similarities, so the network effectively solved a million-way classification problem where each class contained exactly one image and its augmentations. The authors motivated this with an observation about supervised networks: when a network trained on ImageNet labels misclassifies an image of a leopard, the second most likely class it assigns is usually jaguar or cheetah, not bookcase. Visual similarity is already implicit in the structure of the data. One does not need class labels to learn it; one only needs to ask the network to tell individual images apart. At around the same time, Aäron van den Oord, Yazhe Li and Oriol Vinyals introduced Contrastive Predictive Coding, which trained a network to predict future elements of a sequence in a learned latent space, scoring the true future against a set of distractors. They formalised the scoring with a loss they called InfoNCE and showed that minimising it maximises a lower bound on the mutual information between the context and the target. CPC was applied to audio, text and images, where the "sequence" was a column of patches read from top to bottom. These two lines of work established the template that would dominate the next three years: produce two related views of the data, map each through a network to an embedding, and train the network so that related embeddings are similar and unrelated ones are different. The target was no longer a rotation angle or a permutation index. It was the identity of the image itself, as expressed through whatever transformations the designer chose to apply. The next breakthroughs would come from understanding what those transformations should be, and from scaling the approach with better engineering. Before following that story, it is necessary to introduce the architecture on which most of the later chapters depend. Chapter 2. Vision Transformers: An Architecture Without Assumptions The history of self-supervised vision is not chronological with respect to its architectures. Contrastive learning first succeeded on convolutional networks, in 2019 and 2020, before vision transformers had been introduced. But almost every self-supervised method of consequence after 2021 is built on the vision transformer, and several of the most important ideas, masked image modeling above all, only became practical because of it. It is worth understanding the architecture before following the methods, because many of the properties that make modern self-supervised representations useful, and several of their peculiar failure modes, are properties of the transformer rather than of the training objective. From convolutions to patches A convolutional neural network processes an image by sliding small learned filters across it. Each filter looks at a small neighbourhood, typically three by three pixels, and produces a response at every location. Stacking many such layers, with occasional downsampling, lets later layers see progressively larger regions of the image. This design builds in two strong assumptions, usually called inductive biases. The first is locality: what matters at a pixel is mostly its immediate neighbourhood. The second is translation equivariance: the same filter is applied everywhere, so a feature detected in one corner of the image would be detected in the same way in another. These biases match the structure of natural images well. Edges and textures are local, and a cat is a cat wherever it appears in the frame. They are a large part of why convolutional networks learned efficiently from datasets of a million images. The transformer, introduced for machine translation by Ashish Vaswani and colleagues in 2017, makes almost no such assumptions. It operates on a set of tokens, each represented as a vector. In each layer, every token computes a weighted combination of all the other tokens, with weights determined by how relevant each token is judged to be. This operation is called self-attention. Each token produces three vectors from its current representation: a query, a key and a value. The attention weight from token A to token B is computed from the similarity between A's query and B's key, normalised across all tokens with a softmax; token A's new representation is then the weighted sum of all tokens' values. After attention, each token passes independently through a small two-layer network, the multilayer perceptron or MLP block. Residual connections and layer normalisation hold the whole stack together. Multiple attention "heads" run in parallel, each with its own query, key and value projections, so that different heads can attend to different kinds of relationship. Nothing in this design says that nearby tokens are more relevant than distant ones, or that the same pattern should be detected identically at different positions. The only spatial information comes from position embeddings, learned or fixed vectors added to each token to tell the network where it came from. Everything else must be learned. In 2020, Alexey Dosovitskiy and colleagues at Google published "An Image is Worth 16x16 Words", which applied a nearly unmodified transformer to images. The method was disarmingly simple. Cut the image into a grid of non-overlapping square patches, typically 16 by 16 pixels, so that a 224-by-224 image becomes a grid of 14 by 14, or 196, patches. Flatten each patch into a vector and pass it through a single linear layer to produce a token embedding. Add a position embedding to each. Prepend one extra learnable token, called the class token or CLS token, whose final representation will be used to summarise the whole image. Feed the sequence of 197 tokens through a standard transformer encoder. For classification, attach a linear classifier to the final CLS token. This model, the vision transformer or ViT, came in a family of sizes named after their BERT counterparts. ViT-Base has 12 layers, a token width of 768 and about 86 million parameters. ViT-Large has 24 layers, width 1,024 and about 307 million parameters. ViT-Huge has 32 layers, width 1,280 and about 632 million parameters. The notation ViT-B/16 means ViT-Base with 16-pixel patches; ViT-L/14 means ViT-Large with 14-pixel patches. Smaller patches mean more tokens, finer spatial resolution and much more computation, because self-attention's cost grows with the square of the number of tokens. Later work added ViT-Small (about 22 million parameters), ViT-giant (around a billion) and larger variants still. The cost of having no assumptions The original ViT paper reported a result that framed the next several years of research. Trained on ImageNet-1K alone, the vision transformer underperformed comparable ResNets. Trained on the 14-million-image ImageNet-21K, it roughly matched them. Trained on JFT-300M, Google's internal dataset of around 300 million images, it surpassed them, and the gap grew with model size. The authors' interpretation was that the convolutional network's inductive biases are helpful when data is scarce, but become a limitation when data is abundant, because a model that must learn locality from data can also learn when locality does not apply. This finding has a direct bearing on self-supervised learning. If vision transformers need very large amounts of training signal to realise their advantages, and if labelled data is the scarce resource, then a method that can extract training signal from unlabelled images is exactly what the architecture needs. Conversely, a self-supervised objective that can exploit the transformer's flexibility, in particular its ability to process a variable set of tokens and to relate any token to any other, may achieve things a convolutional network cannot. Data-efficient training of ViTs on ImageNet-1K alone was demonstrated within months. Hugo Touvron and colleagues at Facebook AI Research showed with DeiT in late 2020 that heavy data augmentation, strong regularisation and distillation from a convolutional teacher network let a ViT reach competitive accuracy on ImageNet-1K without external data. DeiT's recipe, with its battery of augmentations such as RandAugment, Mixup and CutMix and its long training schedules, became the standard supervised baseline against which self-supervised ViTs were measured. Three properties that matter for self-supervision Three characteristics of the vision transformer turned out to be particularly important for self-supervised learning. Tokens can be dropped. A convolutional network operates on a dense grid and has no natural way to skip regions of the input; a masked region still has to be filled with something, usually zeros or a learned mask value, and the network still pays the computational cost of processing it. A transformer operates on a set of tokens. If one removes three quarters of the patch tokens from the sequence, the remaining quarter can be processed exactly as before, at roughly a quarter of the cost of the attention and MLP layers. This seemingly minor property is what made the masked autoencoder, described in Chapter 6, both efficient and effective. The class token and the patch tokens offer two kinds of output. A ViT produces one output vector per patch plus one for the class token. The class token summarises the image, which makes it the natural target for image-level objectives such as contrastive learning. The patch tokens retain spatial information, which makes them natural targets for dense objectives such as masked prediction, and natural inputs for segmentation. Methods can attach different losses to the two outputs, and the most successful recent methods do exactly that. A convolutional network also produces a spatial feature map, but its final layers are heavily downsampled, and global pooling is the standard way to obtain an image-level vector; there is no separate learned summary token that attends to all positions. Attention maps are directly inspectable. In every layer, every head of a ViT computes an explicit set of weights describing how much each token attends to every other. The weights from the class token to the patch tokens in the final layer show which parts of the image the summary representation relies on. In a supervised ViT these maps are often diffuse or noisy. In some self-supervised ViTs, as Chapter 5 describes, they turn out to trace the outlines of objects with remarkable precision. Because the maps are an explicit part of the computation rather than a post-hoc explanation, they can be used directly as a crude segmentation, which opened a route to unsupervised object discovery. The weaknesses that came with the design The transformer's flexibility has costs beyond its appetite for data. The most obvious is computation. Self-attention scales quadratically with the number of tokens. A 224-pixel image with 16-pixel patches gives 196 tokens; a 518-pixel image with 14-pixel patches, as used by some recent models for dense prediction, gives 1,369 tokens, and attention over that many tokens is roughly fifty times more expensive per layer. This limits the resolution at which ViTs can be trained and pushes practitioners towards larger patches, which in turn limits the spatial detail of the patch features. Architectures such as the Swin Transformer, introduced by Ze Liu and colleagues in 2021, restored some locality by computing attention within shifted local windows and building a hierarchical feature pyramid, which made transformers more practical for detection and segmentation at high resolution. Swin and similar hierarchical designs are used with some self-supervised methods, notably SimMIM, but the plain ViT remains the dominant backbone for self-supervised pretraining, partly because its uniform token structure is easier to mask. A second cost is training instability. Large ViTs trained with some self-supervised objectives can diverge or suffer sudden drops in accuracy partway through training. Xinlei Chen, Saining Xie and Kaiming He documented this in their 2021 MoCo v3 paper, finding that the instability originated largely in the first layer, the linear projection that turns patches into tokens, and that simply freezing that layer at its random initialisation made training markedly more stable. Later large-scale efforts relied on a collection of stabilising techniques, including careful learning-rate schedules, layer-scale parameters that initialise each residual branch with a small multiplier, and normalisation of queries and keys. A third weakness took longer to surface. Timothée Darcet, Maxime Oquab, Julien Mairal and Piotr Bojanowski reported in 2023 that large, well-trained ViTs, both supervised and self-supervised, develop a small number of patch tokens with abnormally high norms, typically located in uninformative background regions. These tokens appear to be repurposed by the network as scratch space to store global information, and they show up as bright artifacts in attention maps and degrade dense features. The fix the authors proposed, adding a handful of extra learnable "register" tokens to the input sequence that are discarded at output, gave the network somewhere else to store that information and cleaned up the maps. The episode is a reminder that a model with few built-in assumptions will use its capacity in ways nobody designed, and that some of those uses are counterproductive for the tasks one cares about. Why the architecture changed the question It would be wrong to think of the vision transformer simply as a better backbone that self-supervised methods happened to adopt. It changed the shape of the design space. Before ViT, the dominant self-supervised methods were joint-embedding methods: they produced a single vector per image from a convolutional network and compared vectors across views. Masked prediction, which had transformed natural language processing through BERT in 2018, had been attempted in vision, most notably with context encoders, but it did not work nearly as well as contrastive learning. With convolutional networks, there was no natural unit to mask, masking introduced a mismatch between pretraining (images with holes) and downstream use (complete images), and the network wasted computation on the holes. The vision transformer removed each of these obstacles. Patches gave a natural unit of masking. Dropping tokens removed the computational waste. The mismatch between masked and complete inputs turned out to matter less than expected, because the transformer could learn to process any subset of tokens. And because the transformer's patch outputs retained spatial structure through all layers, the same network could be trained with an image-level objective on its class token and a patch-level objective on its other tokens simultaneously. The result was that, after 2021, the field had two broad families of methods to choose from, and the option of combining them. Joint-embedding methods, which compare representations of different views, are the subject of the next three chapters. Masked modeling methods, which reconstruct hidden parts of a single view, are the subject of Chapters 6 and 7. In both families, the central question remained what the network should be asked to predict, and the transformer simply made more answers to that question feasible. Chapter 3. Contrastive Learning: Invariance as the Target Between late 2019 and mid-2020, self-supervised vision went from a promising curiosity to a practical alternative to supervised pretraining. The methods responsible, MoCo and SimCLR above all, were built on the instance-discrimination idea introduced at the end of Chapter 1, and their success rested less on new theory than on getting a small number of design choices right. Those choices are instructive, because each one is a decision about what the representation should be made to keep and what it should be allowed to discard. The contrastive recipe Every contrastive method of this period follows the same recipe. Take an image from the unlabelled training set. Apply two independent random transformations to it, producing two "views". Pass each view through an encoder network to produce a feature vector. Treat the two views of the same image as a positive pair. Treat views of other images as negatives. Train the encoder so that each view's feature is more similar to its positive partner than to any negative. The standard loss is a softmax over cosine similarities, essentially the InfoNCE loss from Contrastive Predictive Coding. For a given view, one computes its similarity to its positive partner and to every negative, divides each similarity by a small number called the temperature, and applies a cross-entropy loss that treats the positive as the correct class among all candidates. Lower temperatures sharpen the distribution, making the loss focus on the hardest negatives, those most similar to the anchor. The SimCLR paper called its version NT-Xent, for normalised temperature-scaled cross-entropy. Notice what this loss defines as the target. There is no reconstruction of pixels, no geometric puzzle. The target is a relationship: views of the same image should be close, views of different images should be apart. Whatever the transformations alter, the representation is pushed to ignore. Whatever distinguishes one image from another, the representation is pushed to keep. The augmentations therefore act as the designer's specification of nuisance variation, and the set of negatives acts as the specification of what counts as different. MoCo: a large, consistent dictionary The first method to make this recipe competitive with supervised pretraining in transfer was Momentum Contrast, or MoCo, published by Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie and Ross Girshick in late 2019. The authors framed contrastive learning as building a dictionary. Each training step looks up a query, the representation of one view, against a dictionary of keys that contains its positive partner and many negatives. Two properties of this dictionary matter. It should be large, so that the negatives cover a representative sample of the data and the task is hard. And it should be consistent, so that all the keys are produced by the same, or nearly the same, encoder; if keys computed at different points in training come from very different networks, the comparison becomes noisy. The memory bank of instance discrimination was large but inconsistent, because each stored vector had been computed whenever its image was last visited, possibly many thousands of updates ago. Using only the other images in the current mini-batch as negatives was consistent but small. MoCo resolved the tension with two devices. Keys were stored in a queue: each new batch of keys was pushed in and the oldest batch pushed out, which allowed a dictionary of 65,536 keys, far larger than any batch. And keys were computed by a separate "momentum encoder" whose weights were not trained by gradient descent but updated as a slowly moving average of the query encoder's weights, with a momentum coefficient of 0.999. Because the momentum encoder changes very slowly, keys from several batches ago are still nearly consistent with fresh ones. MoCo's headline result was not its linear-probe accuracy on ImageNet, which at 60.6 percent for a standard ResNet-50 still trailed supervised pretraining. It was transfer. When fine-tuned for object detection and segmentation on PASCAL VOC and COCO, MoCo-pretrained networks matched or outperformed ImageNet-supervised networks on several of the tasks the authors tested. For the first time, a representation learned without a single label was a better starting point for some practical vision tasks than one learned from a million labels. The momentum encoder deserves emphasis, because it reappears throughout the rest of this booklet in different guises. A slowly evolving copy of the network that produces targets for the network being trained is the core mechanism of BYOL, DINO, iBOT, data2vec, I-JEPA and DINOv2. MoCo introduced it as a way of keeping a dictionary consistent. Later methods discovered that it does something more important: it lets the network bootstrap its own targets without collapsing. SimCLR: what actually matters A few months later, in early 2020, Ting Chen, Simon Kornblith, Mohammad Norouzi and Geoffrey Hinton published SimCLR, which dispensed with the queue and momentum encoder and instead used very large batches, up to 8,192 images, so that the other images in the batch provided enough negatives. SimCLR's contribution was as much an ablation study as a method. By varying one component at a time, the authors identified which ingredients mattered and by how much. Three findings were especially important. The composition of augmentations is critical. SimCLR tested pairs of augmentations, such as cropping, colour distortion, rotation, blur, noise and cutout, and found that no single augmentation produced good representations, but the combination of random cropping with strong colour distortion was far better than any other pair. The explanation is a shortcut. If two random crops of the same image are compared without colour distortion, the network can match them simply by comparing their colour histograms, which are usually similar across crops of the same photograph and different across photographs. That solves the contrastive task without learning anything about shapes or objects. Colour distortion removes this shortcut and forces the network to use other cues. Here is the pretext-task lesson of Chapter 1 again: the network will exploit whatever cheap signal distinguishes positives from negatives, and the augmentations must be designed to deny it every cheap signal. A nonlinear projection head improves the representation beneath it. SimCLR attached a small multilayer perceptron, the projection head, on top of the encoder, and computed the contrastive loss on the projection head's output rather than on the encoder's output. After training, the projection head was thrown away and the encoder's output was used as the representation. This improved linear-probe accuracy by more than ten percentage points compared with computing the loss directly on the encoder output. The authors' interpretation is directly relevant to the theme of this booklet. The contrastive loss encourages invariance to the augmentations, so the layer at which the loss is computed discards information such as colour and orientation. The projection head gives that discarding somewhere to happen. The layer beneath it can retain information that the contrastive objective does not need but downstream tasks might. In effect, the projection head is a buffer between the target and the representation, so that the representation is not forced to be as narrow as the target. Contrastive learning benefits from scale more than supervised learning does. Larger batches, longer training and wider networks all helped SimCLR more than they helped supervised baselines. A ResNet-50 with four times the standard width, trained with SimCLR, reached 76.5 percent top-1 linear-probe accuracy on ImageNet, matching a standard supervised ResNet-50. The standard-width SimCLR ResNet-50 reached 69.3 percent. MoCo v2, released shortly afterwards by Xinlei Chen, Haoqi Fan, Ross Girshick and Kaiming He, showed that adding SimCLR's projection head and stronger augmentations to MoCo produced better results than SimCLR without the need for enormous batches, reaching 71.1 percent with a standard ResNet-50 after 800 epochs. The two lines of work converged: the queue and momentum encoder solved the engineering problem of obtaining many consistent negatives, and the augmentations and projection head solved the problem of what to learn. What contrastive learning optimises Why does this work at all? A useful answer was given by Tongzhou Wang and Phillip Isola in 2020, who showed that the contrastive loss, in the limit of many negatives, decomposes into two terms. One is alignment: positive pairs should have nearly identical features. The other is uniformity: features should be spread as evenly as possible over the surface of the unit hypersphere on which normalised vectors live. Alignment alone could be satisfied trivially by mapping every image to the same point. Uniformity prevents this, because it penalises features that cluster together. The negatives in a contrastive loss exist to enforce uniformity. This decomposition clarifies what the representation learns. Alignment makes it invariant to the augmentations. Uniformity makes it preserve as much information as possible about which image is which, subject to that invariance. The result is a representation that keeps whatever distinguishes images and is robust to cropping, colour changes and blur. Because natural images that look alike tend to be semantically related, keeping what distinguishes images while ignoring nuisance variation yields features in which semantically similar images end up close together, even though the network was never told about semantics. Yonglong Tian and colleagues sharpened this picture in a 2020 paper titled "What Makes for Good Views for Contrastive Learning?" They argued that the ideal views share only the information relevant to the downstream task and differ in everything else. If the views share too little, the network cannot learn the relevant information. If they share too much, the network can match them using irrelevant shared information, such as colour statistics. There is a sweet spot, and good augmentation design aims for it. They called this the InfoMin principle. The limits of invariance Contrastive learning's success came with constraints that follow directly from its target. The invariances are chosen by hand, and they are not universally right. If the augmentations include colour distortion, the representation becomes partly insensitive to colour. That is fine for recognising cars, which come in every colour, and harmful for distinguishing ripe fruit from unripe fruit or identifying bird species where plumage colour is a key feature. Tete Xiao and colleagues made this point explicitly in 2021 in a paper titled "What Should Not Be Contrastive in Contrastive Learning", showing that representations trained with rotation or colour augmentations performed worse on downstream tasks that depended on those properties, and proposing to learn separate embedding subspaces for different invariances. Rotation invariance is useful for aerial imagery, where there is no canonical up, and harmful for natural photographs, where up matters. The augmentations that work for ImageNet encode assumptions about ImageNet. Random crops assume object-centric images. The central augmentation of contrastive learning is random cropping. Two crops of the same image are treated as views of the same thing. On ImageNet, where most images contain a single prominent object roughly centred in the frame, two random crops usually both contain part of that object, so pulling them together teaches the network about the object. On scene images, such as a street with many objects, two random crops may show a car and a tree, and pulling them together teaches the network that cars and trees are the same. Senthil Purushwalkam and Abhinav Gupta showed in 2020 that much of contrastive learning's success depends on this object-centric bias, and that the occlusion invariance induced by aggressive cropping does not translate into invariance to viewpoint or to changes in an object's pose. Contrastive methods trained on uncurated scene-centric data, such as COCO or random web images, generally performed worse than those trained on ImageNet, a pattern that later shaped the data curation strategies of Chapter 10. Negatives include false negatives. In instance discrimination, every other image is a negative, including other images of the same kind of object. Two photographs of golden retrievers are pushed apart just as hard as a golden retriever and a teapot, at least in principle. In practice, uniformity on a hypersphere with millions of points leaves room for semantically similar images to remain relatively close, and lower temperatures concentrate the loss on the hardest negatives, which are disproportionately likely to be false negatives. Various methods attempted to identify and remove false negatives, or to treat nearest neighbours in feature space as additional positives, as in NNCLR by Debidatta Dwibedi and colleagues in 2021. The representation is global. Standard contrastive learning compares one vector per view. The loss says nothing about where in the image the features came from or how local regions relate. This makes contrastive representations good for image classification and weaker for dense tasks such as segmentation and detection, where per-pixel or per-region features matter. Dense variants, such as DenseCL by Xinlong Wang and colleagues in 2021, applied the contrastive loss between corresponding locations in the two views, and improved dense transfer, but the fundamental design of contrastive learning remained centred on the whole image. Negatives are expensive. SimCLR needed batches of thousands of images, and MoCo needed a queue and a momentum encoder, because the uniformity term requires many negatives. This raised an obvious question. Are negatives truly necessary, or are they just one way of preventing the representation from collapsing to a single point? If there were another way to prevent collapse, the negatives, along with their false negatives and their demand for large batches, could be dropped. That question produced the methods of the next chapter, and the answer turned out to be surprising. The significance of 2020 It is easy, from the vantage of the present, to treat contrastive learning as a stepping stone superseded by masked modeling and self-distillation. That would undersell it. Contrastive methods established that self-supervised representations could match supervised ones on the hardest standard benchmark, that they could transfer better than supervised features to some detection and segmentation tasks, and that their quality improved with scale faster than supervised learning did. They also established a set of tools that every subsequent method uses in some form: two-view training, strong augmentation, projection heads, momentum encoders and the linear-probe evaluation protocol. Above all, contrastive learning made the design question explicit. The augmentations and negatives together define what the representation should be invariant to and what it should distinguish. Choose them well and the representation captures semantics. Choose them poorly and the network finds a shortcut. Every later method can be understood as a different way of answering the same question, often by changing what is compared, how collapse is prevented, or what space the comparison happens in. Chapter 4. Learning Without Negatives: The Problem of Collapse Every joint-embedding method faces the same threat. If the only instruction is "make the two views of an image produce the same output", the network has a perfect and useless solution: produce the same output for every image. A constant function satisfies the alignment objective exactly. This failure is called representational collapse, and preventing it is the central technical problem of joint-embedding self-supervised learning. Contrastive methods prevent collapse with negatives: by pushing different images apart, they make the constant function costly. The methods in this chapter found other ways, and in doing so revealed that the mechanism preventing collapse shapes the representation as much as the objective it accompanies. BYOL: the result that should not have worked In June 2020, Jean-Bastien Grill and colleagues at DeepMind published "Bootstrap Your Own Latent", or BYOL. Its design looked at first like a recipe for collapse. BYOL has two networks. The online network consists of an encoder, a projection head and an additional small MLP called the predictor. The target network has an encoder and projection head of the same architecture but no predictor, and its weights are an exponential moving average of the online network's weights, exactly like MoCo's momentum encoder. Given two augmented views of an image, the online network processes one and its predictor tries to output the target network's projection of the other. The loss is simply the squared distance between the normalised prediction and the normalised target, and the procedure is applied symmetrically by swapping the views. No gradient flows into the target network; it changes only through the moving average. There are no negatives at all. Nothing in the loss rules out collapse. If both networks output a constant, the loss is zero. Yet BYOL did not collapse. It learned representations that reached 74.3 percent linear-probe accuracy on ImageNet with a standard ResNet-50, better than SimCLR and MoCo v2 at the time, and it was notably more robust to smaller batch sizes and to the removal of colour distortion from the augmentations, presumably because without negatives there was less pressure to exploit colour histograms as a shortcut. The obvious question was why. A blog post by Abe Fetterman and Josh Albrecht shortly afterwards argued that batch normalisation, used in BYOL's projection and prediction heads, might be supplying an implicit form of contrast: by normalising each feature across the images in a batch, batch normalisation prevents all images from having identical features. The BYOL authors responded with a follow-up, led by Pierre Richemond, showing that BYOL still worked when batch normalisation was replaced with group normalisation and weight standardisation, which do not mix information across images, provided the initialisation was handled carefully. Batch statistics could help, but they were not the essential mechanism. SimSiam: it is the stop-gradient The cleanest answer came in November 2020 from Xinlei Chen and Kaiming He, with a method called SimSiam, short for simple Siamese networks. SimSiam stripped BYOL down further. It removed the momentum encoder, so that both branches used the same network with the same weights. It kept the predictor on one branch. And it kept one crucial detail: the target branch's output was treated as a constant during backpropagation, a "stop-gradient" operation, so the network was only updated to move the prediction towards the target, never to move the target towards the prediction. SimSiam did not need negatives, large batches or a momentum encoder, and it reached linear-probe accuracies around seventy percent with a ResNet-50. When the stop-gradient was removed, it collapsed immediately to a constant output. When the predictor was removed, it also collapsed. The paper's interpretation was that SimSiam implicitly performs an alternating optimisation, similar to the expectation-maximisation algorithm or to k-means. At each step, the target can be seen as a fixed estimate of the "true" representation of each image, averaged over augmentations, and the network is updated to match it. Because the target is held fixed during each update, the trivial solution of moving both outputs to the same constant is not directly reachable by gradient descent. Yuandong Tian, Xinlei Chen and Surya Ganguli supplied a more formal analysis in 2021, studying the learning dynamics of linear versions of these networks. They showed that the interplay between the predictor and the stop-gradient creates dynamics in which the predictor aligns with the principal directions of the data's correlation structure, and that collapse is avoided because the eigenvalues of the learned representation do not all decay to zero. They also showed that the predictor could be set directly from the correlation statistics, without training it at all, in a method they called DirectPred. The broad conclusion was that non-contrastive methods avoid collapse through their training dynamics rather than through their loss: the loss permits collapse, but the path that gradient descent takes, constrained by the asymmetric architecture and the stop-gradient, does not lead there. This is a subtle and important point for the argument of this booklet. In contrastive learning, the target, invariance plus uniformity, is fully specified by the loss. In BYOL and SimSiam, the target is the output of the network itself, frozen or slowly averaged, and the representation that emerges depends on the dynamics of the bootstrap as much as on the loss. The network is chasing a moving version of itself. What it ends up learning is whatever structure in the data can sustain that chase without degenerating. Redundancy reduction: Barlow Twins and VICReg A different line of work avoided collapse by adding explicit terms to the loss that ruled out degenerate solutions, without needing negatives. Barlow Twins, introduced in 2021 by Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun and Stéphane Deny, was named after the neuroscientist Horace Barlow, who proposed in 1961 that sensory processing aims to reduce redundancy, recoding inputs so that their components become statistically independent. The method computes, over a batch, the cross-correlation matrix between the features of the two views: entry i,j measures how much feature i of the first view correlates with feature j of the second. The loss pushes this matrix towards the identity. Diagonal entries near one mean that each feature is the same across the two views, which is invariance. Off-diagonal entries near zero mean that different features carry different information, which is redundancy reduction. Collapse to a constant is impossible, because a constant feature has no variance and cannot correlate perfectly with anything. Collapse to a few redundant features is penalised by the off-diagonal term. Barlow Twins was simple and did not need asymmetric networks, stop-gradients, momentum encoders or large batches. It reached about 73 percent linear-probe accuracy with a ResNet-50. One curious finding was that it benefited from very high-dimensional projector outputs, up to 8,192 dimensions or more, unlike contrastive methods. VICReg, from Adrien Bardes, Jean Ponce and Yann LeCun in 2021, separated the same ideas into three explicit terms: variance, invariance and covariance. The invariance term is the mean squared distance between the two views' embeddings. The variance term is a hinge loss that keeps the standard deviation of each embedding dimension, measured across the batch, above a threshold, which prevents collapse to a constant. The covariance term pushes the off-diagonal entries of each branch's own covariance matrix towards zero, which decorrelates the dimensions. Because each branch's variance and covariance are regularised separately, the two branches do not even need to share architecture or weights, which allowed VICReg to be applied to pairs of different modalities. These methods make the anti-collapse mechanism a visible part of the target. The representation is told not only to be invariant across views but also to spread its information over many independent dimensions. That second instruction is a statement about what a good representation looks like, and it is an assumption, not a law: there is no guarantee that downstream tasks are best served by decorrelated features. SwAV: clustering as the target A fourth approach used clustering to prevent collapse. SwAV, short for Swapping Assignments between Views, was introduced in 2020 by Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski and Armand Joulin, building on DeepCluster. SwAV maintains a set of learnable prototype vectors, a few thousand of them, which act as cluster centres. For each view, it computes a soft assignment of the view's feature to the prototypes, called a code. The training task is to predict the code of one view from the feature of the other view, hence the swap. Collapse would occur if every image were assigned to the same prototype. SwAV prevents this by computing the codes under an equipartition constraint: within each batch, the assignments must be spread evenly over the prototypes. The constraint is enforced with a few iterations of the Sinkhorn-Knopp algorithm, a classical method for balancing a matrix so that its rows and columns sum to fixed values. Assignments are computed online, within each batch, rather than by running k-means over the whole dataset as DeepCluster did. SwAV also introduced multi-crop, a data augmentation strategy that would prove to be one of the most durable contributions of this period. Instead of two views per image at the standard resolution, SwAV used two global crops at 224 pixels plus several additional small crops at 96 pixels, each covering a small part of the image. The small crops are cheap to process, so the extra views add little computation, and asking the network to relate small local crops to the global view encourages it to learn local-to-global correspondences: what a small part of an object implies about the whole. SwAV with multi-crop reached 75.3 percent linear-probe accuracy with a ResNet-50, then the best result for a self-supervised method with that architecture, and multi-crop improved other methods when added to them. Table 2 sets out the families of joint-embedding method by how each one prevents collapse, because that choice largely determines what the method costs and what it learns. Table 2. How joint-embedding methods prevent representational collapse. Method family Examples Anti-collapse mechanism Main cost or limitation Contrastive SimCLR, MoCo Negatives enforce uniformity Needs many negatives; false negatives Asymmetric bootstrap BYOL, SimSiam Predictor plus stop-gradient or momentum target Theory subtle; sensitive to architecture details Redundancy reduction Barlow Twins, VICReg Explicit variance and decorrelation terms Needs large projector; extra hyperparameters Online clustering SwAV Equipartition of cluster assignments Choice of prototype count; batch-level balancing Self-distillation DINO (Chapter 5) Centring and sharpening of teacher outputs Many interacting hyperparameters Dimensional collapse and measuring quality without labels Complete collapse, where every image maps to the same point, is easy to detect. A subtler failure is dimensional collapse, in which the representation uses only a small subspace of the available dimensions. Li Jing, Pascal Vincent, Yann LeCun and Yuandong Tian showed in 2022 that contrastive methods, despite their negatives, can suffer from dimensional collapse: the spectrum of the embedding's covariance matrix decays sharply, so that many dimensions carry almost no information. They traced this to the interaction between strong augmentations and the implicit regularisation of deep networks, and proposed a variant, DirectCLR, that applied the loss to a fixed subset of the representation. Dimensional collapse matters because it limits what downstream tasks can extract from the representation, and it points towards a practical idea: the effective dimensionality of a representation can be measured without labels. Quentin Garrido and colleagues proposed RankMe in 2023, a smooth estimate of the rank of the embedding matrix, and showed that it correlated well with downstream performance across many methods and hyperparameter settings. Such label-free diagnostics are valuable because evaluating every candidate representation on downstream tasks is expensive, and because in specialised domains, the labels needed for evaluation may themselves be scarce. Convergence: they are more alike than they look By 2022, a number of papers had argued that contrastive and non-contrastive methods are more similar than their presentations suggest. Garrido and colleagues showed formally that, under reasonable assumptions, contrastive objectives and covariance-regularisation objectives such as VICReg are closely related, one regularising the similarity structure across samples and the other across dimensions, and that many reported performance differences shrink when hyperparameters and projector designs are matched. Randall Balestriero and Yann LeCun related several of these methods to classical spectral embedding techniques. The large "Cookbook of Self-Supervised Learning", published in 2023 by Balestriero and many co-authors, catalogued these connections and the practical tricks that matter in each family. The practical lesson is that the choice among joint-embedding methods is less about which loss is correct and more about which set of trade-offs one accepts: batch size, sensitivity to augmentation, stability, computational cost, and the shape of the resulting feature space. All of them learn a representation that is invariant to the chosen augmentations and that spreads information about images over many dimensions. All of them, in their convolutional incarnations, learn mainly global image-level features. What none of the methods so far achieved was a representation that was strong at both the image level and the pixel level, without fine-tuning. That came from combining the bootstrapped target of BYOL with the multi-crop strategy of SwAV, applying them to a vision transformer, and noticing what the transformer's attention maps had learned to do. That combination was called DINO. Chapter 5. Self-Distillation and the Emergence of Objects In April 2021, a team at Facebook AI Research and Inria led by Mathilde Caron published a paper with an unusual title for a machine learning methods paper: "Emerging Properties in Self-Supervised Vision Transformers". The method it introduced, DINO, was a refinement of ideas from BYOL and SwAV applied to vision transformers. Its accuracy on ImageNet was excellent but not shocking. What made the paper influential was what its authors found when they looked inside the trained network. The attention maps of the class token, which no one had told anything about objects, traced the outlines of the main objects in each image, separating foreground from background with a clarity that supervised vision transformers did not show. Self-supervision had produced, without any annotation, a model that seemed to know where things were. This chapter explains how DINO works, why its properties emerge, and how its successors, iBOT, DINOv2 and DINOv3, combined the DINO objective with masked prediction to produce what are now the most widely used general-purpose visual encoders. How DINO works DINO stands for self-distillation with no labels. Distillation, in ordinary machine learning, means training a small student network to imitate the outputs of a large, already-trained teacher. In DINO, there is no pretrained teacher. The teacher is built from the student during training, in the same way as BYOL's target network: its weights are an exponential moving average of the student's weights. Teacher and student share the same architecture, a vision transformer followed by a projection head that ends in a softmax over a large number of output dimensions, 65,536 in the standard configuration. Each training image is augmented into a set of crops using SwAV's multi-crop strategy: two global crops covering more than half of the image at 224 pixels, and several local crops covering smaller regions at 96 pixels. The teacher sees only the global crops. The student sees all of them. For every pair of a teacher view and a different student view, the loss is the cross-entropy between the teacher's output distribution and the student's output distribution: the student is trained to output the same probability distribution over the 65,536 dimensions as the teacher, for a different view of the same image. Because local crops go only to the student, the student must learn to infer, from a small patch of an image, what the teacher would say about the whole. This local-to-global correspondence is a major source of DINO's semantic quality. There are no negatives, no predictor network and no explicit decorrelation term. Collapse is prevented by two operations applied to the teacher's outputs, and the balance between them is the most delicate part of the method. Centring subtracts a running mean from the teacher's outputs before the softmax. Without it, one output dimension could come to dominate for all images, so that every image would be assigned to the same "class". Subtracting the mean, computed as an exponential moving average over batches, prevents any single dimension from being active for everything. On its own, however, centring encourages the opposite failure: a uniform output distribution, the same for every image. Sharpening uses a low temperature in the teacher's softmax, which makes its output distribution peaked. On its own, sharpening encourages collapse to a single dominant dimension. Combined with centring, the two effects cancel each other's failure modes. Centring pushes towards uniformity across dimensions; sharpening pushes towards confident, peaked assignments for each image. The result is that each image is assigned confidently to a small number of output dimensions, and different images are assigned to different ones. The authors described the output dimensions as something like learned pseudo-classes, although no one tells the network what they mean. One unexpected finding was that the teacher consistently outperformed the student throughout training. The authors linked this to a known effect in optimisation: an exponential moving average of a model's weights, sometimes called Polyak-Ruppert averaging, behaves like an ensemble of recent models and is often better than any single one. The better teacher provides better targets, which improve the student, which in turn improves the teacher through the averaging. This positive feedback loop is the essence of self-distillation, and it explains why the method's representation keeps improving well past the point where a contrastive loss would have saturated. What DINO learned DINO's representations were strong on standard benchmarks. A ViT-Base with 8-pixel patches reached 80.1 percent linear-probe accuracy on ImageNet, and a k-nearest-neighbour classifier on frozen features, which involves no training at all beyond storing the features of labelled examples, reached well above 75 percent with several configurations. The k-NN result is worth dwelling on. It means that, in DINO's feature space, images of the same class are already each other's nearest neighbours, so the representation has organised the dataset semantically without ever seeing a label. For contrastive methods of the same period, k-NN accuracy lagged linear-probe accuracy by a wider margin. The emergent properties were more striking still. The authors visualised the attention weights from the class token to the patch tokens in the last layer of the network. For many images, these maps highlighted the main object and suppressed the background, producing what looked like a segmentation mask. Different attention heads often focused on different objects or different parts of the same object. When the attention maps were thresholded to keep the patches carrying most of the attention mass, they overlapped substantially with ground-truth object masks on the PASCAL VOC dataset, much more so than those of a supervised ViT trained on ImageNet labels. The patch features also supported video object segmentation: on the DAVIS 2017 benchmark, simply propagating segmentation labels from the first frame to later frames by nearest-neighbour matching of DINO patch features gave competitive results, without any training on video. Why should a self-supervised objective produce object segmentation? An informal explanation follows from the training task. The student must produce, from a local crop, the same distribution that the teacher produces from a global view. If the local crop contains part of a dog, the student needs to know that this patch belongs to the dog, not to the grass, to produce the right output. The class token learns to attend to the patches that determine the image's identity, which in an object-centric dataset are the patches belonging to the main object. A supervised network, by contrast, can classify an image using whatever features are discriminative for its label, including background context, such as water for boats or snow for wolves, and its attention has no reason to concentrate on the object's extent. DINO's objective is a richer target than a class label, because it asks for the same answer from every part of the object, and the attention maps reflect that. There is a qualification. DINO was trained on ImageNet, which is object-centric. The emergent segmentation is clearest on images with a single salient object, and the class-token attention maps do not, on their own, separate multiple instances of the same class or reliably segment cluttered scenes. The emergent properties are a sign that the representation encodes objectness, not that it is a solved segmentation model. Chapter 9 describes how the field turned these properties into practical unsupervised segmentation methods. iBOT: adding a local target DINO's loss operates only on the class token. The patch tokens are shaped indirectly, through their role in computing the class token. This is a limitation for dense tasks, where patch features matter directly. iBOT, introduced later in 2021 by Jinghao Zhou and colleagues at ByteDance, Johns Hopkins University and elsewhere, added a second, patch-level objective. The name stands for image BERT pretraining with online tokenizer. The student receives a masked version of a global crop, with a random subset of its patch tokens replaced by a learnable mask token. The teacher receives the unmasked crop. For each masked patch, the student must output the same distribution over prototypes that the teacher outputs for the corresponding unmasked patch. The teacher, which is still an exponential moving average of the student, acts as an online tokenizer: it converts each patch into a soft "visual word" that the student must predict. This is masked image modeling of the kind introduced in Chapter 6, but with the target taken from the network's own feature space rather than from pixels or from a separately trained tokenizer. iBOT kept DINO's class-token loss alongside the patch-level loss. The combination improved linear-probe and fine-tuning accuracy over DINO and, more importantly, improved dense transfer: semantic segmentation, object detection and the semantic consistency of patch features. Patches belonging to the same kind of object part, such as wheels or faces, clustered together in iBOT's feature space across different images. This was the first clear instance of the synthesis that defines the current state of the art: a global invariance objective, which teaches the representation what an image is about, combined with a local masked-prediction objective, which teaches it what each region is, with both targets computed in the network's own evolving latent space. DINOv2: scale and curation In April 2023, Maxime Oquab, Timothée Darcet, Théo Moutakanni and a large team at Meta AI released DINOv2. The method was, in their words, mostly a matter of combining existing techniques and making them work at scale, and the paper's significance was the demonstration that a self-supervised model trained on enough carefully curated data could produce general-purpose features, usable frozen across a wide range of tasks, that matched or exceeded the best features then available from any source, including models trained with text supervision. The training objective combined the DINO class-token loss and the iBOT patch-level loss, with separate projection heads for each. Several refinements were added. The centring step of the teacher was replaced with the Sinkhorn-Knopp normalisation from SwAV, which balances assignments more robustly. A regulariser based on the Kozachenko-Leonenko entropy estimator, called KoLeo, encouraged features within a batch to spread out uniformly by penalising small distances to each sample's nearest neighbour. A short final phase of training at a higher resolution, 518 pixels, improved the quality of patch features for dense tasks. And a variety of engineering measures, including efficient attention implementations and sharding of the model across devices, made it possible to train a ViT-giant with about 1.1 billion parameters. The data was as important as the method. Earlier attempts to train self-supervised models on large uncurated collections of web images had produced disappointing results, because such collections are dominated by a few kinds of image, such as product shots, memes and text-heavy screenshots, and contain many near-duplicates. DINOv2's authors built a curated dataset, LVD-142M, through an automatic pipeline. They began with about 1.2 billion images from the web, removed near-duplicates, and then used the embeddings of a self-supervised model to retrieve, from the uncurated pool, images similar to those in a set of curated datasets, including ImageNet and a variety of fine-grained and domain-specific collections. The result was about 142 million images whose distribution resembled the diversity of curated datasets while being far larger. Curation required no labels, only similarity in an existing embedding space. The largest DINOv2 model was then distilled into smaller models, ViT-Small, Base and Large, by training them to match its outputs, which gave better small models than training them directly. These distilled models were released openly and quickly became standard tools. DINOv2's frozen features were strong across a striking range of tasks. With only a linear classifier on top, the ViT-giant reached about 86 percent top-1 accuracy on ImageNet. On fine-grained classification benchmarks it was competitive with or better than the strongest openly available CLIP models of the time. On dense tasks such as semantic segmentation and monocular depth estimation, where a linear layer was trained on top of frozen patch features, it was substantially better than text-supervised models, whose patch features are weaker because their training objective operates almost entirely at the image level. DINOv2's features also served as the backbone of many later systems, including the Depth Anything family of monocular depth estimators and several foundation models in computational pathology. DINOv3 and the problem of dense feature degradation In August 2025, Oriane Siméoni and colleagues at Meta released DINOv3, which scaled the approach to a ViT with about 7 billion parameters trained on a curated set of about 1.7 billion images. The paper identified and addressed a problem that had limited further scaling of DINO-style training. As training runs grew longer, global metrics such as ImageNet linear-probe accuracy kept improving, but the quality of dense patch features, measured by segmentation performance, peaked and then declined. Patch features became noisier and less spatially consistent, as if the network were trading local structure for global discrimination. The DINOv3 authors introduced a technique they called Gram anchoring to counter this. The Gram matrix of an image's patch features records the pairwise similarities between all patches: which patches look alike to the network. Gram anchoring adds a loss that keeps the student's patch-similarity structure close to that of an earlier checkpoint of the model, taken from a point in training when dense features were still of high quality. The network is free to change its features, but not the pattern of which patches resemble which. With this regulariser, dense performance recovered and continued to improve alongside global performance. DINOv3 also included a high-resolution adaptation phase, released a family of smaller models distilled from the 7-billion-parameter teacher, including both ViT and convolutional variants, and trained a separate version on satellite imagery to show that the same recipe transferred to a very different domain. The reported results put its frozen features at or near the top of a wide range of dense and global benchmarks. The Gram anchoring story is a telling instance of the booklet's theme. The DINO and iBOT objectives define targets that, in principle, should produce good global and local features. At very large scale and very long training, however, the global objective came to dominate, and the local structure that the dense tasks needed was gradually sacrificed. The remedy was to add another target, this time explicitly about the relational structure among patches. As models grow, what the target fails to protect, they eventually lose. What self-distillation established The DINO family showed three things that remain central to the field. First, a bootstrapped target in latent space, produced by a momentum teacher, can yield representations whose frozen features are immediately useful, without fine-tuning. This distinguishes self-distillation from the masked autoencoders of the next chapter, whose features are strong after fine-tuning but weaker when frozen. Second, self-supervised vision transformers acquire spatial knowledge, about where objects are and which regions belong together, that supervised training does not induce as clearly. This knowledge is the foundation of the zero-shot and unsupervised segmentation methods described in Chapter 9. Third, the most effective objectives are hybrids. A global target for semantic identity and a local target for spatial structure, both computed against the network's own evolving features, outperform either alone. The local target in iBOT and DINOv2 is a form of masked image modeling, and its origins lie in a parallel line of research that approached self-supervised vision from the opposite direction, by asking the network not to be invariant but to reconstruct. Chapter 6. Masked Image Modeling: Reconstruction as the Target Contrastive learning and self-distillation define their targets through invariance: different views of the same image should agree. Masked image modeling defines its target through completion: given part of an image, predict the rest. The idea is the visual counterpart of the masked language modeling that made BERT, introduced by Jacob Devlin and colleagues in 2018, the foundation of modern natural language processing. In BERT, about 15 percent of the words in a sentence are hidden and the model must predict them from context. To do so, it must learn grammar, word meanings and a good deal of world knowledge. The hope was that hiding parts of images and predicting them would force a vision model to learn the structure of objects and scenes in the same way. Chapter 1 described an early attempt, the context encoder of 2016, which removed a large central region of an image and asked a convolutional network to fill it in. It worked, but not well enough to compete with contrastive methods. The vision transformer changed the situation, and in 2021 masked image modeling rapidly went from an underperforming idea to the method that set records for fine-tuned accuracy. BEiT: predicting visual words The first masked image modeling method to show the approach's potential with vision transformers was BEiT, introduced in June 2021 by Hangbo Bao, Li Dong, Songhao Piao and Furu Wei at Microsoft Research. The name stands for Bidirectional Encoder representation from Image Transformers, a deliberate echo of BERT. The difficulty in translating BERT to images is that words are discrete symbols drawn from a vocabulary, while image patches are continuous arrays of pixel values. BERT predicts which word was hidden, a classification over the vocabulary. What should a vision model predict about a hidden patch? BEiT's answer was to borrow a tokenizer. It used the discrete variational autoencoder trained by OpenAI for the DALL-E image generator, which maps each patch of an image to one of 8,192 discrete codes. These codes acted as visual words. During pretraining, BEiT masked about 40 percent of an image's patches, choosing them in contiguous blocks rather than independently, replaced them with a learnable mask token, and trained a ViT to predict the discrete code of each masked patch. BEiT's features, after fine-tuning, outperformed both supervised pretraining and the contrastive methods of the time on ImageNet classification and ADE20K semantic segmentation. But the reliance on a separately trained tokenizer was awkward. The tokenizer had been trained on a different dataset for a different purpose, it added a second large network to the pipeline, and it was not obvious that its codes captured what one wanted the encoder to learn. Later versions, such as BEiT v2 in 2022, replaced the DALL-E tokenizer with one trained to reproduce the features of a CLIP model, which improved results substantially but imported text supervision into what had been a purely visual method. The question of what target the masked patches should be predicted against had become central. MAE: just predict the pixels In November 2021, Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár and Ross Girshick published "Masked Autoencoders Are Scalable Vision Learners". The masked autoencoder, or MAE, took the most direct possible answer to the target question: predict the raw pixels of the masked patches. No tokenizer, no momentum teacher, no augmentations beyond simple cropping and flipping. Two design decisions made MAE work where earlier pixel-reconstruction methods had not. A very high masking ratio. MAE masked 75 percent of the patches, chosen uniformly at random. This is far more than BERT's 15 percent, and the reason lies in a difference between language and images. Language is dense with information: each word carries meaning that is hard to infer from a few neighbours. Images are highly redundant: neighbouring patches are strongly correlated, and a missing patch can often be filled in convincingly by interpolating its surroundings, with no understanding of what the image depicts. At low masking ratios, the reconstruction task can be solved by local interpolation. At 75 percent, too few neighbours remain for interpolation to work; the network must infer the missing content from a holistic understanding of the scene. The MAE paper's reconstructions illustrate the point: from a quarter of the patches of a photograph of a bus or a dog, the model produces a blurry but structurally correct reconstruction of the whole, including parts of the object that are entirely hidden. An asymmetric encoder-decoder design. The MAE encoder, a standard ViT, processes only the visible patches, the unmasked 25 percent, without any mask tokens. A separate, much smaller decoder then receives the encoded visible patches together with mask tokens placed at the positions of the missing patches, and reconstructs the pixels. The decoder in the standard configuration had eight transformer blocks with a width of 512, and accounted for only a small fraction of the computation per token compared with the encoder. Because the encoder sees only a quarter of the tokens, and attention cost grows with the square of the sequence length, pretraining was much faster than processing full images, by a factor of three or more in the authors' measurements. After pretraining, the decoder is discarded and the encoder is used on complete images for downstream tasks. This asymmetry was possible only because of the transformer's ability to operate on an arbitrary subset of tokens, the first of the three properties described in Chapter 2. It was also conceptually important. Because the encoder never sees mask tokens, it is not distracted by learning to handle artificial placeholder inputs that never appear at test time, a mismatch that had troubled BEiT and earlier inpainting approaches. The loss was the mean squared error between the reconstructed and original pixels, computed only on the masked patches. One small detail turned out to matter: normalising the pixel values within each target patch, by subtracting its mean and dividing by its standard deviation, improved representation quality. Normalisation removes each patch's overall brightness and contrast from the target, so the network spends less effort predicting low-frequency intensity and more on local structure. MAE's results were remarkable for their scaling behaviour. Fine-tuned on ImageNet-1K, with no external data at any stage, an MAE-pretrained ViT-Huge reached 87.8 percent top-1 accuracy at a resolution of 448 pixels, the best result at the time among methods using only ImageNet-1K data. Larger models benefited more from MAE pretraining than smaller ones, and supervised training of the same large ViTs on ImageNet-1K alone overfitted badly. MAE also transferred well to detection and segmentation on COCO and ADE20K, outperforming supervised pretraining. A follow-up by Yanghao Li, Hanzi Mao, Ross Girshick and Kaiming He in 2022 showed that plain, non-hierarchical ViTs pretrained with MAE could be used directly as backbones for object detection with only minor modifications, an approach called ViTDet. And when Meta trained its Segment Anything Model in 2023, the image encoder was an MAE-pretrained ViT-Huge. SimMIM, MaskFeat and the search for the right target MAE was not alone. Within weeks, several groups published closely related methods that varied the target. SimMIM, from Zhenda Xie and colleagues at Microsoft Research Asia, also reconstructed raw pixels, but processed mask tokens in the encoder rather than dropping them, used a single linear layer as the decoder, and worked with hierarchical architectures such as the Swin Transformer as well as plain ViTs. The paper's message was that a very simple setup, with relatively large masked patches and a light prediction head, sufficed; the elaborate tokenizer of BEiT was unnecessary. MaskFeat, from Chen Wei and colleagues at Meta, asked the network to predict not pixels but histograms of oriented gradients, or HOG, the hand-crafted feature descriptor introduced by Navneet Dalal and Bill Triggs in 2005 for pedestrian detection. HOG summarises the local distribution of edge orientations in a region and is relatively insensitive to small changes in brightness and colour. MaskFeat found that HOG targets worked as well as or better than pixels and discrete tokens, and that the method extended naturally to video. These variations made the target question explicit. Pixels contain everything, including noise, exact colour and fine texture that may be irrelevant to recognition. HOG features discard colour and absolute intensity and retain local shape. Discrete tokens from a learned tokenizer compress patches into a vocabulary, retaining whatever the tokenizer was trained to preserve. Features from another network, such as CLIP, retain whatever that network learned to care about. Each target embodies a decision about what in an image deserves to be predicted, and the representation inherits that decision. Table 3 summarises the main choices. Table 3. Masked image modeling methods and the targets they predict. Method (year) Prediction target Source of target What the target emphasises BEiT (2021) Discrete visual token per patch Pretrained dVAE tokenizer Whatever the tokenizer preserves MAE (2021) Normalised raw pixels The image itself Local structure, texture, layout SimMIM (2021) Raw pixels The image itself Same as MAE, with simpler design MaskFeat (2021) HOG descriptors Fixed hand-crafted transform Edge orientation and local shape iBOT (2021) Teacher's patch distribution Momentum teacher Semantic part identity BEiT v2 (2022) Tokens from a CLIP-distilled tokenizer Text-supervised teacher Language-aligned semantics data2vec (2022) Averaged teacher layer outputs Momentum teacher Contextualised latent features Video and the redundancy argument The redundancy argument for high masking ratios predicts that the optimal ratio should be even higher for video, where consecutive frames are nearly identical. This turned out to be the case. VideoMAE, by Zhan Tong, Yibing Song, Jue Wang and Limin Wang in 2022, and a concurrent paper by Christoph Feichtenhofer, Haoqi Fan, Yanghao Li and Kaiming He, applied masked autoencoding to video clips divided into spatio-temporal cubes. Both found that masking 90 percent or more of the cubes worked best. With so little visible, the model cannot copy a missing patch from the same position in a neighbouring frame and must instead model motion and object persistence. The extremely high masking ratio also made video pretraining, which is otherwise very expensive, much more affordable, because the encoder processes only a small fraction of the cubes. The weakness: frozen features MAE had a conspicuous weakness that its authors reported openly. When evaluated by linear probing, freezing the encoder and training only a linear classifier on top, MAE representations were markedly worse than those of contrastive and self-distillation methods. A ViT-Large pretrained with MAE fell well short of the same architecture trained with MoCo v3 under linear evaluation, even though MAE won clearly after full fine-tuning. The MAE paper explored partial fine-tuning, unfreezing only the last few transformer blocks, and found that tuning even one or two blocks closed most of the gap. The information needed for classification was present in MAE's features, but not in a linearly separable form at the output. Why? The answer follows from the target. A pixel-reconstruction objective rewards the network for retaining everything needed to reproduce pixels: textures, colours, exact positions, lighting. It does not reward the network for organising that information into abstract categories, because pixels are not organised by category. The final layers of the MAE encoder are, in effect, specialised to supply the decoder with what it needs for reconstruction. The semantic abstractions that a linear classifier could use are present, distributed through the representation, but they are not the dominant structure of the output space. Namuk Park and colleagues made this comparison systematically in a 2023 study titled "What Do Self-Supervised Vision Transformers Learn?" They found that contrastive learning and masked image modeling produce qualitatively different representations. Contrastive ViTs capture global shape and low-frequency information, and their later layers' attention becomes highly homogeneous, with all tokens attending to similar regions, which makes the class token a good image summary but leaves the patch tokens less diverse. Masked image modeling ViTs capture high-frequency texture and local detail, their attention remains diverse and local through the depth of the network, and their representations are more useful in the later layers only after fine-tuning reorganises them. The two families are, in a real sense, complementary. Randall Balestriero and Yann LeCun put the critique in its sharpest form in a 2024 paper titled "Learning by Reconstruction Produces Uninformative Features for Perception". Their analysis showed that pixel reconstruction objectives favour the directions in image space with the most variance, which are dominated by low-level, low-frequency components that carry relatively little information for recognising objects, while the features most useful for perception explain comparatively little pixel variance. A reconstruction model must therefore be trained for a long time, and with a high masking ratio, before it begins to learn the perceptually useful features, and even then they are not its primary structure. What masked image modeling established Masked image modeling changed self-supervised vision in four ways. It gave the field a pretraining method whose quality scaled reliably with model size and that did not depend on hand-designed augmentations. Contrastive and self-distillation methods are bound to their augmentation pipelines; MAE needs only random masking and cropping. It showed that the right masking ratio for images is very high, and that high masking is both a better learning signal and a large computational saving. It produced representations that are particularly good starting points for fine-tuning on dense tasks, because they retain local spatial detail through all layers. And it exposed, by contrast with joint-embedding methods, the cost of a target defined in pixel space. Predicting pixels forces the model to care about pixels, and much of what is in pixels is not what one wants a visual representation to know. The obvious next step was to keep the masking but change the space in which the prediction is made: instead of predicting the pixels of the hidden patches, predict their representations. iBOT had already done this within the DINO framework. The next chapter follows the methods that made latent prediction their central principle. Chapter 7. Predicting in Latent Space The previous two chapters described two families of method that seem to pull in opposite directions. Joint-embedding methods such as DINO compare abstract representations of different views and learn semantic, linearly usable features, but depend on hand-designed augmentations and learn local structure only indirectly. Masked autoencoders reconstruct hidden pixels and learn rich local detail without needing augmentations, but spend capacity on pixel-level information that recognition does not need, and their frozen features are weak. A natural question is whether one can have the strengths of both: the masking of MAE, which gives a dense, local learning signal without augmentation, and the abstract target space of joint-embedding methods, which focuses learning on what is semantically meaningful. The answer is to predict the representation of the hidden content rather than the content itself. This idea is the most direct expression of the booklet's theme, because it makes the space of the target an explicit design choice. iBOT, described in Chapter 5, was an early instance. This chapter follows the methods that made it their organising principle, and the theoretical argument for why it should work. The argument for abstraction Consider what a pixel-reconstruction model must do with the region of an image that shows the leaves of a tree. To reconstruct those pixels, it must predict the precise position of every leaf, the exact shade of green, the pattern of light and shadow. Almost none of this is predictable from the visible context. The model can know that there is foliage there, roughly what colour it is, and roughly where the tree's outline lies, but the precise arrangement of leaves is effectively random. A pixel-reconstruction loss nonetheless penalises the model for every wrong leaf. The best the model can do is predict a blurry average, and much of its learning effort goes into getting that average slightly less wrong. Now imagine the target is not the pixels of the foliage but a representation of the region, produced by a network that has learned to encode "foliage, green, part of a tree" and to ignore the precise leaf arrangement. Predicting that representation is a much better-posed problem. The predictable aspects of the region are in the target and the unpredictable details are not. The model's learning is concentrated on what can actually be inferred from context, which is precisely the semantic and structural information that downstream tasks need. Yann LeCun has argued for this position at length, most fully in a 2022 position paper titled "A Path Towards Autonomous Machine Intelligence". He distinguishes three broad architectures. A generative architecture, such as an autoencoder, predicts the input itself, and must therefore model every detail of it. A joint-embedding architecture, such as SimCLR or DINO, encodes two inputs and makes their representations agree, without any prediction. A joint-embedding predictive architecture, or JEPA, encodes two inputs, and then uses a predictor to predict the representation of one from the representation of the other. Because the prediction is made in representation space, the encoder is free to discard information that is unpredictable, and the predictor need only model what is predictable. LeCun's broader claim is that this is how intelligent systems should learn models of the world: not by predicting every pixel of the future, but by predicting abstract states. The difficulty, of course, is that predicting a representation brings back the problem of collapse. If the encoder that produces the target is trained along with the predictor, the easiest solution is for the encoder to produce a constant representation that the predictor can trivially match. Every latent-prediction method must therefore include one of the anti-collapse mechanisms of Chapter 4. In practice, nearly all of them use the BYOL recipe: a momentum teacher that produces the targets, a predictor on the student side, and a stop-gradient. data2vec: one objective for every modality The first method to frame latent prediction as a general principle was data2vec, introduced in early 2022 by Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu and Michael Auli at Meta AI. Its ambition was to use exactly the same learning objective for speech, text and images, differing only in how the input was converted into tokens and masked. In data2vec, a teacher network, an exponential moving average of the student, processes the complete, unmasked input. The student processes a masked version. For each masked position, the student must regress the teacher's representation at that position. The target is not the teacher's final output but the average of the outputs of the teacher's top several transformer layers, each normalised. Averaging over layers produces a richer target than any single layer, and because the teacher sees the full input, its representation of each position is contextualised: it reflects not only what is at that position but how it relates to everything else. This last point distinguishes data2vec from iBOT and from masked pixel prediction. In MAE, the target for a masked patch is that patch's pixels, which depend only on the patch itself. In data2vec, the target for a masked patch is the teacher's representation of that patch in the context of the whole image, which encodes, for instance, that this patch is part of a dog's head that is facing left. Predicting a contextualised target asks the student to infer not just local content but its role in the scene. On images, data2vec matched or slightly exceeded MAE and BEiT on fine-tuned ImageNet accuracy. A successor, data2vec 2.0, released later in 2022, focused on efficiency. It computed the teacher's target once per image and reused it for several different masked versions processed by the student, used a lightweight convolutional decoder, and, like MAE, did not process masked tokens in the encoder. It reached comparable accuracy with a fraction of the pretraining time of MAE. I-JEPA: predicting blocks without augmentation In January 2023, Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun and Nicolas Ballas introduced the Image-based Joint-Embedding Predictive Architecture, or I-JEPA, the most explicit implementation of LeCun's proposal for images. I-JEPA has three components: a context encoder, a target encoder that is an exponential moving average of the context encoder, and a predictor, itself a narrow vision transformer. From each image, several target blocks are sampled, rectangular regions each covering a modest fraction of the image, in the original configuration four blocks each spanning roughly 15 to 20 percent of the image area. A large context block is also sampled, and any patches overlapping the target blocks are removed from it. The target encoder processes the whole image, and its output patch representations within each target block become the targets. The context encoder processes only the visible context patches. The predictor receives the context encoder's output together with mask tokens carrying the position embeddings of the target block, and must output the target encoder's representations for each patch in the block. The loss is the average squared distance between predicted and target representations. Three design choices stand out. No hand-designed augmentations. I-JEPA uses no colour distortion, no blur and no multi-crop. The only transformation is masking. This frees the method from the hand-chosen invariances that limit contrastive and self-distillation methods, which Chapter 3 described as one of their central weaknesses. Large, semantic target blocks. The targets are contiguous blocks large enough to contain object parts, rather than scattered individual patches. Predicting a large block from context requires inferring what object part is there, not interpolating textures from neighbours. The authors found that block size and the informativeness of the context were important: too small a target and the task becomes local; too little context and the task becomes impossible. Prediction in representation space. Because the target is the target encoder's representation, the predictor need not model the unpredictable details of the hidden region. The I-JEPA paper reported that its representations were considerably better than MAE's under linear probing, approaching the linear-probe performance of augmentation-based joint-embedding methods without using their augmentations, and better than those methods on some low-level tasks such as object counting and depth prediction, where augmentation invariance discards useful information. It was also efficient: the authors reported training a ViT-Huge with 14-pixel patches on ImageNet in under 72 hours on 16 A100 GPUs, considerably less than comparable methods required. Why latent prediction learns different features A theoretical analysis by Etai Littwin and colleagues at Apple in 2024 made precise the intuition with which this chapter began. Studying simplified, deep linear versions of the two approaches, they showed that reconstruction objectives such as MAE prioritise features with high variance in the input, whereas JEPA-style objectives prioritise features that are highly predictive across views, which they called high-influence features, and that this bias grows stronger with network depth. In natural images, high-variance directions are dominated by low-level intensity patterns, while predictive features are more likely to be semantic. This complements the Balestriero and LeCun critique of reconstruction described in Chapter 6: reconstruction learns what varies most, latent prediction learns what can be predicted best, and the two are not the same. The practical implication is that latent prediction can learn semantic features without the hand-designed invariances that augmentations impose. This matters especially for domains where the right invariances are unknown or differ from those suited to natural photographs: medical imaging, where colour may be diagnostically important; satellite imagery, where scale and orientation carry meaning; scientific imagery, where the relevant structures may look nothing like everyday objects. From images to video and world models The JEPA principle has been pushed furthest in video, where its argument is strongest. Predicting future video frames pixel by pixel is extraordinarily hard and largely wasted effort: the exact motion of every leaf on a tree in the next second is not predictable, but the fact that a ball thrown upward will come down is. A model that predicts in representation space can concentrate on the latter. V-JEPA, introduced by Adrien Bardes and colleagues at Meta in early 2024, applied the I-JEPA recipe to video, masking large spatio-temporal regions of clips and predicting their representations. Trained on around two million videos from public datasets, it produced frozen video features that performed well on action recognition and motion-sensitive tasks under frozen evaluation, with an attentive probe trained on top. V-JEPA 2, released in June 2025, scaled the approach to a billion-parameter model trained on more than a million hours of video and images. The authors then trained an action-conditioned version on a small amount of unlabelled robot interaction video, on the order of tens of hours from the DROID dataset, and used it to plan simple manipulation tasks, such as picking and placing objects, with robot arms in new environments without collecting data in those environments or training on task-specific rewards. The planning worked by searching for actions whose predicted future representation matched the representation of a goal image. The robot results are early and the tasks simple, but they illustrate the idea that motivates the JEPA programme: a representation trained to predict what is predictable about the world should support not only recognition but also reasoning about the consequences of actions. Whether this scales to complex, long-horizon tasks is an open question. Removing the heuristics Latent prediction methods inherited the anti-collapse machinery of BYOL: momentum teachers, stop-gradients and predictors whose stability depends on many hyperparameters. This makes them effective but somewhat fragile, and it leaves the question of why they do not collapse partly unanswered. In November 2025, Randall Balestriero and Yann LeCun proposed a method called LeJEPA that aims to remove these heuristics. Their argument starts from a theoretical claim: under broad assumptions about downstream tasks, the embedding distribution that minimises worst-case downstream prediction error is an isotropic Gaussian, with equal variance in every direction and no correlations. They introduced a regulariser called Sketched Isotropic Gaussian Regularization, or SIGReg, which pushes the distribution of embeddings towards an isotropic Gaussian by testing it along many random one-dimensional projections, a computationally cheap approach. With this regulariser in place, a JEPA could be trained without a momentum teacher or stop-gradient, with a single main hyperparameter trading off prediction and regularisation. The work is recent and its long-term significance is not yet settled, but it reflects a broader direction: making the anti-collapse mechanism a principled statement about what a good representation's distribution should look like, rather than a set of training tricks. In that respect, it continues the line from Barlow Twins and VICReg described in Chapter 4. The convergence By this point, the boundary between masked image modeling and joint-embedding learning has largely dissolved. iBOT and DINOv2 are joint-embedding methods with a masked latent prediction term. data2vec and I-JEPA are masked prediction methods with a joint-embedding teacher. All of them use a momentum teacher to produce targets in a learned representation space, all of them include some form of masking, and they differ mainly in whether they also use augmentation-based global invariance, in the shape and size of the masked regions, and in whether the predictor is a separate network or the encoder itself. The design question has thus been refined into three sub-questions. What is the target computed from: pixels, a fixed transform, a learned tokenizer, or the network's own teacher? Is it global, local or both? And what transformations relate the student's input to the teacher's: masking, augmentation or both? The strongest general-purpose encoders of the present, DINOv2 and DINOv3, answer: the network's own teacher; both global and local; and both masking and augmentation. I-JEPA and its video successors answer: the network's own teacher; local but semantic; and masking only. Each answer produces a representation with a recognisable character, and the choice between them depends on what the representation will be used for. There is one further source of targets that this booklet has not yet discussed, and it is the one most widely used in practice: language. The next chapter considers what happens when the target is not a view of the image at all, but a sentence describing it. Chapter 8. Language as a Target: CLIP, Zero-Shot Recognition and the Case for Pure Vision Every method in the preceding chapters learns from images alone. The training signal comes from the relationship between parts or views of the same image. There is another abundant source of free supervision: the text that accompanies images on the web. Photographs on news sites have captions; images on shopping sites have product descriptions; images embedded in web pages have alt-text written for accessibility or search engines. This text is noisy, inconsistent and often unrelated to what the image shows, but there are billions of examples, and nobody had to be paid to produce them for machine learning. Methods that learn from image-text pairs are not self-supervised in the strict sense, because the text is an external signal rather than something derived from the image. They are usually called weakly supervised or language-supervised. But they occupy the same practical niche as self-supervised methods, learning general-purpose visual representations without manually curated labels, and they introduced the capability that most people now associate with modern vision models: zero-shot recognition. Any account of how visual features are learned without human labelling must explain them, and must also explain the evidence that has accumulated since 2023 about what language supervision adds and what it costs. CLIP In January 2021, Alec Radford, Jong Wook Kim and colleagues at OpenAI introduced CLIP, short for Contrastive Language-Image Pre-training. The method is contrastive learning of the kind described in Chapter 3, with one change: the two "views" of each training example are an image and its accompanying text. CLIP has two encoders, one for images, a ResNet or vision transformer, and one for text, a transformer. Each maps its input to a vector in a shared embedding space. In each training batch of N image-text pairs, the model computes the cosine similarity between every image and every text, an N by N matrix of scores. The N matching pairs on the diagonal are positives; all other combinations are negatives. The loss is a symmetric cross-entropy, asking each image to pick out its own text from among all texts in the batch and each text to pick out its own image. The original CLIP used batches of 32,768 pairs and a dataset of 400 million image-text pairs collected from the web, which the paper called WebImageText. The result was a model whose image embeddings were aligned with the embeddings of natural-language descriptions. This alignment made possible a new way of using a vision model. To classify an image into one of a set of categories, one does not need to train a classifier. One writes a short text prompt for each category, such as "a photo of a dog", "a photo of a cat", and so on, encodes each prompt with the text encoder, and assigns the image to the category whose text embedding is most similar to the image's embedding. The text encoder synthesises a classifier from the category names. This is zero-shot classification: the model is applied to a task it was never trained on, with no task-specific examples. The best CLIP model, a ViT-Large with 14-pixel patches evaluated at 336 pixels, reached 76.2 percent zero-shot top-1 accuracy on ImageNet. That matched the accuracy of a ResNet-50 trained with supervision on ImageNet's 1.28 million labelled images, without CLIP having used any of them. The choice of prompt mattered: the template "a photo of a {label}" worked better than the bare label, and averaging the embeddings of many templates, such as "a blurry photo of a {label}" and "a drawing of a {label}", worked better still. CLIP was also strikingly robust. On variants of ImageNet designed to test distribution shift, such as ImageNet-Sketch, which contains drawings of the ImageNet classes, and ImageNet-R, which contains renditions such as paintings, cartoons and sculptures, supervised ImageNet models lose much of their accuracy. Zero-shot CLIP lost much less. The CLIP paper argued that this "effective robustness" arose because the model had never been fitted to the particular statistics of ImageNet's training set. Later analysis, notably by Alex Fang and colleagues in 2022, traced the robustness mainly to the diversity of the training data rather than to language supervision as such. The limits of a caption CLIP's strengths and weaknesses both follow from its target. The target is a caption, and a caption is a human's choice of what is worth saying about an image. Captions name objects and sometimes their attributes, actions and settings. They rarely describe spatial layout precisely, almost never mention the count of objects unless it is notable, and say nothing about textures, exact positions, geometry or the many visual properties that are obvious to a human and therefore unremarkable. A caption for a photograph of a kitchen might say "modern kitchen with marble countertops", and say nothing about the position of the sink relative to the window. CLIP's representation inherits these priorities. It is very good at recognising the kinds of things people name, and comparatively poor at tasks that require fine spatial reasoning, counting, or distinguishing visual details that captions do not mention. Several studies documented these weaknesses. The CLIP paper itself reported that zero-shot CLIP performed poorly on tasks such as counting objects in synthetic scenes, estimating distances, and fine-grained classification of car models or aircraft variants. It was susceptible to "typographic attacks": an apple with a piece of paper reading "iPod" stuck to it would be classified as an iPod, because the model had learned to read text in images and to weight it heavily, as captions often describe what is written in an image. Shengbang Tong and colleagues showed in 2024, in a paper titled "Eyes Wide Shut?", that there are many pairs of visually distinct images that CLIP embeds almost identically, for instance images differing in the orientation of an object or the presence of a small detail. Multimodal language models built on CLIP's vision encoder inherited these blind spots and failed to answer simple visual questions about them. Supplementing CLIP features with features from DINOv2, a purely self-supervised model, reduced these failures. The most consistent weakness concerns dense features. CLIP's contrastive loss operates on a single image-level embedding. Like the image-level contrastive methods of Chapter 3, it provides no direct signal for patch-level representations, and CLIP's patch tokens are poorly localised: they do not reliably correspond to the objects at their locations. This is why, in the frozen-feature comparisons of the DINOv2 paper, text-supervised models fell well behind DINOv2 on segmentation and depth estimation, while being competitive on image-level classification. It is also why efforts to use CLIP for zero-shot segmentation, discussed in the next chapter, required substantial adaptation. Successors and refinements CLIP's approach was rapidly scaled and refined. ALIGN, from Chao Jia and colleagues at Google in 2021, used a dataset of about 1.8 billion image and alt-text pairs with minimal filtering, and showed that scale could compensate for noise. The LAION project released openly available datasets of image-text pairs, culminating in LAION-5B in 2022, and the OpenCLIP project used them to train and release open CLIP models; a 2023 study by Mehdi Cherti and colleagues measured how OpenCLIP's performance scaled with data, model size and compute. DataComp, a benchmark introduced by Samir Yitzhak Gadre and colleagues in 2023, held the model and training procedure fixed and asked participants to improve results purely by filtering a common pool of image-text pairs, which established that data curation was among the most powerful levers available. MetaCLIP, from Hu Xu and colleagues at Meta in 2023, attempted to reproduce the curation strategy behind CLIP's original dataset by balancing the distribution of training examples over a large list of concepts. The loss function was also improved. SigLIP, introduced in 2023 by Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov and Lucas Beyer at Google, replaced CLIP's softmax over the whole batch with an independent sigmoid loss on each image-text pair, treating each as a binary question of whether the image and text match. This removed the need to normalise across the batch, made training more memory-efficient, and performed better at smaller batch sizes. SigLIP models became widely used as vision encoders for multimodal language models. SigLIP 2, released in early 2025, added a captioning objective, self-distillation and masked prediction to the training recipe, explicitly to improve localisation and dense features. In other words, it borrowed the local, latent-prediction targets of the self-supervised methods in Chapters 5 to 7 to repair the weakness of a global, caption-level target. A related line of work used the strengths of each family to cover the other's weaknesses. Models such as AM-RADIO, from Mike Ranzinger and colleagues at NVIDIA in 2024, distilled several teachers, including a CLIP model, DINOv2 and the Segment Anything Model, into a single student encoder, aiming to combine CLIP's language alignment with DINOv2's dense features and SAM's boundary precision. Conversely, Meta's dino.txt work trained a text encoder to align with frozen DINOv2 features, and DINOv3 released a text-aligned variant, giving purely self-supervised visual features a route to zero-shot classification. Does vision need language? The success of CLIP raised a pointed question for self-supervised vision. If text supervision is free at web scale and yields zero-shot capability, why bother learning from images alone? There are several answers, and they connect directly to the booklet's argument about targets. The first is that language is a lossy target. A caption encodes what a human chose to say, and much of what matters visually is never said. A representation trained only to match captions will be shaped by that selection, just as a representation trained on ImageNet labels is shaped by the taxonomy of those labels, if less narrowly. Self-supervised targets derived from the image itself are not subject to that selection. The second is that many important image domains have no text. Microscopy images, satellite images, industrial inspection images, medical scans, and the camera feeds of robots and vehicles rarely come with captions, and where they do, such as radiology reports, the reports are specialised and limited in quantity. For these domains, self-supervised learning from images alone is often the only scalable option, and the methods of Chapters 5 to 7 have been widely applied to them. The third answer is empirical. For several years, a common view held that language supervision was necessary for the best performance on multimodal tasks, such as visual question answering with a language model. In 2025, David Fan, Shengbang Tong and colleagues at Meta and New York University tested this directly in a paper titled "Scaling Language-Free Visual Representation Learning". They trained DINO-style self-supervised models and MAE models on the same two-billion-image dataset used for a MetaCLIP baseline, scaling model size from about one billion to seven billion parameters, and evaluated all of them as vision encoders for a multimodal language model on a broad suite of visual question-answering benchmarks. The self-supervised models' performance kept improving with model size and data where the CLIP models' gains slowed, and at the largest scales the self-supervised models matched CLIP on average across the suite. On tasks involving reading text and charts in images, where CLIP had a clear advantage, the gap narrowed substantially when the self-supervised models were trained on a subset of data richer in text-heavy images. The authors' conclusion was that the advantage previously attributed to language supervision was largely an advantage of training data and scale. None of this makes language supervision unimportant. For zero-shot classification and retrieval by text query, a shared image-text embedding space is the most direct solution, and CLIP-style models remain the standard tool. But the evidence suggests that the core visual knowledge in these models does not have to come from language, and that the two kinds of target are complementary rather than competing. Language supervision supplies alignment with human concepts and names. Self-supervision supplies the visual structure that names leave out. Table 4 sets the three main families of label-free encoder against the properties that most often decide which to use. Table 4. Characteristic properties of the main families of label-free visual encoders. Property Self-distillation (e.g. DINOv2) Masked autoencoding (e.g. MAE) Language-supervised (e.g. CLIP, SigLIP) Training signal Image views and masked patches Hidden pixels Paired captions Frozen global features Strong Weak without tuning Strong Frozen dense features Strong Moderate Weak to moderate Native zero-shot classification No, needs text alignment No Yes Best use Frozen backbone for dense and global tasks Initialisation for fine-tuning Text retrieval, open-vocabulary tasks Usable without text Yes Yes No The next chapter turns to how these encoders are actually used: how their quality is measured, and how their features are turned into classifiers and segmentation maps, with and without task-specific labels. Chapter 9. From Features to Tasks: Evaluation, Classification and Segmentation Without Labels A self-supervised encoder is not an end in itself. Its value lies in what can be built on it: a classifier for a new set of categories, a segmentation model for medical scans, a depth estimator for a robot. This chapter is about that final step, and about the closely related question of how one decides whether one representation is better than another. The two topics are inseparable. The way a representation is evaluated determines which methods appear to work, and several of the field's apparent disagreements turn out to be differences in evaluation protocol rather than in representation quality. How representations are measured Four protocols dominate the evaluation of self-supervised representations, and they measure different things. Table 5 summarises them. Table 5. Standard protocols for evaluating self-supervised visual representations. Protocol What is trained Labels needed What it measures Favours k-nearest neighbours Nothing; labelled features are stored Labelled reference set Whether semantic neighbours are already close Joint-embedding, self-distillation Linear probe One linear layer on frozen features Full labelled training set Linear accessibility of task information Joint-embedding, self-distillation, CLIP Attentive or shallow probe A small attention layer or MLP on frozen features Full labelled training set Information present in frozen tokens, less strictly linear Masked modeling and latent prediction gain most Full fine-tuning Entire network Full labelled training set Quality as an initialisation Masked image modeling Low-shot Linear layer or fine-tuning One to a few percent of labels, or a handful per class Label efficiency Methods with clustered, semantic features k-nearest-neighbour classification stores the frozen features of every labelled training image and classifies a test image by a vote among its nearest neighbours in feature space. Nothing is trained. A high k-NN accuracy means the representation has already arranged images so that semantically similar ones are close, which is the most direct measure of whether self-supervision has discovered semantic structure. Linear probing freezes the encoder and trains a single linear layer on its features using the labelled training set. It asks whether the information needed for the task is linearly accessible. It is cheap, it isolates the representation from the effects of further training, and it was the standard headline metric of the contrastive era. Fine-tuning initialises a network with the self-supervised weights and trains all of them on the labelled task. It measures the quality of the representation as a starting point for optimisation, which is a different property from the quality of the frozen features. A representation can be a superb initialisation without being linearly separable, and MAE is the canonical example. Low-shot evaluation trains a classifier, linear or fine-tuned, using only a small fraction of the labels, such as one percent or ten percent of ImageNet, or a handful of examples per class. It tests label efficiency, which is often the practical reason for using self-supervision in the first place. The disagreement between masked image modeling and joint-embedding methods described in Chapter 6 is, in large part, a disagreement between protocols. Under full fine-tuning, MAE beats contrastive methods. Under linear probing, the reverse holds. Neither result is wrong; they measure different properties. The question for a practitioner is which property matters. If the downstream task has plenty of labels and compute for fine-tuning is available, MAE-style pretraining is an excellent choice. If the encoder will be used frozen, for instance because it must serve many tasks at once, or because there are too few labels to fine-tune a large network without overfitting, then the frozen-feature metrics matter, and self-distillation methods such as DINOv2 are generally stronger. The trend since 2023 has been towards frozen evaluation as the primary standard, for three reasons. Frozen features let a single encoder serve many tasks without the cost of storing a separate fine-tuned network for each. They reveal the quality of the representation directly rather than through the lens of further optimisation. And with very large models, fine-tuning every parameter for every task is often impractical. The DINOv2 paper made this explicit by evaluating almost everything with frozen features, and V-JEPA's evaluation was framed around frozen features with an attentive probe, a small cross-attention layer that pools patch tokens into a prediction and gives masked-prediction methods, whose information is distributed across tokens rather than concentrated in a class token, a fairer test. Evaluation also involves choices of benchmark. ImageNet remains the reference, but it rewards a particular kind of object-centric, single-label recognition. A representation can be strong on ImageNet and weak on fine-grained species identification, satellite imagery, texture classification, or medical images. Broad suites of transfer tasks, and evaluations on out-of-distribution variants of ImageNet, are now routinely reported alongside ImageNet accuracy, and a representation's average performance across many tasks is a better indicator of generality than its peak on one. Classification with frozen features For image classification, using a self-supervised encoder is straightforward. Extract the class token, or the average of the patch tokens, for each labelled training image, and train a linear classifier or a small multilayer perceptron on these features. With strong modern encoders, this simple recipe often matches or approaches the accuracy of fine-tuning for moderately sized datasets, at a small fraction of the cost. Two details commonly matter in practice. The first is which layer's output to use. For self-distillation models, the final layer's class token is usually best for classification, and concatenating the class token with the averaged patch tokens, or with the outputs of the last few layers, often helps further; the DINOv2 linear evaluation used such concatenations. For masked-modeling models, intermediate layers may be more linearly informative than the final layer, for the reasons discussed in Chapter 6. The second is resolution. Self-supervised models trained at one resolution may perform better when evaluated at a somewhat higher one, especially for small objects, and most recent models include a high-resolution adaptation phase at the end of training precisely so that they behave well at the resolutions used downstream. When labels are extremely scarce, the k-NN structure of self-supervised feature spaces becomes useful directly. With a handful of labelled examples per class, one can classify new images by their nearest labelled neighbours, or by similarity to the average feature of each class, with no training at all. Mahmoud Assran and colleagues' Masked Siamese Networks, published in 2022, were designed specifically for this low-shot regime, combining masking with a DINO-like objective, and reported strong ImageNet accuracy with only one percent of the labels or a few images per class. Semantic segmentation with frozen features Semantic segmentation assigns a class label to every pixel of an image. It is the dense task on which the differences between self-supervised representations are most visible. The simplest protocol is a linear segmentation probe. Freeze the encoder, extract the patch tokens for each image, train a linear layer that maps each patch token to class scores, and upsample the resulting low-resolution map to the full image size. With a 14-pixel patch size and an input of 518 pixels, the patch grid is 37 by 37, so a linear probe produces a coarse segmentation that is then bilinearly upsampled. On the ADE20K benchmark, which contains 150 categories of objects and scene elements across indoor and outdoor scenes, and on the Cityscapes and PASCAL VOC benchmarks, linear probes on DINOv2 and DINOv3 patch features produce segmentations of genuinely useful quality, far better than those obtained from CLIP patch features under the same protocol. This is the clearest practical consequence of the local, patch-level targets that iBOT introduced and that DINOv2 and DINOv3 retained and protected. That a linear layer on frozen features can segment a scene means the representation has already organised its patch features so that patches belonging to the same kind of object or surface lie in similar regions of feature space, regardless of where they appear or what image they come from. Nothing in the training objective named those categories. The structure emerges because patches of the same kind of thing are predictable from similar contexts, and are therefore assigned similar targets by the teacher. For higher accuracy, the frozen features can feed a heavier decoder, such as a multi-scale head or a Mask2Former-style transformer decoder, or the whole network can be fine-tuned. The patch size sets a limit on spatial precision: a 14- or 16-pixel patch cannot resolve boundaries within itself. Several methods have been proposed to upsample frozen features to pixel resolution while preserving their semantics, among them FeatUp, from Stephanie Fu, Mark Hamilton and colleagues in 2024, which learns an upsampler trained to be consistent with the low-resolution features under small image transformations. These approaches are useful for tasks that need sharp boundaries, but they do not change the basic fact that the semantic content of the segmentation comes from the self-supervised patch features. Segmentation with no labels at all The most ambitious use of self-supervised features is segmentation without any human labels, either to discover objects in an image or to assign consistent semantic categories to pixels across a dataset. This is where the emergent properties of DINO, described in Chapter 5, were turned into practical methods. Object discovery from patch similarity. LOST, introduced in 2021 by Oriane Siméoni and colleagues, used frozen DINO features to localise the main object in an image. It identified a seed patch that was similar to few other patches, on the reasoning that foreground patches are less widespread in an image than background patches, then expanded the seed to include patches similar to it, and produced a bounding box. TokenCut, from Yangtao Wang and colleagues in 2022, built a graph in which each patch was a node and edges were weighted by the similarity of DINO features, then applied the normalised cut algorithm, a classical spectral method for partitioning graphs introduced by Jianbo Shi and Jitendra Malik in 2000, to split foreground from background. Luke Melas-Kyriazi and colleagues' Deep Spectral Methods, also in 2022, computed eigenvectors of a similar patch-affinity matrix and found that they decomposed images into meaningful segments. All three relied on the same observation: in a self-supervised ViT's feature space, the patches of a single object are much more similar to each other than to the background. Unsupervised instance segmentation. CutLER, introduced in 2023 by Xudong Wang, Rohit Girdhar, Stella Yu and Ishan Misra, extended these ideas to multiple objects. Its MaskCut procedure repeatedly applied normalised cuts to DINO patch features, masking out each discovered object and cutting again, to extract several object masks per image. These masks were then used as pseudo-labels to train a standard instance-segmentation detector, which was refined over several rounds of self-training in which the detector's own confident predictions became the next round's labels. The resulting detector, trained without any human annotation, could detect and segment objects across a variety of benchmarks, improving substantially on previous unsupervised methods, though still well below fully supervised detectors. Unsupervised semantic segmentation. Assigning consistent category labels to pixels across a whole dataset, without labels, is harder than discovering objects in a single image. STEGO, introduced in 2022 by Mark Hamilton and colleagues at MIT, Microsoft and Google, observed that the correlations between DINO patch features across different images are already semantically consistent: patches showing people in one image are similar to patches showing people in another. STEGO trained a small segmentation head on top of frozen DINO features to distil these correspondences into compact, cluster-friendly features, and then clustered them. On benchmarks such as COCO-Stuff and Cityscapes, it improved significantly on earlier unsupervised semantic segmentation methods. The clusters it produced are not named categories; a human or a language model must still assign names to them, or they must be matched to ground-truth categories for evaluation. These methods have a characteristic pattern. None of them trains a new representation. All of them exploit the structure already present in frozen self-supervised features, using classical techniques such as graph cuts, spectral clustering or self-training. They are, in effect, ways of reading out what the self-supervised objective has already taught the network. Zero-shot and open-vocabulary segmentation Unsupervised segmentation discovers regions without names. Open-vocabulary segmentation aims to segment regions corresponding to arbitrary categories named by text at test time, such as "the red umbrella" or "lichen on the rock", without having been trained on those categories. This requires connecting visual features to language, so it draws on language-supervised models. The difficulty, as Chapter 8 explained, is that CLIP's patch features are poorly localised. MaskCLIP, from Chong Zhou, Chen Change Loy and Bo Dai in 2022, found that a small modification to CLIP's final attention layer, essentially using the value embeddings of each patch directly rather than the attention-pooled output, produced patch features that could be compared with text embeddings to give a rough zero-shot segmentation. A series of later methods improved on this by training segmentation models on datasets with a limited set of labelled categories while keeping their classifiers in CLIP's text space, so that new categories could be added at test time by writing their names. Approaches such as LSeg, OpenSeg, ODISE and FC-CLIP took variations of this path. They are not strictly zero-shot, because they use some labelled segmentation data, but they generalise to categories never seen in training. The Segment Anything Model, released by Alexander Kirillov and colleagues at Meta in 2023, approached the problem from another angle. SAM is a promptable segmentation model: given an image and a prompt, such as a point, a box or a rough mask, it outputs a segmentation of the indicated object. It was trained on SA-1B, a dataset of about 11 million images and 1.1 billion masks produced through a data engine in which an earlier version of the model proposed masks that annotators corrected, and later produced masks fully automatically. SAM's image encoder was a ViT-Huge pretrained with MAE. SAM does not know what objects are called, but it segments almost anything it is pointed at with sharp boundaries, and it is widely combined with an open-vocabulary detector or a CLIP-like model that supplies the names and locations, in systems such as Grounded SAM. SAM's success illustrates the point of Chapter 6 from the other side: MAE pretraining, whose frozen features are weak for classification, is an excellent initialisation for a model fine-tuned to produce precise masks. Beyond segmentation The same frozen-feature approach extends to other dense tasks. Monocular depth estimation, predicting the distance of every pixel from a single image, has been transformed by self-supervised encoders. The Depth Anything models, introduced by Lihe Yang and colleagues in 2024, used a DINOv2 encoder, and their authors reported that the DINOv2 initialisation was important to their quality. Keypoint correspondence, matching the same physical point across images of different objects of the same category, can be done by nearest-neighbour search on DINO patch features, as Shir Amir and colleagues showed in 2021, and has since been applied to tasks such as transferring part annotations between images. Robot learning systems routinely use frozen DINOv2 or similar features as visual inputs for policies, because those features are robust and carry spatial information. In each case the pattern is the same as in segmentation. The self-supervised objective, by asking the network to predict local structure from context, has organised patch features so that geometric and semantic regularities are linearly or nearly linearly accessible. The downstream task reads that organisation out. The more a target protects local structure during pretraining, the more such tasks the representation supports without further training. Chapter 10. Scale, Data and the Ways Self-Supervision Fails The promise of self-supervised learning was always partly a promise of scale. If the training signal comes from images themselves, the limit on data is the number of images one can collect, and the limit on model size is the compute one can afford. By 2025, the largest self-supervised vision models had reached about seven billion parameters and had been trained on well over a billion images. Scaling to that point taught the field three lessons that were not obvious at the outset. Uncurated data is a poor substitute for curated data, even when there is far more of it. Domain-specific self-supervised models can outperform general ones by wide margins in fields such as pathology and remote sensing. And large self-supervised models fail in characteristic ways, each of which can be traced to something their training target does not protect. The data problem The obvious way to scale self-supervision is to train on more images, and the obvious source of more images is the web. Early attempts gave mixed results. In 2021, Priya Goyal and colleagues at Facebook AI Research trained a model called SEER, using SwAV on a RegNet convolutional network with about 1.3 billion parameters, on one billion random, uncurated public Instagram images. After fine-tuning, SEER reached 84.2 percent top-1 accuracy on ImageNet, which showed that self-supervision could make use of uncurated data at large scale. A follow-up in 2022 scaled the approach to ten billion parameters and argued that models trained this way were more robust and showed smaller performance gaps across demographic groups on some fairness benchmarks than models trained on ImageNet. But in frozen-feature evaluations, models trained on uncurated data consistently lagged models trained on curated datasets such as ImageNet, even when the curated datasets were far smaller. The reason is that uncurated web data is badly distributed. It contains enormous numbers of near-duplicates, it is dominated by a small number of common kinds of image, and rare but important visual concepts appear very infrequently. A self-supervised objective treats every image as equally important. On uncurated data, most of the training signal is spent on the most common types of image, and the long tail of visual concepts receives little attention. Contrastive and self-distillation methods are particularly sensitive to this, because their targets depend on comparisons between images, and a batch dominated by similar images provides weak contrast. DINOv2's response, described in Chapter 5, was to curate automatically. Starting from a large uncurated pool, the authors deduplicated it and then retrieved images similar to those in a collection of curated datasets, producing LVD-142M. Huy Vo and colleagues, many of them DINOv2 authors, developed this idea further in 2024 in a paper on automatic data curation through clustering. Their approach applied hierarchical k-means clustering to the embeddings of an uncurated pool and then sampled evenly across clusters at each level of the hierarchy, so that the resulting dataset was balanced across visual concepts rather than dominated by the most common ones. Models trained on data curated this way approached or matched the performance of those trained on manually curated datasets. DINOv3's training set of about 1.7 billion images was assembled with a combination of clustering-based balancing and retrieval. The lesson is uncomfortable for the original promise of self-supervision. Labels are not needed, but curation is, and curation is a form of supervision in disguise: it encodes a judgement about which images are worth learning from. The judgement can be automated, using embeddings from earlier self-supervised models, but it still reflects choices about what distribution the model should learn from. The 2025 language-free scaling study described in Chapter 8 made the point in a different way, showing that the composition of the training data, in particular its share of images containing text, decided whether self-supervised models could match CLIP on text-heavy tasks. Scaling models Self-supervised vision models have grown in two directions: more parameters, and more training. Both have exposed problems. The jump from ViT-Large to ViT-giant and beyond required substantial engineering. Training instabilities, which MoCo v3 had found in the patch-embedding layer of much smaller models, recurred in new forms. Large models need careful learning-rate warm-up, stabilising architectural details such as layer-scale and query-key normalisation, mixed-precision arithmetic, and the sharding of parameters and optimiser states across many devices. These are not glamorous contributions, but much of the improvement from DINO to DINOv2 came from them. A more fundamental question is whether self-supervised models scale as predictably as language models do. The evidence is mixed. Masked autoencoders showed steady improvements with model size on fine-tuning tasks. DINOv2-style models improved with scale on both global and dense frozen evaluations up to about a billion parameters. The 2025 language-free scaling work found that DINO-style models kept improving on visual question answering up to seven billion parameters, where CLIP's gains were flattening. But DINOv3's authors found that, without Gram anchoring, very long training of a seven-billion-parameter model degraded dense features even as global metrics improved, so scale on its own did not reliably improve every property. The general pattern is that scaling improves whatever the target directly rewards, while properties that the target protects only indirectly can erode. The compute required is very large. Training a seven-billion-parameter vision transformer on a billion and a half images is affordable only to a handful of organisations. One consequence is that the field's most capable self-supervised models come from a small number of industrial laboratories, and their release as open weights, as with DINOv2, DINOv3 and many MAE and CLIP variants, is what makes them available to everyone else. Distillation into smaller models, which all recent families provide, partly addresses the cost of using them: a ViT-Small or ViT-Base distilled from a seven-billion-parameter teacher runs on modest hardware and retains much of the teacher's quality. Domain-specific self-supervision The strongest argument for self-supervised learning in practice may not be general-purpose web models at all, but models trained on large unlabelled collections from specialised domains where labels are scarce and expensive. Computational pathology offers the clearest example. Hospitals hold vast archives of whole-slide images, digitised tissue sections that can each run to billions of pixels, most of them without detailed annotations. Since 2023, several research groups have trained self-supervised foundation models on these archives. UNI, from Richard Chen, Ming Lu, Faisal Mahmood and colleagues, published in Nature Medicine in 2024, trained a ViT-Large with DINOv2 on more than 100 million tissue image patches drawn from over 100,000 whole-slide images, and evaluated it on dozens of clinical tasks, including cancer subtyping and tissue classification. Virchow, from Eugene Vorontsov and colleagues at Paige and Memorial Sloan Kettering, also published in Nature Medicine in 2024, trained a ViT-Huge with DINOv2 on about 1.5 million whole-slide images. Prov-GigaPath, from Hanwen Xu and colleagues at Microsoft and Providence, published in Nature in 2024, combined a DINOv2-trained tile encoder with a slide-level encoder pretrained by masked autoencoding, using over a billion tiles from more than 170,000 slides. All three reported clear gains over models pretrained on natural images, and many pathology studies now begin from one of these encoders rather than from ImageNet weights. Ophthalmology offers another. RETFound, from Yukun Zhou, Pearse Keane and colleagues at University College London and Moorfields Eye Hospital, published in Nature in 2023, pretrained a vision transformer with MAE on about 1.6 million unlabelled retinal images. Fine-tuned with small labelled datasets, it outperformed comparison models at detecting a range of eye diseases and showed promise in predicting some systemic conditions, such as heart failure and myocardial infarction, from retinal images. Remote sensing provides a third. Satellite and aerial imagery is plentiful and largely unlabelled, but differs from natural photographs: it has no canonical orientation, often contains more than three spectral bands, and its objects of interest range from individual trees to entire cities. SatMAE, from Yezhen Cong and colleagues in 2022, adapted masked autoencoding to multispectral and time-series satellite data. Jamie Tolan and colleagues at Meta and the World Resources Institute trained a DINOv2 model on high-resolution satellite imagery and used it in 2024 to produce global maps of forest canopy height from single images. DINOv3 released a separate model trained on satellite imagery alongside its web-image model. These domain models illustrate the booklet's argument from another angle. The choice of target interacts with the domain. Colour-distortion augmentations, standard for natural images, can destroy diagnostically important information in stained tissue, where colour carries meaning, so pathology models adjust their augmentations. Rotation invariance, harmful for natural photographs, is appropriate for overhead imagery and for tissue sections, which have no canonical orientation. Masked modeling and latent prediction, which do not depend on augmentation, transfer to new domains more readily for this reason. Domain experts adapting self-supervised methods are, in effect, redesigning the target to fit what matters in their images. How large self-supervised models fail Several failure modes of self-supervised vision models have been documented. Each corresponds to something the training target does not protect. Artifact tokens. As described in Chapter 2, large ViTs develop a small number of high-norm tokens in low-information background regions, which the network uses as storage for global information. These degrade attention maps and dense features. The cause is that nothing in the target requires every patch token to represent its own patch faithfully; the network is free to repurpose some of them. Adding register tokens, as proposed by Darcet and colleagues, gives the network a designated place for global storage and largely removes the artifacts. Dense feature degradation. As DINOv3's authors documented, long training with global and masked objectives can gradually erode the spatial consistency of patch features while global metrics continue to improve. The cause is that the global objective is a stronger signal than the local one at large scale, and the local structure is protected only indirectly. Gram anchoring adds an explicit target for the relational structure among patches. Inherited invariances. Augmentation-based methods become insensitive to whatever their augmentations change. A model trained with heavy colour distortion will have weaker colour features, which harms tasks where colour matters. The cause is direct: the target tells the network to discard that information. Object-centric bias. Methods that pull random crops together assume that crops of the same image share content, an assumption that holds for object-centric images and fails for cluttered scenes. Models trained on scene-centric data with such methods learn less, and models trained on object-centric data may represent multi-object scenes poorly. Memorisation. Self-supervised models can memorise aspects of their training images. Casey Meehan, Florian Bordes and colleagues at Meta showed in 2023, in work on what they called déjà vu memorisation, that a self-supervised model given only the background crop of a training image could often be used to infer the identity of the foreground object, at rates that exceeded what could be explained by general correlations between backgrounds and objects. This has implications for privacy when models are trained on sensitive images, such as medical scans or personal photographs. The cause is that instance discrimination and its successors reward distinguishing individual training images, which can mean encoding idiosyncratic details of each. Social bias. Self-supervised models trained on web images reflect the distribution of those images, including the over- and under-representation of groups of people and the associations present in the data. Self-supervision removes the biases of a human labelling scheme, but not those of the image collection itself. Several studies have found biased associations in the embeddings of self-supervised models, and the fairness claims made for uncurated-data models such as SEER apply to specific benchmarks rather than establishing an absence of bias. What the failures share Each failure mode is, at bottom, a case of the network doing exactly what the target asks and nothing more. Artifact tokens appear because nothing asks every token to represent its patch. Dense degradation appears because the local target is weaker than the global one. Inherited invariances appear because the augmentations demand them. Memorisation appears because the target rewards telling individual images apart. Bias appears because the target takes the data distribution as given. This makes them tractable. Each has a remedy that changes the target or the data: registers give global information a home, Gram anchoring protects patch structure, augmentation choices can be adjusted for the domain, data curation can rebalance the distribution, and differential privacy or deduplication can limit memorisation. None of these remedies is free, and some, such as debiasing, are only partially understood. But the pattern suggests a general diagnostic for anyone using a self-supervised encoder: when it fails at something, ask whether its target ever required it to succeed at that thing. Usually the answer is no. Conclusion: Choosing What to Predict The history traced in this booklet can be told as a sequence of methods, each surpassing the last on some benchmark. Told that way, it suggests that the field is converging on a single best recipe, which one should adopt and stop thinking about. The argument of the preceding chapters points to a different conclusion. Self-supervised vision has not converged on an answer; it has converged on a question, and on a much better understanding of how the answer to that question shapes what a representation knows. The question is what the network is asked to predict, from what, and in what space. Pretext tasks predicted geometric facts, rotations and permutations, which networks could often determine from low-level cues. Contrastive learning predicted identity across augmentations, which made the hand-designed augmentations into the specification of what to ignore, and produced strong global features with little local structure. Non-contrastive methods showed that the same invariance could be learned without negatives, so long as something else, a stop-gradient, a decorrelation term, or a balanced clustering, held the representation open. Masked autoencoders predicted pixels, which gave a dense signal free of augmentation bias but committed the representation to modelling pixel detail that perception does not need. Latent prediction moved the masked target into the network's own feature space, so the representation could concentrate on what is predictable. Language supervision predicted captions, which aligned images with human concepts at the cost of everything captions leave unsaid. And the strongest general-purpose encoders of the present combine a global invariance target with a local masked target, both in latent space, and then add further targets, registers and Gram anchoring among them, to protect properties the first two do not. At every stage, the representation learned what its target demanded and let go of what the target did not protect. That is the single most useful thing to understand about self-supervised vision, and several practical consequences follow from it. What follows for practice Choose an encoder by what it was asked to predict, not by its ImageNet score. A practitioner choosing a pretrained encoder for a new task should ask what properties the task needs and which training target would have produced them. If the task requires dense, spatially precise features used frozen, such as segmentation, depth estimation, correspondence or robotic perception, a self-distillation model with a strong local target, such as DINOv2 or DINOv3, is usually the best starting point. If the task will be solved by fine-tuning a large model on a substantial labelled dataset, an MAE-style initialisation is excellent, and its weak frozen features do not matter. If the task needs open-vocabulary recognition or retrieval by text, a language-supervised model is the direct route, ideally combined with a self-supervised encoder where spatial detail matters. If the task depends on properties that augmentations would destroy, such as colour, orientation or absolute scale, prefer methods that rely on masking rather than heavy augmentation, or check the augmentation pipeline of the model being considered. The ImageNet linear-probe score of an encoder is a poor guide to any of these decisions. Evaluate with the protocol that matches the use. The disagreement between masked modeling and contrastive learning was, in large part, a disagreement between fine-tuning and linear probing. Anyone comparing encoders for a specific application should use the protocol that matches how the encoder will be deployed, frozen or fine-tuned, with abundant labels or few, on in-distribution images or shifted ones. Label-free diagnostics such as the effective rank of the embedding are useful for screening candidates, but they are no substitute for a small evaluation on the actual task. In a specialised domain, consider training rather than borrowing. The pathology, ophthalmology and remote-sensing models described in Chapter 10 show that a self-supervised model trained on large unlabelled collections from the target domain often outperforms the best general models by wide margins. For organisations that hold such collections, the labels they lack are less of an obstacle than they might appear. The investment required is compute and data engineering rather than annotation, and the design choices, particularly the augmentations, should be made with domain knowledge about what in the images matters. Treat data curation as part of the method. The scaling results of the past three years point to data distribution as a decisive variable. Uncurated data underperforms curated data even at much larger scale, and balancing a dataset over visual concepts, which can be done automatically using embeddings, often improves results more than changing the objective. Curation is not a detail to be delegated; it is a statement about what the model should learn from, and it deserves the same scrutiny as the loss function. When a model fails, ask whether its target ever required success. The failure modes in Chapter 10, artifact tokens, dense degradation, inherited invariances, memorisation and bias, are all cases of a network not being asked to do something. That gives a practical diagnostic. When a self-supervised encoder performs poorly on some property, the first question is whether anything in its training target, its augmentations or its data rewarded that property. The answer usually indicates the remedy. What remains unsettled Several important questions are still open. The first is whether a single objective can produce representations that are optimal across both global and dense tasks at every scale, or whether growing models will always require additional targets to protect properties that the main objective neglects. The history from iBOT to Gram anchoring suggests the latter, but the work on principled regularisers such as SIGReg points towards objectives that might need fewer patches. The second is the relationship between self-supervised and language-supervised learning. The 2025 evidence that self-supervised models can match CLIP on multimodal benchmarks when trained on comparable data weakens the claim that language is necessary for strong visual representations. It does not settle what the ideal combination is. A model that learns its visual structure from images and its conceptual vocabulary from text, with neither target dominating, seems the natural goal, and current systems that distil several encoders into one, or align a text encoder to frozen self-supervised features, are early approximations. The third concerns prediction over time. Images are snapshots, but the visual world is dynamic, and the strongest argument for latent prediction is that it allows a model to learn what is predictable about how scenes change without modelling every pixel of the future. The video and robotics results of V-JEPA 2 are promising, but the tasks are simple, and whether representations learned by predicting the future in latent space will support complex planning and physical reasoning is not yet known. The fourth is theoretical. The field has good intuitions about why its methods work and an expanding set of formal analyses, of alignment and uniformity, of the dynamics that prevent collapse, of the biases of reconstruction versus prediction, of optimal embedding distributions. It does not yet have a theory that predicts, from a target, a data distribution and a model size, what representation will result and how well it will serve a given task. Most progress is still empirical, and most design decisions are still made by running experiments. The broader lesson The central lesson of self-supervised vision applies beyond vision. A learning system extracts from its data whatever its objective makes useful, and nothing else. Supervised learning on labels inherits the taxonomy of the labels. Contrastive learning inherits the invariances of its augmentations. Reconstruction inherits the statistics of pixels. Language supervision inherits the priorities of human description. Self-distillation inherits whatever structure its own teacher has found, which is why it can discover objects and parts that no one named, and also why it must be prevented from gradually forgetting them. None of these is a neutral window onto the visual world. Each is a choice, and the quality of a self-supervised representation is the quality of that choice, multiplied by the scale and diversity of the data on which it is made. The methods described in this booklet have become very good at extracting visual structure without human labels. The human judgement has not disappeared from the process. It has moved, from labelling individual images to deciding what the network should be asked to predict. Glossary Attentive probe. A small cross-attention layer trained on top of frozen patch tokens to pool them into a prediction; a less restrictive alternative to a linear probe. Augmentation. A random transformation applied to an image during training, such as cropping, colour distortion or blur. In joint-embedding methods, augmentations define what the representation should ignore. Class token (CLS token). An extra learnable token added to a vision transformer's input sequence, whose final output serves as a summary representation of the whole image. Collapse. A failure in which a self-supervised network maps all inputs to the same output, or to a low-dimensional subspace (dimensional collapse), satisfying its objective without learning useful features. Contrastive learning. Training a network so that representations of related views (positives) are similar and those of unrelated inputs (negatives) are dissimilar. Fine-tuning. Continuing to train all of a pretrained network's parameters on a labelled downstream task. Joint-embedding architecture. A design that encodes two views of an input separately and trains their representations to agree, as in SimCLR, BYOL and DINO. Joint-embedding predictive architecture (JEPA). A design that encodes two related inputs and uses a predictor to predict the representation of one from the other, as in I-JEPA and V-JEPA. k-NN evaluation. Classifying test images by a vote among their nearest labelled neighbours in a frozen feature space, with no training. Linear probe. Training a single linear classifier on frozen features to measure how linearly accessible task information is. Masked image modeling. Hiding part of an image and training a network to predict something about the hidden part: its pixels, a discrete token, a hand-crafted descriptor or a learned representation. Momentum encoder (teacher). A copy of the network whose weights are an exponential moving average of the trained network's weights, used to produce stable targets. Multi-crop. An augmentation strategy using two large global crops and several small local crops per image, encouraging local-to-global correspondence. Patch token. The representation of one square patch of an image inside a vision transformer. Pretext task. A task invented so that its labels can be generated automatically from unlabelled data, solved not for its own sake but to induce useful features. Projection head. A small network placed between the encoder and the self-supervised loss, discarded after training, which lets the encoder retain information the loss would otherwise suppress. Register token. An extra learnable token added to a vision transformer's input and discarded at output, giving the network a place to store global information so that patch tokens are not repurposed. Stop-gradient. An operation that treats a value as a constant during backpropagation, so that the loss updates only one branch of a network. Zero-shot classification. Classifying images into categories specified only at test time, typically by comparing image embeddings with text embeddings of category names. Further Reading Randall Balestriero, Mark Ibrahim, Vlad Sobal and colleagues, "A Cookbook of Self-Supervised Learning" (2023). A long, practical survey written by many of the researchers behind the methods in this booklet. It catalogues the families of method, the connections between them, and the training details that decide whether they work. The best single reference for anyone implementing self-supervised learning. Ting Chen, Simon Kornblith, Mohammad Norouzi and Geoffrey Hinton, "A Simple Framework for Contrastive Learning of Visual Representations" (ICML 2020). The SimCLR paper. Its ablations of augmentations, projection heads and batch size remain the clearest demonstration of how the design of a contrastive target shapes what is learned. Mathilde Caron and colleagues, "Emerging Properties in Self-Supervised Vision Transformers" (ICCV 2021). The DINO paper. Short, readable and full of the observations about attention maps and k-NN structure that redirected the field towards self-distillation and dense features. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár and Ross Girshick, "Masked Autoencoders Are Scalable Vision Learners" (CVPR 2022). The MAE paper. A model of clear experimental writing, including an honest account of the method's weakness under linear probing. Maxime Oquab, Timothée Darcet, Théo Moutakanni and colleagues, "DINOv2: Learning Robust Visual Features without Supervision" (TMLR 2024; preprint 2023). Describes the combination of DINO and iBOT objectives, the automatic data curation pipeline, and the frozen-feature evaluations that made DINOv2 a standard backbone. Oriane Siméoni and colleagues, "DINOv3" (2025). The current large-scale state of the self-distillation approach, including the account of dense feature degradation and Gram anchoring. Mahmoud Assran and colleagues, "Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture" (CVPR 2023). The I-JEPA paper, the most direct implementation of latent prediction without hand-designed augmentations. Yann LeCun, "A Path Towards Autonomous Machine Intelligence" (2022). A position paper setting out the argument for joint-embedding predictive architectures and for prediction in representation space as the basis of world models. Speculative in places, but essential for understanding the motivation behind JEPA. Alec Radford and colleagues, "Learning Transferable Visual Models From Natural Language Supervision" (ICML 2021). The CLIP paper. Its extensive evaluation section documents both the strengths of zero-shot recognition and the weaknesses that follow from a caption-level target. David Fan, Shengbang Tong and colleagues, "Scaling Language-Free Visual Representation Learning" (2025). The controlled comparison of self-supervised and language-supervised encoders at matched data and scale, and the strongest current evidence on whether vision needs language. Namuk Park and colleagues, "What Do Self-Supervised Vision Transformers Learn?" (ICLR 2023). A careful empirical comparison of the representations learned by contrastive and masked-modeling ViTs, showing their complementary character. Timothée Darcet, Maxime Oquab, Julien Mairal and Piotr Bojanowski, "Vision Transformers Need Registers" (ICLR 2024). A short paper that identifies artifact tokens in large ViTs and offers a simple remedy; a good illustration of how a model with few built-in assumptions uses its capacity in unintended ways. Hashtags: #SelfSupervisedLearning #SelfSupervisedVision #ComputerVision #DeepLearning #ArtificialIntelligence #MachineLearning #RepresentationLearning #VisualRepresentationLearning #VisionTransformers #ViT #MaskedAutoencoders #MaskedImageModeling #ContrastiveLearning #SimCLR #MoCo #BYOL #SimSiam #DINO #DINOv2 #DINOv3 #iBOT #ImageJEPA #LatentSpacePrediction #SelfDistillation #VisualPretraining #UnsupervisedLearning #ZeroShotLearning #ImageClassification #SemanticSegmentation #TransferLearning

Download the Book (PDF): Introduction At about three in the morning, somewhere in the second half of a thirty-six-hour hackathon, a certain kind of silence settles over the room. The music has been turned down. The pizza boxes have migrated to the corners. Most of the people still awake are hunched toward their screens with a particular stillness, shoulders drawn up, faces a hand's width closer to the monitor than they were at dinner, fingers moving but nothing else. If you walked around that room with a notebook you would see the same thing again and again: bodies that have not changed position in an hour, sometimes two. Nobody notices, because nobody is thinking about their body. They are thinking about a race condition, a failing deployment, a demo that has to work in five hours. That stillness is the subject of this book. Not the hunch, not the slouch, not the chin poking toward the screen, but the stillness underneath all of them. Most students in computer science and information technology have been told, at some point, to sit up straight. They have seen the diagrams of the ideal workstation with its right angles at the elbow, hip and knee, and the spine stacked like a column of coins. Many have felt a vague guilt about failing to live up to it. Some have bought a posture-correcting brace, an ergonomic chair they could not really afford, or a phone app that buzzes when they lean forward. And a good number of them still end the semester with a stiff neck, aching forearms, tight shoulders, and a nagging sense that something about the way they work is wearing them down. The argument of this book is that they have been aiming at the wrong target. There is no single correct way to sit at a keyboard, and the research of the past fifteen years has been steadily dismantling the idea that there is. What your neck, wrists and shoulders respond to is not whether your posture matches a textbook picture but how long you hold any one position, how much load you put through tissues that have not been prepared for it, and what the rest of your life is doing to your sensitivity to pain: how much you sleep, how stressed you are, how strong and active you are away from the desk. The best posture, as physiotherapists increasingly put it, is your next one. The practical skill this book teaches is how to keep changing it, and how to build a body that tolerates long hours of focused work without complaint. That is not an excuse to ignore your setup, or a claim that aches are all in your head. Some positions do concentrate load in ways that matter over a long session. A laptop on a bed genuinely forces your neck and wrists into places they will not tolerate for six hours. Specific tissues in the wrist and forearm can become irritated, and occasionally a nerve problem develops that needs a clinician's attention. This book takes all of that seriously. What it rejects is the idea that one fixed, virtuous position will protect you, and the anxiety that comes with it. Anxiety about posture turns out to be part of the problem, not a harmless side effect. Who this book is for The reader I have in mind is a university student who spends a great deal of time at a computer and intends to spend a great deal more. You might be a first-year student discovering that a problem set can eat an entire weekend, a fourth-year student living in the lab during capstone season, a teaching assistant grading on a laptop at the kitchen table, or someone who competes in programming contests and hackathons for fun. You might be doing a degree in software engineering, data science, cybersecurity, information systems, or any other field where the keyboard is the main tool. The advice applies equally to anyone else who works in long stretches at a screen, but the examples come from the world you know: all-nighters before a submission deadline, pair programming in a cramped lab, debugging sessions that stretch past midnight, the peculiar physical state of a team that has been awake since the opening ceremony of a weekend-long event. You do not need any background in anatomy. Where a technical term is useful, it is explained in plain language. Every stretch and exercise is described in words, step by step, with the position you start in, the movement you make, what you should feel, how long to hold or how many times to repeat, and what to do if it hurts. You can read a description once and do the movement at your desk without looking at anything else. How the book is organised The first chapter looks at the evidence on posture and pain, and explains why the idea of a single perfect position has not survived contact with research. The second chapter examines what actually happens to the body during a long coding session: the sustained low-level muscle work, the lack of movement, the effect of sleep loss and stress, the strain on the eyes, and how these combine. Chapters Three, Four and Five deal with the three regions that trouble programmers most, the wrists and forearms, the neck, and the shoulders and upper back. Each explains what tends to go wrong, what the evidence says about common fears, and then gives a set of targeted stretches and exercises. Chapter Six is about the workspaces students actually have, which are rarely the ones in ergonomics brochures, and how to make them good enough. Chapter Seven turns the whole thing into a routine you can use during a marathon, built around the natural pauses in programming work. Chapter Eight steps back from the desk to the larger factors that decide how much strain your body can absorb, including strength, sleep and stress, and sets out clearly when an ache needs a professional's attention. A small number of tables summarise information that is genuinely easier to compare side by side. Everything else is prose, because the reasoning matters as much as the instructions. If you understand why variety beats perfection, you will be able to improvise sensibly in situations this book never describes, which is exactly what a good programmer does with any other principle. A note on health advice This book is educational. It is written to help you understand your body and look after it during demanding periods of work, and it draws on published research and established clinical practice. It is not a substitute for individual assessment and advice from a doctor, physiotherapist or other qualified clinician who can examine you. If you have pain that is severe, getting steadily worse, waking you at night, accompanied by numbness, weakness or clumsiness in your hands, or linked to any other symptom that worries you, please see a clinician rather than relying on stretches. Chapter Eight describes these warning signs in more detail. Most universities have a student health service that can see you or point you in the right direction, and many have physiotherapy or sports medicine services on campus. The stretches and exercises here are gentle, and most people can do them safely. Still, if any movement produces sharp pain, pins and needles that linger afterwards, or a feeling that something is wrong, stop and get it checked. Mild stretching discomfort is normal. Pain that makes you wince is information, not an obstacle to push through. A different kind of discipline Programmers are often good at discipline of a certain kind. They can hold a complex system in their heads for hours, ignore distractions, and push through frustration until the tests pass. That capacity is valuable. But the same focus that makes a long session productive also makes it easy to forget that you have a body at all, until it announces itself with a stiff neck or a burning forearm. The discipline this book asks for is different and, in some ways, harder. It is the discipline of interrupting yourself, briefly and regularly, when you least feel like it. It is the discipline of moving on purpose while your mind is somewhere else, of treating your working position as something to vary rather than something to perfect, and of looking after the parts of your life that have nothing to do with code, because they turn out to matter to how your code-writing body feels. None of it takes much time. Most of it takes less than a minute. What it takes is a change in what you believe good posture is. That is where we begin. Chapter 1: The Myth of the Perfect Posture Ask a room of students to demonstrate good sitting posture and almost all of them will do the same thing. They will pull their shoulders back, lift their chest, tuck their chin slightly, and arch the lower back a little. They will hold it with visible effort for about as long as the demonstration lasts, then sink back into whatever they were doing before. Ask them why that position is good and you will hear a set of confident explanations: it keeps the spine in its natural curves, it reduces pressure on the discs, it stops the muscles from getting weak, it prevents long-term damage. These beliefs are held not only by students but by many health professionals, teachers, parents and furniture advertisers. They feel like common sense. They are also, for the most part, not supported by the evidence. This chapter explains why, not to encourage you to slump for the rest of your life, but to replace a rigid and anxious idea of posture with one that is more accurate and more useful. What the research actually shows If a particular posture were reliably protective, you would expect three things to be true. Experts would agree on what that posture is. People who habitually sit or stand in that posture would have less pain than people who do not. And changing people's posture toward it would reduce their pain. None of these holds up well. Start with agreement. In 2012, Kieran O'Sullivan and colleagues showed 295 physiotherapists in four European countries a set of nine photographs of people sitting, ranging from slumped to very upright, and asked them to choose the best sitting posture. Most chose one of two images, but those two were quite different from each other, and the choice varied between countries. Physiotherapists who chose the more upright posture also tended to hold more negative beliefs about back pain. A later study by Vasileios Korakakis and colleagues asked 544 physiotherapists in Greece a similar question about sitting and standing. They too favoured some version of upright sitting with a curve in the lower back, and almost never chose slouched positions, but they defended quite different postures with the same arguments about natural curves and muscle activity. The authors noted that these preferences persisted despite a lack of strong evidence that any specific posture leads to better health outcomes. In other words, the people most likely to give you posture advice do not agree on what the ideal is, and they use similar reasoning to justify different answers. That is what you would expect to see if the ideal were more a cultural belief than a measured fact. Now look at whether posture predicts pain. For the lower back, a large umbrella review published in 2019 by Christopher Swain and colleagues examined 41 systematic reviews on spinal posture, sitting, standing, bending, twisting and other physical exposures in relation to low back pain. Some reviews reported associations, others did not, and the reviews that relied only on prospective studies, the kind best able to show what comes first, were the least consistent. The authors concluded that there is no consensus that these physical exposures cause low back pain. Association has been documented, they wrote, but it does not provide a causal explanation. For the neck, the picture is similar. A study of more than a thousand 17-year-olds in Western Australia, led by Karen Richards and published in 2016, photographed each participant sitting and grouped them into four distinct neck posture types, from upright to slumped with the head forward. There was no significant difference in the odds of neck pain or headache between these groups. What did differ was psychological and lifestyle profile: those in the slumped, head-forward group had higher odds of depressed mood, and those in the upright group exercised more often. A Brazilian study of 150 young adults aged 18 to 21, published in 2018, looked specifically at the head-down posture people adopt while texting, the so-called text neck. Neither the participants' own view of their neck posture nor physiotherapists' assessments of it were associated with neck pain. Perhaps the most relevant study for anyone heading into a computer-heavy career followed 171 office workers from the first few months of a new job through the following year. The researchers, Bahar Shahidi and colleagues at the University of Colorado, measured a wide range of possible risk factors at the start, including physical, psychological and nervous system measures, then tracked who developed chronic neck pain that interfered with daily life. The factors that predicted it were depressed mood, lower endurance in the muscles at the back of the neck, and a weaker capacity of the nervous system to damp down pain signals. The things people usually worry about, such as the angle of the head, were not among the predictors that emerged. None of these studies, alone, proves that posture never matters. Each has limits, and cross-sectional studies in particular cannot say what causes what. But together they point in one direction. If there were a strong, simple relationship between the shape you hold at a desk and the pain you develop, it would be easier to find than this. Where the fear comes from If the evidence is so equivocal, why is the belief in perfect posture so strong? Part of the answer is simple repetition. Parents tell children to sit up straight, schools reinforce it, and generations of office-safety posters show the same idealised figure. Part of it is aesthetics: an upright posture is associated with confidence and attentiveness, and a slouch with laziness. Diane Slater and colleagues, writing in 2019 in the Journal of Orthopaedic and Sports Physical Therapy, argued that a large commercial industry has grown up around these beliefs, selling products and treatments that claim to correct posture and prevent pain, despite the absence of strong evidence that doing so works. Another part of the answer is vivid but misleading numbers. You may have seen the claim that bending your head forward to look at a phone puts the equivalent of 60 pounds, roughly 27 kilograms, of force on your neck. This figure comes from a short paper published in 2014 by the spinal surgeon Kenneth Hansraj in the journal Surgical Technology International, which used a model to estimate the forces on the cervical spine at different head angles. The model is a calculation, not a measurement of injury. The number spread widely in news stories and infographics, often presented as if it described damage being done. What it does not account for is that neck muscles, joints and discs are built to handle loads like this; you tilt your head forward to read, to cook, to tie your shoelaces, to look at your food, thousands of times a day. Tissues adapt to load. A force being large on paper does not make it harmful, any more than the forces involved in walking up stairs make stairs dangerous. The most important consequence of all this is not intellectual. It is emotional. When people believe their posture is damaging them, they tend to become watchful and tense. They hold themselves rigidly upright, brace their muscles, and feel a small jolt of alarm every time they catch themselves slouching. Pain researchers have shown repeatedly that beliefs and expectations shape how much pain people feel and how much it limits them. Peter O'Sullivan and colleagues, in a widely read editorial on facts everyone should know about back pain, listed the belief that pain is caused by poor posture among the common misconceptions, and emphasised that fear and protective guarding can themselves keep pain going. A student who spends a whole all-nighter holding a stiff military posture, afraid to relax, may end the night more sore than one who slumped, shifted, fidgeted and stood up now and then. Load, time and tolerance If posture is not the villain, what is? A more useful way to think about the body at a desk uses three ideas: load, time and tolerance. Load is the mechanical demand on a tissue. When your head is balanced directly over your shoulders, the muscles at the back of your neck have less work to do than when it juts forward. When your wrists are bent sharply back while typing, the tendons and the structures in the carpal tunnel experience a different load than when they are nearer straight. Posture does change load. That part of the traditional story is true. Time is how long that load is sustained without a change. A position that is perfectly comfortable for ten minutes can become uncomfortable after an hour and genuinely painful after four. The muscles doing the work fatigue, local blood flow in constantly contracted muscle is reduced, and the nervous system becomes increasingly alert to signals from the area. Almost any position, however textbook-perfect, becomes a problem if you hold it long enough. Try sitting in the ideal upright posture without moving for an hour and you will discover this for yourself. Tolerance is how much load, for how long, a particular person's tissues and nervous system can handle before they complain. Tolerance varies enormously between people and within the same person from week to week. It goes up with strength, fitness and gradual exposure. It goes down with sleep deprivation, stress, low mood, illness and sudden spikes in workload. A student who has trained regularly, slept seven or eight hours and feels on top of their coursework can tolerate a long session in an imperfect chair far better than the same student in exam week after three short nights. Put these three together and the practical picture changes. You can reduce load by setting up your workspace sensibly, which Chapter Six discusses. You can limit time by changing position often, which is the heart of Chapter Seven. And you can raise tolerance by building strength and protecting sleep and mood, which Chapter Eight covers. Posture is one lever on one of the three. It is a lever worth pulling when it is easy, but it is far from the whole machine. This also explains a pattern many programmers recognise. The aches rarely come from a single bad afternoon. They come from a crunch: a week of assignment deadlines followed by a hackathon, or the last month of a capstone project. During those periods, time in any one position goes up, sleep goes down, stress goes up, and exercise usually disappears. Load, time and tolerance all move in the wrong direction together. It is not surprising that the body objects. A better definition of good posture If there is no single correct position, what should you aim for? A working definition that fits the evidence is this: good posture is any position that is comfortable, that you can hold without effort for the task in front of you, and that you leave before it stops being comfortable. Several things follow from that definition. First, good posture is plural. You might start a session sitting fairly upright with both feet on the floor, move to leaning back with your keyboard on your lap while reading documentation, stand for a while at a counter, and sit cross-legged for a spell. All of these can be good postures. The mistake is not choosing any one of them; it is staying in it. Second, good posture is relaxed. If you have to hold a position with conscious effort, you are adding muscle tension to the load you are already carrying. A slightly slouched but relaxed position is often better tolerated over a session than a strained upright one. This does not mean collapsing into the worst possible shape; it means not bracing. Third, good posture is responsive. Discomfort is useful information. A mild ache in your neck after an hour is not evidence of damage. It is your body suggesting that it would like to do something else for a while. Responding promptly to these small signals is far more protective than ignoring them in favour of some predetermined ideal. Fourth, good posture includes the eyes and hands. Programmers often think of posture as a matter of the spine, but where you look and how your hands meet the keyboard and mouse shape the position of everything else. If the text on your screen is too small, your head drifts forward to read it. If the mouse is far to the side, your shoulder hitches up to reach it. Fixing the visual and manual demands of your setup often does more for your neck and shoulders than any instruction to sit straighter. Consider an illustrative case. A second-year student notices a dull ache at the base of her skull by late afternoon most days. She has read that forward head posture is bad, so she spends her lab sessions trying to keep her chin tucked and her back rigid against the chair. The ache gets worse, and now her upper back hurts too. When she describes her day to a physiotherapist at the campus clinic, a different picture emerges. She sits in the same chair for four- or five-hour stretches without getting up. She is sleeping about five hours a night during a heavy assignment period. She has stopped going to the gym. Her laptop screen is small, and she has the font size set low to fit more code on screen, so she leans in to read it. The advice she receives is not to hold her head in a better position. It is to increase the font size, raise the laptop on a stand with a separate keyboard, get up every thirty to forty-five minutes, go back to some form of regular exercise, and stop bracing. Within a few weeks the ache is much less frequent. Nothing about her ideal posture changed. What changed was how long she stayed still, how hard she worked to see, how rested and active she was, and how much tension she carried in trying to be perfect. That pattern, which is typical rather than exceptional, is the model for the rest of this book. What to keep and what to let go It would be a mistake to swing from one rigid rule to another and conclude that position is irrelevant. Some traditional advice remains sensible because it reduces load cheaply, not because it creates a magical protective alignment. It makes sense to have your screen roughly at eye level or slightly below so you are not craning up or hunching down for hours. It makes sense to have your keyboard at a height that lets your shoulders relax and your forearms sit roughly level. It makes sense to keep the mouse close rather than out at arm's length. It makes sense to have your feet supported. These are good defaults, and later chapters explain how to achieve them with student-budget equipment. What is worth letting go of is the moralising: the idea that there is a correct shape and that every deviation from it is a small failure. Let go, too, of the idea that pain in your neck or back is a sign of structural damage caused by slouching, and of the fear that makes people hold themselves stiffly. Let go of devices that buzz or pull you into a single position, which at best train you to hold one position for longer. Replace them with a few simple ideas. Your body likes variety. Your tissues adapt to what you ask of them gradually, and they complain when the demand rises faster than they can adapt. Your nervous system's sensitivity to pain is shaped by sleep, stress and mood as well as by mechanical load. And the most effective thing you can do at a keyboard is to move a little, often, and on purpose. The next chapter looks more closely at what a long coding session does to the body, so that when we turn to specific regions and stretches, you will understand not just what to do but why it helps. Chapter 2: What a Coding Marathon Does to the Body Programming looks like one of the least physical activities imaginable. The only visible movements are small ones: fingers on keys, a hand on a mouse, eyes moving across lines of text. It is tempting to conclude that nothing much is happening to the body at all. In fact a lot is happening, and understanding it explains why a day of apparently effortless work can leave you feeling as if you had spent it doing something much harder. This chapter looks at five things a long coding session does to you: it holds some muscles in constant low-level contraction, it removes the ordinary movement your tissues depend on, it demands intense and repetitive work from small structures in the hands and forearms, it strains the visual system in a way that drags the rest of the body along, and, in the specific setting of deadline crunches and hackathons, it combines all of this with sleep loss, stress and poor recovery. Each factor alone is manageable. The combination is what makes coding marathons hard on the body. The quiet work of holding still When you sit at a keyboard, several muscle groups are working the whole time, even if you feel entirely relaxed. The muscles at the back of the neck hold your head up and stop it falling forward. The upper part of the trapezius, the broad muscle that runs from the base of the skull across the top of the shoulders, helps support the weight of your arms, especially when your hands are raised to a keyboard or reaching for a mouse. The muscles of the forearm keep the wrist and fingers poised over the keys, ready to move. None of these muscles is working hard. But none of them gets to switch off either. This kind of sustained, low-intensity muscle activity has been studied for decades in occupational health, particularly in relation to office and assembly-line work. One influential explanation, often called the Cinderella hypothesis and proposed by the Swedish researcher Göran Hägg in the early 1990s, starts from how muscles recruit their fibres. When a muscle contracts gently, it uses a small set of motor units, the smallest and most easily activated ones, and these tend to be recruited first and released last. At low, steady levels of effort, the same small group can end up doing the work for long periods while their neighbours rest. Like Cinderella, they are first up and last to bed. The hypothesis proposes that these overworked fibres are where trouble can begin. It remains a model rather than a settled fact, but it captures something that matches experience: the problem with desk work is not heavy load but relentless, unvarying light load. A related effect involves blood flow. A muscle that is continually contracted, even gently, has less opportunity for the pumping action that comes from alternating contraction and relaxation. Circulation through it is less vigorous than in a muscle that is moving rhythmically. Many people notice the result as a dull, heavy ache that eases quickly once they get up and move around. That quick relief is a clue. It suggests that the discomfort is about the state of the tissue at that moment, not about structural damage, and that the fix is to change what the muscle is doing. This is why the same neck that happily carries your head through an afternoon of walking, talking and looking around can ache after two hours of staring at a monitor. Walking uses more muscle effort than sitting, but the effort constantly varies. Staring at a monitor uses less, but it never lets up. The loss of ordinary movement The body is built for movement, and many of its tissues depend on it. Joints are nourished partly through the movement of fluid in and out of their cartilage as they are loaded and unloaded. Discs between the vertebrae exchange fluid as you move and change position. Nerves slide and glide through the tissues around them as your limbs move, and they are sensitive to sustained compression or stretch. Muscles, as we have seen, depend on rhythmic contraction to keep blood flowing. In ordinary life you do not have to think about any of this, because ordinary life involves constant fidgeting and repositioning. People shift their weight, cross and uncross their legs, turn to look at things, reach, stand up, stretch without noticing. Watch anyone in a lecture hall and you will see dozens of small movements in a few minutes. Deep focus switches much of this off. One of the striking things about programmers in a state of concentration is how still they become. The mind is so absorbed in holding the structure of a problem that the body's small restless adjustments get suppressed. This is part of what makes flow feel good, and it is part of what makes it physically costly. An hour of focused debugging may involve far less movement than an hour of reading a novel, because the novel does not demand the same mental grip. There is also a broader health dimension. The World Health Organization's 2020 guidelines on physical activity and sedentary behaviour, summarised by Fiona Bull and colleagues in the British Journal of Sports Medicine, recommend that adults limit sedentary time and replace some of it with physical activity of any intensity. The guidelines state that evidence was insufficient to set a specific threshold for how much sitting is too much. For the purposes of this book, the more immediate point is local: long bouts without movement leave your neck, shoulders, wrists and back in a single state for too long, and they tell you so. Small structures, heavy traffic Typing and mousing are fine motor tasks performed at high repetition. A productive session involves many thousands of keystrokes, repeated small adjustments of a mouse or trackpad, and a surprising number of keyboard shortcuts that require the hand to stretch or contort: holding down modifier keys with the little finger while reaching with the index finger, or pressing combinations that involve three fingers spread across the keyboard. Programmers use more of these than most computer users. Text editors and development environments reward keyboard fluency, and many developers spend much of their day in terminals and editors driven almost entirely by key combinations. The muscles that move the fingers are mostly in the forearm. Their tendons run through the wrist into the hand, passing through narrow channels and under bands of connective tissue. On the palm side, the tendons that bend the fingers pass through the carpal tunnel along with the median nerve, which supplies sensation to the thumb, index, middle and part of the ring finger. On the back of the forearm, the muscles that lift the wrist and fingers attach near the outer part of the elbow. When the demand on these structures rises faster than they can adapt, the result can be pain in the forearm, discomfort around the wrist, or an ache at the outer elbow. What matters is not just the number of keystrokes but the conditions under which they happen. A wrist held bent back for hours changes the space in the carpal tunnel and the angle at which tendons work. A hand clutching a mouse tightly adds constant contraction to the forearm muscles. A thumb doing all the work on a laptop trackpad takes a disproportionate share of the load. Tension from stress travels into the hands too: people under deadline pressure often grip harder and type harder than they need to. Chapter Three looks at all of this in detail. The eyes lead, the body follows Programming is visually demanding. Code is dense, full of small symbols whose exact form matters. A misplaced semicolon or bracket can change the meaning of a line. Many developers use small fonts to fit more code on screen, work across several windows at once, and switch attention constantly between editor, terminal, browser and documentation. Digital eye strain, sometimes called computer vision syndrome, is common among heavy screen users. A 2018 review by Amy Sheppard and James Wolffsohn in BMJ Open Ophthalmology estimated that its prevalence may be 50 percent or more among computer users, and described two broad groups of symptoms: those related to the effort of focusing and coordinating the eyes at close range, and those related to dry eye, partly because people blink less often and less completely when concentrating on a screen. Typical complaints include tired or sore eyes, blurred vision, dryness, and headaches. For the purposes of this book, the key point is that the eyes and the neck work as a system. When text is hard to see, the head moves toward it. When a screen sits too low, the head tips forward to look down at it. When there is glare on the screen, people tilt and twist to avoid it. When the prescription in someone's glasses is out of date, they lean in. Much of what gets called poor posture at a computer is actually the body doing exactly what it should, which is to put the eyes where they can see. Trying to correct the posture without addressing the visual demand is fighting the wrong battle. That is why this book treats font size, screen height and lighting as posture tools. Sleep, stress and the hackathon effect Everything described so far applies to any long day at a computer. What makes a coding marathon different is that it rarely happens in isolation. It usually coincides with deadlines, competitions or crises, which bring their own physiological baggage. The first is sleep loss. Hackathons are commonly structured around a continuous period of twenty-four to forty-eight hours, and many participants sleep little or not at all. Assignment deadlines produce the same pattern in a less theatrical form: a string of late nights as submission day approaches. Sleep and pain are closely linked. A 2013 review in The Journal of Pain by Patrick Finan, Burel Goodin and Michael Smith examined the prospective and experimental evidence and concluded that sleep impairments reliably predict new episodes and flare-ups of chronic pain, and that sleep disturbance appears to be a stronger predictor of pain than pain is of sleep disturbance. Experimental studies suggest that lack of sleep interferes with the body's own mechanisms for dampening pain signals. In practical terms, the same neck position that barely registers after a good night's sleep can feel painful after a night without one. The second is stress. Deadlines, competition, uncertainty about whether the demo will work and the pressure of performing in a team all raise arousal. Stress tends to show up in the body as increased muscle tension, particularly around the shoulders, jaw and hands. It also makes people less likely to take breaks, because stopping feels like losing time. And, as the office-worker study discussed in Chapter One suggested, mood appears to play a role in whether neck pain develops and persists. The third is the collapse of ordinary recovery. During a crunch, exercise tends to disappear first, followed by regular meals, then daylight and fresh air. Caffeine and sugar replace sleep. None of these alone is catastrophic for a few days, but together they reduce the body's tolerance at exactly the moment when the demand on it is highest. Consider the arc of a typical weekend hackathon for an illustrative team of four students. Friday evening begins with excitement and plenty of moving around: introductions, brainstorming at a whiteboard, walking to get food. By midnight the team has split up the work and each member is heads-down. Between midnight and morning, movement falls to almost nothing. One student works on a laptop on a low table, perched on a beanbag. Another has a borrowed monitor but no separate keyboard. Coffee replaces water. Saturday brings a brief revival with breakfast and a sponsor workshop, then a long afternoon of integration problems and mounting tension. By Saturday night, with little sleep behind them, several team members have aching necks, one has a burning feeling along the outside of her forearm, and everyone is more irritable. Sunday morning's final push, followed by a demo, is performed on bodies that have spent thirty-six hours in the worst possible combination of long static positions, poor setups, sleep loss and stress. Nothing in that weekend is dramatic, and most of the aches will fade within days. But the pattern repeated across a degree, and then across a career, is worth interrupting. Doing so does not require giving up hackathons or all-nighters. It requires knowing where the strain comes from and building simple countermoves into the way you work. Mapping the strain Table 1 draws together the sources of strain described in this chapter. For each, it summarises what happens in the body, the early signs that tell you it is building up, and the simplest countermove. The countermoves are developed fully in later chapters. Table 1. Sources of strain during a long coding session. Source of strain What happens Early signs Simplest countermove Sustained low-level muscle work Same neck, shoulder and forearm muscles never switch off Dull, heavy ache that eases when you move Change position every 30 to 45 minutes Loss of ordinary movement Joints, nerves and muscles stay in one state for hours Stiffness when you finally stand up Short movement breaks tied to natural pauses High-repetition hand use Tendons and forearm muscles take heavy, unvaried load Forearm tightness, wrist or outer elbow ache Lighter touch, varied input, forearm stretches Visual demand Head drifts toward small text, low screens or glare Tired eyes, leaning in, headache Larger font, raise screen, look into the distance Sleep loss and stress Pain sensitivity rises and muscles tense Aches feel worse than the work explains Protect some sleep, deliberate relaxation Read the table as a diagnostic tool rather than a checklist. When you notice discomfort, ask which of these sources is most active right now. Are you simply too still? Is your hand working harder than it should? Are you straining to see? Are you exhausted? Usually the answer is more than one, and the most effective response addresses the one you can change most easily in that moment. The body as a system you can tune It may help to think about all of this in terms you already use. A long coding session is a sustained workload running on hardware with finite resources. Thermal throttling is not a sign that a processor is broken; it is a sign that it has been run too hard for too long without adequate cooling. The fix is not to replace the processor but to manage the load, improve the cooling and schedule the work more sensibly. Your body's aches during a marathon are similar. They are feedback from a system operating near the edge of its current tolerance. The analogy has a limit, and the limit is encouraging. Unlike a processor, your body upgrades itself in response to the demands you place on it, provided you increase them gradually and allow recovery. Muscles that are trained get stronger and more enduring. Tissues that are loaded progressively become more tolerant. Nervous systems that are well rested and not under chronic threat become less sensitive. This is why the later chapters of this book include not only stretches, which offer short-term relief and a change of position, but also strengthening exercises and broader lifestyle measures, which build the capacity that makes long sessions easier over time. The next three chapters take the body region by region, starting where the work is most concentrated: the hands, wrists and forearms. Chapter 3: Wrists, Hands and Forearms For most programmers the hands are the body. They are where the work leaves you and enters the machine. They are also where many students feel the first unmistakable signs of strain: a tightness along the forearm after a long session, a sore spot at the outer elbow, an ache in the thumb from a trackpad, or tingling fingertips that raise a frightening question about carpal tunnel syndrome and whether a career in computing is compatible with having working hands. This chapter starts by putting that fear in proportion, then explains the more common and more manageable forms of hand and forearm strain, then describes practical changes to how you use your keyboard and mouse, and finally gives a set of stretches and movements you can do at your desk. Carpal tunnel syndrome in perspective Carpal tunnel syndrome is a condition in which the median nerve is compressed as it passes through the carpal tunnel, a narrow channel on the palm side of the wrist formed by the small wrist bones and a tough band of tissue across them. Classic symptoms include numbness, tingling or pain in the thumb, index, middle and part of the ring finger, often worse at night or on waking, sometimes relieved by shaking the hand. In more advanced cases, people may notice weakness or clumsiness in gripping, or wasting of the muscles at the base of the thumb. It is widely assumed that heavy computer use causes carpal tunnel syndrome. The evidence is more reassuring than that assumption. A large Danish study led by Johan Hviid Andersen, published in JAMA in 2003, followed several thousand members of a trade union whose work involved computers for one year. The prevalence of symptoms consistent with carpal tunnel syndrome, confirmed by clinical interview, was low, and the occurrence of night-time symptoms lower still. The study found an association between heavy mouse use, more than twenty hours per week, and the risk of possible carpal tunnel syndrome, but no statistically significant association with keyboard use. The authors concluded that computer use does not pose a severe occupational hazard for developing carpal tunnel symptoms. A 2014 meta-analysis by Zakia Mediouni and colleagues, pooling the available epidemiological studies, was unable to show a clear association between computer use and carpal tunnel syndrome, while noting that certain work circumstances might be associated with it. This does not mean programmers never develop carpal tunnel syndrome. It occurs in the general population for many reasons, including pregnancy, diabetes, thyroid conditions, wrist fractures and family history, and computer users are part of that population. It means that the typical aches of heavy keyboard work are usually something else, and that tingling in your fingers after a long session is not automatically evidence that your keyboard is damaging a nerve. If you have persistent numbness or tingling, particularly at night, or any weakness in your hand, see a clinician. There are straightforward ways to assess it and effective treatments. But do not spend your degree dreading it. What usually goes wrong instead The more common problems are less dramatic and more directly connected to how you use your hands. The first is simple muscle fatigue and overload in the forearm. The muscles that bend and straighten your fingers and wrist sit in the forearm, and during long typing sessions they work continuously. When the demand outstrips their conditioning, they feel tight, heavy or achy, often along the top of the forearm near the elbow. This is usually the body's equivalent of the dull ache in your legs after an unusually long walk. It fades with rest, movement and a gradual increase in capacity. The second is irritation where the tendons of these muscles attach to bone. The most familiar example is pain on the outside of the elbow, where the muscles that lift the wrist and fingers attach. This is often called tennis elbow, and its clinical name is lateral elbow tendinopathy or lateral epicondylalgia. Despite the name, it is common in people who have never played tennis, including heavy mouse users. It tends to hurt when gripping or lifting with the palm facing down, and can be stubborn once established. It responds best to reducing aggravating loads while gradually strengthening the forearm, rather than to complete rest. The third involves the thumb. Using the thumb to operate a laptop trackpad, a phone, or the space bar with force can irritate the tendons that run along the thumb side of the wrist. This sometimes develops into a condition called De Quervain's tenosynovitis, with pain at the base of the thumb and along the wrist on that side, especially when gripping or moving the thumb. If the thumb side of your wrist hurts when you make a fist with your thumb tucked inside and bend your wrist toward your little finger, it is worth having it checked. The fourth is the diffuse, hard-to-locate discomfort that many heavy computer users describe as aching, burning or tired hands and forearms. This may involve several of the above, along with the effects of sustained muscle tension, stress and poor sleep described in Chapter Two. It usually responds well to the combination of changes in this chapter. Table 2 sets out common symptoms, what they might suggest, and a sensible first response. It is a guide for deciding when to manage something yourself and when to seek assessment, not a tool for diagnosis. Table 2. Hand and forearm symptoms and sensible first responses. Symptom What it might suggest First response Tight, heavy forearm that eases with rest Muscle fatigue from sustained typing Breaks, lighter touch, forearm stretches Pain at outer elbow when gripping palm-down Irritated tendon attachment (tennis elbow) Reduce mouse grip, gradual strengthening; see a clinician if persistent Pain at thumb side of wrist with thumb use Irritated thumb tendons Rest the thumb, vary input devices; get it assessed if it persists Tingling in thumb and first fingers, especially at night Possible median nerve irritation See a clinician, particularly if persistent Weakness, dropping things, muscle wasting at thumb base Nerve involvement needing assessment See a clinician promptly How you touch the machine Before any stretch, the most effective thing you can do for your hands is to change how you use them. Several changes are simple, cheap and make a disproportionate difference. Lighten your touch. Many people press keys far harder than they need to, bottoming them out with force on every stroke. Next time you type, notice how hard you strike. Then try deliberately typing more softly, letting the key register without slamming into the end of its travel. Keyboards differ a great deal in how much force they require; if you type all day, it is reasonable to try a few and choose one that feels light and responsive to you. Loosen your grip. The same applies to the mouse. Under pressure, people clutch it, and that sustained grip loads the forearm muscles continuously. A mouse should rest under your hand, guided by it rather than squeezed. Move it with your whole arm from the shoulder and elbow for large movements, rather than flicking it only with the wrist. If you notice your mouse hand has been hovering tensely, let it drop to the desk or your lap between uses. Keep your wrists roughly neutral. You do not need a perfect angle, but it helps to avoid the extreme of holding your wrists sharply bent back for hours, which happens when a keyboard is too high or tilted steeply up at the back, or when you rest the heels of your hands on the desk and lift your fingers up to reach the keys. Many keyboards have little feet at the back that tilt them upward; for many people, folding these away makes the wrist angle more comfortable. Avoid bending the wrists sharply sideways too, which happens when a keyboard is very narrow for your shoulders or when you reach far for keys. Vary your input. Programmers have a real advantage here, because so many tasks can be done in several ways. You can alternate between keyboard shortcuts and mouse for navigation. You can switch the mouse to your other hand for part of the day; it feels clumsy at first but most people adapt within days for simple tasks like scrolling and clicking. You can alternate between a mouse and a trackpad, or between a conventional mouse and a vertical one that holds your hand in a handshake position. Evidence that any particular device prevents problems is limited, but alternating between devices spreads load across different muscles and positions, which fits the central principle of this book. Reduce awkward chords. If a particular key combination makes your hand contort, remap it. Many developers remap the rarely used caps lock key to control or escape, putting a frequently used modifier under the strong index or middle finger rather than the little finger. Editors and operating systems allow extensive customisation of shortcuts. Spending an hour setting up bindings that suit your hands is time well spent. Reduce how much you type. This sounds absurd for a programmer, but a surprising amount of typing can be avoided. Snippets, autocomplete, templates and well-chosen tools can remove a lot of repetitive keystrokes. Voice input and dictation tools are increasingly capable and can take over tasks such as writing documentation, commit messages and emails during periods when your hands need a break. Consider an illustrative case. A third-year student working on a large project develops a sore spot on the outside of his right elbow that throbs after long sessions. He has been using a small, lightweight wireless mouse that he grips tightly, and he spends most of each day navigating a large codebase with it. He changes three things: he learns the keyboard shortcuts for navigation in his editor and uses them for much of what he used the mouse for, he switches to a larger mouse that his hand can rest over rather than clutch, and he starts a simple forearm strengthening routine of the kind described below. The elbow does not settle overnight, but over several weeks it becomes steadily less irritable. Nothing about the shape of his sitting position was involved. Stretches and movements for the hands and forearms The following movements are designed to be done at your desk, during short pauses in work. They serve two purposes: to give the muscles and tendons a change from the sustained positions of typing, and to maintain comfortable range of movement. None of them should hurt. You should feel a mild, comfortable stretch or simply movement. If any of them produces sharp pain, or tingling or numbness that persists after you stop, skip that movement and seek advice. Wrist flexor stretch. This stretches the muscles on the palm side of the forearm, which bend the wrist and fingers. Hold one arm straight out in front of you at shoulder height, palm facing up. With your other hand, gently hold the fingers of the outstretched hand and draw them down and back toward the floor, so the wrist bends backward and the palm faces away from you. Keep the elbow straight but not locked. You should feel a gentle stretch along the underside of the forearm. Hold for about twenty to thirty seconds, breathing normally, then release and repeat with the other arm. If the stretch is too strong, bend the elbow slightly. Wrist extensor stretch. This stretches the muscles on the back of the forearm, which lift the wrist and fingers and attach near the outer elbow. Hold one arm straight out in front of you with the palm facing down. With your other hand, gently press on the back of the outstretched hand so the wrist bends down and the fingers point toward the floor. For a slightly stronger stretch, gently curl the fingers into a loose fist first. You should feel the stretch along the top of the forearm, possibly toward the outer elbow. Hold for twenty to thirty seconds, then switch sides. Prayer stretch and reverse. Place your palms together in front of your chest, fingers pointing up, as if praying. Keeping the palms pressed together, slowly lower your hands toward your waist until you feel a stretch on the palm side of your wrists and forearms. Hold for fifteen to twenty seconds. Then turn the hands so the backs touch and fingers point down, and gently raise the hands toward your chest to stretch the other side. Hold again. This is a convenient way to stretch both forearms at once without holding your arm out in a lab full of people. Tendon glides. This sequence moves the finger tendons through their full range and is commonly used by hand therapists. Start with your hand held up in front of you, fingers straight and together, pointing to the ceiling. Then, one position at a time, holding each for about three seconds: bend only the knuckles at the ends of the fingers and the middle knuckles so your fingers form a hook, with the large knuckles still straight; make a full fist with the thumb resting outside the fingers; straighten the fingers again; bend only the large knuckles so the fingers stay straight and form a flat tabletop shape pointing forward; then bend the large and middle knuckles while keeping the fingertips straight, so the fingertips touch the base of your palm. Return to straight fingers between each position. Go through the sequence five times on each hand. It should feel easy and smooth. Finger spreads and thumb circles. Spread your fingers as wide as comfortably possible, hold for three seconds, then close them into a loose fist. Repeat ten times. Then, with your hand relaxed, draw slow circles with the tip of your thumb, five in each direction. These counter the narrow, cramped positions of keyboard work. Forearm rotation. Hold your elbows bent at your sides, forearms pointing forward, as if holding a tray. Slowly turn your palms up to face the ceiling, then down to face the floor, going as far as is comfortable in each direction. Repeat ten times. This moves the forearm through a rotation that typing and mousing, which keep the palms turned down, rarely use. Gentle median nerve glide. Nerves need to slide through surrounding tissues as your limbs move. This movement encourages that gliding gently, and can be a comfortable way to relieve the feeling of stiffness in the arm after typing. Start with your arm relaxed at your side and your head upright. Raise the arm out to the side to about shoulder height, elbow bent, palm facing forward. Slowly straighten the elbow while gently bending the wrist back so the palm faces outward, as if pushing lightly against a wall beside you. As you straighten the arm, tilt your head gently toward that arm. Then return: bend the elbow, relax the wrist, and tilt your head away. Move slowly and smoothly through this back and forth about five to ten times. The aim is gentle movement, not a stretch. You may feel mild tension along the arm, but you should not provoke tingling. If you do, reduce the range or skip this movement. People with diagnosed nerve problems should use nerve glides only on the advice of a clinician. Shake out and let go. At the end of any hand routine, let your arms hang at your sides and gently shake your hands for a few seconds, then let them rest completely, feeling the weight in your fingers. This is less a stretch than a reminder to release the tension your hands have been holding. Building forearm capacity Stretching offers relief and a change, but the most durable protection for your hands comes from making the forearm stronger. This matters especially if you have developed pain at the outer elbow, where gradual strengthening is a mainstay of management. You do not need a gym for this. Wrist extension with a light weight. Sit with your forearm resting on your thigh or a table, palm facing down, your hand hanging over the edge. Hold a light weight, such as a full water bottle or a can of food. Slowly lift the back of your hand toward the ceiling, then lower it slowly over about three seconds. Do ten to fifteen repetitions, rest, and do a second set. Start lighter than you think you need. Some mild discomfort during the exercise is acceptable if it settles quickly afterwards and is no worse the next day. If it is, reduce the weight. Wrist flexion with a light weight. Same position, but palm facing up. Curl the hand up toward the ceiling, then lower it slowly. Same repetitions. Grip squeezes. Squeeze a soft ball or rolled-up sock firmly for three to five seconds, then release. Repeat ten times. Grip strength supports the whole forearm. Rotations with a lever. Hold one end of a hammer, a light dumbbell with weight on one end, or even a full bottle, with your forearm resting on a table and your wrist free. Slowly rotate the forearm so the palm turns up, then down. The further your hand is from the weighted end, the harder this becomes. Two or three sessions a week of these exercises, done steadily over a couple of months, build forearms that tolerate far more typing than untrained ones. Combine them with the changes in how you use your hands, and the aches that many programmers assume are simply part of the job usually become much less common. Putting it into practice You do not need to do everything in this chapter every day. A reasonable approach is to adopt the changes in how you touch the machine permanently, pick two or three of the stretches you like best to do several times a day during natural pauses, and add the strengthening exercises a few times a week. Chapter Seven shows how to fit these into the rhythm of a long session. The hands are the end of a chain that starts at the neck and passes through the shoulders and elbows. Tension higher up affects how the hands feel, and the nerves that supply the hand begin in the neck. The next chapter moves up that chain to the neck, the region most often blamed on posture, and the one where the evidence most sharply contradicts the conventional warnings. Chapter 4: The Neck and Head No part of the body carries more posture guilt than the neck. The phrase "forward head posture" has become a kind of shorthand for everything supposedly wrong with the way people use computers and phones. Students are warned that their necks are being slowly deformed, that the curve of their cervical spine is being flattened or reversed, that the weight on their necks has multiplied, and that a lifetime of problems awaits. Meanwhile, neck pain genuinely is common among heavy computer users, and it can be miserable, bringing with it headaches, difficulty concentrating, and a sense of carrying a heavy weight on your shoulders by the end of the day. This chapter tries to hold both of these facts together. Neck pain at the computer is real and worth addressing. But the explanation that centres on a single bad posture is weak, and the most effective responses are different from the ones that explanation suggests. What the evidence says about the forward head Chapter One introduced several studies. It is worth drawing out what they mean for the neck in particular. The study of more than a thousand 17-year-olds in Australia found four distinct ways of sitting, including an upright posture and two different forward-head postures, and found no significant difference in neck pain or headaches between them. The study of young adults in Brazil found no association between the head-down position used while texting and neck pain. And the study of new office workers in Colorado found that those who went on to develop chronic interfering neck pain over their first year were distinguished by lower endurance in the neck extensor muscles, lower mood, and a less effective capacity of the nervous system to inhibit pain, rather than by head posture. The last finding deserves attention because it points somewhere practical. The muscles at the back of the neck are what hold your head up and resist its tendency to fall forward when you look down or lean toward a screen. When those muscles have plenty of endurance, they can do this work for long periods without fatiguing. When they have less, they tire sooner, and fatigue in a muscle doing sustained work is felt as ache. This suggests a reframing: the problem may be less that your head is forward and more that the muscles holding it have not been prepared for the amount of work you are asking of them, combined with how your nervous system is responding to their fatigue. That reframing does not mean the position of your head is irrelevant. When your head is well forward of your shoulders, the muscles at the back of the neck do work harder to hold it. Over a long session, that extra work adds up. The point is that there are two ways to deal with it: reduce how much you ask of the muscles, by adjusting your setup so you do not need to crane forward for hours, and increase what they can deliver, by building their endurance. The traditional approach focuses entirely on the first, and usually tries to achieve it by willpower. A more effective approach does both, using the environment rather than willpower for the first. Why your head moves forward, and how to stop fighting it Your head does not drift forward toward a screen because you are lazy. It moves forward for reasons, and each reason suggests a fix that is more reliable than trying to hold your head back. It moves forward to see. If the text is small, if the screen is far away, if there is glare, or if your glasses are not right for the distance, your head moves closer. The fix is to make seeing easy. Increase the font size in your editor and terminal until you can read comfortably without leaning. Many programmers resist this because they want to see more lines of code at once. A larger or second screen can give you both. Reduce glare by angling the screen away from windows and overhead lights. And if you find yourself squinting or leaning in despite a reasonable setup, have your eyes tested. Your needs for the intermediate distance of a computer screen may differ from your needs for reading or distance vision, and an optometrist can advise. It moves forward and down because the screen is low. A laptop on a desk puts the screen well below eye level, and your head and neck bend down to look at it. Working like this for a short period is fine. Working like this for six hours is a heavy demand on the neck. The most effective single change for many students is to raise the laptop screen, using a stand or a stack of books, and use a separate keyboard and mouse. A widely used starting point is to place the top of the screen at or slightly below eye level, so that your gaze falls slightly downward toward the middle of the screen, at a distance of roughly an arm's length. Treat this as a starting point to adjust for comfort, not a rule. It moves forward because you are leaning in with interest. Engagement tends to pull people toward what they are looking at, especially when concentrating hard. This is harmless in itself; the problem is only if it lasts for hours without change. The fix is not to suppress it but to break it up with movement. It moves forward because the chair or desk invites it. If your chair has no back support that you can comfortably use, or your desk is so deep that the screen is far away, or you sit forward on the edge of your seat to reach the keyboard, your head follows. Adjusting the setup so that you can sit back and still reach and see comfortably reduces this. Chapter Six covers setups in detail. Headaches and the neck Many heavy computer users experience headaches by the end of a long day. Some of these are tension-type headaches, often described as a band-like pressure around the head or a dull ache in the forehead or back of the head. Others arise from the neck itself, felt at the base of the skull and sometimes spreading forward. Some are linked to eye strain, some to dehydration, caffeine withdrawal or lack of sleep, and some are migraines, which have their own triggers and characteristics. During a coding marathon, several of these causes tend to converge. You may be drinking coffee irregularly, forgetting to drink water, skipping meals, sleeping little, straining your eyes and holding your neck still. It is unsurprising that headaches are a familiar part of hackathon culture. The practical response involves several strands: drink water regularly, eat real food, keep caffeine intake consistent rather than in large irregular bursts, give your eyes regular breaks, and move your neck. Any headache that is sudden and severe, described as the worst headache of your life, or accompanied by fever, stiff neck, confusion, weakness, difficulty speaking or changes in vision, needs urgent medical attention. In the United States that means calling 911; other countries have their own emergency numbers. Headaches that are new and persistent, steadily getting worse, or different from your usual pattern also need to be assessed by a clinician. Giving your eyes a break Because the eyes lead the neck, looking after them is part of looking after your neck. A commonly recommended habit is the 20-20-20 rule: every 20 minutes, look at something about 20 feet away, roughly six metres, for about 20 seconds. The precise numbers are a memorable convention rather than a tested prescription, but the principle is sound. Looking into the distance relaxes the focusing effort your eyes make at close range. It also naturally involves lifting your head, changing your gaze and, if you turn to look out of a window or across a room, moving your neck. Blinking matters too. As the review by Sheppard and Wolffsohn noted, people blink less often and less completely when concentrating on a screen, which contributes to dry, irritated eyes. During your distance breaks, blink slowly and deliberately a few times. If your eyes are persistently dry or sore, a pharmacist or optometrist can advise on lubricating drops and other measures. A few adjustments to the screen help as well. Match its brightness to the surrounding room, so it is neither a glowing rectangle in a dark space nor washed out by a bright one. Many developers prefer dark editor themes; there is no strong evidence that dark or light themes are better for the eyes in general, so choose what you find comfortable, and consider switching between them as ambient light changes. Naps, beanbags and waking up stiff One neck problem is peculiar to marathons: the stiffness that comes from sleeping in the wrong place. At hackathons and during deadline nights, people doze with their head on a desk, slumped sideways in a chair, or curled on a beanbag, and wake with a neck that will barely turn. This is usually a sustained, end-of-range position held for an hour or more without the small movements that normally happen during sleep, rather than any injury. It tends to ease over a day or two with gentle movement, warmth and ordinary activity. If you plan to nap during an event, bring something to lie down on, such as a camping mat or a folded jacket for a pillow, so that your neck is supported roughly in line with your body. A short sleep lying flat is better for your neck, and for your pain sensitivity, than a longer one folded over a keyboard. Stretches and movements for the neck The following movements give the neck a change from sustained positions and maintain comfortable range. Move slowly and gently. The neck is not a region for forcing or bouncing. You should feel movement or a mild stretch, never sharp pain, dizziness, or pins and needles into your arms. If any of those occur, stop and seek advice. Chin nods. This is a small, gentle movement that engages the deep muscles at the front of the neck and counters the tendency for the head to poke forward. Sit or stand tall but relaxed, looking straight ahead. Without tipping your head down, gently draw your chin straight back, as if making a slight double chin, so that your head glides backward over your shoulders. You should feel a gentle stretch at the base of the skull and a mild engagement at the front of the neck. Hold for about three to five seconds, then relax. Repeat five to ten times. The movement is small; you are not trying to push your head back as far as possible. Think of it as resetting where your head sits rather than holding it there permanently. Neck rotations. Sitting tall and relaxed, slowly turn your head to look over one shoulder, as far as is comfortable. Pause for a breath or two, then turn slowly to the other side. Repeat five times each way. This movement is often the first to feel stiff after long periods of looking straight ahead at a screen. Side bends. Sitting tall, slowly tilt your head so your ear moves toward your shoulder, without lifting the shoulder to meet it. You should feel a stretch along the opposite side of your neck. Hold for about fifteen to twenty seconds, breathing slowly, then return to the middle and repeat on the other side. For a slightly stronger stretch, reach the hand on the stretching side down toward the floor, or gently hold the seat of your chair with that hand. Upper trapezius stretch. This targets the muscle across the top of the shoulders that often feels tight and knotted after computer work. Sitting tall, hold the seat of your chair with your right hand to keep that shoulder down. Tilt your head to the left, bringing your left ear toward your left shoulder, then turn your face slightly toward the ceiling. You can rest your left hand lightly on the top of your head to add a little weight, but do not pull. You should feel a stretch along the right side of your neck and the top of your right shoulder. Hold for twenty to thirty seconds, then switch sides. Levator scapulae stretch. The levator scapulae is a muscle running from the side of the neck down to the top inner corner of the shoulder blade, and it is a common source of the ache people feel where the neck meets the shoulder. Sitting tall, hold the seat of your chair with your right hand. Turn your head about 45 degrees to the left, as if looking toward your left armpit, then gently nod your chin down toward your chest. You can rest your left hand on the back of your head for a little extra weight. You should feel the stretch at the back and right side of the neck toward the shoulder blade. Hold for twenty to thirty seconds, then switch sides. Look up and around. Stand up, and slowly look up at the ceiling, letting your chest lift a little with your gaze, then look down at the floor, then slowly around the room from one side to the other. This combines neck movement with a change of visual distance and is an easy way to break up a long spell of screen-staring. Building neck endurance The Colorado study suggests that neck muscle endurance matters. Strengthening and endurance training for the neck and shoulders is also the form of exercise with the best evidence for reducing neck pain in office workers, as the next chapter describes. A simple exercise can start building endurance at home. Deep neck flexor hold. Lie on your back on the floor or a firm bed, with your knees bent and your head resting on the surface, without a pillow or with a thin folded towel. Perform the chin nod described above, gently drawing your chin back and down as if nodding yes, so that the back of your neck lengthens along the floor. Keep the movement small and do not lift your head. Hold this gentle nod for ten seconds while breathing normally, then relax. Repeat up to ten times. As it becomes easier, you can progress by lifting your head a centimetre or two off the surface while maintaining the nod, holding for a few seconds at first and building up gradually. Stop if you feel strain in the front of the neck or if your chin pokes up. Prone head lift. Lie on your front with your forehead resting on a folded towel, arms by your sides. Gently draw your chin in, then lift your head just slightly off the towel, keeping your gaze toward the floor, so that the back of your neck works to hold the weight of your head. Hold for five seconds, then lower. Repeat five to ten times. Progress by increasing the hold time gradually. This works the extensor muscles at the back of the neck that hold your head up during screen work. Done a few times a week, these exercises gradually increase the neck's capacity for sustained work. They are not a replacement for movement breaks during long sessions, but they make those sessions easier to tolerate. Consider an illustrative first-year student who is anxious about her neck because she has read about text neck and has noticed that her head sits forward in photographs. She has intermittent neck aches and headaches, usually worst during assignment weeks. Instead of trying to correct her head position all day, she enlarges the text in her editor, raises her laptop on a stand, sets a gentle reminder to look out of the window every twenty minutes or so, and adds the endurance exercises above to a short routine three evenings a week. She also notices that her headaches are much worse on days when she forgets to drink water and drinks several large coffees in quick succession, and adjusts accordingly. Her head is still, in photographs, a little forward of her shoulders. Her neck bothers her far less. The neck sits above the shoulders and upper back, and tension and fatigue there often feed into each other. The next chapter looks at that region, and at the evidence for the most effective form of exercise for desk workers' neck and shoulder pain. Chapter 5: Shoulders, Upper Back and the Rest of the Chain If the neck is where programmers feel guilty, the shoulders are where they carry their stress. Ask a group of students at the end of a heavy week where they feel tension and many will point to the ridge running from the neck to the top of each shoulder. Some will describe a burning spot between the shoulder blades. Others will talk about a sense that their shoulders have crept up toward their ears and stayed there. This chapter looks at why the shoulders and upper back take so much strain at a computer, describes the exercise approach with the strongest evidence for relieving it, and gives a set of stretches and strengthening movements. It ends by looking briefly at the lower back and hips, which are not the focus of this book but are part of the same chain of structures that spend long hours sitting. Why the shoulders take the load The shoulder is an unusual joint. Most of its stability comes not from bony structure but from muscles. The arm hangs from the shoulder blade, and the shoulder blade itself floats on the back of the rib cage, held in place and moved by a set of muscles that attach it to the spine, the ribs, the skull and the arm. When you raise your hands to a keyboard, reach forward for a mouse, or hold your arms slightly out from your body, these muscles are working to support the weight of your arms and keep the shoulder blades stable. The upper trapezius is one of the most involved. It helps lift and support the shoulder blade, and it works whenever your arms are held forward or up. When a desk or keyboard is too high, you have to lift your shoulders slightly to reach it, and the upper trapezius works continuously. When your mouse is far out to the side, the shoulder on that side hitches up and forward. When you are stressed, the same muscles tend to tighten as part of a general protective response. It is no coincidence that this is the area so many desk workers feel as tight and knotted. Research in occupational health has paid particular attention to pain in this muscle in people doing monotonous, computer-intensive work. The condition is sometimes called trapezius myalgia, which simply means muscle pain in the trapezius. The Cinderella hypothesis described in Chapter Two was developed partly with this kind of pain in mind. The upper back, meanwhile, spends long periods in a relatively fixed, gently rounded position while you work. This is not dangerous. The thoracic spine is designed to round. But the lack of variety means it rarely moves through its full range, and the muscles between the shoulder blades hold a sustained, low-level effort. Many people experience the result as stiffness or a nagging ache that is most noticeable when they finally stand up and stretch. Strength training beats stretching alone A particularly clear piece of evidence comes from Denmark, where Lars Andersen and colleagues at the National Research Centre for the Working Environment ran a randomised trial published in 2008 in the journal Arthritis and Rheumatism. They recruited 48 women from workplaces with monotonous, computer-intensive jobs who had chronic neck muscle pain diagnosed as trapezius myalgia. The women were randomly assigned to ten weeks of specific strength training targeting the neck and shoulder muscles, to general fitness training on an exercise bicycle with relaxed shoulders, or to a reference intervention without physical activity. The group doing specific strength training had a large and lasting reduction in their worst pain. The cycling group had only a small, short-lived reduction. This trial was small and involved a particular group, so it should not be over-read. But it fits with a broader pattern. A 2017 systematic review and meta-analysis by Shereen Louw and colleagues at Stellenbosch University examined trials of exercise for office workers with non-specific neck pain. Most of the included trials used strengthening exercise, and the meta-analysis found a clinically significant reduction in pain with strengthening compared with no exercise. The evidence for endurance and stretching exercises was more limited, simply because fewer trials had tested them. The practical implication is important. Stretching feels good during a long session and provides a welcome change of position, and this book recommends it for that reason. But if you want to make your shoulders and neck substantially more resilient to long hours at a computer, the evidence favours making the muscles stronger. Strengthening does not require a gym membership or heavy weights. A resistance band, a pair of water bottles, a backpack loaded with books, or your own body weight can provide enough load to start. Stretches and movements for the shoulders and upper back The following movements can be done at or near your desk. As before, they should feel comfortable: a mild stretch or easy movement, never sharp pain. If you have a known shoulder problem or any movement provokes pain that lingers, skip it and seek advice. Shoulder rolls. Sit or stand tall and relaxed. Slowly lift your shoulders up toward your ears, then roll them back, drawing the shoulder blades gently together, then down, then forward, tracing a large, smooth circle. Do five slow circles backward, then five forward. Finish by letting your shoulders drop completely and feeling them settle. Many people are surprised by how far their shoulders drop at the end, which shows how much tension they were holding. Shrug and release. Lift both shoulders toward your ears as high as you can, hold for five seconds, squeezing firmly, then let them drop suddenly and completely. Repeat three to five times. Tensing a muscle deliberately and then releasing it can make it easier to feel and let go of tension you were not aware of. Doorway chest stretch. Stand in a doorway with your forearm resting vertically against the frame, elbow at about shoulder height and bent to a right angle, as if you were raising your hand to answer a question. Step forward gently with the leg on the same side until you feel a stretch across the front of your chest and shoulder. Keep your shoulder down and relaxed rather than hunched. Hold for twenty to thirty seconds, then switch sides. You can vary the stretch by moving your elbow slightly higher or lower on the frame. If there is no doorway nearby, you can clasp your hands behind your back, straighten your arms gently and lift them slightly away from your body while opening your chest. Cross-body shoulder stretch. Bring one arm straight across the front of your body at about chest height. With your other hand, hold the arm just above the elbow and gently draw it closer to your chest. You should feel a stretch at the back of the shoulder. Hold for twenty to thirty seconds, then switch sides. Seated thoracic extension. This movement uses the backrest of your chair to encourage the upper back to extend, a movement it rarely makes during computer work. Sit toward the back of a chair whose backrest top reaches roughly the middle of your back, around the bottom of your shoulder blades. Clasp your hands gently behind your head to support it, elbows pointing forward. Slowly lean back over the top of the backrest, letting your upper back arch over it and your elbows open out toward the ceiling. Go only as far as is comfortable, hold for a breath or two, then return to upright. Repeat five times. If your chair has a high backrest or is unstable, do this seated on a stool while simply lifting your chest and gazing slightly upward, or skip it. Never do it on a chair that might tip. Seated rotation. Sit tall with your feet flat on the floor. Cross your arms over your chest, or hold the back of the chair with one hand. Slowly turn your upper body to one side, leading with your chest and letting your head follow, keeping your hips facing forward. Go as far as is comfortable, hold for a breath, then return and turn to the other side. Repeat five times each way. Open book. This is a floor exercise for a longer break or at home. Lie on your side with your knees bent up toward your chest, and your arms straight out in front of you, palms together, at shoulder height. Keeping your knees together and resting on the floor, slowly lift your top arm up toward the ceiling and over to the other side, following it with your eyes and letting your upper body rotate, as if opening a book. Go as far as is comfortable, pause for a breath or two, then return. Repeat five to ten times, then switch sides. Wall slides. Stand with your back against a wall, feet a little way out from it, and your head, upper back and bottom touching the wall if that is comfortable. Raise your arms so your upper arms are out to the sides at shoulder height and your elbows are bent at right angles, forearms pointing up, with the backs of your hands and forearms against the wall as far as is comfortable. Slowly slide your arms up the wall, straightening the elbows, then slide them back down. Keep the movement smooth and within comfort; many people cannot keep their hands fully against the wall, and that is fine. Repeat eight to ten times. Strengthening the shoulders and upper back The following exercises build strength in the muscles that support the shoulder blades and neck. Do them two or three times a week, not necessarily at your desk. Start with a load that lets you complete the repetitions with good control and some effort by the end, and increase it gradually over weeks. Mild muscle soreness a day or two after starting is normal. Pain in the joint, or pain that is worse the next day and does not settle, suggests reducing the load. Shoulder shrugs with weight. Stand holding a weight in each hand, such as dumbbells, full water bottles or shopping bags, arms hanging by your sides. Lift your shoulders straight up toward your ears, hold briefly, then lower slowly. Do ten to fifteen repetitions for two or three sets. Dumbbell exercises of this kind formed the basis of the strength programme in the Danish trial described above, and shrugs directly train the upper trapezius to be stronger and more enduring. Rows. Tie a resistance band around a sturdy post or close it in a door at about chest height, and hold one end in each hand, arms straight out in front of you. Pull your hands back toward your ribs, bending your elbows and drawing your shoulder blades together, then return slowly. Without a band, you can do a bent-over row: stand with one hand and knee resting on a bench or bed, back roughly flat, and pull a weight in the other hand up toward your hip, then lower it. Do ten to fifteen repetitions for two or three sets on each side. Band pull-aparts. Hold a resistance band in front of you at shoulder height with both hands, arms straight, hands about shoulder width apart. Pull the band apart by moving your hands out to the sides, squeezing your shoulder blades together, until your arms form a T. Return slowly. Do twelve to fifteen repetitions for two or three sets. Lateral raises. Stand holding a light weight in each hand at your sides. With a slight bend in the elbows, raise your arms out to the sides until they are roughly level with your shoulders, then lower slowly. Keep the weights light; this exercise does not need heavy loads to be effective. Do ten to twelve repetitions for two or three sets. Push-ups against a wall, desk or floor. Push-ups train the muscles at the front of the chest and shoulders and those that hold the shoulder blades against the ribs. Start with your hands on a wall and your feet a step or two back, then progress to a sturdy desk, then to the floor on your knees, then to full push-ups, as your strength improves. Lower yourself slowly and push back up. Do as many as you can with control, up to about fifteen, for two or three sets. This is not an elaborate programme. Twenty minutes, two or three times a week, is enough to start. The benefit comes from consistency over months, and it compounds: stronger shoulder muscles make every long session easier to tolerate, which makes it easier to keep working without pain, which makes it easier to keep exercising. The lower back and hips This book focuses on the wrists, neck and shoulders, where programmers most commonly report problems. But long hours of sitting also affect the lower back and hips, and the same principles apply. As Chapter One described, the evidence that particular sitting postures cause low back pain is weak and inconsistent. The best-supported advice is broadly the same as for the neck: vary your position, avoid long unbroken spells in any one posture, stay active and strong, and do not treat a slouch as a catastrophe. If you find that a particular way of sitting consistently makes your back uncomfortable, change it, and use the discomfort as a signal to move rather than a cause for alarm. Some simple movements help. Stand up and gently arch backward with your hands on your hips, then bend forward to let your arms hang toward the floor, then stand tall again. While standing, rock your hips in slow circles. For the front of the hips, which spend hours in a bent position while sitting, try a kneeling hip flexor stretch: kneel on one knee on a cushion, with the other foot flat on the floor in front of you and that knee bent at about a right angle. Keeping your upper body upright, gently shift your weight forward until you feel a stretch at the front of the hip of the kneeling leg. Hold for twenty to thirty seconds, then switch sides. Tighten your buttock muscles on the kneeling side slightly to deepen the stretch comfortably. For strength, the same general principle holds as for the shoulders. Exercises such as squats, lunges, bridges and simply walking and climbing stairs build the capacity of the back, hips and legs. The WHO guidelines introduced in Chapter Two recommend muscle-strengthening activities involving all major muscle groups on at least two days a week for adults. That recommendation is about general health, but its benefits extend directly to how comfortable you are after a long day at a desk. A worked example Consider an illustrative fourth-year student who spends most of her final year in the lab working on a machine learning capstone. By midyear she has a persistent ache between her shoulder blades and tightness along the tops of her shoulders, worse by evening. She tried stretching several times a day, which felt pleasant but made little lasting difference. A physiotherapist at her university's sports medicine clinic suggests a different approach. First, they look at her setup: her lab desk is fixed at a height that suits taller students, so she lifts her shoulders slightly to type all day. She lowers her chair to bring her forearms level with the keyboard, adds a footrest made from a sturdy box so her feet are supported, and moves her mouse right next to the keyboard. Second, she starts a strengthening routine of shrugs, rows, band pull-aparts and push-ups, twice a week in her room using a resistance band and a backpack filled with textbooks. Third, she continues her stretches, but now uses them primarily as a reason to get up and change position during natural pauses in her work. Over the following two months the ache between her shoulder blades becomes much less frequent. On heavy days she still notices it, but it no longer follows her home. What made the difference was not a better posture held with effort, but a combination of lower demand on the shoulder muscles, greater strength in them, and regular changes of position. That combination, reducing demand where it is easy, building capacity steadily, and breaking up long periods of stillness, is the whole approach of this book. The next chapter turns to the first of these three: setting up the workspaces students actually use. Chapter 6: The Workspace You Actually Have Ergonomics guides tend to picture an office worker with a fully adjustable chair, a height-adjustable desk, a monitor on an arm, a separate keyboard and mouse, good lighting and a quiet room. Students rarely have any of that. They work at dorm desks built for the average height of someone else, at library tables shared with strangers, in computer labs where the chairs have been broken in odd ways by years of use, at kitchen tables, on sofas, in cafés, on trains, on the floor and, very often, in bed. Much of their work is done on a single laptop that goes everywhere with them. This chapter is about making those real workspaces good enough. The aim is not a perfect setup. As the previous chapters have argued, there is no perfect position to set up for. The aim is to remove the biggest, most sustained demands on your neck, shoulders and wrists, cheaply, so that the variety of positions you move through during a session are all reasonably comfortable. The laptop problem The laptop is the single most important piece of equipment in most students' working lives, and it has a fundamental design conflict. The screen and the keyboard are joined together. If the keyboard is at a comfortable height for your hands, the screen is too low for your eyes. If you raise the screen to eye level, the keyboard is too high for your hands. There is no position of a laptop alone that makes both comfortable for long periods. For short sessions, this does not matter. Checking email, reading a few pages, or writing a quick fix on a laptop in any position is fine. For long sessions, the conflict matters a great deal, because your neck is bent forward to look down at the screen for hours while your arms work in a cramped position close to your body. The solution is to separate the screen from the input. Raise the laptop so the screen is closer to eye level, and use a separate keyboard and mouse at a lower height. A laptop stand does this well, but a stack of sturdy books, a box, or an upturned drawer organiser works just as well. A basic external keyboard and mouse cost relatively little and can be used for years. If you have access to an external monitor, perhaps in a lab or one you have bought secondhand, that is even better; you can place the laptop to one side as a second screen or close it entirely. If you can only change one thing about the way you work, this is probably the one to change. It transforms a laptop from a device that forces a compromise into one that can be set up comfortably almost anywhere there is a table. Carry the equipment with you if you work in several places. A foldable laptop stand, a compact keyboard and a small mouse fit in most backpacks. Some students keep one set at home and one in a locker near their lab. Adjusting what you can Whatever desk and chair you have, a few questions guide sensible adjustments. Can you see comfortably without leaning in? If not, raise the screen, bring it closer, increase the font size, reduce glare, or have your eyes checked. A reasonable starting point is a screen roughly an arm's length away with its top edge at about eye level or a little below. If you wear progressive lenses or bifocals, you may find a lower screen more comfortable because you look through the lower part of the lens for near vision; an optometrist can advise on glasses designed for computer distances. Can your shoulders relax while your hands are on the keyboard? When your arms hang loosely at your sides and you bend your elbows, your forearms should be able to rest roughly level with the keyboard without lifting your shoulders. If the desk is too high, raise your chair. If raising the chair leaves your feet dangling, put something under them: a box, a stack of books, a backpack, or a footrest. If the desk is too low, which is less common, raise the keyboard on something stable or look for a higher surface. Is your mouse close by? Reaching far to the side or forward for a mouse, repeated many times, keeps your shoulder working. Keep the mouse next to the keyboard. Compact keyboards without a number pad let the mouse sit closer to the centre of your body. If you use a number pad heavily, you can buy a separate one and place it on the other side, or on the left if you mouse with your right hand. Can you sit back? A backrest that supports you lets the muscles of your back and neck relax. If your chair's backrest is far behind you when you are close enough to reach the keyboard, bring the keyboard closer or put a cushion behind your back. A rolled towel or small cushion at the lower back is a cheap way to make an unsupportive chair more comfortable. Are your feet supported? Dangling feet encourage perching on the front edge of the seat, which tends to pull the rest of the body forward. Feet resting flat on the floor or a footrest let you sit back and relax. None of these adjustments requires a specific angle or measurement. They are about comfort and relaxation. Once you have made them, stop fiddling. The setup is a base from which to move, not a position to hold. Common student setups Table 3 compares several workspaces students commonly use, the main problem each tends to create, and a cheap fix. It is not an exhaustive list, but it covers most of the places where long coding sessions actually happen. Table 3. Common student workspaces, their main problems and cheap fixes. Workspace Main problem Cheap fix Best used for Laptop on a desk Screen too low, neck bent for hours Raise laptop, add keyboard and mouse Long sessions once fixed Laptop in bed or on a sofa Neck and wrists bent, no support Pillow under laptop, change position often Short reading or review Library or café table Fixed height, often too high Carry stand and keyboard, adjust chair if possible Medium sessions Computer lab Shared, poorly adjusted chairs and screens Take a minute to adjust chair and screen Long sessions if adjusted Standing at a counter Fatigue in legs and back over time Alternate with sitting, shift weight Short to medium spells Floor or low table Deep forward bend of neck and back Use as one position among several Short spells only Notice the last column. Many of these setups are fine for some kinds of work and some durations. Lying in bed with a laptop to read a paper or review a pull request is not going to harm you. Working in bed on a laptop for six hours is a heavy demand. The problem is almost never a single position but spending too long in one that makes your body work hard. Lab computers deserve special mention because many students use them without adjusting anything. The chair was set by the last person, the screen may be at an odd height, the keyboard may be tilted up on its feet. Spending a minute at the start of each session adjusting the chair height, the backrest, the screen and the keyboard pays for itself many times over. If lab chairs or monitors are broken or unadjustable, it is reasonable to report it to whoever runs the lab. Many departments will fix such things when asked. Standing desks and other equipment Sit-stand desks have become popular, and they have a real benefit: they make it easy to change position. A 2018 Cochrane review led by Nipun Shrestha found low-quality evidence that sit-stand desks reduce sitting time at work, by an average of about 100 minutes per workday in the short term and less in the medium term. The review found little evidence about their effects over longer periods. It is worth being clear about what this means. A sit-stand desk is useful because it adds a position to your repertoire, not because standing is inherently healthier than sitting. Standing still for hours has its own problems, including fatigue and discomfort in the legs, feet and lower back. The goal is to alternate, not to replace one static posture with another. If you have access to a sit-stand desk, in a lab or library for instance, use it to change positions during a long session. If you do not, a kitchen counter, a high shelf or a box on a table can provide a standing option for part of the time. Treadmill desks, cycling desks and balance boards are sometimes promoted as ways to keep moving while working. The Cochrane review found their effects on sitting time unclear or inconsistent, and most students do not have access to them. They are not necessary. Ergonomic chairs can be comfortable and are worth having if you can afford one and will spend years at a desk. But an expensive chair does not protect you from the effects of sitting still for hours, and a cheap chair adjusted sensibly, with a cushion and a footrest, can be perfectly adequate. Do not let the belief that you need expensive equipment stop you from making the simple changes described above. Posture-correcting braces, shoulder straps and devices that buzz when you slouch are widely marketed. As Chapter One explained, the idea that holding a particular posture prevents pain is poorly supported, and devices designed to keep you in one position work against the principle of variety. They are best avoided. Split and tented keyboards, which divide the keys into two halves angled apart or raised in the middle, allow the wrists and forearms to sit in a less twisted position. Some programmers find them much more comfortable, especially those with wide shoulders or existing wrist discomfort. Others dislike them. They take time to adapt to. If you are curious, try one before buying, perhaps by borrowing a friend's for a few days. Vertical mice, which hold the hand in a handshake position rather than palm-down, reduce the forearm rotation that a conventional mouse requires. The evidence that they prevent problems is limited, but many people with forearm discomfort find them more comfortable, and they add variety when alternated with a conventional mouse or trackpad. Accessibility tools and alternative input For some students, especially those with existing hand, wrist or arm conditions, or who are going through a flare-up of pain, reducing hand use substantially for a period may help. Several options exist. Voice dictation built into most operating systems can handle writing prose, such as documentation, reports and emails. Specialised voice-coding tools allow people to write and navigate code by speech, using spoken commands for symbols, editing actions and navigation. They take time to learn but are used productively by some professional developers who cannot type for long periods. Foot pedals can take over common keys such as modifiers or mouse clicks. Eye-tracking tools can control the pointer. If you have a condition that affects your ability to type or use a mouse, even temporarily, your university's disability or accessibility services office can often help. Depending on your institution and country, they may be able to arrange assistive technology, extra time for assessments, or other adjustments. In the United States, for instance, universities have legal obligations to provide reasonable accommodations for students with disabilities, but the details and procedures vary between institutions, and systems differ in other countries. It is worth finding out what your university offers before you need it. Pair programming, lectures and shared screens Some of the most awkward positions students adopt have nothing to do with their own desk. Pair programming, a common practice in courses and team projects, often means two people crowded around one laptop, one of them typing while the other leans in sideways to read the screen. Group debugging sessions can put four people around a single monitor, most of them twisted at an angle for an hour. In lecture halls, students balance laptops on fold-out tablets far too small for them and crane down at the screen while also looking up at the projector. The same principles apply. If you pair program regularly, use a larger shared monitor or a second screen mirroring the first, so the navigator can sit square to a screen rather than twisted. Swap the driver and navigator roles regularly, which is good practice anyway, and let the swap be an occasion to stand up and change seats. Many teams now pair remotely even when sitting in the same building, each person at their own screen with a shared editing session; this sounds antisocial but is often far more comfortable. In lectures, accept that the setup is poor and keep laptop sessions there short, or take notes by hand for part of the time. An illustrative case Consider a second-year student who shares a small flat with three others. She has no desk in her bedroom, so she works either in bed or at the kitchen table. The kitchen chairs are hard wooden ones with no adjustment, the table is a little high for her, and the overhead light produces glare on her laptop screen in the evenings. By the middle of the semester she has a stiff neck most days, and her right wrist aches after long sessions. She spends a modest amount of money on a foldable laptop stand, a basic keyboard and a mouse, and makes the rest from what is at hand. A firm cushion on the kitchen chair raises her so her forearms sit level with the keyboard; a stack of old textbooks under the table gives her feet somewhere to rest. She angles the table so the overhead light no longer reflects off the screen and uses a desk lamp to one side instead. In bed, she restricts herself to reading and reviewing, with the laptop propped on a pillow and a timer to remind her to get up. On days when the kitchen is busy, she works at the university library, where she carries the same stand and keyboard. None of this approaches an ergonomics brochure. All of it makes her usual positions easier to tolerate, and it gives her three distinct places to move between during a long day. Setting up for a hackathon Hackathons are the extreme case, and they deserve their own planning. You will usually be working in an unfamiliar space, on furniture chosen for events rather than for long work, for a long time, with little sleep. A few preparations make a significant difference. Pack a portable setup: a laptop stand, a keyboard and a mouse. Together they take little space and transform whatever table you end up at. If you have a spare monitor and can carry it, consider bringing it. Choose your spot deliberately if you can. A table at a reasonable height with a chair that has a backrest is better than a beanbag in a corner. Near a window is better than a windowless room, both for your eyes and for your sense of time. Plan to use several positions during the event. Spend some time working at a table, some standing at a counter or high table if one is available, and some in a more relaxed position for reading or planning. Change between them deliberately. Bring a water bottle and keep it full. Bring some real food. Plan when you will sleep, even briefly, and where. The next chapter shows how to build movement into the working rhythm of a long session, whether in a hackathon, a lab or your room. The setup gives you a comfortable base; the routine keeps you moving from it. Chapter 7: Movement Snacks and the Marathon Routine Every chapter so far has come back to the same conclusion: what your body most needs during a long session at a keyboard is change. Not a better fixed position, but a steady rhythm of different ones, punctuated by brief spells of movement. This chapter turns that principle into practice. It explains what the evidence does and does not say about breaks, shows how to fit movement into the natural rhythm of programming work, gives a short routine you can use during a session, and offers a plan for surviving a hackathon or a deadline crunch in better shape. What the evidence says about breaks It would be convenient to report that research has established the ideal frequency and length of breaks for computer workers. It has not. A 2019 Cochrane review by Tessy Luger and colleagues examined randomised trials of work-break schedules for preventing musculoskeletal symptoms in healthy workers. They found only six studies, involving 373 workers, all at high risk of bias. The evidence that adding more breaks, or different kinds of breaks, reduced musculoskeletal pain, discomfort or fatigue was of low quality, and the trials did not show a considerable effect. The authors called for larger, better studies. This is worth being honest about, because many sources present specific break schedules as if they were proven. They are not. What the evidence shows is that the question has not been well studied, not that breaks do not help. Several other lines of reasoning support regular changes of position. The physiology described in Chapter Two, sustained low-level muscle activity, reduced circulation in continuously contracted muscle, and the lack of the ordinary movement that tissues depend on, all point toward regular change. Many people find, from direct experience, that the dull ache of a long session eases quickly when they get up and move. And the Cochrane review on reducing sitting at work, discussed in Chapter Six, included one study in which short breaks of one to two minutes every half hour reduced total sitting time more than two long breaks a day did. That study measured sitting, not pain, and its evidence was rated low quality, but it suggests that frequent short breaks are a practical way to break up long periods of stillness. The sensible conclusion is this: nobody knows the optimal break schedule, and it probably differs between people and tasks. But short, frequent changes of position and brief bouts of movement are cheap, pleasant, unlikely to do harm, and consistent with everything we understand about why long static sessions cause discomfort. Treat the specific intervals in this chapter as reasonable starting points to adjust, not as prescriptions. Using the natural rhythm of programming The biggest obstacle to taking breaks is not ignorance but interruption. Programming involves holding a large, fragile structure in your head, and anything that disrupts it can cost you minutes of reconstruction. Being told to stand up every twenty minutes regardless of what you are doing feels like being asked to drop a stack of plates. The solution is to attach movement to moments when your concentration is already interrupted. Programming has many of these, and most programmers spend them staring at the screen waiting. Consider how many times in a typical session you: · start a build, a compile or a test suite and wait for it to finish; · deploy something and wait for it to come up; · run a training job, a data pipeline or a long query; · push a commit and wait for continuous integration to report back; · install dependencies or pull a large container image; · wait for a teammate to review a change or answer a question; · finish a function, fix a bug or close a ticket, and pause before starting the next. Each of these is an opportunity. The build takes forty seconds: stand up and do the chest stretch in the doorway. The test suite takes three minutes: walk to the water fountain and back, doing shoulder rolls as you go. The deployment is waiting on something: do the hand and forearm stretches while watching the logs. You have just fixed a hard bug: stand up, look out the window at something far away, and let your eyes and neck move before diving into the next problem. Attaching movement to these moments has two advantages. It costs no concentration, because your concentration was already interrupted. And it creates a habit linked to a regular cue, which is far more reliable than trying to remember to move. Within a few weeks, many people find that starting a test suite automatically prompts them to stand. Completion points are especially good moments. Finishing a unit of work is a natural boundary, and taking a minute to move before starting the next one can also give your mind a chance to reset. Many programmers report that solutions to stubborn problems arrive when they step away from the screen, which is a further argument for doing so. For longer spells of uninterrupted work, when you are deep in a problem and nothing forces a pause, a gentle timer can help. The Pomodoro technique, which divides work into blocks of about twenty-five minutes separated by short breaks, is popular among students and works for many people. Others prefer longer blocks of forty-five or fifty minutes. The point is not the exact number but having a reliable prompt. Choose a prompt that is easy to postpone for a minute or two so that you can reach a sensible stopping point, rather than one that forces you to abandon a thought midway. Micro-movements while you work Not all movement needs a break. Many small changes can happen while you keep working, especially during reading-heavy tasks like reviewing code or reading documentation. Change how you sit every so often. Lean back and read with the keyboard on your lap. Sit forward with your elbows on the desk. Cross your legs, then uncross them. Sit on one foot for a while. Shift your weight from one side to the other. These positions are not bad posture; they are variety. Move your feet and legs under the desk. Point and flex your ankles, press your heels into the floor, lift your knees alternately. It helps circulation and reminds you that you have a body. Let your hands drop between bursts of typing. When you pause to think, let your hands rest on your thighs or hang by your sides rather than hovering over the keyboard. Look away. Every time you pause to think, look at something far away rather than at the screen. Blink. Stand to read. If you can stand at a counter or high surface, do some of your reading there. Code review and documentation work particularly well standing. Take calls and meetings on your feet. Stand-up meetings are called that for a reason; take them literally. If your team meets online, you can often stand or walk around with headphones. A five-minute routine For longer breaks, perhaps once every hour or two, it helps to have a short, familiar routine that moves all the regions covered in this book. Table 4 sets out a routine of about five minutes, drawing on stretches and movements described in Chapters Three, Four and Five. You can do it standing beside your desk. Once you have learned it, it requires no thought. Table 4. A five-minute desk routine. Movement How Approximate time Region Stand, look far away, blink Look out a window or across the room 30 seconds Eyes, neck Shoulder rolls, shrug and release Five circles each way, three shrugs 45 seconds Shoulders Chin nods and neck rotations Five nods, five turns each side 45 seconds Neck Doorway or hands-behind chest stretch Hold each side about 20 seconds 45 seconds Chest, shoulders Wrist flexor and extensor stretches Hold each about 20 seconds, both arms 60 seconds Forearms, wrists Tendon glides and finger spreads One cycle of each, both hands 30 seconds Hands Standing back arch and forward hang Three slow repetitions each 30 seconds Back, hips Short walk To the water fountain or window and back 30 seconds or more Whole body This routine is a template, not a script. If your forearms are the main problem, spend longer on them. If your neck is fine but your upper back is stiff, swap in seated rotations. If you are in a lab and feel self-conscious, skip the movements that look strange and do the rest. The routine's value lies in its familiarity: because you have done it many times, starting it requires no decisions. It is also worth noticing how you feel before and after. Many people discover that five minutes of movement improves not only physical comfort but also alertness, which is valuable in a long session. Planning a hackathon or deadline crunch Hackathons and deadline crunches compress everything this book discusses into an intense period. A little planning helps a great deal. Before the event, prepare your body. If you know a long event or crunch is coming, keep up your regular exercise in the weeks beforehand, and try to arrive reasonably rested rather than already sleep-deprived. Pack your portable setup as described in Chapter Six, along with a water bottle, some decent food and anything you need for sleeping. At the start, set up your space properly. Take ten minutes to arrange your laptop stand, keyboard and mouse and adjust your chair. It feels like wasting time when everyone else is diving in, but it will pay off many times over. During the event, anchor movement to team rhythms. Hackathon teams have natural synchronisation points: stand-ups, integration attempts, meals, sponsor talks, demo rehearsals. Make each one a movement point. Suggest standing up for team discussions. When someone is running a long build, have everyone stretch. A team that moves together is more likely to keep moving than an individual trying to remember alone. Rotate positions deliberately. Plan to spend some of the time at a table, some standing, some in a more relaxed position for planning and reading, and some walking, perhaps to get food or fresh air. Protect some sleep. Even a short sleep during a long event is far better than none, both for your pain sensitivity, as described in Chapter Two, and for the quality of your work. Many experienced hackathon participants plan a few hours of sleep in shifts, so the team keeps making progress while each member rests. Error rates rise with fatigue, and a bug introduced at five in the morning can cost more time than the sleep would have. Keep your intake steady. Drink water regularly. Eat actual meals when you can. If you use caffeine, spread it out rather than drinking a large amount at once, and avoid it in the hours before you plan to sleep. After the event, recover. Sleep properly. Move. Go for a walk. Return to your normal exercise routine within a day or two. Expect some residual stiffness and treat it with gentle movement rather than alarm. If anything is still bothering you after a week or so, or is getting worse, consider having it checked. Consider an illustrative team of three students who have attended several hackathons. At their first, they worked in whatever positions they happened to land in, drank energy drinks, and finished the weekend with aching necks and one member with a painful wrist that took weeks to settle. For their next event, they agreed on a few rules. Everyone brought a stand and keyboard. Every integration attempt was preceded by the whole team standing up for a few minutes of the routine above, which also served as a moment to talk through the plan. They slept in shifts of about three hours each. They ate dinner away from their laptops. They finished the event tired, as everyone does, but in noticeably better shape, and they felt that their final demo had been better prepared because the breaks had given them regular moments to step back and think. When discomfort arrives mid-session Even with good habits, there will be sessions when something starts to hurt: a burning forearm at one in the morning, a neck that has stiffened during a long debugging hunt, a headache building behind the eyes. What you do in the next few minutes shapes how the rest of the session goes, and it is worth having a simple sequence ready so that you do not have to think. First, stop and take stock. Rather than pushing on and hoping it passes, spend thirty seconds noticing what you feel and where. Is it an ache, a burn, a stiffness, a tingling? Is it in the muscle, around a joint, or spreading along a limb? Run through the sources in Table 1 in Chapter Two. Have you been completely still for a long time? Are you gripping or pounding the keys? Are you straining to see? Are you exhausted, hungry, dehydrated or anxious? Second, change the thing you can most easily change. If you have been still, get up and do the routine above. If your grip is tight, consciously soften it and use the keyboard for navigation for a while. If you are straining to see, enlarge the text and look into the distance. If you are exhausted, consider whether a short sleep would serve the project better than another hour of degraded work. Third, change your position for the next stretch of work. Move to a different chair, stand for a while, swap your mouse to the other hand, or switch to a task that uses your hands differently, such as reading documentation, sketching a design on paper, or talking through a problem with a teammate. Most programming projects have plenty of useful work that does not involve typing. Fourth, judge by the trend rather than the moment. Discomfort that eases within minutes of a change is behaving as you would expect for the ordinary fatigue of a long session, and it is safe to keep working while you continue to vary your position. Discomfort that keeps climbing despite changes, that becomes sharp, or that brings numbness, tingling or weakness in the hand is a signal to stop for the night and, if it persists, to get it assessed. Chapter Eight describes these signals in more detail. Finally, resist the two opposite mistakes. One is ignoring the signal and grinding on for hours, which is how a minor irritation becomes a problem lasting weeks. The other is panicking, stopping all activity and assuming something is badly damaged. Nearly all mid-session aches are the body asking for a change, and the right response is to give it one. Making it stick The difference between knowing what to do and doing it is, as with any habit, mainly a matter of cues and repetition. A few strategies help. Anchor movement to cues that already happen, as described above. Builds, tests, commits and completions are ideal. Make it social. If you work with others, agree to move together. A study group that stands up every forty-five minutes is more likely to keep doing it than an individual. Pair programmers can use role swaps as movement points. Make it visible. Some programmers use a small physical reminder, such as a sticky note on the monitor or an object on the desk, as a prompt to move. Start small. Pick one or two movements and one or two cues, and build from there. It is better to do a thirty-second stretch at every test run than to plan an elaborate routine you never actually do. Notice the benefits. Pay attention to how you feel at the end of days when you moved regularly compared with days when you did not. For most people, the difference is clear enough within a week or two to become its own motivation. The routine and habits described in this chapter manage the effects of long sessions on the body. But the body's tolerance for those sessions is shaped by much more than what happens at the desk. The final chapter looks at the broader picture: strength, fitness, sleep, stress, and when to seek professional help. Chapter 8: Beyond the Desk It is natural to look for the causes of desk-related aches at the desk. The chair, the screen, the keyboard, the angle of the neck: these are visible, concrete and easy to blame. This book has not dismissed them. But it has argued repeatedly that what happens at the desk is only part of the story. How much strain your body can absorb depends heavily on what happens away from it: how strong and active you are, how well you sleep, how stressed and low you feel, and how you think about the aches when they come. This final chapter turns to those broader factors. It then sets out, as clearly as possible, when an ache is something to manage yourself and when it needs a professional's attention, and how to find that help as a student. Capacity is built away from the keyboard Chapter One introduced the idea of tolerance: how much load, for how long, your tissues and nervous system can handle before they complain. The most reliable way to raise that tolerance is regular physical activity, and in particular strength training. The World Health Organization's 2020 guidelines, summarised by Fiona Bull and colleagues, recommend that adults do at least 150 to 300 minutes of moderate-intensity aerobic activity per week, or 75 to 150 minutes of vigorous activity, or an equivalent combination, along with muscle-strengthening activity involving all major muscle groups on two or more days a week. They also recommend reducing sedentary time, and emphasise that some activity is better than none. For a student, 150 minutes of moderate activity might mean walking or cycling to campus most days, a couple of sessions of a sport, dancing, swimming, or any activity that raises your heart rate and breathing. Muscle strengthening might mean the exercises described in Chapters Three, Four and Five, bodyweight exercises in your room, a gym session, climbing, or a sport that involves strength. Many universities offer affordable or free access to gyms, pools, fitness classes and recreational sports, and joining a club is often the easiest way to make activity regular, because it adds a social commitment. These recommendations are about general health, and they are worth following for that reason alone: regular activity is associated with lower risk of heart disease, diabetes, depression and many other conditions. But their benefits for the specific strains of computer work are direct. The Danish trial in Chapter Five showed that targeted strength training substantially reduced neck and shoulder pain in computer workers. The Colorado study in Chapter One found that neck muscle endurance predicted who developed chronic neck pain. Strong forearms tolerate more typing. Fit bodies recover faster from long sessions. And regular exercise improves sleep and mood, which, as described below, affect pain sensitivity in their own right. One of the ironies of student life in computing is that exercise often disappears exactly when it is most needed. When deadlines pile up, the gym session is the first thing to go. Protecting even a small amount of activity during busy periods, perhaps a twenty-minute strength routine twice a week and a daily walk, pays off disproportionately. It is not time taken from your work; it is maintenance on the machine that does the work. Sleep as a pain modulator Chapter Two described the evidence linking sleep and pain: the review by Finan and colleagues found that sleep impairments reliably predict new and worsening pain, and that sleep appears to affect pain more strongly than pain affects sleep. Experimental studies suggest that sleep loss weakens the body's internal systems for dampening pain. For students, this has a simple and uncomfortable implication. The all-nighter that feels like a heroic effort to meet a deadline is also a reliable way to make your neck, shoulders and wrists hurt more, both that night and in the following days. The aches of a hackathon are not only the result of bad chairs and long hours; they are partly the result of sleep deprivation turning up the volume on every signal your body sends. This does not mean you must never stay up late. It means treating sleep as a real resource rather than a luxury. Some practical points follow. Try to keep a reasonably regular sleep schedule during ordinary weeks, so that the occasional late night is an exception rather than the norm. Plan work to avoid last-minute crunches where you can; this is easier said than done, but starting assignments earlier is one of the most effective interventions for both grades and bodies. During a necessary crunch, protect some sleep, even a few hours, rather than none. Avoid caffeine in the late afternoon and evening on nights you intend to sleep. And if you have ongoing difficulty sleeping, it is worth raising with your student health service, because effective treatments for insomnia exist. Stress, mood and the nervous system Pain is not simply a readout of tissue damage. It is produced by the nervous system as a protective response, integrating signals from the body with context, expectation, mood and past experience. This is well established in pain science, and it is why the same physical strain can feel mild on a good day and severe on a bad one. For programmers, several consequences follow. Stress tends to increase muscle tension, especially in the shoulders, jaw and hands, adding to the physical load. It also sensitises the nervous system, making it quicker to produce pain. Low mood is associated with neck pain, as the Australian and Colorado studies both suggested, though the direction of the relationship is not always clear. And worry about pain itself, especially the belief that aches signal damage, can make pain worse and more persistent. This is not a claim that pain is imaginary or that it is your fault. Pain influenced by stress and mood is real pain. The point is that there are more levers available than purely mechanical ones. Managing stress through activity, sleep, time with friends, time outdoors, and whatever else genuinely helps you, is part of managing the physical strain of computer work. Some specific practices can help during long sessions. Slow breathing, with a longer out-breath than in-breath, is a simple way to reduce arousal for a moment. A body scan, where you briefly bring attention to each area of your body in turn and deliberately relax it, can reveal and release tension in the shoulders, jaw and hands that you had not noticed. Neither needs special training; a minute of either during a break is useful. It also helps to hold a reassuring and accurate understanding of what aches mean. Most aches during long computer sessions are the body's request for a change, not evidence of damage. Understanding this lets you respond calmly rather than anxiously, which is itself protective. If stress or low mood are more than a passing feature of busy weeks, if you feel persistently down, hopeless, unable to enjoy things, or overwhelmed, please talk to someone. University counselling services exist for exactly this and are used by many students. If you are in crisis or thinking about harming yourself, contact emergency services immediately. In the United States, you can call or text 988 to reach the 988 Suicide and Crisis Lifeline, or call 911 in an emergency. Other countries have their own emergency numbers and crisis services, and your university can tell you what is available locally. Counting the whole week's load When students think about the strain on their hands, neck and shoulders, they usually count only coursework. But the body does not distinguish between an hour spent on an assignment and an hour spent on anything else that uses the same structures in the same way. A realistic account of your load has to include everything. For many computing students, that means gaming, often with a mouse held in a fast, tense grip for long evenings, or a controller that keeps the thumbs working hard. It means phone use, which adds hours of thumb work and head-down reading on top of time at a keyboard. It may mean a part-time job that involves a till, a warehouse scanner or more computer work. It may mean playing an instrument, climbing, rowing, weightlifting or other activities that place heavy demands on the forearms and grip. None of these is a problem in itself. Many of them, particularly sport and music, bring real benefits, and climbing and weightlifting build exactly the forearm and shoulder strength that protects against computer strain. The issue is the total, and particularly sudden changes in the total. Tissues adapt well to load that rises gradually; they complain when it spikes. A student who adds a new online game on top of a heavy assignment week, or who starts climbing three times a week in the same month as a hackathon, is asking a lot of the same tendons all at once. A useful habit when something starts to ache is to ask what has changed in the past two or three weeks, across your whole life rather than only at your desk. Often the answer is obvious once you look: a new game, a new sport, a new job, a new course with a heavy coding load, a new phone habit, fewer nights of sleep. Reducing the spike, not necessarily stopping the activity, is usually enough to let things settle. When you then build back up, do it gradually over a few weeks rather than returning straight to the full load. The same thinking applies to the start of each academic year. After a summer with less computer work, the first weeks of a demanding semester can represent a sharp rise in load. Easing into long sessions, and keeping up activity and sleep during that transition, helps the body adapt rather than protest. When to seek help Most aches from computer work are self-limiting and respond well to the measures in this book. But some symptoms need professional assessment, and it is important to recognise them. The following warning signs mean you should see a clinician rather than relying on self-management. For the hands and arms, see a clinician if you have numbness, tingling or burning in your fingers that persists or keeps coming back, particularly at night or on waking; weakness in your hand, dropping things, or difficulty with fine tasks such as buttons; wasting of the muscles at the base of the thumb; or pain that is steadily getting worse despite reducing aggravating activities. For the neck, see a clinician if your neck pain is accompanied by pain, numbness, tingling or weakness spreading down one or both arms; if it followed an injury such as a fall or collision; if it is severe and not easing after a few weeks; or if it is associated with headaches that are new, persistent or getting worse. Seek urgent care, by calling emergency services, for a sudden, severe headache unlike any you have had before; neck pain with fever, rash, confusion or a stiff neck that stops you touching your chin to your chest; sudden weakness, numbness or drooping on one side of the face or body; difficulty speaking or understanding speech; or loss of coordination or balance. These can indicate serious conditions that need immediate attention. For the back, seek urgent care if you have back pain with numbness around the genitals or buttocks, new difficulty controlling your bladder or bowels, or progressive weakness in your legs. More generally, see a clinician for any pain that is severe, constant, getting worse over weeks, waking you at night, associated with unexplained weight loss, fever or feeling unwell, or simply worrying you. There is no penalty for having something checked that turns out to be minor. Finding help as a student Most universities have a student health service offering general medical care, often at low or no cost to students, depending on the institution and country. This is usually the best first stop. A general practitioner or nurse can assess you, advise on management, and refer you to specialists if needed. Many universities also have physiotherapy, sports medicine or athletic training services, sometimes attached to a sports centre or a health sciences department. Physiotherapists, called physical therapists in the United States, are well placed to assess musculoskeletal pain, advise on exercise, and help you manage aches from computer work. Some universities offer ergonomic assessments for students with ongoing problems. Costs and access vary widely. In the United States, access often depends on your health insurance, which many universities offer or require; check what your student plan covers, including whether you need a referral for physical therapy and whether there are co-payments. In countries with national health systems, you may be able to see a physiotherapist through that system, though waiting times vary, or through your university. Your student health service can explain the options where you are. If an ongoing condition affects your ability to use a computer for your studies, contact your university's disability or accessibility services, as described in Chapter Six. They can often arrange assistive technology or academic adjustments. It may help to know what to expect. A first physiotherapy appointment for desk-related pain usually involves a conversation about your symptoms, work and habits, followed by a physical examination of movement, strength and, where relevant, nerve function. Most people leave with an explanation of what is likely going on, some advice on reducing aggravating loads, and a small number of exercises to do at home. Passive treatments such as massage may feel pleasant, but the lasting benefit usually comes from the exercises and changes you carry on with afterwards. When you see a clinician, it helps to describe not only the symptom but also the context: how much you use a computer, how your workspace is set up, how you have been sleeping, what exercise you do, and what makes the pain better or worse. This lets them give advice suited to your actual life. Bringing it together Consider one last illustrative case, which draws together the threads of this chapter. A third-year student preparing for internship interviews has been spending long evenings on coding practice problems while carrying a full course load. He has stopped playing football with friends, sleeps five or six hours most nights, and feels anxious about his job prospects. His neck and shoulders ache most evenings, and occasionally he feels tingling in his right hand. At the student health service, a doctor examines him, finds no signs of nerve damage, and explains that the tingling, which comes and goes and is not present at night, is most likely related to sustained positions and muscle tension rather than a nerve condition, while asking him to return if it becomes persistent, occurs at night, or is accompanied by weakness. The doctor refers him to the campus physiotherapist, who reviews his setup and suggests the kinds of changes described in Chapter Six, teaches him a strengthening routine, and encourages him to return to football. The doctor also suggests he talk to the counselling service about his anxiety, and discusses his sleep. Over the following two months, he raises his laptop and uses an external keyboard, builds movement into his practice sessions, does his strength exercises twice a week, plays football again on weekends, and makes an effort to be in bed by midnight on most nights. He also sees a counsellor a few times. The aches do not vanish entirely, but they become occasional rather than constant, and the tingling stops. He notices that his anxiety about interviews has also eased, partly because he is sleeping better and partly because football gives him a break from thinking about them. No single change solved his problem. The combination did, because the problem itself was a combination: of load, time and tolerance, of desk and life. That is the typical shape of the strain that computer work places on the body, and it is why the most effective response is broad rather than narrow. The conclusion that follows draws out what this means for how you approach the years of computing ahead of you. Conclusion: From Correct Form to Good Variety The idea this book set out to challenge is easy to state: that there is a correct way to sit at a computer, and that holding it will protect you. Having followed the evidence through the neck, the shoulders and the hands, through the physiology of long sessions and the realities of student workspaces, we can now say something more precise about what should replace it, and why the replacement matters more than it might first appear. What follows from the evidence The first consequence is a shift in what you pay attention to. If posture were the main cause of strain, the sensible thing to monitor would be the shape of your spine. Because it is not, the more useful things to monitor are time and trend: how long you have been in one position, and whether your discomfort eases when you change it or keeps climbing. These are things you can actually observe during a session, unlike the precise angle of your neck. They are also things you can act on immediately. A programmer who notices "I have not moved for ninety minutes" has something to do about it. A programmer who notices "my head is forward" has, at most, a vague instruction to try harder. The second consequence is that the most important interventions are cheap and ordinary. Raising a laptop screen and adding a separate keyboard costs little. Making the font bigger costs nothing. Standing up while a test suite runs costs nothing. Strength exercises with a resistance band or a backpack full of books cost almost nothing. Protecting sleep and continuing to play a sport cost only the discipline to prioritise them. The equipment that is most heavily marketed, from posture braces to expensive chairs to buzzing sensors, sits at the edges of what matters, and some of it works against the principle of variety. The third consequence is less obvious. Once you see strain as a matter of load, time and tolerance, it becomes clear that the body at the keyboard is not a fixed quantity to be protected but a capacity to be developed. A first-year student who builds forearm and shoulder strength, learns to move during natural pauses, and keeps up regular activity will, by the final year, be able to handle a volume and intensity of computer work that would have hurt at the start. This is the opposite of the traditional message, which treats computer work as a slow erosion to be minimised. The more accurate message is that your tolerance for computer work can grow, provided you build it deliberately and do not let the demands outrun it. What remains unsettled Honesty requires admitting how much is still unknown. The research on break schedules, reviewed in Chapter Seven, is thin and of low quality, and nobody can yet say with confidence how often or how long breaks should be. The evidence on specialised equipment such as vertical mice and split keyboards is limited. Most of the studies of posture and pain are cross-sectional, capturing a moment rather than following people over time, and so cannot fully settle what causes what. The best trials of exercise for neck and shoulder pain in computer workers are relatively small. And almost none of this research has been conducted on students specifically, still less on the peculiar conditions of hackathons and deadline crunches. This uncertainty cuts in two directions. It means that confident prescriptions, whether of exact break intervals or of a single ideal posture, overstate what is known. But it does not mean anything goes. The broad picture is consistent across many lines of evidence: rigid attachment to one posture is poorly supported, sustained stillness is uncomfortable, strength and activity help, sleep and mood matter, and fear about posture can make things worse. Acting on that broad picture is reasonable even while the details are debated. There is also an open question worth noting for anyone considering a career in computing. The tools of programming are changing quickly. Code completion and AI-assisted development already alter how much typing a given piece of work requires, and voice and other input methods continue to improve. It is not yet clear how these changes will affect the physical demands of the work. They may reduce keystroke load while increasing time spent reading and reviewing, which shifts strain from the hands toward the eyes and neck. The principles in this book should apply whatever the tools, because they concern how bodies respond to sustained, unvarying work rather than any particular device, but the specific balance of strains may look different in ten years. What to do differently If there is one change to take from this book, it is to stop trying to find the right position and start building a habit of leaving positions. Every session, whatever your setup, move often, on purpose, and early, before discomfort forces you to. Beyond that, a few specific changes are worth making now rather than when something starts to hurt. Separate your laptop screen from your keyboard for any long session. Enlarge your text until you can read without leaning. Attach movement to the natural pauses of programming work, especially builds, tests and completions. Learn a short routine well enough that you can do it without thinking. Add strength training for your forearms, neck and shoulders two or three times a week. Treat sleep as part of your workflow, not a competitor to it. And when aches arrive, respond with curiosity and change rather than with alarm or stubbornness. There is a cultural point too. Computing culture often celebrates endurance: the all-nighter, the thirty-six-hour hackathon, the developer who works until the problem is solved regardless of the hour. Some of that is harmless and even joyful. But the culture could do more to treat movement and recovery as part of skilled work rather than a distraction from it. Teams can build movement into their rhythms. Hackathon organisers can provide places to stand, to stretch and to sleep. Instructors running long lab sessions can build in pauses. Students who have learned to look after themselves can make it normal for others to do the same, simply by standing up when the build starts. The stillness described at the beginning of this book, the room full of motionless bodies at three in the morning, is not a sign of dedication. It is a sign that the people in the room have been taught to ignore their bodies in the service of their work. The more productive habit, and the more sustainable one across a long career, is to keep both in view: to hold the problem in your mind while letting your body move. Your best posture really is your next one. The skill is remembering to take it. Notes Chapter 1 1. O'Sullivan, K., O'Sullivan, P., O'Sullivan, L., & Dankaerts, W. (2012). What do physiotherapists consider to be the best sitting spinal posture? Manual Therapy, 17(5), 432-437. https://doi.org/10.1016/j.math.2012.04.007 1. Korakakis, V., O'Sullivan, K., O'Sullivan, P. B., et al. (2019). Physiotherapist perceptions of optimal sitting and standing posture. Musculoskeletal Science and Practice, 39, 24-31. https://doi.org/10.1016/j.msksp.2018.11.004 1. Swain, C. T. V., Pan, F., Owen, P. J., Schmidt, H., & Belavy, D. L. (2020). No consensus on causality of spine postures or physical exposure and low back pain: A systematic review of systematic reviews. Journal of Biomechanics, 102, 109312. https://doi.org/10.1016/j.jbiomech.2019.08.006 (Published online in 2019.) 1. Richards, K. V., Beales, D. J., Smith, A. J., O'Sullivan, P. B., & Straker, L. M. (2016). Neck posture clusters and their association with biopsychosocial factors and neck pain in Australian adolescents. Physical Therapy, 96(10), 1576-1587. https://doi.org/10.2522/ptj.20150660 1. Damasceno, G. M., Ferreira, A. S., Nogueira, L. A. C., Reis, F. J. J., Andrade, I. C. S., & Meziat-Filho, N. (2018). Text neck and neck pain in 18-21-year-old young adults. European Spine Journal, 27(6), 1249-1254. https://doi.org/10.1007/s00586-017-5444-5 1. Shahidi, B., Curran-Everett, D., & Maluf, K. S. (2015). Psychosocial, physical, and neurophysiological risk factors for chronic neck pain: A prospective inception cohort study. The Journal of Pain, 16(12), 1288-1299. https://doi.org/10.1016/j.jpain.2015.09.002 1. Slater, D., Korakakis, V., O'Sullivan, P., Nolan, D., & O'Sullivan, K. (2019). "Sit up straight": Time to re-evaluate. Journal of Orthopaedic and Sports Physical Therapy, 49(8), 562-564. https://doi.org/10.2519/jospt.2019.0610 1. Hansraj, K. K. (2014). Assessment of stresses in the cervical spine caused by posture and position of the head. Surgical Technology International, 25, 277-279. 1. O'Sullivan, P. B., Caneiro, J. P., O'Sullivan, K., et al. (2020). Back to basics: 10 facts every person should know about back pain. British Journal of Sports Medicine, 54(12), 698-699. https://doi.org/10.1136/bjsports-2019-101611 Chapter 2 1. Hägg, G. M. (1991). Static work loads and occupational myalgia: A new explanation model. In P. A. Anderson, D. J. Hobart, & J. V. Danoff (Eds.), Electromyographical Kinesiology (pp. 141-144). Amsterdam: Elsevier. This is the original statement of what became known as the Cinderella hypothesis. 1. Bull, F. C., Al-Ansari, S. S., Biddle, S., et al. (2020). World Health Organization 2020 guidelines on physical activity and sedentary behaviour. British Journal of Sports Medicine, 54(24), 1451-1462. https://doi.org/10.1136/bjsports-2020-102955 1. Sheppard, A. L., & Wolffsohn, J. S. (2018). Digital eye strain: Prevalence, measurement and amelioration. BMJ Open Ophthalmology, 3(1), e000146. https://doi.org/10.1136/bmjophth-2018-000146 1. Finan, P. H., Goodin, B. R., & Smith, M. T. (2013). The association of sleep and pain: An update and a path forward. The Journal of Pain, 14(12), 1539-1552. https://doi.org/10.1016/j.jpain.2013.08.007 Chapter 3 1. Andersen, J. H., Thomsen, J. F., Overgaard, E., et al. (2003). Computer use and carpal tunnel syndrome: A 1-year follow-up study. JAMA, 289(22), 2963-2969. https://doi.org/10.1001/jama.289.22.2963 1. Mediouni, Z., de Roquemaurel, A., Dumontier, C., et al. (2014). Is carpal tunnel syndrome related to computer exposure at work? A review and meta-analysis. Journal of Occupational and Environmental Medicine, 56(2), 204-208. https://doi.org/10.1097/JOM.0000000000000080 Chapter 4 1. The studies discussed in this chapter are those in notes 4, 5, 6 and 12. Chapter 5 1. Andersen, L. L., Kjaer, M., Søgaard, K., Hansen, L., Kryger, A. I., & Sjøgaard, G. (2008). Effect of two contrasting types of physical exercise on chronic neck muscle pain. Arthritis and Rheumatism, 59(1), 84-91. https://doi.org/10.1002/art.23256 1. Louw, S., Makwela, S., Manas, L., Meyer, L., Terblanche, D., & Brink, Y. (2017). Effectiveness of exercise in office workers with neck pain: A systematic review and meta-analysis. South African Journal of Physiotherapy, 73(1), 392. https://doi.org/10.4102/sajp.v73i1.392 Chapter 6 1. Shrestha, N., Kukkonen-Harjula, K. T., Verbeek, J. H., Ijaz, S., Hermans, V., & Pedisic, Z. (2018). Workplace interventions for reducing sitting at work. Cochrane Database of Systematic Reviews, 2018(6), CD010912. https://doi.org/10.1002/14651858.CD010912.pub4 Chapter 7 1. Luger, T., Maher, C. G., Rieger, M. A., & Steinhilber, B. (2019). Work-break schedules for preventing musculoskeletal symptoms and disorders in healthy workers. Cochrane Database of Systematic Reviews, 2019(7), CD012886. https://doi.org/10.1002/14651858.CD012886.pub2 1. The finding on short breaks every half hour is from the review in note 19. Chapter 8 1. The physical activity recommendations are from the guidelines in note 11; the sleep and pain evidence is from the review in note 13. Further Reading Butler, D. S., & Moseley, G. L. Explain Pain. 2nd ed. Adelaide: Noigroup Publications, 2013. An accessible, illustrated introduction to modern pain science, explaining why pain is produced by the nervous system as a protective response and why understanding this can help. Useful for anyone whose aches have become a source of worry. O'Sullivan, P. B., et al. "Back to basics: 10 facts every person should know about back pain." British Journal of Sports Medicine, 2020. A short, readable editorial that sets out common misconceptions about back pain, including those about posture. Freely available online and easily found by its title. Slater, D., et al. "'Sit up straight': Time to re-evaluate." Journal of Orthopaedic and Sports Physical Therapy, 2019. A brief viewpoint by physiotherapy researchers on why beliefs about posture are so widespread and why the evidence does not support them. Bull, F. C., et al. "World Health Organization 2020 guidelines on physical activity and sedentary behaviour." British Journal of Sports Medicine, 2020. The research summary behind the WHO recommendations on activity, strength training and sitting. The WHO also publishes shorter public versions of the guidelines on its website. Sheppard, A. L., & Wolffsohn, J. S. "Digital eye strain: prevalence, measurement and amelioration." BMJ Open Ophthalmology, 2018. An open-access review of what is known about eye strain from screen use and how to manage it, written for clinicians but readable by students. Andersen, L. L., et al. "Effect of two contrasting types of physical exercise on chronic neck muscle pain." Arthritis and Rheumatism, 2008. The Danish trial showing that specific strength training reduced neck and shoulder muscle pain in computer-intensive workers. A good example of how a small, well-designed trial can change practice. United States Occupational Safety and Health Administration (OSHA). Computer Workstations eTool. A free online guide to setting up computer workstations, with practical checklists for monitors, keyboards, chairs and desks. Treat its specific positions as starting points rather than rules, in keeping with the approach of this book. Hargrove, T. A Guide to Better Movement: The Science and Practice of Moving With More Skill and Less Pain. Seattle: Better Movement, 2014. A thoughtful book by a movement teacher on how the nervous system shapes movement and pain, with practical ideas for moving with more variety and less tension. Hashtags: #PostureForProgrammers #PostureForCoders #ProgrammerHealth #CoderWellness #DeveloperWellness #ComputerErgonomics #WorkstationErgonomics #HealthyCodingHabits #CodingMarathons #HackathonWellness #TechStudentHealth #PhysicalStrainRelief #MusculoskeletalHealth #NeckPainRelief #ShoulderPainRelief #WristPainPrevention #ForearmHealth #RepetitiveStrainInjury #DigitalEyeStrain #MovementBreaks #DeskStretches #Microbreaks #PostureAwareness #ActiveSitting #SedentaryBehavior #StrengthTraining #WorkplaceWellness #SleepAndRecovery #BudgetFriendlyErgonomics #SustainableProgramming

Download the Book (PDF): Introduction A telephone line in the 1990s carried speech at 64 kilobits per second. A modern neural codec can carry the same voice, recognisably the same person, at a little over one kilobit per second, and a language model can continue that stream of bits as if it were text: finishing a sentence in the speaker's voice, translating it into another language while keeping the timbre, or answering a question before the questioner has quite finished drawing breath. These capabilities arrived quickly, mostly between 2020 and 2025, and they arrived together. That is not a coincidence. They share a single enabling idea, and understanding that idea is the most useful thing a newcomer to the field can acquire. The idea is this: speech becomes tractable for large-scale learning once it is turned into a sequence of tokens whose design deliberately separates what is said from who says it and how. Everything else in the contemporary speech stack, from transcription to synthesis to real-time conversation, is downstream of that representational choice. A recognition system is, at bottom, a model that maps acoustic representations to text tokens. A synthesis system maps text tokens, plus a sample of a voice, to acoustic tokens, and then renders those tokens back into a waveform. A voice translation system maps acoustic tokens in one language to acoustic tokens in another while carrying the speaker's identity across. A conversational voice engine does all of this in a loop, under a latency budget measured in a few hundred milliseconds. In each case the quality ceiling, the failure modes and the speed of the system are set by how the audio was tokenised, at what rate, and with what information kept or thrown away. This booklet argues that claim and works through its consequences. The argument is that the representation layer, which is often treated as plumbing, is the actual site of the field's main design decisions. Choose a tokeniser that runs at 75 frames per second with eight codebooks, and your language model must generate six hundred tokens for every second of audio; choose one that runs at 12.5 frames per second, and conversational latency suddenly becomes achievable on a single graphics card. Choose semantic tokens that were trained to discard speaker information, and you gain clean pronunciation but must reintroduce the voice somewhere else; choose acoustic tokens that keep everything, and you gain fidelity but invite the model to copy the accent of the reference speaker into a language they never spoke. These are not implementation details. They are the engineering questions that decide whether a product works. Why the word "foundation" is apt The phrase "foundation model" was coined for large models trained on broad data at scale and then adapted to many downstream tasks. Speech has its own versions. Self-supervised encoders such as wav2vec 2.0, HuBERT and WavLM learned representations of speech from tens of thousands of hours of untranscribed audio, and those representations turned out to be useful for recognition, speaker verification, emotion detection and, crucially, as the source of discrete units for synthesis. Weakly supervised recognisers such as OpenAI's Whisper, trained on 680,000 hours of audio paired with transcripts found on the internet, showed that a single model could transcribe dozens of languages robustly without per-domain tuning. Neural audio codecs such as SoundStream, EnCodec and the Descript Audio Codec showed that a learned encoder and decoder could compress audio to a few kilobits per second while leaving a discrete code that language models could read and write. And codec language models such as AudioLM and VALL-E showed that, once audio is a token sequence, the machinery that made text generation work can be pointed at speech. What makes these foundation models rather than merely large models is that each of them becomes a component in many other systems. A HuBERT encoder appears inside voice conversion systems, speech translation systems and synthesis evaluation tools. A neural codec appears as the output layer of text-to-speech systems, as the tokeniser of spoken dialogue models, and as the compression scheme for transmission. Whisper appears not only as a transcriber but as the automatic judge that measures how intelligible a synthesis system is. The field has become a set of interlocking parts, and the joints between those parts are, almost always, token interfaces. What this booklet covers, and what it does not The scope is the computational audio pipeline for human speech: how speech is captured as a representation, how it is transcribed, how it is synthesised, how a voice is preserved when it is cloned or carried into another language, and how all of this is made fast enough for conversation. It covers acoustic tokenisation in two families, semantic units derived from self-supervised encoders and acoustic codes derived from neural codecs, and explains why most modern systems use both. It covers neural vocoders, the components that turn representations back into sound, because the last stage of synthesis is where many perceptual failures appear and because the codec decoder and the vocoder have converged into the same kind of network. It covers multilingual zero-shot voice transfer, the ability to speak in a voice heard for only a few seconds, in a language the original speaker may never have used. And it covers streaming latency, because a conversational voice engine is judged first by whether it responds at a human pace, and the techniques that achieve that pace reach back into every earlier design decision. Music generation, general sound-event modelling and environmental audio are outside the scope except where their techniques bear directly on speech. So is the deep mathematics of signal processing; the booklet explains what a mel spectrogram is and why it is used, but it does not derive the short-time Fourier transform. Where specific numbers are given, such as bitrates, frame rates, dataset sizes and reported latencies, they come from the published descriptions of the systems concerned. Where the field is genuinely unsettled, as it is on several questions of evaluation and on the governance of voice cloning, the text says so. A note on the reader The intended reader is technically literate but not a speech specialist: a software engineer moving into voice products, a researcher from text or vision who keeps meeting audio tokenisers in papers, a product manager who needs to understand why the voice assistant is slow, a policy professional trying to see what is actually possible with three seconds of someone's voice. No prior knowledge of acoustics or speech recognition is assumed. Terms are defined when they first appear, and a short glossary at the end collects the ones that recur. The shape of the argument The chapters run in the order the argument needs. The first establishes the historical arc from hand-built pipelines to end-to-end and foundation models, because the current architecture makes sense only as a reaction to what came before. The second looks at what a speech signal contains, physically and informationally, because the case for factorised tokens rests on the observation that a waveform carries several distinct kinds of information at wildly different bit rates. The third and fourth treat the two families of tokens: semantic units, which keep content and discard identity, and neural codec tokens, which keep almost everything and must be organised so that a model can use them. With representations in hand, the fifth chapter turns to transcription, the direction from sound to text, and the sixth to vocoders, the last mile from representation back to sound. The seventh treats synthesis as language modelling, the paradigm that made zero-shot voice cloning practical. The eighth examines voice preservation directly: what a voice is, how identity is measured, and where it is lost. The ninth carries the voice across a language boundary, the hardest version of the problem, and the tenth confronts latency, the constraint that tests every earlier choice at once. The eleventh deals with misuse, detection and consent, which are not an afterthought but a design requirement once three seconds of audio suffice to clone a voice. The conclusion draws out what follows for anyone building, buying or regulating these systems. The thread that holds all of this together is the token: what it encodes, how fast it ticks, and what it throws away. Keep that thread in hand and the rest of the field, which can look like an undifferentiated thicket of acronyms, resolves into a small number of choices with understandable consequences. Chapter 1. From Pipelines to Foundation Models For most of its history, speech technology was built as a chain of separately engineered parts, each with its own specialists, its own training data and its own failure modes. The modern stack is the result of collapsing that chain, first into end-to-end models trained on transcribed speech, then into large pretrained models that learn from audio with little or no transcription at all. Understanding what was removed, and what had to be reinvented in learned form, explains a great deal about why today's systems are built the way they are. The classical recognition pipeline A speech recogniser from around 2005 had four main components. A front end converted the waveform into a sequence of feature vectors, usually mel-frequency cepstral coefficients computed every ten milliseconds over a twenty-five millisecond window. An acoustic model scored how well each short stretch of features matched each of a few thousand context-dependent phonetic states. A pronunciation lexicon, written largely by hand, listed how each word in the vocabulary was spelled out in phonemes. And a language model, typically an n-gram model estimated from large amounts of text, scored how plausible each sequence of words was. A decoder searched the combined space for the best word sequence. The acoustic model was a hidden Markov model whose states emitted features according to Gaussian mixture models. Each phoneme was modelled as a short left-to-right chain of states, and phonemes were made context-dependent by conditioning on their neighbours, so that the "t" in "stop" and the "t" in "top" had different models. Because there were far too many such triphones to train separately, their states were clustered with decision trees built from linguistic questions: is the left neighbour a nasal, is the right neighbour a vowel. The result was a system that worked, that was deployed in dictation software and call centres for two decades, and that required a great deal of linguistic expertise to build for each new language. Synthesis followed a parallel path. Concatenative systems recorded a single speaker for many hours, segmented the recordings into units such as diphones or half-phones, and at runtime selected and joined units to match the target text. When the right units existed in the database, the output could be strikingly natural, because it was literally the speaker's voice. When they did not, audible joins and prosodic discontinuities appeared. Statistical parametric synthesis, built on the same hidden Markov machinery as recognition, generated vocoder parameters frame by frame. It was smoother and more flexible than concatenation but had a characteristic buzzy, muffled quality, a consequence both of averaging over training examples and of the simple source-filter vocoders used to render the parameters. Every one of these components embodied a decision about representation. Mel cepstra discarded phase and fine spectral detail on the theory that recognition did not need them. Phonemes were the unit because linguists had described languages in phonemes. The vocoder in parametric synthesis represented speech as a fundamental frequency, a voicing decision and a smooth spectral envelope because that was what source-filter theory said speech was. The representations were designed by people, and the systems were only as good as the designs. Neural networks enter, then take over The first large change came between roughly 2009 and 2012, when deep neural networks replaced Gaussian mixtures as the scorers of hidden Markov model states. These hybrid systems kept the lexicon, the language model and the decoder, but the acoustic model now learned its own internal features from stacked frames of filterbank energies. Error rates on conversational telephone speech fell substantially, and within a few years every major commercial recogniser had adopted the approach. The lesson, which would repeat throughout the decade, was that a learned representation beat a designed one once there was enough data to learn it. The next step was to remove the pipeline itself. Connectionist temporal classification, introduced by Alex Graves and colleagues in 2006, gave a way to train a network to emit a sequence of labels from a longer sequence of acoustic frames without knowing the alignment between them in advance. The network outputs a distribution over labels plus a special blank symbol at every frame, and the training objective sums over all alignments that collapse to the correct transcript. A CTC model can emit characters directly, so the pronunciation lexicon is no longer required. Attention-based encoder-decoder models, popularised for speech by the Listen, Attend and Spell system in 2016, went further: an encoder reads the whole utterance, and a decoder generates the transcript one token at a time while attending to the encoder output, implicitly learning both the alignment and a language model. The recurrent neural network transducer, also introduced by Graves in 2012, combined a CTC-like encoder with a prediction network that conditions on previous outputs, producing a model that is both end-to-end and naturally suited to streaming. Synthesis underwent a similar transformation. In 2016 DeepMind's WaveNet generated raw audio one sample at a time with a deep stack of dilated causal convolutions, producing speech that listeners rated far closer to natural recordings than either concatenative or parametric systems. WaveNet was far too slow to run in real time at first, but it established that a neural network could model the waveform directly. In 2017 and 2018, Google's Tacotron and Tacotron 2 replaced the linguistic front end and acoustic model of parametric synthesis with a single sequence-to-sequence network that read characters and wrote mel spectrograms, which a WaveNet then rendered into audio. Tacotron 2 was reported to reach a mean opinion score close to that of professionally recorded speech on its single-speaker test set. The pattern across both directions was the same. Hand-designed intermediate representations, such as phonetic states, lexicons and vocoder parameters, were replaced by learned ones. What remained were two interfaces that everyone still agreed on: text at one end, and a mel spectrogram or waveform at the other. The problem of labelled data End-to-end models are hungry. A hybrid system could be trained reasonably on a few hundred hours of transcribed speech, because the lexicon and language model carried a great deal of prior knowledge. An end-to-end model must learn spelling, pronunciation and grammar from the transcribed audio itself, and it improves steadily into the tens of thousands of hours. Transcribed speech is expensive. A professional transcriber may need several hours to produce an accurate transcript of one hour of conversational audio, and for most of the world's languages there is no large transcribed corpus at all. Untranscribed audio, on the other hand, is abundant. Radio archives, podcasts, audiobooks and video soundtracks exist in quantities that dwarf any transcribed collection. The question that drove speech research from about 2019 onwards was how to learn from that audio without transcripts. The answer came from self-supervised learning, the same family of ideas that had transformed text processing with BERT. Facebook AI's wav2vec 2.0, published in 2020, encodes raw audio with a convolutional feature extractor into a sequence of latent vectors at roughly fifty per second, masks a portion of them, and trains a Transformer to identify the correct quantised latent for each masked position among a set of distractors. The quantised latents are learned jointly, so the model discovers its own discrete inventory of speech sounds. After pretraining on 53,000 hours of unlabelled audiobook speech, the paper reported that fine-tuning on as little as ten minutes of transcribed speech produced a usable English recogniser, and fine-tuning on the standard 960 hours of LibriSpeech produced state-of-the-art error rates at the time. HuBERT, published by Facebook AI in 2021, took a different route to the same end. Rather than learning quantised targets jointly, it generates targets offline by clustering acoustic features with k-means, trains a Transformer to predict the cluster identity of masked frames, then reclusters the model's own intermediate representations and trains again. The first iteration clusters mel-frequency cepstra; later iterations cluster the model's hidden states, which are far more phonetic. Microsoft's WavLM, published in 2022, added a denoising objective, in which the model must predict targets for clean speech from inputs mixed with noise or overlapping speakers, and trained on 94,000 hours of diverse audio. WavLM proved particularly strong on tasks involving speaker identity and overlapping speech, which is one reason its representations appear repeatedly in voice conversion and synthesis evaluation. These models changed the economics of speech recognition for low-resource languages, and they changed something more subtle as well. Their intermediate representations, when clustered, produce discrete units that correlate strongly with phonemes but are discovered without any phonetic labels. That made it possible, for the first time, to treat untranscribed speech as a sequence of symbols, and to train language models directly on spoken audio. The chapter on semantic tokens returns to this in detail. Scale and weak supervision Self-supervision solved the problem of learning representations without labels, but the fine-tuned recognisers built on top of it were often brittle. A model fine-tuned on audiobook speech struggled with telephone audio, accented speech or background music. OpenAI's Whisper, released in September 2022, took a different approach. It is an ordinary encoder-decoder Transformer, trained in the ordinary supervised way, but on 680,000 hours of audio paired with transcripts collected from the internet. Those transcripts are noisy and imperfect, which is why the approach is called weak supervision, but the sheer scale and diversity of the data made the model unusually robust. About 117,000 hours covered 96 languages other than English, and a further 125,000 hours paired non-English audio with English translations, so a single model could transcribe many languages and translate them into English. Whisper's significance for this booklet is twofold. First, it showed that a single recognition model could serve as general infrastructure: robust enough to transcribe audio it had never been tuned for, in many languages, with punctuation and capitalisation. Second, its multitask format, in which special tokens in the decoder prompt specify the language, the task and whether to produce timestamps, demonstrated that speech tasks could be expressed as instructions to a sequence model rather than as separate systems. That framing has since become standard. Google's Universal Speech Model, described in 2023, pushed scale further, pretraining a two-billion-parameter encoder on twelve million hours of YouTube audio spanning more than three hundred languages and then fine-tuning on smaller labelled sets. Meta's Massively Multilingual Speech project, also from 2023, took a different tack: it built a labelled dataset from readings of religious texts available in more than 1,100 languages, combined it with self-supervised pretraining, and released recognition and synthesis models covering that many languages, along with a language identification model covering about four thousand. These projects established that coverage of the long tail of the world's languages was a matter of finding data and pretraining at scale, not of building new systems language by language. The generative turn The final piece of the modern picture came from the generative side. Once speech could be represented as discrete units, the question arose whether a language model trained on those units could generate speech the way a text language model generates text. Meta's Generative Spoken Language Modeling work in 2021 showed that a model trained on HuBERT units could produce plausible-sounding babble with recognisable words. Google's AudioLM, published in 2022, combined two kinds of tokens: semantic tokens derived from a self-supervised encoder, which carried linguistic content and long-range structure, and acoustic tokens from the SoundStream neural codec, which carried the fine detail needed for high-fidelity reconstruction. Given a few seconds of a speaker, AudioLM could continue in the same voice, with the same recording conditions, saying linguistically coherent if semantically aimless things. In January 2023 Microsoft's VALL-E applied the same idea to text-to-speech. It treated synthesis as conditional language modelling over EnCodec codec tokens: given phonemes for the target text and the codec tokens of a three-second recording of a speaker, generate the codec tokens of that speaker saying the target text. Trained on 60,000 hours of English audiobook speech, VALL-E could clone voices it had never heard during training, and it preserved not only timbre but the acoustic environment and emotional tone of the prompt. This was the moment zero-shot voice cloning moved from a research curiosity to something that everyone in the field, and soon everyone outside it, had to take seriously. What the collapse left behind Looking back across this history, a few things stand out. The classical pipeline had explicit, human-designed representations at every joint: phonemes, lexicons, triphone states, vocoder parameters. The end-to-end era removed most of them but kept text and spectrograms as the agreed interfaces. The foundation-model era replaced even the spectrogram with learned discrete representations, of two kinds, and made those representations the interface between components. That last change is what makes the current generation of systems composable. A synthesis system can be built from a text encoder, a token language model, a codec decoder and a vocoder, each trained separately and each replaceable. A speech translation system can reuse a recogniser's encoder, a text translation model and a synthesis decoder. A conversational engine can wire a streaming recogniser to a text language model to a streaming synthesiser, or it can replace the whole chain with a single model that reads and writes codec tokens directly. In each case the critical design question is what the tokens at each joint contain and how fast they arrive. It also left behind some old problems in new forms. The classical systems failed in ways that engineers could diagnose: a missing word in the lexicon, a poorly trained triphone, a bad join. Modern systems fail in ways that are harder to trace: a recogniser that invents a fluent sentence during a silence, a synthesiser that skips a word or repeats a phrase, a cloned voice that drifts toward a different speaker halfway through a paragraph. Most of these failures, as the following chapters show, can be traced back to the representation: to what the tokens encode, how they are aligned to text, and what information the model is forced to infer because the tokens did not carry it. Chapter 2. What a Speech Signal Contains Before deciding how to tokenise speech, it helps to be precise about what a speech signal actually carries. The waveform recorded by a microphone is a single stream of pressure measurements, but it encodes at least four quite different kinds of information: the linguistic message, the identity of the speaker, the manner in which the message is delivered, and the acoustic circumstances of the recording. These kinds of information occupy very different amounts of the signal's capacity, change at very different rates, and matter to very different tasks. The whole case for factorised tokens rests on this observation. How speech is produced The standard physical account of speech production is the source-filter model. Air from the lungs passes through the larynx, where the vocal folds either vibrate, chopping the airflow into a quasi-periodic train of pulses, or remain open, allowing turbulent flow. The vibration rate is the fundamental frequency, usually written F0, which listeners perceive as pitch. Adult male voices typically sit somewhere around 85 to 180 hertz in conversation, adult female voices around 165 to 255 hertz, with wide individual and situational variation. Above the larynx, the throat, mouth and nasal passages form a tube whose shape is constantly changed by the tongue, lips, jaw and soft palate. That tube acts as a filter, amplifying some frequencies and attenuating others. The resonant peaks of the filter are called formants, and the pattern of the first two or three formants is what distinguishes one vowel from another. Consonants add bursts, frication noise and rapid formant transitions. The model is a simplification, but it captures the essential point that two largely independent things are being combined. The source carries pitch and voice quality, such as breathiness or creak. The filter carries the shape of the vocal tract, which changes rapidly to produce different speech sounds and also reflects the fixed anatomy of the speaker: the length of the vocal tract, the size of the nasal cavity, the shape of the palate. A child, an adult woman and an adult man saying the same vowel produce formants at systematically different frequencies because their vocal tracts differ in length, and listeners normalise for this effortlessly. This already suggests a first decomposition. What is being said lives mostly in the time-varying pattern of the filter. Who is saying it lives partly in the fixed characteristics of the filter and partly in the typical range and quality of the source. How it is being said, its intonation, stress, rhythm and emotional colour, lives in the trajectory of F0 over time, in loudness, in timing and in voice quality. A model that wants to change one of these while preserving the others must find some way to pull them apart. From waveform to spectrogram A digital waveform is a sequence of samples. Telephone speech is sampled at 8,000 samples per second, wideband speech at 16,000, and high-fidelity speech synthesis commonly uses 22,050, 24,000 or 44,100. At 16,000 samples per second with 16 bits per sample, uncompressed speech occupies 256 kilobits per second. A single second of audio is 16,000 numbers, which is a very long sequence for any model to process sample by sample. The short-time Fourier transform turns the waveform into a sequence of spectra. A window, typically twenty to fifty milliseconds long, slides along the signal in steps, typically ten to twelve and a half milliseconds, and at each position the Fourier transform reports how much energy is present at each frequency. Stacking these spectra gives a spectrogram, a two-dimensional array with time along one axis and frequency along the other. Each frequency bin has both a magnitude and a phase. Speech models have traditionally kept the magnitude and discarded the phase, partly because magnitude is what the ear is most sensitive to and partly because phase is hard to model: it wraps around, behaves almost randomly from frame to frame, and has no simple perceptual interpretation. The mel spectrogram adds a perceptual warping. Human hearing resolves low frequencies much more finely than high ones; the difference between 200 and 300 hertz is obvious, while the difference between 8,000 and 8,100 hertz is barely perceptible. The mel scale approximates this, spacing filters linearly at low frequencies and logarithmically at higher ones. A typical mel spectrogram for speech models has 80 or 128 mel bands, computed every 10 to 12.5 milliseconds, with log compression of the magnitudes to approximate the ear's roughly logarithmic sensitivity to loudness. Whisper, for example, takes 80-channel log-mel spectrograms computed with a ten-millisecond hop in its original versions, and 128 channels in its third large version. The mel spectrogram became the default intermediate representation of neural speech processing for good reasons. It is compact, about 80 numbers every ten milliseconds compared with 160 raw samples. It is smooth enough that neural networks can predict it well with regression losses. And it preserves almost everything a listener needs to understand the speech and recognise the speaker. What it discards, mainly phase and fine spectral detail at high frequencies, is exactly what the vocoder must later reconstruct. That reconstruction problem, and why it was hard, is the subject of the chapter on vocoders. The information budget The most illuminating way to look at a speech signal is to ask how many bits per second each kind of information requires. The answers differ by orders of magnitude. Start with the linguistic message. Conversational English runs at roughly 10 to 15 phonemes per second. English has around 40 phonemes, so each one carries at most a little over five bits if all were equally likely, and considerably less once the predictability of real phoneme sequences is taken into account. The phonetic content of speech therefore amounts to something on the order of fifty to a hundred bits per second. Looked at at the level of words, a speaker producing 150 words a minute delivers two and a half words per second; at roughly ten bits per word for a reasonably predictable vocabulary, the figure is in the same tens-of-bits range. The message itself is tiny. Speaker identity is different in kind. It does not change from moment to moment; it is, to a first approximation, a fixed set of parameters for the whole utterance. Speaker verification systems summarise a voice in a fixed-length embedding, commonly 192 or 256 numbers, extracted from a few seconds of speech. In information terms, identity costs a modest number of bits once and almost nothing thereafter. What makes it hard is not its size but its entanglement: the same acoustic features that carry phonetic content also carry identity, because both are expressed through the shape of the vocal tract. Prosody sits between the two. Pitch contours, durations and loudness change on the scale of syllables and phrases, a few times per second, and carry information about emphasis, sentence type, emotion and attitude. Prosody needs more bits than the phonetic message and has more variability: the same sentence can be spoken in many acceptable ways, which is one reason why early neural synthesisers trained with regression losses produced flat, averaged intonation. Finally there is everything else: the room's reverberation, the microphone's frequency response, background noise, the codec the audio passed through on its way to the recording, breaths, lip smacks, the fine stochastic texture of aspiration and frication. This residual is by far the largest consumer of bits in a high-fidelity recording. A codec that reconstructs speech at broadcast quality needs several kilobits per second precisely because it must reproduce all of this detail, most of which carries no linguistic or identity information at all. The disparity is the key fact. The message needs tens of bits per second, identity needs a few hundred bits once, prosody needs perhaps a few hundred bits per second, and acoustic detail needs thousands. A representation that treats all of these together, such as a raw waveform or a mel spectrogram, forces a model to learn a great deal of irrelevant structure in order to get at the part it needs. A recogniser does not care about room reverberation. A synthesiser should not have to choose the room reverberation when it is trying to choose the next word. A voice translation system wants to change the message completely while leaving identity untouched. Entanglement and why it is hard to undo If the different kinds of information occupied different parts of the signal, separating them would be easy. They do not. Consider a few examples. The formants that distinguish the vowel in "beat" from the vowel in "bit" are shifted systematically for a speaker with a shorter vocal tract. A recogniser must learn to normalise this shift to recover the phoneme; a synthesiser must learn to apply it to produce the right speaker. The same acoustic dimension carries both content and identity. Pitch carries prosodic information, such as the rise at the end of a yes-or-no question in English, but in tone languages such as Mandarin, Cantonese, Vietnamese or Yoruba, pitch also distinguishes words. The Mandarin syllable "ma" with a high level tone, a rising tone, a dipping tone and a falling tone means four different things. A representation that treats pitch purely as prosody, separable from content, will fail on these languages. This becomes important in cross-lingual voice transfer, where a voice from a non-tonal language must be made to speak a tonal one, and vice versa. Accent lives in the fine detail of how phonemes are realised: the precise vowel qualities, the timing of voicing onset in stops, the presence or absence of certain reductions. Accent is part of identity in the sense that listeners use it to recognise people, but it is also partly linguistic, because it is tied to the language and dialect being spoken. When a voice is carried into a new language, it is far from obvious which parts of the accent should travel with it. Recording conditions leak into everything. A self-supervised encoder trained on clean audiobooks will represent a phoneme recorded on a telephone differently from the same phoneme recorded in a studio, and a codec language model prompted with a noisy recording will reproduce the noise as faithfully as the voice, because from its perspective the noise is part of the acoustic pattern it was asked to continue. None of these problems has a perfect solution, and the field has learned to live with partial separation. The practical question is always which kinds of information a given representation keeps, which it suppresses, and what the downstream model must therefore supply on its own. The two strategies for representation Two broad strategies have emerged, and the following two chapters examine each. The first strategy starts from the recognition side. Self-supervised encoders such as HuBERT and WavLM are trained to predict masked parts of speech from context. Because the most predictable structure in speech is linguistic, their intermediate layers come to represent phonetic content strongly. Speaker identity and recording conditions are partially present but weaker in the middle layers, and clustering those representations into a small number of discrete units suppresses them further. The resulting semantic tokens, or units, carry roughly the information a phoneme transcription would carry, plus some prosodic and contextual detail, at a rate of around fifty per second. They are excellent for modelling what is said and poor for reconstructing exactly how it sounded. The second strategy starts from the compression side. Neural audio codecs train an encoder, a quantiser and a decoder together to reconstruct the input waveform as faithfully as possible within a bit budget. Their tokens carry everything needed for reconstruction, which means they carry the message, the speaker, the prosody and the room, all mixed together in whatever way the codec found most efficient. They are excellent for reconstructing audio and less convenient for modelling it, because a language model trained on them must learn the linguistic structure from tokens that were never designed to expose it. Most capable modern systems use both, either in sequence, generating semantic tokens first and acoustic tokens conditioned on them, or by building codecs whose first layer of tokens is trained to look like semantic tokens while later layers carry the acoustic residual. The factorisation that the information budget suggests, content separate from identity separate from detail, is approximated by the structure of the tokens. Where that approximation fails, the failures show up in exactly the places the information budget predicts: identity leaking through content tokens, content errors appearing when acoustic tokens are asked to carry too much, and accent travelling where it was not wanted. What this means for system design Three practical lessons follow from the information budget, and they recur throughout the rest of the booklet. First, the frame rate of a representation matters more than its dimensionality. A language model generates tokens sequentially, and its cost, both in computation and in latency, scales with the number of tokens per second of audio. A representation at fifty frames per second needs four times as many steps as one at 12.5. Since the linguistic message changes only ten to fifteen times a second, there is room to reduce frame rates substantially, and much recent progress in conversational systems has come from doing exactly that. Second, a representation should be chosen for what the downstream model needs to control. A recogniser wants content and invariance to everything else. A synthesiser wants content it can control from text, identity it can control from a reference, and detail it can leave to a decoder. A voice converter wants content from one source and identity from another. Using the same representation for all of these is convenient, but it forces trade-offs. Third, information that a representation discards must be supplied from somewhere. If semantic tokens discard the speaker, the synthesis decoder must be told who is speaking. If a codec's first layer discards fine detail, later layers or the vocoder must restore it. Errors in a system often arise not where information is lost but where it is reintroduced, because that is where the model must guess. Chapter 3. Semantic Tokens: Discovering Units Without Labels A semantic token is a discrete symbol, drawn from a vocabulary of a few hundred to a few thousand, that stands for a short stretch of speech and encodes mainly what was said. The term is somewhat misleading, because these tokens carry phonetic rather than semantic information in the linguist's sense: they distinguish sounds, not meanings. But the label has stuck, and it usefully contrasts them with acoustic tokens, which encode how the speech sounded in full detail. Semantic tokens are the bridge between the continuous world of audio and the symbolic world in which language models operate, and their properties determine a great deal about what speech language models can and cannot do. How units are made The most common recipe is simple. Take a pretrained self-supervised encoder such as HuBERT or WavLM. Run a large amount of speech through it and extract the hidden representation from one of its intermediate layers, one vector every twenty milliseconds. Fit a k-means clustering model to a sample of these vectors with some fixed number of clusters, commonly 100, 200, 500 or 1,000. To tokenise new audio, run it through the encoder, take the same layer's vectors, and replace each with the index of its nearest cluster centre. The result is a sequence of integers at fifty per second. The choice of layer matters. Self-supervised encoders develop a rough hierarchy across their depth. Early layers stay close to the acoustic input and retain a good deal of speaker and channel information. Middle-to-upper layers become more phonetic, which is what masked prediction rewards, since predicting a masked frame from context is easiest if the model has figured out which speech sound belongs there. The very last layers of some models become specialised to the pretraining objective and slightly less useful. Probing studies, which train simple classifiers on each layer to predict phone identity or speaker identity, consistently find that phonetic information peaks somewhere in the upper middle of the network and that speaker information is strongest near the bottom. Practitioners choose the layer that maximises the former while minimising the latter for their purpose; for the base HuBERT model, layer six was a common choice in early unit-based work, and for larger models a layer around two-thirds of the way up is typical. The number of clusters is a trade-off. With too few, distinct phonemes are merged and intelligibility suffers when the units are turned back into speech. With too many, the clusters begin to subdivide phonemes by speaker, pitch or context, and the units become less purely linguistic and harder for a language model to learn. Experiments in the Generative Spoken Language Modeling work at Meta compared 50, 100 and 200 units and found the middle range a good compromise for unit-based language modelling; later systems have used larger vocabularies, particularly when the units feed directly into a synthesiser that benefits from more detail. A common post-processing step is deduplication. Because speech sounds last for several frames, a unit sequence contains long runs of the same token. Collapsing each run into a single token shortens the sequence by a factor of two to three and makes it look more like a phoneme string, which helps language models learn linguistic structure. The cost is that duration information is discarded. If deduplicated units are used for synthesis, durations must be predicted separately, much as in a classical text-to-speech system. What units capture, and what they do not Semantic units derived this way are a surprisingly good approximation of a phonetic transcription. Measured against forced-aligned phoneme labels, clusters from a good HuBERT layer show high purity: most frames assigned to a given cluster belong to a small number of phonemes, and most frames of a given phoneme fall in a small number of clusters. Unit sequences can be decoded into text with a lightweight model, and they can be resynthesised into intelligible speech with a vocoder trained to map units to audio. They are not a perfect phonetic transcription, and the differences matter. Units are frame-level rather than segment-level, so they encode duration implicitly as run length. They are context-sensitive, because the encoder's representation of a sound depends on its neighbours, so the same phoneme in different contexts may map to different units. And they are not perfectly speaker-invariant. Clustering suppresses speaker information but does not eliminate it, and studies of speaker identification from unit sequences find that some identity leaks through, often enough to distinguish male from female speakers and sometimes enough to identify individuals at above-chance rates. The leakage is stronger with larger cluster counts and lower layers. Prosody is captured only partially. Units carry duration through run lengths and some stress information through context, but fundamental frequency is largely absent, because pitch variation within a phoneme does not help predict which phoneme a masked frame belongs to and so the encoder has little reason to represent it. This is a feature for some purposes and a defect for others. A voice conversion system that takes content from units and identity from a reference speaker benefits from the absence of the source speaker's pitch. A speech-to-speech system that wants to preserve the emphasis and intonation of the input must recover prosody from somewhere else. Recording conditions are suppressed to a degree that depends on the encoder. WavLM's denoising pretraining makes its units more robust to noise and overlap than HuBERT's. No unit inventory derived from a self-supervised encoder is fully channel-invariant, but in practice units are far more stable across recording conditions than spectrograms or codec tokens. Units as the vocabulary of spoken language models The most consequential use of semantic units has been as the vocabulary of language models trained on speech. Meta's Generative Spoken Language Modeling, published in 2021, trained a Transformer language model on HuBERT units derived from thousands of hours of audiobook speech and then resynthesised its outputs with a unit-to-speech decoder. The resulting speech was locally fluent and contained real words and short phrases, though it lacked the long-range coherence of text language models trained on comparable compute. The gap is instructive. A unit sequence at fifty tokens per second, or perhaps twenty per second after deduplication, is far longer than the corresponding text, which might be three or four tokens per second at the subword level. A language model with a fixed context window therefore sees far less linguistic content when trained on units, and much of its capacity is spent modelling the low-level regularities of how units follow each other within a single sound. Later work addressed this in several ways. One approach, used by AudioLM and its descendants, is hierarchical: a first model generates semantic tokens, capturing linguistic structure, and a second generates acoustic tokens conditioned on them, capturing detail. Another approach, now dominant, is to start from a pretrained text language model and teach it to read and write speech units, so that the linguistic knowledge comes from text and only the mapping to and from speech must be learned from audio. Meta's SpiritLM, for example, interleaved text tokens and speech units in training sequences so that a single model could continue in either modality. The choice of AudioLM's semantic tokenizer is worth noting. It used w2v-BERT, a Google model that combines contrastive learning in the style of wav2vec 2.0 with masked prediction in the style of BERT, and took k-means units from an intermediate layer. AudioLM's ablations showed that these semantic tokens were far better than codec tokens at producing linguistically coherent continuations but poor at reconstructing the speaker's voice when decoded on their own. That observation, that one token type cannot do both jobs well, became the founding principle of the two-stage architectures that followed. Supervised semantic tokens Self-supervised units have an awkward property: they are optimised for masked prediction, not for any particular downstream purpose, and the cluster structure that emerges from k-means is not guaranteed to align with linguistic categories. An alternative is to derive tokens from a model trained explicitly for recognition. If an encoder has been trained to produce transcripts, its internal representations are by construction aligned with linguistic content. Alibaba's CosyVoice, published in 2024, adopted this idea. It inserts a vector quantisation layer into the encoder of a multilingual speech recogniser and trains the whole system for recognition, so that the quantised codes must carry whatever the recogniser needs to transcribe. The resulting supervised semantic tokens run at 25 per second in the original system. CosyVoice's authors reported that these tokens gave better content consistency and speaker similarity in zero-shot synthesis than unsupervised alternatives. The trade-off is that the tokeniser now inherits the recogniser's biases: it represents what the recogniser was trained to hear, in the languages it was trained on, and may discard paralinguistic information that a self-supervised encoder would have retained, such as laughter or hesitation sounds that the transcripts did not mark. Other systems have taken intermediate positions, for example by training a codec whose first quantiser is encouraged to match the representations of a self-supervised encoder, which is the approach of SpeechTokenizer and of Kyutai's Mimi codec. These hybrid designs are discussed in the next chapter, because they belong as much to the codec family as to the semantic one. Evaluating a semantic tokeniser There is no single accepted measure of a semantic tokeniser's quality, but several are in common use, and understanding them clarifies what the tokens are for. Phone purity and phone-normalised mutual information measure how closely units align with phoneme labels on a forced-aligned dataset. They reward units that behave like a phonetic transcription. ABX discrimination, a measure developed for the Zero Resource Speech Challenge series, asks whether the representation of a sound is closer to another instance of the same sound than to an instance of a minimally different sound, across speakers and contexts. It rewards representations that are phonetically discriminative and speaker-invariant. Resynthesis intelligibility measures how understandable speech is when units are turned back into audio by a vocoder, typically by transcribing the resynthesised speech with a strong recogniser and computing word error rate. It rewards units that preserve enough information to reconstruct intelligible speech. Speaker leakage measures how well a classifier can identify the speaker from the units, and lower is better if the goal is disentanglement. Downstream language-modelling quality, measured with benchmarks such as sWUGGY, which tests whether a spoken language model prefers real words over phonotactically plausible non-words, and sBLIMP, which tests grammatical preferences, measures how useful the units are as a vocabulary for spoken language modelling. These measures do not always agree. A tokeniser with more clusters may score better on resynthesis intelligibility and worse on speaker leakage. A tokeniser from a deeper layer may score better on ABX and worse on resynthesis. The right choice depends on what the tokens are for, which is the general lesson of this booklet in miniature. The limits of the semantic approach Semantic tokens solved one problem, making spoken content tractable for language models, and in doing so exposed others. Because they discard much of the acoustic detail, any system built on them must reconstruct that detail somewhere else, and the reconstruction stage becomes the bottleneck for quality. Because they run at fifty frames per second before deduplication, they impose a heavy sequence-length burden on language models. Because they are derived from encoders trained predominantly on English or on a limited set of languages, their quality degrades for languages and accents underrepresented in pretraining. And because they partially suppress prosody, systems that rely on them alone tend to produce speech whose intonation is plausible but generic. The response to these limits has not been to abandon semantic tokens but to combine them with a second representation that carries what they discard. That second representation is the neural codec token, and its design involves a different set of engineering trade-offs. Chapter 4. Neural Codecs and Acoustic Tokens A neural audio codec is a learned compressor. An encoder network turns a waveform into a sequence of continuous vectors at a reduced frame rate; a quantiser maps each vector to one or more entries in learned codebooks; a decoder network turns the sequence of codebook entries back into a waveform. The whole system is trained end to end so that the reconstruction sounds as close as possible to the original at a target bitrate. The codebook indices are the acoustic tokens. Unlike semantic units, they were not designed to expose linguistic structure; they were designed to reconstruct sound. That they can also serve as the vocabulary of a generative model is a consequence of their being discrete, compact and complete. Residual vector quantisation The core technical problem in a neural codec is how to quantise a high-dimensional continuous vector into a small number of bits without losing too much. The naive approach, a single codebook, scales badly: to spend 60 bits per frame with one codebook would need two to the sixtieth power entries, which is absurd. The solution adopted by Google's SoundStream, published in 2021, and by nearly every codec since, is residual vector quantisation. Residual vector quantisation uses a stack of codebooks, each of modest size, typically 1,024 entries. The first codebook quantises the encoder's output vector to its nearest entry. The difference between the original vector and that entry, the residual, is passed to the second codebook, which quantises it in turn. The residual of that step goes to the third, and so on. The decoder sums the selected entries from all codebooks to approximate the original. With eight codebooks of 1,024 entries, each frame is described by eight indices of ten bits each, eighty bits in total, using codebooks with only 8,192 entries altogether. The structure produces a natural coarse-to-fine hierarchy. The first codebook captures the largest components of the encoder output, which tend to correspond to the broad spectral shape and much of the phonetic content. Successive codebooks refine the approximation with progressively finer detail. This hierarchy is what makes residual quantisation attractive for generative modelling: a model can generate the first codebook, which carries most of the perceptually important structure, with an expensive autoregressive process, and fill in the later codebooks more cheaply. SoundStream also introduced quantiser dropout. During training, the number of codebooks used is randomly truncated, so the decoder learns to reconstruct from any prefix of the codebook stack. A single trained model can then operate at multiple bitrates by simply using fewer or more codebooks. SoundStream operated at 24 kilohertz and was evaluated from 3 to 18 kilobits per second. At 3 kilobits per second, its authors reported that it outperformed the Opus codec at 12 kilobits per second in listening tests. Training uses a combination of losses. Reconstruction losses in the time and frequency domains keep the output close to the input. Adversarial losses, from discriminators that try to distinguish real from reconstructed audio, push the output toward perceptual realism rather than toward the blurry average that regression losses alone produce. Feature-matching losses compare the discriminators' internal activations for real and reconstructed audio. And commitment losses keep the encoder outputs close to their quantised values so that the codebooks remain useful. The adversarial component is essential; without it, codecs at low bitrates sound muffled, for the same reason that early neural vocoders did. EnCodec, DAC and the bitrate arithmetic Meta's EnCodec, published in 2022, refined the SoundStream design and became the most widely used codec in early generative speech work, largely because its code and weights were openly released. The 24 kilohertz EnCodec model produces 75 frames per second, each frame quantised with up to 32 codebooks of 1,024 entries. Each codebook therefore contributes 75 times 10, or 750, bits per second, and the model supports bitrates of 1.5, 3, 6, 12 and 24 kilobits per second by using 2, 4, 8, 16 or 32 codebooks. VALL-E used EnCodec at 6 kilobits per second, meaning eight codebooks, so each second of speech became 600 tokens arranged as 75 frames of eight. EnCodec also introduced a small Transformer language model over the codes for additional entropy coding, and a loss balancer that normalised gradients from the different loss terms so that their relative weights could be set intuitively. Descript's Audio Codec, usually called DAC and published by Kumar and colleagues in 2023, pushed fidelity further. Among its improvements were factorised and normalised codes, in which lookup happens in a low-dimensional projected space with normalised vectors, which improved codebook utilisation, a known problem in which many codebook entries are never used and the effective bitrate is lower than the nominal one. DAC also used the periodic "snake" activation function borrowed from the BigVGAN vocoder, improved discriminators and modified quantiser dropout. Its 44.1 kilohertz model compresses full-band audio to about 8 kilobits per second using nine codebooks at roughly 86 frames per second, and the authors reported that it outperformed EnCodec at comparable bitrates across speech, music and environmental sound. The arithmetic of these designs has a direct consequence for anything that generates codec tokens. The number of tokens a model must produce per second of audio is the frame rate multiplied by the number of codebooks it must model. At EnCodec's 75 frames per second and eight codebooks, that is 600 tokens per second. At DAC's 86 frames and nine codebooks, it is about 775. A text language model generating a typical reply might produce three or four subword tokens per second of speech it describes. A codec language model producing the same reply as audio must generate one to two orders of magnitude more tokens. Every architectural trick in codec-based synthesis, from the autoregressive and non-autoregressive split of VALL-E to the delay patterns of later models, is a way of managing that multiplication. Lower frame rates and fewer codebooks The pressure to reduce token counts has driven codec design since 2023 in two directions: lower frame rates and fewer codebooks. Lower frame rates trade temporal resolution for sequence length. Speech sounds last tens of milliseconds, and the linguistic message changes only ten to fifteen times per second, so there is no fundamental reason a codec frame must be shorter than about 40 to 80 milliseconds, provided the decoder can reconstruct the fine temporal structure within each frame. Kyutai's Mimi codec, released in 2024 alongside the Moshi dialogue model, runs at 12.5 frames per second at 24 kilohertz, each frame representing 80 milliseconds of audio. With eight codebooks of 2,048 entries, its bitrate is 12.5 times 8 times 11, or 1,100 bits per second, and it produces 100 tokens per second rather than EnCodec's 600 at a comparable codebook count. Mimi uses Transformer layers in both encoder and decoder in addition to convolutions, which helps the decoder reconstruct detail over the longer frames. Fewer codebooks trade bitrate for simplicity of modelling. A single-codebook codec produces one token per frame, so a standard autoregressive language model can generate it without any special handling of the codebook dimension. WavTokenizer, published in 2024, compresses 24 kilohertz audio into a single codebook at 40 or 75 tokens per second, using a large codebook and an attention-augmented decoder. BigCodec, also from 2024, targets very low bitrates around a kilobit per second with a single large codebook. These codecs sacrifice some fidelity relative to multi-codebook designs at the same bitrate, but they simplify generation enough that several recent synthesis systems have adopted them. A related development is finite scalar quantisation, introduced in image modelling and adopted in some audio codecs, which replaces learned codebooks with a fixed grid: each dimension of a low-dimensional projection is rounded to one of a few levels, and the code is the tuple of levels. This avoids codebook collapse entirely, since every grid point is reachable, and has proved competitive with learned residual quantisation in several systems. Spark-TTS and some versions of the CosyVoice family use variants of it. Table 1 sets the main codecs discussed here side by side on the dimensions that matter most for generative use. Table 1. Representative neural codecs and their token budgets. Codec (year) Sample rate Frame rate Codebooks used Bitrate Tokens per second SoundStream (2021) 24 kHz 75 Hz up to 24 3 to 18 kbps up to 1,800 EnCodec (2022), as in VALL-E 24 kHz 75 Hz 8 of 32 6 kbps 600 DAC (2023) 44.1 kHz about 86 Hz 9 about 8 kbps about 775 SpeechTokenizer (2023) 16 kHz 50 Hz 8 4 kbps 400 Mimi (2024) 24 kHz 12.5 Hz 8 1.1 kbps 100 WavTokenizer (2024) 24 kHz 40 or 75 Hz 1 0.5 or 0.9 kbps 40 or 75 Source: the published papers and model cards for each codec. Figures are for the configurations most often used in speech generation; several codecs support others. The trend in the table is clear: from hundreds of tokens per second toward tens, and from many parallel codebooks toward few or one. The cost is paid in reconstruction fidelity at the high end of the frequency range and in the burden placed on the decoder, which must synthesise more detail from less information. That burden is the point at which codec decoders and neural vocoders become, in effect, the same component. Semantic distillation: making codec tokens readable Acoustic tokens from a codec trained purely for reconstruction are hard for a language model to learn. Their first codebook captures whatever large-scale structure was most useful for reconstruction, which correlates with phonetic content but also with speaker, pitch and channel. A model trained to generate them from text must learn the linguistic mapping through this tangle. SpeechTokenizer, published in 2023, proposed a direct remedy: train the codec with an additional loss that encourages the first residual codebook to match the representations of a HuBERT teacher, while the remaining codebooks carry the residual acoustic detail. The first layer then behaves like a semantic token stream, and the rest like acoustic tokens. A synthesis system can generate the first layer from text with an autoregressive model, which handles content and timing, and the remaining layers with a non-autoregressive model, which handles voice and detail. The authors reported that this split improved the content accuracy of a VALL-E-style system and that the first layer carried substantially less speaker information than in an undistilled codec. Mimi adopted the same principle, distilling representations from WavLM into its first codebook, but with an important difference: because Mimi is designed for streaming, its encoder is causal, while the WavLM teacher is not. The distillation target is a non-causal representation, and the codec learns to approximate it causally. Mimi's authors also reported that training with adversarial and feature-matching losses alone, dropping the usual reconstruction loss, improved perceived quality even though objective reconstruction measures got worse, a reminder that perceptual realism and sample-level fidelity are different goals. Microsoft's NaturalSpeech 3, published in 2024, took factorisation further with its FACodec, which uses separate quantisers for content, prosody, timbre and acoustic detail, trained with supervision and adversarial disentanglement objectives so that each subspace carries its intended attribute. Its aim was to let a synthesis model generate each attribute separately, with timbre supplied from a reference. Whether such explicit factorisation outperforms the simpler semantic-first split in practice remains a matter of active comparison; what is clear is that the field has converged on the idea that the tokens should be organised so that content sits in an identifiable place. Modelling multiple codebooks Even with semantic distillation and lower frame rates, a codec with several codebooks produces a two-dimensional grid of tokens: time along one axis, codebook depth along the other. A language model must generate this grid, and there are several ways to arrange it. The simplest is to flatten it: generate all codebooks for frame one, then all for frame two, and so on. This preserves full dependency structure but multiplies the sequence length by the number of codebooks. With eight codebooks at 75 frames per second, a ten-second utterance becomes 6,000 tokens, which is expensive to generate and stretches a model's context. VALL-E split the problem. An autoregressive Transformer generated the first codebook frame by frame, conditioned on the text and the prompt, and a non-autoregressive Transformer then generated each subsequent codebook in parallel across all frames, conditioned on the previous codebooks. This captured the sequential, content-bearing structure with an autoregressive model and the residual detail with a cheaper parallel one. Meta's MusicGen, though designed for music, introduced a delay pattern widely adopted for speech: codebook k at frame t is generated at step t plus k, so that at each step the model emits one token per codebook, but each codebook is offset in time. This keeps the number of autoregressive steps close to the number of frames while allowing each codebook to condition on the coarser ones of the same frame. Moshi used a two-level Transformer: a large temporal Transformer runs once per frame and produces a context vector, and a small depth Transformer runs over the codebooks within that frame, generating them sequentially. The large model's cost scales with the frame rate, and the small model's cost scales with the number of codebooks, which is much cheaper than running the large model for every codebook. Several later systems, including Sesame's Conversational Speech Model, adopted a similar split. Each arrangement embodies a judgement about which dependencies matter. Content and timing need the full sequential context. Fine acoustic detail mostly depends on the coarse tokens of the same frame and on nearby frames. Matching the model's structure to the token structure is one of the main levers for making codec language models both fast and accurate. What codec tokens get wrong Codec tokens carry everything, and that is both their strength and their weakness. A codec language model prompted with three seconds of a speaker in a reverberant room will continue in that room. Prompted with a noisy telephone recording, it will continue with telephone noise. This is ideal for faithful continuation and unhelpful for producing clean output from a poor reference, which is the common case when cloning a voice from real-world recordings. Codec tokens also carry the source's accent and speaking style intimately, because those are part of the acoustic pattern. In cross-lingual transfer this produces accent leakage, discussed in a later chapter. And codec tokens are brittle under generation in a particular way. Because each token conditions everything after it, a single bad token early in an autoregressive sequence can push the model into a region it never saw during training, producing a stretch of mumbled or repeated speech. Text language models have the same vulnerability, but text has a structure that tends to pull generation back on course; acoustic tokens, especially from codecs without semantic distillation, have less. Much of the engineering of codec-based synthesis, from sampling strategies to alignment constraints to the move toward non-autoregressive generation, is aimed at controlling this brittleness. The broad lesson is that codec tokens are a medium, not a model. They make high-fidelity generation possible by giving language models a vocabulary for sound, but they do not by themselves organise that vocabulary in a way that makes speech easy to model. The organising work is done by semantic distillation, by hierarchical generation, and by the architectural choices of the models that read and write them. Chapter 5. Transcription End to End Automatic speech recognition is the oldest and most commercially mature part of the speech stack, and it is also the part where foundation models have had the most measurable effect. A modern recogniser is a single network that reads acoustic features and writes text, trained on hundreds of thousands to millions of hours of audio. It is used directly for dictation, captioning, meeting notes and voice search, and indirectly as the ear of every conversational system and the judge of every synthesis evaluation. Its design involves three main decisions: how the encoder represents audio, how the decoder turns that representation into text, and how the system handles the trade-off between accuracy and latency. Encoders: from recurrent networks to Conformers The encoder's job is to turn a sequence of acoustic feature frames into a sequence of vectors that make the linguistic content easy to read off. Early end-to-end systems used stacks of bidirectional recurrent networks, typically long short-term memory layers. These were superseded first by Transformers, which handle long-range context more efficiently, and then, from 2020, by the Conformer, introduced by Gulati and colleagues at Google. The Conformer interleaves self-attention, which captures global context across the utterance, with convolution modules, which capture local patterns such as the spectral transitions within and between phonemes. Each block sandwiches a multi-head self-attention module and a convolution module between two half-step feed-forward modules. The combination proved consistently better than either pure Transformers or pure convolutional networks for speech, and Conformer variants now form the encoder of most production recognisers and many research systems. Google's Universal Speech Model, NVIDIA's Parakeet and Canary families, and many open-source toolkits are built on Conformer or closely related designs such as the FastConformer, which downsamples more aggressively at the input to reduce computation. Before the encoder proper, nearly every system reduces the frame rate. Feature frames arrive every ten milliseconds, a rate that is wasteful for an encoder with quadratic attention cost. A convolutional subsampling stage typically reduces the rate by a factor of four to eight, to 40 or 80 milliseconds per encoder frame. Whisper uses two convolutional layers, one with stride two, to bring its 100-per-second mel frames down to 50 per second, so a 30-second input window becomes 1,500 encoder positions. NVIDIA's FastConformer subsamples by eight, to 80 milliseconds. The choice affects both cost and the granularity of alignment: a decoder cannot place a word boundary more precisely than one encoder frame. Increasingly, the encoder is initialised from a self-supervised model rather than trained from scratch. A wav2vec 2.0, HuBERT, WavLM or BEST-RQ encoder pretrained on unlabelled audio can be fine-tuned with a recognition head, and for low-resource languages this initialisation is often the difference between a working system and a failed one. Google's USM used BEST-RQ, a pretraining method that predicts the indices of a random, frozen projection-and-codebook quantiser applied to masked spectrogram frames, which avoids the need to learn the quantiser and scaled to its twelve-million-hour pretraining set. Decoders: CTC, attention and transducers Three decoder families dominate, and each makes a different trade between simplicity, accuracy and streaming suitability. Connectionist temporal classification places a linear layer on top of the encoder that, at every frame, outputs a probability distribution over the output vocabulary plus a blank symbol. A transcript is obtained by taking the most likely symbol at each frame, collapsing repeats, and removing blanks. Training maximises the total probability of all frame-level paths that collapse to the correct transcript, computed efficiently with dynamic programming. CTC's great virtue is its simplicity and speed: decoding is a single pass with no autoregressive loop, and it naturally supports streaming if the encoder does. Its weakness is a conditional independence assumption: the output at each frame does not depend on outputs at other frames, so the model cannot learn spelling or grammar dependencies beyond what the encoder captures. In practice, large CTC models with strong encoders compensate well, and external language models can be fused at decoding time to correct residual errors. Meta's MMS recognisers and many fine-tuned wav2vec 2.0 models use CTC. Attention-based encoder-decoder models add an autoregressive decoder that generates the transcript one token at a time, attending at each step to the full encoder output. Because each output conditions on all previous outputs, the decoder learns an internal language model, and these models typically achieve the best accuracy on offline tasks. Whisper is an attention encoder-decoder, as are many systems that integrate a large text language model as the decoder. The weaknesses are the cost of autoregressive decoding, the difficulty of streaming, since attention over the whole encoder output requires the whole utterance, and a characteristic failure mode: the decoder can lose its place in the audio, producing repeated phrases or text that has no basis in the input. This failure, usually called hallucination, is discussed below. The recurrent neural network transducer, or RNN-T, adds a prediction network that conditions on previous non-blank outputs, and a joint network that combines the prediction network's state with the encoder's state at each frame to produce a distribution over the vocabulary plus blank. The model can emit zero or more tokens at each encoder frame. Training sums over all alignments, like CTC, but the outputs are no longer conditionally independent. The transducer combines CTC's natural alignment, which makes streaming straightforward, with some of the attention decoder's ability to model output dependencies. Google moved its on-device recognition to RNN-T models around 2019, and transducers remain the most common choice for streaming production systems. The token-and-duration transducer, a variant introduced by NVIDIA, predicts how many frames to skip after each emission, which substantially speeds up inference. Hybrid training, in which a single encoder is trained with both CTC and attention losses, is common. The CTC loss encourages monotonic alignment, which stabilises the attention decoder and reduces hallucination, and the CTC branch can be used to rescore or constrain attention-based hypotheses. Whisper and the multitask format Whisper deserves separate attention because it established the template for general-purpose recognition. Its architecture is unremarkable: a standard Transformer encoder and decoder, operating on 30-second windows of log-mel spectrogram. What made it distinctive was the training data, 680,000 hours drawn from the internet, and the multitask format. Every Whisper training example begins with a short sequence of special tokens: a start-of-transcript marker, a language tag, a task tag specifying transcription or translation into English, and a flag indicating whether timestamps should be predicted. Timestamp tokens, quantised to twenty milliseconds, are interleaved with text when requested. A previous-text token allows the preceding transcript to be supplied as context. At inference, these tokens act as instructions, and the same model performs language identification, transcription in about a hundred languages, translation to English, and coarse word timing. The weak supervision had consequences beyond scale. Because Whisper was trained on transcripts written by people for people, often with punctuation, capitalisation and inverse text normalisation already applied, it produces readable output directly: "$5.00" rather than "five dollars", proper sentence breaks, sensible paragraphing. This is what users want, and it is also a nuisance for evaluation, because standard word error rate requires normalising the reference and hypothesis to a common form. Whisper's authors released a text normaliser for exactly this reason, and comparisons between recognisers are only meaningful when normalisation is consistent. Later versions extended the data. Whisper large-v3, released in November 2023, was trained on one million hours of weakly labelled audio and four million hours of audio pseudo-labelled by the earlier large-v2 model, and moved from 80 to 128 mel channels. Whisper large-v3-turbo, released in 2024, reduced the decoder from 32 layers to 4, making decoding several times faster with a modest loss of accuracy, a choice that reflects where the computation in an encoder-decoder recogniser actually goes: the encoder runs once per window, but the decoder runs once per token. Hallucination and other failure modes Large encoder-decoder recognisers fail in a way that classical systems rarely did. Given silence, music, noise or speech in an unexpected language, the decoder may generate fluent text with no basis in the audio. Common patterns include repeating the last phrase several times, inserting stock phrases associated with video subtitles in the training data such as thanks for watching, or continuing a plausible sentence past the point where the speaker stopped. These hallucinations are dangerous precisely because they are fluent: a reader of the transcript has no cue that the text was invented. Studies of Whisper transcriptions of medical and conversational audio have documented hallucinated content, including in some cases entire invented sentences, and the risk is highest in recordings with long pauses, as in speech from people with aphasia. Several mitigations are standard. Voice activity detection before recognition removes silent and non-speech segments so that the decoder is never asked to transcribe nothing. Constraining the decoder with a CTC alignment or with timestamp consistency rules prevents it from drifting. Thresholds on the decoder's average log-probability and on the compression ratio of the output, which is high when text repeats, flag likely failures for retry with different decoding settings. Whisper's reference implementation applies temperature fallback: if the output at temperature zero fails these checks, decoding is retried at progressively higher temperatures. None of these is a complete solution; the underlying cause is that an autoregressive decoder with a strong internal language model can produce probable text without strong evidence, and only architectures with tighter coupling between output and audio, such as CTC and transducers, avoid it by construction. Other failure modes are more familiar. Accuracy drops on accents, dialects and speaking styles underrepresented in training data, and the drop is not uniform: a 2020 study by Koenecke and colleagues found that five commercial recognisers of the time produced substantially higher error rates for Black American speakers than for white American speakers on comparable conversational speech. Proper nouns, technical vocabulary and code-switched speech remain difficult. Overlapping speakers degrade any single-speaker recogniser, which is why meeting transcription systems combine recognition with diarisation, the task of determining who spoke when. Measuring accuracy Word error rate is the standard metric. It counts the substitutions, deletions and insertions needed to turn the hypothesis into the reference, divided by the number of reference words. It is simple and widely reported, and it has well-known weaknesses. It weights all words equally, so missing "not" counts the same as missing "the". It depends heavily on normalisation, as already noted. It is undefined for languages without word boundaries such as Chinese, Japanese and Thai, where character error rate is used instead. And it can exceed 100 percent if the hypothesis contains many insertions, which hallucinating models occasionally manage. Benchmark results should be read with these caveats. LibriSpeech, built from public-domain audiobooks, remains the most cited English benchmark, but its clean read speech is far easier than conversation, and top systems now score around two percent on its clean test set, close to the noise floor of its reference transcripts. More informative are aggregate benchmarks across diverse conditions. The Hugging Face Open ASR Leaderboard averages error rates across several English datasets covering read, spontaneous, accented and telephone speech, and also reports inverse real-time factor, a measure of speed. For multilingual evaluation, FLEURS, a dataset of the same sentences read in about a hundred languages, and Common Voice, a crowdsourced corpus, are standard. Table 2 sets out the main families of current recognisers on the dimensions a practitioner usually cares about. Table 2. Families of contemporary speech recognisers. Family Representative systems Decoder Streaming suitability Characteristic strength Characteristic weakness Weakly supervised encoder-decoder Whisper, Canary Attention Poor without adaptation Robust, readable, multilingual Hallucination on silence and noise SSL encoder with CTC head MMS, fine-tuned wav2vec 2.0 CTC Good with causal encoder Very wide language coverage, fast Weaker output dependencies, needs LM fusion Conformer transducer Parakeet-TDT, on-device assistants RNN-T or TDT Excellent Low latency, no hallucination by design Less natural formatting, harder to train LLM-integrated Encoder plus text LLM decoder Attention via LLM Limited Strong context, instruction following Cost, latency, inherited hallucination Source: published descriptions of the named systems. Integrating large language models The newest family connects a speech encoder to a pretrained text language model. A projection layer or small adapter maps encoder outputs into the language model's embedding space, and the combined system is trained, often with the language model frozen or lightly tuned, to produce transcripts or answer questions about the audio. The language model contributes vast knowledge of spelling, names, grammar and context, which helps with rare words and long-form coherence, and it allows recognition to be steered with instructions: transcribe this, but spell the product names as follows; transcribe and also note the speaker's emotion. The approach has costs. A large language model decoder is far more expensive than a CTC head or transducer prediction network, which matters for both server costs and latency. It inherits the hallucination tendencies of attention decoders, sometimes in amplified form, because a strong text prior makes invented content more fluent. And it raises questions about what the encoder should pass to the language model: continuous embeddings at the encoder's frame rate, which are informationally rich but long, or compressed or discretised representations, which are shorter but may lose detail. This is the same representation question that runs through the rest of the field, now arising at the joint between a speech encoder and a text model. Streaming recognition A recogniser in a conversational system must produce text while the user is still speaking, both to show partial results and to begin downstream processing early. Streaming imposes two constraints. The encoder cannot look arbitrarily far into the future, so it must operate on chunks of audio with limited lookahead. And the decoder must commit to outputs before the utterance is complete. Transducers and CTC models handle the second constraint naturally, because they align outputs to encoder frames. The first is handled by restricting the encoder's attention to past frames plus a small, fixed lookahead, typically a few hundred milliseconds, or by processing audio in chunks with attention allowed within each chunk and to a cache of previous chunks. Each design trades latency against accuracy: more lookahead means better accuracy but slower emission. Many production systems run in two passes, a streaming first pass that produces fast partial results and a non-streaming second pass that rescores or corrects them once the utterance ends. The most important streaming decision, however, is not inside the recogniser but around it: deciding when the user has finished speaking. This is endpointing, and it dominates perceived latency in conversational systems. A recogniser that transcribes perfectly but waits a full second of silence before declaring the turn over will feel sluggish. One that declares the turn over at the first pause will interrupt users who were merely thinking. The chapter on latency returns to this, because it is one of the clearest cases where the fastest component in the pipeline is not the one that determines how fast the system feels. Recognition as infrastructure A final point about recognition bears on everything that follows. Because strong recognisers are now cheap and widely available, they have become measuring instruments for the rest of the field. Synthesis systems are evaluated by transcribing their output and computing word error rate against the input text; voice conversion and translation systems are evaluated the same way. Recognisers also label training data: Whisper large-v3's pseudo-labelled hours are one example, and many synthesis datasets are built by transcribing large audio collections automatically. Both uses inherit the recogniser's biases. A synthesis system that produces speech in an accent the evaluating recogniser handles poorly will appear less intelligible than it is. A dataset transcribed by a model that hallucinates in long silences will contain hallucinated transcripts, which a synthesis model will then learn. The recogniser is no longer only an application; it is part of the field's infrastructure, and its errors propagate. Chapter 6. Neural Vocoders and the Last Mile Every synthesis system ends with a component that produces a waveform. In classical systems this was a signal-processing vocoder that rendered pitch and spectral envelope parameters through a source-filter model. In the neural era it is a network that takes some intermediate representation, a mel spectrogram, a set of semantic units or a stack of codec tokens, and generates audio samples. The term vocoder has survived the transition, though modern neural vocoders bear little resemblance to their ancestors. They matter disproportionately to perceived quality, because whatever the upstream model gets right, the vocoder can still ruin with a metallic buzz, a smeared consonant or a pitch artefact, and listeners notice these failures immediately. Why the problem is hard A mel spectrogram is a lossy description of sound. It records the energy in each of about eighty frequency bands every ten milliseconds or so, and it discards phase entirely. To turn it back into a waveform, the vocoder must invent a phase for every frequency component in every frame, consistent with its neighbours, so that the reconstructed signal is smooth and natural. It must also fill in the fine spectral detail between mel bands and the sample-level texture of noise components such as frication and breath. Classical approaches to phase reconstruction, notably the Griffin-Lim algorithm, iteratively estimate a phase consistent with a given magnitude spectrogram. Griffin-Lim works, in the sense that its output is intelligible, but it produces a characteristic phasey, robotic quality, because many different phase configurations are consistent with the same magnitudes and the algorithm has no knowledge of which ones sound natural. The neural insight was that the space of natural-sounding speech waveforms is far smaller than the space of waveforms consistent with a given spectrogram, and a network trained on real speech can learn where the natural ones lie. The other difficulty is rate. A spectrogram frame describes about 256 samples at 22 or 24 kilohertz. A vocoder must upsample by that factor, generating hundreds of samples for each input frame, while keeping the output consistent with periodic structure, such as the harmonics of a voiced sound, that spans many frames. Speech is also highly sensitive to errors in periodicity. A slight inconsistency in the period of a voiced sound from cycle to cycle is perceived as roughness or buzz; a discontinuity at a frame boundary is heard as a click. Autoregressive vocoders WaveNet, introduced by DeepMind in 2016, was the first neural model to generate convincing raw speech. It predicts each audio sample from all previous samples, using a stack of dilated causal convolutions whose receptive field doubles with each layer, so that a modest number of layers can see thousands of samples into the past. The original model quantised samples to 256 levels with mu-law companding and predicted a categorical distribution over them; later variants used mixtures of logistic distributions to model continuous values. Conditioned on linguistic features or mel spectrograms, WaveNet produced speech that listeners rated far closer to natural recordings than any previous system. It was also extremely slow, because generating one second of 24 kilohertz audio meant 24,000 sequential forward passes through a deep network. DeepMind's Parallel WaveNet, published in 2017, solved this for deployment by distilling the autoregressive model into a parallel student based on inverse autoregressive flows, trained to match the teacher's output distribution. The student could generate all samples at once and was deployed in Google Assistant. The distillation procedure was intricate, however, and the field looked for simpler routes. WaveRNN, published by DeepMind in 2018, took a different route to speed: a single-layer recurrent network, heavily optimised and sparsified, that could generate samples in real time on a mobile processor. LPCNet, published by Jean-Marc Valin and Jan Skoglund in 2019, combined a WaveRNN-style network with classical linear prediction: a linear predictor estimates each sample from previous ones using coefficients derived from the spectral envelope, and the network models only the prediction residual. Because the residual is much simpler than the full signal, the network can be small, and LPCNet ran in real time on a single phone core. Its hybrid design anticipated a recurring theme: whenever signal-processing knowledge can be built into the vocoder's structure, the learned part can be smaller and more reliable. Flows, GANs and the move to parallel generation The next generation of vocoders abandoned sample-by-sample generation entirely. Two approaches emerged around 2019. Flow-based vocoders, notably NVIDIA's WaveGlow, model the waveform with an invertible network that transforms simple Gaussian noise into audio, conditioned on the spectrogram. Training maximises the exact likelihood of real audio, and generation is a single parallel pass. WaveGlow produced high-quality output but required a large network, around 88 million parameters, to model the complex distribution of waveforms with invertible transformations. Generative adversarial vocoders proved more efficient and became dominant. A generator network upsamples the spectrogram to a waveform through a series of transposed convolutions interleaved with residual blocks, and one or more discriminator networks are trained to distinguish generated audio from real. The generator learns to fool the discriminators, which pushes its output toward the distribution of natural speech rather than toward an average. MelGAN, published in 2019, showed that a compact GAN could vocode faster than real time on a CPU. Parallel WaveGAN, from 2020, combined adversarial training with a multi-resolution short-time Fourier transform loss, which compares the spectrograms of real and generated audio at several window sizes and stabilises training. HiFi-GAN, published by Jungil Kong, Jaehyeon Kim and Jaekyoung Bae in 2020, became the reference design for the next several years. Its key innovation was the multi-period discriminator, a set of sub-discriminators each of which reshapes the one-dimensional waveform into a two-dimensional array by folding it at a fixed period, using prime periods such as 2, 3, 5, 7 and 11 samples. Each sub-discriminator then examines the audio for periodic structure at its period. Because voiced speech is made of periodic components, this gives the discriminators a direct way to penalise inconsistent periodicity, which was the main source of buzzy artefacts in earlier GAN vocoders. HiFi-GAN combined this with a multi-scale discriminator operating on progressively downsampled audio and with a mel-spectrogram reconstruction loss. Its largest configuration reported quality close to that of the ground-truth recordings in listening tests on a single-speaker dataset, while generating audio many times faster than real time on a GPU. HiFi-GAN and its variants became the default vocoder in research and open-source systems, and its discriminators were adopted by neural codecs. Universal vocoders Early neural vocoders were trained on a single speaker and degraded on others. For zero-shot synthesis, where the target voice is unseen, a vocoder must generalise to any speaker, any language and ideally any recording condition. This is the universal vocoder problem. BigVGAN, published by NVIDIA researchers in 2022 and presented at ICLR in 2023, addressed it with two main changes to the HiFi-GAN design. First, it replaced the leaky ReLU activations in the generator with a periodic activation called snake, defined as x plus the square of the sine of a learned frequency times x, divided by that frequency. Snake gives the network an inductive bias toward periodic outputs, which helps it extrapolate to pitches and harmonics not seen in training. Second, because periodic nonlinearities generate high-frequency components that can alias when the signal is downsampled, BigVGAN applied anti-aliasing filters around each activation. Trained on a diverse dataset and scaled to over a hundred million parameters, BigVGAN generalised notably better than HiFi-GAN to unseen speakers, languages, singing and even some non-speech audio. Its snake activation was subsequently adopted by the DAC codec, and BigVGAN variants remain among the most widely used universal vocoders. Vocoding in the frequency domain A different line of work questioned why vocoders should generate waveforms directly at all. Upsampling from frame rate to sample rate through stacks of transposed convolutions is computationally expensive and prone to checkerboard artefacts. An alternative is to predict the short-time Fourier coefficients, magnitude and phase, at the frame rate, and to convert them to a waveform with the inverse short-time Fourier transform, a fast, deterministic operation that does the upsampling for free. Vocos, published by Hubert Siuzdak in 2023, adopted this design. Its backbone operates entirely at the frame rate, with no upsampling layers, and outputs a magnitude and a phase for each frequency bin; phase is represented in a way that respects its periodic nature. The inverse transform produces the waveform. Vocos reported quality comparable to HiFi-GAN and BigVGAN at a fraction of the computational cost, and because the expensive part of the network runs at the frame rate rather than the sample rate, it is particularly attractive for low-latency synthesis. Vocos has also been used as the decoder in codec designs, including WavTokenizer, that reconstruct waveforms from tokens. The frequency-domain approach reflects a broader rethinking. The difficulty of phase was one of the original reasons for discarding it in the mel spectrogram. Once networks could model phase well enough, it made sense to have them predict it directly at the frame rate rather than to infer it implicitly through sample-level generation. Diffusion and flow-matching vocoders Denoising diffusion models, which learn to generate data by reversing a gradual noising process, entered vocoding in 2020 with WaveGrad and DiffWave. They produce high-quality audio and are straightforward to train, since they need no adversarial component, but they require many iterative denoising steps at inference, making them slower than GAN vocoders. Subsequent work reduced the step count substantially, and diffusion and flow-matching decoders became common in a slightly different role: not as vocoders from mel spectrograms but as decoders that generate mel spectrograms or continuous latents from coarse token sequences, after which a fast GAN vocoder renders the waveform. This is the arrangement in CosyVoice, where a flow-matching model maps semantic tokens and a speaker embedding to mel spectrograms and a HiFi-GAN vocoder produces audio, and in several other two-stage systems. Table 3 compares the vocoder families on the properties that determine where each is used. Table 3. Neural vocoder families compared. Family Examples Generation Speed Typical role today Autoregressive WaveNet, WaveRNN, LPCNet Sample by sample Slow, except small hybrids Low-power devices (LPCNet), historical reference Flow WaveGlow Parallel, invertible Fast but large Largely superseded GAN, time domain HiFi-GAN, BigVGAN Parallel upsampling Very fast Default mel-to-waveform vocoder, codec decoders GAN, frequency domain Vocos Frame-rate STFT prediction Fastest Low-latency synthesis, codec decoders Diffusion or flow matching DiffWave, WaveGrad, CosyVoice decoder Iterative denoising Slower, step-dependent Token-to-spectrogram decoders before a GAN vocoder Source: published descriptions of the named systems. The convergence of codec decoders and vocoders A neural codec's decoder takes a sequence of quantised vectors at frame rate and generates a waveform. A neural vocoder takes a sequence of spectrogram frames at frame rate and generates a waveform. Architecturally, these are the same problem, and the solutions have converged. Codec decoders use the same upsampling generators, the same snake activations, the same multi-period and multi-resolution discriminators as GAN vocoders. Some codecs, such as WavTokenizer, use a Vocos-style frequency-domain decoder. This convergence has practical consequences. In a system that generates codec tokens directly, such as VALL-E or Moshi, the codec decoder is the vocoder, and there is no separate mel spectrogram stage. In a system that generates semantic tokens and then decodes them to a mel spectrogram, as CosyVoice does, a conventional vocoder is still needed. In both cases, the last stage of the pipeline is a GAN-trained upsampling network, and its properties, including its artefacts, its generalisation to unseen voices and its latency, are largely shared. It also explains why codec bitrate and vocoder quality are intertwined. A low-bitrate codec places a heavier burden on its decoder, which must synthesise more detail from less information. At Mimi's 1.1 kilobits per second, the decoder is in effect a generative model, inventing plausible high-frequency detail that the tokens do not specify. The perceptual quality of such a decoder depends on how well its adversarial training has taught it what natural detail sounds like, and its failures look like vocoder failures: a slight metallic sheen, smeared transients, imprecise sibilants. Characteristic artefacts Anyone who listens to a great deal of synthetic speech learns to recognise vocoder artefacts, and they are worth describing, because they are the most common reason a technically correct synthesis system sounds wrong. Buzziness or metallic timbre arises from inconsistent periodicity: harmonics that do not quite line up from cycle to cycle. Multi-period discriminators and snake activations were developed largely to suppress it, and it is now rare in well-trained vocoders on in-domain speech, but it reappears on unusual voices, very high or very low pitches, and singing. Smearing of transients, such as the burst of a plosive or the onset of a fricative, arises when the vocoder's temporal resolution is too coarse or when the upstream model has blurred the spectrogram. Low-frame-rate codecs are particularly vulnerable. Noise-floor problems include hiss in what should be silence, or the opposite, an unnaturally dead silence that sounds gated. Both arise from the vocoder's learned model of background, and both are sensitive to the recording conditions of the training data. Pitch artefacts include sudden octave jumps or warbling in sustained vowels, typically when the upstream model produces spectrograms that are ambiguous about pitch. Aliasing produces faint high-frequency whistles or tones, particularly in vocoders with nonlinear activations and no anti-aliasing. Boundary clicks arise in streaming synthesis when audio is generated in chunks and the chunks do not join smoothly. This is a latency-related artefact and is discussed in the chapter on streaming. Vocoders under streaming constraints A conversational system wants to begin playing audio as soon as possible, which means the vocoder must produce output for the first few frames before later frames are available. GAN vocoders built from convolutions can do this if their convolutions are causal or if they are run on overlapping chunks with enough left context to avoid boundary artefacts. Non-causal convolutions, which look at future frames, add latency equal to their future receptive field. Frequency-domain vocoders like Vocos need one frame of lookahead for the inverse transform's overlap-add, which at typical hop sizes is a few tens of milliseconds. Codec decoders designed for streaming, such as Mimi's, are built causal throughout, so they can decode each 80-millisecond frame as soon as its tokens are available. Diffusion and flow-matching decoders are harder to stream, because each denoising step operates on a block of frames, and they are typically run on chunks of a few hundred milliseconds with careful handling of the boundaries. CosyVoice 2, published in late 2024, introduced chunk-aware causal flow matching specifically to make its decoder streamable. The vocoder, in other words, is not a neutral last step but one of the places where streaming latency is won or lost. Chapter 7. Synthesis as Language Modelling Text-to-speech has passed through three architectural generations in a decade. The first neural generation, exemplified by Tacotron 2, mapped characters to mel spectrograms with an attention-based sequence-to-sequence model. The second, exemplified by FastSpeech 2 and VITS, replaced attention with explicit duration prediction and made synthesis non-autoregressive and robust. The third, beginning with VALL-E in 2023, recast synthesis as language modelling over discrete audio tokens, and in doing so made zero-shot voice cloning a routine capability. The third generation has since diversified into autoregressive, non-autoregressive and hybrid forms, and the trade-offs among them are the central design questions in contemporary synthesis. The problems that language modelling solved The second-generation systems were excellent at what they were designed to do: produce natural speech in a voice they had been trained on, quickly and reliably. FastSpeech 2, published by Microsoft researchers in 2020, predicted the duration of each phoneme with a separate module, expanded the phoneme sequence accordingly, and generated the spectrogram in parallel, with additional predictors for pitch and energy. VITS, published by Jaehyeon Kim and colleagues in 2021, combined a variational autoencoder, normalising flows and adversarial training into a single model that went from text to waveform, learning alignment with a monotonic search. Both could be made multi-speaker by adding a speaker embedding, and with enough speakers in training they could generalise somewhat to new voices given an embedding extracted from a reference recording. But that generalisation was limited. A speaker embedding is a compact summary, a few hundred numbers, and it captures timbre and broad characteristics but loses much of what makes a voice distinctive: idiosyncratic prosody, speaking rate, the particular way a person reduces unstressed syllables. Models trained on a few hundred or a few thousand speakers learned a space of voices that was too narrow to cover the variety of real speakers. And the regression losses typically used to train spectrogram predictors produced averaged prosody: given many acceptable ways to say a sentence, the model learned to say it in the mean way, which sounds flat. Language modelling over audio tokens addressed both problems. Instead of summarising the reference voice in an embedding, a codec language model receives the reference audio itself as a prompt, as a sequence of tokens, and continues it. Everything about the reference that can be expressed in tokens, timbre, accent, speaking rate, recording conditions, emotional tone, is available to the model as context. And because the model predicts a distribution over discrete tokens and samples from it, it produces varied, natural prosody rather than an average. The cost of this approach is that the model must be trained on very large and diverse data, so that the space of voices it has seen is broad enough to contain almost any new speaker, and that sampling introduces new failure modes. VALL-E and the codec language model VALL-E, published by Microsoft in January 2023, made the approach concrete. It was trained on LibriLight, about 60,000 hours of English audiobook speech, transcribed automatically. Each training example was converted into a phoneme sequence for the text and an EnCodec token sequence at 6 kilobits per second for the audio, eight codebooks at 75 frames per second. At inference, the model received the phonemes of the reference transcript followed by the phonemes of the target text, along with the codec tokens of a three-second reference recording, and generated codec tokens for the target text. The generation was split in two. An autoregressive Transformer generated the first codebook, token by token, conditioned on all phonemes and on the first-codebook tokens of the prompt. This stage determined content, timing and the coarse acoustic shape, and because it was autoregressive it learned duration implicitly, choosing when to move on to the next phoneme. A non-autoregressive Transformer then generated codebooks two through eight, one codebook at a time but all frames in parallel, each conditioned on the phonemes, on the full prompt and on the previously generated codebooks. The result was decoded with the EnCodec decoder. VALL-E's evaluations reported substantially higher speaker similarity to the reference than the best previous zero-shot systems, and listeners judged its outputs as more similar to the reference speaker. It also reproduced the reference's acoustic environment, so that a prompt recorded on a telephone produced output that sounded like a telephone, and its emotional tone. Microsoft did not release the model, citing the risk of misuse, and subsequent versions, including VALL-E 2 in 2024, which its authors reported reached human parity on certain benchmarks, were likewise described in papers only. The weaknesses were as instructive as the strengths. VALL-E's autoregressive stage sometimes skipped words, repeated them, or got stuck in loops, producing long stretches of noise or silence. The word error rate of its output, measured by transcribing it, was higher than that of the non-autoregressive systems it outperformed on similarity. These robustness problems arise because nothing in the model enforces a monotonic alignment between phonemes and audio: the model learns to advance through the text, but it can lose track. VALL-E 2 addressed this with repetition-aware sampling, which falls back to random sampling when a token repeats too often, and with grouped code modelling, which predicts several frames per step to shorten the sequence. Variations on the autoregressive theme Several systems followed VALL-E's general approach while changing its components. SPEAR-TTS, from Google in 2023, split synthesis into two translation tasks: text to semantic tokens, and semantic tokens to acoustic tokens, the latter in the style of AudioLM. Because the second stage does not involve text, it can be trained on untranscribed audio, and the first stage needs only a small amount of paired data. The authors reported good results with as little as fifteen minutes of transcribed speech for the text-to-semantic stage, using back-translation to generate synthetic training pairs. Many open systems released since 2024 use a pretrained text language model as the autoregressive backbone, extending its vocabulary with audio tokens. This brings strong text understanding, which helps with pronunciation of unusual words, numbers and abbreviations, and with prosody that reflects meaning, such as appropriate emphasis on contrastive words. Llasa, published in 2025, initialised from a Llama text model and generated single-codebook speech tokens, and its authors reported that scaling both training compute and inference-time compute, for instance by sampling several candidates and selecting with a verifier, improved results. Systems built in this way blur the line between a speech synthesiser and a multimodal language model that happens to output speech. Non-autoregressive generation with flow matching The robustness problems of autoregressive codec models encouraged a return to non-autoregressive generation, now with much more powerful generative models than the regression-based FastSpeech generation had used. The key technique is flow matching, a training method for continuous generative models closely related to diffusion. A flow-matching model learns a velocity field that transports samples from a simple noise distribution to the data distribution along approximately straight paths; generation integrates this field from noise to data in a modest number of steps. Meta's Voicebox, published in 2023, applied flow matching to speech as a text-guided infilling task. Given the text of an utterance and the audio with some portion masked out, the model generates mel spectrogram frames for the masked region that are consistent with the surrounding audio and the text. Zero-shot synthesis becomes a special case: the reference recording is the unmasked context, and the target utterance is the masked region to be filled in. Voicebox used a separate duration model to determine how many frames each phoneme should occupy, which gave it a hard monotonic alignment and correspondingly good robustness. Trained on 60,000 hours of English audiobooks and a multilingual set covering six languages, it reported lower word error rates than VALL-E with comparable speaker similarity, and it could also perform editing, denoising and style transfer, because all of these are forms of infilling. Like VALL-E, it was not publicly released. E2 TTS, from Microsoft in 2024, and F5-TTS, from a team led by researchers at Shanghai Jiao Tong University later the same year, simplified the design further by removing the duration model and the phoneme alignment altogether. The text, as characters, is simply padded with filler tokens to the length of the target audio and supplied to the flow-matching model alongside the masked spectrogram, and the model learns to align them internally. The only duration decision is the total length of the output, which can be estimated from the ratio of text length to reference length. F5-TTS replaced the U-Net-style Transformer of E2 with a diffusion Transformer and added a convolutional text encoder, and its authors reported faster training and better robustness. Its code and weights were released openly, and it became one of the most widely used open zero-shot synthesis systems. Masked generative models offer a third non-autoregressive option. MaskGCT, published in 2024 by the Amphion team, generates discrete tokens rather than continuous spectrograms, but does so by iterative unmasking: starting from a fully masked sequence, it predicts all tokens in parallel, keeps the most confident, re-masks the rest, and repeats. Like E2 and F5, it requires no explicit alignment. It uses two stages, text to semantic tokens and semantic to acoustic tokens, each masked. Hybrid architectures The most successful production-oriented systems combine an autoregressive stage, which handles content, timing and long-range prosody, with a non-autoregressive stage, which handles acoustic detail. The autoregressive stage operates on relatively few tokens with high linguistic content, so it is fast and its robustness problems are contained. The non-autoregressive stage operates on continuous features with a powerful generative model, so it produces high-fidelity output without sequential generation. CosyVoice, published by Alibaba in 2024, is the clearest example. An autoregressive Transformer generates supervised semantic tokens, at 25 per second, from text and a speaker embedding. A conditional flow-matching model then generates mel spectrograms from the semantic tokens, the speaker embedding and the reference spectrogram, and a HiFi-GAN vocoder produces the waveform. Because semantic tokens carry content but little speaker information, the autoregressive stage can focus on what is said, and the flow-matching stage is responsible for making it sound like the target speaker. CosyVoice 2 and 3 refined this with finite scalar quantisation for the tokeniser, initialisation of the autoregressive model from a pretrained text language model, and chunk-aware streaming. ByteDance's Seed-TTS, described in 2024, used a similar structure, an autoregressive token model followed by a diffusion-based acoustic model, trained on very large data, and its authors reported that listeners could not reliably distinguish its output from real recordings on their test sets. It also explored a fully non-autoregressive variant and reinforcement-learning-based fine-tuning to improve robustness and emotional control. Table 4 summarises the three paradigms. Table 4. Architectural paradigms for zero-shot speech synthesis. Paradigm Representative systems What is generated Alignment Main strength Main weakness Autoregressive codec LM VALL-E, Llasa Codec tokens, sequentially Learned implicitly Natural, varied prosody; strong prompt fidelity Skips, repeats, loops; slow for long outputs Non-autoregressive flow or mask Voicebox, F5-TTS, MaskGCT Spectrogram frames or tokens, in parallel Duration model or learned from padding Robust, fast, good for editing Total length must be set; prosody can be flatter Hybrid CosyVoice, Seed-TTS Semantic tokens, then continuous features AR over semantic tokens Balances robustness and naturalness; streamable Two models to train; cascaded errors Source: published descriptions of the named systems. Text front ends still matter The language-modelling paradigm did not abolish the text front end, the component that turns written text into something a synthesiser can pronounce. Numbers, dates, currencies, abbreviations, acronyms and symbols must be expanded: "Dr." may be doctor or drive, "1/2" may be one half or January second, "$3.5M" must become three point five million dollars. Some systems handle this with rule-based text normalisation before synthesis; others rely on a language-model backbone to learn it from data. The latter works well for common cases and poorly for rare ones, and errors here are among the most noticeable to users, because they are content errors rather than acoustic ones. Pronunciation is a related issue. Systems that take characters rather than phonemes must learn spelling-to-sound rules from data, which is easy for languages with regular orthographies such as Spanish or Finnish and harder for English, whose spelling is notoriously irregular, or for Chinese and Japanese, where the same characters can have different readings in different words. Many systems therefore still use a grapheme-to-phoneme converter for some languages, or accept phoneme input as an override, so that product names and unusual words can be pronounced correctly. How synthesis is evaluated Evaluation of zero-shot synthesis rests on three measurements, all imperfect. Intelligibility is measured by transcribing the synthesised speech with a strong recogniser, commonly Whisper large-v3 for English and Paraformer for Chinese, and computing the word or character error rate against the input text. This captures skipped, repeated and mispronounced words, but it inherits the recogniser's biases and its error floor; a synthesis system cannot score better than the recogniser's own error rate on real speech of the same kind, and differences of a fraction of a percentage point between systems are often within noise. Speaker similarity is measured by extracting speaker embeddings from the reference and the synthesised speech with a speaker verification model, commonly a WavLM-based model fine-tuned for verification, and computing the cosine similarity between them. This captures timbre and some speaking style but is insensitive to aspects of identity that verification models do not encode, and it can be gamed: a system that copies the reference's recording conditions will score higher than one that produces clean speech in the same voice, because the verification model is partially sensitive to channel. Naturalness and quality are measured by human listening tests, typically mean opinion scores on a five-point scale or comparative tests in which listeners choose between two samples, and increasingly by learned predictors of mean opinion score such as UTMOS. Human tests are expensive and vary with listener population and instructions; learned predictors are cheap but only as reliable as their training data, and they can be fooled by artefacts their training did not include. Standard test sets help comparability. LibriSpeech test-clean and its subset selected by VALL-E's authors, and Seed-TTS's released evaluation sets in English and Chinese, are widely used. But because most zero-shot systems now achieve error rates near the recogniser's floor and similarity scores within a few hundredths of each other, the differences that remain are often in aspects that none of the automatic metrics capture well: whether the prosody suits the meaning, whether the voice remains consistent across a long passage, whether emotional tone is appropriate. These remain largely matters for careful listening. Where the paradigm stands Synthesis as language modelling has delivered what it promised: zero-shot voice cloning from a few seconds of reference audio, at a quality that listeners often cannot distinguish from real speech in short samples. The open questions have shifted. They concern long-form consistency, since a model that produces convincing ten-second samples may drift over a ten-minute audiobook chapter; controllability, since prompting with reference audio gives limited control over emotion and style independently of identity; robustness on difficult text; and the handling of languages beyond the handful on which most systems are trained. They also concern the voice itself: what exactly is being preserved when a system clones a speaker, and how to tell whether it has succeeded. That is the subject of the next chapter. Chapter 8. Voice Preservation and Zero-Shot Cloning Voice preservation means producing speech that a listener would accept as coming from a particular person, in content that person did not actually say. It covers several applications that are technically similar and socially very different: a person with a degenerative disease recording their voice so that a speech-generating device can speak in it later; a film being dubbed so that an actor appears to speak another language in their own voice; a customer-service system speaking in a consistent branded voice; a podcast being edited by retyping its transcript; and, at the other end of the spectrum, fraud and impersonation. All rest on the same capability, and all raise the same technical question: what, exactly, is a voice, and how much of it does a model need to see in order to reproduce it? What identity consists of From the listener's side, a voice is recognised through a mix of cues. The most stable are anatomical: the length and shape of the vocal tract, which fix the general placement of formants; the mass and stiffness of the vocal folds, which fix the typical pitch range and voice quality. Layered on these are learned habits: accent and dialect, the realisation of particular vowels and consonants, speaking rate, rhythm, characteristic intonation patterns, hesitation habits, laughter. And there is a layer of situational variation: the same person sounds different when tired, excited, ill, whispering, or speaking to a child. People recognise familiar voices across all of this variation, and listeners familiar with a speaker are sensitive to small departures from their habitual style that strangers would not notice. From the machine's side, identity is usually operationalised through speaker embeddings. A speaker verification network is trained to map utterances to fixed-length vectors such that utterances from the same speaker are close together and utterances from different speakers are far apart. The d-vector, introduced by Google in 2014, averaged frame-level activations of a network trained to classify speakers. The x-vector, from Johns Hopkins in 2018, used a time-delay neural network with statistics pooling and became the standard for several years. ECAPA-TDNN, published in 2020, added channel attention and multi-layer feature aggregation and is now among the most widely used. More recent verification models are built on self-supervised encoders, and WavLM-based verifiers are standard in synthesis evaluation. These embeddings capture the stable anatomical layer well and the habitual layer partially. They are trained to be invariant to content, which is what verification requires, so they discard much of what is distinctive about how a person speaks rather than how they sound. They are also, to varying degrees, sensitive to recording conditions, because channel differences are hard to separate from speaker differences in training data. This mismatch between what the evaluation measures and what listeners perceive as identity is a recurring difficulty in voice preservation. Three ways to condition on a voice There are three broad ways to give a synthesis model a target voice, and they differ in what they transmit. The first is a fixed speaker embedding: extract a vector from reference audio with a verification model or a learned speaker encoder, and feed it to the synthesiser as a conditioning input. This is compact and fast, and it allows voices to be stored, interpolated and compared. But it passes only what the embedding captures, which is mostly timbre. Multi-speaker systems of the FastSpeech and VITS generation used this approach, and their zero-shot performance was limited by the narrowness of the embedding. The second is in-context prompting: supply the reference audio itself, as a token or spectrogram sequence, as the prefix of the sequence the model is generating. The model then continues the prefix, and whatever it can infer from the prefix about the speaker is available. This is how VALL-E, Voicebox, F5-TTS and most current zero-shot systems work. It transmits far more than an embedding: speaking rate, accent, voice quality, the acoustic environment and the emotional tone of the reference all influence the output. That richness is both the reason in-context systems clone voices so convincingly and the reason they are hard to control: a reference recorded in an echoey room produces echoey output, and an angry reference produces angry output even if the target text is calm. The third is fine-tuning: adapt the synthesis model's weights, or a small set of added weights, on recordings of the target speaker. This requires more data, typically minutes to hours rather than seconds, and some computation, but it produces the most faithful reproduction of a speaker's habitual style, because the model can learn patterns that would not show up in a few seconds of audio. Professional voice cloning services commonly offer both an instant mode, based on in-context prompting, and a professional mode, based on fine-tuning with longer recordings. Most systems now combine at least two. CosyVoice, for example, conditions its flow-matching decoder both on a speaker embedding and on the reference spectrogram as an in-context prompt; the embedding provides a stable identity anchor across the utterance, and the prompt provides detail. How much audio is enough Zero-shot systems advertise reference lengths of three to fifteen seconds, and the numbers are meaningful but need interpretation. Three seconds, as in the VALL-E paper's evaluations, is enough to capture timbre and broad pitch range, and to produce output that a speaker verification model scores as similar and that strangers judge as the same voice. It is generally not enough to capture distinctive prosody, accent subtleties or habitual phrasing, and people familiar with the speaker are more likely to notice that something is off. Ten to thirty seconds captures more, and systems that use longer prompts typically show higher similarity scores. Beyond a minute or so, in-context prompting runs into practical limits on sequence length, and fine-tuning becomes the better route. The content of the reference matters as much as its length. A reference with varied intonation, several sentence types and a range of phonemes gives the model more to infer from than a reference of the same length with monotone delivery. A reference with background noise, music or other speakers contaminates the output. A reference in one emotional register biases the output toward it. Practical voice cloning workflows therefore spend effort on reference selection and cleaning: choosing segments with clear speech, removing noise with a speech enhancement model, and sometimes concatenating several short segments with different characteristics. Voice conversion Voice conversion is the task of changing the speaker of an existing recording while keeping its content and, ideally, its prosody. It is closely related to synthesis but starts from speech rather than text, which means the content and timing are given and only the identity must be changed. The standard modern approach exploits the factorisation described in earlier chapters. Extract content representations from the source speech using a self-supervised encoder or semantic tokeniser, which carry what was said with relatively little speaker information. Extract identity from reference speech of the target speaker. Decode the combination with a synthesis decoder or vocoder. The quality of the conversion depends on how speaker-free the content representation is: any source-speaker information that leaks through will pull the output toward the source voice. kNN-VC, published in 2023 by Matthew Baas, Benjamin van Niekerk and Herman Kamper, showed how far this could be taken with a strikingly simple method. It extracts WavLM features from both the source and a pool of target-speaker audio. For each source frame, it finds the k nearest target frames in feature space and replaces the source frame with their average. The resulting sequence of target-speaker features is vocoded with a HiFi-GAN trained on WavLM features. Because WavLM features of the same phoneme from different speakers are close in feature space, nearest-neighbour matching tends to select target frames with the same phonetic content, and because the replacement frames come from the target speaker, they carry the target's identity. With a few minutes of target audio, kNN-VC produced conversions competitive with far more complex systems, which is a vivid demonstration of how much speaker-independent phonetic structure self-supervised representations contain. Conversion in real time, for live streaming or gaming, imposes latency constraints that rule out models with long lookahead. Several open systems, including the community-developed Retrieval-based Voice Conversion, which combines HuBERT-style features with nearest-neighbour retrieval and a VITS-derived decoder, are widely used for this. Their popularity has also made them a common tool of impersonation, which is one reason real-time conversion features prominently in discussions of detection. Where identity is lost Voice preservation fails in characteristic ways, and each failure traces to a specific point in the pipeline. Speaker drift occurs when a long utterance starts in the target voice and gradually moves toward a different one, typically toward an average voice or toward a speaker prominent in training data. It arises in autoregressive systems because the model's context window covers the prompt at the start of generation but the prompt's influence weakens as more generated tokens accumulate. Systems mitigate it by reinserting the prompt periodically, by conditioning every step on a fixed speaker embedding, or by generating long passages in segments, each prompted afresh. Timbre averaging occurs when the model reproduces the general category of the voice, its approximate pitch and broad quality, but not its individual character. It is most common with embedding-only conditioning and with speakers far from the training distribution: very young or very old speakers, speakers with voice disorders, speakers of underrepresented languages and accents. A model trained predominantly on adult speakers of standard varieties will tend to normalise unusual voices toward the centre of its training distribution. Prosodic mismatch occurs when the timbre is right but the speaking style is wrong: the cloned voice sounds like the person reading someone else's lines. Short references often lack enough prosodic information for the model to capture habitual style, and fixed embeddings encode almost none of it. Channel transfer occurs when the output reproduces the reference's recording conditions. This preserves something real, the acoustic signature of the recording, but it is usually not what the user wants and it inflates similarity scores. Accent leakage, which occurs when the target is in a different language from the reference, is the subject of the next chapter. Voice banking and clinical uses Some of the most consequential applications of voice preservation are clinical. People with motor neurone disease, also known as amyotrophic lateral sclerosis, often lose the ability to speak as the disease progresses, and many use speech-generating devices. For decades those devices offered a small set of generic synthetic voices. Voice banking, in which a person records their own speech while they still can so that a personalised synthetic voice can be built from it, has existed since the concatenative era, but it traditionally required hours of carefully read recordings, often more than a person whose speech was already deteriorating could manage. Neural synthesis, and particularly fine-tuning from a few minutes of audio or zero-shot cloning from seconds, has changed this substantially. A personalised voice can now be built from much shorter recordings, and in some cases from existing recordings such as home videos or voicemails made before the illness. Several commercial providers of speech-generating devices and voice cloning services have offered such voices for people with ALS and other conditions, and charities in the field have promoted voice banking as early as possible after diagnosis. Message banking is a related practice: recording specific phrases, in the person's own voice and with their own intonation, for later playback. It preserves authentic recordings rather than synthesising new ones, and many clinicians recommend combining it with voice banking, because a banked message carries the full expressiveness of the original speech in a way that synthesis still does not entirely achieve. Clinical voice preservation also raises particular technical difficulties. When speech has already been affected by disease, a synthesis system cloned from current recordings will reproduce the affected speech, which is usually not what the person wants. Some systems attempt to repair this, generating speech that keeps the speaker's timbre while correcting articulation, but doing so means deliberately departing from the reference, and the balance between faithfulness and repair is a judgement the person concerned should make. Consent and the definition of a voice Voice preservation raises a question that has no purely technical answer: when is it legitimate to clone a voice? The technical facts constrain the answer. A few seconds of publicly available audio, such as a clip from a podcast, a voicemail greeting or a video, are enough for a convincing zero-shot clone. Any consent regime that depends on controlling access to recordings of a person's voice is therefore weak, because such recordings are abundant for almost everyone who has spoken in public or online. The consent mechanisms that do exist are mostly procedural. Commercial voice cloning services commonly require the user to affirm that they have the right to use the voice, and some require a verification recording in which the speaker reads a randomly generated phrase, which makes it harder to clone someone from found audio alone. Professional voice actors increasingly negotiate contractual terms covering whether and how synthetic versions of their voices may be made and used; the dispute between the SAG-AFTRA union and video game companies, which led to a strike beginning in 2024, turned substantially on such terms. Some jurisdictions have begun to legislate, as the chapter on misuse discusses. These mechanisms shape what a technically sophisticated system should do. A system that accepts arbitrary reference audio and clones it without friction is easy to build; the constraint on doing so is not technical but a matter of design choices about verification, logging, watermarking and refusal. Voice preservation done responsibly is not just a synthesis problem but an access-control problem, and the components that solve the latter belong in the same architecture as the components that solve the former. Chapter 9. Crossing Languages: Zero-Shot Cross-Lingual Voice Transfer Cross-lingual voice transfer means producing speech in one language in the voice of a speaker heard only in another. A Spanish-speaking executive's address delivered in Japanese, in her own voice; a documentary narrator heard in English and released in Hindi without hiring a new narrator; a patient who speaks only Tagalog hearing a doctor's English explanation in the doctor's own voice, translated. When the reference is only a few seconds long and the speaker never recorded anything in the target language, this is zero-shot cross-lingual transfer, and it is the hardest version of voice preservation. It tests every representational choice discussed so far, because it requires a system to change the content completely, change the language, and keep the identity, while knowing that identity and language are tangled together in the signal. Why it is harder than monolingual cloning A monolingual zero-shot system works by inferring from the reference everything it needs to continue in the same voice. In the cross-lingual case, some of what it infers is wrong for the target. The reference speaker's vowels are realised in the patterns of their language; the target language may have different vowels or the same vowels realised differently. The reference's rhythm reflects whether the source language is stress-timed, like English, or syllable-timed, like Spanish, or mora-timed, like Japanese. Its intonation patterns reflect source-language conventions for questions, emphasis and phrase boundaries. Its consonants include allophones that the target language does not use. All of this is part of what a codec language model sees as the voice, and all of it will, by default, be carried into the output. The result is accent leakage: output in the target language that sounds as if spoken by a non-native speaker with the reference's native accent. Sometimes this is exactly right. A real bilingual speaker with a recognisable accent sounds like themselves partly because of that accent, and dubbing that removes it can sound uncanny. But often the goal is native-sounding speech in the target language, in a voice recognisably the speaker's, and accent leakage then counts as failure. The tension is not between right and wrong but between two different definitions of preserving a voice. The second difficulty is data. Monolingual zero-shot synthesis in English benefits from hundreds of thousands of hours of training audio. Most languages have far less, and cross-lingual training needs not just multilingual data but data in which the same speakers appear in several languages, or some other way of teaching the model that timbre and language are separable. Bilingual speech corpora with the same speakers are rare and small. Most systems must learn the separation implicitly from monolingual data in many languages, trusting that the model will discover that timbre varies across speakers within a language in the same way it varies across speakers between languages. The third difficulty is phonetic coverage. A speaker of a language without a particular sound has never produced it in the reference. When the target language requires it, the model must invent how that speaker would produce it, extrapolating from the general structure of their voice. For a sound that exists in many languages, the model has seen many speakers produce it and can generalise. For a sound specific to a few languages, such as the clicks of Xhosa and Zulu, the pharyngeal consonants of Arabic or the retroflex series of many South Asian languages, the extrapolation is harder and errors are more likely. Language as a controllable variable The first systematic approach to cross-lingual transfer in the codec-language-model era was VALL-E X, published by Microsoft in 2023. It extended VALL-E to two languages, English and Chinese, with training data in both, and added explicit language identity tokens to the model's input. At inference, the model received the source-language phonemes and codec tokens of the reference, the target-language phonemes of the text to be spoken, and a language token indicating the target. The authors reported that without the language token the output carried a strong foreign accent, and that the language token substantially reduced it, giving the user a dial for the accent: specify the target language to reduce the reference accent, or specify the source language to preserve it. The broader lesson is that language identity must be represented explicitly somewhere in a cross-lingual system, so that the model can be told which of the patterns in the reference belong to the speaker and which belong to their language. Language tokens, language embeddings and language-specific phoneme inventories all serve this purpose. Systems that rely on universal phoneme sets, such as the International Phonetic Alphabet, gain the ability to share sounds across languages but lose the implicit language marker that language-specific inventories carry, and typically compensate with an explicit language token. YourTTS, published by researchers associated with the Coqui project in 2022, took the speaker-embedding route to multilingual zero-shot synthesis. Built on VITS, it was trained on English, Portuguese and French and conditioned on a speaker embedding from a pretrained verification model, with a language embedding as an additional input. Because the speaker embedding was trained for verification, it captured timbre while largely discarding accent, so YourTTS produced relatively native-sounding target-language speech with moderate speaker similarity. Coqui's later XTTS, described in a 2024 paper, combined a GPT-style autoregressive model over audio tokens with a speaker conditioning module that encoded the reference into a fixed set of latent vectors, and supported more than a dozen languages. Coqui closed as a company at the start of 2024, but XTTS remained in wide use through its open release. Disentanglement by design The most robust cross-lingual systems are those whose representations separate content from identity before the language boundary is crossed. The hybrid architecture of CosyVoice illustrates why. Its autoregressive model generates supervised semantic tokens, which come from a multilingual recogniser and therefore carry content in a largely language-appropriate and speaker-independent form. The target-language tokens are generated from target-language text, so their phonetic content is native by construction. Only in the second stage, where a flow-matching model renders these tokens as speech, does the speaker's identity enter, through a speaker embedding and a reference spectrogram. Because the semantic tokens fix what sounds are made, the decoder's job is to make them in the speaker's voice, not to decide which sounds to make, and accent leakage is limited to the subtler aspects of realisation that the decoder controls. This is the factorisation argument of earlier chapters in its most practical form. When content tokens carry a speaker's accent, the accent crosses the language boundary with them. When content tokens are derived from a recogniser trained to be speaker-invariant, and identity is supplied separately, the accent mostly stays behind. Systems that use codec tokens throughout, without a semantic stage, face the hardest version of the problem, because their tokens carry everything at once. The trade-off, as always, is that what is discarded must be supplied. A cross-lingual system that strips accent from the content stream may also strip habitual prosody and expressive detail that listeners consider part of the voice. Speaker embeddings capture timbre; they do not capture how a person laughs or pauses. Cross-lingual output from strongly factorised systems therefore tends to sound like the speaker's timbre on a generic native speaker's delivery. Whether that counts as preservation depends on what the listener cares about. Speech-to-speech translation with voice preservation The most demanding application of cross-lingual transfer is speech-to-speech translation, in which the input is speech in one language and the output is speech in another, carrying the same meaning in the same voice. The classical approach is a cascade: recognise the source speech, translate the text, synthesise the target. A cascade with a zero-shot synthesiser conditioned on the source speech can preserve voice reasonably well, and each component can use the best available model for its task. But a cascade discards everything in the source that does not survive transcription: emphasis, emotion, pacing, hesitation, laughter. And it accumulates latency, because each stage waits for the previous one. Direct speech-to-speech translation models attempt to translate without a text bottleneck. Google's Translatotron, published in 2019, was the first end-to-end model to map source spectrograms directly to target spectrograms. Translatotron 2, in 2021, improved quality and introduced a voice retention mode trained only on speaker-consistent pairs, which preserved the source speaker's voice in the output while making it difficult to use the model to produce a different speaker's voice, a deliberate design choice to limit misuse. Translatotron 3, in 2023, explored training without parallel speech data. Meta's work moved toward unit-based translation. Its UnitY architecture, published in 2023, translated speech first into target-language text and then into target-language discrete units, which a unit vocoder rendered as speech, in a single model trained end to end with intermediate text supervision. This design underpinned SeamlessM4T, released in August 2023, a single multitask model covering speech recognition, text translation, speech-to-text translation and speech-to-speech translation across roughly a hundred languages for speech input. The Seamless family released at the end of 2023 added SeamlessExpressive, which aimed to preserve the vocal style, speech rate and pauses of the source in the translated output, and SeamlessStreaming, discussed in the next chapter. SeamlessExpressive is instructive. Its developers explicitly targeted the prosodic dimensions that cascades and unit-based systems discard: speaking rate, pause placement, and the expressive style of the speaker, alongside timbre. They did this with a separate expressive encoder that extracts a style representation from the source and conditions the unit generator and the vocoder on it. The difficulty is that prosody is partly language-specific: an English rise on a question does not map directly onto the intonation of a question in French or Mandarin. Preserving prosody across languages means preserving its function, such as emphasis, uncertainty or excitement, rather than its literal contour, and current systems achieve this only partially. Kyutai's Hibiki, released in early 2025, applied the full-duplex multistream architecture of the Moshi dialogue model to simultaneous speech translation from French to English, generating translated speech in the speaker's voice while the speaker was still talking. Its authors reported results ahead of earlier simultaneous systems on translation quality, speaker similarity and naturalness, and the design illustrates how voice transfer, translation and streaming have converged on the same token-based machinery. Tone, rhythm and the limits of transfer Some language pairs strain the idea of a portable voice more than others. When the target is a tone language and the source is not, the model must generate lexical tones from the text in a pitch range consistent with the speaker. The speaker's habitual pitch range can be estimated from the reference, but how that speaker would realise Mandarin's third tone, or Cantonese's six contrastive tones, is pure extrapolation, and systems that carry the speaker's intonation habits from a non-tonal language can distort the tones and change word meanings. In the reverse direction, a tone-language reference contains pitch movements that are lexical rather than expressive, and carrying them into a non-tonal target as if they were prosody produces odd intonation. Rhythm presents similar issues. A syllable-timed language spoken with the vowel reduction and stress patterns of a stress-timed one sounds strongly accented, and vice versa. Duration models trained on one language's rhythm and applied to another's text can produce timing that is phonetically correct but rhythmically foreign. Script and segmentation add a further layer. Languages without spaces between words, such as Chinese, Japanese and Thai, require segmentation before phoneme conversion; languages with complex orthographies such as Arabic, where short vowels are usually not written, require diacritisation or a model that infers vowels from context. Errors at these stages are content errors that no amount of voice preservation can repair. Evaluating cross-lingual transfer Evaluation of cross-lingual voice transfer extends the monolingual metrics and adds new ones. Intelligibility is measured with a recogniser for the target language, which requires a strong recogniser in that language and is less reliable for low-resource targets. Speaker similarity is measured by comparing embeddings from the source-language reference and the target-language output, which raises a problem: speaker verification models are themselves somewhat sensitive to language, and similarity scores for the same speaker in two languages are typically lower than for the same speaker in one, even for genuinely bilingual speakers. Cross-lingual similarity scores therefore cannot be compared directly with monolingual ones. Accent is the hardest to measure. Human ratings of nativeness are the usual approach, sometimes supplemented by accent classifiers, but both depend on who is rating and against what standard. A Mexican Spanish speaker rating synthetic Spanish from an English reference may judge nativeness differently from a Castilian one. For speech-to-speech translation, meaning preservation is measured by transcribing the output and comparing it with reference translations using text translation metrics, and expressive preservation by comparing prosodic features or by human judgements of whether the output conveys the same emphasis and emotion as the source. The absence of standard bilingual evaluation data with the same speakers in both languages is a persistent obstacle. Without it, it is hard to say what a correct cross-lingual output would sound like, because the real speaker never produced one. Coverage and fairness The final issue is which languages get this capability at all. Most zero-shot cross-lingual systems are trained on a small set of high-resource languages: English, Chinese, Spanish, French, German, Japanese, Korean, Portuguese and a handful of others. Projects such as Meta's MMS have extended basic synthesis to over a thousand languages, but those systems are mostly single-speaker per language and not designed for zero-shot voice transfer. The gap between the languages in which a person's voice can be carried across and those in which it cannot is wide, and it tracks the existing inequalities of language technology. There is a further subtlety. Accent leakage affects speakers of underrepresented languages in a particular way. When a speaker of Yoruba or Bengali uses a cross-lingual system to produce English, the system has seen relatively little speech from their accent group, so its model of their voice is less accurate and the transfer is more likely to drift toward a generic voice. When a speaker of English uses the same system to produce Yoruba or Bengali, the system has seen relatively little target-language speech, so the output is more likely to be mispronounced. Voice transfer works best for those already best served, which is a constraint that only broader data collection and evaluation can address. Chapter 10. Streaming Latency in Conversational Voice Engines A conversational voice engine is judged first by its timing. A system that understands perfectly and speaks beautifully but takes two seconds to reply feels broken; a system that replies in a quarter of a second feels attentive, even if what it says is ordinary. This is not a matter of impatience. Human conversation runs on a tight rhythm, and departures from that rhythm carry meaning: a long pause before a reply signals reluctance, difficulty or disagreement, whether or not the speaker intends it. A machine that pauses for a second before every answer is, from the listener's point of view, constantly hesitating. Getting latency right therefore means understanding the human baseline, decomposing the machine's delay into its parts, and attacking each part with techniques that reach back into every representational choice discussed so far. The human benchmark Conversation analysts have long observed that speakers take turns with remarkably small gaps. A cross-linguistic study by Tanya Stivers and colleagues, published in 2009, measured the timing of responses to yes-or-no questions in ten languages from around the world and found that the most common gap in every language was within about 200 milliseconds of the end of the question, with modest differences between languages in average delay. Since producing even a single word takes a speaker considerably longer than 200 milliseconds to plan, from retrieving the concept to articulating the first sound, the short gaps imply that listeners begin planning their reply while the other person is still talking, predicting both the content and the end of the turn. Overlaps of a few hundred milliseconds are common and generally unremarkable; gaps much longer than half a second start to be heard as meaningful. This sets an ambitious target. A machine that matches human turn-taking must begin speaking within a few hundred milliseconds of the user stopping, which is less time than it takes to transmit a network packet across a continent and back several times. Earlier voice assistants fell far short of it. When OpenAI introduced GPT-4o in May 2024, it described the voice mode it was replacing, which chained a separate transcription model, a text language model and a synthesis model, as having average latencies of 2.8 seconds with GPT-3.5 and 5.4 seconds with GPT-4, and reported that GPT-4o could respond to audio input in as little as 232 milliseconds, with an average of 320 milliseconds. Kyutai, describing its Moshi model in September 2024, reported a theoretical latency of 160 milliseconds and a practical latency of about 200 milliseconds. These figures put the best systems close to the human benchmark, and they were achieved by architectural changes rather than by faster hardware alone. Decomposing the delay The latency of a voice engine is the time from the user's last sound to the system's first sound. In a cascaded system, it decomposes into a chain of contributions, and it is worth listing them, because each has its own techniques. Audio capture and transmission: the user's device must buffer audio into packets, typically 10 to 20 milliseconds each, and send them to a server. Network round-trip times vary from tens of milliseconds within a region to a few hundred across continents or on congested mobile links, and jitter buffers add further delay to smooth out irregular packet arrival. Endpointing: the system must decide that the user has finished speaking. A simple voice activity detector with a silence threshold, say 500 to 800 milliseconds, adds that entire threshold to every reply. This is often the single largest contributor in naive systems. Final recognition: once the end of the turn is declared, the recogniser must produce its final transcript. A streaming recogniser that has been transcribing all along may need only a few tens of milliseconds to finalise; a non-streaming one that starts only at the endpoint may need hundreds. Language model time to first token: the text model must process the prompt, including conversation history and system instructions, and produce its first output token. For a large model this involves a prefill pass over possibly thousands of tokens, then one decoding step, and typically takes from under a hundred milliseconds to several hundred depending on model size, prompt length and hardware. Synthesis time to first audio: the synthesiser must produce its first chunk of audio. A non-streaming synthesiser waits for the full text and then generates the full waveform; a streaming one starts as soon as the first few words are available and emits audio in chunks. Playback buffering: the client must receive the first audio chunk and begin playing it, usually after accumulating a small buffer to avoid underruns. Table 5 gives an indicative budget for a cascaded system, distinguishing a naive configuration from an optimised one. The figures are illustrative ranges drawn from the components' typical behaviour described in this chapter, not measurements of any particular product. Table 5. Indicative latency budget for a cascaded voice engine. Stage Naive configuration Optimised configuration Main technique Capture and network, one way 50 to 150 ms 20 to 60 ms Small frames, regional servers, persistent connections Endpointing 500 to 1,000 ms 100 to 300 ms Semantic turn detection, speculative processing Final recognition 200 to 500 ms 10 to 50 ms Streaming recogniser, incremental finalisation Language model first token 300 to 1,000 ms 50 to 200 ms Smaller model, prompt caching, speculative start Synthesis first audio 500 to 2,000 ms 50 to 150 ms Streaming synthesis from partial text Network return and playback buffer 50 to 150 ms 20 to 60 ms Small chunks, adaptive buffering Source: illustrative ranges assembled from the component behaviour described in the text; actual figures vary widely with hardware, model size and network. The naive column sums to well over two seconds, close to the figures OpenAI reported for its earlier cascade. The optimised column sums to a few hundred milliseconds. Every row has a technique, and the remaining sections examine the most important ones. Endpointing and turn prediction Because endpointing dominates naive latency, it is the first place to look. A silence-based detector faces an unavoidable trade-off: a short threshold responds quickly but interrupts users who pause mid-sentence to think, while a long threshold avoids interruptions but makes every reply slow. People pause within turns frequently, for breath, for word-finding, for emphasis, and those pauses can easily exceed half a second. Humans resolve this by predicting turn ends from content and prosody, not just silence. A sentence that is syntactically complete and ends with falling intonation is likely finished; one that ends on "and" or with a rising, continuing intonation is not. Modern systems emulate this with turn-detection models that take the streaming transcript and sometimes prosodic features and estimate the probability that the user has finished. Several open-source voice agent frameworks now include such models, built from small language models fine-tuned on conversational transcripts. With a good turn predictor, the silence threshold can be shortened when the model is confident the turn is complete and lengthened when it is not. A complementary technique is speculative processing. The system begins generating a response before the endpoint is confirmed, using the partial transcript, and discards the work if the user resumes speaking. This converts endpointing latency into wasted computation, a trade worth making when computation is cheaper than user attention. The cost is complexity: speculative work must be cancellable, and responses must be regenerated if the final transcript differs materially from the partial one. Streaming recognition and incremental understanding A streaming recogniser, typically a transducer or CTC model with a causal or limited-lookahead encoder, produces partial transcripts continuously. At the endpoint, the transcript is nearly complete and only the last few hundred milliseconds of audio need final processing. The latency cost is that streaming models are somewhat less accurate than their non-streaming counterparts, because they cannot use future context; production systems often accept this, sometimes running a non-streaming rescoring pass in parallel to correct the transcript for the conversation history without delaying the reply. Incremental understanding goes further: feeding the partial transcript to the language model as it arrives, so that the model's prefill over the conversation history and the early part of the user's turn is complete before the turn ends. Prompt caching, offered by most large language model serving systems, means that the conversation history need not be reprocessed on every turn; only the new user input must be prefilled, which shortens time to first token substantially. Streaming synthesis A non-streaming synthesiser needs the whole text before it starts, and the language model produces text at a few tens of tokens per second, so waiting for the full reply adds seconds. Streaming synthesis begins as soon as the first phrase is available. The simplest approach segments the language model's output at sentence or clause boundaries and synthesises each segment as it completes. This is easy to implement with any synthesiser and cuts latency to the time needed to generate and synthesise the first clause, but it introduces prosodic discontinuities at segment boundaries, because each segment is synthesised without knowledge of what follows. True incremental synthesis generates audio from a text stream with limited lookahead, so that the synthesiser can start speaking after a few words. This requires a synthesis model that can work causally over text. Autoregressive codec models are naturally suited, since they generate audio left to right, but they need some lookahead on the text to choose appropriate prosody: the intonation at the start of a sentence depends on whether it is a question. Hybrid systems such as CosyVoice 2 interleave text tokens and speech tokens in a fixed ratio, so that the model reads a few text tokens, emits a few speech tokens, reads more text, and so on, and they make the flow-matching decoder chunk-aware so that it can render audio in short blocks. Non-autoregressive systems that generate an entire utterance at once, such as F5-TTS, are harder to stream and are usually applied to short segments. Within synthesis, the time to first audio has its own components. The token model must generate enough tokens for the first chunk; the decoder, whether a codec decoder, a flow-matching model or a vocoder, must render them; and any lookahead in the decoder adds delay. A codec at 12.5 frames per second produces 80 milliseconds of audio per frame, so the first chunk can be as short as one or two frames. A codec at 75 frames per second produces about 13 milliseconds per frame but requires many more token-generation steps per second of audio, which raises the throughput needed to stay ahead of playback. This is where frame rate directly determines latency: lower frame rates mean fewer sequential steps per unit of audio and hence less computation on the critical path. Real-time factor is the relevant throughput measure: the time needed to generate a second of audio. A system with a real-time factor of 0.2 generates a second of audio in 200 milliseconds and can easily stay ahead of playback once it starts. A system whose real-time factor approaches 1 will underrun, producing gaps, unless it buffers more before starting, which raises latency. Latency and throughput are separate constraints, and a voice engine must satisfy both. Removing the cascade The cascade imposes a structural limit: each stage must produce enough output for the next stage to begin. Even with every stage streaming, there is a minimum delay at each joint. And the text bottleneck in the middle discards paralinguistic information, such as emotion, emphasis and hesitation, that a human listener would respond to. End-to-end spoken dialogue models remove the cascade by having a single model read and write speech tokens directly. GPT-4o was described by OpenAI as a single model trained end to end across text, vision and audio, so that all inputs and outputs are processed by the same network. Its technical details have not been published, but its reported latencies indicate what is possible when recognition, reasoning and synthesis are not separated. Moshi, published by Kyutai in September 2024 with open weights, is the best-documented example. Its architecture brings together several ideas discussed in earlier chapters. It uses the Mimi codec at 12.5 frames per second, with semantic distillation into the first codebook. It is built on a 7-billion-parameter text language model called Helium, extended to model audio. It uses a two-level structure: a temporal Transformer runs once per 80-millisecond frame, and a small depth Transformer generates the codebooks within each frame. And it models two audio streams simultaneously, one for the user and one for itself, so that at every frame it both listens and decides whether and what to say. Moshi also generates a time-aligned text stream for its own speech, which its developers call an inner monologue, predicting text tokens just ahead of the corresponding audio tokens. They reported that this improved the linguistic quality of its output, because the text provides a scaffold that the audio tokens follow. Moshi's theoretical latency of 160 milliseconds comes from the codec's 80-millisecond frame plus an 80-millisecond acoustic delay between its semantic and acoustic tokens. Its reported practical latency of about 200 milliseconds on a single graphics card puts it near the human benchmark. The design shows how directly latency follows from representation: at 75 frames per second with eight codebooks, the same architecture would need to run its temporal model six times as often, and the latency budget would not close. Full duplex and barge-in Human conversation is not strictly turn by turn. Listeners produce backchannels, short responses such as "mm-hm" or "right", while the other person talks; speakers overlap at turn transitions; and listeners interrupt. A half-duplex system, which either listens or speaks but never both, cannot do any of this. It must wait for the user to stop before it can respond, and if the user starts talking while it speaks, it must detect the interruption, stop, and switch back to listening. Barge-in, the ability to interrupt the system, is the minimum requirement for natural interaction. It needs fast voice activity detection on the user's channel while the system is speaking, which in turn needs acoustic echo cancellation, because the user's microphone picks up the system's own voice from the loudspeaker. Without effective echo cancellation, the system hears itself and interrupts itself, or fails to distinguish the user's interruption from its own output. Once an interruption is detected, the system must stop playback, discard any queued audio, and update its record of the conversation to reflect what it actually said before being interrupted, since the user heard only part of the reply. This last step is easy to overlook and important: a language model that believes it delivered a full answer that was in fact cut off will respond incoherently to the next turn. Full-duplex models such as Moshi go further, modelling both channels at once so that the model can backchannel, overlap and yield naturally. This removes explicit endpointing entirely: the model decides from moment to moment whether to speak, based on everything it has heard. The cost is that such models are harder to control. They may interrupt when they should wait, or remain silent when they should respond, and their behaviour depends on conversational training data that is scarce and expensive to collect. Deployment realities Laboratory latencies are measured on fast hardware with short contexts and no load. Deployed systems face additional delays. Graphics processors shared among many simultaneous conversations must batch requests, which increases throughput but adds queueing delay; a system tuned for latency runs at lower utilisation and higher cost per conversation. Long conversation histories increase prefill time unless cached. Telephony adds its own delays: the public switched telephone network and the codecs it uses add tens of milliseconds and reduce audio bandwidth to 8 kilohertz, which degrades both recognition and synthesis. Mobile networks add variable latency and packet loss, which jitter buffers smooth at the cost of further delay. There is also a perceptual subtlety. Users perceive latency differently depending on what the system does in the gap. A short acknowledgement such as "sure" or "let me check", generated quickly while the substantive reply is prepared, makes a longer total latency feel acceptable, much as human speakers use fillers to hold the floor. Some systems deliberately generate such openers from a fast small model while a larger model prepares the content. Conversely, a reply that starts quickly but then stalls mid-sentence because synthesis fell behind is perceived as worse than one that started slightly later and flowed without interruption. Latency as a design constraint The overall picture is that latency is not a property of any one component but of the whole architecture, and the decisions that most affect it are made far upstream. The frame rate of the codec determines how many sequential steps synthesis must take. The causality of the encoder determines whether recognition can stream. The choice between cascade and end-to-end determines whether there is a text bottleneck in the middle. The representation of turn-taking determines whether endpointing is a silence threshold or a learned prediction. A team that treats latency as an optimisation to be applied at the end will find most of its options already closed off. A team that treats it as a constraint from the start will choose tokenisers, models and interfaces that leave room to meet it. Chapter 11. Trust, Misuse and Governance The capabilities described in the preceding chapters have a direct and obvious dark side. A system that can clone a voice from three seconds of audio can clone anyone whose voice has been recorded and published, and a system that can make that voice speak in real time can be used to impersonate them on a telephone call. These are not hypothetical risks. Voice cloning has been used in fraud against companies and families, in political disinformation, and in harassment. Any account of speech foundation models that treated these uses as a footnote would misrepresent the field, because the people building these systems spend a growing share of their effort on detection, provenance and access control, and because regulation is beginning to shape what can be built and deployed. The shape of the threat The misuse cases cluster into a few kinds. The first is financial fraud by impersonation. Criminals use a cloned voice of a family member, typically claiming to be in trouble and needing money urgently, or of a company executive, instructing an employee to make a payment. Consumer protection agencies in several countries have issued warnings about such schemes, and banks that use voice biometrics for authentication have had to reconsider them. The second is political disinformation. In January 2024, voters in New Hampshire received automated telephone calls using a synthetic voice imitating President Joe Biden, discouraging them from voting in the state's primary election. The incident led the United States Federal Communications Commission to issue a declaratory ruling in February 2024 that calls using AI-generated voices fall within the Telephone Consumer Protection Act's restrictions on artificial or prerecorded voices, making such robocalls illegal without the called party's prior consent. The political consultant responsible was later fined by the Commission and faced criminal charges in New Hampshire, of which a jury acquitted him in 2025; the regulatory ruling, not the prosecution, proved the lasting consequence. The third is non-consensual content: synthetic speech putting words in a person's mouth to embarrass, harass or defame them, often combined with synthetic video. The fourth is erosion of trust in genuine recordings. Once it is widely known that convincing synthetic speech is easy to produce, people can dismiss real recordings as fakes. This liar's dividend, as legal scholars Robert Chesney and Danielle Citron called it in 2019, may in the long run be as damaging as the fakes themselves. What makes these threats tractable or intractable is largely technical. The questions are whether synthetic speech can be detected, whether it can be marked at creation so that it can be identified later, and whether access to cloning can be controlled. Detection Detecting synthetic speech is the oldest of these approaches. The ASVspoof challenge series, begun in 2015, has provided shared datasets and evaluation protocols for detecting spoofing attacks against automatic speaker verification, including synthesis, voice conversion and replay of recordings. Successive editions have tracked the advance of synthesis methods; ASVspoof 5, held in 2024, included attacks generated with recent neural methods and crowdsourced data recorded in diverse conditions. Detectors learn to distinguish real from synthetic speech from cues that synthesis tends to get wrong. Early cues were obvious: vocoder artefacts in the high frequencies, unnatural phase patterns, absence of fine breath and noise structure. Detectors based on self-supervised encoders such as wav2vec 2.0 and WavLM, fine-tuned on spoofing data, became the strongest approach, because those encoders capture subtle regularities of natural speech that synthesisers do not perfectly reproduce. The fundamental difficulty is generalisation. A detector trained on outputs of known synthesis systems often performs poorly on outputs of new ones, because the artefacts it learned to detect were specific to the systems it saw. Each advance in synthesis quality removes some of the cues detectors rely on. Detectors are also sensitive to channel effects: compression, resampling and transmission over a telephone line can destroy the subtle artefacts they look for, and can make genuine speech look suspicious. Detection is therefore an arms race in which the detector is structurally at a disadvantage, and it is best understood as one layer of defence rather than a solution. Watermarking and provenance Watermarking takes the opposite approach: instead of trying to recognise synthetic speech from its imperfections, the synthesiser embeds an imperceptible signal at creation that a detector can later recognise. If every responsible synthesis system watermarks its output, then speech carrying a watermark can be identified as synthetic, and in some schemes its source can be identified too. Audio watermarking long predates neural synthesis, but neural methods have made it more robust. Meta's AudioSeal, published in 2024, trains a generator that adds a watermark to audio and a detector that recognises it, jointly, with the detector working at the level of individual samples so that it can localise watermarked segments within a longer recording, which matters when synthetic speech is spliced into real recordings. The training includes augmentations that simulate common transformations, such as compression, filtering, resampling and noise, so that the watermark survives them. Google's SynthID, originally developed for images, has been extended to audio produced by Google's generative models. Several commercial synthesis providers watermark their output and offer classifiers that check for their own watermark. Watermarking has limits. It works only if the synthesis system applies it, so open-weight models, which anyone can run without the watermarking step, are not covered. Watermarks can be removed or degraded by an attacker who knows or can estimate the scheme, and robustness to removal trades against imperceptibility and capacity. And a watermark proves only that a recording was produced by a cooperating system; the absence of a watermark does not prove a recording is genuine. Provenance standards address the problem from the other side, by attaching signed metadata to genuine recordings that records how they were captured and edited. The Coalition for Content Provenance and Authenticity, known as C2PA, has developed a standard for such credentials, and some cameras and editing tools support it. For audio, provenance metadata is easily stripped when files are re-encoded or transmitted through telephony, so it complements watermarking rather than replacing it. Access control and consent at the system level The most direct way to reduce misuse is to control who can clone what. Several design measures are in use. Services may require users to affirm that they have consent to clone a voice, and may take action against accounts found to violate this. Some require the voice owner to record a verification statement, often a randomly generated phrase, so that the cloned voice is linked to a live recording by the person concerned; this defeats cloning from found audio alone, though not from a cooperating or coerced speaker. Some maintain lists of voices that may not be cloned, such as prominent political figures, and use speaker verification to reject references that match them. And some keep logs linking generated audio to the account that produced it, so that misuse can be traced. Model release policies are another lever. Microsoft's VALL-E and Meta's Voicebox were described in research papers but not publicly released, explicitly because of misuse concerns. OpenAI previewed a voice cloning system called Voice Engine in March 2024, which could clone a voice from a fifteen-second sample, but limited it to a small number of partners and stated that it was taking a cautious approach to broader release. Google's Translatotron 2 was designed so that its voice-preservation mode could reproduce only the voice of the person actually speaking in the input, which made it unsuitable for impersonation. Meanwhile, capable open-weight zero-shot systems such as XTTS, F5-TTS, CosyVoice and many others are freely available, so the protective effect of withholding any single model is limited. The field's practical consensus is that withholding the most capable systems buys time and sets norms but does not prevent capable cloning from being widely accessible. Voice biometric authentication, which some banks and telephone services use to verify customers, deserves particular mention, because zero-shot cloning directly undermines it. Verification systems can incorporate spoofing detection, and some do, but given the generalisation problems of detection, relying on voice alone for authentication is increasingly hard to justify. Security guidance has moved toward treating voice as one factor among several rather than as a sufficient credential. Law and regulation Legal responses have developed on several fronts. In the United States, the FCC's 2024 ruling brought AI-generated voices in robocalls under existing telephone consumer protection law. Tennessee's ELVIS Act, signed in March 2024, extended the state's protection of a person's name and likeness explicitly to their voice, including simulations of it, with the music industry in mind. Other states have passed laws addressing synthetic media in elections, and federal proposals to create a broader right against unauthorised digital replicas have been introduced in Congress. In the European Union, the AI Act, which entered into force in August 2024, includes transparency obligations in Article 50 that bear directly on synthetic speech. Providers of systems that generate synthetic audio must ensure that outputs are marked in a machine-readable format and detectable as artificially generated, and deployers of systems that generate deepfakes must disclose that the content has been artificially generated or manipulated, with some exceptions for evidently artistic or satirical work. These transparency obligations apply from August 2026. The requirement for machine-readable marking makes watermarking, or an equivalent technical measure, effectively mandatory for providers serving the European market, and it is likely to shape practice well beyond Europe. In China, regulations on deep synthesis services, in force since January 2023, require providers to label synthetic content and to obtain consent before editing biometric information such as voices, and later rules on labelling AI-generated content added more detailed requirements. Labour agreements are a further form of governance. SAG-AFTRA's agreements with film and television studios after the 2023 strikes, and the long dispute with video game companies that followed, established terms for consent and compensation when performers' voices are digitally replicated. These agreements matter beyond their immediate parties, because they establish norms for what consent to voice cloning means in practice: specific, informed, tied to particular uses and revocable for some purposes. What builders should do For anyone building systems with voice cloning capability, the technical and regulatory landscape implies a set of practices that are becoming standard. Watermark all synthetic output with a robust scheme and provide a means for third parties to check for the watermark. Verify consent for cloned voices through a mechanism stronger than a checkbox, ideally involving a live recording by the voice owner. Block cloning of voices on protected lists and of references that appear to be scraped from public figures. Log generation in a way that allows misuse to be traced without unnecessary retention of personal data. Disclose synthetic speech to listeners where they might otherwise assume it is human, as in customer service calls. And treat voice biometrics as a weak factor in any authentication flow the system touches. None of these measures is complete on its own, and some are in tension with others: logging aids accountability but raises privacy concerns, and verification recordings are themselves voice data that must be protected. But together they make a meaningful difference to how easily a given system can be misused, and increasingly they are not optional. The limits of technical defence The deeper point is that the technical defences described here can reduce misuse but cannot eliminate it, because the capability is now widely distributed. Open-weight models capable of convincing zero-shot cloning run on consumer hardware, and no watermarking scheme or access control can reach them. The long-term defence against voice impersonation will therefore be partly social: a general understanding that a familiar voice on a telephone is not proof of identity, and verification habits, such as calling back on a known number or agreeing family code words, that do not depend on the voice. Several consumer protection agencies already recommend exactly these habits. The voice, which for all of human history has been one of the most reliable signals of who is speaking, has become a signal that can be forged cheaply, and institutions and individuals are still adjusting to that fact. Conclusion The argument of this booklet has been that the representation layer is where the speech stack's real decisions are made. Having followed that argument through transcription, synthesis, voice preservation, cross-lingual transfer, latency and misuse, it is worth asking what follows from it: what the argument predicts, what it leaves open, and what someone working with these systems should do differently because of it. What the argument predicts If the token is the contract between components, then improvements in tokenisation should propagate across the whole stack, and they have. The move from 75-frame-per-second codecs to 12.5-frame-per-second codecs did not merely make synthesis cheaper; it made full-duplex dialogue on a single graphics card possible, because it cut the number of sequential steps per second of audio by a factor of six. Semantic distillation into the first codebook did not merely improve codec language models; it made the same token stream usable for content modelling and for reconstruction, which is what allowed Moshi's architecture to work at all. Supervised semantic tokens did not merely improve CosyVoice's word error rate; they gave cross-lingual synthesis a content stream that was largely free of the reference speaker's accent. In each case a change at the representation layer opened capabilities several components away. The argument also predicts where the next improvements will come from. Three are visible already. The first is further reduction of frame rate and token count without loss of intelligibility or identity, through codecs whose decoders are increasingly generative models in their own right. Every halving of the token rate halves the sequential work of every model that reads or writes those tokens. The second is better factorisation: tokenisers that place content, prosody, identity and channel in separately addressable parts of the representation, so that a model can change one while holding the others fixed. Explicitly factorised codecs exist, but none has yet clearly displaced the simpler semantic-first designs, and the field has not settled how to separate prosody from content in languages where pitch is lexical. The third is tighter integration with text language models, so that speech tokens and text tokens share a model whose linguistic knowledge comes mostly from text. This is already the dominant design for new systems, and it shifts the main research question from how to model speech to how to map speech into and out of a model that already understands language. What remains unsettled Several questions remain genuinely open, and a reader should treat confident claims about them with caution. Evaluation is the most important. Word error rate measured by a recogniser, speaker similarity measured by a verification model and naturalness measured by a learned predictor have each become standard because they are cheap and reproducible, but each measures something narrower than what listeners care about. Most current zero-shot synthesis systems are near the floor of the first and the ceiling of the second on standard test sets. The differences that remain, in long-form consistency, appropriateness of prosody, fidelity to a speaker's habitual style, and nativeness in a second language, are not well captured by any automatic metric. Until better measures exist, claims that one system is better than another should be read with attention to what was actually measured. The definition of voice preservation is unsettled in a deeper sense. A cross-lingual system must decide whether to keep the speaker's accent or remove it, and both choices are defensible. A clinical system must decide whether to reproduce speech already affected by disease or to repair it. A dubbing system must decide how much of the original performer's prosody to keep when the target language's prosody works differently. These are not engineering questions with correct answers. They are choices about what a voice is, and they should be made deliberately, with the people whose voices are involved, rather than by default through whatever the model happens to do. The balance between cascaded and end-to-end conversational architectures is also unsettled. End-to-end models have demonstrated latency close to the human benchmark and the ability to handle paralinguistic information that cascades discard. Cascades remain easier to control, to audit and to update, because each component can be inspected and replaced, and because the text in the middle provides a natural point at which to apply safety filters and business logic. Many production systems still use optimised cascades, and the gap in latency between a well-optimised cascade and an end-to-end model is smaller than the gap between either and a naive cascade. Which architecture prevails may depend less on raw capability than on which is easier to make reliable. Governance is unsettled most of all. Watermarking is becoming mandatory in some jurisdictions but cannot reach open-weight models. Detection generalises poorly. Consent mechanisms can be circumvented. The legal treatment of voice as a protected attribute is developing unevenly across jurisdictions. It is not yet clear whether the combination of technical, legal and social defences will be enough to preserve the voice as a usable signal of identity, or whether institutions will simply have to stop treating it as one. What to do differently For an engineer building a voice product, the practical implication is to choose representations first and models second. Before choosing a synthesis model or a recogniser, decide what the system must control, what it must preserve and how fast it must respond, then choose tokenisers and interfaces that make those requirements achievable. A system that must respond in under 300 milliseconds needs a low-frame-rate representation, a causal encoder and streaming synthesis from partial text; no amount of later optimisation will rescue a design that chose otherwise. A system that must carry voices across languages needs a content representation that is largely speaker-free and an explicit representation of language identity. A system that will be used for clinical voice banking needs a way to separate timbre from articulation. For a researcher, the implication is that work on representations has outsized leverage. A better tokeniser improves recognition, synthesis, conversion and dialogue at once. So does a better evaluation method, which is the most underinvested part of the field relative to its importance. For anyone buying or deploying these systems, the implication is to ask the questions the representation layer raises. What frame rate and token budget does the system use, and what does that imply for latency at scale? What happens to the speaker's accent when the system crosses languages, and can that be controlled? How does the system behave on silence, noise and unusual voices, and how were those cases evaluated? Is synthetic output watermarked, and how is consent for cloned voices verified? For policymakers, the implication is that the technical capability to clone a voice from a few seconds of audio is now general, cheap and widely distributed, and that policy built on the assumption that it can be contained will fail. Policy that requires marking of synthetic output by commercial providers, that protects voice as a personal attribute, that sets standards for consent and that encourages verification practices not dependent on voice is more likely to work, because it does not rely on the capability being scarce. The thread The thread through all of this has been simple. Speech carries a message that needs tens of bits per second, an identity that needs a few hundred bits once, a manner of delivery that needs a few hundred bits per second and acoustic detail that needs thousands. The systems that work best are those whose representations respect this structure, separating what can be separated and making explicit what must be supplied. When a system fails, by hallucinating text in a silence, drifting away from a cloned voice, carrying an accent where it was not wanted or pausing too long before a reply, the failure can almost always be traced to a point where the representation carried the wrong information, at the wrong rate, in the wrong place. Learning to see that point is the main skill the field now requires. Glossary Acoustic token. A discrete symbol produced by a neural codec's quantiser, carrying the information needed to reconstruct audio in detail, including speaker, prosody and recording conditions as well as content. Autoregressive model. A model that generates a sequence one element at a time, each conditioned on all previous elements. CTC (connectionist temporal classification). A training objective and decoding method that lets a network emit a label sequence from a longer sequence of frames without a predetermined alignment, using a blank symbol and collapsing repeats. Codebook. A learned table of vectors used by a quantiser; each input vector is replaced by the index of its nearest entry. Endpointing. Deciding when a speaker has finished their turn, so that a system can respond. Flow matching. A method for training continuous generative models that learn a velocity field transporting noise to data along approximately straight paths, closely related to diffusion. Frame rate. The number of representation vectors or token positions per second of audio. Fundamental frequency (F0). The rate of vocal-fold vibration in voiced speech, perceived as pitch. Mel spectrogram. A time-frequency representation of audio whose frequency axis is warped to approximate human pitch perception, with log-compressed magnitudes and no phase. Neural codec. A learned encoder, quantiser and decoder trained jointly to compress and reconstruct audio at a low bitrate. Residual vector quantisation (RVQ). A quantisation scheme using a stack of codebooks, each quantising the residual error left by the previous ones, producing coarse-to-fine tokens. RNN-T (recurrent neural network transducer). A recognition architecture that combines an acoustic encoder, a prediction network over previous outputs and a joint network, well suited to streaming. Self-supervised learning. Training a model on unlabelled data with an objective derived from the data itself, such as predicting masked portions. Semantic token. A discrete unit, typically from clustering a self-supervised encoder's representations or from a quantised recogniser, that encodes mainly phonetic content. Speaker embedding. A fixed-length vector summarising a speaker's voice, produced by a model trained for speaker verification. Vocoder. The component that converts an intermediate representation, such as a mel spectrogram or codec tokens, into a waveform. Word error rate (WER). The number of word substitutions, deletions and insertions needed to turn a recogniser's output into the reference transcript, divided by the number of reference words. Zero-shot voice cloning. Synthesising speech in the voice of a speaker not seen during training, from a short reference recording supplied at inference time. Further Reading The literature of this field lives mostly in conference papers and preprints. The following are the primary sources most worth reading in full, grouped by the chapters they support. Representations and self-supervision Baevski, Zhou, Mohamed and Auli, "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations" (NeurIPS 2020). The paper that made self-supervised pretraining the default starting point for speech recognition, with its demonstration of usable recognition from minutes of labelled data. Hsu, Bolte, Tsai, Lakhotia, Salakhutdinov and Mohamed, "HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units" (IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021). The source of the clustering-based targets that became the standard recipe for semantic units. Chen and colleagues, "WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing" (IEEE Journal of Selected Topics in Signal Processing, 2022). Explains the denoising objective that makes WavLM strong on speaker and overlap tasks, and hence ubiquitous in voice conversion and evaluation. Lakhotia and colleagues, "On Generative Spoken Language Modeling from Raw Audio" (Transactions of the ACL, 2021). The first systematic study of language models trained on discovered speech units, with careful evaluation of unit choices. Neural codecs Zeghidour, Luebs, Omran, Skoglund and Tagliasacchi, "SoundStream: An End-to-End Neural Audio Codec" (2021). Introduces residual vector quantisation with quantiser dropout in its now-standard form. Défossez, Copet, Synnaeve and Adi, "High Fidelity Neural Audio Compression" (2022). The EnCodec paper, clear on the bitrate arithmetic and the loss balancing that make codecs trainable. Kumar, Seetharaman, Luebs, Kumar and Kumar, "High-Fidelity Audio Compression with Improved RVQGAN" (NeurIPS 2023). The DAC paper, useful for its analysis of codebook utilisation. Défossez and colleagues, "Moshi: a speech-text foundation model for real-time dialogue" (Kyutai, 2024). Describes both the Mimi codec and the Moshi dialogue model; the single most useful document on how representation choices determine conversational latency. Recognition Radford, Kim, Xu, Brockman, McLeavey and Sutskever, "Robust Speech Recognition via Large-Scale Weak Supervision" (2022). The Whisper paper, essential for the multitask format and for its frank discussion of evaluation and normalisation. Gulati and colleagues, "Conformer: Convolution-augmented Transformer for Speech Recognition" (Interspeech 2020). The encoder design that most production recognisers now use. Pratap and colleagues, "Scaling Speech Technology to 1,000+ Languages" (2023). The Massively Multilingual Speech paper, on extending recognition and synthesis to the long tail of languages. Vocoders and synthesis Kong, Kim and Bae, "HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis" (NeurIPS 2020). The reference GAN vocoder; its multi-period discriminator is worth understanding in detail. Siuzdak, "Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis" (2023). A concise case for frequency-domain vocoding. Wang and colleagues, "Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers" (2023). The VALL-E paper, which reframed synthesis as language modelling. Le and colleagues, "Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale" (NeurIPS 2023). The flow-matching alternative, and a good account of synthesis as infilling. Du and colleagues, "CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens" (2024). The clearest statement of the hybrid architecture and of supervised semantic tokens. Chen and colleagues, "F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching" (2024). A widely used open system and a readable account of alignment-free non-autoregressive synthesis. Cross-lingual transfer and translation Zhang and colleagues, "Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling" (2023). The VALL-E X paper, on language tokens as a control for accent. Seamless Communication team, "Seamless: Multilingual Expressive and Streaming Speech Translation" (Meta, 2023). Documents SeamlessExpressive and SeamlessStreaming, including their approach to preserving prosody across languages. Conversation and turn-taking Stivers and colleagues, "Universals and cultural variation in turn-taking in conversation" (Proceedings of the National Academy of Sciences, 2009). The cross-linguistic study behind the human latency benchmark. Levinson and Torreira, "Timing in turn-taking and its implications for processing models of language" (Frontiers in Psychology, 2015). Explains why short gaps imply that listeners plan replies while still listening, the cognitive fact that full-duplex models try to emulate. Misuse and governance San Roman and colleagues, "Proactive Detection of Voice Cloning with Localized Watermarking" (ICML 2024). The AudioSeal paper, on sample-level watermark detection. Chesney and Citron, "Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security" (California Law Review, 2019). The source of the liar's dividend concept and still the best general framing of the legal issues. Regulation (EU) 2024/1689, the Artificial Intelligence Act, Article 50. The transparency obligations for synthetic audio that apply in the European Union from August 2026. Hashtags: #SpeechFoundationModels #SpeechAI #ArtificialIntelligence #DeepLearning #SpeechRecognition #AutomaticSpeechRecognition #EndToEndSpeechRecognition #SpeechSynthesis #TextToSpeech #NeuralTextToSpeech #VoicePreservation #VoiceCloning #ZeroShotVoiceCloning #SpeakerIdentity #SpeakerEmbeddings #VoiceConversion #CrossLingualVoiceTransfer #SpeechToSpeechTranslation #MultilingualSpeechAI #NeuralAudioCodecs #SpeechTokenization #SemanticTokens #AcousticTokens #SelfSupervisedSpeechLearning #WhisperAI #Wav2Vec2 #HuBERT #NeuralVocoders #RealTimeSpeechAI #ResponsibleVoiceAI

Download the Book (PDF): Introduction: The Model Has to Fit the Machine A trained neural network is, at the moment training ends, a large file of floating-point numbers and a description of how to combine them. On a workstation with a data-centre GPU, that file is a minor detail. It loads in a second, it sits comfortably in tens of gigabytes of memory, and the arithmetic that consumes it is fed by a power supply that thinks nothing of drawing several hundred watts. Move the same network to a phone, a doorbell camera, a hearing aid, a vibration sensor bolted to a factory pump, or a battery-powered tag on a shipping pallet, and every one of those comfortable assumptions fails at once. The microcontroller in the sensor may have 256 kilobytes of SRAM and two megabytes of flash. It may run at a few hundred megahertz with no operating system, no file system, and no floating-point unit worth the name. It may have to last three years on a coin cell. The phone is far richer, but it shares its memory bandwidth with the display, the camera pipeline, and every other application, and it throttles itself the moment the chassis warms. Neither device will tolerate a model that was designed with no thought for where it would run. This booklet is about closing that gap: taking neural networks that work and making them run well on devices that are small, cheap, power-limited, and frequently disconnected. The techniques have names that every practitioner eventually meets — quantization, pruning, knowledge distillation, hardware acceleration on neural processing units — and there is no shortage of papers describing each. What is in short supply is a clear account of how they fit together, why some of them pay off handsomely on one device and not at all on another, and how a working developer should decide what to do first. Why run the model on the device at all It is worth being plain about the reasons, because they shape every later decision. The first is latency. A round trip to a cloud endpoint costs tens to hundreds of milliseconds before the model even starts, and the variance is worse than the mean. A wake-word detector, a gesture recogniser, a safety interlock on a machine tool, or a camera that must decide whether to record cannot wait on a cellular link. On-device inference turns an unpredictable network delay into a predictable compute delay that the engineer controls. The second is availability. Devices go into basements, fields, mines, aircraft holds, and moving vehicles. A product whose core function disappears when the connection does is a fragile product. Offline inference makes intelligence a property of the device rather than a property of the network. The third is privacy and data minimisation. Audio that never leaves the microphone cannot be intercepted, subpoenaed, or mishandled on a server. For health monitoring, home devices, and anything that listens or watches, processing locally and transmitting only a result — "a person is present", "the bearing is degrading" — is often the only design regulators and customers will accept. The fourth is energy and cost at scale. Radios are expensive in energy. On a typical low-power sensor node, transmitting raw data costs far more than computing on it locally, so a model that reduces a stream of samples to an occasional event can extend battery life dramatically. At fleet scale, the same logic applies to money: millions of devices each sending a few seconds of audio an hour to a server is a substantial inference bill that on-device processing simply removes. None of these benefits arrive automatically. They arrive only if the model actually fits, runs fast enough, and keeps enough of its accuracy after being made to fit. That is the engineering problem this booklet addresses. The controlling idea The single idea that organises everything that follows is this: on edge devices, the scarce resource is memory — how much of it a model occupies and how many bytes it moves — far more than arithmetic, and every optimization technique should be judged by what it does to memory on the specific hardware that will run it, measured on that hardware. That claim sounds narrow, but it reorders the whole subject. It explains why converting a network from 32-bit floating point to 8-bit integers is nearly always the first and most valuable step: it cuts storage and memory traffic by a factor of four before it does anything to arithmetic. It explains why removing ninety per cent of a network's weights can produce a model that is no faster at all, because scattered zeros in a dense layout still have to be fetched or skipped with bookkeeping that costs more than it saves. It explains why a small student network distilled from a large teacher is often a better bet than an aggressively compressed version of the teacher itself. And it explains why a neural processing unit with impressive headline throughput can deliver disappointing results on a model whose activations do not fit in its local SRAM, or whose operators it cannot execute, forcing the work back onto the CPU. The second half of the idea — measured on the target — matters as much as the first. Edge optimization is full of proxies that mislead: parameter counts that ignore activation memory, operation counts that ignore memory traffic, desktop benchmarks that ignore the target's cache sizes and instruction set, and accuracy figures computed in floating point that silently differ from what the integer kernels on the device actually compute. The developer who measures on the device, early and repeatedly, will out-ship the developer who optimizes proxies. What this booklet covers, and what it leaves out The chapters run in the order a practitioner meets the problems. The first sets out the budget: how memory, energy, and latency interact on real edge hardware, and why moving data dominates computing on it. The second argues that the most powerful optimization happens before any compression technique is applied, in the choice and sizing of the architecture itself. The next four chapters take the main compression techniques in turn. Two are devoted to quantization, because it is the technique with the largest and most reliable payoff and the one where subtle mistakes cost the most accuracy: the first explains the integer arithmetic precisely, the second deals with the cases where simple post-training quantization breaks and what to do about them. A chapter on pruning separates the kinds of sparsity that hardware can exploit from the kinds that only look good in a paper. A chapter on knowledge distillation explains how to train small networks that are better than their size suggests. The final three chapters move from the model to the machine. One examines the specialised accelerators — micro-NPUs in microcontrollers and the larger neural engines in phones — and what they demand of a model. One covers the toolchains that turn a trained network into something a device can execute, and how to profile the result honestly. The last works through a complete pipeline, from requirements to a deployed and monitored model, showing the order in which decisions should be made and where they typically go wrong. Some things are deliberately left out. Training on the device, federated learning, and on-device personalisation are real and growing fields, but they are a different problem from making inference fit. Analogue and in-memory computing are promising but not yet where most developers will ship. Large language models appear where they illuminate a technique — weight-only quantization to four bits was developed largely for them, and phones now run models with a few billion parameters — but this is not a guide to deploying chat assistants. The focus is on the broad middle of edge AI: vision, audio, sensor, and small language models running on phones, embedded Linux boards, and microcontrollers. How to read it The book assumes that you can train a neural network in a mainstream framework and that you know what a convolution, a matrix multiplication, and a learning rate are. It does not assume embedded-systems experience; where hardware detail matters, it is explained. Code appears only where a short fragment makes a mechanism concrete. Where the text cites specific numbers, they come from named sources listed in the notes, and where a figure varies too much across devices to be stated honestly, the text says so rather than inventing one. Read straight through if the subject is new. If you already ship models to devices and want a specific answer, the chapters stand reasonably well alone, but the first chapter's account of the memory budget is the lens through which the rest is written, and it repays a few minutes even for experienced readers. The field moves quickly. Toolchains have been renamed within the last two years — TensorFlow Lite is now LiteRT, and PyTorch's ExecuTorch reached its 1.0 release in October 2025 — and new accelerators appear every few months. The principles in this booklet change much more slowly. A model that respects the memory budget, uses arithmetic the hardware supports natively, and is validated on the target will run well on next year's chips as well as this year's. CHAPTER 1 The Edge Budget: Memory, Energy, and Time Every edge deployment begins, whether the developer acknowledges it or not, with a budget. The device has a fixed amount of storage, a fixed amount of working memory, a fixed rate at which it can move data, a ceiling on the power it can draw and the heat it can shed, and a deadline by which an answer is needed. The model must fit inside all of these at once. Exceeding any one of them is fatal: a model that is accurate and fast but needs twice the available RAM simply does not run. The common mistake is to think of this budget mainly in terms of arithmetic — how many multiply-accumulate operations the network needs per inference, and how many the processor can execute per second. Arithmetic matters, but on edge hardware it is rarely the binding constraint. The binding constraints are almost always about memory: how many bytes the model occupies, how many bytes are alive at the same moment during inference, and how many bytes must be moved between levels of the memory hierarchy to get the answer. This chapter sets out that budget concretely, because every later technique is best understood as a way of spending it more wisely. Three classes of device "Edge" covers an enormous range of hardware, and the budget looks very different across it. Three classes are worth distinguishing. Phones and tablets carry system-on-chip processors with several CPU cores, a capable GPU, and, on nearly every recent flagship and many mid-range parts, a dedicated neural processing unit. They have gigabytes of DRAM, tens of gigabytes per second of memory bandwidth, and abundant storage. Their constraints are subtler than size. Memory bandwidth is shared with everything else on the device. Sustained workloads cause thermal throttling within minutes, so a model that benchmarks well for ten seconds may run markedly slower in a twenty-minute video call. Battery drain is visible to users and punished in app-store reviews. And the hardware is fragmented: an Android application may meet hundreds of different chipsets, each with its own accelerator and driver quirks. Embedded application processors — the class that runs embedded Linux in smart cameras, industrial gateways, robots, and appliances — sit in the middle. They typically have one to four Arm Cortex-A cores, hundreds of megabytes to a few gigabytes of DRAM, and increasingly a modest NPU rated at somewhere between half a TOPS and a few TOPS (trillions of operations per second, almost always quoted for 8-bit integer arithmetic). They run an operating system with virtual memory and a file system, which makes deployment familiar, but they are cost-sensitive parts where every dollar of bill-of-materials matters across a production run. Microcontrollers are the extreme. A typical modern part built around an Arm Cortex-M core has between 128 kilobytes and a few megabytes of on-chip SRAM, between 512 kilobytes and a few megabytes of on-chip flash, a clock somewhere between 50 and 800 megahertz, and no operating system beyond, perhaps, a small real-time kernel. Power consumption in active mode is measured in milliwatts, and in sleep in microwatts. These are the devices that end up in sensors, wearables, and battery-powered equipment by the billions, and they are where the budget is tightest and the techniques in this booklet matter most. The machine-learning community that targets them calls itself TinyML, after the book of that name by Pete Warden and Daniel Situnayake.1 The same principles apply across all three classes, but the numbers shift by three or four orders of magnitude, and so does the relative importance of each constraint. On a phone, the question is usually whether a model runs fast enough without draining the battery or overheating. On a microcontroller, the first question is whether it fits at all. Storage and working memory are different budgets A model consumes two kinds of memory, and confusing them is the most common source of early failure. The first is storage for the weights: the learned parameters, plus the description of the graph. On a phone or embedded Linux board, weights live in a file and are loaded or memory-mapped into DRAM. On a microcontroller, they usually live in flash and are read directly from it during inference, without ever being copied into RAM — a technique called execute-in-place for code and, by analogy, read-in-place for weights. This is why a microcontroller with 256 kilobytes of SRAM can run a model with 400 kilobytes of weights: the weights never need to be in SRAM at all, as long as flash read speed is adequate. The second is working memory for activations: the intermediate tensors produced by each layer and consumed by the next. These must be writable, so on a microcontroller they live in SRAM. Activations are transient — a tensor can be discarded once every layer that reads it has run — so the memory needed is not the sum of all activation tensors but the maximum, over the course of inference, of the tensors that are alive at the same moment. That figure is called peak activation memory, and it is frequently the binding constraint on small devices. Peak activation memory behaves very differently from parameter count. Convolutional networks for vision typically have their largest activations in the early layers, where spatial resolution is high, and their largest weights in the late layers, where channel counts are high. A network can therefore have very few parameters and still need a great deal of RAM. MobileNetV2, one of the most widely used mobile architectures, has roughly 3.4 million parameters and around 300 million multiply-accumulates at its standard 224-by-224 input; its early layers produce tensors of 112 by 112 spatial positions with 32 or more channels, so even in 8-bit form a single such tensor occupies around 400 kilobytes, and a layer needs its input and output alive simultaneously. That network, as designed, will not run in the SRAM of most microcontrollers regardless of how its weights are compressed. The researchers behind MCUNet, discussed in the next chapter, made exactly this observation: the activation footprint, not the weight footprint, was the barrier to running ImageNet-scale vision on microcontrollers.2 Runtimes for microcontrollers make this budget explicit. LiteRT for Microcontrollers — the runtime formerly called TensorFlow Lite Micro — requires the developer to provide a single statically allocated block of memory, the tensor arena, from which all activations and scratch buffers are carved at initialisation time.3 No dynamic allocation happens during inference. The runtime plans the arena by examining the lifetime of every tensor and packing tensors whose lifetimes do not overlap into the same addresses. If the plan does not fit, initialisation fails, and the developer learns precisely how many bytes are missing. That explicitness is a virtue: it forces the activation budget into view at the start of a project rather than at the end. On phones and Linux boards the same two budgets exist, but they are softer. Exceeding available memory does not fail cleanly; it causes paging, cache thrashing, or the operating system killing the application. The discipline of knowing both numbers — weight bytes and peak activation bytes — is just as valuable there. Moving data costs more than computing on it The deeper reason memory dominates is energy. In digital hardware, the energy cost of an arithmetic operation is small compared with the energy cost of fetching its operands from anywhere other than a register, and the cost rises steeply the further away the data lives. The most widely cited figures come from Mark Horowitz's analysis of a 45-nanometre process, which Song Han and colleagues reproduced when making the case for network compression.4 The absolute values belong to an older process node and have all fallen since, but the ratios between them — which are what matter — have remained broadly similar, because they are governed by the physics of driving wires and memory arrays rather than by transistor switching. Table 1 sets them out. Table 1. Approximate energy per operation in a 45 nm process. Operation Energy (pJ) Relative to int add 32-bit integer add 0.1 1 32-bit float add 0.9 9 32-bit integer multiply 3.1 31 32-bit float multiply 3.7 37 32-bit read, small SRAM 5 50 32-bit read, off-chip DRAM 640 6,400 Source: M. Horowitz, 45 nm energy table, as reproduced in Han et al., "Learning both Weights and Connections for Efficient Neural Networks" (NeurIPS 2015). Two lessons follow. First, a read from off-chip DRAM costs over a hundred times more than a read from a small on-chip SRAM, and thousands of times more than an addition. A network whose weights stream from DRAM on every inference spends most of its energy on that streaming, not on the arithmetic. Second, narrower data is cheaper twice over: an 8-bit integer operation costs a small fraction of a 32-bit floating-point one — Horowitz's ISSCC 2014 keynote put an 8-bit integer addition at around 0.03 picojoules and an 8-bit multiplication at around 0.2 — and, more importantly, four times as many 8-bit values fit through the same memory port, into the same cache line, and into the same SRAM. This is the single most important fact in edge AI, and it is why the techniques in this booklet are ranked as they are. Quantization reduces bytes moved and stored. Architectural choices that keep activations small enough for on-chip SRAM avoid the most expensive tier of the hierarchy entirely. Pruning helps only when the hardware can avoid fetching the pruned weights. Accelerators win mainly because they place large local SRAM next to the arithmetic units and schedule data movement carefully, not merely because they have more multipliers. Arithmetic intensity and the memory wall A useful way to make this precise is arithmetic intensity: the number of arithmetic operations performed per byte moved from memory. Every processor has a ratio between its peak compute rate and its memory bandwidth. If a workload's arithmetic intensity is below that ratio, the processor spends its time waiting for data and the workload is memory-bound; above it, the arithmetic units are the bottleneck and the workload is compute-bound. This is the reasoning behind the roofline model, a standard performance-analysis tool in high-performance computing that applies just as well to a phone NPU. Different layer types sit in very different places. A standard 3-by-3 convolution over a large feature map reuses each weight at every spatial position and each input value across many output channels; its arithmetic intensity is high and it tends to be compute-bound. A fully connected layer evaluated on a single input — the normal case at the edge, where batch size is one — uses each weight exactly once. Its arithmetic intensity is roughly one multiply-accumulate per weight fetched, which places it firmly in memory-bound territory on almost any hardware. Depthwise convolutions, which the next chapter discusses, have low arithmetic intensity as well, because each filter touches only one channel. The consequence for language models on phones is stark and easy to compute. Generating each token of output from a transformer requires reading essentially all of the model's weights once. If a three-billion-parameter model is stored at four bits per weight, that is about 1.5 gigabytes per token. A phone with 60 gigabytes per second of usable memory bandwidth can therefore generate at most around forty tokens per second from that model, however many TOPS its NPU advertises — and in practice somewhat fewer, because bandwidth is shared and never fully utilised. Halve the bytes per weight and the ceiling doubles. No amount of additional arithmetic capability changes this bound; only moving fewer bytes does. The same reasoning applies, at smaller scale, to a microcontroller reading weights from flash. Flash is slower to read than SRAM, and many microcontrollers place a small cache or prefetch buffer in front of it. A model whose weights are read once per inference in a predictable order streams well; one whose access pattern jumps around defeats the cache and runs far slower than its operation count predicts. Latency is more than inference time The third element of the budget is time. Developers usually measure the time from handing the runtime an input tensor to receiving an output tensor, and that number matters. But the end-to-end latency a user or a control loop experiences includes several other components that are easy to overlook. Pre-processing converts raw sensor data into the tensor the model expects. For audio this typically means computing a spectrogram or mel-frequency features from a window of samples, which on a microcontroller can cost as much as a small neural network. For vision it means resizing, cropping, colour conversion, and normalisation. On a phone, the camera frame may need to be copied from a hardware buffer to a CPU-accessible one, and copies of multi-megabyte images are not free. Accelerator dispatch adds overhead whenever work moves between processors. Handing a graph or a sub-graph to an NPU or GPU involves driver calls, synchronisation, and sometimes copying tensors between memory regions. For a large model this overhead is amortised; for a small one it can dominate. A model that runs in two milliseconds on the CPU may take four on an accelerator if dispatch costs three. Post-processing turns raw outputs into decisions: non-maximum suppression for object detectors, smoothing and thresholding for keyword detectors, decoding for sequence models. These steps often run on the CPU and are rarely optimised with the same care as the network. Cold start is the time to load the model, initialise the runtime, and, on some accelerators, compile the graph for the specific hardware. On phones, first-run accelerator compilation can take seconds. Applications that load a model on demand will feel sluggish unless this cost is hidden or cached. Finally, latency must be considered under realistic conditions. A phone benchmark run immediately after boot, with the device cool and nothing else running, measures the best case. The relevant figure is the sustained latency after several minutes of continuous operation, with the application's own user interface and the camera pipeline competing for the same resources. Energy per inference and duty cycling For battery-powered devices, the meaningful metric is not power but energy per inference, measured in millijoules or microjoules, combined with how often inference runs. A model that draws 50 milliwatts for 20 milliseconds uses one millijoule per inference. Run once per second, that is an average of one milliwatt — manageable for a device with a reasonable battery. Run continuously, it is 50 milliwatts, which drains a coin cell in hours. This is why so many edge systems are structured as cascades. A tiny, always-on model — sometimes running on a dedicated low-power core or even on dedicated hardware within the sensor — watches for a cheap signal: sound above a threshold, motion in the frame, a vibration pattern outside the normal envelope. Only when it fires does the device wake a larger model on a more capable processor. Smart speakers use exactly this structure, with a minimal wake-word detector gating a far larger speech pipeline. The design principle is that the expensive model should run as rarely as possible, and the cheap model should be cheap enough to run always. Energy per inference also depends heavily on how quickly the processor can return to sleep. A faster inference, even at higher instantaneous power, often uses less total energy, because the static power of the rest of the system — memory, clocks, voltage regulators — is paid for every millisecond the device is awake. This "race to sleep" logic is one of the strongest arguments for hardware acceleration on microcontrollers: a micro-NPU that finishes an inference ten times sooner lets the whole chip return to its microwatt sleep state ten times sooner. Writing the budget down The practical outcome of this chapter is a habit: before choosing an architecture or applying any compression technique, write the budget down as numbers. For a given target, that means at least the following. • Maximum weight storage, in bytes, after accounting for application code, the runtime, and any over-the-air update scheme that needs a second copy of the firmware. • Maximum working memory available for activations and scratch buffers, after accounting for the stack, the heap, sensor buffers, and communication buffers. • Latency deadline, end to end, including pre- and post-processing, at sustained rather than peak performance. • Energy per inference and inference rate, if the device is battery-powered, and the resulting average power. • The accuracy floor below which the product does not work, measured on data that represents the deployed environment rather than the benchmark. Each number should have a margin. Firmware grows, sensor buffers get larger, and a model that fits with three kilobytes to spare in the first prototype will not fit in the shipping product. With these numbers in hand, every later decision becomes a question of trade-offs within a known envelope rather than a search for "the best model". A network that exceeds the activation budget must change its architecture or input resolution; no amount of weight compression will help. A network that fits but misses its latency deadline needs faster arithmetic — integer kernels, an accelerator, or fewer operations. A network that fits and is fast enough but drains the battery needs fewer bytes moved per inference or fewer inferences. The techniques in the rest of this booklet are tools for each of those situations, and knowing which situation you are in is most of the battle. CHAPTER 2 Architecture First: Designing Within the Budget Most discussions of edge AI start with compression: take a large model and shrink it. That framing is natural, because compression techniques are well defined and make for satisfying before-and-after comparisons. But it gets the order of operations wrong. The most powerful decision a developer makes is the choice of architecture and its size, and that decision is made before any compression technique is applied. A network designed for the budget from the outset, then quantized, will almost always beat a network designed for a server and then compressed until it fits. The reason is simple. Compression techniques remove redundancy. They can take a model that is twice as large as it needs to be and bring it close to its necessary size, and with care they can go somewhat further. What they cannot do is change the shape of the computation: the spatial resolution at which early layers run, the way channels interact, the operators used, the depth of the network. Those choices determine the peak activation memory, the arithmetic intensity, and the operator compatibility that the previous chapter identified as the real constraints. Get them wrong, and compression is fighting the architecture. The factorised convolution The single architectural idea that made mobile vision practical is the depthwise separable convolution, popularised by Google's MobileNets in 2017.1 It is worth understanding precisely, because its strengths and weaknesses explain much of what follows. A standard convolution with a k-by-k kernel, C_in input channels, and C_out output channels does two things at once. It mixes information spatially, combining each pixel with its neighbours, and it mixes information across channels, combining every input channel into every output channel. For a feature map of H by W positions, it costs k² × C_in × C_out × H × W multiply-accumulates. A depthwise separable convolution splits those two jobs. First, a depthwise convolution applies one k-by-k filter to each input channel separately, mixing spatially but not across channels; it costs k² × C_in × H × W. Then a pointwise convolution — a 1-by-1 convolution — mixes across channels at each position; it costs C_in × C_out × H × W. The ratio of the factorised cost to the standard cost works out to 1/C_out + 1/k². With 3-by-3 kernels and a reasonably large number of output channels, that is a little over one-ninth: roughly an eight- to nine-fold reduction in arithmetic and a similar reduction in parameters, for a modest loss in accuracy. MobileNetV1 built a whole network from these blocks and added two knobs that remain the simplest and most effective way to fit a network to a budget. The width multiplier scales the number of channels in every layer by a factor, typically between 0.25 and 1.0; because both parameters and computation scale roughly with the square of channel count in pointwise layers, halving the width cuts cost by close to four. The resolution multiplier scales the input image size; because computation and activation memory scale with the number of spatial positions, halving resolution cuts both by four. A designer with a fixed budget can trade between them, and the MobileNet paper's experiments showed that accuracy falls off smoothly rather than abruptly as either knob is turned down. MobileNetV2, published the following year, introduced the inverted residual block that dominates efficient architectures to this day.2 Each block takes a thin input, expands it with a 1-by-1 convolution to a wider intermediate representation (typically six times wider), applies a depthwise convolution in that wide space, then projects back down to a thin output with another 1-by-1 convolution — with no activation function after the projection, a choice the authors called a linear bottleneck, because applying a non-linearity in a low-dimensional space destroys information. A residual connection joins the thin input to the thin output. The design keeps the tensors that pass between blocks small, which is good for memory, while doing the expressive work in a wider space inside the block. MobileNetV3 refined this with neural architecture search and with small, hardware-motivated choices.3 Its designers replaced the swish activation with "hard swish", a piecewise-linear approximation that is cheap to compute and, importantly, behaves well under integer quantization, where smooth functions involving exponentials are awkward. That is a recurring lesson: an operation that is cheap in floating point on a GPU may be expensive or numerically fragile in 8-bit integers on a microcontroller, and architectures intended for the edge should be chosen with the integer version in mind. When efficiency on paper is not efficiency on silicon The depthwise separable convolution reduces operation counts dramatically, and on CPUs, where performance tracks operation counts reasonably well, it delivers. But it has a weakness that matters on accelerators. Depthwise convolutions have very low arithmetic intensity: each filter touches only one channel, so there is little reuse of data, and the operation tends to be limited by memory movement. Accelerators built around large arrays of multiply-accumulate units — systolic arrays and their relatives — achieve their headline throughput only when the work can be laid out to keep the whole array busy. A depthwise layer often cannot, and utilisation drops sharply. Google's own accelerator designers confronted this directly. EfficientNet-EdgeTPU, a family tuned for the Coral Edge TPU, replaced the expand-then-depthwise pattern in its early layers with a single ordinary 3-by-3 convolution — a "fused" block — that performs more arithmetic but maps much better to the accelerator's matrix units, and runs faster in practice. EfficientNet-Lite, a variant intended for mobile CPUs and accelerators, removed the squeeze-and-excitation modules and replaced swish with ReLU6, again because those components were awkward to quantize or poorly supported. MobileNetV4, published in 2024, made the cross-hardware problem its explicit goal: its "universal inverted bottleneck" block can take several forms, including the fused and plain variants, and the authors searched for configurations that perform well simultaneously on mobile CPUs, DSPs, GPUs, and dedicated accelerators such as the Apple Neural Engine and the Edge TPU.4 The general point is that operation counts are a proxy, and on accelerators a misleading one. The architecture that is best for a phone CPU may not be best for the same phone's NPU, and the one that is best for a server GPU is almost never best for either. The only reliable metric is latency measured on the target hardware, which is why the most effective architecture search methods for the edge build measured latency directly into their objective. MnasNet, a precursor to MobileNetV3, did exactly this, running candidate models on physical phones during the search and rewarding accuracy and real latency together rather than accuracy and operation count. Designing for peak memory: the MCUNet lesson On microcontrollers, the binding constraint is usually not latency but the activation budget, and here the most instructive work is MCUNet, from Ji Lin, Song Han, and colleagues, presented at NeurIPS in 2020.5 Their goal was to run ImageNet-scale image classification on off-the-shelf microcontrollers with a few hundred kilobytes of SRAM and one or two megabytes of flash — something that had widely been considered out of reach. MCUNet had two parts designed together. TinyNAS performed architecture search, but with an important first stage: before searching for the best architecture, it searched for the best search space. It considered many combinations of input resolution and width multiplier, discarded those that could not satisfy the memory constraints of the target, and among the remainder preferred the space whose randomly sampled models that fit the constraint had higher computational capacity — on the reasoning that a space in which feasible models can afford more computation is likely to contain more accurate ones. Only then did it search for a specific architecture. TinyEngine was an inference library co-designed with the search, which generated specialised code for each model at compile time, avoided the overhead of a general interpreter, and used memory-scheduling techniques to reduce peak activation memory. The result was a model reaching about 70.7 per cent top-1 accuracy on ImageNet on a commercial microcontroller — a figure comparable to the original MobileNetV1 on a phone, achieved within a few hundred kilobytes of RAM. The specific number matters less than the method. The search was constrained by peak memory from the start, the inference engine was designed jointly with the model, and both were evaluated on the real device. A follow-up, MCUNetV2, attacked the activation peak more directly with patch-based inference. Because the largest activations in a vision network occur in the first few layers, MCUNetV2 runs those layers on small spatial patches of the input one at a time, holding only a patch's worth of activations in memory, and assembles the results only once the feature map has shrunk enough to fit. Overlapping receptive fields between patches cause some computation to be repeated, which the authors mitigated by redistributing the network's receptive field, but the trade of a little extra arithmetic for a large reduction in peak memory is exactly right on a device where memory, not arithmetic, is scarce. The two knobs that matter most: resolution and width Architecture search is powerful but requires infrastructure that many teams do not have. For the common case — choosing a well-known efficient family and fitting it to a device — two knobs do most of the work, and they should be turned deliberately. Input resolution affects activation memory and computation quadratically, and it is frequently set far higher than the task requires. A person-detection model for a doorbell that must decide whether anyone is in the frame does not need a 224-by-224 input; the Visual Wake Words benchmark, built specifically for microcontroller-scale person detection, is commonly run at 96 by 96, an input with less than a fifth of the pixels.6 Detecting a face at a distance needs more resolution; detecting a raised hand at arm's length needs less. The right resolution is the smallest at which the task's accuracy on representative data stays above the floor, and it is found empirically by training at several resolutions and comparing. This experiment is cheap and almost always worth running before anything else. Width affects weight storage roughly quadratically in pointwise layers and activation memory linearly. Reducing width is the most natural way to bring a model's weights inside a flash budget. It also interacts with hardware in a way that is easy to miss: many accelerators and optimised CPU kernels process channels in groups — multiples of 8, 16, or 32 are common — so a layer with 36 channels may run no faster than one with 48, and one with 33 may be dramatically slower than one with 32. Choosing channel counts that align with the target's natural vector width is a free improvement. Depth is a less effective knob for edge budgets. Removing layers reduces weight storage and computation but does little to the activation peak, which is set by the widest high-resolution layer, and it tends to cost more accuracy per unit of savings than reducing width or resolution. Choosing operators the device can run An architecture is not only a shape; it is a set of operators, and every operator must exist, efficiently, on the target. This is a hard constraint that architecture papers rarely mention. Microcontroller runtimes support a curated set of operators, and an application includes only those it registers, to save flash. Common operators — convolution, depthwise convolution, fully connected, pooling, ReLU and ReLU6, add, reshape, softmax — have optimised integer kernels, such as Arm's CMSIS-NN library for Cortex-M processors.7 Less common operators may have only a slow reference implementation, or none. A network that uses a novel activation function, an unusual normalisation, or a dynamic shape may convert cleanly on a desktop and then fail, or run ten times slower than expected, on the device. Accelerators are stricter still. A micro-NPU supports a fixed list of operators with specific constraints on kernel size, stride, channel count, and data type. Any operator outside that list is executed on the host CPU, which means splitting the graph, copying tensors, and losing much of the accelerator's benefit. Chapter 7 returns to this in detail, but the architectural lesson belongs here: check the operator support list of every runtime and accelerator you may target before committing to an architecture, and prefer operators that appear on all of them. Some specific choices recur often enough to state plainly. Batch normalisation is free at inference, because it can be folded into the weights and bias of the preceding convolution; layer normalisation and group normalisation cannot be folded and require computing statistics at run time, which is costly and awkward in integer arithmetic. ReLU and ReLU6 are trivial in integers and, with ReLU6, conveniently bound the activation range, which helps quantization; smooth activations involving exponentials are more expensive and are best replaced by piecewise-linear approximations. Fixed input shapes allow memory planning at compile time; dynamic shapes, beloved of research code, frustrate it. Audio and sensor models are a different shape Much of the literature on efficient architectures concerns images, but a large share of real edge workloads are audio and sensor streams: keyword spotting, sound-event detection, vibration monitoring, gesture recognition from accelerometers, and anomaly detection on industrial signals. These have their own architectural conventions, and the budget arguments apply to them in a somewhat different way. For audio, the network rarely consumes raw waveform samples. The usual input is a compact time-frequency representation — a log-mel spectrogram or mel-frequency cepstral coefficients — computed over a window of one second or so, giving a small two-dimensional array of perhaps forty or fifty time steps by ten to forty frequency bands. Yundong Zhang and colleagues at Arm, in a 2017 study titled "Hello Edge: Keyword Spotting on Microcontrollers", compared several network families on this kind of input under microcontroller memory and compute limits, and found that a depthwise separable convolutional network — the DS-CNN — gave the best accuracy within the budget. The keyword-spotting reference model in the MLPerf Tiny benchmark, discussed in Chapter 8, is a network of this type with under forty thousand parameters. The lesson is that the feature extraction does a great deal of the work, and the network itself can be tiny. That shifts the budget. On such a device, computing the spectrogram may cost as much time and energy as running the network, and the feature extractor's exact parameters become part of the model's definition: change the window length or the number of mel bands and the trained network no longer applies. Treating the feature extraction as part of the architecture, designed and versioned with it, avoids a whole class of deployment bugs. Streaming inputs also allow an optimisation that images do not. A keyword spotter typically re-runs every few tens of milliseconds on a window that overlaps almost entirely with the previous one. A naive implementation recomputes every layer for the whole window each time. A streaming implementation keeps the intermediate results for the parts of the window it has already seen, in small state buffers, and computes only the new time steps — turning each convolution over time into an incremental update. For networks built from one-dimensional or causal convolutions over time, this can reduce the work per step by an order of magnitude at the cost of a few kilobytes of state. It requires the architecture to be designed for it, with convolutions that look only backwards in time, which is another reason to settle the deployment pattern before the architecture. For low-rate sensor data such as accelerometer or vibration readings, the networks are smaller still, and sometimes a neural network is not the right tool at all. Hand-designed features — spectral peaks, root-mean-square levels, statistical moments over a window — fed to a small classifier or a gradient-boosted tree can match a neural network on simple tasks at a fraction of the cost. The neural approach earns its place when the patterns are too varied to engineer features for, and the budget arguments of this booklet then apply in full. Transformers at the edge Transformers deserve a word, because they increasingly appear in edge workloads — in on-device language models, in speech recognition, and in vision models that combine convolutional and attention layers. Their characteristic operation, self-attention, has computation and memory that grow with the square of sequence length, and it relies on softmax and layer normalisation, both of which are awkward in low-precision integer arithmetic. Two adaptations have made them more practical. Hybrid architectures use convolutions in early, high-resolution stages, where attention would be prohibitively expensive, and introduce attention only on smaller feature maps. And hardware has begun to catch up: the third generation of Arm's micro-NPU, the Ethos-U85 announced in April 2024, added native support for operators such as matrix multiplication, transpose, and gather that transformer networks require, precisely because customers wanted to run them on embedded devices.8 Even so, for a microcontroller-class target with a tight memory budget, a well-designed convolutional or recurrent network is usually the safer choice unless the task genuinely needs attention. Small models trained well A final point about architecture is really a point about training. A small network trained with the same recipe as a large one is often under-trained relative to its potential. Small networks benefit disproportionately from longer training schedules, careful data augmentation, label smoothing, and — above all — from learning to imitate a larger, more accurate model, a technique Chapter 6 covers at length. The gap between a small architecture trained naively and the same architecture trained well can be several percentage points of accuracy, larger than the loss from quantizing it. That makes the order of work clear. First, choose an efficient architecture family whose operators the target supports. Second, find the smallest resolution and width that can meet the accuracy floor, measuring peak activation memory and on-device latency as you go. Third, train that network as well as you can, including distillation from a stronger teacher. Only then apply the compression techniques of the following chapters, starting with quantization — which, for a well-designed network, is usually all the compression it needs. CHAPTER 3 Quantization: The Arithmetic of Integers If a developer could apply only one optimization to a model bound for an edge device, it should be quantization: representing weights and activations with fewer bits than the 32-bit floating point used in training. Converting to 8-bit integers cuts weight storage by a factor of four, cuts memory traffic by the same factor, lets the processor use integer instructions that are cheaper and often wider in parallel, and — crucially — makes the model eligible to run on neural processing units, most of which execute only integer arithmetic. For a well-designed convolutional network, the accuracy cost is usually under a percentage point and frequently negligible. Those benefits are reliable enough that quantization is often treated as a button to press: run the converter, ship the result. Usually that works. When it does not, the failure is usually silent — the model runs, produces plausible outputs, and is quietly a few points less accurate than it should be, or catastrophically wrong on inputs outside the range seen during conversion. Understanding the arithmetic precisely is what lets a developer tell the difference, diagnose it, and fix it. This chapter sets out the arithmetic; the next deals with the cases where the simple approach breaks. Mapping real numbers to integers The scheme used by almost every edge runtime is uniform affine quantization, formalised for integer-only inference by Benoit Jacob and colleagues at Google in a paper presented at CVPR in 2018, which became the basis of TensorFlow Lite's integer path.1 It maps a real value r to an integer q through two parameters: a positive real scale S and an integer zero point Z. The mapping from integer back to real is r = S × (q − Z). Going the other way, a real value is quantized by dividing by the scale, rounding to the nearest integer, adding the zero point, and clamping to the representable range — for signed 8-bit integers, −128 to 127. The scale sets the step size: two adjacent integers represent real values S apart. The zero point sets which integer represents the real value zero. Jacob and colleagues insisted that real zero be exactly representable, for a practical reason: neural networks pad tensors with zeros, and ReLU produces exact zeros in abundance, so if zero were represented with even a small error, that error would be added systematically across every padded border and every inactive unit. To choose S and Z for a tensor, one needs the range of real values it will hold, from some minimum to some maximum. The scale is that range divided by the number of integer steps available — 255 for 8 bits — and the zero point is chosen so that the minimum maps to −128 and zero maps exactly to an integer. Every real value inside the range is then represented with an error of at most half a step. Values outside the range are clamped to its edges, which can produce much larger errors. A small worked example makes this concrete. Suppose calibration shows that an activation tensor ranges from −1.0 to 3.0. The range spans 4.0, divided across 255 steps, so the scale is about 0.0157. To place real zero exactly on an integer, compute where −1.0 would fall relative to zero — about 64 steps below it — and set the zero point so that −1.0 maps to −128; that makes the zero point −64. Real zero is now represented exactly by the integer −64. A real value of 1.0 divides by the scale to about 63.75, rounds to 64, and adds the zero point to give the integer 0; dequantizing gives 0.0157 × (0 − (−64)), or about 1.004 — an error of under half a step, as promised. A real value of 5.0, outside the calibrated range, would compute to an integer well above 127 and be clamped to 127, which dequantizes to about 3.0: an error of 2.0, more than a hundred times larger than the rounding error. Every value the calibration data did not anticipate is treated this way. This immediately exposes the central trade-off of quantization. A wide range accommodates rare large values without clamping, but makes the step size large, so ordinary values are represented coarsely. A narrow range represents ordinary values finely, but clamps the outliers. The first kind of error is called rounding error; the second, clipping error. Choosing the range is choosing the balance between them, and much of the craft of quantization is in that choice. Symmetric, asymmetric, per-tensor, per-channel Two further choices shape how a scheme behaves. Symmetric quantization fixes the zero point at zero and uses a range symmetric around it. It simplifies the arithmetic, because the zero-point terms disappear from many calculations, but wastes half the integer range on a tensor whose values are all positive. Asymmetric quantization uses the zero point to shift the range so it fits the data. The common convention, and the one used in the LiteRT integer specification, is to quantize weights symmetrically, since trained weights are roughly centred on zero, and activations asymmetrically, since activations after ReLU are non-negative and would waste half their range under a symmetric scheme.2 Granularity determines how many tensors share a scale. Per-tensor quantization uses one scale for an entire weight tensor; per-channel quantization uses a separate scale for each output channel of a convolution or each output row of a fully connected layer. The difference is large in practice. After batch normalisation is folded into a convolution's weights, different output channels can have weight ranges that differ by an order of magnitude or more. A single per-tensor scale must accommodate the widest channel, leaving the narrow channels represented by only a handful of distinct integer values. Early experience with quantizing MobileNet architectures, whose depthwise layers are particularly prone to this, found that per-tensor weight quantization could cause severe accuracy losses that per-channel quantization largely eliminated.3 Per-channel weight quantization is now the default in mainstream toolchains, and it costs almost nothing at inference time: the per-channel scales are folded into the requantization step described below. Activations, by contrast, are almost always quantized per-tensor, because per-channel activation scales would complicate the arithmetic of every subsequent layer. How an integer layer actually computes The heart of integer-only inference is that a quantized layer can compute its output using only integer multiplication, integer addition, and a single fixed-point rescale at the end. It is worth tracing the arithmetic once, because it explains several otherwise puzzling details of quantized models. Consider one output of a fully connected layer: a sum over products of weights and inputs, plus a bias. Substitute the quantized representation of each — weight as S_w × (q_w − Z_w), input as S_x × (q_x − Z_x), output as S_y × (q_y − Z_y) — and rearrange. The output integer q_y turns out to equal the output zero point, plus a multiplier M times a sum of products of zero-point-adjusted integers, plus the bias, where M is the product of the weight and input scales divided by the output scale. Everything inside the sum is integer arithmetic. Each product of two 8-bit values fits in 16 bits, and the sum of many such products is accumulated in a 32-bit integer accumulator, which has ample headroom for the hundreds or thousands of terms in a typical layer. The terms involving zero points can be precomputed or rearranged so that the inner loop is a plain integer dot product. The bias is stored as a 32-bit integer with a scale equal to the product of the weight and input scales and a zero point of zero, so that it can be added directly to the accumulator. This is why quantized models keep biases at 32 bits even when everything else is 8 bits: the bias is added at the accumulator's precision, and quantizing it coarsely would introduce a systematic offset into every output. The only non-integer quantity is the multiplier M, a real number typically between zero and one. Jacob and colleagues showed that it can be represented as a 32-bit fixed-point integer multiplier and a right shift, so that the final rescaling is an integer multiplication followed by a rounding shift. This requantization step converts the 32-bit accumulator back to an 8-bit output in the output tensor's scale. With per-channel weights, each output channel has its own multiplier and shift. Finally, a fused activation function costs nothing. ReLU becomes clamping the output integer to be no less than the integer representing zero; ReLU6 additionally clamps it to be no greater than the integer representing six. These clamps are folded into the saturation that requantization must perform anyway. Three practical consequences follow. First, the output scale of each layer is fixed at conversion time from the calibration data, and values that would exceed it at run time are silently saturated. Second, an operator for which no integer kernel exists breaks the chain, forcing a dequantize-compute-requantize sequence in floating point that is slow and, on accelerators, may move the computation to another processor. Third, because the requantization arithmetic is specified exactly, a correct integer kernel on a microcontroller and a correct integer kernel on a desktop should produce bit-identical outputs — which makes the desktop interpreter, running the converted model, the right reference against which to test the device. Formats beyond int8 Eight-bit integer is the workhorse, but not the only option. The main alternatives are set out in Table 2. Table 2. Common numeric formats for edge inference. Format Bits Size vs FP32 Typical edge use Hardware support FP32 32 1× Training; reference All CPUs; slow on MCUs without FPU FP16 16 0.5× Phone GPUs and NPUs Most phone SoCs BF16 16 0.5× Training; some newer NPUs Limited at the edge INT8 8 0.25× Default for CNNs, MCUs, NPUs Nearly universal INT4 4 0.125× LLM weights; some NPUs Growing; often unpacked to INT8 Float16 halves storage with essentially no accuracy loss for most models and runs natively on phone GPUs. It is a reasonable choice when the target is a GPU and the model is sensitive to integer quantization, but it does not unlock integer-only NPUs and it offers nothing on a microcontroller without half-precision hardware. Dynamic range quantization — a LiteRT conversion mode — stores weights as 8-bit integers but keeps activations in floating point, computing activation scales on the fly. It requires no calibration data and delivers the fourfold storage saving, but the arithmetic remains largely floating point, so it is useful mainly for CPU deployment of models, such as recurrent networks, where full integer conversion is troublesome. 16-bit activations with 8-bit weights, sometimes written as int16x8, are supported by several runtimes and by Arm's Ethos-U NPUs. They give activations 256 times as many levels while keeping weight storage at 8 bits, and are a useful middle path for audio and sensor models whose activations have high dynamic range. Four bits and below are covered in the next chapter. They matter mainly for weights, especially in language models, where memory bandwidth dominates. Post-training quantization: calibration Post-training quantization, usually abbreviated PTQ, converts a trained floating-point model without further training. Weight ranges can be read directly from the weights. Activation ranges cannot — they depend on the input — so PTQ requires a calibration step: run a representative sample of inputs through the floating-point model, record the values each activation tensor takes, and choose a range for each. The calibration set does not need to be large — a few hundred samples is a common starting point — but it must be representative of what the device will see. This is where many silent failures originate. A model for an outdoor camera calibrated only on daytime images will set activation ranges that clip badly at night. A keyword spotter calibrated on clean studio recordings will saturate on loud, noisy audio. Calibration data should be drawn from the same distribution as deployment data, including its extremes, and ideally from the validation set rather than the training set. The simplest range estimator takes the observed minimum and maximum. It never clips calibration data but is easily dominated by rare outliers, producing a large scale and coarse resolution for typical values. Better estimators deliberately clip. A percentile estimator ignores the most extreme fraction of observed values. A mean-squared-error estimator searches for the range that minimises the total quantization error on the calibration data. An entropy-based estimator, used in NVIDIA's TensorRT, chooses the range that minimises the information lost, measured as the divergence between the original and quantized distributions. For activations with long tails, clipping estimators are usually substantially better than min-max. In LiteRT, full-integer post-training quantization takes a few lines of configuration. The essential parts are a function that yields representative inputs and a declaration that the model must use integer operators throughout: converter = tf.lite.TFLiteConverter.from_keras_model(model) converter.optimizations = [tf.lite.Optimize.DEFAULT] converter.representative_dataset = representative_samples converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8] converter.inference_input_type = tf.int8 converter.inference_output_type = tf.int8 tflite_model = converter.convert() Requiring integer-only operators is important for microcontroller and NPU targets: without it, the converter may silently leave unsupported operators in floating point, producing a model that runs on a desktop but not on the device. Setting the input and output types to integers removes floating-point conversions at the model's boundaries, which matters on processors without a floating-point unit; the application then quantizes its inputs and dequantizes outputs itself, using the scales and zero points stored in the model. PyTorch's current path, export-based quantization, follows the same logic: capture the model graph with torch.export, insert observers according to a quantizer configuration supplied for the target backend, run calibration data through, and convert. The backend-specific quantizer matters, because it encodes which operators the target can fuse and which quantization schemes its kernels expect. Repairs that need no training Before resorting to retraining, several data-free or calibration-only techniques can rescue a model that quantizes badly. Two from Markus Nagel and colleagues at Qualcomm are widely implemented.4 Cross-layer equalization addresses the problem of channels with wildly different ranges. Because ReLU commutes with positive scaling, the weights of one channel in a layer can be multiplied by a factor and the corresponding input weights of the next layer divided by the same factor without changing the network's function. Choosing these factors to equalise channel ranges makes per-tensor quantization far more accurate, and helps even per-channel schemes by balancing ranges across adjacent layers. Bias correction addresses the observation that quantization error in weights is often biased — on average it shifts a layer's outputs in one direction. That expected shift can be estimated from the batch-normalisation statistics or from calibration data and subtracted from the layer's bias, removing a systematic error that would otherwise propagate through the network. A third technique, AdaRound, also from Nagel's group, questions the assumption that each weight should be rounded to its nearest integer.5 Rounding each weight independently minimises each weight's own error but not the error in the layer's output. AdaRound learns, using a small amount of unlabelled calibration data and a layer-by-layer optimisation, whether to round each weight up or down so as to minimise output error. It requires no labels and no end-to-end training, yet frequently recovers a large share of the accuracy lost to naive rounding, especially at four bits. Evaluate the model you will ship The last rule of this chapter is procedural and important. The accuracy of a quantized model must be measured on the quantized model, running in an interpreter that uses the same integer kernels as the device, over the full validation set. Not on the floating-point model. Not on a "fake-quantized" simulation inside the training framework, whose rounding and saturation may differ in small but consequential ways. Not on a few dozen examples. The comparison worth making is three-way: the floating-point model, the quantized model in the desktop interpreter, and the same quantized model on the device, run on identical inputs. The first gap measures quantization loss and tells you whether the techniques of the next chapter are needed. The second should be zero, or nearly so; if it is not, something is wrong in the deployment — a pre-processing mismatch, an input scaling error, an operator implemented differently on the device — and finding it now is far cheaper than finding it in the field. CHAPTER 4 When Quantization Breaks: Diagnosis, Training, and Low Bit-Widths Post-training quantization to 8 bits works for most convolutional networks most of the time. This chapter is about the rest of the time: the models that lose five or ten points of accuracy on conversion, the models whose outputs collapse on certain inputs, and the push below 8 bits, where post-training methods begin to fail as a matter of course and where much of the recent research effort, driven by language models, has been concentrated. The approach is deliberately diagnostic. When a quantized model underperforms, the instinct is to reach for the heaviest tool available — quantization-aware training — and retrain. Sometimes that is right. More often, the loss is concentrated in a small number of layers for an identifiable reason, and a targeted fix is faster, cheaper, and more robust than retraining the whole network. Finding where the error comes from Quantization error is introduced at every layer and accumulates through the network, but it is rarely spread evenly. A typical poorly quantizing model has one or two layers responsible for most of the loss. The first task is to find them. The most direct method is layer-wise comparison. Run the same input through the floating-point model and the quantized model, capture every intermediate tensor, dequantize the quantized ones, and compute a measure of agreement for each layer — commonly the signal-to-quantization-noise ratio, the ratio of the floating-point tensor's power to the power of the difference between the two. A healthy 8-bit model typically shows a ratio that declines gradually through the network. A sudden drop at one layer marks where the trouble is introduced. Most toolchains provide a debugger for exactly this; LiteRT's quantization debugger and the numeric-comparison utilities in PyTorch's quantization tooling both report per-layer statistics. A complementary method is sensitivity analysis: quantize the network one layer at a time, leaving all others in floating point, and measure end-to-end accuracy for each. Layers whose quantization alone causes a large drop are sensitive; the rest can safely be quantized. Running the analysis in reverse — quantize everything, then restore one layer at a time to floating point — answers the more practical question of which single layer, if kept at higher precision, recovers the most accuracy. The causes that these analyses reveal fall into a handful of recurring patterns. Outlier activations. A small number of values in an activation tensor are far larger than the rest, forcing a wide range and a coarse step size. This is common after layers without a bounding activation and is endemic in transformer models, as discussed below. The fixes are a clipping range estimator, replacing an unbounded activation with a bounded one such as ReLU6 and retraining, or keeping that tensor at 16 bits. Per-tensor weight scales on uneven channels. Visible as poor agreement in particular convolution layers, especially depthwise ones. The fix is per-channel quantization or, where the target supports only per-tensor scales, cross-layer equalization. Mismatched scales at joins. Residual additions and concatenations combine tensors that were quantized with different scales. The runtime must requantize them to a common scale before combining, losing precision in whichever input has the finer scale. If one branch has a much wider range than the other, the narrow branch is effectively crushed. Inspecting the scales on both inputs of every addition and concatenation is a fast check. Sensitive first and last layers. The first layer consumes raw input, whose precision may matter more than intermediate features; the last layer produces logits or regression outputs whose small differences decide the answer. Keeping these at higher precision is a long-standing practice and costs little, since they are usually a small share of total computation — though on an accelerator, any layer left at higher precision may fall back to the CPU, a cost that must be measured. Nonlinear functions. Sigmoid, tanh, softmax, exponentials, and division are implemented in integer runtimes with lookup tables or polynomial approximations. Their outputs have fixed quantization parameters, and inputs outside the expected range saturate. Attention layers and gated recurrent units concentrate these functions and are often where quantization of sequence models goes wrong. Mixed precision Once sensitive layers are identified, the cheapest remedy is often mixed precision: keep those layers at 16-bit activations, or in floating point, and quantize the rest to 8 bits. Because a handful of layers usually accounts for most of the error and a small share of the computation, the cost in speed and size is modest. The constraint is hardware. On a CPU, mixing precisions costs a few conversion operations. On an NPU that executes only 8-bit operations, a floating-point layer splits the graph and sends part of the work back to the host processor, with tensor copies and synchronisation on each side. Arm's Ethos-U NPUs support 16-bit activations natively, which makes int16 activations on sensitive layers an attractive middle ground on those parts. The right choice depends on what the target executes natively, which is another reason to settle the target early. More systematic methods exist for choosing a precision per layer. Hessian-aware approaches such as HAWQ, from Zhen Dong, Amir Gholami, and colleagues, use second-order information about the loss to estimate each layer's sensitivity and allocate bits accordingly; search-based methods treat the bit-width of each layer as a hyperparameter and optimise it against measured latency. For most deployments, simple sensitivity analysis followed by keeping one to three layers at higher precision captures most of the available benefit. Quantization-aware training When targeted fixes are not enough — or when the target bit-width is below 8 — the model must learn to tolerate quantization. Quantization-aware training, or QAT, does this by simulating quantization during training, so that the optimiser sees, and compensates for, the error it introduces. The mechanism is the insertion of fake quantization operations into the training graph. Each fake-quantize operation takes a floating-point tensor, quantizes it to the target grid, and immediately dequantizes it, so that the tensor remains floating point but takes only the values the integer model will be able to represent. The forward pass therefore computes what the quantized model will compute, including rounding and clipping errors, and the loss reflects them. The difficulty is the backward pass. Rounding is a step function whose derivative is zero almost everywhere, which would stop gradients dead. The standard remedy is the straight-through estimator, an idea analysed by Yoshua Bengio and colleagues in 2013: in the backward pass, treat the rounding operation as if it were the identity, passing the gradient through unchanged, while zeroing it for values that were clipped.1 The estimator is mathematically crude but works remarkably well in practice, because the weights being trained are still floating-point values that accumulate small updates until they cross a rounding boundary. A refinement called learned step size quantization, introduced by Steven Esser and colleagues at IBM in 2020, makes the quantization scale itself a trainable parameter, with a carefully derived gradient.2 Rather than fixing ranges from calibration statistics, the network learns the scales that minimise the training loss, trading rounding error against clipping error layer by layer. LSQ and its successors are the basis of most high-quality low-bit QAT. Several practical points determine whether QAT succeeds. • Start from a well-trained floating-point model and fine-tune, rather than training with fake quantization from scratch. A short fine-tuning run — a few percent of the original training schedule — at a reduced learning rate is usually enough at 8 bits; more is needed at 4 bits. • Handle batch normalisation correctly. At inference, batch normalisation is folded into the preceding convolution's weights, so the weights that are quantized are the folded ones. QAT must simulate this folding, or it will train the network to tolerate the wrong quantization error. Toolchains handle this, but custom training loops often do not. A common practice is to freeze batch-normalisation statistics partway through QAT so that the folded weights stop shifting. • Match the target's scheme exactly. Symmetric or asymmetric, per-tensor or per-channel, the rounding mode, and which operators are fused must all match what the target runtime will do. A model trained for one scheme and deployed with another loses much of the benefit. • Evaluate the converted model, not the QAT model. The fake-quantized training graph approximates the integer model; the converted model is the integer model. Small discrepancies in rounding or fusion can make the converted model worse than the training graph suggests. At 8 bits, QAT typically closes most of the gap between a poorly quantizing model and its floating-point original. Its costs are engineering ones: a training pipeline that must be maintained alongside the floating-point one, additional hyperparameters, and longer iteration cycles. It is worth paying when PTQ with targeted fixes still falls short of the accuracy floor, and it is essentially mandatory below 8 bits for convolutional networks. Recurrent and stateful models Recurrent networks — LSTMs and GRUs in particular — remain common at the edge for audio, sensor, and time-series tasks, and they quantize less gracefully than convolutional networks. The reason is state. A recurrent layer carries a hidden state, and in the LSTM's case a separate cell state, from one time step to the next. Quantization error introduced at one step is fed back at the next and can accumulate over long sequences, so that a model that agrees closely with its floating-point original for the first few steps drifts away from it later. The gates make matters worse. Each LSTM gate passes a linear combination of inputs and state through a sigmoid or tanh, and the cell state is updated by multiplying gate outputs together and adding the results. Those elementwise products and additions combine quantities with very different ranges, and the cell state can grow over time in ways that a range fixed at calibration does not anticipate. For this reason, fully integer LSTM implementations, including the one in LiteRT, typically keep the cell state at 16 bits even when weights and most activations are at 8, and use carefully specified fixed-point approximations for the gate non-linearities. Several practices help. Calibrate on sequences at least as long as those seen in deployment, so that the recorded ranges include the state's growth over time. Evaluate on long sequences, comparing floating-point and quantized outputs step by step to see whether and where drift sets in. Where the architecture can be chosen, prefer designs whose state is bounded, or replace recurrence with causal convolutions over time, which quantize like any other convolution and have no feedback path for errors to circulate in. And if a recurrent model must be quantized aggressively, quantization-aware training on full-length sequences is usually needed, because only training can teach the network to keep its state within a range the integer arithmetic represents well. Below eight bits Fewer bits per weight means less storage and less memory traffic, which, given the arguments of Chapter 1, is attractive. But accuracy falls faster as bit-width shrinks, and hardware support thins out. At 4 bits, each weight has only sixteen possible values. Convolutional networks can be brought to 4-bit weights with 8-bit activations with modest accuracy loss using QAT and techniques such as LSQ; 4-bit activations are considerably harder. The hardware question is whether the target can compute directly in 4 bits. Some NPUs can; many cannot, and instead unpack 4-bit weights to 8 bits in their weight decoder or kernel. In that case the arithmetic is no faster, but storage and memory traffic are halved, which is precisely the benefit that matters most on memory-bound workloads. At 2 bits and 1 bit, networks become ternary or binary, and multiplication can be replaced by sign flips, bit operations, and population counts. Binary networks such as XNOR-Net drew considerable research interest because of the dramatic theoretical savings, but they have struggled to match the accuracy of 8-bit models on hard tasks and have seen limited production use, partly because mainstream processors and NPUs are not built to exploit them. They remain worth knowing about for extremely constrained targets and for simple tasks where the accuracy gap is tolerable. Language models: where low-bit quantization grew up Most of the recent progress in low-bit quantization has come from large language models, for a reason Chapter 1 made explicit: generating each token requires reading every weight, so token rate is bounded by memory bandwidth divided by model size in bytes. Every bit removed from each weight raises that ceiling proportionally. That economic pressure has produced a body of techniques that edge developers can now use directly, as phones and embedded boards increasingly run language models with a few billion parameters. The first obstacle was outliers. Tim Dettmers and colleagues, in work published in 2022 as LLM.int8(), observed that transformer language models above a certain size develop a small number of activation features with magnitudes far larger than the rest, concentrated in a few channels and appearing consistently across layers.3 Per-tensor 8-bit quantization of those activations destroyed accuracy. Their remedy was to split the matrix multiplication, computing the few outlier dimensions in 16-bit floating point and the rest in 8-bit integers. SmoothQuant, from Guangxuan Xiao, Ji Lin, and colleagues, took a different route: because the difficulty lies in activations while weights are easy to quantize, it mathematically migrates part of the activation range into the weights, scaling activation channels down and the corresponding weight rows up by the same factors, after which both quantize well to 8 bits.4 This is the same function-preserving rescaling idea as cross-layer equalization, applied to a new problem. The second development was weight-only quantization at 3 and 4 bits, which keeps activations in 16-bit floating point and quantizes only the weights. Because token generation is memory-bound, this delivers most of the speed benefit while sidestepping the activation-outlier problem. Two methods dominate. GPTQ, from Elias Frantar and colleagues and published at ICLR in 2023, quantizes weights one column at a time within each layer and, after quantizing each column, updates the remaining unquantized weights to compensate for the error just introduced, using second-order information estimated from a small calibration set.5 It brought models with over a hundred billion parameters to 3 and 4 bits in a few GPU-hours with small accuracy loss. AWQ, activation-aware weight quantization, from Ji Lin and colleagues and recognised with a best-paper award at MLSys 2024, starts from the observation that a small fraction of weight channels — those multiplied by large activations — matter disproportionately to output quality.6 Rather than keeping them in higher precision, which is awkward for hardware, AWQ scales those channels up before quantization so they are represented more finely, with the inverse scaling absorbed elsewhere. The method needs no backpropagation and generalises well across domains. Both methods are usually combined with group-wise quantization: instead of one scale per output channel, weights are divided into groups of, typically, 32 to 128 consecutive values, each with its own scale and sometimes its own zero point. Smaller groups track local variation more closely at the cost of storing more scales; a group of 128 with a 16-bit scale adds about an eighth of a bit per weight. Widely used local inference tools, such as llama.cpp and its GGUF file format, offer a family of such block-quantized formats at various bit-widths and group arrangements. For the most aggressive settings, QAT has returned. Model developers increasingly release checkpoints trained with quantization in the loop so that their 4-bit versions lose less quality than post-training methods allow; Google, for instance, published quantization-aware-trained 4-bit versions of its Gemma 3 open models in 2025. For an edge developer, the practical guidance is simple: when a vendor provides a QAT-trained low-bit checkpoint for the model you need, prefer it to quantizing the full-precision model yourself. A decision order Pulling the chapter together, a sensible sequence for quantizing a model for the edge looks like this. Start with full-integer post-training quantization to 8 bits with per-channel weights and a clipping range estimator, calibrated on representative data. Evaluate the converted model on the full validation set. If the loss exceeds the budget, run per-layer comparison and sensitivity analysis to locate the problem, and apply targeted fixes: better ranges, equalization, bias correction, AdaRound, or higher precision on a few layers, checking each time that the target hardware supports the resulting mixture without costly fallbacks. Only if those fail, or if the target requires fewer than 8 bits, move to quantization-aware training, matched exactly to the target's scheme. For language models, start from weight-only 4-bit quantization with GPTQ or AWQ and group-wise scales, or from a vendor's QAT checkpoint. At every step, the criterion is the same: accuracy of the converted model, measured on the target's arithmetic, against the floor set in the budget. CHAPTER 5 Pruning: Sparsity the Hardware Can Use Neural networks are over-parameterised. A trained network typically contains a large fraction of weights that contribute almost nothing to its output, and those weights can be removed — set to zero and left there — with little or no loss of accuracy, especially if the network is fine-tuned afterwards. This observation is as old as modern neural networks themselves: Yann LeCun, John Denker, and Sara Solla described "Optimal Brain Damage" in 1990, using second-derivative information to decide which weights to remove, and Babak Hassibi and David Stork refined it as "Optimal Brain Surgeon" a few years later. The modern revival came from Song Han and colleagues, whose 2015 paper showed that simply removing the weights with the smallest magnitudes, then retraining, could reduce the parameter count of AlexNet ninefold, from 61 million to 6.7 million, and of VGG-16 thirteenfold, from 138 million to 10.3 million, with no loss in accuracy.1 Their follow-up, Deep Compression, combined pruning with weight sharing and Huffman coding to shrink AlexNet's storage from 240 megabytes to 6.9 — a factor of 35 — and VGG-16's from 552 megabytes to 11.3, a factor of 49.2 Those numbers launched a large research field. They also created a lasting misunderstanding. A network with 90 per cent of its weights set to zero is not, in general, ten times faster or ten times smaller in memory, and on many devices it is neither faster nor smaller at all. Whether pruning pays off depends almost entirely on the pattern of the zeros and on whether the target hardware and software can exploit that pattern. This chapter is about the difference. Why scattered zeros are not free Consider a weight matrix in which 50 per cent of the entries, chosen individually by magnitude, have been set to zero. This is unstructured sparsity: the zeros fall wherever the small weights happened to be, with no regular arrangement. Stored densely, the matrix is exactly as large as before; the zeros occupy space like any other value. To save storage, it must be stored in a sparse format that records only the non-zero values and their positions. A common format, compressed sparse row, stores each non-zero value and its column index, plus a pointer to the start of each row. If values are 8-bit integers and column indices are 16-bit integers, each non-zero entry costs three bytes, compared with one byte per entry stored densely. At 50 per cent sparsity, the sparse representation is larger than the dense one. Cleverer encodings — relative indices in fewer bits, bitmaps marking non-zero positions — reduce the overhead, but the break-even point for unstructured sparsity on 8-bit weights is commonly well above 50 per cent, and the savings remain modest until sparsity is very high. Computation is worse. Dense matrix and convolution kernels are fast because they process contiguous blocks of data with vector instructions: a single instruction multiplies and accumulates sixteen or thirty-two values at once, and the memory system streams data predictably. Skipping individual zeros requires reading indices, gathering scattered input values, and breaking the regular pattern that vector units and caches depend on. Below high sparsity levels, a sparse kernel processing only the non-zeros is slower than a dense kernel processing everything, zeros included. This combines with the quantization of the previous chapters. A dense 8-bit weight is already one byte; there is little left for unstructured sparsity to save once indices are paid for. Han's original results were reported on 32-bit floating-point weights, where each removed weight saved four bytes and sparse bookkeeping was comparatively cheap. Granularity: the decisive choice The remedy is to prune in patterns the hardware can exploit. Pruning methods are best classified by granularity — the size and shape of the units removed — because granularity determines what hardware benefit is possible. Table 3 summarises the main options. Table 3. Pruning granularity and the benefit hardware can realise. Pattern Removes Accuracy cost Benefit on generic hardware Needs special support Weight Single weights (unstructured) Lowest Little, except compression Sparse kernels at high sparsity Block Small tiles of weights Low to moderate Some with block kernels Block-sparse kernels N:M N of every M weights Low None Yes, e.g. 2:4 units Channel Whole output channels or filters Higher Full: smaller dense model No Layer Whole layers Highest Full No At one end, unstructured pruning removes individual weights. It achieves the highest sparsity for a given accuracy, because it has the most freedom to choose what to remove, but delivers the least on ordinary hardware. At the other end, structured pruning removes entire output channels of a convolution — equivalently, whole filters — or entire neurons of a fully connected layer. The result is not a sparse network at all but a smaller dense one: a convolution with 64 output channels pruned to 40 is simply a convolution with 40 output channels, and the next layer's input channels shrink correspondingly. Every runtime, every kernel library, and every accelerator benefits immediately, because the network is just narrower. The cost is accuracy: removing whole channels is a coarser operation than removing individual weights, so less can be removed before accuracy suffers. Between them lie two intermediate forms. Block sparsity removes small rectangular tiles of weights, matching the tile size that a kernel or accelerator processes at once; when a whole tile is zero, it can be skipped without per-element bookkeeping. N:M or semi-structured sparsity requires that in every group of M consecutive weights, at most N are non-zero. The best-known case is 2:4 sparsity, introduced in hardware with NVIDIA's Ampere generation of GPUs: in every group of four weights, two must be zero, and the hardware stores only the two survivors plus a two-bit index for each, and skips the zero multiplications, doubling peak math throughput for eligible layers.3 The same pattern has reached the embedded world: Arm lists native hardware support for 2:4 sparsity among the features of its Ethos-U85 micro-NPU.4 N:M sparsity sits close to unstructured pruning in accuracy, because within each small group the pruner still chooses freely, while giving the hardware a perfectly regular pattern to exploit. The storage arithmetic for N:M sparsity shows why it is attractive. Under the 2:4 pattern, each group of four 8-bit weights is stored as two 8-bit values and two 2-bit position indices: twenty bits instead of thirty-two, or about 62 per cent of the dense size. That is a smaller saving than the nominal 50 per cent sparsity suggests, because of the indices, but it is achieved without any irregularity in the data layout, and on hardware that skips the zero multiplications the compute saving is the full half. The hidden difficulty of structured pruning Structured pruning sounds simple — remove channels, get a narrower network — but in modern architectures the channels of different layers are tied together, and removing one requires removing others consistently. In a plain chain of convolutions, pruning an output channel of one layer means removing the matching input channel of the next layer and nothing else. Residual networks are not plain chains. Wherever two tensors are added, as in every residual connection, they must have the same number of channels in the same order, so a channel can be removed from one only if the corresponding channel is removed from the other and from every other layer feeding or reading that same addition. In a MobileNetV2-style network, all the blocks within a stage share their residual channels, so a single pruning decision on the residual path touches many layers at once. Concatenations, depthwise convolutions (whose input and output channels correspond one to one), grouped convolutions, and attention heads impose further couplings. Getting these dependencies wrong produces a model that fails to run or, worse, runs with silently misaligned channels. Early structured-pruning code typically handled them by hand for each architecture. More recent tools build a dependency graph of the network automatically and prune coupled channels as groups; the DepGraph method of Gongfan Fang and colleagues, presented at CVPR in 2023 and implemented in the open-source Torch-Pruning library, is one widely used example. A practical consequence is that the residual channels of efficient architectures are often left unpruned, with pruning concentrated on the wide expansion layers inside each block, which are not coupled to anything outside it and which are, conveniently, where much of the computation lies. There is also a hardware subtlety. A structurally pruned network can end up with channel counts such as 37 or 53 — whatever the pruning criterion happened to leave. As Chapter 2 noted, many kernels and accelerators process channels in groups of 8, 16, or 32, so such counts waste part of every vector operation. Rounding pruned channel counts to the target's natural multiple usually costs a negligible amount of accuracy and recovers that waste. The practical rule is to choose granularity by asking what the target can exploit. On a microcontroller CPU with no sparse support, unstructured sparsity helps only storage compression, and only if the toolchain compresses weights; channel pruning helps everything. On an NPU with N:M support, N:M sparsity is the obvious choice. Pruning in a pattern the hardware ignores produces a model that is smaller in a paper and identical on the device. Where unstructured sparsity still pays Unstructured pruning is not useless at the edge, but its benefits arrive by two specific routes. The first is compression for storage and transfer. A weight tensor with many zeros compresses well with general-purpose or specialised entropy coding, even when stored densely. This matters for over-the-air model updates, where bytes transmitted cost energy and money, and for accelerators that decompress weights on the fly. Arm's Ethos-U NPUs, for instance, store weights in a compressed format produced by the Vela compiler and decompress them in hardware as they are fetched, so that zeros and repeated values reduce both the flash footprint and the bandwidth needed to stream weights to the accelerator. On such a target, unstructured sparsity and clustering translate into real savings in the resource that matters most. The second is specialised sparse kernels at high sparsity. Erich Elsen and colleagues at Google showed, in work presented at CVPR in 2020, that sparse versions of MobileNet-style networks could run substantially faster than their dense counterparts on mobile Arm CPUs, using kernels designed for the sparse 1-by-1 convolutions that dominate those networks — but at sparsity levels around 70 to 90 per cent, not 50.5 That work found its way into the XNNPACK library used by mobile runtimes. The lesson is that unstructured sparsity can pay on CPUs, but only with purpose-built kernels and only at sparsity levels that require careful training to reach without unacceptable accuracy loss. Weight clustering, a close relative of pruning, deserves mention here. Instead of setting weights to zero, it groups each layer's weights into a small number of clusters — sixteen, say — and replaces each weight with its cluster's centroid, so that the layer contains only sixteen distinct values. Deep Compression used this as its weight-sharing stage. Like unstructured sparsity, clustering does not speed up ordinary arithmetic, but it makes weights far more compressible, and the TensorFlow Model Optimization Toolkit offers it alongside pruning and quantization for exactly that purpose. How to prune: criteria and schedules The choice of which units to remove has generated an enormous literature, but the simple criteria remain strong. For unstructured pruning, weight magnitude is the standard: small weights contribute less. For channel pruning, common criteria include the magnitude of each filter's weights, such as its L1 norm, used by Hao Li and colleagues in an influential 2017 paper on pruning filters,6 or the scaling factor of the batch-normalisation layer that follows each channel — the basis of "network slimming", which adds a sparsity-inducing penalty on those factors during training so that unimportant channels are driven towards zero and can be removed cleanly. More elaborate criteria use gradients or second-order information to estimate each unit's effect on the loss. They can help, especially at high sparsity, but the gains are less consistent than many papers suggest. A 2020 survey by Davis Blalock and colleagues, "What Is the State of Neural Network Pruning?", examined some eighty pruning papers and found that inconsistent baselines, architectures, and metrics made most results impossible to compare, and that simple magnitude pruning often performed about as well as more complicated methods when compared fairly.7 For a practitioner, that is liberating: start with magnitude-based criteria and spend effort on the schedule and fine-tuning instead. The schedule matters more than the criterion. Removing a large fraction of weights at once damages a network badly; removing them gradually, with training in between, lets the remaining weights adapt. Michael Zhu and Suyog Gupta's 2017 study "To prune, or not to prune" proposed a widely adopted gradual pruning schedule in which sparsity rises from zero to its target over the course of training, quickly at first and more slowly as it approaches the target, following a cubic curve.8 They also reported a result relevant to Chapter 2: for a fixed memory budget, large networks pruned to sparsity often outperformed small dense networks of the same non-zero parameter count. That finding is real, but it compares non-zero parameters, not memory or latency on hardware that cannot exploit unstructured sparsity — exactly the distinction this chapter has been drawing. The Lottery Ticket Hypothesis, published by Jonathan Frankle and Michael Carbin at ICLR in 2019, offered a striking explanation of why pruning works.9 They found that dense networks contain sparse subnetworks which, if reset to their original initial weights and trained in isolation, can match the accuracy of the full network — winning tickets in an initialisation lottery. The result deepened understanding of over-parameterisation and spawned a large follow-on literature, but it is not itself a deployment technique: finding the ticket requires training the full network, often repeatedly. Its practical message is mostly reassurance that heavily pruned networks can be genuinely capable, not degraded approximations. For language models, one-shot methods have been developed because retraining is so expensive. SparseGPT, from Elias Frantar and Dan Alistarh, prunes large models to 50 per cent or more unstructured sparsity in one pass using the same second-order, layer-wise reconstruction ideas as GPTQ, and Wanda, from Mingjie Sun and colleagues, achieves competitive results with a criterion as simple as weight magnitude multiplied by input activation norm. Both support N:M patterns, which is where their speed benefit on hardware comes from. Pruning with quantization Pruning and quantization combine, but order and coordination matter. The usual sequence is to prune first, fine-tune to recover accuracy, then quantize — with quantization-aware training if needed. Naive QAT on a pruned model may, however, nudge zeroed weights away from zero, destroying the sparsity pattern. Toolchains therefore offer sparsity-preserving variants of QAT that hold pruned weights at zero throughout; the TensorFlow Model Optimization Toolkit calls its version "collaborative optimization" and supports chains such as pruning followed by clustering followed by quantization, each stage preserving what the previous one produced. There is also a question of whether both are worth doing. After 8-bit quantization of a well-designed efficient architecture, the remaining room for pruning to help — on hardware without sparse support — is mostly via structured pruning, which is to say via making the network narrower. If the architecture was already sized to the budget as Chapter 2 recommends, there may be little left to take. When to prune The honest summary is that pruning is a powerful technique whose value at the edge is highly conditional. It is clearly worth applying when: • The target hardware supports a sparsity pattern natively — N:M sparsity on a capable NPU or GPU, or weight compression in an accelerator's fetch path — and the pruning matches that pattern. • The network is over-sized for the task and structured channel pruning can produce a narrower dense network, particularly when retraining a smaller architecture from scratch is impractical. • Model download size matters and unstructured sparsity or clustering will make the weights compress better for transfer. • Very high unstructured sparsity is achievable and the runtime has sparse kernels for the dominant layer types. It is usually not worth the effort when the target is a CPU or NPU without sparse support, the network is already a compact architecture sized for the budget, and the only form of sparsity on offer is unstructured. In that situation, the bytes and time are better recovered through quantization and a smaller architecture — and, as the next chapter argues, by training that smaller architecture to punch above its weight. CHAPTER 6 Knowledge Distillation: Teaching Small Networks Quantization and pruning start from a trained network and remove precision or parameters from it. Knowledge distillation works differently. It leaves the large network intact and uses it as a teacher, training a separate, smaller network — the student — to reproduce its behaviour. The student can have whatever architecture the device demands: a compact convolutional network sized to a microcontroller's activation budget, a shallow transformer that fits in a phone's memory, a design whose every operator the target NPU supports. Distillation does not compress the teacher; it transfers what the teacher learned into a body built for the edge. This makes distillation the natural partner of the architecture-first approach in Chapter 2. Choose an efficient architecture sized for the budget, then train it with the help of the best model you can obtain, and it will usually be noticeably more accurate than the same architecture trained on labels alone. Then quantize it. In many edge projects, that sequence — design small, distil, quantize — beats any attempt to compress a large model directly. Soft targets and temperature The idea that a small model can learn from a large one predates deep learning's dominance. Cristian Bucilă, Rich Caruana, and Alexandru Niculescu-Mizil showed in 2006 that a large ensemble of models could be compressed into a single small network by training the small network to match the ensemble's outputs on a large set of unlabelled and synthetic examples. The form most practitioners use today was set out by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean in their 2015 paper "Distilling the Knowledge in a Neural Network".1 Their central observation concerns what a trained classifier's output contains. A network that classifies an image of a husky outputs a probability distribution over all classes. The highest probability goes to "husky", but the rest is not uniform noise: "wolf" and "malamute" receive far more probability than "teapot". Those relative probabilities on the wrong classes encode what the network has learned about similarity between classes — information that a one-hot label, which says only "husky" and nothing else, does not contain. Hinton and colleagues called this the teacher's dark knowledge. The difficulty is that a confident teacher assigns tiny probabilities to the wrong classes, so their differences have almost no effect on a training loss. The remedy is temperature. A classifier's probabilities come from applying the softmax function to its logits, the raw pre-softmax scores. Dividing the logits by a temperature T greater than one before the softmax softens the distribution, raising small probabilities and lowering the largest, so that the relative structure among the wrong classes becomes visible. The student is trained to match the teacher's softened distribution at the same temperature, measured by Kullback-Leibler divergence. In the standard formulation, the student's loss is a weighted sum of two terms: the ordinary cross-entropy against the true labels, computed at temperature one, and the divergence from the teacher's softened distribution, computed at temperature T and multiplied by T squared. The T-squared factor is easy to overlook and important: softening the distributions shrinks the gradients of the distillation term by roughly that factor, and multiplying back keeps the two terms in balance as temperature changes. At inference, the student runs at temperature one like any classifier. Two hyperparameters therefore govern the basic method: the temperature, and the weight between the label loss and the distillation loss. Neither has a universal best value; practitioners typically try a handful of temperatures in the low single digits up to around ten, and weightings that put most of the emphasis on the distillation term when the teacher is strong. Both are cheap to tune relative to the gains available. Why it works The reasons distillation helps are worth understanding, because they guide its use. The first is a richer signal per example. Each training example, labelled one-hot, provides the student with a single bit of class information per class. The teacher's distribution provides a graded similarity structure over all classes. The student learns from more information per example, which matters most when it has limited capacity and cannot afford to waste it on distinctions the data does not support. The second is regularisation. Training a network to match a soft distribution rather than a hard label discourages it from becoming over-confident, much as label smoothing does, and tends to improve generalisation and calibration. The third, and in practice often the largest, is the use of unlabelled data. The distillation term needs only the teacher's outputs, not true labels. Any in-domain data the teacher can process — and edge applications often have abundant unlabelled data from the sensors themselves — becomes training data for the student. Hinton and colleagues called the data used for distillation the transfer set; making it large and representative is one of the most effective levers available. That last point connects distillation to a widely used edge workflow that rarely calls itself distillation: pseudo-labelling with a large model. A team building a tiny detector for, say, delivery packages on doorsteps may have millions of unlabelled frames from cameras in the field and no budget to label them. A large, accurate model — perhaps a general-purpose vision model too big for any edge device — labels the frames, and the tiny model trains on those labels. This is distillation through hard labels rather than soft distributions, and it loses some of the dark knowledge, but it lets the student learn from a vast in-domain dataset that would otherwise be unusable. Where the teacher's soft outputs can be stored, they should be; where only its decisions are available, pseudo-labels still help substantially. Distilling features as well as outputs Matching only the final output leaves much of the teacher's knowledge unused. A family of methods also asks the student to match the teacher's intermediate representations. FitNets, from Adriana Romero and colleagues at ICLR in 2015, introduced "hints": an intermediate layer of the teacher serves as a target for an intermediate layer of the student, with a small learned regressor mapping the student's features to the teacher's dimensions.2 This allowed thin, deep students to be trained that would otherwise have been hard to optimise. Attention transfer, from Sergey Zagoruyko and Nikos Komodakis, asks the student to match spatial maps derived from the teacher's activations, indicating where in the input the teacher's features respond, which avoids having to match full feature tensors of different widths. Other methods match the relationships between examples — for instance, the similarity between the representations of pairs of inputs — rather than the representations themselves. Feature-based distillation is most useful when the student and teacher share a broad structure, so that their intermediate layers correspond naturally, and for tasks beyond classification. Object detection and segmentation outputs are harder to distil purely at the output level, because the outputs are structured and the background dominates; matching intermediate feature maps, particularly around objects, has proved more effective there. Language models offer the clearest demonstrations of feature distillation's value. DistilBERT, from Victor Sanh and colleagues at Hugging Face in 2019, halved the number of layers in BERT and trained the student with a combination of the teacher's soft output distribution, the ordinary language-modelling loss, and a loss aligning the directions of the student's and teacher's hidden states; the result was 40 per cent smaller and 60 per cent faster while retaining 97 per cent of BERT's language-understanding performance on the GLUE benchmark.3 TinyBERT, from Xiaoqi Jiao and colleagues, went further by distilling attention matrices and hidden states layer by layer, both during general pre-training and again during task-specific fine-tuning. The capacity gap and the patient teacher An intuitive assumption is that a better teacher always produces a better student. It does not. Jang Hyun Cho and Bharath Hariharan, in a 2019 study of distillation's efficacy, found that very large, highly accurate teachers often produced worse students than smaller teachers, because the student lacked the capacity to mimic them, and that a teacher stopped early in its training sometimes worked better than a fully trained one.4 Seyed Iman Mirzadeh and colleagues proposed bridging large gaps with a teacher assistant: distil from the large teacher into a medium-sized network, then from that into the student. A complementary and more optimistic result came from Lucas Beyer and colleagues at Google, in a CVPR 2022 paper titled "Knowledge distillation: A good teacher is patient and consistent".5 They argued that many disappointing distillation results stemmed from how the training was run rather than from a fundamental capacity limit. Two practices mattered most. Consistency: the teacher and student must see exactly the same input, including the same random augmentation — which rules out the common shortcut of precomputing the teacher's outputs on unaugmented images. Patience: distillation benefits from far longer training than ordinary supervised learning, because the student is fitting a rich function rather than a set of labels, and gains continued over schedules of hundreds or thousands of epochs. Following these practices, they distilled a large teacher into a standard ResNet-50 that set a new accuracy level for that architecture on ImageNet. These findings sit together comfortably. A student does have a capacity limit, and a teacher far beyond it may not help. But most students are well short of that limit when trained briefly and inconsistently, and the practical first step is to train longer, with consistent augmentation, before concluding that a smaller teacher is needed. There is a subtler caution. Samuel Stanton and colleagues, in "Does Knowledge Distillation Really Work?" at NeurIPS 2021, showed that even when distillation improves a student's accuracy, the student often agrees with its teacher's predictions much less closely than one would expect — a gap they traced to the difficulty of optimisation rather than to a lack of capacity.6 The practical implication is that a distilled student should be evaluated as a model in its own right, on accuracy against ground truth and on the failure cases that matter, rather than assumed to inherit the teacher's behaviour. Distillation for language models on devices Distillation has become central to how small language models are built. Rather than training a small model from scratch on next-token prediction, developers increasingly train it to match a larger model's full predicted distribution over the vocabulary at each position — a much richer signal than the single correct next token. Google's technical report for its Gemma 2 open models, for example, describes training the smaller models in the family with knowledge distillation from a larger model rather than with standard next-token prediction alone, and attributes much of their quality to it. Other model families have followed similar approaches. A second form, sequence-level distillation, described by Yoon Kim and Alexander Rush in 2016 for translation, trains the student on complete outputs generated by the teacher rather than on token-level distributions. Its modern descendant is the widespread practice of generating synthetic training data with a large model to fine-tune a small one for a narrow task. For an edge developer who needs a small on-device model to perform one job well — extracting fields from receipts, classifying support messages, summarising notifications — fine-tuning a compact model on a large set of teacher-generated examples is often the most effective route to acceptable quality within a phone's budget. Distillation and quantization together Distillation and quantization interact well. The cleanest combination is to distil a floating-point student first, then quantize it with the methods of Chapters 3 and 4. When quantization-aware training is needed, the teacher can remain in the loop: the floating-point teacher supervises the fake-quantized student, so that the quantized student is trained to match a full-precision target rather than hard labels. This is sometimes called quantization-aware distillation, and it frequently recovers more accuracy than QAT on labels alone, especially at low bit-widths. A special case uses the floating-point version of the same model as the teacher for its own quantized version; the soft targets guide the quantized network back towards the behaviour it had before quantization. Mistakes that quietly waste distillation Distillation is simple enough to implement in a few lines, which makes it easy to implement slightly wrongly. Several errors recur, and each costs accuracy without producing an obvious failure. The first is running the teacher in training mode. If the teacher contains dropout or batch normalisation, it must be in evaluation mode while producing targets; otherwise dropout randomly corrupts its outputs and batch normalisation uses the statistics of the student's current mini-batch rather than its own learned statistics. The student then imitates a noisier, worse teacher than the one that was evaluated. The second is caching teacher outputs computed on unaugmented inputs and then training the student on augmented ones. This is tempting because it saves the cost of running the teacher during student training, but it means the student is asked to produce, for a cropped and colour-shifted image, the teacher's output for the original — a mismatch that Beyer and colleagues identified as a principal reason distillation underperforms. If teacher outputs must be cached for cost reasons, cache them for a fixed set of augmented views and train the student on exactly those views. The third is mishandling temperature. The student's softened outputs must be computed at the same temperature as the teacher's for the distillation term, while the label term and inference use temperature one; forgetting the T-squared factor silently shifts the balance between the terms as temperature changes, so that tuning temperature also tunes, unintentionally, the weighting. The fourth is distilling on the wrong data. A transfer set drawn only from the labelled training set wastes the method's greatest strength. A transfer set drawn from a different distribution — public web images for a model that will see infrared night frames — transfers the teacher's knowledge about the wrong world. The most valuable transfer data is unlabelled data from the deployment environment. The fifth is evaluating against the teacher instead of the task. A student that agrees with its teacher on 95 per cent of inputs has learned the teacher's mistakes along with its successes. The measures that matter are accuracy against ground truth and behaviour on the failure cases the product cares about. Distillation also extends beyond classification in ways that matter for edge work. For regression outputs — a predicted distance, a remaining-useful-life estimate for a machine component — the student can be trained to match the teacher's predictions directly, which smooths noisy labels in much the way soft targets do. For object detectors, matching the teacher's class scores and box predictions on the anchors or regions where the teacher found objects, plus intermediate features, works better than matching outputs everywhere, where background dominates. And self-distillation, in which a network is trained to match a previous generation of itself with the same architecture, often yields a modest but reliable improvement at no cost in size — a useful last step when no larger teacher is available. A practical recipe For a developer adding distillation to an edge project, the following sequence captures most of the benefit with modest effort. 1. Obtain the strongest teacher you can run during training. It need not fit on any device, and it can be an ensemble or a large pretrained model fine-tuned for the task. 2. Choose the student architecture for the device, as in Chapter 2, and verify that it fits the budget before training. 3. Assemble the largest representative transfer set you can, including unlabelled in-domain data from the deployment environment. 4. Train with the combined loss — labels plus softened teacher outputs — computing the teacher's outputs online on the same augmented inputs the student sees. 5. Train for longer than you would without distillation, and watch validation accuracy rather than training loss to decide when to stop. 6. If the student underperforms and the teacher is very much larger, try a smaller or earlier-stopped teacher, or an intermediate teacher assistant. 7. Quantize the student, using the teacher to supervise quantization-aware training if post-training quantization falls short. Distillation is the one technique in this booklet that typically improves a model's accuracy at a fixed size, rather than trading accuracy for size. That makes it the best investment available once the architecture has been sized to the device. The next chapter turns from the model to that device — and in particular to the specialised hardware that increasingly determines what a well-designed model can achieve on it. CHAPTER 7 Micro-NPUs and Neural Engines: What Accelerators Demand For most of the history of TinyML, running a neural network on a microcontroller meant running it on the CPU. Optimised kernel libraries squeezed remarkable performance from Cortex-M cores — Arm's CMSIS-NN, first described in 2018, reported several-fold improvements in speed and energy over naive implementations by exploiting the cores' single-instruction, multiple-data instructions1 — but a general-purpose core running at a few hundred megahertz can only do so much. A modest vision model might take hundreds of milliseconds, too slow for interactive use and too energy-hungry for continuous operation. That has changed. Neural processing units, long standard in phones, have moved down into microcontrollers and low-cost application processors. Arm announced its first micro-NPU, the Ethos-U55, in February 2020, alongside the Cortex-M55 processor, claiming that the pair could deliver up to 480 times the machine-learning performance of previous Cortex-M systems.2 Silicon vendors have since shipped parts built on that design and its successors, and several have designed NPUs of their own. A developer choosing hardware for an edge AI product today can buy a microcontroller with a dedicated accelerator for a few dollars. Accelerators change the economics of edge AI dramatically, but only for models that suit them. This chapter explains what an NPU actually is, what it demands of a model, and how to tell whether a given model will run well on one. What an NPU is An NPU is not a faster CPU. It is a fixed-function or narrowly programmable engine built around three components. The first is an array of multiply-accumulate units, typically 8-bit integer multipliers feeding wider accumulators. A micro-NPU might have between 32 and a few thousand such units; a phone NPU has many more. Each clock cycle, the array performs one multiply-accumulate per unit, so a 256-unit array at 500 megahertz performs 128 billion multiply-accumulates per second — conventionally quoted as 256 billion operations per second, or 0.256 TOPS, because each multiply-accumulate counts as two operations. The second is local SRAM close to the array, together with direct memory access engines that move data between that SRAM and the system's larger, slower memory. The array is only useful if it is fed, and feeding it is the main design problem. A good NPU tiles each layer's computation so that blocks of weights and activations are loaded into local memory, reused as many times as possible, and written back only when finished. The third is a command stream: a precompiled sequence of instructions describing which operations to perform, on which tiles, in which order, with which data movements. The NPU does not interpret a neural network graph at run time. An offline compiler translates the model into this command stream ahead of time, deciding the tiling, scheduling, and memory placement. The quality of that compiler matters as much as the hardware. Many NPUs add specialised units: a weight decoder that decompresses weights on the fly as they are fetched; elementwise units for additions and activations; lookup tables for non-linear functions; and pooling units. Each covers a specific set of operations, and the combination of all of them defines exactly what the NPU can do. The Ethos-U family as a worked example Arm's Ethos-U line is the most widely licensed micro-NPU design and illustrates the general pattern well, because its toolchain is open and its constraints are documented. The Ethos-U55 is designed to sit beside a Cortex-M processor in a microcontroller. It can be configured with 32, 64, 128, or 256 multiply-accumulate units per cycle, supports 8-bit weights with 8-bit or 16-bit activations, and decompresses weights in hardware. The Ethos-U65 is a larger relative intended for systems with more memory bandwidth, including those with an application processor; NXP's i.MX 93, for example, pairs Cortex-A55 cores with an Ethos-U65. The Ethos-U85, announced in April 2024 as the third generation, scales from 128 to 2,048 units per cycle, which Arm quotes as 256 billion to 4 trillion operations per second at a one-gigahertz clock, and added hardware support for transformer operations such as matrix multiplication, transpose, and gather, as well as native 2:4 sparsity.3 The deployment flow is the same across the family. The developer trains a model, quantizes it to 8-bit integers (or 16-bit activations with 8-bit weights), and exports it as a LiteRT flatbuffer. Arm's open-source Vela compiler then reads that file, identifies the operators the NPU can execute, compiles them into a command stream, compresses their weights, and writes out a new model file in which the supported portions are replaced by a single custom operator containing the command stream. At run time, LiteRT for Microcontrollers executes the model on the Cortex-M; when it reaches the custom operator, it hands the command stream to the Ethos-U driver, and the NPU runs it. Any operators Vela could not place on the NPU remain in the model as ordinary operators and run on the Cortex-M, using CMSIS-NN kernels where available. Vela produces a report, and reading it is the single most useful thing a developer can do when targeting an Ethos-U part. It lists, operator by operator, whether each was placed on the NPU or left for the CPU, and why; it estimates cycle counts and memory traffic for the NPU portions; and it reports peak SRAM usage. A model that Vela maps entirely to the NPU will typically run many times faster than on the CPU alone. A model in which a few operators in the middle of the network fall back to the CPU may run disappointingly, because each fallback splits the NPU's work into separate command streams, forces intermediate tensors through memory, and puts the comparatively slow CPU on the critical path. Operator coverage is the first constraint Every NPU supports a defined list of operators, each with its own conditions. A convolution may be supported only for certain kernel sizes, strides, and dilation factors; a pooling operation only up to a maximum window; an elementwise operation only when its inputs' shapes broadcast in particular ways; a reshape only in some positions. Activation functions beyond the common ones may be handled through lookup tables with specific constraints on quantization parameters. The lists are published in each vendor's documentation and enforced by their compilers. For model design, this means the operator constraints of the target accelerator should be checked before an architecture is committed to — the same advice Chapter 2 gave for runtimes, with higher stakes. The commonest causes of fallback are familiar: an unusual activation function, a normalisation layer that cannot be folded, an operator introduced by a framework's export path that the developer never wrote explicitly (a transpose inserted to change memory layout, a cast between data types, a dynamic reshape), and post-processing operations such as non-maximum suppression appended to the end of a detection model. Some of these can be fixed by changing the export settings; others require changing the model; post-processing usually belongs on the CPU anyway and should be separated from the network. A useful discipline is to run the target compiler on a candidate architecture in its first, untrained form — random weights quantized with dummy calibration data — before investing in training. The compiler's report on operator placement and memory use depends on the architecture, not on the trained weights, and an incompatibility discovered in the first hour costs nothing. The cost of a single fallback is worth reasoning through, because it is routinely underestimated. Imagine a network of twenty layers in which the NPU can run every layer except the tenth. Without that fallback, the whole network is one command stream: the NPU runs from input to output, keeping intermediate tensors in its working memory and overlapping data movement with computation. With it, the work splits into three pieces. The NPU runs layers one to nine and writes the ninth layer's output to SRAM; the CPU reads it, runs layer ten with its own kernels, and writes the result back; the NPU starts again with layers eleven to twenty. The CPU portion may be only one layer out of twenty, but the CPU might be ten or twenty times slower than the NPU at that operation, and the hand-offs add synchronisation, cache maintenance, and lost overlap. It is common for one or two fallbacks to account for a large share of total inference time, which is why a placement report showing "nearly all" operators on the NPU is not the good news it sounds like. Memory is still the second constraint The arguments of Chapter 1 apply to accelerators with extra force, because an NPU's arithmetic is so fast that almost any layer can become memory-bound. On a microcontroller with an Ethos-U55, the NPU usually has no large memory of its own; it reads weights from flash or external memory and reads and writes activations in the microcontroller's SRAM. If weights must be fetched from slow external flash, the NPU waits on them; Vela's compression helps, and so does placing weights in faster internal memory when they fit. If activations do not fit in on-chip SRAM, they must spill to external RAM, and throughput falls sharply. Vela offers several memory modes reflecting different system designs — everything in SRAM, weights in flash with activations in SRAM, or a dedicated SRAM used as a cache in front of slower memory on the larger Ethos-U parts — and the right choice depends on the board, not the model. This is why advertised throughput is a ceiling rather than a prediction. The TOPS figure quoted for an NPU assumes every multiply-accumulate unit is busy every cycle. Real models achieve some fraction of that, depending on how well each layer's shape fits the array, how much each layer reuses its data, and whether memory can keep up. Arm, for instance, describes the Ethos-U85 as reaching up to 85 per cent utilisation on popular networks, which is a good result and still implies a gap between the headline and practice;3 depthwise convolutions and small layers typically fall well short of whatever figure a vendor quotes. Comparisons between accelerators on TOPS alone are close to meaningless, all the more so because vendors do not always quote the same data type — some figures assume 4-bit arithmetic or count skipped operations in sparse layers. A landscape of edge accelerators The range of available accelerators is wide and changes quickly, but a few representative examples, set out in Table 4, illustrate the classes a developer is likely to meet. Two cautions apply to reading it. The Ethos-U entries are licensed designs rather than chips, so their real throughput depends on the configuration and clock speed each silicon vendor chooses; the other entries are specific products. And the figures are peak ratings quoted by their makers, on the terms each maker chose, so they indicate scale rather than support a direct comparison. Table 4. Representative edge neural accelerators. Part Host Peak int8 Notes Ethos-U55 Cortex-M 32–256 MACs/cycle Licensed IP; many MCU vendors Ethos-U65 Cortex-M or A Up to 512 MACs/cycle Higher-bandwidth systems Ethos-U85 Cortex-M or A 256 GOPS–4 TOPS at 1 GHz Transformer ops; 2:4 sparsity Neural-ART STM32N6 (M55) 600 GOPS ST's in-house NPU, 2024 Edge TPU USB/PCIe module 4 TOPS About 2 TOPS per watt Source: vendor announcements from Arm (2020, 2024), STMicroelectronics (December 2024), and Google Coral documentation. Beyond the Ethos-U designs, which appear in microcontrollers from vendors such as Alif Semiconductor, Infineon, and Himax as well as in NXP's application processors, several silicon makers now design their own. STMicroelectronics' STM32N6, launched in December 2024, pairs an 800-megahertz Cortex-M55 with an in-house accelerator it calls the Neural-ART Accelerator, rated at 600 billion operations per second, together with a hardware image-processing pipeline for camera input. NXP's MCX N microcontrollers carry its own eIQ Neutron NPU. Google's Coral Edge TPU, available as a USB accelerator, a PCIe card, or a module, is rated at 4 TOPS at about two TOPS per watt and runs models compiled with its own Edge TPU compiler. It is worth noting a less dramatic but important development alongside the NPUs: vector extensions to the microcontroller CPU itself. The Cortex-M55 and Cortex-M85 implement Helium, Arm's M-profile vector extension, which lets the CPU process several 8-bit values per instruction. Arm's own figure for the Cortex-M55 alone was up to fifteen times the machine-learning performance of earlier Cortex-M cores.2 Helium matters even in systems with an NPU, because it accelerates the operators that fall back to the CPU, as well as pre-processing such as audio feature extraction. Phone NPUs and their gateways Phone NPUs are far more powerful than micro-NPUs, but developers rarely program them directly. They are reached through vendor or platform software layers, and the choice of layer often matters more than the hardware. On Apple devices, the Neural Engine is reached through Core ML. The developer converts a model to Core ML's format, and the framework decides at run time how to divide the work among the CPU, GPU, and Neural Engine. The developer can express a preference but not force placement, and operators or configurations the Neural Engine does not support run elsewhere. Apple's tools report which compute unit each layer ran on, and checking that report is the equivalent of reading Vela's. On Android, the landscape is more fragmented. Google's Neural Networks API, intended as a common interface to vendor accelerators, was deprecated as of Android 15, and the recommended path is now to use a runtime — LiteRT or ExecuTorch — with delegates or backends for specific accelerators: a GPU delegate, and vendor-specific delegates for Qualcomm's Hexagon NPU, MediaTek's APU, and others. Each delegate takes the portion of the graph it supports and leaves the rest on the CPU, exactly as Vela does on a microcontroller. ExecuTorch's 1.0 release in October 2025 listed production-ready backends for Qualcomm's AI Engine, Apple's Core ML, Arm's Ethos-U, Vulkan GPUs, and XNNPACK on CPUs, with further vendor backends at earlier stages.4 The practical difficulty on Android is that a model tuned for one vendor's NPU may run poorly or fall back to the CPU on another's. Applications that must run across many devices commonly ship a model that runs well on CPU and GPU everywhere, with accelerator delegates enabled where they have been validated, and fall back gracefully elsewhere. Choosing hardware with the model in mind For teams choosing hardware rather than inheriting it, the questions follow from everything above. What does the model actually need? Compute the multiply-accumulates per inference and the inference rate. A keyword spotter running a few million multiply-accumulates ten times a second does not need an NPU; a capable Cortex-M with Helium handles it with ample margin. A camera-based detector running hundreds of millions of multiply-accumulates at several frames per second does. Does the accelerator support the model's operators? Run the vendor's compiler on the candidate model and read the placement report. Partial support can be worse than none. Does the memory system support the accelerator? Check how much on-chip SRAM is available for activations and how fast weights can be fetched. An NPU starved by slow external flash will not reach its rated performance. Is the toolchain mature? An accelerator is only as good as its compiler, runtime, and documentation. Open, well-documented flows with active communities save weeks of debugging. What is the energy per inference, measured? Vendor figures are useful starting points, but energy per inference on the developer's own model, including the CPU, memory, and idle power consumed while waiting, is what determines battery life. These questions point to a single conclusion. An accelerator is not a feature that a model benefits from automatically; it is a target that the model must be built for. The compression techniques of the preceding chapters produce models that are small and integer-valued; the accelerator rewards models that are also made of operators it supports, arranged in shapes it handles efficiently, with activations that fit in its memory. The next chapter turns to the software that connects the two — the toolchains that turn a trained model into something a device can execute, and the profiling that shows whether it worked. CHAPTER 8 From Checkpoint to Device: Toolchains, Runtimes, and Honest Measurement A trained model is not a deployable artefact. Between the checkpoint a training script saves and the binary that runs on a phone or microcontroller lies a chain of tools — graph capture, conversion, quantization, backend compilation, runtime integration — each of which can change the model's behaviour in small ways, and any of which can silently break it. Most of the time lost in edge AI projects is lost here, not in model design. A model that scored well in a notebook and then misbehaves on the device has usually been altered somewhere along this chain, or is being fed different input than it was trained on. This chapter describes the chain as it stands in 2026, the runtimes that sit at its end, and the measurement practices that make problems visible early. The tool names will change; the shape of the chain and the discipline of measuring along it will not. The shape of the chain Every deployment path, whatever its branding, performs the same five steps. Capture. The model's computation is extracted from the training framework as a static graph: a fixed list of operators, their connections, and their weights, free of the Python control flow that training code relies on. In TensorFlow, this is a SavedModel or a traced Keras function. In PyTorch, the current mechanism is torch.export, which traces the model into a graph of standard operators with explicit shapes. Conversion. The captured graph is translated into the representation the target runtime understands: a LiteRT flatbuffer, an ExecuTorch program, an ONNX file, a Core ML package. Conversion maps framework operators onto runtime operators, folds constants, fuses patterns such as convolution followed by batch normalisation and activation into single operators, and may rewrite parts of the graph to suit the runtime. Quantization. Weights and activations are converted to integers, using the calibration and training techniques of Chapters 3 and 4. Depending on the path, quantization happens before conversion (quantization-aware training in the framework), during it (LiteRT's converter calibrates as it converts), or as a separate stage applied to the captured graph (PyTorch's export-based quantization). Backend compilation. Where an accelerator is involved, a vendor compiler or delegate partitions the graph, compiling the supported portions into the accelerator's own format — an Ethos-U command stream from Vela, a Core ML program for the Neural Engine, a Qualcomm context binary for Hexagon — and leaving the rest for the CPU. Integration. The result is embedded in the application: loaded from a file on a phone, compiled into the firmware image as a constant array on a microcontroller. The application provides inputs, invokes the runtime, and consumes outputs. At each step, the model can change. Conversion can substitute an operator with a numerically different implementation. Fusion changes the order of floating-point operations. Quantization introduces error by design. Backend compilation may use different rounding or saturation from the reference kernels. And integration can feed the model input that differs from training input in ways that no tool can detect. Knowing where each change can enter is the first defence against it. The runtimes Three runtime families cover most edge deployments today. LiteRT, formerly TensorFlow Lite and renamed by Google in September 2024, remains the most widely deployed. Models are stored as flatbuffers — a serialisation format that can be read in place without parsing or copying, which suits memory-mapped loading on phones. On Android and embedded Linux, an interpreter executes the model with optimised CPU kernels, chiefly from the XNNPACK library, and hands sub-graphs to delegates for GPUs and NPUs. Despite its origins, LiteRT now accepts models authored in PyTorch and JAX as well as TensorFlow through Google's AI Edge conversion tools. LiteRT for Microcontrollers, formerly TensorFlow Lite Micro, is a separate, stripped-down runtime for devices without an operating system. Its design, described by Robert David and colleagues at MLSys in 2021, follows from the constraints of Chapter 1.1 It performs no dynamic memory allocation during inference; the application supplies the tensor arena, and the runtime plans all activations within it at initialisation. It includes only the operators the application registers through an operator resolver, so that unused kernels do not consume flash. It runs the same flatbuffer format as the larger runtime, and it accepts optimised kernel implementations — CMSIS-NN for Cortex-M, vendor libraries for DSPs — in place of its portable reference kernels. ExecuTorch is PyTorch's on-device runtime, which reached its 1.0 release in October 2025.2 Its flow begins with torch.export, lowers the exported program to a restricted operator set suitable for edge devices, and then hands partitions of the graph to backends — XNNPACK for CPUs, Core ML, Qualcomm's AI Engine, Arm's Ethos-U, Vulkan, and several others — before serialising the result as a program file the runtime executes. A selective build mechanism links only the operators a given model uses. Its main attraction for many teams is that it removes a framework conversion step: a model developed in PyTorch stays in PyTorch's operator semantics until it reaches the backend, reducing one common class of conversion mismatches. Meta reports that it runs inside its own applications on billions of devices. Alongside these sit ONNX Runtime, which offers a mobile build and execution providers for various accelerators and is common where models arrive in ONNX format from mixed toolchains; vendor toolchains such as ST's tools for STM32 parts and NXP's eIQ, which wrap the general runtimes or generate their own code for each vendor's chips; and code-generating compilers, which skip the interpreter entirely and emit C code specialised to one model, as MCUNet's TinyEngine did. Code generation can save memory and time by eliminating interpretive overhead and planning memory more aggressively, at the cost of a less general deployment flow. The choice among them is usually dictated by the target: the vendor's supported path for a given accelerator, the platform's preferred runtime for a phone, and the team's training framework. Where there is freedom, the deciding factors should be operator coverage on the target, maturity of the backend for the specific accelerator, and quality of the profiling tools — the last being consistently underrated. The pre-processing trap The most common cause of a model that works in development and fails on the device is not quantization or conversion. It is a mismatch between how inputs were prepared during training and how they are prepared on the device — what machine-learning engineers call training-serving skew. Training pipelines typically prepare inputs with high-level library calls: decode an image, resize it with a particular interpolation method, convert its colour space, normalise it with particular means and standard deviations. Audio pipelines compute spectrograms with particular window sizes, hop lengths, and mel filterbanks, apply a logarithm, and perhaps normalise. On the device, those steps are reimplemented, often in C, often by a different engineer, often with a different library — and small differences abound. An image arrives in BGR rather than RGB order. A camera delivers YUV data and the conversion uses a slightly different matrix. Resizing uses nearest-neighbour rather than bilinear interpolation. An audio feature extractor uses a different window function, a different number of mel bands, or a natural logarithm where training used base ten. Each difference shifts the input distribution, and the model, which has never seen such inputs, degrades. Quantization adds a subtler variant. A fully integer model expects its inputs already quantized with the input tensor's scale and zero point, which are stored in the model. If the application quantizes inputs with different parameters — or, very commonly, feeds unsigned 8-bit pixel values to a model that expects signed 8-bit inputs — the model sees systematically shifted data. Three practices guard against this. Use the same pre-processing code, or a bit-exact port of it, in training and on the device; where possible, generate the device's feature extraction code from the same source, or move pre-processing into the model graph itself so that it is converted with everything else. Record golden test vectors — raw sensor inputs, the pre-processed tensors computed from them by the training pipeline, and the expected outputs — and check them on the device, comparing each stage separately. And include the device's own sensor in data collection as early as possible, so that the model is trained on data captured through the same microphone, lens, and analogue front end it will see in the field. Testing for equivalence Chapter 3 recommended a three-way comparison: floating-point model, quantized model in the desktop interpreter, quantized model on the device. The full chain suggests a more complete ladder of checks, each comparing adjacent stages on identical inputs. 1. Training framework, floating point, versus converted model, floating point. Differences should be at the level of floating-point rounding; anything larger indicates a conversion problem. 2. Converted floating point versus converted integer, on the desktop interpreter. This measures quantization loss, and should be checked on the full validation set. 3. Desktop integer versus device integer, CPU only. For a correct integer implementation this should be bit-exact or nearly so; differences indicate kernel bugs, different library versions, or integration errors. 4. Device CPU versus device with accelerator. Small differences are possible where the accelerator uses different rounding; large ones indicate a compiler or driver issue. 5. Golden sensor input versus device pre-processing. Run the device's feature extraction on recorded raw input and compare with the training pipeline's output. Automating these comparisons, so that each new model version runs through them before release, is one of the highest-value investments an edge AI team can make. The checks are cheap; the failures they catch are expensive. Profiling where the time goes A single end-to-end latency figure tells the developer whether the model meets its deadline, but not why it does or does not. For that, per-operator profiling on the target is essential. On microcontrollers, LiteRT for Microcontrollers includes a profiler interface that records the time spent in each operator invocation, and Cortex-M processors provide a hardware cycle counter that can time any section of code to the cycle. On Android and embedded Linux, LiteRT's benchmark tool runs a model on the device for a fixed number of iterations, with warm-up runs, and reports average latency and per-operator breakdowns with or without delegates. ExecuTorch provides developer tools that record execution traces on the device and map them back to the original PyTorch operators. Apple's Xcode generates performance reports for Core ML models showing which compute unit each layer ran on and how long it took. Per-operator profiles routinely reveal surprises. A handful of operators dominate; often one or two layers account for half the latency. An operator that was expected to run on the accelerator is on the CPU. A reshape or transpose inserted by the converter is costing more than the convolution beside it. Pre-processing, running outside the model, takes longer than the model. Each of these suggests a specific fix, and none is visible from end-to-end timing alone. Measuring energy For battery-powered devices, energy must be measured, not estimated. The standard method is to power the device through an instrument that records current at high time resolution — a dedicated power analyser or one of the inexpensive current-profiling tools common in embedded development — and to integrate current times voltage over an inference to obtain energy. The device should toggle a pin at the start and end of inference so that the trace can be aligned with the computation. Measurements should cover the whole cycle: waking from sleep, capturing sensor data, pre-processing, inference, post-processing, any radio activity, and returning to sleep. Inference alone is often not the largest consumer. A model that halves inference energy but requires the processor to stay awake longer for pre-processing may save nothing. Benchmarks and what they are for Public benchmarks offer a common reference point for comparing hardware and software stacks. The most relevant for microcontroller-class devices is MLPerf Tiny, developed under MLCommons by a consortium led by Colby Banbury, Vijay Janapa Reddi, and colleagues from more than fifty organisations, and first described in 2021.3 It defines four tasks with fixed datasets, reference models, and accuracy thresholds, and measures latency and, optionally, energy on submitted systems. Table 5 lists its tasks. Table 5. The MLPerf Tiny benchmark tasks. Task Dataset Reference model Size Quality target Keywords Speech Commands DS-CNN 52.5 KB 90% top-1 Person Visual Wake Words, 96×96 MobileNetV1 325 KB 80% top-1 Images CIFAR-10 ResNet 96 KB 85% top-1 Anomaly ToyADMOS FC autoencoder 270 KB 0.85 AUC Source: Banbury et al., "MLPerf Tiny Benchmark" (NeurIPS Datasets and Benchmarks, 2021), Table 1. The reference models are instructive in themselves. They show the scale of model that serves real tasks on microcontrollers: tens to a few hundred kilobytes of weights, input resolutions far below what server models use, and accuracy targets set at what the application needs rather than what a leaderboard rewards. Published results from the benchmark's rounds are a useful starting point for comparing hardware platforms on comparable workloads. They are not a substitute for measuring the developer's own model. Benchmark submissions are heavily tuned; the reference models use operators that every platform supports well; and a system that leads on keyword spotting may be ordinary on a model with different operators or memory behaviour. Benchmarks narrow the candidate list. Measurements of the actual model on the actual hardware make the decision. Fitting it into firmware and apps A few integration details recur often enough to note. On microcontrollers, the model is usually compiled into the firmware as a constant byte array, placed in flash, and aligned as the runtime requires; misalignment produces crashes that look like model bugs. The tensor arena must be sized by measurement — the runtime can report how much of the arena was actually used after a test inference — with a margin for future model updates. Over-the-air update schemes that keep two firmware images double the flash required for the model, and some products store the model in a separate flash partition so that it can be updated independently of the application. On phones, models are often too large to ship inside the application package and are downloaded on first use, which raises questions of versioning, integrity checking, and compatibility between model versions and application versions. Accelerator compilation caches should be kept between launches so that cold-start costs are paid once. Pin the toolchain, version the model Every tool in the chain changes from release to release, and those changes can alter the deployed model. A new converter version may fuse operators differently, choose a different implementation of an activation function, change the default quantization scheme for some operator, or produce a model that an older on-device runtime cannot read. A new accelerator compiler may tile layers differently, changing both latency and, occasionally, numerical results. A team that regenerates its model with whatever tool versions happen to be installed will eventually ship a model that differs from the one it tested, without anyone having changed the model. The defence is the same as for any other build artefact: make it reproducible. Pin the exact versions of the training framework, converter, quantization tools, and backend compiler used to produce each released model, ideally in a container image that can be rebuilt later. Record the calibration data set, by content hash rather than by file path. Embed metadata in the model file itself — a version string, the hash of the source checkpoint, the toolchain versions, and the input pre-processing parameters — so that a device in the field can report exactly which model it is running and a developer can reconstruct how it was produced. LiteRT's flatbuffer format supports arbitrary metadata for this purpose, and the other formats have equivalents. Runtime and model must also stay compatible. On a microcontroller, the runtime is compiled into the firmware, so a new model that uses an operator the deployed firmware did not register, or relies on a newer version of an operator's kernel, will fail to initialise on devices in the field. Firmware should check a model's required operator set and schema version before switching to it, and keep the previous model available if the check fails. On phones, where the runtime is part of the application or the operating system, the same check belongs in the code that downloads models. None of this is glamorous, and none of it appears in papers. It is, nonetheless, where edge AI projects succeed or fail. The final chapter assembles everything — budget, architecture, compression, hardware, toolchain, and measurement — into a single pipeline, and walks it end to end. CHAPTER 9 A Complete Pipeline: From Requirement to Fielded Model The preceding chapters examined techniques one at a time. Real projects do not encounter them that way. They encounter a product requirement, a device, a deadline, and a stream of decisions whose order matters as much as their content. This chapter walks through two projects end to end — one on a microcontroller, one on a phone — to show how the pieces fit together and where projects typically go wrong. The projects are composites, built from the patterns that recur across many real deployments; the numbers in their budgets are design choices of the kind a team would make, not measurements of any particular product. Project one: a battery-powered person detector The requirement: a small, battery-powered sensor mounted in a corridor or doorway must report whether a person is present, for building occupancy analytics. It must run for at least two years on its battery, must never transmit images, and must report presence within about one second of a person entering its view. The device is a microcontroller with a Cortex-M55 core, an Ethos-U55 micro-NPU configured with 128 multiply-accumulate units, around 1.5 megabytes of SRAM, several megabytes of flash, a low-power monochrome image sensor, and a low-energy radio. Step one: write the budget Following Chapter 1, the team writes the budget before touching a model. For memory, the firmware — real-time kernel, radio stack, sensor driver, update logic, and the inference runtime — is estimated at a few hundred kilobytes of flash, and the update scheme keeps two images, so the model's weights are allowed half a megabyte with margin. For SRAM, the frame buffer for a low-resolution greyscale image, the radio's buffers, and the stack consume a known share; the tensor arena is allowed 400 kilobytes. For time, one second end to end is generous for inference, but energy turns it into a tighter constraint. The battery's capacity and the two-year target give an average power budget, measured in tens of microwatts. Sleep current consumes much of it. Each wake-capture-infer-report cycle must therefore cost little enough that the expected number of cycles per day fits within what remains. That arithmetic immediately shapes the design. Running the neural network continuously at several frames per second would blow the energy budget many times over. The device needs a cascade: a cheap trigger that runs always, and the neural network only when the trigger fires. The team chooses a hardware motion-detection mode in the image sensor, which compares successive low-resolution frames internally and raises an interrupt when enough pixels change, with the microcontroller asleep in between. When woken, the device captures frames and runs the network on a few of them before returning to sleep. The neural network's job is now to distinguish people from the other things that cause motion — doors swinging, light changes, a cleaning robot, a passing reflection. Step two: collect data through the device The team's first data collection uses the actual sensor, lens, and mounting height, not a public dataset. Public person-detection datasets such as Visual Wake Words are useful for pre-training, but they consist mostly of colour photographs taken at eye level in good light, while the device sees greyscale frames from a ceiling-height angle, often in dim corridors with fluorescent flicker. A few dozen prototype units installed in representative buildings for several weeks capture frames whenever the motion trigger fires. This produces exactly the distribution the network must handle: what the trigger sends it. Labelling is the bottleneck. Following Chapter 6, the team uses a large, accurate person detector, far too big for any edge device, to label the captured frames automatically, and has people review a sample — particularly the frames where the large model was uncertain — to correct errors and estimate label quality. Privacy rules for the collection are agreed with the building owners in advance; the prototype units are configured for data collection only, with signage and retention limits, and the production devices will never store or transmit images. The team also sets aside slices of the evaluation data: night-time frames, frames with partial occlusion, frames where a person is far from the sensor, frames with flicker. Overall accuracy can hide failures concentrated in one slice, and the product's credibility depends on all of them. Step three: choose and size the architecture The team selects a MobileNet-style architecture, as recommended in Chapter 2, and checks two constraints before training anything. First, operator support. A candidate network is instantiated with random weights, converted to an 8-bit LiteRT model with a few dummy calibration samples, and passed through Vela. The report shows every operator mapped to the NPU except the final softmax, which is moved out of the model and computed trivially in application code. It also shows that one candidate using a hard-swish activation in an early layer is placed on the NPU through a lookup table, while another variant's unusual pooling configuration falls back to the CPU; the latter is changed. Second, size. The team trains small proxy models at three input resolutions — 64, 96, and 128 pixels square — and three width multipliers, measuring both accuracy on the evaluation slices and, through Vela's estimates and early on-device runs, peak SRAM and latency. The highest resolution helps markedly on the "person far from sensor" slice, where a distant figure occupies only a few pixels at lower resolutions, and costs too much memory at the largest width. The chosen design runs at 128 by 128 with a moderate width, at roughly 300 kilobytes of int8 weights and a peak activation footprint comfortably inside the arena. That choice — driven by a slice the team would not have examined without collecting its own data — is the most consequential of the project. Step four: train well, then quantize The chosen architecture is pre-trained on a public person dataset converted to greyscale, then fine-tuned on the team's device-captured data. Fine-tuning uses distillation from the large detector's soft per-frame person score, combined with the reviewed labels, with consistent augmentation for teacher and student — random crops, brightness and contrast changes, simulated sensor noise, and synthetic flicker — and a long schedule, as Chapter 6 recommends. Distillation improves accuracy on the night-time slice most, where the teacher's graded confidence carries information that hard labels lose. Quantization follows Chapter 3: full-integer post-training quantization with per-channel weights, calibrated on several hundred frames drawn from every slice, including the darkest. The first attempt, calibrated on a random sample dominated by daytime frames, loses noticeably more accuracy at night than during the day; recalibrating with a deliberately balanced set fixes most of it. The remaining loss is small enough that quantization-aware training is not needed. The team checks the converted model on the full evaluation set, slice by slice, in the desktop interpreter. Step five: validate on the device The validation ladder from Chapter 8 runs next. Golden test vectors — raw sensor frames, the training pipeline's pre-processed tensors, and the expected integer outputs — are loaded into a test build of the firmware. The ladder catches two problems. The first is in pre-processing: the sensor delivers frames in a slightly different crop from the one the data collection firmware used, because a register setting differs between builds, so the production model sees a marginally zoomed view. The difference is small, but it shifts accuracy on the distant-person slice, and it would have been nearly impossible to diagnose from field reports. The second is in input quantization: the application converts pixels from unsigned to signed integers by subtracting 128, as the model expects, but one code path skips the subtraction after a sensor reset. Both bugs are fixed before any production unit ships. With the ladder clean, the team profiles. The NPU portion of the network takes a few tens of milliseconds; the per-operator profile shows that the image-scaling routine, running on the CPU before inference, takes almost as long. Rewriting it to use the Cortex-M55's Helium vector instructions halves it. The team then measures energy for the whole wake cycle with a current-profiling instrument, toggling a pin at each stage. Inference is not the largest consumer; waking and stabilising the image sensor is. Capturing two frames instead of four per wake, with temporal agreement required between them, saves more energy than any further model optimization could. Step six: decide, smooth, and ship A network outputs a score; the product needs a decision. The team chooses a threshold from the evaluation data by deciding which error is worse — missing a person or reporting a phantom one — and setting the threshold to meet the product's tolerance for each. They add simple temporal logic: presence is reported when two consecutive frames exceed the threshold, and absence only after a period without detections, suppressing flicker in the reported state. This logic is tested on recorded sequences, not just individual frames. Step seven: monitor and update Shipping is not the end. Models degrade in the field when the world they see differs from the world they were trained on — a new building with unusual lighting, a change of carpet, a seasonal shift in daylight. Since the devices must never transmit images, the team designs privacy-preserving telemetry: histograms of the network's output scores, counts of motion triggers and of positive detections per hour, and battery voltage. A shift in the score distribution, or a rise in the ratio of triggers to detections, flags a site for investigation, where — with the building owner's agreement — a separate data-collection unit can be installed temporarily. Model updates travel over the air in the dedicated model partition, signed, versioned, and checked against the firmware's runtime version. They are rolled out to a small fraction of devices first, with telemetry compared against the previous model before wider release, and the previous model retained for rollback. The flash budget set in step one, with its allowance for two images, is what makes this possible. Project two: an on-device language model in a phone app The second project is a phone application that summarises and categorises incoming messages locally, so that their content never leaves the device. The requirement is a short summary within a couple of seconds and a category label, on recent mid-range and flagship phones, without noticeable battery drain or heat. The budget is bandwidth and memory Chapter 1's arithmetic dominates. Generating each output token requires reading all of the model's weights, so the generation rate is bounded by memory bandwidth divided by weight bytes. The application can also claim only a modest share of the phone's memory before the operating system starts closing other applications or, worse, the app itself. Both point to the same conclusion: the fewer bytes the weights occupy, the better on every axis. The team considers models of around one, two, and four billion parameters. At four bits per weight, these occupy roughly half a gigabyte, one gigabyte, and two gigabytes. The largest leaves too little headroom on mid-range phones and generates too slowly to meet the latency target on them. The team chooses a model in the one-to-two-billion range for which the developer has published a quantization-aware-trained 4-bit checkpoint, following Chapter 4's advice to prefer such checkpoints to post-training quantization when they exist. Memory also goes to the key-value cache, which stores intermediate attention results for every token processed so far. Its size grows with the number of layers, the attention dimensions, and the length of the context, and for long inputs it can rival the weights. The team caps input length — long message threads are truncated to their most recent portion — and stores the cache at reduced precision, a supported option in their chosen runtime. Make the small model good at one job A general small model summarises adequately but categorises inconsistently, and its summaries are longer than the interface can show. Following Chapter 6, the team generates a large set of synthetic training examples with a much larger model — realistic messages, with high-quality summaries of the required length and correct category labels — and fine-tunes the small model on them, using parameter-efficient fine-tuning so that the adaptation adds only a small set of extra weights. The fine-tuned model is then re-quantized to 4 bits, and evaluated on held-out real messages contributed with consent by internal testers, slice by slice — short messages, long threads, multiple languages, messages containing mostly links or numbers. Run it where it runs best, and measure it hot The runtime choice follows from the platforms. On iOS, the team uses Core ML and checks Xcode's performance report to confirm which parts run on the Neural Engine. On Android, it uses a runtime with a GPU backend and vendor NPU backends where they have been validated, and a well-optimised CPU path everywhere else, because the device population includes chipsets from several vendors. Measurement is done the way Chapter 1 warned it must be: not on a cool phone straight after boot, but after several minutes of sustained use, with the application's interface active. The first measurements show that the NPU path on one chipset is fast for the first few summaries and then slows markedly as the device warms and throttles. The team schedules summarisation for messages as they arrive in small batches while the phone is idle and charging where possible, and limits the number of interactive summaries generated back to back. Cold-start compilation for the accelerator, several seconds on first launch, is moved to a background task after installation. Fail gracefully Not every phone can run the model well. The application measures a short benchmark on first launch and, on phones below a threshold, falls back to the category label alone, generated by a small classifier distilled from the language model — a model of a few megabytes that runs in milliseconds on any device. The classifier is the fruit of the same distillation pipeline and costs almost nothing to maintain alongside the main model. What the two projects share The two projects differ by several orders of magnitude in model size, yet the sequence of decisions is the same. The budget comes first, and it is written in bytes, milliseconds, and joules, measured on the real device in realistic conditions. Data comes from the deployment environment, through the deployment sensor, and is evaluated in slices that reflect where the product could fail. The architecture is chosen and sized against that budget and checked against the target's operator support before any expensive training. Distillation from a larger model makes the small model better than its size suggests. Quantization is applied with representative calibration and evaluated on the converted model, and heavier techniques are reserved for when lighter ones fail. Deployment is validated stage by stage against golden data, profiled per operator, and measured for energy across the whole cycle, where the largest costs often turn out to lie outside the network. And the model is monitored and updated in the field, with the budget having reserved room for that from the beginning. The specific techniques that mattered most were not the most sophisticated ones. In the first project, the decisive choices were the input resolution, driven by a slice of the data, and the number of frames captured per wake, driven by an energy measurement. In the second, they were the bytes per weight and the decision to specialise a small model for one task. Pruning played no role in either, because neither target could exploit it profitably, and quantization-aware training was needed only where a vendor had already done it. That pattern is typical, and it is the practical meaning of this booklet's controlling idea: the work that pays is the work that respects the memory budget of the actual device and checks itself against that device at every step. Conclusion: Engineering for the Device in Front of You The techniques in this booklet are often presented as a menu: quantize, prune, distil, accelerate, choose any combination. Having worked through each of them against the budget of real devices, a different picture emerges. They are not equals, and they are not interchangeable. They form a hierarchy, ordered by how reliably each one reduces the resource that edge hardware lacks most — bytes stored and bytes moved — on the hardware a developer actually has. At the top of that hierarchy sit two decisions that are not compression techniques at all. The first is the choice of architecture and its size, above all the input resolution and the width of the network, which together set the activation peak that decides whether a model fits a microcontroller and the memory traffic that decides how fast it runs on anything. The second is how well the chosen network is trained, which is where knowledge distillation earns its place: it is the one technique that reliably makes a small model better rather than trading its accuracy for size. Then comes 8-bit integer quantization, which cuts storage and traffic fourfold, unlocks integer accelerators, and costs little accuracy for a well-designed network. Below that sit the conditional techniques: low-bit quantization where memory bandwidth dominates, structured pruning where a network is over-sized, unstructured and semi-structured sparsity where the hardware can exploit them. And underneath all of it sits the work that is never published — matching pre-processing, validating the chain stage by stage, profiling per operator, measuring energy across the whole wake cycle — without which none of the rest can be trusted. What follows for practice Several practical consequences follow from that hierarchy, and they are worth stating as plainly as possible because they run against common habits. Start on the device, not in the notebook. The single most effective change most teams can make is to get a trivial model running on the target hardware in the first week — before collecting serious data or training anything good. A random-weight network passed through the real toolchain, deployed in the real firmware or application, and measured with the real profiler exposes operator gaps, memory limits, pre-processing pipelines, and integration problems while they are still cheap to fix. Projects that reach the device last discover their constraints last. Write the budget in the units that bind. Parameter counts and operation counts are convenient, but they are proxies. The budget that governs a deployment is written in bytes of flash and SRAM, milliseconds of sustained latency, and joules per inference. Every architectural and compression decision should be justified in those units, measured on the target. Spend effort in order. Before reaching for quantization-aware training, sparsity, or custom kernels, ask whether the input resolution is larger than the task needs, whether the network is wider than the budget allows, whether the model has been trained with a strong teacher, and whether post-training quantization has been given representative calibration data. Most of the available improvement usually lies in those questions, and they are the cheapest to answer. Treat the accelerator as a target, not a feature. An NPU accelerates the models built for it and quietly fails to accelerate the rest. Check its operator coverage and memory behaviour before the architecture is fixed, read its compiler's placement report for every model version, and judge it by measured latency and energy on your own network, never by its headline throughput. Evaluate the model you ship, where you ship it. Accuracy is a property of the converted integer model running on the device's kernels, fed by the device's sensor and pre-processing, measured across the slices of data where the product could fail. Any accuracy figure measured elsewhere is an estimate of that one. What remains unsettled The principles above are stable; several parts of the landscape are not. The numeric formats of edge inference are in flux. Eight-bit integers will remain the default for convolutional models on microcontrollers for years, but below that the industry has not settled. Integer formats at four bits and below, block-scaled formats in which groups of values share an exponent — such as the microscaling formats standardised through the Open Compute Project — and low-precision floating-point variants are all competing, with different hardware supporting different subsets. A developer targeting several devices today may need several quantized versions of the same model, and the toolchains for producing and validating them are less mature than those for 8-bit integers. Transformers on microcontroller-class hardware are at an early stage. Accelerators such as the Ethos-U85 have added the operators they require, but the memory demands of attention, the awkwardness of softmax and layer normalisation in integer arithmetic, and the scarcity of well-tested small transformer architectures for sensor tasks mean that convolutional and recurrent networks remain the safer choice for the smallest devices. That balance is likely to shift, but it has not shifted yet. Toolchain fragmentation remains the largest practical cost in the field. Each accelerator has its compiler, each phone vendor its delegate, each runtime its operator set, and a model that runs well through one path may fail or crawl through another. Recent consolidation — LiteRT's acceptance of models from several frameworks, ExecuTorch's growing set of production backends — helps, but developers targeting diverse hardware should still expect to spend real time on per-platform validation. Finally, the line between inference and learning on the device is beginning to blur. Personalising a model to a single user's voice or a single machine's vibration signature, and adapting to drift without sending data off the device, are increasingly practical. They raise their own questions of memory, energy, and validation, and they are beyond the scope of this booklet, but they will draw on the same discipline: the device's budget comes first. The device in front of you Edge AI is sometimes described as a matter of shrinking models until they fit. That description misses what makes the work succeed. The developers who ship good edge models are not, in the main, those with the most elaborate compression pipelines. They are those who know their device's budget precisely, design within it from the start, train small models as carefully as large ones, apply the simple and reliable techniques before the clever ones, and measure everything on the hardware that will run it. Those habits make intelligence a property of the device itself — available without a network, answering within a deadline, keeping private data private, and running for years on a battery. That is the promise of running AI at the edge, and it is kept or broken in the bytes. Notes Chapter 1 1. Pete Warden and Daniel Situnayake, TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers (O'Reilly Media, 2019). 2. Ji Lin, Wei-Ming Chen, Yujun Lin, John Cohn, Chuang Gan, and Song Han, "MCUNet: Tiny Deep Learning on IoT Devices," Advances in Neural Information Processing Systems 33 (NeurIPS 2020). arXiv:2007.10319. 3. Google AI Edge, "TensorFlow Lite is now LiteRT," Google Developers Blog, 4 September 2024; and LiteRT for Microcontrollers documentation, ai.google.dev/edge/litert/microcontrollers. 4. Song Han, Jeff Pool, John Tran, and William J. Dally, "Learning both Weights and Connections for Efficient Neural Networks," Advances in Neural Information Processing Systems 28 (NeurIPS 2015), arXiv:1506.02626, reproducing Mark Horowitz's 45 nm energy table; see also Mark Horowitz, "Computing's Energy Problem (and What We Can Do About It)," IEEE International Solid-State Circuits Conference (ISSCC), 2014. Chapter 2 1. Andrew G. Howard et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications," arXiv:1704.04861, 2017. 2. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 3. Andrew Howard et al., "Searching for MobileNetV3," IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 4. Danfeng Qin et al., "MobileNetV4: Universal Models for the Mobile Ecosystem," European Conference on Computer Vision (ECCV), 2024. arXiv:2404.10518. 5. Lin et al., "MCUNet" (as Chapter 1, note 2); and Ji Lin, Wei-Ming Chen, Han Cai, Chuang Gan, and Song Han, "MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning," NeurIPS 2021. 6. Aakanksha Chowdhery, Pete Warden, Jonathon Shlens, Andrew Howard, and Rocky Rhodes, "Visual Wake Words Dataset," arXiv:1906.05721, 2019. 7. Liangzhen Lai, Naveen Suda, and Vikas Chandra, "CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs," arXiv:1801.06601, 2018. 8. Arm, "Arm Ethos-U85: Addressing the High Performance Demands of IoT in the Age of AI," Arm Newsroom, 9 April 2024. Chapter 3 1. Benoit Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference," CVPR 2018. arXiv:1712.05877. 2. LiteRT 8-bit quantization specification, ai.google.dev/edge/litert (formerly tensorflow.org/lite/performance/quantization_spec). 3. Raghuraman Krishnamoorthi, "Quantizing Deep Convolutional Networks for Efficient Inference: A Whitepaper," arXiv:1806.08342, 2018. 4. Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling, "Data-Free Quantization Through Weight Equalization and Bias Correction," ICCV 2019. 5. Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort, "Up or Down? Adaptive Rounding for Post-Training Quantization," International Conference on Machine Learning (ICML), 2020. Chapter 4 1. Yoshua Bengio, Nicholas Léonard, and Aaron Courville, "Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation," arXiv:1308.3432, 2013. 2. Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha, "Learned Step Size Quantization," International Conference on Learning Representations (ICLR), 2020. 3. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer, "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale," NeurIPS 2022. 4. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han, "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models," ICML 2023. 5. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers," ICLR 2023. 6. Ji Lin et al., "AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration," Proceedings of Machine Learning and Systems (MLSys), 2024. Chapter 5 1. Han et al., "Learning both Weights and Connections for Efficient Neural Networks" (as Chapter 1, note 4). 2. Song Han, Huizi Mao, and William J. Dally, "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding," ICLR 2016. arXiv:1510.00149. 3. Asit Mishra et al., "Accelerating Sparse Deep Neural Networks," arXiv:2104.08378, 2021. 4. Arm, "Arm Ethos-U85" (as Chapter 2, note 8). 5. Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan, "Fast Sparse ConvNets," CVPR 2020. 6. Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf, "Pruning Filters for Efficient ConvNets," ICLR 2017. 7. Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag, "What Is the State of Neural Network Pruning?" MLSys 2020. 8. Michael Zhu and Suyog Gupta, "To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression," arXiv:1710.01878, 2017. 9. Jonathan Frankle and Michael Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks," ICLR 2019. Chapter 6 1. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, "Distilling the Knowledge in a Neural Network," arXiv:1503.02531, 2015. 2. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio, "FitNets: Hints for Thin Deep Nets," ICLR 2015. 3. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf, "DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter," arXiv:1910.01108, 2019. 4. Jang Hyun Cho and Bharath Hariharan, "On the Efficacy of Knowledge Distillation," ICCV 2019. 5. Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov, "Knowledge Distillation: A Good Teacher Is Patient and Consistent," CVPR 2022. 6. Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew Gordon Wilson, "Does Knowledge Distillation Really Work?" NeurIPS 2021. Chapter 7 1. Lai, Suda, and Chandra, "CMSIS-NN" (as Chapter 2, note 7). 2. Arm announcement of the Cortex-M55 and Ethos-U55, February 2020, as reported in "Cortex-M Gets Heavy-Duty Machine Learning," Electronic Design, 11 February 2020. 3. Arm, "Arm Ethos-U85" (as Chapter 2, note 8). 4. PyTorch, "Introducing ExecuTorch 1.0," PyTorch Blog, 22 October 2025. Chapter 8 1. Robert David et al., "TensorFlow Lite Micro: Embedded Machine Learning for TinyML Systems," Proceedings of Machine Learning and Systems (MLSys), 2021. 2. PyTorch, "Introducing ExecuTorch 1.0" (as Chapter 7, note 4). 3. Colby Banbury et al., "MLPerf Tiny Benchmark," NeurIPS Datasets and Benchmarks Track, 2021. arXiv:2106.07597. Further Reading Pete Warden and Daniel Situnayake, TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers (O'Reilly Media, 2019).** The book that named the field. Its project-by-project approach to keyword spotting, person detection, and gesture recognition on microcontrollers remains an excellent hands-on introduction, even though some tool names have since changed. Daniel Situnayake and Jenny Plunkett, AI at the Edge: Solving Real-World Problems with Embedded Machine Learning (O'Reilly Media, 2023).** A broader, workflow-oriented companion that covers problem framing, data collection, evaluation, and responsible deployment for edge AI products rather than individual techniques. Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer, Efficient Processing of Deep Neural Networks (Morgan & Claypool, 2020).** The standard reference on how neural network accelerators are designed, with a rigorous treatment of dataflow, data reuse, and why memory access dominates energy. Essential for anyone who wants to understand what an NPU does with a model. Markus Nagel et al., "A White Paper on Neural Network Quantization" (arXiv:2106.08295, 2021). A clear, practical account of post-training quantization and quantization-aware training from the Qualcomm research group behind several of the techniques in this booklet, with recommended pipelines and debugging advice. Amir Gholami et al., "A Survey of Quantization Methods for Efficient Neural Network Inference" (arXiv:2103.13630, 2021). A thorough survey of quantization schemes, granularities, and training methods, useful for placing any given technique in context. Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste, "Sparsity in Deep Learning: Pruning and Growth for Efficient Inference and Training in Neural Networks," Journal of Machine Learning Research 22 (2021).** A long and careful survey of pruning methods, sparsity patterns, and the hardware conditions under which sparsity yields real speedups. Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao, "Knowledge Distillation: A Survey," International Journal of Computer Vision 129 (2021).** A structured overview of response-based, feature-based, and relation-based distillation and the training schemes that combine them. Vijay Janapa Reddi et al., Machine Learning Systems (open online textbook, mlsysbook.ai).** A freely available textbook from Harvard that treats machine learning as a systems discipline, with substantial material on efficient models, optimization, hardware acceleration, and on-device deployment. Hashtags: #EdgeAI #OnDeviceAI #EmbeddedAI #MobileAI #InternetOfThings #IoT #TinyML #NeuralNetworkOptimization #ModelCompression #DeepLearning #EfficientNeuralNetworks #ModelQuantization #PostTrainingQuantization #QuantizationAwareTraining #INT8Inference #INT4Quantization #NeuralNetworkPruning #StructuredPruning #KnowledgeDistillation #HardwareAwareOptimization #NeuralProcessingUnits #MobileNet #MCUNet #TensorFlowLite #LiteRT #ExecuTorch #MicrocontrollerAI #LowPowerAI #EdgeInference #AIModelDeployment

Download the Book (PDF): Introduction Pick up a small quadcopter, power it on, and take your hands off the sticks without arming the flight controller's stabilization, and it will not fly. It will not even fail gracefully. Within a fraction of a second of leaving the ground, one side will produce slightly more thrust than the other, the airframe will tilt, the tilted thrust will push it sideways, and nothing in the physics of the machine will act to bring it back. A paper aeroplane has more natural stability than a quadcopter. The machine that inspects wind turbines, maps quarries, films weddings and carries blood samples between hospitals is, in its raw mechanical form, an object that wants to fall over. Everything that makes a multirotor useful is therefore software. The four motors and four propellers are only actuators; the aircraft is the control system. A few hundred times a second, a small processor reads a gyroscope, decides how wrong the airframe's rotation is, and changes the speed of each motor by a few percent to correct it. Above that loop sits another one that cares about angle rather than rotation rate, above that one that cares about velocity, above that one that cares about position, and above that the mission logic that decides where the aircraft ought to be at all. Each layer trusts the one beneath it to do its job, and each layer is only as good as that trust is justified. That layered structure is the controlling idea of this booklet. A quadcopter flight controller is a stack of nested feedback loops, each running faster than the one above it, each hiding the messy dynamics of the layer below behind a simpler promise. The inner rate loop promises: "tell me how fast to rotate and I will rotate that fast." The attitude loop promises: "tell me what angle to hold and I will hold it." The position controller promises: "tell me where to be and I will get there smoothly." Waypoint navigation, terrain following, obstacle avoidance and the rest of autonomy are all built by issuing requests to these promises. When a drone oscillates, drifts, overshoots a waypoint, or flies confidently into a tree, the cause can almost always be traced to a specific layer whose promise was broken, and the fix lives in that layer, not the one where the symptom showed up. Understanding the stack this way turns a bewildering list of parameters into a map. ArduPilot, the open-source autopilot this booklet uses as its main example, exposes well over a thousand configurable parameters. Newcomers meet them as a wall of cryptic names: ATC_RAT_RLL_P, INS_HNTCH_FREQ, PSC_VELXY_P, WPNAV_SPEED, EK3_SRC1_POSXY. Read as a stack, they sort themselves. The ATC_RAT family belongs to the innermost loop. The INS family shapes the sensor data before any loop sees it. PSC is the position controller. WPNAV shapes the paths that the position controller is asked to follow. EK3 is the estimator that tells all of them where the aircraft believes it is. Knowing which layer a parameter lives in tells you what it can affect, what it cannot, and which other parameters it depends on. What this booklet covers The book moves up the stack from the bottom. It begins with the physics, because a control system cannot be understood apart from the thing it controls: how four fixed-pitch propellers produce lift, roll, pitch and yaw; why the reaction torque of spinning propellers is the only handle on yaw; and why the system is unstable in the first place. It then turns to sensing. The "gyroscopic stabilization" of a modern drone involves no spinning gyroscope at all, but a vibrating silicon structure a few millimetres across, and the difference matters, because it determines what the sensor gets wrong and how badly vibration corrupts it. From raw sensor data the controller must estimate attitude, and the chapter on sensing explains the filters that do this, from the simple complementary filter to the extended Kalman filters that modern autopilots run. Two chapters are given to the PID controller, because it is the heart of the inner loops and because tuning it is the single most common practical problem pilots and builders face. The first explains what proportional, integral and derivative terms actually do, why the textbook form of PID needs several modifications before it can fly an aircraft, and how ArduPilot arranges its controllers in a cascade. The second is a practical guide to tuning: filtering vibration before touching gains, using logs rather than impressions, recognising the signatures of too much and too little of each term, and understanding what automated tuning can and cannot do for you. The middle of the book climbs to navigation. Knowing where the aircraft is involves satellite navigation, barometers, magnetometers and optical sensors, all fused by an estimator that must constantly decide which of them to believe. Getting from one place to another involves turning a list of waypoints into a smooth trajectory that respects the aircraft's limits on speed, acceleration and jerk. Avoiding obstacles along the way involves sensors that see the world, algorithms that decide what to do about what they see, and a hard-earned understanding of where those algorithms fail. The last chapters deal with the software itself and with safety. ArduPilot is a large, mature codebase with a hardware abstraction layer, a cooperative scheduler, a simulator that lets you fly the real flight code on a laptop, an embedded scripting engine, and a standard protocol, MAVLink, for talking to ground stations and companion computers. Understanding how these fit together is what lets a developer extend the system without breaking the loops that keep it in the air. Finally, a chapter on failsafes and regulation treats safety as an engineering discipline: what the aircraft should do when a sensor lies, a battery sags, or a radio link drops, and what the rules in the United States and Europe expect of the person who sets it flying. Scope and assumptions The focus is the multirotor, and specifically the quadcopter, because it is the dominant civilian platform and the clearest example of a machine that exists only through its control system. Hexacopters and octocopters follow the same principles with more motors to mix. Fixed-wing aircraft and VTOL hybrids appear only in passing; their control problems are different enough to deserve their own treatment. ArduPilot is the main reference platform because it is open, extensively documented, widely used in survey, inspection, agriculture and research, and because its parameter names give concrete handles to abstract ideas. PX4, the other major open-source autopilot, appears where comparison sheds light, and Betaflight, the firmware of choice for acrobatic first-person-view racing, appears where its very different priorities illuminate a tuning question. The principles in this book apply to all of them; the parameter names do not. Where a parameter is named, it is an ArduPilot Copter parameter unless otherwise stated, and readers should check the documentation for their firmware version, since names and defaults change between releases. The applications in view are civilian: hobby flying, aerial photography, mapping and surveying, infrastructure inspection, agricultural spraying, search and rescue support, delivery trials, and academic research. The booklet assumes a reader comfortable with high-school physics and willing to read a few lines of code, but it does not assume a background in control theory. Every concept from that field is introduced when it is first needed, in words rather than equations wherever words will do. One practical warning belongs at the start. Everything described here can be tried on a simulator before it is tried on hardware, and should be. ArduPilot's software-in-the-loop simulator runs the same flight code that runs on the aircraft, and a mistake made in simulation costs nothing. A mistake made with real propellers spinning at tens of thousands of revolutions per minute can cost a finger. The habit of removing propellers before working on a powered aircraft on the bench, and of testing every change in the simulator first, is the most valuable thing any reader can take from this book, and it is worth adopting before reading any further. Chapter 1: Four Propellers and an Unstable Machine A helicopter controls itself by changing the pitch of its rotor blades as they spin. A swashplate tilts the whole rotor disc to steer, collective pitch raises and lowers the total lift, and a tail rotor cancels the torque of the main rotor. It is a mechanically intricate solution, full of linkages, bearings and hinges, and it took decades of engineering to make reliable. The quadcopter replaces all of it with arithmetic. Four fixed-pitch propellers, bolted directly to four electric motors, are the only moving parts. Every manoeuvre the aircraft makes, from a gentle hover to an aggressive flip, is produced by changing the speeds of those four motors relative to one another. This simplicity is why multirotors became cheap and ubiquitous in the 2010s, once three technologies matured at the same time: small brushless motors with fast electronic speed controllers, lithium-polymer batteries with enough energy density to fly for useful periods, and microelectromechanical (MEMS) inertial sensors cheap enough to put in phones. None of these was new, but together they made it possible to build a machine whose mechanical design was trivial and whose control was entirely electronic. The price of that trade is that the quadcopter has no mechanical stability whatsoever. To understand why the flight controller must work so hard, and what it is working against, it helps to look closely at how four spinning propellers produce motion. Thrust, torque and the four controls A propeller is a rotating wing. As it spins, each blade meets the air at an angle and accelerates a column of air downward; by Newton's third law, the air pushes the propeller up. For a fixed-pitch propeller in still air, the thrust it produces is, to a good approximation, proportional to the square of its rotational speed. Double the speed and thrust roughly quadruples. The constant of proportionality depends on the propeller's diameter, pitch, blade shape and the density of the air, which is why the same aircraft needs more throttle to hover on a hot day at a high-altitude site than on a cold morning at sea level. The same blades that push air down also drag on the air as they turn, and that drag produces a torque on the motor. The air resists the propeller's rotation, and by reaction the motor and the frame it is attached to are pushed in the opposite direction. This reaction torque is also roughly proportional to the square of the speed. On a single-rotor helicopter, it would spin the fuselage, which is why the tail rotor exists. On a quadcopter, the solution is to spin two propellers clockwise and two counter-clockwise. In the common X configuration, motors on one diagonal turn one way and motors on the other diagonal turn the other way. When all four run at the same speed, their reaction torques cancel and the frame does not spin. From these two facts, thrust and reaction torque, come the four basic controls. Total thrust, usually called throttle, is changed by speeding up or slowing down all four motors together. If thrust exceeds the aircraft's weight, it climbs; if thrust falls short, it descends. Roll is produced by speeding up the motors on one side and slowing those on the other. More thrust on the left than on the right creates a torque about the aircraft's front-to-back axis, and it rolls to the right. Because one clockwise and one counter-clockwise motor sit on each side, the reaction torques remain balanced and the aircraft does not yaw while it rolls. Pitch works the same way about the side-to-side axis: speed up the rear motors and slow the front ones, and the nose drops. Yaw is the subtle one. To rotate the aircraft about its vertical axis, the controller speeds up the two motors that spin in one direction and slows the two that spin in the other. Total thrust stays the same, so the aircraft does not climb or descend, and the thrust remains symmetric across the frame, so it does not roll or pitch. But the reaction torques no longer cancel, and the net torque turns the frame. A quadcopter yaws by exploiting the very effect its counter-rotating layout was designed to cancel. The yaw handle is weak. Reaction torque is a small fraction of the thrust-induced torques that produce roll and pitch, so a quadcopter yaws far more sluggishly than it rolls. This is why yaw authority is the first thing to run out when an aircraft is overloaded or flying at its limits, and why flight controllers typically limit how much of the motors' range the yaw controller is allowed to use. If roll and pitch need the motors' full range to stay level, yaw must give way. A drone that spins uncontrollably when pushed hard has usually run out of yaw authority, not suffered a software failure. The mixer The flight controller does not think in terms of individual motors. Its attitude and rate controllers produce four abstract demands: a roll torque, a pitch torque, a yaw torque and a total thrust. A component called the mixer translates these into four motor outputs. For an X quadcopter, the arithmetic is simple. Each motor's output is the throttle demand plus or minus the roll demand, plus or minus the pitch demand, plus or minus the yaw demand, with the signs determined by where the motor sits and which way it spins. The front-right motor, for instance, gets less output for a right roll (it is on the side that must drop), less output for a nose-down pitch (the front must drop), and more or less for yaw depending on its spin direction. ArduPilot encodes these factors for each supported frame type in a table inside its motors library, and selecting a frame class and type in the configuration chooses the right table. The same abstraction lets the same controller fly a hexacopter, an octocopter, a Y6 with coaxial motor pairs, or a dozen other layouts. The controller asks for torques; the mixer works out which motors to change. Real mixers must deal with saturation, and this is where the arithmetic stops being trivial. Each motor can only run between a minimum and a maximum. If the throttle is high and the roll controller asks for a large correction, the motors on the high side may hit their ceiling while the others still have room. A naive mixer would clip the saturated motors and produce less roll torque than requested, with a side effect of altering yaw and thrust. ArduPilot's mixer instead prioritises: it preserves roll and pitch first, sacrificing yaw, and if necessary it reduces total thrust to make room for attitude corrections. The logic is that an aircraft which briefly loses some altitude while keeping itself level is recoverable, whereas one that holds altitude while tumbling is not. ArduPilot logs a flag when the motors are saturated in this way, and a log showing frequent saturation is a sign that the aircraft is underpowered for what it is being asked to do. The relationship between motor command and thrust is itself not linear. Electronic speed controllers translate a command into a voltage applied across the motor windings, the motor's speed depends on that voltage, its load and the battery's state of charge, and thrust depends on the square of speed. ArduPilot exposes a parameter called MOT_THST_EXPO that approximates this curve so the mixer can linearise it, along with MOT_SPIN_MIN and MOT_SPIN_MAX for the usable range and battery voltage compensation that scales outputs as the pack sags during a flight. Getting these right matters more than most builders expect. A controller tuned with a linearised thrust curve behaves consistently from hover to full throttle; one that is not will feel crisp at one throttle and sloppy or twitchy at another, and no amount of PID adjustment will fix it everywhere at once. Thrust margin How much spare thrust an aircraft has is the single physical number that most constrains its control. Builders usually express it as a thrust-to-weight ratio: the total thrust all motors can produce at full power, divided by the aircraft's weight. A ratio of two means the aircraft hovers at roughly half its maximum thrust. Because thrust rises with the square of motor speed, half thrust corresponds to roughly seventy percent of maximum speed, and since the relationship between throttle command and speed is not linear either, the hover throttle a log reports will not simply equal the fraction of thrust in use. ArduPilot learns the actual value in flight and stores it as MOT_THST_HOVER, which the altitude controller then uses as its baseline. The margin above hover is what the controller spends on everything else. To tilt and accelerate sideways, the aircraft must increase total thrust so that the vertical component still balances weight; at a 30-degree lean, it needs about 15 percent more thrust than in hover, and at 45 degrees about 41 percent more. To correct a roll, it must raise the motors on one side, and if those motors are already near their limit, the correction is lost. To climb, it needs thrust beyond weight. And to fight a gust, all of these may be needed at once. An aircraft with a thrust-to-weight ratio near two has generous room for all of them. One loaded close to a ratio of 1.5, as heavy survey and spraying aircraft often are at takeoff, hovers at about two-thirds of its maximum thrust and has much less room, so its controller saturates more readily, its maximum safe lean angle is lower, and its tune must be gentler. Racing quadcopters, at the other extreme, may exceed a ratio of eight, hovering near the bottom of their throttle range where small changes in command produce large changes in thrust, which brings its own tuning difficulties. Thrust margin also falls during a flight. As the battery discharges, its voltage drops and the maximum speed each motor can reach falls with it, so the same aircraft that had a ratio of two on a fresh pack may have noticeably less near the end. Voltage compensation keeps the controller's response consistent while there is margin to compensate with, but it cannot create thrust the motors cannot produce. This is one reason a heavy aircraft that handles well at the start of a flight can become sluggish and prone to saturation in wind toward the end of it, and why battery failsafe thresholds, discussed in the last chapter, should be set with control margin in mind as well as flight time. Why it is unstable A conventional aeroplane is designed to be stable. Its centre of gravity sits ahead of the wing's aerodynamic centre, its tailplane produces a restoring moment when the nose pitches up or down, and its wings are often set at a slight dihedral angle so that a sideslip rolls it back towards level. Disturb a well-designed aeroplane and, left to itself, it will drift back toward its trimmed condition. Pilots can fly them hands-off for a while. A hovering quadcopter has none of this. Suppose it hovers perfectly level and a gust tilts it five degrees to the right. Its thrust vector is now tilted too, so part of the thrust pushes it sideways. Nothing in the airframe resists the tilt: there is no tailplane, no dihedral, no pendulum effect worth mentioning, because the thrust acts at roughly the same height as the centre of mass. The aircraft stays tilted, accelerates sideways, and begins to lose altitude because less of its thrust now points up. Any further disturbance adds to the tilt rather than being cancelled. In the language of control theory, the attitude dynamics of a hovering multirotor behave like a double integrator: torque produces angular acceleration, angular acceleration integrates into angular rate, and rate integrates into angle. With no natural restoring force, small errors accumulate. The translational motion is equally unforgiving. Tilt produces horizontal acceleration, acceleration integrates into velocity, and velocity into position. So the chain from a motor speed change to a change in position runs through four integrations: torque to rotation rate, rate to angle, angle (through tilted thrust) to acceleration, and acceleration to velocity and position. Each integration is a delay and an accumulation, and that chain is exactly why the control system is built as a stack of nested loops. Rather than trying to command position directly through four layers of integration, the flight controller closes a fast loop around rotation rate, a slower one around angle, and slower ones still around velocity and position. Each loop sees a plant that is close to a single integrator, which is far easier to control than four in a row. A small amount of aerodynamic damping does exist. When a quadcopter moves through the air, the advancing side of each propeller meets faster air than the retreating side, which produces a phenomenon called blade flapping that tilts the effective thrust vector slightly back against the motion. Propellers also experience a form of drag that rises with forward speed, sometimes called rotor drag or induced drag, which acts a little like friction. These effects matter to high-performance estimators and to research on aggressive flight, but they are far too weak to stabilise the aircraft. For practical purposes, the quadcopter flies only because its controller makes it. Gyroscopic effects and what "stabilization" means The phrase "gyroscopic stabilization" is often used loosely about drones, and it is worth being precise about it. A spinning propeller is a small gyroscope, and spinning gyroscopes resist changes in the direction of their spin axis. When the aircraft pitches, each propeller experiences a gyroscopic torque at right angles to the pitch, a precession effect that tries to roll the aircraft slightly. On a quadcopter with two propellers spinning each way at similar speeds, these gyroscopic torques largely cancel. On aircraft with large, heavy propellers, or during violent manoeuvres where the motor speeds differ greatly, they become noticeable couplings between axes that a good controller must reject. But they do not stabilise the aircraft. A quadcopter's propellers are far too light, and their axes are aligned with the very motion the aircraft must make, for gyroscopic stiffness to do anything useful. The stabilization in a drone comes from a different device that happens to share the name: the rate gyroscope, a sensor that measures how fast the airframe is rotating. It has no spinning parts, and the next chapter explains how it works. The point here is that the stabilization is active, not passive. It consists of measuring rotation and responding to it with motor commands, hundreds or thousands of times a second. Remove the sensor, or slow the loop down, and the aircraft becomes unflyable within moments. How fast must the loop run? The answer depends on how quickly the aircraft can rotate and how quickly disturbances arrive. A small, light racing quadcopter with powerful motors can change its rotation rate in a few milliseconds; a heavy survey hexacopter with large propellers responds more slowly, because large propellers have more rotational inertia and take longer to change speed. The motors themselves are a limit: a brushless motor and propeller cannot change speed instantaneously, and the time constant of that response, typically tens of milliseconds for a small aircraft and longer for a large one, bounds how fast the whole system can react. ArduPilot's main loop runs at 400 hertz by default on Copter, and the fastest inner rate loop can be configured to run faster still on capable hardware. Betaflight, aimed at small agile aircraft, commonly runs its gyro sampling and PID loop in the multi-kilohertz range. Both choices reflect the same principle: the control loop must be several times faster than the fastest dynamics it needs to correct, or it will always be reacting to a situation that has already changed. What the physics asks of the software The physics of a quadcopter thus sets the agenda for everything that follows. The aircraft is unstable, so it needs feedback. The chain from motor to position runs through several integrations, so the feedback should be arranged in nested loops. Yaw authority is weak and motors saturate, so the mixer must prioritise. Thrust is non-linear and varies with battery voltage and air density, so the outputs must be linearised and compensated. Motors and propellers take time to respond, so the loops must be fast and the controller must anticipate rather than merely react. And the aircraft responds to disturbances so quickly that the sensor feeding the inner loop must be fast, clean and trustworthy. That last requirement turns out to be the hardest to meet. A quadcopter is a vibrating machine. Four motors spinning at thousands of revolutions per minute, each turning a propeller that is never perfectly balanced, shake the frame at frequencies that sit uncomfortably close to the bandwidth the control loop needs. The sensor that tells the controller how the aircraft is rotating is also, unavoidably, a vibration sensor. Separating the motion the controller needs to see from the vibration it must ignore is the subject of the next chapter, and it is the foundation on which every later layer rests. Chapter 2: Sensing Rotation and Estimating Attitude The innermost loop of a flight controller needs one number above all others: how fast the aircraft is rotating about each of its three axes, right now. The attitude loop above it needs a second: which way the aircraft is pointing, relative to the ground and to gravity. The first comes almost directly from a sensor. The second has to be estimated, because no single sensor on a small drone measures attitude directly and reliably. This chapter covers both, starting with the device at the centre of it all. The MEMS gyroscope The gyroscope in a modern flight controller is a microelectromechanical system, a structure of silicon etched at the scale of micrometres and packaged in a chip a few millimetres across. Common parts in flight controllers come from manufacturers such as TDK InvenSense, Bosch and STMicroelectronics, and a single package usually holds a three-axis gyroscope and a three-axis accelerometer together, an arrangement called an inertial measurement unit, or IMU. The gyroscope has no spinning rotor. Instead, a tiny proof mass is driven to vibrate back and forth along one axis at a fixed frequency, typically in the tens of kilohertz. When the chip rotates, the Coriolis effect acts on the moving mass and pushes it sideways, perpendicular to both its vibration and the axis of rotation. The sideways displacement is proportional to the rotation rate, and it is detected as a change in capacitance between the mass and fixed electrodes nearby. Electronics on the chip demodulate this signal, filter it, and present the rotation rate as a digital number, typically in degrees per second, over a serial bus to the flight controller's processor. Three properties of this sensor shape everything the flight controller does with it. The first is that it measures rate, not angle. To know the aircraft's attitude from the gyroscope alone, the controller must integrate the rate over time, summing up small rotations to track the total. This works well over short periods but drifts over long ones, because any small constant error in the measured rate, called bias, accumulates into a growing angle error. A bias of a fraction of a degree per second, quite normal for a consumer MEMS part, becomes tens of degrees of error in a minute. Bias also changes with temperature, which is why some flight controllers heat their IMUs to a fixed temperature and why ArduPilot can perform temperature calibration of its inertial sensors. The second is that the sensor is fast and low-noise in the band the controller cares about. Over a few milliseconds, the gyroscope is by far the most trustworthy source of information about rotation on the aircraft. This is why the innermost loop runs directly on gyroscope data with minimal processing. The third, and the one that causes the most trouble, is that the gyroscope responds to vibration. A MEMS gyroscope is itself a vibrating mechanical structure, and it has limited ability to distinguish the rotation it is meant to measure from mechanical shaking of the board it is soldered to. More importantly, the frame genuinely does rotate slightly with each vibration cycle, and the gyroscope faithfully reports that rotation. From the controller's point of view, the airframe is rocking back and forth hundreds of times a second at the motor rotation frequency and its harmonics. That motion is real, but it is not motion the controller can or should correct, and if the controller tries, it will drive the motors back and forth in step with the noise, heating them and amplifying the very vibration it is trying to cancel. There is an extra trap. A digital sensor samples its signal at a fixed rate. Any vibration at a frequency higher than half the sampling rate does not simply disappear; it appears in the samples as a false, lower-frequency signal, an effect called aliasing. A vibration at 1,050 hertz sampled at 1,000 hertz looks exactly like a 50 hertz oscillation, which is squarely in the band the controller acts on. Once aliased, the false signal cannot be removed by any later filter, because it is indistinguishable from real motion. This is why modern IMUs sample internally at several kilohertz and apply anti-aliasing filters before the data leave the chip, why flight controller designers care about the IMU's internal configuration, and why physical vibration isolation of the flight controller and good propeller balance remain important even with sophisticated digital filtering. The rule is that vibration should be prevented mechanically where possible, measured honestly, and filtered digitally only after it has been sampled fast enough to be seen for what it is. The accelerometer and the magnetometer The gyroscope tells the controller how fast it is turning. To know which way is down, the controller needs a reference that does not drift, and on a small aircraft the obvious candidate is gravity. A MEMS accelerometer is a proof mass on a spring, again etched in silicon, whose displacement under acceleration is measured capacitively. Three of them at right angles measure the acceleration of the chip along each axis. At rest, an accelerometer reads one gravity pointing up, because the sensor is being held up against gravity, and the direction of that reading tells the controller which way is down. If the aircraft is tilted, the gravity vector appears tilted in the sensor's frame, and the tilt angle can be computed from simple trigonometry. The catch is that an accelerometer cannot distinguish gravity from any other acceleration. During a hover it reads gravity. During a turn, a climb, a braking manoeuvre or a gust, it reads gravity plus the aircraft's own acceleration, and the computed "down" direction swings away from the true vertical. Worse, a multirotor in steady flight is an object whose thrust always points along its own vertical axis. In a coordinated, unaccelerated tilt the accelerometer mostly sees thrust along the body axis and drag, and the information it carries about tilt is weaker than intuition suggests. Accelerometers are also even more sensitive to vibration than gyroscopes, since vibration is acceleration. They are a reliable reference for "down" only when averaged over seconds, not milliseconds. For heading, the controller needs a reference for which way is north. A three-axis magnetometer measures the direction of the Earth's magnetic field, and once the aircraft's tilt is known, the horizontal component of that field gives magnetic heading. Magnetometers are the most temperamental sensors on a drone. The Earth's field is weak, and the currents flowing to the motors, which can be tens of amps, create magnetic fields of their own that change with throttle. Steel structures, reinforced concrete, power lines and buried pipes distort the local field. For these reasons the compass is usually mounted on a mast with the GPS receiver, away from the power wiring, and ArduPilot provides both a calibration routine that corrects for fixed distortions caused by the airframe itself and a separate compensation for motor current interference. Even so, compass errors remain one of the commonest causes of poor navigation, and later chapters return to them. Fusing the sensors The three sensors have complementary strengths. The gyroscope is excellent over short timescales and drifts over long ones. The accelerometer is noisy and corrupted by manoeuvres over short timescales but gives a stable reference for down over long ones. The magnetometer does the same for heading, with extra vulnerabilities. Combining them so that each covers the others' weaknesses is the problem of attitude estimation. The simplest workable answer is the complementary filter. It integrates the gyroscope to track attitude, and at every step nudges the result a small amount toward the attitude implied by the accelerometer (and magnetometer, for heading). The size of the nudge sets a crossover time. Faster than that, the estimate follows the gyroscope; slower than that, it follows gravity. A crossover of a second or two lets the estimate ride smoothly through short manoeuvres while still slowly correcting gyroscope drift. Mathematically, it is a high-pass filter on the gyroscope's integrated angle added to a low-pass filter on the accelerometer's angle, with the two filters chosen so their responses sum to one at every frequency, which is where the name comes from. The complementary filter is cheap, intuitive and good enough for a surprising range of aircraft. Early hobby flight controllers used versions of it, and refinements remain in wide use. One influential form, the nonlinear complementary filter on the rotation group published by Robert Mahony, Tarek Hamel and Jean-Michel Pflimlin in 2008, adds an integral term that estimates and removes the gyroscope bias rather than merely nudging the angle, and it works directly with rotations rather than Euler angles to avoid their singularities. Another widely implemented approach, described by Sebastian Madgwick in 2010, uses a gradient-descent step to align the measured gravity and magnetic field vectors with their expected directions. ArduPilot's original attitude estimator, known as the DCM algorithm after the direction cosine matrix it uses to represent attitude, belongs to this family, and it still runs in the background today as a fallback. The complementary filter's limitation is that its trust in each sensor is fixed. It cannot recognise that the accelerometer is unreliable during a hard turn and lean on the gyroscope harder, or that a GPS fix has become available and could help separate gravity from the aircraft's own acceleration. For that, the controller needs a filter that maintains not only an estimate but a measure of how uncertain the estimate is. The extended Kalman filter A Kalman filter keeps track of a set of quantities, called the state, together with a matrix describing how uncertain each quantity is and how the uncertainties are correlated. At each step it does two things. It predicts: using a model of how the system evolves, it moves the state forward in time and grows the uncertainty to reflect what the model does not know. Then it corrects: when a measurement arrives, it compares what the sensor reported with what the state predicted it would report, and blends the difference into the state in proportion to how uncertain the prediction was relative to how noisy the sensor is. A confident prediction and a noisy sensor yield a small correction; an uncertain prediction and a precise sensor yield a large one. The weighting is recomputed at every step, so the filter's trust in each sensor changes as conditions change. The classic Kalman filter assumes the system is linear. Attitude and navigation are not, because rotations combine non-linearly and the relationship between position, velocity and sensor readings depends on the attitude. The extended Kalman filter, or EKF, handles this by linearising the model around the current estimate at each step. It is an approximation, and it can misbehave if the estimate strays far from the truth, but for the gentle non-linearities of a drone's motion it works well and has been the workhorse of aircraft navigation for decades. ArduPilot's current estimator, EKF3, is an extended Kalman filter with a state of around two dozen quantities. It tracks attitude, represented as a quaternion to avoid the singularities that Euler angles suffer at ninety degrees of pitch; velocity and position in a local north-east-down frame; the biases of the gyroscopes and accelerometers; the Earth's magnetic field and the aircraft's own magnetic offsets; and wind velocity, which it can estimate when an airspeed sensor or, on multirotors, a configured model of the airframe's drag gives it something to work with. The gyroscope and accelerometer drive the prediction step at high rate. GPS position and velocity, barometric altitude, magnetometer readings, and optionally optical flow, rangefinders, visual odometry and wheel encoders are fused as corrections when they arrive. Because the filter carries the accelerometer bias and the aircraft's velocity in its state, it can separate gravity from manoeuvring acceleration far better than a complementary filter can, provided it has an external velocity reference such as GPS. Two engineering details make EKF3 robust in practice. First, sensors report with delay; a GPS fix may describe where the aircraft was a hundred or more milliseconds ago. Fusing it as if it were current would corrupt the estimate. EKF3 therefore runs on a delayed time horizon, fusing each measurement at the moment it actually describes, and uses a separate fast complementary filter to project the delayed estimate forward to the present for the controllers. Second, every measurement is checked before it is fused. The filter computes how surprising each reading is, given its own uncertainty, and rejects readings whose surprise exceeds a threshold. This innovation gating, as it is called, is what lets the estimator survive a brief GPS glitch or a magnetic disturbance without being dragged off course, and the size of the recent innovations is what ArduPilot reports as the EKF's health. When innovations grow persistently large, the EKF failsafe triggers, a mechanism discussed in the last chapter. ArduPilot can run several instances of the EKF at once, typically one per IMU on boards that carry two or three, and switch between them if one diverges or its sensor fails. This redundancy is one reason serious survey and inspection aircraft fly controllers with multiple IMUs, often of different models and sometimes on separately isolated mountings, so that a vibration or a failure that afflicts one is unlikely to afflict all. What the controllers actually receive It is worth being clear about what each loop sees, because confusion here is a common source of tuning mistakes. The innermost rate loop sees gyroscope data, filtered by a low-pass filter and by notch filters aimed at motor noise, at the full loop rate. It does not use the EKF's output for its feedback, because the EKF is too slow and too smoothed for a loop that must respond in milliseconds. It does use the EKF's estimate of gyroscope bias, which is subtracted from the raw readings. The attitude loop sees the attitude estimate, which comes from the EKF's quaternion propagated to the present. Its job is to compare that estimate with the desired attitude and produce a desired rotation rate for the rate loop. The position and velocity loops see the EKF's position and velocity estimates. They are slow enough that the EKF's latency is acceptable, and they depend completely on its accuracy. A position controller with perfect gains will still wander if the position estimate is wrong. This separation explains why some problems look like tuning problems but are not. An aircraft whose rate loop oscillates has a gain or filtering problem, almost always in the rate controller or in the vibration that feeds it. An aircraft that holds attitude well but drifts slowly in position or "toilet bowls", circling in a growing spiral around the point it is meant to hold, usually has an estimation problem, very often a compass heading error that makes the position controller push in slightly the wrong direction. No change to the rate PID gains will cure toilet-bowling, and no amount of compass calibration will cure a twitchy roll. Knowing which layer owns the symptom is the first step to fixing it, and the next two chapters turn to the layer that most often needs attention: the PID controllers at the bottom of the stack. Chapter 3: The PID Controller and the Cascade The proportional-integral-derivative controller is the most widely used feedback algorithm in engineering. It regulates the temperature of chemical reactors, the speed of conveyor belts, the pressure in gas pipelines and the cruise speed of cars. Its theory was worked out in the early twentieth century, notably by Nicolas Minorsky, who analysed automatic ship steering in a paper published in 1922, and its practical tuning was codified by John Ziegler and Nathaniel Nichols in 1942. It is used so widely not because it is optimal for anything in particular but because it is simple, robust and adequate for an enormous range of problems, and because its three terms correspond to three intuitive ideas about how to correct an error. A quadcopter's inner loops are PID controllers, or close relatives of them. The textbook PID, however, would make a poor flight controller. This chapter explains what each term does, why each needs modification before it can fly, and how ArduPilot arranges its controllers into a cascade so that each one faces a problem it can actually solve. Three ways to correct an error Every feedback controller starts with an error: the difference between what is wanted, called the setpoint or target, and what is measured. For the roll rate controller, the target might be "rotate right at 30 degrees per second" and the measurement "currently rotating right at 20 degrees per second", for an error of 10 degrees per second. The controller's job is to produce an output, here a roll torque demand to the mixer, that reduces the error. The proportional term is the obvious response: push in proportion to how wrong things are. Output equals a gain, P, times the error. A larger error produces a larger correction. Proportional control alone is often surprisingly good, but it has two weaknesses. If the gain is too low, the response is sluggish and the system lags behind its target. If the gain is too high, the correction overshoots: by the time the aircraft has responded, the controller has pushed too hard, the error has reversed, and the controller pushes back too hard the other way. With enough gain, and with the delays inherent in motors, propellers and filters, this becomes a sustained oscillation. There is always a ceiling on P, and the ceiling is set by how much delay the loop contains. The integral term addresses a different problem: persistent error. Suppose the aircraft's centre of gravity is slightly off to one side, so holding a steady roll rate of zero requires the motors on one side to run slightly faster than those on the other. A pure proportional controller can only produce that offset if there is an error to multiply, so the aircraft must always be slightly wrong to generate the correction it needs. The integral term accumulates the error over time and adds a correction proportional to the running total. As long as any error persists, the integral keeps growing, until it supplies exactly the offset needed and the error falls to zero. Integral action is what makes a controller precise in the long run. Its cost is that it responds slowly, and because it remembers past errors, it can keep pushing after the error has reversed, contributing its own overshoot and oscillation if its gain is too high. The derivative term looks at how fast the error is changing and pushes against that change. If the error is shrinking rapidly, the derivative term reduces the output in anticipation of arriving, which damps overshoot. It acts like a shock absorber: it resists motion, not position. Derivative action allows a higher proportional gain to be used without oscillation, which is why it is valuable in flight control. Its weakness is noise. Differentiating a signal amplifies its high-frequency content, so any vibration in the measurement becomes a much larger fluctuation in the derivative term, which then drives the motors. Most of what goes wrong with derivative gain in practice is noise amplification, not control theory. These intuitions can be put compactly. P responds to the present error, I to the accumulated past error, and D to the predicted future error. Table 1 summarises the typical effect of increasing each gain, the trade-off it introduces, and the failure that appears when it is pushed too far. Table 1. Typical effects of raising each PID gain in a multirotor rate loop. Gain Main benefit Main cost Sign of excess P Faster, tighter response Less stability margin Fast oscillation after stick input I Removes steady offsets, holds attitude against disturbances Slower to settle, can wind up Slow wobble, overshoot that drifts back D Damps overshoot, allows more P Amplifies noise Hot motors, high-pitched buzz, jitter FF Crisp response to commands without extra feedback Does nothing for disturbances Overshoot on stick input only What the textbook form gets wrong A PID controller written straight from its definition would cause several problems on an aircraft, and each has a standard fix. The first is derivative kick. If the derivative is taken of the error, then any sudden change in the target, such as a pilot snapping the stick from centre to full deflection, produces an instantaneous spike in the derivative and a violent jolt in the output. One common remedy is to take the derivative of the measurement alone rather than the error, so that the derivative term reacts to how the aircraft is actually moving and ignores sudden changes in what is wanted. Another, which ArduPilot uses, is to keep the derivative on the error but pass both the target and the error through low-pass filters, with separate filter frequencies for target, error and derivative (ATC_RAT_RLL_FLTT, FLTE and FLTD for roll), and to shape the target upstream so it never steps instantaneously. The derivative filter frequency is one of the most important parameters in the whole tuning process, because it sets the balance between derivative usefulness and noise amplification. The second is integrator windup. If the motors are saturated, the controller cannot reduce the error no matter what it demands, but the integral term keeps accumulating. When the saturation finally ends, the integrator holds an enormous stored correction that drives a large overshoot. Practical controllers limit how large the integral can grow, stop integrating while outputs are saturated, and often reset or relax the integrator while the aircraft is on the ground, since it is meaningless to accumulate attitude corrections against the reaction of the landing gear. ArduPilot provides an integrator limit for each axis (ATC_RAT_RLL_IMAX and its equivalents) and suppresses integration when the mixer reports saturation. It also deliberately relaxes the integrators before takeoff, because a drone sitting on uneven ground can otherwise accumulate a correction that tips it over the moment it leaves. The third is that pure feedback is always late. A feedback controller only acts once an error exists, so when the pilot or the navigation code requests a new rotation rate, the controller does nothing until the error appears, and then it must build its output from the error. Feedforward fixes this by adding a term proportional to the target itself, not the error. If the target is "rotate at 100 degrees per second", feedforward immediately produces approximately the output known to give that rate, and the feedback terms only need to correct what feedforward gets wrong. ArduPilot exposes this as ATC_RAT_RLL_FF and its counterparts. Feedforward does nothing for disturbances like gusts, which arrive without a change in target, so it complements feedback rather than replacing it. Betaflight's feedforward works on the rate of change of the pilot's stick, which suits its racing audience's demand for instant response to rapid stick movement. The fourth is that the plant's gain varies. The torque a given output produces depends on battery voltage, air density, and where on its thrust curve each motor is operating. ArduPilot compensates for voltage sag and for the thrust curve in the mixer, as described in Chapter 1, so that the PID gains see a roughly constant plant across the flight. Without this, a tune that was perfect on a fully charged battery would grow sluggish as the pack discharged. Betaflight offers a similar compensation and, in its own way, a mechanism that varies gains with throttle. The general principle is gain scheduling: when the plant changes predictably, change the controller with it. The cascade ArduPilot's attitude control is not one PID loop but a cascade of them, each wrapped around the one below. Understanding the arrangement is essential to tuning it. At the bottom are three rate controllers, one each for roll, pitch and yaw. Each is a full PID with feedforward, operating on gyroscope rate at the main loop rate. Its target is a desired rotation rate in degrees per second; its output is a torque demand to the mixer, normalised to the range of the motors. These are the controllers whose gains appear as ATC_RAT_RLL_P, ATC_RAT_RLL_I, ATC_RAT_RLL_D and so on for pitch (PIT) and yaw (YAW). Above them sit three angle controllers, one per axis. These are much simpler: each is a proportional controller only, with gain ATC_ANG_RLL_P for roll. The attitude error, the difference between the desired and estimated attitude, is multiplied by this gain to produce a desired rotation rate, which becomes the target for the rate controller below. Why no integral or derivative term? Because the rate controller below already handles disturbances and steady offsets, and because the integration from rate to angle means a proportional angle controller wrapped around a well-tuned rate loop already behaves like a well-damped system. Adding I or D at this level would mostly add complexity and new ways to oscillate. The attitude error itself is computed using quaternions, not by subtracting Euler angles, so the controller takes the shortest rotational path from the current attitude to the desired one even when the aircraft is far from level. Above the angle controller, but still within the attitude control library, is input shaping. When a pilot moves a stick or the navigation code requests a new attitude, the request is not passed straight to the angle controller as a step. Instead it is shaped into a smooth trajectory, limited by a maximum angular acceleration (ATC_ACCEL_R_MAX for roll, with equivalents for pitch and yaw) and smoothed by an input time constant (ATC_INPUT_TC). The angle controller then tracks a target that moves in a way the aircraft can physically follow, and the shaped rate of that target is fed forward directly to the rate controller. This arrangement is the core of ArduPilot's crisp but smooth feel: the commanded motion is planned to be achievable, the feedforward path delivers most of it immediately, and the feedback loops only clean up the residue. Above the attitude controller sits the position controller, AC_PosControl, with its own cascade: a proportional position controller produces a desired velocity, a PID velocity controller produces a desired acceleration, and the desired acceleration is converted into a desired lean angle and a thrust demand, which become the targets for the attitude controller. Its parameters are prefixed PSC_: PSC_POSXY_P for horizontal position, PSC_VELXY_P, I and D for horizontal velocity, and PSC_POSZ_P, PSC_VELZ_P and PSC_ACCZ_P, I and D for the vertical axis, which has an extra acceleration loop because vertical acceleration is controlled directly through thrust rather than through tilt. Seen whole, the stack looks like this. A navigation layer decides where the aircraft should be. The position controller turns that into a desired acceleration. The conversion from acceleration to lean angle and thrust turns that into a desired attitude. Input shaping turns the desired attitude into a smooth attitude trajectory with a matching rate feedforward. The angle controller turns the attitude error into a desired rate. The rate controller turns the rate error into a torque demand. The mixer turns torque demands into motor outputs. Each layer runs at a rate appropriate to its dynamics, and each relies on the layers below it. Bandwidth and separation The cascade works only if each inner loop is substantially faster than the loop wrapped around it. This principle, sometimes called time-scale separation, is the reason tuning must proceed from the inside out. An outer loop treats its inner loop as if it were perfect: when the angle controller asks for 50 degrees per second, it assumes it gets 50 degrees per second, immediately. That assumption is only approximately true, and it holds well enough only if the rate loop settles much faster than the angle loop needs. If the rate loop is sluggish, the angle loop's requests are fulfilled late, and the angle loop, seeing its corrections lag, behaves as if it had too much gain. A common rule of thumb in cascaded control is that each inner loop should be several times faster than the outer loop around it. This has direct practical consequences. If the rate loop is badly tuned, raising or lowering the angle P will not fix the aircraft's behaviour; it will only shift the symptoms. If the position loop oscillates, the cause may be a position gain set too high, but it may equally be an attitude loop too slow for the position gain to be safe. And if the estimator's position and velocity are noisy or delayed, the position loop's useful bandwidth is limited no matter how well the attitude loops perform. The position controller can never be faster than its information. It also explains why default gains for outer loops tend to transfer well between aircraft while rate gains do not. The rate loop deals directly with the physics of the particular airframe: its inertia, its motor response time, its propeller size, its vibration. The angle and position loops deal with an idealised plant made uniform by the loops beneath them. A well-tuned rate loop on a small inspection quadcopter and a well-tuned rate loop on a large spraying octocopter present roughly similar behaviour to the angle loop above, even though their rate gains may differ by a large factor. This is the payoff of the nested structure: once the bottom layer is right, the layers above are much more forgiving. Beyond PID Researchers have flown multirotors with a long list of more sophisticated controllers: linear quadratic regulators that compute optimal gains from a model, sliding-mode controllers that are robust to model error, geometric controllers that operate directly on the rotation group and remain well-behaved through extreme attitudes, incremental nonlinear dynamic inversion that uses measured angular acceleration to cancel unmodelled dynamics, and model predictive controllers that optimise a trajectory over a short future horizon at every step. Some of these have reached open-source autopilots in limited form, and they perform impressively in aggressive flight, in heavy wind and on unusual airframes. PID remains dominant in the inner loops of production civilian autopilots for reasons that are worth stating plainly. It needs no detailed model of the airframe, which matters when users build thousands of different aircraft. Its parameters have intuitive meanings that pilots can adjust by observation. Its failure modes are well understood, so a bad tune produces recognisable symptoms rather than mysterious behaviour. And wrapped in the modifications this chapter has described, filtered derivatives, anti-windup, feedforward, input shaping, thrust linearisation and voltage compensation, it performs well enough that the remaining gains from more advanced methods rarely justify their cost in complexity. The modified PID inside a modern autopilot is not the textbook controller; it is a carefully engineered descendant of it. Tuning that controller well is the subject of the next chapter. Chapter 4: Tuning in Practice Ask experienced builders how they tune a new aircraft and most will describe the same order of work, even if they express it differently: make the airframe mechanically sound, measure and filter its vibration, set sensible starting gains from its size, fly it gently and read the logs, then tune the rate loops, automatically or by hand, before touching anything above them. The order matters more than any individual technique. Most bad tunes come not from a wrong gain but from doing a later step before an earlier one, usually by raising and lowering PID gains to fight a problem that is actually vibration, or a thrust curve that was never set. This chapter follows that order. The examples use ArduPilot parameter names and tools, but the sequence applies to any autopilot. Before the first flight The mechanical state of the aircraft sets the ceiling on what any tune can achieve. Propellers should be balanced, undamaged and correctly seated; a chipped propeller can double the vibration level of an otherwise clean airframe. Motor bearings should turn smoothly. Arms should be rigid, because a flexible arm combined with a fast rate loop forms a resonant system that the controller can excite. The flight controller should be mounted so that its IMU is isolated from high-frequency vibration without being so soft that it wobbles relative to the frame at low frequencies. Many modern controllers include internal isolation for their IMUs, and adding a second soft mount on top can do more harm than good. Wires should be secured so they cannot slap against the frame or the controller. The frame must also be configured correctly: frame class and type, motor order and direction, and propeller direction on each motor. ArduPilot's motor test function in Mission Planner and other ground stations spins each motor individually, with propellers removed, to confirm that the motor that spins is the one the mixer expects and that it turns the right way. An incorrect motor order is one of the classic causes of a quadcopter flipping violently on its first attempt to take off, and the motor test exists so that no one has to discover it with propellers fitted. ArduPilot then needs a few physical facts about the aircraft before any loop is tuned. The battery cell count and voltage range set the voltage compensation. The thrust curve exponent, MOT_THST_EXPO, depends mainly on propeller size; smaller propellers generally use a lower value. The minimum spin and arming spin values, MOT_SPIN_MIN and MOT_SPIN_ARM, should be set so that all motors start reliably and never stall in flight. Mission Planner provides an initial parameter setup helper that takes propeller diameter, battery cell count and a few other values and computes reasonable starting points for the thrust curve, the gyro filter, the rate gains, and the angular acceleration limits. These starting values are not a tune, but they are close enough to fly safely on most aircraft, and they put the filters in roughly the right range so that the first flight produces usable data. The first flight should be short, gentle and close to the ground, in a stabilised mode such as Stabilize or AltHold, in a clear area with no one nearby. Its purpose is not to judge how the aircraft feels but to collect logs: a period of steady hover, a few gentle inputs on each axis, and a landing. Those logs drive every decision that follows. Filtering vibration first Vibration that reaches the rate controller is the most common obstacle to a good tune, and the most frequently misdiagnosed. Its signature is that the aircraft cannot tolerate as much P or D as it should, that motors run hot after short flights, and that the logged motor outputs look noisy even in a calm hover. The temptation is to reduce gains until the noise goes away. This works, after a fashion, but it leaves the aircraft sloppy in wind and slow to recover from disturbances. The better approach is to remove the noise from the gyroscope signal so that the gains can be set where the dynamics, rather than the vibration, require. Multirotor vibration is dominated by the motors. Each motor spins at a frequency set by its speed, and each propeller blade passing a given point adds energy at multiples of that frequency. A quadcopter hovering at a motor speed of, say, 6,000 revolutions per minute has a fundamental vibration at 100 hertz, with harmonics at 200, 300 and higher. Larger aircraft with larger propellers spin slower and vibrate at lower frequencies, which brings the noise uncomfortably close to the frequencies the rate controller must respond to. A general low-pass filter strong enough to remove 100 hertz noise from a large aircraft's gyro signal would also add enough delay to compromise the rate loop. The solution is a notch filter: a filter that removes a narrow band of frequencies around a centre while leaving everything else largely intact. Because the motor frequency changes as throttle changes, a fixed notch would be in the right place only at one throttle setting. ArduPilot's harmonic notch filter solves this by moving its centre frequency in flight, tracking the motor frequency and placing additional notches at its harmonics. The filter is enabled with INS_HNTCH_ENABLE, and INS_HNTCH_MODE selects how the frequency is tracked. In throttle-based mode, the filter estimates motor frequency from throttle output, using a reference frequency measured at hover (INS_HNTCH_FREQ) and the hover throttle (INS_HNTCH_REF). In ESC telemetry mode, it reads actual motor speeds reported by electronic speed controllers that support bidirectional DShot or a serial telemetry protocol, and can even run a separate notch for each motor. In FFT mode, the flight controller performs a running spectral analysis of the gyro data in flight and places the notch on whatever peak it finds. ESC telemetry is the most accurate where available, because it follows each motor's real speed rather than an estimate. To set the notch, the builder first flies with the filter disabled and logging enabled for raw gyro data. ArduPilot's batch sampler, controlled by INS_LOG_BAT_MASK and related parameters, records short bursts of high-rate IMU data that can be turned into a frequency spectrum. Tools such as the ArduPilot filter review tool on the project's website, and the FFT views in the UAV Log Viewer and Mission Planner, display the spectrum and show the peaks clearly: usually a dominant peak at the motor frequency and smaller ones at its harmonics, with a broad floor of noise below. The builder sets the notch frequency and bandwidth to cover the dominant peak, enables the filter, flies again, and checks that the peaks have been flattened in the post-filter data. An aircraft that has been properly notched will usually tolerate substantially higher rate gains than it did before, and its motors will run noticeably cooler. After the notch, the static low-pass filter on the gyro (INS_GYRO_FILTER) and the derivative filter in the rate controller (ATC_RAT_RLL_FLTD and its equivalents) can usually be set higher than would otherwise be safe, which reduces the delay in the loop and allows more aggressive control. This is the main reason vibration work comes first: it changes how much performance the loop can achieve before a single PID gain is touched. Reading the logs The single most useful habit in tuning is to make decisions from logs rather than impressions. An aircraft that "feels a bit twitchy" might have too much P, too much D, too little D, a noisy gyro, a loose propeller, or an overly aggressive input shaping, and the feel alone cannot tell them apart. The logs can. ArduPilot records a large amount of data to its onboard storage at each flight. The most important messages for tuning include the rate controller's targets and achieved rates, the PID terms themselves for each axis, the attitude targets and achieved attitudes, the motor outputs, the vibration metrics, and the IMU data. Plotting the desired roll rate against the actual roll rate during a series of stick inputs shows how faithfully the rate loop follows its target. A well-tuned loop tracks the target closely, with a slight lag and a small, well-damped overshoot at most. A loop with too little gain lags visibly and rounds off sharp inputs. A loop with too much gain overshoots and rings, oscillating briefly at a high frequency after each input. A loop with too much I overshoots and drifts back slowly. Vibration metrics are worth checking on every flight. ArduPilot reports vibration levels on each accelerometer axis and counts clipping events, which occur when an acceleration exceeds the sensor's measurement range. Clipping is serious because a clipped accelerometer reports a false average, and the EKF can be misled into a wrong estimate of vertical velocity. The ArduPilot documentation gives guidance on acceptable vibration levels, and a sudden rise in vibration between flights is a reliable sign that something mechanical has changed, such as a damaged propeller or a loose motor. The motor outputs tell their own story. In a calm hover, all four outputs should sit close together. A persistent offset between diagonal pairs indicates a yaw imbalance, often a motor mounted slightly twisted. A persistent offset between sides indicates an off-centre centre of gravity or a weak motor. Outputs that frequently touch their minimum or maximum indicate saturation, which may mean the aircraft is underpowered or that its tune is too aggressive for the available authority. Table 2 gathers the commonest symptoms seen in flight and in the logs, with the layer and the most likely cause for each. It is a starting point for diagnosis, not a substitute for looking at the data. Table 2. Common tuning symptoms and their usual causes. Symptom Where it shows Usual cause First thing to check Fast oscillation after inputs Rate target versus actual Rate P or D too high Reduce rate P and D together Hot motors, buzzing, noisy outputs Motor outputs, gyro spectrum Unfiltered motor noise Harmonic notch setup Slow wobble, overshoot that creeps back Attitude target versus actual Rate I too high or too low Rate I relative to P Sluggish response, rounded inputs Rate target versus actual Rate gains or feedforward too low Rate P, FF, acceleration limits Circling drift around a hold point Position track in Loiter Compass heading error Compass calibration and interference Bouncing in altitude hold Vertical velocity and throttle Vertical gains or hover throttle wrong Learned hover throttle, PSC_ACCZ gains Spinning under hard manoeuvres Yaw target versus actual Yaw authority exhausted Motor saturation, weight, motor alignment Automatic tuning Tuning rate loops by hand is slow and requires experience. ArduPilot provides AutoTune, a flight mode that does most of the work for the rate and angle loops. With the aircraft hovering in a large open area in calm air, the pilot switches to AutoTune, and the aircraft begins performing small, rapid twitches on one axis at a time, typically roll, then pitch, then yaw, as selected by AUTOTUNE_AXES. Each twitch is a step request to the rate or angle controller. AutoTune measures the response, adjusts the D gain upward until it sees a small bounce-back, then reduces it to a safe fraction; it does the same for P; and it then measures the maximum angular acceleration the aircraft can achieve, which it uses to set the acceleration limits and the angle P. The process takes several minutes and a substantial portion of a battery. The pilot can take control at any time by moving the sticks, during which AutoTune pauses and resumes when the sticks are released. When it finishes, the new gains are tested in flight and saved only if the pilot lands and disarms while still in AutoTune mode. AUTOTUNE_AGGR sets how aggressive the resulting tune will be, in effect how much bounce-back AutoTune will accept before backing off. Lower values give a crisper tune with less margin; higher values give a softer one. The default suits most aircraft. AutoTune works best on an aircraft whose vibration has already been filtered, whose thrust curve and hover throttle are correctly set, and which is flown in calm conditions. Wind confuses its measurements, because a gust looks to the algorithm like an unexpected response to its own input. AutoTune has limitations worth knowing. It tunes the rate and angle loops, not the position controller or the navigation layer. It can produce a tune that is too aggressive for very large or very flexible aircraft, where its step inputs excite structural modes. It can struggle with aircraft that have very little yaw authority. And it requires the pilot to keep the aircraft in a safe area while it twitches and drifts, which demands both space and attention. For these reasons, some builders of large aircraft prefer to tune manually or to start from AutoTune's result and refine it by hand. ArduPilot also offers QuickTune, a Lua script that tunes the rate loops by raising each gain until it detects the onset of oscillation and then backing it off by a set margin. It is faster than AutoTune and uses smaller motions, which suits aircraft where AutoTune's twitches are inconvenient. Like any automatic method, it assumes the earlier steps have been done: the frame is sound, the vibration filtered, the thrust curve set. Tuning by hand Manual tuning follows the same logic that AutoTune encodes. Starting from conservative gains, the builder raises the rate D gain on one axis in small steps until a fast oscillation appears after sharp inputs, then reduces it to perhaps half of that value. Then the rate P is raised in the same way, typically kept at or slightly above the value of I. The rate I is then set, in ArduPilot conventionally equal to P for roll and pitch as a starting point, and adjusted if the aircraft does not hold its attitude firmly in wind or drifts back slowly after inputs. Feedforward can be added for a crisper stick response. Only once the rate loops are satisfactory are the angle P values adjusted, and only once the attitude loop is satisfactory are the position controller and navigation speeds considered. Some pilots prefer to tune by injecting test signals rather than by stick inputs. ArduPilot supports a system identification mode that injects a frequency sweep, called a chirp, into a chosen axis and logs the response. From this, the frequency response of the aircraft can be computed offline, showing directly how much gain and phase margin the loop has at each frequency. This is the method control engineers use on full-size aircraft, and it gives a far richer picture of the loop than step responses alone, at the cost of more analysis. It is particularly useful for large, heavy or unusual airframes where the safety margin matters more than the speed of tuning. Whatever the method, a tune is complete only when it has been tested across the aircraft's envelope: with a full and a depleted battery, with and without payload, in calm and windy conditions, at hover and in fast forward flight. A survey aircraft that will carry a heavy camera should be tuned with that camera fitted. An aircraft whose weight changes during flight, such as an agricultural sprayer that empties its tank, needs a tune that is stable at both ends of the range, which may mean accepting less performance at one end. The best tune is the one that is stable everywhere the aircraft will actually fly, not the one that feels sharpest on a single calm test flight. Racing and cinematic aircraft Firmware such as Betaflight, built for small first-person-view aircraft flown in acrobatic and racing styles, approaches tuning with different priorities. Its pilots want the aircraft to follow stick inputs instantly, often at rotation rates exceeding a thousand degrees per second, and they fly in "rate" or "acro" mode, where the sticks command rotation rate directly and the angle loop is not used. Betaflight therefore emphasises the rate loop above all, with sophisticated filtering, RPM-based notch filtering driven by bidirectional DShot motor telemetry, feedforward tied to stick movement, and tuning sliders that adjust groups of gains together. Its blackbox logs are analysed with tools such as the community-developed PID Toolbox and the Betaflight Blackbox Explorer. ArduPilot's defaults favour a different kind of aircraft, one that must hold position, navigate autonomously, and fly stably for long periods with payloads. But the physics is the same, and the lessons transfer in both directions. Betaflight's community demonstrated, for example, how much rate loop performance could be gained by RPM-based notch filtering, an idea that ArduPilot's harmonic notch in ESC telemetry mode embodies. Whichever firmware is used, the order remains: mechanics, then vibration, then rate loops, then everything above them. With the inner loops tuned, the aircraft holds its attitude reliably. The next question is where it is. Chapter 5: Knowing Where You Are A quadcopter with well-tuned attitude loops will hold itself level, but it will not stay in one place. Hovering level in still air, the slightest residual tilt, a few tenths of a degree, produces a sideways acceleration that becomes a drift of metres within seconds. In wind, the drift is immediate. To hold a position, let alone fly to one, the aircraft needs to know where it is, how fast it is moving, and which way it is facing, and it needs to know these things continuously, reliably and with enough accuracy for the task at hand. A hobby aircraft circling a field needs a few metres of accuracy. A survey aircraft producing a map of a construction site needs centimetres. An inspection aircraft flying close to a bridge pier where satellite signals are blocked needs a way to navigate without them at all. This chapter covers the sources of position information available to a civilian drone and how the estimator decides which to trust. It is the layer between the attitude controllers of the previous chapters and the navigation and avoidance algorithms of the next ones, and it is the layer most often responsible for navigation failures that look, at first glance, like something else. Satellite navigation The Global Positioning System and its counterparts, Russia's GLONASS, the European Union's Galileo and China's BeiDou, collectively known as global navigation satellite systems, or GNSS, are the primary source of position for almost every outdoor drone. Each satellite broadcasts a signal containing its precise orbital position and a timestamp derived from an onboard atomic clock. A receiver measures the time of arrival of signals from several satellites and, knowing the speed of light, computes its distance from each, a quantity called a pseudorange because it includes an unknown offset from the receiver's own imprecise clock. With signals from at least four satellites, the receiver can solve for its three position coordinates and its clock offset simultaneously. Modern receivers on drones, such as the u-blox modules that dominate the hobby and professional market, track several constellations at once, which improves both accuracy and availability, since more satellites in view means better geometry and more resilience to signals being blocked. Standard GNSS positioning on a drone typically achieves horizontal accuracy of the order of a couple of metres under an open sky, and vertical accuracy somewhat worse, because all the satellites are above the receiver and their geometry constrains height less well. Several error sources contribute. The signals are delayed passing through the ionosphere and troposphere by amounts that vary with time and location. Satellite clock and orbit predictions contain small errors. Signals reflected off buildings, terrain or the aircraft itself arrive slightly late and corrupt the measurement, an effect called multipath that is especially troublesome near structures, which is precisely where inspection drones work. And the geometry of the satellites in view affects how much these errors translate into position error, a quantity summarised by the dilution of precision. For many purposes, a couple of metres is enough. For precision work, two enhancements are common. Satellite-based augmentation systems such as WAAS in North America and EGNOS in Europe broadcast corrections for the large-scale atmospheric and satellite errors over a region, and most modern receivers apply them automatically when available. Real-time kinematic positioning, or RTK, goes much further. Instead of using only the coded timing signal, an RTK receiver measures the phase of the carrier wave itself, whose wavelength is around nineteen centimetres for the main GPS frequency. By comparing its carrier phase measurements with those of a base station at a precisely known location nearby, and resolving the whole number of wavelengths between receiver and satellite, an RTK system can achieve centimetre-level relative accuracy. Corrections reach the aircraft over a telemetry radio link from a local base station or over the internet from a network of reference stations, using a standard format called RTCM. ArduPilot can inject these corrections into the aircraft's receiver through its ground station link. Two receivers can also give heading. Mounted a known distance apart on the airframe, a pair of RTK-capable receivers can measure the vector between their antennas with centimetre accuracy, and from it the aircraft's yaw angle. ArduPilot supports this "moving baseline" configuration, and it is increasingly used on aircraft where a magnetometer would be unreliable, for instance those flying close to steel structures or carrying high-current payloads. For survey work, the distinction between relative and absolute accuracy matters. RTK gives centimetre accuracy relative to the base station. If the base station's own position is only known to a metre, the whole survey is shifted by that metre. Professional mapping workflows therefore either place the base over a surveyed point, use a correction network tied to a national reference frame, or tie the final map to ground control points measured independently. An alternative to RTK is post-processed kinematic positioning, in which the raw satellite data are recorded on the aircraft and processed after the flight against base station data, giving similar accuracy without a live link. Many mapping drones use this approach to geotag images, while flying with ordinary GNSS for navigation. Altitude and heading GNSS altitude is less accurate than horizontal position and is vulnerable to the same disturbances. Almost every flight controller therefore also carries a barometer, a pressure sensor whose reading falls as the aircraft climbs. Barometers resolve changes of well under a metre and respond quickly, which makes them good for holding altitude over short periods. Their weakness is that air pressure changes with the weather, so absolute barometric altitude drifts over the course of a flight, and that the rotor downwash and airflow over the airframe create local pressure changes. At high forward speed, or when the aircraft descends into its own downwash near the ground, the barometer can report altitude errors of a metre or more. Good flight controller designs enclose the barometer in foam to shield it from airflow, and ArduPilot can compensate for some speed-related pressure errors. The EKF fuses the barometer and GNSS altitude together, using the barometer's short-term stability to smooth GNSS noise and GNSS to correct the barometer's long-term drift. Above terrain, absolute altitude is not always the useful quantity. An aircraft spraying a crop or inspecting a wind turbine blade needs to know its height above the surface beneath or beside it. Downward-facing rangefinders, using lidar, radar or ultrasound, provide this directly. ArduPilot uses rangefinder data both for terrain following and for precision landing, and the EKF can use it as a height source when flying low over flat ground. Terrain databases provide a second option: ArduPilot can download elevation data for the area around a mission from the ground station and use it to fly at a constant height above the ground, without a rangefinder, though only as accurately as the database itself. Heading comes from the magnetometer, the dual GNSS moving baseline described above, or from GNSS velocity during sustained forward flight, since the direction of travel constrains the heading once the aircraft moves fast enough. ArduPilot's EKF3 can also run without a magnetometer at all, using a separate estimator, sometimes called the Gaussian sum filter, that runs a bank of small filters with different heading hypotheses and selects the one most consistent with GNSS velocity as the aircraft manoeuvres. This allows recovery from a compass failure in flight, and it has become an important safety feature, because compass problems are one of the commonest causes of navigation trouble. Table 3 compares the principal sources of position, height and heading information available to a civilian multirotor. Table 3. Principal navigation sources on a civilian multirotor. Source Measures Typical strength Typical weakness Standard GNSS Position, velocity Global, absolute Metre-level error, multipath near structures RTK GNSS Position, heading (dual) Centimetre relative accuracy Needs corrections link, base accuracy Barometer Altitude change Fast, precise short term Weather drift, downwash and airspeed errors Magnetometer Heading Works at rest, cheap Motor currents, steel, local anomalies Rangefinder Height above surface Direct terrain clearance Limited range, surface-dependent Optical flow Velocity over ground Works without GNSS Needs texture, light and known height Visual odometry Position, velocity, attitude Indoor and GNSS-denied use Computation, lighting, drift over distance Navigating without satellites There are many places where satellite signals are weak or absent: inside buildings, under bridges, inside tunnels and mines, beneath dense forest canopy, and close to tall structures where multipath is severe. Inspection is often done exactly there. Several alternatives allow a drone to estimate its motion without GNSS. Optical flow sensors use a small downward-facing camera to track the apparent motion of the ground texture between frames. The flow, combined with the aircraft's height from a rangefinder and its rotation rate from the gyroscope, gives velocity over the ground. ArduPilot can fuse optical flow into EKF3 to allow position hold indoors or in GNSS-denied areas, and the technique is the basis of the stable indoor hover that small consumer drones achieve. It needs a textured surface, adequate light, and a reliable height measurement; over calm water, a uniform floor or long grass moving in the wind, it struggles. Visual odometry and visual-inertial odometry go further, using one or more cameras together with the IMU to track features in the environment and estimate the aircraft's full position and orientation over time. Stereo cameras and depth sensors provide scale directly; monocular systems recover it by combining camera and IMU motion. These systems typically run on a companion computer, a separate small computer carried alongside the flight controller, and send their estimates to ArduPilot over MAVLink, where EKF3 fuses them as an external navigation source. Odometry drifts over distance, since small errors accumulate, so for long flights it is often combined with a map of the environment, an approach called simultaneous localisation and mapping, or SLAM. Motion capture systems, using arrays of infrared cameras tracking reflective markers on the aircraft, give millimetre-accurate position indoors and are standard in university research laboratories. Ultra-wideband radio beacons placed around a site can provide decimetre-level positioning indoors, and ArduPilot supports several such systems. Each of these feeds the same estimator through the same interfaces, which is the value of a general fusion architecture: the controllers above need not know or care whether the position estimate comes from satellites, cameras or beacons. ArduPilot's EKF3 allows several sets of navigation sources to be defined, using parameters of the form EK3_SRC1_POSXY, EK3_SRC1_VELXY, EK3_SRC1_POSZ and EK3_SRC1_YAW for the first set and equivalents for the second and third, and to switch between them in flight, either by a transmitter switch, by a Lua script, or by a command from a ground station or companion computer. An aircraft can thus fly outdoors on GNSS, switch to visual odometry as it enters a building or passes under a bridge, and switch back as it emerges. Managing these transitions so that the estimate does not jump when the source changes is one of the harder practical problems in GNSS-denied navigation, and it is an area of active development. When the sensors lie Every navigation sensor fails in characteristic ways, and the estimator must handle them without letting a bad reading drive the aircraft off course. GNSS glitches are the most familiar. A sudden jump in reported position, caused by multipath, a brief loss of satellites or receiver error, would, if fused naively, tell the position controller that the aircraft had suddenly moved, and the controller would fly it in the opposite direction to correct. The EKF's innovation gating, described in Chapter 2, rejects measurements that disagree too strongly with the prediction, and if the disagreement persists, the filter gradually reconciles with the new data rather than jumping. ArduPilot's pre-arm checks also refuse to arm in modes that need position until the GNSS fix is of adequate quality, with enough satellites and a low enough reported error. GNSS interference is a growing concern. Intentional jamming and spoofing, along with unintentional interference from badly designed electronics, have become more common in several parts of the world, and civil aviation authorities have issued warnings about their effects on both crewed and uncrewed aircraft. A jammed receiver loses its fix entirely, which an aircraft can detect and respond to by switching to a non-GNSS mode or triggering a failsafe. A spoofed receiver is more dangerous, because it reports a plausible but false position. Defences include cross-checking GNSS against inertial and other sensors, monitoring signal strength and consistency, using multi-constellation and multi-frequency receivers, and fitting antennas designed to reject signals from below the horizon. None of these is perfect, and operators flying beyond line of sight in areas with known interference should plan accordingly. Compass errors are subtler and, in practice, more common. A magnetometer affected by motor currents or nearby steel reports a heading offset by some angle. The position controller, believing the aircraft faces a slightly different direction, applies its corrections in the wrong direction, and the aircraft drifts in a curve rather than holding position. The result is the circling toilet-bowl pattern described earlier, which grows as the controller chases an error it is itself creating. Preventing it involves careful compass placement, calibration in the configuration in which the aircraft will fly, motor interference compensation, and refusal to take off near large steel objects or reinforced concrete. Increasingly, ArduPilot's ability to estimate heading from GNSS velocity alone allows the EKF to detect and recover from a bad compass in flight. Barometric errors show up as altitude bounce in fast flight or near the ground, and vibration-induced accelerometer clipping, mentioned in the previous chapter, can corrupt the vertical velocity estimate badly enough that the aircraft climbs or descends unexpectedly. This is one reason vibration matters beyond the rate loop: it reaches the estimator too. The general lesson is that the navigation layer is not a single sensor but a judgement, made continuously by the estimator, about which of several imperfect sources to trust and by how much. When the judgement is right, the controllers above have an accurate, smooth estimate to work with and the aircraft flies precisely. When it is wrong, the controllers will faithfully fly the aircraft to the wrong place. With a trustworthy estimate in hand, the aircraft is ready for the next layer: deciding where to go. Chapter 6: From Waypoints to Trajectories The attitude loops make the aircraft stable. The estimator tells it where it is. The position controller can now hold it at a point or move it toward one. What remains is to decide where the aircraft should go and how it should get there: the navigation layer. For a pilot flying by hand, this layer is the pilot, translating intentions into stick movements. For an autonomous aircraft, it is software that turns a mission, usually expressed as a list of waypoints, into a continuous trajectory that the position controller can follow without demanding more than the aircraft can deliver. The challenge is not geometric. Drawing a path between points on a map is trivial. The challenge is dynamic: producing a sequence of positions, velocities and accelerations that respects the aircraft's limits on speed, acceleration and jerk, that passes through or near the right places, and that leaves enough margin for the position controller to correct disturbances along the way. A path that the aircraft cannot follow is worse than useless, because the position controller will try to follow it anyway and end up saturating its lean angle, overshooting corners, and arriving late and in the wrong place. Flight modes as layers ArduPilot's flight modes are a direct expression of the control stack, and understanding them as such clarifies what each one does and does not do. In Stabilize mode, the pilot's sticks command attitude directly. The roll and pitch sticks set lean angles, the yaw stick sets yaw rate, and the throttle stick sets motor output. Only the attitude and rate loops are active; there is no altitude hold or position control. Acro mode goes one layer lower, with all three sticks commanding rotation rates, so the aircraft holds whatever attitude it was left at. AltHold adds the vertical position controller: the throttle stick now commands climb rate, and centring it holds altitude. Horizontal motion is still controlled through lean angle. Loiter and PosHold add the horizontal position controller, so the sticks now command horizontal velocity or, when centred, hold position. Loiter is designed for smooth position control with braking when the sticks are released; PosHold behaves more like AltHold under stick input and only locks position when the sticks are centred. Guided mode hands the navigation layer to an external source. A ground station or companion computer sends position, velocity or acceleration targets over MAVLink, and the aircraft's position controller follows them. Guided mode is the basis of most custom autonomy, including the obstacle avoidance and companion computer behaviours discussed later. Auto mode runs a stored mission, a list of commands uploaded from a ground station, and it is the mode most associated with autonomous flight. RTL, short for return to launch, climbs to a set altitude and flies home; SmartRTL retraces the path the aircraft actually flew, which is useful when the direct route home is obstructed. Land descends and lands at the current position. Brake stops the aircraft as quickly as possible and holds. Each mode, then, defines which layers of the stack are under software control and which are under the pilot's. Switching modes in flight swaps the source of commands at a given layer without disturbing the layers below. This is why a pilot can switch from Auto to Loiter at any moment and immediately have a stable, position-holding aircraft: the position controller, attitude loops and estimator are already running, and only the source of targets has changed. The mission An ArduPilot mission is a numbered list of items, each a MAVLink mission command with up to seven parameters. Most are navigation commands that tell the aircraft to go somewhere or do something that involves moving; others are actions that happen immediately or conditions that delay the next item. The list is uploaded from a ground station such as Mission Planner or QGroundControl using the MAVLink mission protocol and stored in the flight controller's memory, where it survives a reboot. A handful of commands do most of the work in civilian missions. Table 4 lists the commonest, with their MAVLink command identifiers, which are fixed by the MAVLink standard and shared by all autopilots that implement it. Table 4. Common MAVLink mission commands used in civilian multirotor missions. Command ID What it does Typical use NAV_TAKEOFF 22 Climb vertically to a set altitude First item of most missions NAV_WAYPOINT 16 Fly to a position, optionally pausing Path points, survey lines NAV_SPLINE_WAYPOINT 82 Fly to a position along a smooth curve Cinematic and inspection paths NAV_LOITER_TIME 19 Hold position for a set time Observation, sensor dwell DO_CHANGE_SPEED 178 Set new horizontal speed Slow sections near structures DO_SET_ROI 201 Point the aircraft or gimbal at a location Orbit and facade inspection DO_SET_CAM_TRIGG_DIST 206 Trigger camera every set distance Photogrammetry surveys NAV_RETURN_TO_LAUNCH 20 Fly home and land Last item of most missions Altitudes in a mission can be relative to the home position, relative to sea level, or relative to terrain, and choosing the wrong frame is a classic source of trouble. A mission planned with altitudes relative to home over a hillside site will fly at a constant height above the takeoff point, which may be dangerously low over rising ground or uselessly high over falling ground. Terrain-relative altitudes avoid this, using either a rangefinder or a terrain database. Shaping the path Given a sequence of waypoints, the navigation library must produce a trajectory. The simplest approach is to fly straight from each waypoint to the next at constant speed, stopping at each one. This works but wastes time and energy, and the abrupt changes of direction are hard on both the aircraft and any camera it carries. Every modern autopilot therefore shapes the path to account for the aircraft's dynamics. ArduPilot's waypoint navigation, in the AC_WPNav library, uses the parameters WPNAV_SPEED for horizontal speed, WPNAV_SPEED_UP and WPNAV_SPEED_DN for vertical speeds, WPNAV_ACCEL for horizontal acceleration, and WPNAV_RADIUS for how close the aircraft must come to a waypoint before it is considered reached. In recent versions, ArduPilot generates trajectories using S-curves: velocity profiles in which not only speed and acceleration but also jerk, the rate of change of acceleration, is limited. A trapezoidal profile, with constant acceleration up to cruise speed and constant deceleration down to rest, has instantaneous changes in acceleration at the corners, which demand instantaneous changes in lean angle. The attitude controller cannot deliver those, so the aircraft lags its trajectory at every transition. An S-curve rounds off the corners of the acceleration profile, so lean angle changes smoothly, and the aircraft can follow the trajectory closely. The jerk limit is linked to how quickly the attitude controller can change lean angle, which is one more instance of the outer loop being shaped to respect the inner loop's capabilities. At intermediate waypoints, ArduPilot does not stop unless told to. As the aircraft approaches a waypoint, the navigation library begins the turn toward the next one early enough to blend the two legs into a smooth curve, limited by the acceleration available. The aircraft therefore cuts the corner slightly, passing near rather than exactly through the waypoint. For many tasks this is ideal. For tasks where the aircraft must pass exactly over a point, such as triggering a camera above a ground control marker, a delay parameter on the waypoint forces a stop. Spline waypoints produce smoother curves still. Rather than straight legs joined by short turns, a spline waypoint defines a curve whose shape is determined by the positions of the waypoints before and after it, so the aircraft flies a continuous arc through the whole sequence. Spline paths are favoured for cinematography, where smooth camera motion matters, and for some inspection tasks, where they allow the aircraft to follow a curved structure without stopping. The research literature has gone further. A widely cited approach, published by Daniel Mellinger and Vijay Kumar in 2011, generates trajectories that minimise snap, the fourth derivative of position, through a sequence of waypoints. Because a quadcopter's motor commands relate to the fourth derivative of its position through the chain of integrations described in Chapter 1, a minimum-snap trajectory is in a precise sense the smoothest the motors can follow. Minimum-snap and related polynomial trajectories are widely used in research and in some commercial systems, usually generated on a companion computer and fed to the autopilot as a stream of setpoints in Guided mode. Surveys, inspections and deliveries Civilian missions fall into a few recognisable patterns, each with its own requirements of the navigation layer. Mapping surveys use a lawnmower pattern: parallel lines flown back and forth over an area, with a camera triggered at regular distances so that consecutive images overlap along each line and adjacent lines overlap across. Photogrammetry software then matches features between the overlapping images to reconstruct a three-dimensional model and an orthomosaic, a map corrected for perspective. Typical overlaps are high, often seventy to eighty percent, because the reconstruction depends on seeing each ground point from several angles. Ground sample distance, the size of one image pixel on the ground, is set by the camera's sensor, its lens and the flight height: flying twice as high doubles the ground sample distance, covering more area with less detail. Mission planning tools such as the survey grid function in Mission Planner compute the line spacing, trigger distance and speed from the camera specifications, the desired overlap and the ground sample distance, and generate the waypoints automatically. The navigation requirement is accurate, straight lines at constant speed and height, which means good GNSS, well-tuned position control, and terrain following over uneven ground. A worked example shows how these quantities interact. Suppose the camera has a sensor 13.2 millimetres wide behind an 8.8 millimetre lens, producing images 5,472 pixels wide and 3,648 pixels tall, a common format for one-inch-sensor survey cameras. Ground sample distance is the sensor width multiplied by the flight height, divided by the product of focal length and image width in pixels. At 100 metres, that gives 13.2 times 100 divided by 8.8 times 5,472, or about 2.7 centimetres per pixel. Each image then covers about 150 metres across the flight line and about 100 metres along it, if the camera is mounted with its long side across track. For 75 percent forward overlap, consecutive photographs must be taken every quarter of the along-track footprint, so every 25 metres. For 70 percent side overlap, adjacent lines must be 30 percent of the across-track footprint apart, about 45 metres. These numbers then constrain the flight. If the camera can reliably capture and store an image every two seconds, the aircraft cannot fly faster than 25 metres every two seconds, or 12.5 metres per second, without missing triggers. Motion blur sets a second limit: at 10 metres per second with a shutter speed of one thousandth of a second, the aircraft moves one centimetre during the exposure, comfortably less than one pixel, but at a slower shutter in poor light the blur would approach the ground sample distance and degrade the reconstruction. Wind sets a third: flying lines into and with a strong wind gives very different ground speeds on alternate passes, and the trigger distance command, which fires on distance travelled rather than time, keeps the overlap constant where a fixed time interval would not. Finally, turning at the end of each line costs time and battery, which is why survey planners orient lines along the long axis of the area and, on aircraft that allow it, fly the turns at a lower acceleration limit than the lines. A planner that gets all of this right produces a mission in which the navigation layer's only job is to fly straight, level and steady, which is exactly what a well-tuned position controller does best. Inspection missions are more varied. A facade inspection might fly vertical columns up and down a building at a constant standoff distance, with the camera pointing at the wall. A tower inspection might orbit at several heights. A bridge inspection might fly beneath the deck, without GNSS, using visual odometry or optical flow. These missions often use region-of-interest commands to keep the camera pointed at the structure, and they demand precise positioning close to obstacles, which makes them both the most demanding and the riskiest navigation tasks most civilian drones perform. They are also where obstacle sensing and avoidance, discussed in the next chapter, matter most. Delivery flights combine long, fairly simple cruise segments with demanding endpoints: a precise descent to a drop point or a landing pad, sometimes using a downward camera and a visual marker for precision landing, sometimes lowering the package on a tether while the aircraft hovers. ArduPilot supports precision landing using infrared beacons or visual markers detected by a companion computer, and it supports winch control for tethered delivery. Delivery operations at scale also involve the regulatory and safety requirements of flying beyond visual line of sight, discussed in the final chapter. Guided mode and external planners Missions stored on the aircraft are fixed once uploaded. Many applications need the path to change in flight in response to what the aircraft sees: following a moving subject, tracking a crop row, revising a route around a newly detected obstacle, or searching an area adaptively. For these, the navigation layer moves off the autopilot to a companion computer or ground station that sends targets in Guided mode. ArduPilot's Guided mode accepts targets at several levels. A position target makes the aircraft fly to a point, using the same S-curve shaping as Auto mode. A velocity target makes it fly at a given velocity until told otherwise, with a timeout that brings it to a stop if updates cease, which is a vital safety feature when the external planner might crash. An acceleration target, or a combined position, velocity and acceleration target, lets the external planner specify a full trajectory that the position controller then tracks. Attitude targets bypass the position controller entirely and command the attitude loops directly, for applications that need to control the aircraft's motion more aggressively than the position controller allows. This flexibility is what makes ArduPilot a platform rather than a closed product. The stack is open at every layer: an external program can replace the mission planner, the trajectory generator or even the position controller, while the autopilot continues to provide stabilisation, estimation and failsafes beneath it. The design choice that makes this possible, a clean separation between layers with well-defined interfaces between them, is the same one that makes the stack tunable in the first place. It is also what makes it possible to add the capability that turns a navigating aircraft into one that can safely share space with the world: the ability to sense and avoid obstacles. Chapter 7: Seeing and Avoiding A drone flying a pre-planned mission in an open field needs no knowledge of its surroundings. The planner has already decided where it can safely go, and the aircraft simply follows. As soon as drones work near structures, trees, people and other aircraft, that assumption fails. The world changes, maps are wrong, and the precise positioning that makes close inspection possible also makes small errors consequential. A survey aircraft whose mission was planned from a map made last year may meet a crane that was not there. An inspection aircraft holding position near a tower in gusty wind may be pushed toward it. A delivery aircraft descending to a garden may find a washing line across its path. Collision avoidance is the layer that handles these surprises. It sits between the navigation layer and the position controller, modifying the targets that navigation requests so that the aircraft does not fly into anything it can detect. Like every other layer, it depends on the ones below it: avoidance is only as good as the sensing that informs it and the position control that executes it, and the commonest failure of avoidance systems is not a bad algorithm but a sensor that did not see the obstacle. Sensing the world Several kinds of sensors give a drone information about its surroundings, each with characteristic strengths and blind spots. Single-beam rangefinders, using a laser, ultrasound or radar, measure the distance to whatever lies in one direction. They are cheap, light and simple, and several pointing in different directions give coarse coverage. Ultrasound works only at short range and is confused by soft surfaces and by the noise of propellers. Laser rangefinders give longer range and better precision but can miss thin objects, transparent surfaces and dark materials. Radar works in dust, fog and darkness, and sees through foliage to some extent, at the cost of lower angular resolution. Scanning lidars sweep a laser beam around the aircraft, producing a ring of distance measurements in a horizontal plane or, in more expensive units, a three-dimensional point cloud. A 360-degree scanning lidar mounted on top of an aircraft gives a detailed picture of obstacles at its own height, which suits horizontal avoidance in inspection and indoor flight. ArduPilot supports several such sensors through its proximity library, configured with parameters prefixed PRX. Cameras provide the richest information, from which depth can be recovered by stereo matching between two cameras, by structured light, by time-of-flight measurement, or by algorithms that infer depth from a single image. Depth cameras produce dense three-dimensional data over a cone in front of the aircraft. Their range is limited, their field of view is narrow compared with a lidar, they need light, and they require a companion computer to process their data. But they see thin wires and branches that a scanning lidar's sparse beams may miss, and their output can feed object recognition as well as geometric avoidance. For other aircraft, the main civilian tool is cooperative surveillance. Many crewed aircraft broadcast their position, altitude and velocity using ADS-B, Automatic Dependent Surveillance-Broadcast, and small ADS-B receivers can be carried on drones. ArduPilot's avoidance library, enabled with AVD_ENABLE, can monitor these broadcasts and take action, such as climbing, descending or moving away, when a crewed aircraft is predicted to come too close. ADS-B only detects aircraft that carry a transmitter, which many light aircraft, gliders and helicopters at low altitude do not. Remote ID broadcasts, discussed in the final chapter, provide similar information about other drones but are not designed as a collision avoidance system. Every sensor has blind spots, and the gaps between them define the limits of any avoidance system. A forward-facing depth camera does not see what is beside or behind the aircraft. A horizontal lidar does not see power lines above or below its scanning plane. No common onboard sensor reliably detects every thin wire at useful range. Operators who rely on avoidance should know exactly what their sensors can and cannot see, and plan missions accordingly. Simple avoidance: stop and slide The most basic and robust form of avoidance is to prevent the aircraft from flying too close to any detected obstacle. ArduPilot's simple avoidance, enabled through AVOID_ENABLE, works in pilot-controlled modes such as Loiter and AltHold, as well as in some autonomous modes. It takes the pilot's velocity request and, for each direction in which an obstacle is detected, limits the velocity component toward the obstacle so that the aircraft can stop at a set margin, AVOID_MARGIN, given its maximum deceleration. If the pilot pushes toward a wall, the aircraft slows and stops at the margin. If the pilot pushes diagonally toward the wall, the component toward the wall is removed and the aircraft slides along it. A related behaviour backs the aircraft away if it finds itself closer than the margin, for instance because it was pushed by a gust. The same mechanism enforces geofences, the virtual boundaries discussed in the final chapter: a fence boundary is treated as an obstacle, and the aircraft stops before crossing it. Simple avoidance is a velocity filter: it does not plan a route around an obstacle, only prevents the aircraft from moving into one. This makes it easy to reason about and hard to defeat. Its weakness is that an autonomous mission that encounters an obstacle across its path will simply stop, which is safe but does not complete the mission. Path planning around obstacles To get past an obstacle rather than merely stop in front of it, the aircraft must plan a new path. ArduPilot provides two object avoidance algorithms for this, selected with OA_TYPE, and the wider research literature offers many more. They fall into two broad families, local reactive methods and global planners, which make different trade-offs. Table 5 compares representative examples of each. Table 5. Representative obstacle avoidance approaches for small multirotors. Approach Family Needs a map Strength Weakness Stop and slide Velocity filter No Simple, predictable Does not route around obstacles BendyRuler Local reactive No Handles unexpected obstacles in flight Can be trapped in concave shapes Vector field histogram Local reactive Short-term grid Smooth steering through clutter Local minima, tuning sensitive Velocity obstacles Local reactive No Handles moving obstacles Requires obstacle velocity estimates Dijkstra or A* on a graph Global planner Yes Finds a route if one exists Map must be accurate and current Sampling planners (RRT family) Global planner Yes Copes with complex 3D spaces Computation, path smoothing needed BendyRuler is ArduPilot's local method. At each update, it probes a set of candidate directions around the line toward the destination, extending a short "ruler" in each direction and checking whether the path along it would stay clear of detected obstacles and fences. It picks the clear direction closest to the destination, with a preference for continuing in the direction it chose last time to avoid dithering, and flies that way. It works with live sensor data and needs no map, which makes it suited to obstacles that were unknown when the mission was planned. Like all local methods, it can be trapped: an aircraft approaching the inside of a U-shaped obstacle will find every direction toward the destination blocked and may oscillate or stop. Its parameters allow the probe distance and the preference for continuing in the same direction to be tuned. The vector field histogram, introduced by Johann Borenstein and Yoram Koren in 1991 for ground robots, builds a short-term grid of obstacle densities around the robot, reduces it to a polar histogram of how blocked each direction is, and steers toward the least obstructed direction near the goal. Velocity obstacle methods, developed in the late 1990s by Paolo Fiorini and Zvi Shiller, and their reciprocal variants, compute the set of velocities that would lead to a collision with each moving obstacle within a time horizon, and choose a velocity outside all of them. These methods handle moving obstacles explicitly, which makes them a natural fit for multi-drone coordination, where each aircraft knows the others' velocities. ArduPilot's second algorithm is a global planner based on Dijkstra's shortest path algorithm. It builds a graph whose nodes are points around the edges of known exclusion zones and fences, connects pairs of nodes that can see each other without crossing a boundary, and finds the shortest route from the aircraft to the destination through the graph. Because it needs the boundaries in advance, it works with polygon fences and exclusion zones defined at mission planning time rather than with obstacles seen in flight. It always finds a route if one exists and never gets trapped, but it can only avoid what it knows about. ArduPilot allows the two to be combined, using Dijkstra's for known boundaries and BendyRuler for unexpected obstacles. More capable global planning, used in research and in some commercial systems, works in three dimensions on a map built from sensor data in flight. A common approach represents the environment as a voxel grid or an octree, a hierarchical structure such as the widely used OctoMap library, in which each cell records the probability that it is occupied. A planner such as A* searches a graph of cells for a collision-free path, or a sampling-based planner such as a rapidly exploring random tree grows a set of random collision-free paths until one reaches the goal. These planners run on a companion computer, typically within the Robot Operating System, and send their paths to the autopilot as a stream of targets in Guided mode. Where avoidance fails Avoidance systems fail in recognisable ways, and the patterns are worth knowing because they recur across products and platforms. The commonest failure is a sensor that does not see. Thin wires, bare branches, glass, water, dark matte surfaces and objects outside the sensor's field of view are all frequently missed. An aircraft with forward-facing sensors that is flown sideways or backwards may have no protection at all in the direction of travel. Some commercial consumer drones disable avoidance in their fastest modes, and some disable it in low light. The first rule of avoidance is that its absence must be assumed until its presence has been confirmed for the specific sensor, direction, speed, lighting and obstacle type in question. The second is stopping distance. Avoidance can only work if the aircraft detects an obstacle far enough away to stop or turn before reaching it. A sensor with ten metres of range on an aircraft flying at fifteen metres per second gives less than a second of warning, which may be insufficient once the sensor's own processing delay and the aircraft's deceleration limit are taken into account. Avoidance systems therefore usually limit speed in proportion to sensor range, and operators who raise speed limits without understanding this remove the margin avoidance depends on. It is worth doing the arithmetic once, because it makes the constraint concrete. An aircraft flying at speed v and able to decelerate at a rate a needs a distance of v squared divided by twice a to stop. At 10 metres per second with a deceleration of 5 metres per second squared, a reasonable figure for a survey quadcopter that limits its lean angle, the braking distance is 10 metres. Before braking can begin, the sensor must take its measurement, the processing chain must recognise an obstacle, and the controller must begin to tilt back; if that chain takes 0.2 seconds, the aircraft covers another 2 metres. Add a 2-metre margin so that the aircraft stops short of the obstacle rather than at it, and the sensor must detect obstacles at least 14 metres away. If the sensor reliably detects the obstacle in question only at 8 metres, perhaps because it is a dark, thin structure, the safe speed falls to a little over 6 metres per second. Halving the deceleration, as happens on a heavily loaded aircraft or one that limits its lean angle to protect a gimbal, doubles the braking distance and lowers the safe speed further. Avoidance specifications quoted as a single range figure hide this dependence; the useful question is always at what range a given sensor sees a given obstacle, and how fast the aircraft can be flying for that range to be enough. The same reasoning applies sideways. An aircraft holding a two-metre standoff from a wall in gusty wind relies on its position controller to reject a gust before it closes the gap. If a gust can push the aircraft a metre before the controller responds, the standoff margin has already lost half its value before avoidance has anything to do. Facade and tower inspection procedures commonly specify standoff distances and maximum wind speeds together for this reason, and an operator who reduces one without reconsidering the other has removed a margin the system was relying on. The third is local traps and oscillation, already described. A reactive algorithm in a cluttered environment may wander, dither between two directions, or stop. This is safe in the sense that the aircraft does not collide, but it may then drain its battery hovering, which turns an avoidance problem into a power problem. Mission design and failsafe configuration must anticipate it. The fourth is the interaction between avoidance and position control. If the position estimate is wrong, a map-based planner will plan around obstacles that are not where it thinks they are. If the aircraft is buffeted by wind close to a structure, the position controller may not be able to hold the margin avoidance requests. Avoidance cannot be better than the estimation and control layers beneath it, and testing it in calm conditions does not prove it will work in gusty ones. The last is moving obstacles, especially other aircraft. Most onboard avoidance assumes a static world. ADS-B gives warning of cooperative crewed aircraft, but non-cooperative traffic, birds and other drones are hard to detect at useful range with the sensors a small drone can carry. Detecting and avoiding non-cooperative aircraft, a capability called detect and avoid, is the central technical requirement for routine beyond-line-of-sight operation, and it is the subject of considerable industry and regulatory work, including a consensus standard for small unmanned aircraft, ASTM F3442. For now, most beyond-line-of-sight operations rely on a combination of airspace restrictions, ground-based surveillance, cooperative signals and operational procedures rather than on the aircraft's own sensors alone. Avoidance as a layer The lesson running through this chapter is the same as in the earlier ones. Avoidance is a layer that intercepts the targets flowing from navigation to position control and modifies them. It makes promises to the layer above: "give me a destination and I will try to reach it without hitting anything I can see." It relies on promises from the layers below: that the estimator knows where the aircraft is and the position controller can put it where it is told. When it fails, the cause can usually be found in a specific broken promise: the sensor did not see, the estimator was wrong, the controller could not hold the margin, or the algorithm was trapped. Diagnosing which one is far more productive than treating avoidance as a single feature that either works or does not. Building, configuring and extending this layer, and every other layer of the stack, ultimately means writing and running software. The next chapter turns to the software itself: how ArduPilot is organised, how to run it without an aircraft, and how to add new behaviour without disturbing the loops that keep the aircraft in the air. Chapter 8: Programming the Flight Controller Everything described so far, from the mixer to the obstacle avoidance planner, is software, and in an open-source autopilot all of it can be read, changed and extended. That openness is what has made ArduPilot and PX4 the default platforms for research, for small manufacturers of survey and inspection aircraft, and for a large community of builders who want to understand their machines rather than merely operate them. It also carries responsibility. The same code that can be modified to add a new sensor or a new mission behaviour can be modified to break the loop that keeps the aircraft in the air. Programming a flight controller well means understanding how the codebase is organised, which parts are safety-critical, and which interfaces exist precisely so that new behaviour can be added without touching them. This chapter introduces ArduPilot's architecture, the simulator that makes development safe, the scripting engine that allows behaviour to be added without recompiling, and the MAVLink protocol through which external programs talk to the aircraft. It ends with a comparison with PX4 and a note on how the two projects differ in philosophy. How ArduPilot is organised ArduPilot began in 2009 as a hobby project built around Arduino-based hardware, growing out of the DIY Drones community. It has since become a mature, professionally maintained open-source project, licensed under the GNU General Public License version 3, with a core development team, regular releases, and use in commercial aircraft as well as hobby and research platforms. The single source repository contains firmware for several vehicle types: Copter for multirotors and traditional helicopters, Plane for fixed-wing and VTOL aircraft, Rover for ground and surface vehicles, Sub for underwater vehicles, and a few others. Each vehicle has its own directory containing its flight modes and top-level logic, and all of them share a large collection of libraries. The libraries are where most of the code discussed in this book lives. AP_InertialSensor reads and filters the IMUs, including the harmonic notch. AP_NavEKF3 is the estimator. AC_AttitudeControl contains the attitude and rate controllers, and AC_PosControl the position controller. AP_Motors contains the mixers for every frame type. AC_WPNav handles waypoint navigation and trajectory shaping. AP_Mission stores and executes missions. AC_Avoidance contains simple avoidance and the object avoidance algorithms. GCS_MAVLink implements the communication protocol. AP_Param provides the parameter system that makes all of these configurable without recompiling. Reading the source of a parameter's library, and finding where it is used, is often the quickest way to understand exactly what it does. Beneath the libraries is the hardware abstraction layer, AP_HAL. The flight code never talks to hardware directly; it calls HAL interfaces for timers, serial ports, SPI and I2C buses, PWM outputs, storage and threads. Each supported platform implements these interfaces. The main implementation for flight hardware runs on top of ChibiOS, a small real-time operating system, on STM32 microcontrollers from the F4, F7 and H7 families that dominate current flight controllers. Others run on Linux boards and, crucially, on a desktop computer for simulation. A board's specific wiring, which pins connect to which sensors and outputs, is described in a text file called a hardware definition, from which the build system generates the configuration. Adding support for a new flight controller board is usually a matter of writing that file rather than writing code, which is why ArduPilot supports hundreds of boards. The firmware is built with the waf build system. Configuring for a board and building the Copter firmware looks like this: ./waf configure --board MatekH743 ./waf copter At run time, the vehicle code is driven by a scheduler. The fastest, most critical work, reading the IMU, running the rate and attitude controllers, and writing to the motors, runs in a fast loop at the main loop rate. Everything else is listed in a scheduler table, each task with a target rate and a maximum expected run time in microseconds: the EKF update, GPS reading, navigation, logging, MAVLink communication, battery monitoring, and dozens of others. At each loop, after the fast work is done, the scheduler runs as many due tasks as fit in the remaining time. This cooperative design is simple and predictable, but it means any task that takes longer than expected steals time from others. A new piece of code that blocks, waiting on a slow sensor or doing a long calculation in one go, can delay the loop and degrade control. ArduPilot logs loop timing and reports when the scheduler is overrunning, and developers adding code to the main thread are expected to keep it short and non-blocking, moving anything slow to a separate thread. The simulator The single most important tool for developing on ArduPilot is SITL, software in the loop. SITL compiles the real flight code, the same libraries and vehicle code that run on the aircraft, for a desktop computer, and connects it to a simulated physical model of the vehicle and its sensors. The flight code sees simulated IMU, GPS, barometer and compass readings, computes motor outputs, and the simulator turns those outputs into simulated motion. From the flight code's point of view, nothing is different from real flight. Starting a simulated Copter with a console and a map is one command, run from the vehicle's directory in a development environment: sim_vehicle.py -v ArduCopter --console --map The simulator speaks MAVLink, so a ground station such as Mission Planner or QGroundControl can connect to it exactly as it would to a real aircraft. A developer can plan and fly missions, change parameters, inject sensor failures, simulate wind, and watch how the controllers and estimator respond. SITL can also connect to external physics simulators such as Gazebo and the commercial RealFlight simulator, which provide more realistic dynamics, three-dimensional environments and simulated cameras and lidars for testing perception and avoidance. The ArduPilot project's automated test suite is built on SITL. Every proposed change to the code is run through hundreds of simulated flights that check that the vehicle still takes off, holds position, flies missions, handles failsafes and lands. This is one of the main reasons ArduPilot is trustworthy despite its size and the number of people who contribute to it, and it is a model for anyone developing their own extensions: test in simulation first, automatically if possible, and only then on hardware. Simulation has limits. The simulated vehicle's dynamics are simplified, simulated vibration is not real vibration, and simulated GPS errors are not real multipath. A tune that is perfect in simulation will not be perfect on a real aircraft. But logic errors, wrong signs, unit mistakes, mode transition bugs and failsafe misconfigurations all show up in simulation, and catching them there is far cheaper than catching them in a field. Scripting with Lua Modifying and recompiling the firmware is powerful but heavy: it requires a development environment, careful testing, and maintenance of a private branch that must be updated as ArduPilot evolves. For many applications, ArduPilot's embedded Lua scripting engine is a better choice. Scripts are small text files placed on the flight controller's SD card, in a scripts directory, and enabled with SCR_ENABLE. They run in a sandbox with their own memory allowance and time limits, so a script that errors or runs away is stopped without crashing the flight code. A script typically defines a function that does some work and then returns itself with a delay, so the scripting engine calls it again after that interval. A minimal script that reports altitude to the ground station every two seconds looks like this: function update() local loc = ahrs:get_location() if loc then gcs:send_text(6, string.format("Alt: %.1f m", loc:alt() * 0.01)) end return update, 2000 end return update() Scripts have access to a wide set of bindings: reading the estimator's position and attitude, reading and setting parameters, reading sensors, sending and receiving MAVLink messages, changing flight modes, setting Guided mode targets, controlling servos and relays, and interacting with serial devices. The ArduPilot repository includes a library of example and applet scripts for tasks such as switching EKF sources between GNSS and non-GNSS navigation, running the QuickTune procedure, adding custom flight modes for particular inspection patterns, controlling winches and parachutes, and driving custom sensors. Scripting is well suited to mission logic, payload control, custom failsafe behaviour and sensor integration. It is not suited to anything that must run in the fast loop, such as a new rate controller, because scripts run at a lower priority and much more slowly than compiled code. The division mirrors the stack: the inner loops stay in compiled, tested core code, and new behaviour at the upper layers can be added in scripts that cannot destabilise them. A worked extension It helps to follow one small extension from idea to field, because the sequence matters more than the code. Suppose an inspection operator wants the aircraft, during an Auto mission, to pause and hold position for ten seconds whenever a payload sensor on a serial port reports a reading above a threshold, logging the event and notifying the ground station, and then to resume the mission. This could be implemented by modifying the mission code in C++, but nothing about it touches the inner loops, so it belongs in a script. The script's update function reads the sensor through the serial binding, checks the reading, and when the threshold is exceeded switches the vehicle to a holding mode such as Loiter, records the time, sends a status message, and writes a custom log entry. On later calls it checks whether ten seconds have passed and, if so, switches back to Auto, which resumes the mission from the current item. Several details need care, and each corresponds to a general lesson. The script must do nothing if the pilot has switched to a manual mode in the meantime, since overriding a pilot's deliberate mode change is dangerous; so it checks the current mode before acting and only intervenes while the vehicle is in Auto. It must handle the sensor going silent, rather than waiting indefinitely, so it treats a missing reading as no event rather than blocking. It must not trigger repeatedly while the reading stays high, so it records that an event has been handled until the reading falls. And it must leave the failsafes untouched: if the battery failsafe triggers RTL during the pause, the script must recognise that the vehicle is no longer in its holding mode for its own reasons and stand aside. The script is then tested in SITL, with the serial sensor replaced by a simulated input, often a second script or a parameter the developer toggles. The tests cover the normal case, the case where the pilot takes control during the pause, the case where a failsafe triggers during the pause, the case where the sensor stops reporting, and the case where the reading stays high. Only when each behaves as intended does the script go onto real hardware, first with the aircraft flown in a large open area with an experienced pilot ready to take over, then on a representative mission. At each stage the logs are checked to confirm that the script's mode changes happened when and only when expected. None of these steps is exotic, and together they take far longer than writing the script itself. That ratio is normal and correct. MAVLink and companion computers MAVLink is the message protocol through which an autopilot communicates with ground stations, companion computers, cameras, gimbals and other devices. It defines a set of messages, each a compact binary structure with a numeric identifier: HEARTBEAT, sent once a second by every component to announce its presence and state; ATTITUDE and GLOBAL_POSITION_INT for telemetry; COMMAND_LONG for commands such as arming, takeoff or mode changes; SET_POSITION_TARGET_LOCAL_NED and its global equivalent for Guided mode targets; and many more. Messages carry the system and component identifiers of their sender and target, so several aircraft and devices can share a link. MAVLink 2, the current version, adds message signing to authenticate commands, which matters for any link where an unauthorised party might try to send commands. The protocol is lightweight enough to run over low-bandwidth telemetry radios, and both ArduPilot and PX4 implement it, along with ground stations, camera manufacturers and many other vendors. Several libraries make MAVLink easy to use from a program. pymavlink, the Python library maintained alongside ArduPilot, gives direct access to every message and is widely used for scripting tests and simple applications. The following lines connect to a simulated aircraft, switch it to Guided mode, arm it and command a takeoff to ten metres: from pymavlink import mavutil m = mavutil.mavlink_connection('udp:127.0.0.1:14550') m.wait_heartbeat() m.set_mode('GUIDED') m.arducopter_arm() m.motors_armed_wait() m.mav.command_long_send(m.target_system, m.target_component, mavutil.mavlink.MAV_CMD_NAV_TAKEOFF, 0, 0, 0, 0, 0, 0, 0, 10) MAVSDK, a higher-level library developed in the Dronecode ecosystem with bindings for C++, Python and other languages, offers task-level functions such as arm, take off, upload a mission and start offboard control, and works with both PX4 and, to a large extent, ArduPilot. DroneKit, an older high-level Python library once widely used with ArduPilot, still appears in many tutorials, but its development has largely stalled, and new projects are generally better served by pymavlink or MAVSDK. A companion computer is a separate, more powerful computer carried on the aircraft, typically a single-board computer such as a Raspberry Pi or an NVIDIA Jetson module, connected to the flight controller by a serial link or Ethernet. It runs the tasks the flight controller cannot: image processing, depth perception, mapping, path planning, machine learning and high-bandwidth communication. It talks to the autopilot over MAVLink, reading telemetry and sending Guided mode targets, external navigation estimates, obstacle distances and commands. The flight controller remains responsible for stabilisation, estimation and failsafes, so if the companion computer crashes or its software misbehaves, the aircraft can still stabilise itself and execute a failsafe. Many companion computer applications are built with the Robot Operating System, ROS, a middleware framework widely used in robotics research. For years, the standard bridge between ROS and MAVLink autopilots was MAVROS. More recently, ArduPilot has added native support for ROS 2 through the DDS communication standard, allowing the autopilot to publish and subscribe to ROS 2 topics directly through a lightweight agent on the companion computer. This makes it easier to integrate ArduPilot into the large ecosystem of ROS 2 perception and planning software. PX4 and the choice of platform PX4 is the other major open-source autopilot. It grew out of a research project at ETH Zurich around 2009 led by Lorenz Meier, and it is now hosted under the Dronecode Foundation, part of the Linux Foundation. Its architecture differs from ArduPilot's in several ways. PX4 is structured as a set of independent modules, each a separate task, that communicate through an internal publish-subscribe message bus called uORB. This makes it natural to replace or add modules and to run modules in separate threads. Its estimator, EKF2, is a descendant of the same lineage of work as ArduPilot's EKF3 and shares many ideas. Its default operating system on flight hardware is NuttX, a small POSIX-like real-time operating system. And it is licensed under the permissive BSD three-clause licence, which allows companies to modify it and ship the result without publishing their changes, whereas ArduPilot's GPL requires that modified firmware distributed to others be made available in source form. The licence difference shapes the ecosystems. PX4 is popular with companies building proprietary products on an open base, and with some research groups for its modular architecture. ArduPilot is popular with builders, survey and agricultural operators, and companies that value its breadth of supported hardware and vehicles, its mature failsafes and its community. Both are capable of professional work, both implement MAVLink, both have SITL simulators and integrate with ROS 2, and the control concepts in this book apply to both. The parameter names, mode names and some behaviours differ, and anyone moving between them should read the documentation for the platform at hand rather than assume equivalence. Whichever platform is chosen, the discipline is the same. Change one thing at a time. Test in simulation first. Keep slow or uncertain code out of the fast loop. Keep new behaviour at the highest layer that can do the job. And never forget that the code under development is the only thing keeping a fast-spinning, unstable machine in the air, which is why the last layer of the stack, the one that decides what to do when everything else goes wrong, deserves a chapter of its own. Chapter 9: Failsafes, Rules and the Engineering of Safety Every layer of the control stack makes promises to the layer above it, and every promise can be broken. A propeller cracks, and the rate loop can no longer deliver the torque the attitude loop expects. A battery sags, and the motors can no longer deliver the thrust the position controller assumes. The GNSS receiver loses its fix, and the estimator can no longer tell the position controller where the aircraft is. The radio link drops, and the pilot can no longer tell the aircraft what to do. A well-engineered aircraft does not assume these things will not happen. It decides in advance what it will do when they do. That decision is the failsafe system, and it is the top of the control stack in a different sense from navigation. Navigation decides where to go when everything is working. Failsafes decide what to do when something is not. This chapter describes the failsafes in ArduPilot, the checks that prevent an aircraft from taking off in an unsafe state, the geofences that keep it within bounds, and the regulatory framework that governs civilian drone flight in the United States and Europe. It treats safety as engineering, which means thinking about failure modes, their likelihood, their consequences, and the actions that reduce both. Checks before flight The cheapest failure to handle is one that is caught before takeoff. ArduPilot's pre-arm checks run continuously while the aircraft is disarmed and refuse to allow arming if any of a long list of conditions is not met. The checks include: the IMUs are calibrated, consistent with each other and not reporting excessive bias; the compass is calibrated, its readings are consistent with the expected field strength and with each other if there are several; the GNSS has an adequate fix in modes that require one; the EKF is healthy, its innovations small and its estimates consistent; the battery voltage is above its failsafe threshold; the radio is calibrated; the board voltage is within limits; the logging system is working; required parameters are set to sane values; and the geofence, if enabled, is loaded and the aircraft is inside it. A failed check is reported to the ground station with a message naming the cause. The checks can be disabled individually through ARMING_CHECK, and the temptation to do so when a check is blocking a flight is strong. It should almost always be resisted. A check that fails is almost always reporting a real problem, and the problems pre-arm checks catch, such as a compass that is badly affected by a nearby steel object or an EKF that has not converged, are exactly the problems that cause crashes a few seconds after takeoff. Experienced operators treat a pre-arm failure as information to be understood, not an obstacle to be removed. Beyond automated checks, a physical preflight inspection matters as much: propellers for chips and cracks, motors for free rotation, frame for loose fasteners, battery for swelling and correct charge, payload mountings, and antennas. Professional operations use written checklists for this, partly because they are required to and partly because checklists work. Failsafes in flight ArduPilot provides a set of failsafes, each triggered by a particular kind of failure and each configurable to take one of several actions. The actions form a hierarchy of severity, from continuing the mission, through holding position or returning home, to landing where the aircraft is, to cutting the motors in the extreme case of an aircraft that has left its permitted area and must be stopped regardless of what happens on the ground. The radio failsafe triggers when the aircraft stops receiving valid signals from the pilot's transmitter, configured with FS_THR_ENABLE. The ground station failsafe, FS_GCS_ENABLE, does the same for loss of the telemetry link to the ground station, which matters for aircraft flown entirely from a ground station or companion computer. The default response to either is usually RTL, return to launch: the aircraft climbs to RTL_ALT if it is below that height, flies straight home, descends and lands. A variant, SmartRTL, retraces the path the aircraft flew to reach its current position, which is safer where the direct path home is obstructed. For an aircraft already executing an autonomous mission, the operator may choose to let the mission continue rather than abort, if the mission is designed to finish safely without pilot input. The battery failsafe monitors voltage and, if a current sensor is fitted, consumed capacity. Two thresholds are configurable: a low threshold, BATT_LOW_VOLT and its capacity equivalent, with an action set by BATT_FS_LOW_ACT, typically RTL; and a critical threshold with its own action, typically Land. The thresholds should be set so that the low action leaves enough energy to get home from the farthest point the aircraft will fly, in the strongest headwind it will face, with a margin. Lithium batteries sag under load and recover at rest, so voltage thresholds should be set with the load in mind; capacity-based thresholds, using a current sensor, are more reliable. A worked example shows how the low threshold should be derived rather than guessed. Imagine a survey hexacopter whose current sensor shows it draws an average of 30 amps in cruise, carrying a 16,000 milliamp-hour pack, flying a mission whose farthest point is 1,500 metres from home, in a forecast wind of 6 metres per second. If it returns at an airspeed of 12 metres per second directly into that wind, its ground speed is 6 metres per second and the return leg takes 250 seconds. Add 60 seconds for the climb to return altitude, the descent and the landing, and the return needs a little over 300 seconds of flight, which at 30 amps is about 2,600 milliamp-hours. Lithium-polymer packs should not be routinely discharged close to empty, both for their health and because voltage collapses quickly near the end, so a reserve of perhaps 20 to 25 percent of capacity, here around 3,500 milliamp-hours, should remain on landing. The low capacity threshold therefore needs to be set at roughly 6,000 milliamp-hours remaining, not at the round number of 20 or 30 percent that a default might suggest, and the critical threshold somewhat below it. If the wind is stronger than forecast, or the aircraft is heavier, the numbers shift, which is why operators of long-range aircraft increasingly use failsafe logic that estimates the energy needed to return from the aircraft's current position rather than relying on a fixed threshold. The EKF failsafe triggers when the estimator's health degrades, when its innovations, the disagreement between what it predicted and what sensors report, exceed a threshold set by FS_EKF_THRESH for a sustained period. This usually indicates a compass or GNSS problem. The default action is to switch to a mode that does not require position, such as AltHold, or to land, because an aircraft whose estimator no longer knows where it is cannot safely fly home. An aircraft that triggers the EKF failsafe will drift with the wind unless the pilot takes manual control, which is a strong argument for every operator being capable of flying in a non-position mode. Other failsafes cover vibration high enough to corrupt the estimator, in which case ArduPilot switches to a more vibration-resistant method of estimating vertical velocity; loss of a companion computer's heartbeat in Guided mode; the detection of a crash, when the aircraft finds itself at an extreme attitude and not moving and disarms to protect the motors and people nearby; and the loss of a motor on aircraft with more than four, where ArduPilot can detect the failure and redistribute thrust across the remaining motors. Hexacopters and octocopters are often chosen for professional work precisely because they can survive the loss of a motor, which a quadcopter cannot. Parachute recovery systems, triggered automatically by the autopilot or manually by the pilot, are increasingly used on aircraft that fly over people or carry heavy payloads, and some regulatory frameworks treat them as a key mitigation for operations over populated areas. ArduPilot supports parachute release, with checks that the aircraft is high enough for the parachute to deploy and that the motors are stopped before release. Geofences A geofence is a virtual boundary the aircraft will not cross. ArduPilot supports several types, enabled with FENCE_ENABLE and selected with FENCE_TYPE: a maximum altitude, FENCE_ALT_MAX; a cylinder of radius FENCE_RADIUS around the home position; and arbitrary inclusion and exclusion polygons and circles uploaded from a ground station. If the aircraft breaches a fence, it takes a configured action, typically RTL or Land. In pilot-controlled modes with simple avoidance enabled, the aircraft stops at the fence boundary rather than crossing it, as described in Chapter 7. Fences enforce operational limits in a way that does not depend on the pilot's attention. An operator flying a survey near an airport's controlled airspace can set a fence along the boundary; an operator flying near a road can exclude it; an operator flying a demonstration can keep the aircraft away from the audience. Fences are also a natural way to encode the operational volume and buffers that regulators increasingly expect for complex operations. A fence does not make an aircraft safe on its own, since an aircraft with a failed estimator may not know where the fence is, but as one layer among several it catches a large class of errors. The regulatory frame Civilian drone flight is regulated in almost every country, and an engineer building or programming drones needs to understand the rules that apply, because they shape what the aircraft must be able to do. In the United States, the Federal Aviation Administration regulates small unmanned aircraft under 25 kilograms through Part 107 of Title 14 of the Code of Federal Regulations for commercial operations, with a separate framework for recreational flyers, who must pass a short knowledge test and follow community safety guidelines. Part 107 requires a remote pilot certificate obtained by passing a knowledge test, limits flights to below 400 feet above ground level in most circumstances, requires the aircraft to remain within the pilot's visual line of sight, and restricts flight over people and at night unless specific conditions are met. Rules for flight over people and at night were expanded in 2021, with categories of aircraft defined by the injury they could cause. Operations outside these rules, such as beyond visual line of sight, have required waivers granted case by case. In August 2025, the FAA published a proposed rule, Part 108, that would establish a performance-based framework for routine beyond-visual-line-of-sight operations, with requirements on the aircraft, the operator and the management of air traffic risk, to replace much of the waiver process for activities such as delivery, agriculture and infrastructure inspection. The FAA's Remote ID rule, Part 89, requires most drones to broadcast their identity, position, altitude, velocity and the location of their control station during flight, either through built-in broadcast or through an add-on module. Compliance was required from September 2023, with the FAA giving operators until March 2024 before enforcement. ArduPilot supports Remote ID through an open-source module design and through commercial modules, and Remote ID compliance has become a standard consideration in the design of any aircraft sold or flown in the United States. In the European Union, drone operations are governed by common rules adopted in 2019, which have applied across member states since the end of 2020. They divide operations into three categories by risk. The open category covers low-risk operations within visual line of sight, below 120 metres, with aircraft classified by class marks C0 to C4 according to their mass and safety features, and with subcategories that define how close to people the aircraft may fly. The specific category covers higher-risk operations, including most beyond-line-of-sight flights, and requires an operational authorisation based on a risk assessment, usually carried out using the Specific Operations Risk Assessment methodology, known as SORA, or by following a published standard scenario. The certified category covers the highest-risk operations, such as carrying people or flying large aircraft over crowds, and requires certification of the aircraft, operator and pilot on terms closer to those of crewed aviation. Network and direct Remote ID requirements also apply in Europe for many aircraft. Other countries have their own regimes, broadly similar in principle: registration, pilot competence, altitude limits, line of sight requirements in the basic category, and a risk-based path to more complex operations. Anyone flying professionally needs to know the specific rules where they fly, which change frequently and are best checked directly with the national aviation authority. Safety as a system The engineering lesson of this chapter, and in a sense of this book, is that safety is not a feature but a property of the whole system. A drone does not become safe because it has a return-to-launch button, an obstacle sensor or a parachute. It becomes safe through a chain: a sound airframe, clean sensor data, well-tuned control loops, a trustworthy estimator, sensible navigation, honest avoidance, careful failsafe configuration, a competent operator and a mission planned within the aircraft's capabilities and the rules that apply. Each link depends on the ones before it. A failsafe that triggers RTL is useless if the estimator does not know where home is. A battery failsafe set to a voltage that leaves too little energy to get home will trigger and still lose the aircraft. An avoidance system that cannot see wires gives false confidence near power lines. The risk-based approach that regulators have adopted, epitomised by SORA, asks operators to think in exactly this way: identify the ways an operation can go wrong, estimate the consequences, and apply mitigations until the residual risk is acceptable. For the engineer, that means configuring each failsafe deliberately, testing it in simulation and then on the aircraft, understanding what the aircraft will do in each failure and whether that is what the operator wants, and documenting it. The flight control stack provides the tools. Using them well is a matter of discipline, and of understanding, at every layer, what promise is being made and what happens when it is broken. Conclusion A quadcopter is a machine that cannot fly without software, and the software that flies it has a shape. At the bottom, a rate loop running hundreds of times a second turns gyroscope readings into motor commands. Above it, an angle loop turns attitude errors into rate requests. Above that, a position controller turns position and velocity errors into attitude requests, a navigation layer turns missions into smooth trajectories, an avoidance layer bends those trajectories around what the sensors can see, and a failsafe layer decides what to do when any of this breaks. Feeding all of it, an estimator weighs the testimony of a handful of imperfect sensors and decides, continuously, where the aircraft is and which way it is facing. The value of seeing the system this way is practical rather than theoretical. It tells the builder what to do first and what to leave until later. It tells the pilot where to look when the aircraft misbehaves. It tells the developer where new code can safely go. Three consequences in particular follow from the argument of the preceding chapters, and they are worth stating as rules. Work from the bottom up The first rule is that problems should be fixed in the layer that owns them, and that the layers should be made sound in order from the bottom. A drone whose position hold wanders cannot be cured by retuning its rate loop, and a drone whose rate loop oscillates cannot be cured by slowing its waypoint speed. But the lower layers constrain the upper ones absolutely. If vibration corrupts the gyroscope signal, the rate loop cannot be tuned properly; if the rate loop is sluggish, the angle loop will overshoot; if the compass is wrong, the position controller will circle; if the position estimate is noisy, the trajectory generator's smooth path will be followed roughly. The order of work in Chapter 4, mechanics before vibration, vibration before gains, rate loops before angle loops, angle loops before position, is not a convention. It follows from the way each layer depends on the ones beneath it. This rule applies equally to diagnosis. When something goes wrong in flight, the most productive question is not "what parameter should I change?" but "which layer's promise was broken?" The logs can usually answer it. Rate targets and achieved rates show whether the rate loop delivered. Attitude targets and estimates show whether the angle loop delivered. EKF innovations show whether the estimator trusted its sensors. Position targets and estimates show whether the position controller delivered. Following the chain from the symptom down to the first broken promise finds the cause far faster than adjusting parameters by intuition. Trust the estimate only as far as its inputs The second rule is that every autonomous behaviour is, in the end, a bet on the estimator. Waypoint navigation, terrain following, geofencing, return to launch, precision landing and map-based avoidance all assume that the aircraft knows where it is. When it does, these features are remarkably capable. When it does not, they become dangerous in proportion to their capability, because they will confidently fly the aircraft to the wrong place. A failsafe that returns home is only safe if home is where the estimator thinks it is. An avoidance system that plans around mapped obstacles is only safe if the map and the aircraft agree about where the aircraft is within it. The practical upshot is that the navigation sensors and their configuration deserve as much attention as the control gains, and usually get less. Compass placement and calibration, GNSS antenna placement and quality, barometer shielding, vibration isolation for the accelerometers, and careful configuration of alternative sources for GNSS-denied flight are where many navigation failures begin. Operators who fly in difficult places, near steel structures, under bridges, in areas of known GNSS interference, should treat the estimator's health as the first thing to monitor in flight and should know how to fly the aircraft by hand in a mode that does not depend on it. Keep new behaviour high in the stack The third rule concerns change. Open-source autopilots invite modification, and the layered architecture makes modification safe if it respects the layers. New mission behaviour belongs in mission commands, scripts or a companion computer sending Guided mode targets, not in the attitude controller. New sensors belong behind the estimator's interfaces, not wired directly into the position controller. New avoidance logic belongs in the layer between navigation and position control, where it can modify targets without bypassing the controllers that execute them. The lower the layer, the higher the bar for changing it, and the inner loops should be changed only by people who understand them, after extensive simulation, and with a clear plan for what happens if the change misbehaves in flight. This rule is also why the simulator matters so much. Running the real flight code against a simulated aircraft costs nothing and catches the logic errors, sign errors and mode transition bugs that cause most crashes after software changes. Every change should fly in simulation before it flies for real, and every failsafe should be triggered deliberately in simulation so that its behaviour is known before it is needed. What remains unsettled Some parts of the stack are mature. Rate and attitude control of multirotors is, for practical civilian purposes, a solved problem; well-configured autopilots fly stably and precisely on an enormous variety of airframes, and automatic tuning works well for most of them. GNSS-based navigation in open sky is reliable and, with RTK, precise enough for survey-grade work. Other parts are still moving. Navigation without satellites is improving quickly but remains fragile, especially at transitions between sources and over long distances. Onboard obstacle sensing still struggles with thin wires, transparent surfaces and non-cooperative aircraft, and this limits what can be done beyond visual line of sight without heavy reliance on airspace management and procedure. The regulatory frameworks for routine beyond-line-of-sight operations in the United States and Europe are taking shape around risk-based principles, but the details, and how they will interact with the technical capabilities of detect-and-avoid systems, are still being worked out. Machine learning is entering perception and, in research, control, and how to verify that a learned component behaves safely across the full range of conditions an aircraft meets is an open question. What will not change is the underlying structure. However capable the upper layers become, they will still rest on an estimator that must decide which sensors to believe, and on control loops that must turn requests into motor commands faster than an unstable machine can fall. The reader who understands that structure, who can find the layer that owns a problem, read the logs that show which promise was broken, and add new behaviour where it will not disturb the loops beneath, is equipped not just to tune today's aircraft but to understand whatever autonomy is built on top of them next. Notes Chapter 2 Robert Mahony, Tarek Hamel and Jean-Michel Pflimlin, "Nonlinear Complementary Filters on the Special Orthogonal Group," IEEE Transactions on Automatic Control 53, no. 5 (2008): 1203–1218. Sebastian O. H. Madgwick, "An Efficient Orientation Filter for Inertial and Inertial/Magnetic Sensor Arrays," technical report, University of Bristol, 2010. Chapter 3 Nicolas Minorsky, "Directional Stability of Automatically Steered Bodies," Journal of the American Society of Naval Engineers 34, no. 2 (1922): 280–309. J. G. Ziegler and N. B. Nichols, "Optimum Settings for Automatic Controllers," Transactions of the ASME 64 (1942): 759–768. Chapter 6 Daniel Mellinger and Vijay Kumar, "Minimum Snap Trajectory Generation and Control for Quadrotors," Proceedings of the IEEE International Conference on Robotics and Automation (2011): 2520–2525. Chapter 7 Johann Borenstein and Yoram Koren, "The Vector Field Histogram: Fast Obstacle Avoidance for Mobile Robots," IEEE Transactions on Robotics and Automation 7, no. 3 (1991): 278–288. Paolo Fiorini and Zvi Shiller, "Motion Planning in Dynamic Environments Using Velocity Obstacles," International Journal of Robotics Research 17, no. 7 (1998): 760–772. Armin Hornung, Kai M. Wurm, Maren Bennewitz, Cyrill Stachniss and Wolfram Burgard, "OctoMap: An Efficient Probabilistic 3D Mapping Framework Based on Octrees," Autonomous Robots 34, no. 3 (2013): 189–206. ASTM International, ASTM F3442/F3442M, Standard Specification for Detect and Avoid System Performance Requirements. Chapter 8 Lorenz Meier, Dominik Honegger and Marc Pollefeys, "PX4: A Node-Based Multithreaded Open Source Robotics Framework for Deeply Embedded Platforms," Proceedings of the IEEE International Conference on Robotics and Automation (2015): 6235–6240. Chapter 9 Federal Aviation Administration, 14 CFR Part 107, Small Unmanned Aircraft Systems; 14 CFR Part 89, Remote Identification of Unmanned Aircraft. Federal Aviation Administration, "Normalizing Unmanned Aircraft Systems Beyond Visual Line of Sight Operations," notice of proposed rulemaking, published in the Federal Register, August 2025. Commission Implementing Regulation (EU) 2019/947 on the rules and procedures for the operation of unmanned aircraft; Commission Delegated Regulation (EU) 2019/945 on unmanned aircraft systems and on third-country operators of unmanned aircraft systems. Joint Authorities for Rulemaking on Unmanned Systems (JARUS), Specific Operations Risk Assessment (SORA). Further Reading ArduPilot documentation (ardupilot.org). The Copter wiki is the authoritative reference for every parameter, flight mode and failsafe named in this book, and its tuning process guide, vibration and harmonic notch pages, and EKF source selection pages are the best practical companions to Chapters 4 and 5. The developer section covers building the firmware, SITL, Lua scripting and the codebase's libraries. PX4 User Guide and Developer Guide (docs.px4.io). The equivalent references for PX4, with clear explanations of its controller architecture, uORB messaging and offboard control. MAVLink Developer Guide (mavlink.io). The definitive description of the protocol, its messages, its mission and command microservices, and message signing. Karl Johan Åström and Richard M. Murray, Feedback Systems: An Introduction for Scientists and Engineers** (Princeton University Press, 2008; second edition 2021). A lucid, freely available introduction to feedback control, covering stability, frequency response and PID control with physical intuition rather than heavy formalism. Karl Johan Åström and Tore Hägglund, Advanced PID Control** (ISA, 2006). The standard practical reference on PID controllers, including derivative filtering, anti-windup, feedforward and tuning methods. Randal W. Beard and Timothy W. McLain, Small Unmanned Aircraft: Theory and Practice** (Princeton University Press, 2012). A textbook treatment of modelling, sensing, state estimation, control and path planning for small aircraft, written for engineering students. It focuses on fixed-wing aircraft, but its chapters on sensors, estimation and guidance apply directly to multirotors. Robert Mahony, Vijay Kumar and Peter Corke, "Multirotor Aerial Vehicles: Modeling, Estimation, and Control of Quadrotor," IEEE Robotics and Automation Magazine 19, no. 3 (2012). A concise, authoritative tutorial on quadrotor dynamics, including blade flapping and rotor drag, and on attitude estimation and control. Paul D. Groves, Principles of GNSS, Inertial, and Multisensor Integrated Navigation Systems** (Artech House, second edition 2013). The comprehensive reference on satellite navigation, inertial sensors and the Kalman filter integration that underlies modern autopilot estimators. Sebastian Thrun, Wolfram Burgard and Dieter Fox, Probabilistic Robotics** (MIT Press, 2005). The classic text on probabilistic state estimation, mapping and localisation, including Kalman filters, occupancy grids and SLAM, essential background for GNSS-denied navigation and map-based avoidance. Steven M. LaValle, Planning Algorithms** (Cambridge University Press, 2006). A thorough treatment of motion planning, including graph search and sampling-based planners such as rapidly exploring random trees, and freely available online from the author. Federal Aviation Administration and European Union Aviation Safety Agency websites (faa.gov/uas and easa.europa.eu). The primary sources for current rules on drone operation, Remote ID, beyond-visual-line-of-sight operations and the specific category, which change frequently enough that the regulators' own guidance should be checked before any professional operation. Hashtags: #DroneFlightControl #FlightControlSystems #PIDControl #PIDTuning #AutonomousNavigation #DroneTechnology #QuadcopterControl #UAVEngineering #DroneAutopilot #ArduPilot #PX4Autopilot #Betaflight #ControlSystemsEngineering #FeedbackControl #CascadedPIDControl #AttitudeControl #FlightStabilization #SensorFusion #ExtendedKalmanFilter #InertialNavigation #GNSSNavigation #RTKPositioning #WaypointNavigation #TrajectoryPlanning #ObstacleAvoidance #DroneSensors #HarmonicNotchFiltering #FlightControllerTuning #MAVLink #DroneSafety