Download the Book (PDF): Introduction At a quarter to six on a Tuesday, the parking lot of a community college looks nothing like the brochure. There are no students lounging on the lawn. There are people walking fast from their cars in work clothes, some still wearing lanyards or scrubs or steel-toed boots, one carrying a laptop bag and a fast-food sack in the same hand. A woman sits in her car with the engine running, finishing a phone call about who is picking up her son from practice. A man in a delivery uniform eats a granola bar in four bites while he walks. By six o'clock, most of them are seated in classrooms, and for the next three hours they will be asked to think about organic chemistry, accounting principles, nursing pharmacology, or the causes of the First World War. These are evening students, and there are far more of them than most people imagine. The Lumina Foundation, which tracks the changing profile of American higher education, reports that about a third of college students are 25 or older, that roughly four in five work while enrolled and about 30 percent of those work full time, and that close to one in five has children or other dependents.[1] The eighteen-year-old living in a residence hall, with no job and no one to care for, is no longer the typical student. For a large share of learners, college happens after something else: after a shift, after daycare pickup, after a commute, after a day that has already asked a great deal. This book is for them, and for the partners, parents, advisers, instructors and employers who want to help them finish. It deals with three practical questions that come up again and again for anyone whose classes start when other people are heading home. When and what should I eat, if the dinner hour falls in the middle of a lecture? How should I use caffeine, which is the obvious tool for staying awake through a long evening but also the obvious threat to the sleep I need? And how do I get home safely, whether that means a forty-minute drive on a dark highway, a wait at a bus stop, or a walk across a half-empty campus to a parking garage? Why the usual advice falls short There is no shortage of general wellness advice. Eat balanced meals, get seven or more hours of sleep, avoid caffeine late in the day, don't drive tired. All of it is correct. Very little of it is written for someone whose day is organized differently from the day the advice assumes. Consider the standard instruction to stop drinking coffee in the early afternoon. For someone who works from eight to four and then sits in a statistics lecture from six to nine, that instruction is not merely inconvenient; it asks them to face the hardest intellectual work of the day with the least help. Or consider the common suggestion to eat dinner early and lightly. A student who leaves work at 4:30, drives straight to campus and sits in class until 9:15 has no early dinner to eat, unless she eats it at her desk at three in the afternoon. And advice about "avoiding drowsy driving" is of little use to someone who has no alternative to driving home at ten at night after sixteen hours awake. The problem is not that general guidance is wrong. It is that the guidance was written around a clock the evening student does not live on. Night classes stretch the day at both ends of its natural shape: they push mental effort into hours when the body is preparing to wind down, they break the usual rhythm of meals, and they place the most dangerous part of the day, the trip home, at the point of greatest fatigue. Good habits for this schedule have to be built for this schedule. The central idea The argument of this book is simple enough to state in a sentence: plan the evening backward from the time you intend to be asleep, not forward from the time class starts. Most evening students plan forward. Class is at six, so I have to leave work by five, so I will grab something to eat on the way, so I will have a coffee to get through it, and then I will go home and deal with whatever is left. Every one of those decisions is made to serve the next hour, and the costs of all of them arrive together at the end of the night, when the student is trying to sleep with caffeine still in the bloodstream, a heavy meal still in the stomach, a mind still racing from the lecture and, often, a household that has been waiting for them. Planning backward reverses the logic. If you need to be asleep by 11:30 to get seven hours before a 6:30 alarm, then your last substantial caffeine belongs somewhere around the middle of the afternoon, your largest meal belongs before class rather than after it, your drive home has to be something you can do alert, and the half hour after you walk in the door has a specific job to do. The anchor point is sleep, because sleep is the one part of the day that cannot be compressed indefinitely without the rest of the day falling apart. Everything else can be arranged around it. This is not a counsel of perfection. Evening students have little slack, and plenty of nights will not go to plan. The value of a backward plan is that it tells you which compromises are cheap and which are expensive. Eating a sandwich in the car is a cheap compromise. A large energy drink at the break is an expensive one, because you will pay for it at one in the morning. Driving home alone when you are nodding off at red lights is the most expensive compromise of all, and this book will be blunt about it. What this book covers The chapters move roughly in the order of an evening. Chapter 1 explains the biology that makes night classes feel the way they do: the two systems, one tracking time of day and the other tracking how long you have been awake, that together determine how alert you are at any given hour. Understanding them explains why you can feel lively at seven and wrecked at nine-thirty, and why the late afternoon is so treacherous. Chapter 2 turns that biology into a method, showing how to build an evening plan backward from bedtime and how to adjust it for different schedules. Chapters 3 and 4 take on food and caffeine, the two levers students reach for most often and understand least. Chapter 3 looks at what the research says about late eating and how to feed yourself when there is no dinner hour. Chapter 4 explains how long caffeine really lasts, what the federal guidance on daily intake means in practice, and how to calculate a personal cut-off time. Chapter 5 deals with the classroom itself: how to stay attentive through a two- or three-hour session when the day has already been long. Chapters 6 and 7 cover the journey home, first by car, where drowsiness is the main danger, and then on foot and by public transit, where darkness and isolation change the risks. Chapter 8 addresses the part of the evening most students neglect entirely, the transition from class to sleep, and Chapter 9 widens the lens to the families and workplaces that surround every evening student and that can make these habits either possible or impossible. The conclusion does not summarize; it argues for a particular way of thinking about the whole term, not just the single night. A few notes before starting The scenarios in this book are illustrative composites, built to show how the principles work in realistic situations. They are not case studies of real people. Figures and research findings are drawn from published sources, listed in the Notes at the back. This book offers general information, not individual medical advice. People differ in how they metabolize caffeine, in how their digestive systems handle late meals, in their sleep needs and in their health conditions. If you are pregnant, have a heart condition, anxiety disorder, reflux disease, diabetes, a sleep disorder or any other medical condition, or take medications that interact with caffeine or affect alertness, talk with a doctor, nurse practitioner or pharmacist about how the general guidance here applies to you. If you ever feel chest pain, a racing or irregular heartbeat, or confusion after consuming caffeine or energy products, treat it as a medical emergency. And if the strain of combining school, work and family ever leaves you thinking about harming yourself, call or text 988 to reach the 988 Suicide and Crisis Lifeline in the United States, at any hour. Finally, a word about tone. Evening students are frequently the most motivated people on a campus. They have chosen to add study to lives that were already full, usually for reasons that matter a great deal to them: a better job, a career change, a promise to themselves or to their children. They do not need to be told to try harder. What they need, and what this book tries to offer, is a clearer picture of how their bodies behave in the evening and a set of practical ways to work with that behavior instead of against it. The goal is not only to get through the semester, but to get through it healthy, rested enough to learn, and safely home every night. Chapter 1: The Evening Student's Body Ask a group of night students when they feel worst during a typical class day and the answers cluster in two places. The first is somewhere between three and five in the afternoon, often in the car or on the bus on the way to campus, when the eyelids feel heavy and the idea of sitting through three hours of lecture seems absurd. The second arrives late in the evening, sometimes during the last half hour of class, more often on the way home. Many of the same students will also report something that puzzles them: a stretch in the early evening, around the first break, when they feel surprisingly clear-headed, even energetic, despite having been awake since dawn. None of this is imagined, and none of it is a failure of discipline. It is the predictable output of two biological systems that govern alertness in every human being. Understanding them is the foundation for everything else in this book, because once you know why your energy rises and falls when it does, you can stop fighting the pattern blindly and start planning around it. Two forces that decide how awake you are Sleep scientists have described the regulation of sleep and wakefulness for decades with what is known as the two-process model, first set out by the Swiss researcher Alexander Borbély in 1982.[1] The model has been refined many times since, but its central insight has held up remarkably well. At any moment, how sleepy or alert you feel is the result of two separate processes interacting. The first is sleep pressure, sometimes called the homeostatic sleep drive. It works much like hunger. From the moment you wake up, pressure to sleep begins to build, and it keeps building for every hour you stay awake. Sleep discharges it; staying awake accumulates it. One of the chemical signals involved is adenosine, a molecule that gradually builds up in the brain during waking hours and that promotes sleepiness by acting on specific receptors. This matters for the evening student because caffeine, as Chapter 4 will explain in detail, works mainly by blocking those same adenosine receptors. It does not remove sleep pressure. It hides it. The second process is the circadian rhythm, an internal clock that runs on a cycle of roughly twenty-four hours. Its master timekeeper is a small cluster of cells in the hypothalamus called the suprachiasmatic nucleus, which receives information about light directly from the eyes and uses it to keep the body's many internal rhythms aligned with the outside day. The circadian clock drives changes in body temperature, hormone release, digestion and, crucially, alertness. Unlike sleep pressure, it does not simply increase or decrease; it cycles, pushing toward wakefulness at some times of day and withdrawing that push at others. The two forces interact in a way that can seem counterintuitive. Across a normal day, the circadian system actually pushes harder toward wakefulness as the day goes on, rising into the evening. It does this precisely to offset the steadily mounting sleep pressure. Without it, a person who woke at six in the morning would be struggling to stay awake by mid-afternoon and would be barely functional by evening. Then, a couple of hours before your usual bedtime, the circadian push withdraws, the hormone melatonin begins to rise, and the accumulated sleep pressure is finally allowed to show itself. That is when sleep comes easily, if you let it. This is the most important piece of biology for a night student to understand. The clarity you may feel in the early evening is not a fluke. It is your circadian system working at full strength to keep you awake against a day's worth of accumulated pressure. Researchers have even given part of this period a name. In the 1980s the Israeli sleep scientist Peretz Lavie, studying people who were given brief sleep opportunities around the clock, found that it was unusually hard for participants to fall asleep in the few hours before their normal bedtime, and described this stretch as a "forbidden zone" for sleep.[2] Later researchers often call it the wake maintenance zone. For many evening students, the wake maintenance zone overlaps with the first half of class. It is a genuine asset: a biologically supported window of alertness that happens to line up with lecture time. It also has limits. When it ends, often in the later part of the evening, the full weight of the day's sleep pressure lands at once. That is why many students feel fine at seven and dreadful at nine-thirty. Nothing has gone wrong between those hours. The biological support has simply been withdrawn on its usual schedule. The treacherous late afternoon The first slump of the day, the one that tends to arrive between roughly two and five in the afternoon, has a different explanation. Many people blame it on lunch, and lunch can make it worse, but the dip happens even in people who have not eaten. The sleep researcher Timothy Monk, reviewing decades of work on what is often called the post-lunch dip, concluded that it is a real phenomenon rooted in human biology, one that can occur even when a person has had no lunch and does not know what time of day it is, and that a heavy, carbohydrate-rich meal makes it more pronounced.[3] In the two-process picture, the afternoon dip is a moment when the circadian clock's wake-promoting signal has not yet risen high enough to fully offset the sleep pressure built up since morning. It is a trough between the morning's alertness and the evening's circadian support. For most workers, it is an inconvenience. For night students, it is a hazard, because it tends to coincide with the transition between work and school: the drive to campus, the hurried meal, the wait for class to begin. This has three practical consequences that later chapters will develop. First, the afternoon is the time when caffeine is most tempting and, for many schedules, most defensible, as long as it is timed with bedtime in mind. Second, the commute to campus deserves more caution than most students give it, especially after a physically demanding shift or a short night's sleep. Third, the meal eaten in that window should be designed to avoid deepening the dip, which usually means avoiding a very large, carbohydrate-heavy plate immediately before sitting down in a warm lecture hall. Owls, larks and the rest of us Not everyone's clock runs on the same schedule. People differ in what researchers call chronotype: their natural preference for earlier or later timing of sleep and activity. Extreme "larks" wake early without effort and fade early in the evening; extreme "owls" struggle to wake in the morning and come alive at night. Most people sit somewhere in between. Chronotype is partly inherited and shifts with age: it tends to become later through adolescence, peaks in lateness around the end of the teenage years and early twenties, and then drifts earlier through adulthood. The chronobiologist Till Roenneberg and colleagues, who have collected data on sleep timing from very large numbers of people, have written extensively about how strongly these differences shape daily life.[4] Chronotype matters for night students in two directions. An owl may genuinely thrive in evening classes. The lecture falls in their best hours, and the problem is not staying awake in class but getting up for the job the next morning. For a lark, especially an older student whose chronotype has drifted earlier, a class that runs until nine or ten at night pushes against their biology. They may feel the evening slump earlier and harder, and they may struggle more on the drive home. You probably already know roughly where you fall. If you are unsure, notice what happens on a free week, such as a vacation, when you can sleep and wake without an alarm. The times you settle into after a few days are a reasonable guide to your natural rhythm. It is worth knowing, because the rest of this book's advice should be tuned to it. A lark taking a class that ends at 9:45 has less margin for error on caffeine timing and the drive home than an owl in the same seat. How much sleep you actually need The biology of sleep pressure implies something uncomfortable: the only thing that truly repays it is sleep. In 2015 the American Academy of Sleep Medicine and the Sleep Research Society published a joint consensus statement concluding that adults should sleep seven or more hours per night on a regular basis to promote optimal health, and that regularly sleeping less than seven hours is associated with a range of adverse outcomes, including impaired performance, more errors and greater risk of accidents.[5] Some people, particularly young adults and those recovering from sleep debt, may benefit from more than nine hours. Evening students are at obvious risk of falling short. If class ends at 9:30, the drive takes half an hour, and there is anything at all to do at home, going to bed before 11 is a challenge. If the alarm goes off at 5:45 for work, the arithmetic simply does not reach seven hours. One or two nights like that are survivable. A semester of them, week after week, produces a condition sleep researchers call chronic partial sleep restriction, in which performance declines steadily even though the person often reports feeling only moderately tired. The unsettling finding from this line of research is that people adapt to feeling tired much faster than they adapt to being impaired. Your sense of how well you are functioning becomes an unreliable instrument precisely when you most need it. That unreliability shows itself in several ways that matter to students. Attention wavers more often and for longer. Reading comprehension suffers; you find yourself reading the same paragraph three times. Memory consolidation, much of which happens during sleep, is compromised, which means that the effort you spent in class is partly wasted if the night after it is short. Mood becomes more fragile, and small frustrations at home or work loom larger. At the extreme end are microsleeps, brief lapses of a few seconds in which the brain effectively goes offline, sometimes without the person realizing it. In a lecture, a microsleep costs you a sentence or two of notes. At sixty miles an hour, it can cost everything, which is why Chapter 6 treats drowsy driving as the single most serious risk in the evening student's day. Light: the clock's steering wheel The circadian clock does not keep perfect time on its own. In most people it runs slightly longer than twenty-four hours, and it needs daily correction. The main signal it uses is light, received through specialized cells in the retina that are especially sensitive to short-wavelength, bluish light. Bright light in the morning tends to shift the clock earlier; bright light in the late evening tends to shift it later and suppresses the release of melatonin. This creates a quiet problem for evening students. Classrooms are brightly lit. So are libraries, laptop screens, phone screens and the fluorescent corridors of many campus buildings. A student who spends the hours from six to ten in bright artificial light and then goes home to a lit kitchen and a phone screen is sending the clock a strong message that it is still daytime. Over weeks, this can push the circadian rhythm later, making it harder to fall asleep at the desired hour and harder to wake in the morning. Chapter 8 returns to this in detail, including the experimental evidence on screens before bed, and Chapter 5 explains how the same principle can be used deliberately, since light that is unhelpful at eleven at night can be useful at seven. The morning side of the light equation is just as important and costs nothing. Getting outdoor light, even on an overcast day, within the first hour or two after waking helps anchor the clock to an earlier schedule, which in turn makes it easier to fall asleep after a late class. For a working student, this can be as simple as walking the last part of the commute, drinking morning coffee by a window, or taking a short walk at a morning break. Food and the clock The circadian system does not only govern sleep. Almost every tissue in the body contains its own clock, and these peripheral clocks, including those in the liver, gut and fat tissue, are influenced not just by light but by the timing of meals. The body processes food differently at different times of day. Research on meal timing, reviewed in Chapter 3, suggests that eating the same food later in the day can change appetite hormones and energy use in ways that are not trivial. For now, the key point is that the timing of dinner is not a neutral choice. It interacts with the same biological clock that determines when you will feel sleepy. What this means for your evening Taken together, these systems produce a recognizable daily profile for most evening students. There is a period of reasonable alertness in the morning. There is a dip in the mid-to-late afternoon that coincides with the end of work and the trip to campus. There is a lift in the early evening, supported by the circadian push toward wakefulness, that carries the student into and through the first part of class. And then there is a drop, sometimes quite steep, once that circadian support begins to withdraw in the hours before the usual bedtime, a drop that typically lands during the last part of class or on the journey home. Seen this way, several habits that feel natural turn out to be poorly timed. Drinking the strongest coffee of the day at the class break, around 7:30, spends caffeine during a window in which the body is already doing a good job of keeping you awake, and ensures that much of that caffeine is still active when you are trying to sleep. Eating the biggest meal of the day after class, at ten or eleven at night, puts food into the system right as the clock is preparing for sleep. Driving home without any plan for the late-evening slump treats the most dangerous part of the day as if it were the safest. The good news is that the same knowledge points toward better timing. Caffeine used in the afternoon to cover the dip, and then stopped, can carry you to the evening's natural lift without poisoning your sleep. A real meal before class, sized and composed to avoid deepening the slump, can make a late snack unnecessary. A deliberate plan for the drive home, informed by an honest sense of how tired you are, can turn the riskiest half hour of the day into a manageable one. It also points toward a more forgiving view of yourself. If you have ever felt ashamed of nodding off in a late lecture, or of feeling unable to face a textbook at ten at night, it is worth knowing that you were fighting two of the most powerful regulatory systems in the human body. Willpower plays a role in any demanding schedule, but it works best when it is spent on arranging the day sensibly, not on overriding biology minute by minute. The next chapter shows how to do that arranging. Chapter 2: Planning the Evening Backward from Bedtime Most people plan a school night the way they plan a road trip with no fixed destination: one leg at a time. Class starts at six, so I leave work at five. I am hungry, so I stop for food. I am tired, so I buy coffee. Class ends at nine, so I drive home. I am wired and hungry, so I snack and scroll. I get to bed at half past midnight and wonder why the alarm feels like an assault. Every decision in that sequence is reasonable in isolation. The trouble is that each is made to solve the problem immediately in front of it, and the costs are all deferred to the same place: the night's sleep. By the time those costs arrive, it is too late to do anything about them. This chapter proposes a different way to plan, one that starts from the end of the evening rather than the beginning. Choosing the anchor The anchor of a backward plan is the time you intend to be asleep. Not the time you go to bed, and not the time you intend to start "winding down," but the time you want to actually be asleep. The distinction matters, because most people need somewhere between ten and thirty minutes to fall asleep after lying down, and more on nights when they arrive home stimulated. To choose the anchor, start from the morning. When does your alarm go off on the day after class? Count back at least seven hours, the minimum the American Academy of Sleep Medicine and the Sleep Research Society recommend for adults on a regular basis.[1] If your alarm is at 6:30, your anchor is 11:30 at the latest. If you know you function better with more, and many people do, move it earlier. For some students, this arithmetic produces an anchor that seems impossible. If class ends at 9:45, the commute takes forty minutes and the alarm goes off at 5:30 for an early shift, seven hours would require being asleep by 10:30, which is barely possible even with no delay at all. In that case, the backward plan does not magically create time, but it does something useful: it shows clearly that the schedule is running a deficit, which means the deficit has to be paid somewhere else. That might mean protecting sleep on non-class nights, planning a nap on a day off, adjusting the work schedule on the morning after class, or, in some cases, reconsidering the number of evening classes taken in one term. It is far better to see that problem on paper in the first week than to discover it in week nine, when exhaustion has already begun to erode grades and health. Working backward, step by step Once the anchor is set, the rest of the evening falls into place by counting backward from it. The sequence below is a template, not a script, and the chapters that follow explain each step in detail. 1. Asleep time. The anchor, chosen as above. 1. Lights out. Typically fifteen to thirty minutes before the anchor, more if you tend to lie awake after class. 2. Wind-down. A period of thirty to sixty minutes before lights out, with dim light, no heavy food, no work email and as little screen time as you can manage. Chapter 8 covers what belongs in this window. 3. Arrival home. The wind-down cannot start until you get home, which means the commute has to be accounted for honestly, including any stops. 4. Departure from campus. Class end time plus the minutes it takes to pack up, talk to the instructor, walk to the car or the stop and, if needed, wait for a campus escort or a bus. 5. Caffeine cut-off. A point several hours before the anchor, which Chapter 4 explains how to calculate from caffeine's half-life and the research on its effects on sleep. For most people with a late-evening anchor, this falls somewhere in the afternoon. 6. Main meal. Placed early enough that it is well digested before lying down, ideally two to three hours or more before sleep, as Chapter 3 discusses, which usually means before class or at the break rather than after. Written out this way, the plan looks almost bureaucratic. In practice it takes only a few minutes to work through for a given schedule, and once done it rarely needs revisiting until the timetable changes. What it produces is a small number of fixed points, typically a caffeine cut-off, a meal time, a departure time and a lights-out time, that turn the evening from a series of improvisations into something closer to a routine. Three sample evenings The principle is easiest to see in examples. The three evenings below are illustrative composites, not real people, chosen to represent common situations. The full-time worker with one long weekly class. Consider a 34-year-old logistics coordinator who works from 8:00 to 4:30 and takes an accounting course that meets on Wednesdays from 6:00 to 9:15, twenty-five minutes from home. His alarm goes off at 6:15. Seven hours before that is 11:15, so he sets his anchor at 11:00 to give himself a little margin. Working backward: lights out at 10:40, wind-down beginning at about 10:00, arrival home at about 9:50 after a few minutes of packing up and the twenty-five-minute drive. That leaves him only ten minutes of slack between arriving and starting his wind-down, which tells him that the drive home and whatever he does in his first minutes at home need to be simple. There is no room for a large meal after class, and no room for a wired hour on the couch. His caffeine cut-off, calculated as in Chapter 4, lands in the early to middle part of the afternoon. On Wednesdays, he moves his usual 3:30 coffee to 1:30 and makes it a smaller one. His main meal becomes a packed dinner eaten at his desk at 4:30 or in the car before class, sized to hold him until the morning with perhaps a small snack at the break. The result is not a transformed life. It is an evening in which the three hours of accounting still feel long, but in which he falls asleep at 11 rather than at 12:30, and in which Thursday morning is merely tired rather than wrecked. The parent with two weeknight classes. Now consider a 29-year-old medical assistant and single mother of a seven-year-old who takes two courses on Monday and Thursday evenings from 6:30 to 9:30, twenty minutes from home. Her mother watches her son on class nights. Her alarm goes off at 5:45 because she has to get her son ready for school before a 7:30 shift. Seven hours before 5:45 is 10:45. Working backward from 10:45 is sobering. Class ends at 9:30; she is home by about 10:00 at the earliest. Her mother usually wants to talk for a few minutes before leaving, and her son is sometimes still awake. A realistic lights-out is 10:45 and a realistic asleep time is 11:15, which gives her six and a half hours on class nights. The backward plan does not solve this, but it clarifies the options. She can protect sleep on the other five nights, making sure they are genuinely seven hours or more. She can talk with her mother about shifting the handover conversation to the next day by phone. She can ask her employer whether a slightly later start time is possible on Tuesdays and Fridays, the mornings after class. She can set a firm caffeine cut-off of early afternoon on class days, knowing that anything later will cost her the margin she cannot afford. And she can decide, before the semester begins, that she will not take a third evening class in the same term. The student who takes the bus. Finally, consider a 22-year-old retail worker whose shift ends at 3:00 and who takes a biology class with lab from 5:30 to 9:20. He does not own a car and takes two buses home, a trip that takes about fifty minutes at night because the second bus runs only every half hour after nine. His alarm on the following day is at 8:00 because he works a late shift. His anchor is easier: seven hours before 8:00 is 1:00, so he can be asleep by 12:30 with room to spare. His challenges are different. The long, uncertain transit ride means that his evening's end point is less predictable, and his safety plan for the walk to the stop and the wait matters a great deal, as Chapter 7 discusses. Because he is an owl who naturally stays up late, his risk is not so much short sleep on class nights as drifting later and later, eventually shifting his whole rhythm until he struggles on the mornings when he does have to get up early. His plan focuses less on caffeine and more on light and routine: a consistent lights-out, a dim, quiet wind-down after the bus ride and outdoor light the next morning. Why sleep is the anchor It is fair to ask why sleep should be the anchor rather than something else, such as study time, family time or work performance. There are three reasons. The first is that sleep is the least compressible part of the day. You can eat dinner in fifteen minutes instead of forty. You can commute by a different route. You can study in shorter blocks. But sleep that falls below what your body needs does not simply produce a proportionally shorter night's rest; it produces measurable impairment the next day, and the impairment compounds over consecutive nights. Research on chronic sleep restriction consistently finds that performance keeps declining across successive nights of short sleep even as people's sense of how sleepy they are levels off. In a widely cited laboratory study, Hans Van Dongen, David Dinges and colleagues restricted healthy adults to four, six or eight hours in bed for fourteen consecutive nights. Those limited to six hours showed steadily accumulating deficits in attention and cognitive performance, and by the end of two weeks their deficits were comparable to those seen after up to two nights of total sleep deprivation. Yet their ratings of their own sleepiness rose only modestly after the first few days, and the researchers concluded that participants were largely unaware of how impaired they had become.[2] Six hours a night is a common figure for evening students during a busy term. That study is a good reason to treat it as a warning rather than a norm. The second reason is that sleep serves learning directly. Much of the consolidation of new information into long-term memory takes place during sleep. A student who attends a three-hour lecture and then sleeps five hours has not merely lost two hours of rest; they have undermined the process by which the lecture becomes knowledge. For evening students, whose study time is scarce, this is a waste they can ill afford. The third reason is safety. The most dangerous thing most evening students do on a class night is travel home, and the biggest factor that makes that trip dangerous is sleep debt. A student who consistently gets enough sleep begins each class night with a smaller load of accumulated sleep pressure and faces the drive home with a larger margin of alertness. Anchoring on sleep is, among other things, an anchor on getting home. The weekly view A backward plan for a single evening is useful. A backward plan for a week is better, because the evenings are not independent. Two late nights in a row do more harm than two separated by a normal night, and weekend habits can undo the gains of careful weekdays. Start by marking the nights of the week when you have class and the mornings that follow. Those mornings are your pinch points. If you have any control over your work schedule, it is often worth spending it there: a later start, a lighter task load, a remote morning. If you do not, protect the nights before class nights instead, so that you enter each class evening with as little sleep debt as possible. Next, look at the non-class nights. They are not free time in the sense of time to be spent without limit. They are your recovery nights, and they should be treated as such, with a bedtime at least as early as your class-night anchor and ideally earlier. The temptation, particularly for students who feel they have neglected family or friends on class nights, is to stay up late on the other evenings to make up for it. That is understandable, and occasionally worth it, but as a regular pattern it turns every night into a short one. Finally, watch the weekend. Many adults sleep much later on Saturday and Sunday than during the week, a pattern Till Roenneberg and colleagues have described as social jetlag, because the body experiences it much as it would a flight across time zones.[3] A student whose weekday sleep runs from 11:30 to 6:30 and whose weekend sleep runs from 1:30 to 9:30 is effectively shifting two time zones every weekend. The Monday evening class after that shift can be especially difficult. Some extra sleep on weekends is reasonable if you are carrying a debt, but keeping wake times within an hour or so of the weekday time, and adding sleep at the start of the night rather than the end, keeps the clock steadier. When the plan breaks It will break. Class will run over. The instructor will want to talk. The car will not start. A child will be sick. The shift will run late and you will arrive at class hungry, with no packed dinner and no time to buy one. A backward plan is not a promise that the evening will go as intended; it is a reference point that lets you make better decisions when it does not. On a night when things go wrong, the plan helps you decide which rules to bend and which to hold. If you arrived without dinner, it is better to eat something modest at the break, or on the way home, than to go to bed ravenous or to eat a large meal at eleven. If you are running an hour behind on everything, it is better to cut the wind-down short and go to bed than to insist on the full routine and lose sleep. If you are exhausted and facing a long drive, the rule about driving alert is not one to bend at all; Chapter 6 explains the alternatives. A useful habit is a thirty-second review at the end of each class night, or the next morning, asking a single question: what cost me the most sleep tonight? Over a few weeks, the answer tends to repeat. For some students it is caffeine at the break. For others it is the half hour of scrolling after they get home, or a late meal, or a conversation that ran long. Once you know your particular leak, you can focus your effort there instead of trying to perfect every part of the evening at once. A plan written down For the plan to work, it helps to put it somewhere visible. Some students write their key times, caffeine cut-off, dinner, departure and lights-out, on an index card in their bag or a note on their phone. Others set a small number of phone alarms: one for the last coffee, one for dinner, one for the start of the wind-down. Alarms are not glamorous, but they turn a good intention into a prompt that arrives at the right moment, when your memory of the plan is likely to be crowded out by everything else going on. The rest of this book fills in the detail behind each of those fixed points, starting with the one that most evening students handle worst: the meal that has no natural place in their day. Chapter 3: Dinner Without a Dinner Hour For many evening students, dinner is the meal that disappears. Lunch happens at work. Breakfast, if it happens, is rushed. Dinner, the meal that most households organize the evening around, falls squarely in the hours when the student is either driving to campus or sitting in class. What replaces it tends to follow a pattern: nothing much between a noon lunch and a 9:30 class ending, then a large, fast and often high-fat meal at ten or eleven at night, eaten in the car, in front of the television or standing at the kitchen counter. That pattern has costs that go beyond nutrition in the narrow sense. It leaves the student hungry and distracted in the second half of class, it loads the stomach just before sleep, and, according to a growing body of research, it may shift the way the body handles food in ways that matter over months and years. This chapter looks at the evidence on late eating and then offers practical ways to feed yourself on a schedule that has no dinner hour. What the research says about late eating The study of meal timing, sometimes called chrononutrition, rests on the insight described in Chapter 1: the body's internal clocks extend to the organs that process food, and those organs do not behave identically at all hours. The same meal eaten at different times of day can produce different effects. One of the most carefully controlled experiments on this question was published in 2022 by Nina Vujović, Frank Scheer and colleagues at Brigham and Women's Hospital in Boston.[1] They recruited sixteen adults with overweight or obesity and had each of them go through two laboratory protocols in random order. In both protocols, participants ate exactly the same meals, and researchers controlled their sleep, physical activity and light exposure. The only difference was timing: in one protocol, the meals were scheduled about four hours later in the day than in the other. When participants ate late, they reported more hunger, their ratio of the hunger-promoting hormone ghrelin to the satiety hormone leptin rose, they burned calories at a slower rate while awake, and their fat tissue showed changes in gene activity consistent with storing more fat and breaking down less. The authors concluded that these were converging mechanisms by which late eating may increase the risk of weight gain. This was a small study in a specific group, and the researchers themselves noted that it needs to be extended, including to more women. But it is consistent with other work. An earlier study led by Marta Garaulet followed 420 people in Spain through a twenty-week weight-loss program and compared those who ate their main meal, which in that Mediterranean population was lunch, before 3 p.m. with those who ate it later.[2] The late eaters lost less weight and lost it more slowly, even though their reported calorie intake, diet composition, estimated energy expenditure, appetite hormone levels and sleep duration were similar to those of the early eaters. As an observational study, it cannot prove that meal timing caused the difference, but it points in the same direction as the laboratory evidence. It is important not to overstate what these findings mean for an individual evening student. They do not show that eating after a certain hour causes weight gain on its own, and they do not show that a single late dinner does harm. What they suggest is that, all else being equal, a pattern of shifting most of the day's eating into the late evening is not metabolically neutral, and that moving the main meal earlier, where possible, is a reasonable default. Reflux, sleep and the stomach The more immediate cost of late eating for many people is felt the same night. Lying down with a full stomach makes it easier for stomach contents to flow back into the esophagus, producing the burning sensation of heartburn and, for some, coughing, a sour taste or disrupted sleep. The National Institute of Diabetes and Digestive and Kidney Diseases lists eating meals two to three hours before lying down among the lifestyle changes doctors may recommend for people with gastroesophageal reflux disease.[3] The evidence behind that advice has some direct support. In a case-control study published in 2005, Yasuhiro Fujiwara and colleagues in Osaka compared 147 patients with reflux disease with 294 matched controls and asked how long each person typically waited between finishing dinner and going to bed.[4] After adjusting for smoking, drinking and body mass index, people whose dinner-to-bed time was less than three hours had odds of reflux disease more than seven times higher than those who waited four hours or more. Like any case-control study, it shows an association rather than proof of cause, but it matches both clinical experience and the physiology of digestion. For the evening student, the practical implication is that the post-class meal is badly placed. A student who gets home at ten and eats a large dinner at 10:15 before trying to sleep at 11:15 is doing almost exactly what the reflux guidance warns against. Even people without reflux disease often find that a heavy late meal makes it harder to settle, leaves them feeling uncomfortably full in bed and produces lighter, more broken sleep. There is a counterweight to all of this. Going to bed very hungry can also disrupt sleep, and a student who eats almost nothing between noon and midnight is likely to find both the second half of class and the night that follows harder than they need to be. The goal is not to stop eating at a particular hour at all costs. It is to move the bulk of the evening's food earlier and to keep anything eaten after class small and easy to digest. Three ways to place the meal There is no single right answer to where dinner belongs, because class schedules vary. But most evening schedules can be handled with one of three basic patterns, summarized in Table 1 and discussed below. Before class. For students who have a gap of an hour or more between the end of work and the start of class, or who can eat at work late in the afternoon, the simplest pattern is to eat a real, moderate dinner between about four and six. This fits well with the biology in Chapter 1: it arrives after the worst of the afternoon dip, it is digested well before sleep and it carries the student through class. The main risk is eating too large a meal just before sitting down, which can deepen drowsiness in the first hour of class. A moderate plate, with protein, vegetables and some whole-grain carbohydrate, tends to work better than a very large, starchy one, in line with Monk's observation that a high-carbohydrate meal can accentuate the post-lunch dip.[5] Split dinner. For students who come straight from work to class with no time to eat, or whose classes run late, a split pattern often works best: a substantial snack or small meal late in the afternoon, perhaps in the car or at a desk before leaving work, and a second small meal at the class break. The two halves together make a dinner. Neither is large enough to cause sleepiness, and by the time class ends there is no strong hunger driving a late meal at home. Small late meal. For some students, particularly those whose classes end early in the evening, or whose schedules make eating before class impossible, the only option is to eat after class. In that case, the aim is to make that meal modest, to eat it as soon as possible after class rather than just before bed and to favor foods that are easy to digest. A bowl of soup and bread, yogurt with fruit, eggs and toast or leftovers in a small portion are all better choices than a drive-through meal eaten at 10:45. Table 1. Where the evening meal can go, by schedule type. Schedule Main food At the break After class Gap of an hour or more before class Moderate dinner, 4:30 to 6:00 Water; fruit if hungry Nothing, or a small snack Straight from work, class ends by 9:30 Substantial snack before leaving work Second half of dinner Small snack only if needed Class ends after 9:30, long commute Split: late-afternoon meal and break meal Protein-based portion Small, light snack at least an hour before bed No chance to eat before class Small meal right after class Something to prevent extreme hunger Keep portion modest; eat soon after arriving What to eat, and how to make it happen The quality of an evening student's food depends less on nutritional knowledge than on logistics. Most adults know that vegetables, whole grains, beans, fish and lean proteins are good for them and that meals built mostly from fried food, refined starches and sugary drinks are not. The problem is that a person leaving work at 4:45 to reach a 6:00 class does not have the chance to buy, let alone prepare, a balanced meal, and the food that is fast and available on the way to campus is overwhelmingly the second kind. The solution is almost always to make the decision earlier in the day or earlier in the week, when you have more time and less hunger. A few approaches work well for many students. Cook once, eat twice. On a non-class night or on the weekend, cook a double portion of something that keeps and travels well: a grain bowl, a stew, a pasta dish with vegetables, a batch of rice and beans. Pack half into containers for class nights. This single habit does more for most evening students' diets than any amount of good intention at 5 p.m. Build a portable dinner. A dinner does not need to be hot or elaborate. A wrap with chicken or beans and vegetables, a container of pasta salad, a hard-boiled egg or two with a piece of fruit and some whole-grain crackers, or hummus with vegetables and pita can all serve as a real meal. What matters is that it includes some protein, some fiber from vegetables, fruit, beans or whole grains, and enough substance to hold you for several hours. Stock a class bag. Keep a small supply of shelf-stable foods in your bag or car for nights when the plan fails: nuts, a nut butter packet, whole-grain crackers, dried fruit, a shelf-stable milk or soy milk carton, a protein bar you actually like. These are not ideal dinners, but they keep you out of the vending machine and prevent the extreme hunger that leads to the late drive-through meal. Drink water. Mild dehydration can contribute to fatigue and headaches, and evening students who drink coffee through the afternoon and then sit in dry, heated or air-conditioned classrooms often drink little else. A refillable water bottle is one of the simplest tools in this book. Keep in mind that drinking a great deal in the last hour before bed may wake you during the night, so front-load fluids earlier in the evening. Plan the break. Many evening classes include a ten- or fifteen-minute break halfway through. It is short, and it is often spent in line at a vending machine or coffee cart. If you know what you will eat at the break, and have it with you, you can use the whole ten minutes to eat, stretch and get some air. Know the room rules. Some classrooms and labs do not allow food, and in laboratory courses eating is often banned for good safety reasons. Check at the first session, and if food is not permitted, plan to eat in a hallway or lounge at the break, or shift more of the meal to before class. Many instructors of evening classes are sympathetic to students eating quietly in lectures, since they know their students have come straight from work, but it is courteous to ask, and to choose foods that are not noisy or strongly scented. When the budget is tight Evening students are more likely than traditional students to be supporting others on limited incomes, and food insecurity is a serious and common problem in higher education. The Lumina Foundation reports that 22 percent of college students reported low or very low food security in the preceding thirty days.[6] Advice about packing balanced meals means little to a student who cannot reliably afford groceries. If this describes your situation, it is worth knowing that many colleges now run campus food pantries, sometimes with evening hours or pickup options designed for working students, and that student services offices can often help with applications for public benefits such as the Supplemental Nutrition Assistance Program, where eligibility rules for students are specific and sometimes confusing. Asking is not a sign of failure. Colleges run these programs because hunger is a direct obstacle to finishing a degree, and students who are fed learn better. On a limited budget, the most economical approach to class-night meals is usually the same as the healthiest: cooking in batches at home from inexpensive staples such as rice, dried or canned beans, lentils, eggs, oats, frozen vegetables, bananas and seasonal produce, and packing it. The most expensive approach, and usually the least nutritious, is buying food on the way to class several nights a week. Special situations Some students need to be more careful with meal timing than others. People with diabetes, particularly those who use insulin or certain oral medications that can cause low blood sugar, should talk with their clinician or diabetes educator before changing the timing or size of their evening meal, since an evening class can easily lead to a long gap without food or a mismatch between medication and meals. People with reflux disease, as discussed above, have particular reason to finish eating well before lying down. Pregnant students, students with a history of disordered eating and anyone following a medically prescribed diet should treat the patterns in this chapter as starting points for a conversation with a professional rather than as instructions. It is also worth noting what this chapter does not recommend. It does not recommend skipping meals as a strategy to stay alert, which tends to backfire through distraction, irritability and overeating later. It does not recommend any particular diet or rigid time window. The evidence on late eating supports a modest, practical shift, moving the main evening meal earlier and keeping late food light, not a set of rules to be followed at any cost. Dinner as part of the plan The previous chapter proposed working backward from the time you want to be asleep. Food fits into that plan naturally. If the anchor is 11:30, the main meal should ideally be finished by about 8:30, and preferably much earlier. On most evening schedules, that means before class or at the break, not after. Once that is decided, the rest follows: what to pack, when to eat it and what to keep in the bag for nights when the plan fails. The student who makes this one change, from a large late meal to an earlier main meal and a small late snack if needed, often notices a difference within a week or two. The second half of class is less distracted, the drive home is not accompanied by the smell of fries, and falling asleep becomes easier. The next chapter turns to the other lever that most evening students pull without thinking: caffeine. Chapter 4: Caffeine by the Clock Caffeine is the evening student's oldest ally. It is cheap, legal, available in every building on campus and effective. It genuinely does improve alertness, reaction time and the ability to sustain attention on dull tasks, and for someone facing three hours of lecture after a full working day, those are not small benefits. Used well, it can carry a tired student through the afternoon dip and into the evening. Used badly, it quietly steals the one thing that would have made the next day easier: sleep. The difference between using it well and using it badly is almost entirely a matter of timing. This chapter explains how caffeine works, how long it lasts, how much is too much and how to calculate a personal cut-off time that fits an evening schedule. What caffeine does, and what it does not do Chapter 1 described adenosine, one of the chemical signals that builds up in the brain the longer you are awake and that contributes to the feeling of sleep pressure. Caffeine's main effect is to block the receptors that adenosine acts on.[1] With those receptors occupied, the brain registers less of the sleep pressure that has accumulated, and you feel more alert. This mechanism explains both caffeine's power and its limits. It does not create energy, and it does not repay sleep debt. The adenosine is still there, waiting. When the caffeine wears off, the underlying sleep pressure returns, often abruptly, which is one reason why people who rely on repeated doses through the day can feel suddenly exhausted in the late evening. It also explains why caffeine works better on moderate tiredness than on severe sleep deprivation: when sleep pressure is very high, blocking some of its signal is not enough to restore normal function. Caffeine is absorbed quickly from the gut. Most people begin to feel its effects within roughly half an hour, which is why fatigue-management guidance for drivers and shift workers commonly notes that caffeine takes about thirty minutes to start working.[2] Peak levels in the blood typically come within an hour or so of drinking it. After that, the body begins to clear it, and the pace of that clearing is the most important fact in this chapter. Half-life: the number that matters most Caffeine is broken down mainly by the liver, and the rate at which that happens is usually described in terms of half-life, the time it takes for the amount in the body to fall by half. In the average adult, the half-life of caffeine is approximately five hours.[1] That average hides a great deal of variation. According to the same clinical review, smoking can shorten the half-life by up to 50 percent, and pregnancy, especially in the final trimester, can lengthen it to as much as fifteen hours. Genetics, liver function and certain medications also influence it, so two people drinking the same cup of coffee at the same time can carry very different amounts of caffeine to bed. The implications of a five-hour half-life for an evening student are striking when you work them through. Table 2 shows how much of a 200-milligram dose, roughly what you might get from a medium brewed coffee, would remain in the body at various times after drinking it at 5:00 p.m., assuming a half-life of three, five or eight hours. The figures are simplified estimates that ignore the time it takes to absorb the dose and treat clearance as a steady exponential decline, but they give an honest sense of the scale involved. Table 2. Estimated caffeine remaining from a 200 mg dose taken at 5:00 p.m. Clock time Hours later Half-life 3 h Half-life 5 h Half-life 8 h 7:00 p.m. 2 126 mg 152 mg 168 mg 9:00 p.m. 4 79 mg 115 mg 141 mg 11:00 p.m. 6 50 mg 87 mg 119 mg 1:00 a.m. 8 31 mg 66 mg 100 mg 6:00 a.m. 13 10 mg 33 mg 65 mg Calculated from the standard half-life formula; the five-hour average half-life is from Evans, Richards and Battisti (StatPearls). Look at the middle column. A student with an average half-life who drinks a medium coffee at 5:00, just before a 6:00 class, still has nearly 90 milligrams of caffeine in the body at 11:00 p.m., which is roughly the amount in a small cup of coffee. In effect, they are going to bed shortly after drinking one. A student whose half-life happens to be longer is going to bed with more caffeine in their system than a typical can of cola contains, and will still carry a meaningful amount at breakfast. The evidence on caffeine and sleep How much does this actually matter? The most useful study for evening students was published in 2013 by Christopher Drake and colleagues at the Henry Ford Hospital sleep center in Detroit.[3] They gave healthy adults 400 milligrams of caffeine, about the amount in two to three large coffees, at one of three times: at bedtime, three hours before bedtime or six hours before bedtime, and compared the results with a placebo. Participants slept at home, and their sleep was measured both by their own reports and by a portable monitor. All three timings significantly disrupted sleep compared with placebo. Even when the caffeine was taken six hours before bed, objectively measured total sleep time was reduced by more than an hour. Just as important, participants tended not to notice the disruption as fully as the monitor recorded it. The authors concluded that their findings supported advice to avoid substantial caffeine for at least six hours before bedtime. Six hours is a sobering number for someone with an 11:30 anchor: it points to a cut-off around 5:30 p.m. for a large dose, which is before many evening classes even start. And the study used a fixed, fairly large dose, so it does not tell us precisely how smaller amounts would behave. But taken together with the half-life figures in Table 2, it strongly suggests that the classic evening student habit, a large coffee or energy drink before class or at the break, is costing real sleep, whether or not the student feels it. How much is too much? Timing is one question; total amount is another. The U.S. Food and Drug Administration has cited 400 milligrams a day as an amount not generally associated with dangerous, negative effects for most healthy adults, describing it as roughly two to three 12-ounce cups of coffee.[4] The FDA also notes that people vary widely in their sensitivity to caffeine and in how fast they metabolize it, and that pregnant or breastfeeding people, and those who take certain medications or have certain health conditions, may need to limit it more and should ask their health care provider. The same FDA guidance warns that toxic effects, such as seizures, can occur with rapid consumption of around 1,200 milligrams of caffeine, and it specifically cautions against pure and highly concentrated caffeine products, which can have serious consequences including death. A single teaspoon of powdered caffeine, the Mayo Clinic notes, can contain roughly as much caffeine as twenty-eight cups of coffee.[5] Such products have no place in a student's study routine. The 400-milligram figure is a ceiling, not a target, and it says nothing about timing. A student who drinks 300 milligrams of caffeine a day, well within the FDA figure, can still have badly disrupted sleep if much of it comes after four in the afternoon. Knowing how much caffeine you actually consume is harder than it sounds, because the content of common drinks varies widely. Table 3 shows the ranges the FDA reports for 12-ounce servings of several common drinks. Table 3. Caffeine in a 12-fluid-ounce serving of common drinks. Drink Typical caffeine range Brewed coffee 113 to 247 mg Energy drink 41 to 246 mg Caffeinated soft drink 23 to 83 mg Decaffeinated coffee, per 8 oz 2 to 15 mg Source: U.S. Food and Drug Administration, "Spilling the Beans: How Much Caffeine Is Too Much?" (2024). Two features of these figures deserve attention. First, the ranges are wide: a 12-ounce brewed coffee from one shop may contain twice as much caffeine as the same size from another. Large coffee-shop sizes are often 16 or 20 ounces, which can push a single drink toward or beyond 300 milligrams. Second, decaffeinated coffee is not caffeine-free. For most people the small amount is irrelevant, but for someone who is very sensitive, several cups of decaf in the evening are not nothing. Caffeine also turns up in places students may not think of: tea, both black and green; chocolate and cocoa; some pre-workout powders and "focus" supplements; caffeinated water and gum; and some over-the-counter pain relievers, which list caffeine among their active ingredients. Reading labels for a week can be an eye-opening exercise. Calculating your personal cut-off Putting the pieces together gives a practical method for setting a caffeine cut-off time on class days. Start with your anchor, the time you want to be asleep, from Chapter 2. Count back at least six hours, in line with the Drake study. That is your default cut-off for any substantial dose. For an 11:30 anchor, it falls at 5:30 p.m.; for an 11:00 anchor, at 5:00 p.m. Then adjust for what you know about yourself. If you have noticed that even an afternoon coffee keeps you awake, or if you are pregnant, taking medication that slows caffeine clearance or are simply a "slow metabolizer" by experience, move the cut-off earlier, perhaps to eight hours before your anchor. If you smoke, your caffeine may clear faster, although the far larger health gain would come from quitting smoking, which your campus health service can help with. Next, adjust for dose. The six-hour rule was based on a large dose. A smaller amount, such as a cup of black tea or a small coffee, taken close to the cut-off will leave less in your system at bedtime. A useful way to think about it is that the later in the day you drink caffeine, the smaller the amount should be. To see how this works in practice, return to the logistics coordinator from Chapter 2, whose anchor on class nights is 11:00. Six hours earlier is 5:00 p.m., so on paper a coffee at 4:30 before leaving work would just about qualify. But he knows from experience that a large afternoon coffee keeps him awake, and his usual drink is a 16-ounce brewed coffee that could easily contain well over 200 milligrams. So he moves his cut-off to about 1:30, well over eight hours before his anchor, and switches his late-afternoon drink to a small black tea, which contains far less caffeine, or to decaf when he wants only the ritual. His total daily caffeine barely changes. Its timing changes a great deal. Finally, test it. For two weeks, follow your cut-off strictly and pay attention to how quickly you fall asleep, how often you wake and how you feel in the morning. Then, if you want, experiment carefully with small changes. Your own experience, gathered deliberately rather than assumed, is the best guide you have, with the caveat from the Drake study that people tend to underestimate how much caffeine is disturbing their sleep. Using caffeine strategically For most evening students with a typical schedule, the backward plan leads to a pattern something like this: caffeine in the morning as usual; a moderate dose in the early to middle afternoon, timed to cover the post-lunch dip and the transition to campus; and then nothing, or only a small amount of something mild, from the late afternoon on. The early-evening lift described in Chapter 1 does much of the work of keeping you alert through the first half of class, and the tools in Chapter 5, movement, light and active engagement, can help with the second half. This is harder than it sounds, because the moment of greatest temptation is the class break, when the energy of the evening lift has begun to fade and the coffee cart is right there. A student who knows why the break coffee is expensive, and who has a plan for what to do instead, is far more likely to resist it than one who simply tries to say no. There is one important exception, which Chapter 6 addresses in detail: when you are too drowsy to drive safely. Safety guidance for drowsy drivers recommends pulling over to rest, and a combination of caffeine and a short nap can provide temporary relief when a driver has to continue.[2] A student who is dangerously sleepy at the wheel should not let concern about sleep quality stop them from using caffeine in that situation, although the real solution is a schedule that avoids reaching that state. Energy drinks Energy drinks deserve a separate word, because they are especially popular with students and especially easy to misuse. Some contain no more caffeine than a cup of coffee; others contain much more, and it is easy to drink them quickly and cold, taking in a large dose in a few minutes. Many also contain large amounts of sugar and other stimulant ingredients whose effects in combination are less well studied. Two cautions matter most. First, the FDA's concern about rapid consumption of large doses applies with particular force to energy drinks and energy shots. Drinking several in a short period, especially on an empty stomach or after little sleep, can produce a racing heart, tremors, anxiety and nausea. Anyone who experiences chest pain, a very rapid or irregular heartbeat, fainting, confusion or seizures after consuming caffeine should seek emergency medical care. Second, mixing energy drinks with alcohol is a poor idea, because the stimulant can mask the feeling of intoxication without reducing actual impairment, which can lead people to drink more and to believe they are fit to drive when they are not. Tolerance, dependence and cutting back People who consume caffeine regularly develop tolerance to some of its effects, meaning that the same dose produces less alertness over time. This is one reason why caffeine use tends to creep upward during a demanding semester. Regular users can also experience withdrawal symptoms, such as headache, fatigue, difficulty concentrating and irritability, when they stop suddenly, typically beginning within a day or so and fading over several days.[6] If you decide that you are drinking more caffeine than you want, or that your afternoon and evening caffeine needs to shift earlier, it is usually better to make the change gradually, reducing by a modest amount every few days, than to stop all at once in the middle of a busy week. A good time to start is a break between terms, or a week with fewer demands. Replacing a later cup with decaffeinated coffee or herbal tea can preserve the ritual, which for many people is part of the appeal, without the stimulant. Caffeine in the backward plan Caffeine fits into the backward plan as one of its fixed points: a cut-off time, written down and ideally set as a phone reminder, that protects your sleep. Around that point sit the other choices: how much to drink earlier, what to drink instead later and how to handle the break. Of all the changes described in this book, moving caffeine earlier is among the easiest to make and the quickest to show results. Many students find that within a week or two they fall asleep faster after class and wake less tired, without any other change to their routine. The next chapter looks at what to do instead of caffeine when the lecture is long and your attention is flagging. Chapter 5: Staying Sharp in the Seat There is a particular kind of silence that settles over an evening classroom at about 8:15. The instructor is still talking. The slides are still changing. But half the room has the fixed, slightly glazed look of people who are physically present and mentally elsewhere. A few are fighting it openly, shifting in their seats, rubbing their eyes, taking sips from empty cups. One or two have lost the fight, their heads dropping slowly forward and jerking back up. Nobody in that room is lazy. Most of them have been awake for fourteen hours or more, many have worked a full shift and all of them are now sitting still, in a warm room, listening to a single voice, at the point in the evening when the circadian lift described in Chapter 1 is beginning to fade. The conditions could hardly be better designed to produce drowsiness. The previous chapter argued that reaching for more caffeine at this point is usually a poor trade. This chapter is about what to do instead. None of the strategies here is as powerful as a good night's sleep, but together they can make a real difference to how much of a long evening class you actually absorb. Arrive less depleted The best time to prepare for the second half of class is before the first half begins. Much of what determines how you feel at 8:15 has already been decided by 5:30: how much you slept the night before, whether you ate a sensible meal, whether you spent the afternoon dip in a state of mounting exhaustion or managed to take the edge off it. For students whose schedule allows it, one of the most effective tools is a short nap in the late afternoon. Research on napping in healthy adults, reviewed by Catherine Milner and Kimberly Cote, finds that short naps can improve alertness and performance, and that very brief naps, on the order of ten to twenty minutes, have the advantage of producing benefits quickly while minimizing sleep inertia, the grogginess that follows waking from deeper sleep.[1] In one well-known study, Amber Brooks and Leon Lack compared naps of different lengths taken in the afternoon after a night of restricted sleep and found that a ten-minute nap produced prompt and sustained improvements in alertness and performance, while longer naps were followed by a period of grogginess before any benefits appeared.[2] A short nap in a parked car, in a quiet corner of the library or on a break room couch between work and class can therefore be worth more than a coffee, and it has no effect on the caffeine level you take to bed. The key is to keep it short. Set an alarm for twenty minutes, allowing a few minutes to fall asleep, and get up when it goes off. A long nap late in the afternoon can leave you groggy for the start of class and, by reducing sleep pressure, can make it harder to fall asleep at your anchor time. Some people combine the two: they drink a coffee immediately before a short nap, so that the caffeine begins to take effect at about the time they wake. This is the same approach recommended for drowsy drivers in Chapter 6. For an evening student, it only makes sense if the timing still respects the caffeine cut-off from Chapter 4, which in practice means it works best for classes that start in the late afternoon. If you park in a campus lot or garage to nap, choose a well-lit, busy area, lock the doors and let someone know where you are. Chapter 7 discusses campus safety in more detail. Use your body The single most reliable way to fight drowsiness in class, short of sleeping, is to stop sitting still. Posture and movement both matter. Sleepiness is harder to maintain when you are upright and engaged than when you are slumped and passive, and even small amounts of movement raise alertness temporarily. At the class break, get up and walk. A few minutes of brisk walking, ideally outside or at least out of the classroom, is more effective than standing in line for a vending machine. Climbing a flight or two of stairs is even better. Swing your arms, stretch, breathe deeply. The goal is not exercise in any meaningful fitness sense but a brief change in physiological state. During class, sit upright with both feet on the floor. If the room and instructor allow it, stand at the back for part of the session; many instructors are happy for adult students to do this if asked beforehand, and some classrooms have high tables for exactly this purpose. Small movements, such as flexing your feet, shifting position or rolling your shoulders, are unobtrusive and can help. If you are fighting sleep hard, it is better to step out quietly for a two-minute walk to the water fountain than to spend twenty minutes nodding. Regular physical activity outside class also helps, both by improving the quality of sleep and by building general fitness. The federal Physical Activity Guidelines for Americans recommend that adults get at least 150 to 300 minutes of moderate-intensity aerobic activity a week, along with muscle-strengthening activity on two or more days.[3] For an evening student, that can sound like one more demand on an overfull schedule, but it can often be built into existing routines: walking or cycling part of the commute, a brisk walk at lunch, a few sessions a week on non-class days. Vigorous exercise very close to bedtime may make it harder for some people to fall asleep, so on class nights it is usually better to be active earlier in the day. Use light Chapter 1 described how light influences the circadian clock. Light also has a more immediate effect: bright light, particularly light rich in the shorter blue wavelengths, has an alerting effect on the brain, independent of its effects on the clock.[4] This is part of the reason why evening screen time can make it harder to fall asleep, a problem Chapter 8 discusses. But in the context of an evening class, the same effect can be used deliberately. If you have a choice of where to sit, choose a well-lit part of the room rather than a dim corner. If the instructor dims the lights for slides, sit closer to the screen or the windows. At the break, step into a brightly lit hallway or lobby rather than a dark courtyard. These are small effects, but they add up. There is a tension here, of course. The same light that helps you stay alert at 7:30 may, if it continues all evening, push your internal clock later and make it harder to fall asleep. The answer is to use light when you need alertness, in class and on the drive home, and then to dim it deliberately once you are home and moving toward sleep. Engage actively Passive listening is far more conducive to drowsiness than active engagement. Students who take notes by hand, ask questions, work through problems as the instructor presents them and participate in discussions tend to report less difficulty staying awake than those who simply listen. There is also good reason to think that this kind of active processing improves learning itself. A few techniques work well for evening students. Sit near the front, where it is harder to drift and easier to engage. Take notes in your own words rather than trying to transcribe the slides; the effort of summarizing keeps your mind working. Prepare a question or two from the reading before class, so you have a reason to listen for particular points. When the class includes group work or discussion, participate fully rather than letting others carry it; talking is one of the most effective antidotes to drowsiness there is. It also helps to know the structure of the evening. If you can see from the syllabus or slides which topics will be covered when, you can plan to be at your most attentive for the material that matters most and use the less demanding stretches as recovery. Mind the environment Warm, stuffy rooms promote drowsiness. If you can, dress in layers so you can shed a sweater when the room gets warm, and choose a seat near a window, door or air vent. A cool drink of water can help, and staying hydrated through the evening, as Chapter 3 suggested, prevents the mild dehydration that can add to fatigue and headaches. Some students chew gum or eat something crunchy to stay alert. Evidence on these strategies is limited, but if they work for you and your instructor does not object, they are harmless. Be careful with eating as a way of staying awake, though: grazing on sugary snacks through the second half of class tends to add calories without solving the underlying problem. Put the hard thinking in the right hours Evening students often assume that the time to study is after class, while the material is fresh. It is an appealing idea and, for most people on most nights, a poor one. By the time a three-hour class ends and the trip home is done, the circadian support that carried you through the early evening has usually withdrawn, sleep pressure is at its peak for the day, and the anchor time from Chapter 2 is close. Reading a chapter at 10:30 p.m. under those conditions tends to produce the familiar experience of turning pages without retaining them, and every minute spent doing it comes directly out of sleep, which is itself when much of the day's learning is consolidated. A better approach is to protect a short period for review within a day of class, but not on the night itself. Twenty or thirty minutes the next morning, at lunch or on the commute if you ride transit, spent going back over your notes and filling in gaps, is usually worth more than an hour of bleary effort the night before. Heavier work, such as problem sets, papers and reading for the next session, generally belongs on non-class days, weekends or the earlier part of the day, when your attention is freshest. For students with fixed work hours, even a regular early-morning block of forty-five minutes before the household wakes can be the most productive study time of the week, provided it does not come at the cost of short sleep. The same logic applies to preparation. Skimming the reading or the slides for the next class, even briefly, gives you a scaffold for the lecture, so that you spend the evening connecting new material to something you have already seen rather than meeting it cold. Students who arrive prepared consistently find it easier to stay engaged, which in turn makes it easier to stay awake. Preparation is, among other things, a fatigue strategy. An evening class, hour by hour It can help to see these tools arranged across a single session. Consider an illustrative three-hour class running from 6:00 to 9:00, taken by a student who has worked a day shift. In the late afternoon, between leaving work and class, the student eats the first half of a split dinner, as described in Chapter 3, and, if there is time, takes a ten- to twenty-minute nap in a safe place. Any caffeine for the day was finished in the early afternoon. Arriving a few minutes early, the student takes a seat in a well-lit part of the room near the front and looks over the slides or reading. From 6:00 to about 7:15, the early-evening lift is at work. This is the time for the most demanding material, and the student takes notes in their own words and asks at least one question. At the break, the student walks for five minutes, ideally up and down a flight of stairs or outside, eats the second half of dinner and refills the water bottle, skipping the coffee cart. From about 7:30 to 9:00, attention is harder to hold. The student sits upright, stands at the back for part of the time if the instructor allows it, and uses small movements to stay alert. If the fight against sleep becomes intense, a brief walk to the water fountain is better than twenty minutes of nodding. At the end, rather than lingering to start homework, the student packs up, notes any questions to review the next day and turns attention to the trip home. Nothing in this sequence is dramatic, and none of it requires unusual discipline. It simply lines up the student's choices with the biology of the evening instead of working against it. Talk to your instructors Many instructors who teach evening classes are acutely aware of the challenges their students face and structure their sessions accordingly: with breaks, changes of activity and a variety of teaching methods. Others, particularly those who also teach day classes, may simply transplant a daytime lecture into the evening without adjustment. If you find a particular class very hard to stay alert in, it is reasonable to raise it, politely and constructively. An instructor may be willing to add a short break, to allow students to stand at the back, to break up a long lecture with a brief activity or to post slides or recordings so that a moment of inattention does not mean losing material. Evening students are often more comfortable than younger students in having this kind of adult conversation, and instructors often appreciate the feedback. If you have a disability or health condition that affects your alertness or energy, your college's disability services office can help arrange formal accommodations. When sleepiness is a warning Occasional drowsiness in a long evening class is normal. Persistent, overwhelming sleepiness, the kind that makes it hard to stay awake in class even when you have slept reasonably well, is not, and it deserves attention. Several common sleep disorders cause excessive daytime sleepiness. Obstructive sleep apnea, in which breathing repeatedly stops or becomes shallow during sleep, is common among adults, particularly those who snore loudly, are overweight or have high blood pressure, and it often goes undiagnosed. Insomnia, in which people have persistent difficulty falling or staying asleep, is also common, and it frequently responds well to a structured treatment called cognitive behavioral therapy for insomnia. Depression, some medications and a range of medical conditions can also cause fatigue. If you find yourself falling asleep in class despite reasonable sleep, if others have told you that you snore heavily or stop breathing in your sleep, if you regularly struggle to stay awake while driving or if your fatigue is getting worse over time, talk to a clinician. Many colleges have health services with evening hours; your own primary care provider is another option. Sleep disorders are treatable, and treating them can transform both academic performance and safety. Accept some limits Finally, it is worth being honest about what these strategies can and cannot do. They can help a reasonably rested student stay alert and engaged through a long evening class. They cannot make a severely sleep-deprived student function well, and no amount of standing, walking or note-taking will fully make up for a week of five-hour nights. If you find that you are routinely fighting sleep in class despite doing everything in this chapter, the problem is almost certainly upstream: in the amount of sleep you are getting, the timing of your caffeine or the overall load you are carrying. The backward plan in Chapter 2 is the place to start looking for the answer. The same honesty matters even more once class ends. A student who has fought drowsiness through the last hour of a lecture is not going to be magically alert for the drive home, and that drive is the subject of the next chapter. Chapter 6: The Drive Home Of all the risks an evening student faces, the drive home after a late class is the most serious, and it is the one most often dismissed. Students who would never consider driving after three drinks will routinely drive forty minutes on a dark highway after sixteen hours awake, a full shift and a three-hour lecture, with the radio turned up and the window cracked open, trusting that they will be fine because they have always been fine before. Most of the time they are. But drowsy driving is not a problem of averages. It is a problem of the one night when the microsleep lasts four seconds instead of one, on the one stretch of road where a four-second lapse is enough. This chapter explains how common drowsy driving crashes are, why they are especially dangerous, how to recognize when you are too tired to drive and what to do about it. It is the most important chapter in this book, and the one where the advice is least negotiable. How big is the problem? The National Highway Traffic Safety Administration estimates that in 2017, 91,000 police-reported crashes involved drowsy drivers, leading to an estimated 50,000 people injured and nearly 800 deaths.[1] NHTSA's own drowsy driving page also reports 644 deaths in drowsy-driving-related crashes in 2024. The agency acknowledges that these figures are widely believed to understate the problem, because, unlike alcohol, drowsiness leaves no measurable trace after a crash. It is identified only if the driver admits to it or an investigator infers it, and a driver who has died cannot be interviewed. Research using other methods suggests that the true toll is much higher. A 2014 analysis by Brian Tefft for the AAA Foundation for Traffic Safety, using a national sample of crashes investigated in detail, estimated that a drowsy driver was involved in about 6 percent of crashes in which a vehicle was towed, 13 percent of crashes in which someone was hospitalized and 21 percent of crashes in which someone was killed.[2] A later AAA Foundation study, led by Justin Owens, examined in-vehicle video recorded in the moments before more than 700 real-world crashes, measuring the proportion of time drivers' eyes were closed. It found signs of drowsiness in between 8.8 and 9.5 percent of all crashes and in between 10.6 and 10.8 percent of the more serious crashes that would be reported to police, far above the share identified in official statistics.[3] The exact number is uncertain, but the pattern is not. Drowsy driving is a leading contributor to serious and fatal crashes, and it is badly undercounted. Why drowsy crashes are so dangerous Drowsy driving crashes have a distinctive profile. According to NHTSA, they occur most often between midnight and 6 a.m. or in the late afternoon, frequently involve a single driver with no passengers, often happen on rural roads and highways and typically show no evidence of braking before the crash.[1] Every element of that profile is relevant to evening students. The late afternoon is the time of the post-lunch dip described in Chapter 1, and it is when many students are driving from work to campus. Late evening is not quite the midnight to 6 a.m. peak, but for a student who has been awake since 5:30 in the morning, 10 p.m. represents more than sixteen hours of wakefulness, and a longer commute or a later class pushes the drive into the highest-risk window. Students almost always drive home alone. And many community colleges and commuter campuses are located on the edges of towns, with a significant part of the trip home on highways or rural roads. The absence of braking is the most frightening part of the profile. It reflects the fact that a drowsy driver who crashes has often, for a few seconds, simply stopped driving. In a microsleep, the brain briefly disengages from the outside world. The driver does not see the curve, the stopped car or the edge of the road and so does not react to it. Unlike a distracted driver who glances up and swerves, a sleeping driver may hit an obstacle at full speed. That is one reason why drowsy crashes are disproportionately severe. It is often said that driving drowsy is like driving drunk, and there is truth in the comparison. The National Institute for Occupational Safety and Health, drawing on the research literature, notes that being awake for 17 hours produces impairment similar to that of a blood alcohol concentration of 0.05 percent, and that being awake for 24 hours is similar to a blood alcohol concentration of 0.10 percent, above the legal limit for driving in every U.S. state.[4] This line of work traces back to a well-known 1997 study in Nature by Drew Dawson and Kathryn Reid, which compared the performance effects of sustained wakefulness with those of alcohol and concluded that moderate levels of fatigue produced impairment greater than the legally prescribed level of alcohol intoxication in many countries.[5] Consider what this means in practice. A student who wakes at 5:30 a.m., works a full day and drives home from a class that ends at 9:45 p.m. will have been awake for more than sixteen hours by the time they reach the car and seventeen by the time they get home. If they slept poorly the night before, or have been getting six hours a night all week, the effective impairment may be considerably greater. The same student would never consider driving after a few drinks. The biological reality is not so different. NIOSH adds a crucial caution: combining even a small amount of alcohol with extended wakefulness makes performance worse still.[4] A single beer with dinner before class, or a glass of wine at a work event beforehand, is a far larger risk for a student who will drive home late than it would be for someone driving home at six after a normal day. Knowing when you are too tired The central difficulty with drowsy driving is that people are poor judges of their own impairment. The Van Dongen study described in Chapter 2 found that people restricted to six hours of sleep a night for two weeks became steadily more impaired while reporting only modest increases in sleepiness. Many drivers who have had a drowsy crash or near miss report that they did not feel dangerously tired until the moment it happened. That does not mean that the warning signs are invisible, only that you have to look for them deliberately rather than waiting for an overwhelming feeling of tiredness. Research from the Loughborough University sleep laboratory found that drivers in a simulator generally did acknowledge feeling sleepy before their driving deteriorated, which suggests that noticing and acting on sleepiness is possible.[6] The problem is less that drivers do not feel sleepy than that they talk themselves out of stopping. Table 4 lists common warning signs, drawn from NHTSA and NIOSH guidance, and what each should prompt you to do.[1][4] The rule is simple: if you notice any of them, treat it as a signal to stop driving as soon as you safely can, not as a signal to try harder. Table 4. Drowsy driving warning signs and what to do. Warning sign What it means What to do Frequent yawning, heavy eyelids, rubbing your eyes Sleep pressure is high and rising Plan a stop; do not start a long drive this way Drifting from your lane or hitting a rumble strip Attention is lapsing Pull over at the next safe place Missing an exit or a sign you normally notice Brief lapses are occurring Pull over at the next safe place No memory of the last few miles Possible microsleeps Stop driving now, as soon as it is safe Head nodding or jerking upright You are falling asleep Stop driving now, as soon as it is safe Before you even start the car, ask yourself a few honest questions. How long have I been awake? How much did I sleep last night, and how much this week? Did I struggle to stay awake in class? Have I had any alcohol or medication that could make me drowsy? If the answers worry you, the decision about how to get home should be made in the parking lot, not on the highway. What works and what does not When drivers feel drowsy, they tend to reach for a familiar set of remedies: turning up the radio, opening the window, turning the air conditioning to cold, singing, slapping their own cheeks, chewing gum. These strategies share a feature: they may make you feel more alert for a few minutes without substantially reducing your actual risk. The same Loughborough researchers tested two of these strategies directly. In a simulator study published in 1998, Louise Reyner and Jim Horne had sleep-restricted young drivers drive for two and a half hours in monotonous conditions, sometimes with cold air blowing on their faces and sometimes with the radio playing. Neither had a significant effect on lane-drifting incidents overall. The radio reduced drivers' reported sleepiness for much of the drive, but the effects of cold air were only transient, and the authors concluded that these measures were at best temporary expedients, perhaps enough to help a driver reach a safe place to stop.[7] This is a sobering finding, because it means that the radio can make you feel better without making you safer. What does work, in the short term, is stopping and addressing the sleepiness itself. In an earlier study, Horne and Reyner compared three things sleepy drivers might do during a thirty-minute rest break between two monotonous simulated drives: take a nap of less than fifteen minutes, drink coffee containing 150 milligrams of caffeine or drink decaffeinated coffee as a placebo. Both the nap and the caffeine significantly reduced driving impairment and sleepiness. Simply taking a break, without either, proved ineffective.[6] NHTSA's guidance reflects this evidence. It advises that the only true protection against drowsy driving is getting adequate sleep, and it suggests that drivers who feel drowsy pull over at a safe place and consider a combination of caffeine and a short nap of about twenty minutes, noting that coffee alone is not enough for a driver who is seriously sleep-deprived.[1] NIOSH's guidance for shift workers similarly recommends pulling over and napping for fifteen to twenty minutes, and notes that because caffeine takes about thirty minutes to begin working, drinking it just before a short nap can mean that it takes effect around the time you wake.[4] Two caveats matter. First, these countermeasures are temporary. They can buy you some alertness for a limited time, enough, perhaps, to finish a modest drive home safely, but they do not remove the underlying sleep debt, and a driver who uses them and then drives for another two hours on a dark highway is running out of margin. Second, the stop itself has to be safe. Pulling onto the shoulder of a highway to nap is dangerous. Choose a well-lit rest area, a busy gas station or a campus lot you know, lock the doors and set an alarm. Alternatives to driving tired The best strategy for drowsy driving is not a trick for getting through the drive but a plan that avoids the drive when you are too tired to make it safely. For evening students, this requires some advance thought, because the options that exist at ten at night are fewer than those at five in the afternoon. Carpool with a classmate. Sharing a drive with another student from the same class means that one of you can drive while the other stays alert and talks, and that you can swap if one of you becomes too tired. It also halves the number of nights each person drives, if you alternate. Many evening classes have students who live in the same area; asking at the first session is easy. Arrange a backup ride. Before the term starts, identify someone who could pick you up on a night when you are too tired to drive safely: a partner, a family member, a friend. Agree on it explicitly, so that when the night comes, the call is a plan rather than a favor. Keep enough money or credit available for a taxi or rideshare in case your backup is unavailable. Nap before you drive. If you feel too drowsy to drive at the end of class, a twenty-minute nap in a locked car in a well-lit, busy part of a campus lot, possibly combined with caffeine, may make the drive home safer. It is not a solution for a long drive after severe sleep loss, and it has costs for your sleep that night, but it is far better than driving drowsy. Stay over. For students with long commutes, particularly those who drive an hour or more on rural roads, it may occasionally make sense to stay nearby, with a friend or relative close to campus, on nights when fatigue is severe. Change the schedule. If you find that you regularly finish class too tired to drive safely, that is not a problem to solve with coffee. It is a sign that your overall schedule, the combination of work hours, class times, commute length and sleep, is not sustainable, and the backward plan in Chapter 2 should be revisited. Options may include a later start at work on the morning after class, taking fewer evening classes in a term, choosing online or hybrid sections for some courses or moving closer to campus. None of these is easy, but they are far easier than the alternative. Before you drive, not only during Most of the work of avoiding drowsy driving happens long before the drive itself. The student who consistently sleeps seven or more hours, keeps caffeine to the afternoon, eats a sensible dinner before class and moves during the break arrives at the car in a far better state than one who has done none of those things. In that sense, the whole of this book is about the drive home. A few specific preparations also help. Check your medications, including over-the-counter antihistamines, sleep aids and some cold remedies, for warnings about drowsiness and operating vehicles; if something you take regularly carries such a warning, talk to your pharmacist or doctor about timing or alternatives. Avoid alcohol entirely before a late class if you will be driving afterward. Be especially careful in the week after the change to or from daylight saving time, and in the first weeks of a term, when schedules are new and sleep is often disrupted. A note for the people who love evening students If you share a home with an evening student, you can help in ways that matter. Offer to pick them up on nights when they seem exhausted, without making them feel judged for needing it. Keep your phone on when they are driving home. If they call and say they are pulling over to rest, tell them that is the right decision. And if you notice that they are consistently arriving home drained, help them think through whether their schedule can be changed. Drowsy driving is the one part of the evening student's day where the consequences of a bad night can be permanent. Everything else in this book is about feeling better and learning more. This chapter is about getting home. Chapter 7: Walking, Waiting and Riding After Dark Not every evening student drives home. Many take the bus or the train, some walk or cycle, and nearly all of them, even those with cars, spend part of the evening on foot: crossing a parking lot, walking from a classroom building to a garage, waiting at a stop, walking the last few blocks from a station to their front door. These moments rarely get the attention that the drive home deserves, but they carry their own risks, and those risks are shaped by the same two facts that shape everything else in this book. It is dark, and you are tired. This chapter covers the practical side of getting home safely when you are not behind the wheel: traffic safety for pedestrians and cyclists, personal safety on campus and in transit and the resources most colleges provide but many evening students never use. The traffic risk is the larger one When people think about safety on the way home at night, they usually think first about crime. It is a legitimate concern, and later sections of this chapter address it. But for most people walking at night in the United States, the greater statistical danger is traffic. NHTSA's figures for 2022 show that 7,522 pedestrians were killed in traffic crashes in the United States that year.[1] The details, summarized in Table 5, describe a problem concentrated in exactly the conditions evening students encounter. Table 5. Pedestrian deaths in U.S. traffic crashes, 2022. Factor Share of pedestrian deaths Occurred in the dark 78% Occurred between 6:00 and 8:59 p.m. 26% Occurred in urban areas 85% Occurred away from intersections 75% Source: NHTSA, Traffic Safety Facts 2022 Data: Pedestrians (DOT HS 813 590, 2024). More than three-quarters of pedestrian deaths happen in the dark, and the three hours from 6:00 to 9:00 in the evening, when many evening classes are in session or just ending, account for about a quarter of the total, more than any other three-hour period. Most deaths happen in urban areas, and most happen away from intersections, where drivers are least likely to expect someone crossing. NHTSA's review of pedestrian safety countermeasures also notes that nighttime pedestrian fatalities rose much faster than daytime deaths over the decade from 2010 to 2019.[2] The reasons are not mysterious. Drivers see pedestrians much later at night, especially pedestrians in dark clothing on poorly lit roads. Drivers are also tired in the evening, and some are impaired. And pedestrians themselves, tired after a long day, are more likely to cross mid-block, to misjudge a gap in traffic or to be looking at a phone. Being seen, and seeing The single most effective thing a pedestrian can do at night is to be visible. Many people greatly overestimate how visible they are to drivers after dark. From the sidewalk, you can see a car's headlights from far away, and it is easy to assume the driver can see you just as clearly. But a person in dark clothing can be very hard to pick out against a dark background until a car is close, especially if the driver is dazzled by oncoming lights or distracted. Some simple measures make a large difference. Wear or carry something light-colored, and ideally something retroreflective: a reflective band on a bag or ankle, a reflective strip on a jacket, a small clip-on light. Reflective material shines back toward the source of light, which means that it lights up in a driver's headlights in a way that ordinary light-colored fabric does not. Keeping a reflective vest or band in your bag, to put on for the walk to the bus or across a dark campus road, is a small habit that can matter a great deal. Cross at marked crossings and intersections wherever possible, and where there are signals, use them, even if it means a longer walk. Make eye contact with drivers before stepping in front of a car, and do not assume that because you can see them, they can see you. Walk facing traffic if there is no sidewalk, so that you can see approaching vehicles. Put the phone away while crossing streets and walking through parking lots, which are full of cars reversing out of spaces and drivers looking for exits rather than pedestrians. Fatigue matters here too. The same lapses of attention that make drowsy driving dangerous make tired walking riskier, and a student who has struggled to stay awake through the last hour of class is more likely to step off a curb without looking properly. If you notice that you are tired, slow down and be deliberate about crossings. Alcohol deserves a mention. Impaired walking is a real contributor to pedestrian deaths, and a student who has a drink after class with classmates before walking to a station is taking on more risk than they may realize. Cycling at night For students who cycle to evening classes, visibility is even more critical. Many states require bicycles ridden at night to have a white front light and a red rear reflector or light, and in any case these are the minimum sensible equipment. A bright rear light, ideally one that can flash, and reflective material on the pedals, wheels or ankles make a cyclist much easier to see. A helmet is essential. Plan a route that uses bike lanes or quieter, well-lit streets, even if it is slightly longer, and ride predictably. If you are very tired after class, it may be safer to walk the bike across complex intersections or, if transit allows bikes, take part of the trip by bus. Personal safety: an honest perspective Personal safety after dark is a subject that can easily slide into fear, and fear is not useful. Most evening students will never experience a crime on their way home. Campuses are, on the whole, relatively safe places. At the same time, evening students are often moving through campuses and neighborhoods at hours when they are quieter, alone and tired, and some sensible habits can reduce the already modest risk further. The most reliable of these habits have less to do with self-defense than with planning and awareness. Know your route before you need it, including where the well-lit paths are, where the emergency phones are and where there are likely to be other people. Park in a well-lit area close to the building you will leave from, not the building you arrive at, since arriving in daylight and leaving in darkness can make a spot that seemed fine at five feel very different at ten. Walk with a classmate when you can; the simplest safety arrangement is to leave with someone going the same way. When walking alone, stay alert to your surroundings. Headphones at high volume and absorption in a phone make it harder to notice what is happening around you. Have your keys ready before you reach your car or front door, so that you are not standing at a door searching through a bag. Glance into the back seat and around the car before getting in, and lock the doors once you are inside. None of this needs to become a source of anxiety; these are simply the habits of people who move around at night regularly. Trust your instincts. If something about a situation feels wrong, whether it is a person, a place or a sense of being followed, act on the feeling without waiting for proof. Go back into a building, move toward other people, call campus security or cross to the other side of the street. The social awkwardness of being wrong is a trivial price. What your campus provides Most colleges offer safety resources designed for exactly the situations evening students face, and it is striking how many evening students never learn about them. Spend fifteen minutes in the first week of term finding out what yours provides. The details vary, but common resources include the following. Escort services. Many campuses run safety escort programs, sometimes called safe walk or safe ride, in which a campus security officer, trained student staff member or volunteer will walk or drive you from a building to your car, a transit stop or another campus location after dark. Some services run on a schedule; others respond on request by phone or app. Escort programs exist to be used, and requesting one is not an admission of fear. It is a sensible choice, particularly for students leaving late from a part of campus that is quiet at night. Emergency phones. Many campuses have emergency call stations, often marked by blue lights, that connect directly to campus police or security. Know where they are on your usual route. Campus police or security. Save the campus police or security number, both emergency and non-emergency, in your phone. For life-threatening emergencies, call 911; but for situations like feeling uneasy about someone in a parking lot, a campus security line may be the fastest way to get someone on the scene. Safety apps and alerts. Many colleges use apps or text systems that let students contact security, share their location with a friend or security staff during a walk and receive alerts about emergencies. Sign up for emergency notifications as soon as you enroll. Shuttles. Some campuses run evening shuttles between buildings, parking areas and nearby transit stations, which can remove the least comfortable walks from your route entirely. It is also worth knowing what information your college is required to share. Under a federal law known as the Clery Act, colleges that take part in federal student aid programs must publish an annual security report, including crime statistics for the campus and certain nearby areas and a description of their safety policies; issue timely warnings about crimes that pose a continuing threat; and, for those with campus police or security departments, maintain a daily crime log available to the public.[3] Evening students can use these documents to understand where and when incidents have occurred on their campus and what the college does about them. Riding transit at night For students who use buses or trains, the ride home has its own logistics. Service is often less frequent in the evening, so a missed connection can mean a long wait. Check schedules in advance, and use real-time arrival apps where available so that you can wait inside a building rather than at an exposed stop. Where possible, choose stops that are well lit and busy, even if it means a slightly longer walk. On the vehicle, sit near the driver or in a car with other passengers. If someone makes you uncomfortable, move, and tell the driver or transit staff if needed. Many transit systems have text or app-based reporting for harassment and safety concerns; find out whether yours does. Fatigue is a hazard on transit too, though of a different kind. Falling asleep and missing your stop is a common experience for tired evening students, and in some places it can leave you at the end of a line late at night, far from home. Setting a phone alarm for a few minutes before your stop, or a location-based reminder if your phone supports one, is a simple safeguard. The walk from the final stop to your door deserves the same attention as the walk on campus. Know the route, keep it well lit where possible and consider asking someone at home to watch for you or to walk to meet you on nights when you feel uneasy. The practical kit A handful of small items, kept permanently in a bag or car, handle most of the minor emergencies that can turn a late journey into a stressful one. A phone charger or small battery pack matters more than people expect: a phone with no charge at 10 p.m. cannot summon an escort, check a bus time, share a location or call for help. A small flashlight, or at least familiarity with the phone's light, helps in dark parking lots and on unlit paths. A reflective band or vest takes up almost no space. Some cash or a transit card with credit on it covers the night when the card reader fails or a rideshare is needed. For drivers, a membership in a roadside assistance program, or at least the number of a towing service, a basic emergency kit and a habit of not letting the fuel tank run low before class nights, prevent the particular misery of a breakdown on a dark road after a long day. It is also worth having a short list of numbers stored in your phone and, ideally, written on a card in your wallet in case the phone is lost or dead: campus security, a backup ride, a taxi company and someone at home. If something does happen Despite every precaution, things sometimes go wrong. If you are in immediate danger or witness a crime in progress, call 911, or use a campus emergency phone, and move toward light and other people. If you are able, note details such as descriptions, vehicles and the direction someone went, but your safety comes before any of that. After an incident, even one that ends without physical harm, it is common to feel shaken for days or weeks, and evening students often try to push through without telling anyone, because they have work the next day and class the next week. Reporting to campus police or security is your choice in most situations, but doing so can help the college warn others and, where relevant, investigate. Colleges also have staff whose job is to support students after harassment, assault or other crimes, whether or not the student makes a formal report. These may include counseling services, victim advocates and the office responsible for handling reports of sexual harassment and violence. They can help with practical issues too, such as changing class sections or arranging an escort for the rest of the term. For anyone affected by sexual violence, the National Sexual Assault Hotline, run by RAINN, offers free, confidential support at any hour at 1-800-656-4673. For anyone whose distress becomes overwhelming, including thoughts of self-harm, the 988 Suicide and Crisis Lifeline is available by call or text at 988. Letting someone know One habit ties all of this together: let someone know when you leave and when you arrive. A brief text to a partner, family member or friend saying you are leaving campus, followed by another when you get home, creates a simple safety net. If something goes wrong, someone will know sooner. Many phones now offer location-sharing features that do this automatically for a set period; used with care and consent, they can reduce both risk and worry for the people waiting at home. Planning the journey as part of the evening In the backward plan from Chapter 2, the trip home was one of the fixed points, and this chapter adds detail to it. For a driver, the plan is about alertness. For a pedestrian, cyclist or transit rider, it is about visibility, awareness and logistics: knowing the route, the schedule and the resources available and deciding in advance how you will handle a night when you are more tired than usual or a situation feels wrong. None of this should make the evening journey feel frightening. Most nights, it will be uneventful: a walk to the car, a bus ride, a text to say you are home. The point of preparing is to keep it that way. Chapter 8: Landing the Evening The front door closes behind you at 10:05. You have been awake for more than sixteen hours. You are tired in the bone-deep way that comes from a full day of work followed by three hours of concentration. And yet, as many evening students discover, you are not ready to sleep. Your mind is still running through the lecture, the problem set due next week, the email from your manager, the permission slip your daughter needs signed. Your body feels exhausted, but your brain feels switched on. This is one of the most common complaints of evening students, and it is the part of the evening they are least likely to plan for. The class has a start and end time. The drive home has a route. But the hour between walking in the door and falling asleep is usually left to chance, and chance tends to fill it with a phone, a snack, a television and a bedtime that slides steadily later. This chapter is about that hour: why it is hard, and how to use it well. Why you feel wired Several things combine to make falling asleep after a late class harder than it should be. The first is simple mental activation. Sustained concentration on new and challenging material is stimulating, and the brain does not switch off the moment the lecture ends. Many students replay what they learned, worry about what they did not understand or start mentally planning the work to come. This kind of cognitive arousal is one of the main contributors to difficulty falling asleep. The second is light. Chapter 1 explained that bright light in the evening signals the circadian clock that it is still daytime, suppresses melatonin and can push the clock later. Most students spend the evening in brightly lit classrooms, then drive home through lit streets, then walk into a lit house and pick up a lit phone. By the time they try to sleep, their bodies have received a strong and continuous daytime signal. The third is whatever caffeine is still circulating. As Chapter 4 showed, a coffee at five in the afternoon can leave the equivalent of a small cup still active at eleven. Students who drink caffeine at the class break are often carrying more than that. The fourth, for many, is the transition itself. Arriving home, particularly to a household with children, a partner or other responsibilities, often means stepping straight from one role into another: from student to parent, from student to partner, from student to the person who notices that the dishes are not done. The emotional shift can be as stimulating as the lecture. None of these is a sign that something is wrong. They are the predictable effects of the schedule. The answer is to build a deliberate transition, a wind-down period that takes each of them in turn and gives your mind and body the signals they need to shift toward sleep. The wind-down window In the backward plan from Chapter 2, the wind-down was a period of roughly thirty to sixty minutes before lights out. What goes in that window matters less than that it is consistent, calm and different from what came before. Over time, a consistent routine becomes a signal in itself: the brain learns that these steps lead to sleep. Dim the lights. As soon as you get home on a class night, lower the lighting. Use lamps rather than overhead lights, and warmer, dimmer bulbs where possible. This is one of the simplest and most effective changes, and it directly counters the bright-light signal of the evening. Manage screens. The evidence here is worth knowing in some detail. In a carefully controlled laboratory study published in 2015, Anne-Marie Chang, Charles Czeisler and colleagues at Brigham and Women's Hospital compared reading on a light-emitting e-reader with reading a printed book in the hours before bedtime. When participants read on the light-emitting device, they took longer to fall asleep, felt less sleepy in the evening, secreted less melatonin, showed a later timing of their circadian clock and were less alert the next morning.[1] The authors noted that a national survey had found that 90 percent of Americans used some kind of electronic device at least a few nights a week within an hour of bedtime. Screens also affect sleep through what is on them. Email, news, social media and messages are designed to be engaging, and they often raise exactly the kind of mental activation that makes sleep harder. A realistic goal for most students is not to abolish screens in the evening but to put a boundary around them: no work email after class, no social media in bed and the phone charging somewhere other than the bedside for the last half hour before lights out. Night-mode settings that warm the screen color may reduce some of the light's effect, but they do not address the stimulation of the content. Close the loops. Much of the mental activity that keeps evening students awake is about unfinished business. A simple, well-tested technique is to write it down. In a 2018 study, Michael Scullin and colleagues at Baylor University asked young adults to spend five minutes before bed either writing a to-do list of tasks they needed to complete in the coming days or writing about tasks they had already completed. Those who wrote to-do lists fell asleep significantly faster, and the more specific their lists, the faster they fell asleep.[2] For an evening student, this can become a fixed part of the wind-down: five minutes with a notebook, writing down what needs to happen tomorrow and when. The act of writing it down seems to allow the mind to stop holding it. Do something calm. Reading a printed book or magazine, taking a warm shower, gentle stretching, listening to quiet music or a podcast that is interesting but not gripping, a few minutes of slow breathing or a brief conversation with someone at home can all serve. What matters is that it is calm, pleasant and not work. Keep food light. If you are hungry, have a small snack, not a meal, as discussed in Chapter 3. A piece of fruit, a small yogurt, a slice of toast or a handful of nuts will settle hunger without loading your stomach before sleep. The bedroom itself The room you sleep in matters too, and for evening students it often needs a little more attention than for people who go to bed earlier in a quieter house. Sleep specialists commonly recommend a bedroom that is dark, quiet and comfortably cool, and each of these can be harder to achieve at eleven at night in a busy household or an apartment building. Darkness is the easiest to fix. Blackout curtains or blinds shut out streetlights, and an eye mask costs very little and works almost anywhere. Cover or turn away bright standby lights on chargers and electronics. If you need to get up in the night, a small, dim night light is better than switching on the overhead light. Noise is harder, particularly for students who come home to a household that is still awake: a partner watching television, older children, roommates. Earplugs help many people, and a fan or a white-noise machine can mask intermittent sounds such as voices, traffic or doors, which tend to be more disruptive than steady ones. It is also reasonable to ask the people you live with for some quiet after a certain hour on class nights; the next chapter discusses how to have that conversation. Temperature varies more between individuals, but most people sleep better in a room that is slightly cooler than they would choose for sitting around in the evening. If you share a bed with a partner who runs warmer or cooler, separate blankets can solve more arguments than a thermostat. For students whose partners go to bed hours earlier and are woken when they come in, a few practical arrangements help: getting ready for bed in another room, keeping clothes for the morning laid out elsewhere and using a small light rather than the main one. Some couples find that on class nights, the late partner sleeping in another room altogether is the kindest arrangement for both. There is no need to treat this as a sign of a problem in the relationship; it is logistics. Alcohol is not a sleep aid Many adults use a drink in the evening to unwind, and after a long day and a demanding class, a glass of wine or a beer can seem like a reasonable reward. It is worth understanding what alcohol actually does to sleep. A 2013 review of the research by Irshaad Ebrahim, Colin Shapiro and colleagues found that alcohol shortens the time it takes to fall asleep and, at higher doses, increases deep sleep in the first half of the night, but that it increases sleep disruption in the second half of the night and, at moderate and high doses, reduces the total proportion of dreaming, or REM, sleep.[3] The authors concluded that alcohol should not be used as a sleep aid. In other words, alcohol helps you fall asleep faster and then makes the rest of your night worse. For an evening student already short of sleep, that is a poor trade. If you enjoy a drink, the best place for it is on an evening without class, well before bed and never before driving. When sleep will not come Even with a good wind-down, some nights you will lie awake. Sleep specialists generally advise against lying in bed for long periods trying to force sleep, because the bed can gradually become associated with frustration and wakefulness. If you find yourself awake for what feels like more than twenty minutes or so, get up, go to another room with dim light and do something quiet and boring until you feel sleepy, then return to bed. Do not check the time repeatedly; clock-watching tends to increase anxiety about sleep. Occasional nights like this are normal. If they become the pattern, if you regularly lie awake for long periods, wake frequently or wake too early and cannot get back to sleep, and it has been going on for weeks, talk to a clinician. Chronic insomnia is common, and the treatment recommended as first-line by major medical bodies is not medication but cognitive behavioral therapy for insomnia, a structured program that addresses the thoughts and habits that keep insomnia going. It is available through sleep clinics, some therapists and, increasingly, through validated digital programs. A caution about sleep aids Some evening students turn to sleep aids, either over-the-counter products or prescriptions, to force sleep after a late class. This deserves caution for a reason that connects directly to the rest of this book: some sleep medications can leave people impaired the next morning, including while driving. In 2013, the U.S. Food and Drug Administration required lower recommended doses for certain insomnia drugs containing zolpidem, because of evidence that blood levels in some patients, particularly women, remained high enough the next morning to impair activities requiring alertness, such as driving.[4] Many over-the-counter sleep aids contain sedating antihistamines, which can also cause next-day drowsiness. Melatonin is widely sold as a dietary supplement in the United States, which means it is not regulated in the same way as a medicine, and the dose and timing that might be appropriate vary considerably between people and purposes. If you are considering any sleep aid, including melatonin, discuss it with a clinician or pharmacist, especially if you drive early in the morning. The morning after Landing the evening well includes the next morning. On the day after a late class, a few habits help keep the circadian clock anchored and the sleep debt from compounding. Keep a regular wake time. It is tempting to sleep in after a late night if your schedule allows, and occasionally that is reasonable. But a large shift in wake time pushes the clock later, making the next night's sleep harder. If you need more sleep, a modest adjustment of an hour or less is usually better than a large one, and an early-afternoon short nap is often better still. Get morning light. Outdoor light in the first hour or two after waking is the most powerful signal the clock receives to stay on an earlier schedule. A short walk, a coffee by a window, or getting off the bus a stop early can all provide it. This is the counterweight to the bright evening light of class. Use caffeine in the morning. The morning after a late class is exactly when caffeine is most useful and least costly. It can help with the grogginess of a short night, and it will be long cleared by the following evening. Be kind to yourself. The morning after a class night is often the hardest part of the week. It helps to schedule less demanding tasks for it where you can, and to accept that you may not be at your best. Three wind-downs What a good wind-down looks like depends heavily on who is waiting at home. Three illustrative examples show how the same principles adapt to different households. Consider first a student who lives alone. She arrives home at 10:05 and her anchor is 11:30. She switches on two lamps rather than the ceiling light, plugs her phone in on the kitchen counter and puts the kettle on for a caffeine-free herbal tea. While it steeps, she writes a short list of what needs to happen tomorrow, including the one reading she wants to review at lunch. She takes a shower, reads a novel in bed for twenty minutes and turns out the light at 11:05. Nothing in the sequence is special, which is the point; after a few weeks, it runs almost automatically. Now consider a father of two young children whose partner has handled bedtime. He gets home at 9:50. The children are asleep, but his partner has been waiting to talk, and he wants to hear about the evening. They sit together in the kitchen with the lights low for twenty minutes, deliberately keeping the conversation to the day rather than to the family budget or next week's scheduling problems, which they have agreed to save for the weekend. He writes his to-do list while his partner gets ready for bed, then reads for fifteen minutes. The conversation is part of his wind-down rather than an obstacle to it, because they have agreed what kind of conversation it is. Finally, consider a student who shares an apartment with two roommates who are still up and watching a film when she gets home at 10:30, with an anchor of midnight. She says hello, eats a yogurt in the kitchen rather than joining them, and goes to her room. She uses a fan for background noise and wears an eye mask because light shows under the door. Her wind-down is shorter than she would like, but it still has the same elements: dim light, the list, a calm activity and a consistent lights-out. She has asked her roommates to keep the volume down after midnight on weeknights, and on most nights they do. In each case the routine is modest, and none of them follows it perfectly every week. What they share is that the last hour has a shape, decided in advance, rather than being whatever happens to fill it. The evening, reconsidered Most evening students think of their class night as ending when class ends. In practice, the night ends when you fall asleep, and the last hour matters as much as any other. A student who arrives home, dims the lights, writes tomorrow's list, eats a small snack if needed, reads for twenty minutes and turns off the light at a consistent time is giving themselves a very different night from one who scrolls on the couch until 12:30 with a bowl of leftovers and a glass of wine. The difference is not willpower. It is having decided in advance what the last hour will look like. The next chapter turns to the people who share that last hour: the families, partners and employers who surround every evening student. Chapter 9: The People at Home and at Work Every evening student lives inside a web of other people's schedules. A class from six to nine is not only three hours of the student's time; it is three hours in which someone else is putting the children to bed, or the dog is not walked, or a parent is not visited, or a shift cannot be covered. The morning after is not only the student's hard morning; it is the morning their manager expects them at their desk at eight and their partner expects them to take the children to school. Wellness advice that treats the student as an isolated individual, free to set their own bedtime and meal schedule, misses most of what makes these habits hard to keep. The Lumina Foundation reports that roughly four in five college students work, about 30 percent of them full time, and that close to one in five has children or other dependents.[1] Among evening students those proportions are, by the nature of the schedule, often higher. For them, the question is not only what to do, but how to arrange the rest of life so that doing it is possible. This chapter is about those arrangements: at home, at work and with the college itself. The conversation at home The single most useful thing many evening students can do for their health during a term is to have one clear conversation with the people they live with before it starts. Not a vague announcement that "school is going to be busy," but a specific discussion of what class nights will look like and what each person needs. It helps to bring the backward plan from Chapter 2 to that conversation. If you can say, "On Tuesdays and Thursdays I'll be home at about ten, and I need to be asleep by 11:30, so I need the time between ten and eleven to be quiet," the people you live with have something concrete to work with. They can see what you need and where they fit. Vague requests tend to produce vague cooperation; specific ones are much easier to honor. Several topics are worth covering. Who handles dinner, children's bedtimes, homework help, pets and household tasks on class nights? What is the plan when a child is sick on a class night? Who can pick you up if you are too tired to drive, as discussed in Chapter 6? What does the household need from you on the nights you are home, and on weekends, so that the load does not simply shift permanently onto someone else? How will you stay connected, since a partner who has handled every Tuesday and Thursday evening alone for fifteen weeks may feel something closer to loneliness than resentment? That last question is easy to overlook. An evening program changes the rhythm of a household in ways that affect everyone. Partners may feel they have become single parents two nights a week. Children may miss bedtime stories. Older parents who rely on a student for company or care may feel neglected. Acknowledging these costs openly, and agreeing on ways to protect time together, such as a weekly meal, a regular Saturday morning, or a phone call from the car before class, helps prevent the slow build-up of friction that can undermine both the relationship and the student's ability to keep going. Children and the evening student Student parents face some of the hardest scheduling problems in higher education. Child care that covers evening hours is often difficult to find and expensive, and the evening class may fall exactly across dinner, bath and bedtime, the most labor-intensive part of a young child's day. Some colleges operate campus child care centers, and some of those offer evening care or drop-in options for students; a federal program called Child Care Access Means Parents in School helps fund campus-based child care for low-income student parents at participating institutions. It is worth asking your college's student services office what is available, since these programs are often underused simply because students do not know they exist. Community organizations, family members, other student parents and neighbors who can trade care are other possibilities. When children are old enough to understand, involving them in the plan can help. A child who knows that Mom has class on Tuesdays, that Grandma will do bedtime and that Mom will be there for breakfast often copes better than one for whom the absence is unexplained. Some student parents find that sharing a little of what they are learning, or letting children see them study, turns the class into something the family is doing together rather than something taking a parent away. Many adult students cite their children as a reason for returning to school; letting children see that effort can be a quiet form of teaching. Student parents also need to be realistic about their own sleep. A parent of a baby or toddler who wakes in the night is carrying a sleep debt before any class begins, and the advice in this book about caffeine timing, drowsy driving and protecting sleep applies with extra force. If night wakings are shared with a partner, it is reasonable to ask that the class-night parent be spared them on those nights, and to repay the favor on other nights. The conversation at work For students who work, the workplace can either support or undermine every habit in this book. The morning after a late class is the pinch point of the week, and whether an employer allows even a small amount of flexibility around it can make the difference between a sustainable schedule and an exhausting one. It is worth considering carefully whether and how to raise your studies at work. Many employers are supportive, and some offer formal tuition assistance, flexible scheduling or study leave; others may be indifferent or, occasionally, see studies as a sign that an employee plans to leave. You know your own workplace best. But where it is reasonably safe to do so, a straightforward conversation with a manager can produce practical help: a later start on the morning after class, not being scheduled for late shifts on class days, the ability to leave on time on class afternoons rather than being asked to stay for a late meeting, or a quiet space to eat before leaving. As with the conversation at home, specific requests work better than general ones. "Could I start at 9:00 rather than 8:00 on Wednesdays, and make up the hour on Friday?" is easier for a manager to say yes to than "I'm going to be really busy this term." Framing the request around your work performance, such as being sharper on the morning after class, can also help, and it has the virtue of being true. For students who work shifts, particularly rotating or night shifts, the challenges are greater. Shift work itself disrupts the circadian clock, and combining it with evening classes can leave very little room for regular sleep. If this describes your situation, prioritize a consistent sleep period as much as your rota allows, be especially vigilant about drowsy driving and consider discussing your schedule with a clinician, because the health effects of combining shift work with study can accumulate quietly over a term. The college as a partner Colleges that serve large numbers of evening students increasingly recognize that those students need services at times they can actually use them. Advising, tutoring, counseling, financial aid, the library and health services may offer evening hours, online appointments or both. Many evening students assume these services are only for daytime students and never ask. Ask. Several college resources are particularly relevant to the themes of this book. Counseling services can help with the stress, anxiety and low mood that often accompany a heavy load. Health services may be able to assess persistent fatigue or sleep problems. Disability services can arrange accommodations for health conditions that affect energy, alertness or scheduling. Basic needs programs, including food pantries, emergency grants and help with public benefits, address the practical pressures that undermine wellness, as Chapter 3 discussed. Campus safety offices provide escort services and safety information, as Chapter 7 described. Instructors, too, are partners. Most evening instructors understand the lives of their students and will respond reasonably to a respectful conversation about a problem: an occasional late arrival because of a shift change, a missed class because of a sick child or a request for recordings or slides. Telling an instructor early that you are juggling work and family, before a crisis, tends to produce more flexibility than explaining it after one. Knowing your limit Perhaps the most important decision an evening student makes about wellness happens before a term begins: how many courses to take. The pressure to finish quickly is real. Tuition costs, the desire for a better job and the sense that life is on hold until the degree is done all push toward a heavier load. But the arithmetic of the backward plan applies at the level of the term as well as the evening. Each additional evening class is another night with late caffeine temptations, a displaced dinner, a late drive and a short sleep, and another morning after. There is no universal right number. A student with a flexible job, supportive family and short commute may manage three evening classes comfortably; another with a long commute, young children and a physically demanding job may find that one is the limit. What matters is making the decision deliberately, with an honest look at the whole week, rather than by default. Finishing a program a term or two later, in good health, is a better outcome than burning out halfway through and withdrawing. If you find partway through a term that the load is unsustainable, talk to an adviser before the withdrawal deadline. Dropping one course to protect the others, and your health, is a legitimate choice, not a failure. When the plan meets a bad week Every term contains at least one week when everything goes wrong at once: a child with a fever on a class night, a mandatory late meeting, a parent in the hospital, a car in the shop, an exam the next morning. The habits in this book tend to collapse in exactly those weeks, and that is when they matter most, because a week of crisis is also a week of short sleep and high stress, often with more driving than usual. It helps to decide in advance what your fallback rules are, so that you are not making them up at eleven at night. A reasonable set might look like this. Sleep is protected before study: if something has to give, it is the reading, not the night. Driving is only done alert: if you are too tired, you use a backup ride, a nap or a later departure, whatever the inconvenience. Missing one class is recoverable: a quick message to the instructor and a classmate's notes are almost always better than attending exhausted and driving home impaired. And food is simplified rather than skipped: in a crisis week, a supermarket sandwich and a piece of fruit before class is a perfectly good dinner. It also helps to have agreed with the people around you what happens in a crisis week. Who covers child care if your usual arrangement falls through? Can a classmate share notes? Is there someone at work who can cover the morning after class if you have been up at the hospital? These arrangements are easier to set up in week one, when everyone is calm, than in week nine, when they are needed. What supporters can do This book is written mainly for students, but many of its readers will be the people around them: partners and parents, advisers and instructors, managers and college administrators. For them, the evidence in the earlier chapters points toward some concrete forms of help. Partners and family members can take on class-night tasks without keeping score, protect the quiet hour after the student gets home, offer rides on exhausting nights without judgment and keep the conversation on class nights light, saving difficult household decisions for another time. Employers can recognize the morning after class as the hardest part of the week and allow flexibility around it where the work permits, avoid scheduling late meetings or overtime on class afternoons and make tuition assistance and study-leave policies easy to find and use. Many of these cost little and are repaid in a more rested, more loyal employee. Instructors of evening classes can build in a real break, vary activities through a long session so that students are not asked to listen passively for three hours, post materials so that a moment of drowsiness does not mean lost content, and end on time, since for a student with a long drive or a late bus, ten extra minutes can mean a missed connection or a later, riskier trip home. Colleges can match their services to the hours evening students keep. That means food available on campus in the evening, not only vending machines; counseling, advising and health appointments at times working students can attend; safety escort services and shuttles that run until after the last class lets out rather than stopping before it; well-lit parking areas and paths; and orientation materials that tell evening students, specifically, what is available to them. None of these is exotic. They simply reflect the fact that a large share of students now attend at night. Stress, mood and when to get help Combining work, family and study is stressful, and some degree of stress is part of the bargain. But chronic short sleep, irregular meals and constant pressure can combine to wear down mood and resilience over a term, and evening students are often reluctant to seek help, both because of time and because they feel that struggling is simply the price of their choice. Watch for signs that stress has become something more: persistent low mood or loss of interest in things you normally enjoy, feeling hopeless, irritability that is damaging relationships, withdrawal from people, significant changes in appetite, reliance on alcohol or other substances to cope or an inability to function at work or in class. These deserve attention from a professional, whether through college counseling, your primary care provider or an employee assistance program if your employer offers one. If you ever have thoughts of harming yourself, or feel that you cannot go on, reach out immediately. In the United States, you can call or text 988 to reach the 988 Suicide and Crisis Lifeline at any hour. If you are in immediate danger, call 911. Borrowing strength Evening students often describe themselves as doing this alone: going to class alone, driving home alone, studying alone after everyone else is asleep. In practice, most of them are carried by a network of people who make it possible. Recognizing that network, asking clearly for what you need from it and thanking the people in it are not only courtesies. They are part of what makes the habits in this book sustainable. The same goes for fellow evening students. The person in the next seat is probably dealing with a version of your problems. Classmates can share rides, compare notes, form study groups that meet online at convenient times, walk together to the parking lot and simply provide the reassurance that someone else understands. The first night of class is a good time to find them. Conclusion It is tempting to read a book like this as a list: eat dinner earlier, stop caffeine by mid-afternoon, nap for twenty minutes rather than an hour, wear something reflective, dim the lights when you get home. The list is real, and every item on it is worth doing. But looked at together, the chapters point to something more useful than a list, and it is worth drawing out, because it changes how an evening student should think about the whole term rather than a single night. One decision, not many The first conclusion is that the evening student's apparently separate choices are really one choice seen from different angles. Dinner timing, caffeine timing, how you use the class break, how you get home and what you do in the last hour before bed all look like separate problems with separate solutions. They are not. Each one either protects or erodes the same thing: the night's sleep. A late coffee costs sleep through caffeine still in the blood. A late, heavy meal costs it through reflux and discomfort. A bright, screen-filled last hour costs it through a delayed clock and an activated mind. And lost sleep, in turn, makes every other choice harder the next day: you reach for more caffeine, you eat less carefully, you drive home more tired. This is why the book began with a single rule, to plan backward from the time you intend to be asleep, and why that rule turns out to be more than a scheduling trick. It is a way of seeing that the evening is a system, and that the system has one output that matters more than the others. When you choose the anchor first, most of the other decisions stop being willpower contests at the moment of temptation and become arithmetic done in advance. It also explains a pattern that many students notice once they start: changing one habit tends to make others easier. The student who moves caffeine earlier falls asleep faster, wakes less exhausted, needs less caffeine the next afternoon and finds the drive home less of a struggle. The student who eats a proper meal before class stops arriving home ravenous and so stops eating at eleven. The effects compound in both directions, which is why a modest change made consistently tends to be worth more than a dramatic one abandoned after a week. Not all compromises are equal The second conclusion concerns what to do when the plan fails, which it will. Evening students live with little slack, and no one keeps every habit every week. What the chapters make clear is that the costs of different compromises are not remotely equal, and that it pays to know the difference before the difficult night arrives. Most compromises are recoverable. A dinner eaten late, a coffee drunk after the cut-off, a night of scrolling, a skipped wind-down: each costs some sleep, some comfort, perhaps some learning. A few bad nights are absorbed by a few good ones. The body's systems for regulating sleep and appetite are resilient, and a student who slips and then returns to the plan loses little in the long run. One compromise is different in kind. Driving while drowsy carries a risk that cannot be recovered from if it goes wrong. The research in Chapter 6 on the prevalence of drowsy crashes, their severity and people's poor ability to judge their own impairment means that the ordinary logic of "just this once" does not apply. A thousand uneventful drives do not make the next one safe. This asymmetry suggests a simple ordering of priorities for a hard week. Be flexible about food. Be reasonably disciplined about caffeine, since it is the easiest lever and the quickest to repay. Protect sleep as much as the week allows. And treat the rule about driving only when alert as absolute, arranging backup rides, naps or delayed departures in advance, so that on the night you need them, you are following a plan rather than making a hard decision while exhausted. The term is the unit The third conclusion is that the evening, which is where this book has focused, is not really the right unit for thinking about an evening student's health. The term is. Sleep debt accumulates across nights. The Van Dongen study discussed in Chapter 2 found that people restricted to six hours a night kept getting worse over two weeks, while feeling only slightly sleepier. A student who averages six hours through a fifteen-week term is not experiencing fifteen weeks of the same mild tiredness; they are likely experiencing a slow decline in attention, memory and mood that they may not notice until it shows up in grades, relationships or a near miss on the road. The late-eating research, too, concerns patterns rather than single meals. The consequences of the evening schedule are cumulative, and so the most powerful decisions about it are the ones made at the level of the term. That shifts attention from the individual night to a handful of structural choices, most of which are made before the first class. How many evening courses to take at once. Which nights to take them on, and whether the morning after each can be protected at work. How long the commute is, and whether there is a backup way home. Who will cover the household on class nights, and whether that arrangement is fair and sustainable for them. Which college services exist at hours you can use. These decisions are less vivid than choosing between coffee and water at the break, but they set the conditions in which every nightly choice is made. A student who gets them right will find the nightly habits far easier; a student who gets them wrong may find that no amount of nightly discipline is enough. This is why the chapter on families, workplaces and colleges is not an afterthought. The evening student's wellness is a shared project whether or not anyone acknowledges it. The partner who does bedtime, the manager who allows a 9:00 start on Wednesdays, the instructor who ends on time and the college that runs its escort service until after the last class all contribute to whether the student sleeps, eats and gets home safely. Making those contributions explicit, and asking for them clearly, is part of the work. A different measure of success Evening students tend to measure success by endurance: getting through the class, getting through the week, getting through the term. The capacity to endure is real and admirable, and many evening students have far more of it than their daytime peers. But endurance is a poor guide to health, because it rewards pushing past exactly the signals, fatigue, hunger, drowsiness at the wheel, that the body uses to protect itself. A better measure is sustainability. Can you keep this schedule up for the length of the program without your health, your relationships or your safety paying for it? If the answer is yes, the details of this book are refinements. If the answer is no, the most important thing you can change is not your caffeine cut-off but your schedule, and there is no shame in changing it. A degree finished a term later, by someone who is well rested, still married, still close to their children and never had the crash they came close to, is a better outcome by any measure that matters. The people in the parking lot at a quarter to six, eating granola bars and finishing phone calls, have already shown that they have the determination to do something hard. What they need is not more determination but a way of arranging their evenings, and their terms, that lets that determination last. That is what these habits are for: not to make night classes easy, which they will never be, but to make them survivable, healthy and safe, one well-planned evening at a time. Hashtags: #HealthyHabits #NightClasses #EveningStudents #StudentWellness #NonTraditionalStudents #AdultLearners #EveningClassWellness #HealthyStudentLifestyle #SleepHygiene #SleepHealth #CircadianRhythm #SleepManagement #CaffeineManagement #HealthyMealPlanning #MealTiming #HealthyEatingHabits #StudentNutrition #EnergyManagement #FatigueManagement #DrowsyDrivingPrevention #CampusSafety #LateNightSafety #StressManagement #WorkStudyBalance #WorkLifeStudyBalance #TimeManagement #StudentMentalHealth #HealthyStudyHabits #EveningRoutine #LifelongLearning

Download the Book (PDF): Introduction A handbag with a flap, a strap and a small padlock hanging from the front is, in the eyes of American copyright law, a useful article. So is a quilted jacket, a pair of stretch leggings, a sheepskin boot and a sneaker with a rubber toe cap. None of them, as designed objects, belongs to anyone for very long, and many of them belong to no one at all. A competitor can study the season's most admired silhouette on Monday, send a technical drawing to a factory on Tuesday, and have a close copy in stores within weeks, and in most cases nothing in the federal statute books will stop it. Yet the same competitor who copies the silhouette cannot copy the padlock if it has become the recognized signature of a particular house, cannot use the red lacquered sole that shoppers associate with one Parisian shoemaker on a shoe that is otherwise a different color, and cannot sell a bag stamped with interlocking letters it has no right to use. In one case the copier is a fast-fashion retailer doing what the industry has always done. In another it is an infringer liable for the brand owner's damages. In the third it is a counterfeiter who may owe up to two million dollars in statutory damages per mark and who may, if the government takes an interest, go to prison. The difference between those outcomes is the subject of this book. Fashion law, as a working field, is largely the practice of knowing where the lines fall: what the law will protect in a garment or accessory, how the owner can enforce that protection against an industry that copies at industrial speed, and, increasingly, what the law demands of the company in return. For the last of those questions has changed the most. A decade ago, a book on fashion law could treat supply chains as a matter of contract and cost. Today a brand that cannot say where its cotton was grown may find its shipments detained at the border, a brand that tells shoppers its jacket is sustainable must be ready to prove what that means, and a brand selling into California or the European Union will soon pay for what happens to its clothes after customers throw them away. The argument The controlling claim of this book is that fashion law protects signals rather than designs, and that the law is now beginning to insist that those signals be true. Consider what the three branches of the subject have in common. Trade dress and trademark protect a product's capacity to tell a shopper who made it. They do not reward creativity as such; they reward the accumulated association between a look and a source. Counterfeiting law exists because a fake product sends a false signal about origin, and the most serious remedies in American trademark law are reserved for that falsehood. And the new supply-chain mandates, whether they concern forced labor, chemical safety, textile waste or green marketing, are at bottom rules about origin too: where the fiber came from, under what conditions it was spun and sewn, what it is made of, and whether the story printed on the hang tag is accurate. Seen this way, the modern fashion company faces a single legal problem wearing three outfits. It must build distinctive signals, defend them against false copies, and make sure that its own signals about provenance and virtue can withstand scrutiny. The capabilities that serve one task tend to serve the others. A brand that can trace a garment from gin to shop floor can prove its goods are authentic, answer customs officers asking about Xinjiang cotton, and substantiate a recycled-content claim. A brand that cannot do any of these things is exposed on all three fronts at once. What the book covers The chapters move from protection to enforcement to accountability. Chapter 1 explains why the United States has never given fashion design the robust protection that literature, music and software enjoy, and what little the copyright and design patent systems do offer. It sets out the tools available and the price of each, and it contrasts the American position with the European Union's more generous regime for designs. Chapters 2 and 3 are about trade dress, the doctrine that does most of the heavy lifting for fashion houses. Chapter 2 explains how a product's appearance can become a protectable indicator of source, and why the Supreme Court has made that hard to prove for product designs. Chapter 3 examines the limits: the functionality doctrine, which keeps useful and aesthetically necessary features in the public domain, the treatment of color and pattern, and the cases on stripes, soles and boots that show where courts draw the boundary. Chapter 4 turns to copying in the market: the knockoff, the counterfeit and the newest figure in the story, the openly advertised "dupe." It examines what separates lawful imitation from infringement and how recent litigation involving fast-fashion platforms and mass retailers has tested the line. Chapters 5 and 6 cover counterfeiting. Chapter 5 sets out the legal framework, from statutory damages and ex parte seizures to the liability of landlords, marketplaces and resellers. Chapter 6 describes the enforcement machinery that has grown up around online counterfeiting, including the mass-defendant "Schedule A" lawsuits in Chicago, the new federal marketplace transparency law, and the end of the customs exemption that allowed billions of dollars of low-value parcels to enter the country duty-free. Chapters 7 and 8 address sustainable supply chains. Chapter 7 covers forced labor and human rights: import bans, the Uyghur Forced Labor Prevention Act, disclosure statutes, the limited reach of American tort law over overseas factories, and the European due diligence directive in its recently reduced form. Chapter 8 covers environmental mandates: extended producer responsibility for textiles, bans on destroying unsold goods, restrictions on chemicals, the French campaign against ultra-fast fashion, and the rapidly tightening law of green marketing claims. The conclusion draws the threads together and sets out what the analysis implies for designers, brand owners, retailers and their counsel. A note on scope The book is primarily concerned with United States law, because that is where the largest body of fashion litigation has been decided and where many of the industry's disputes are fought even when the parties are foreign. Where the European Union or its member states lead, which in sustainability they plainly do, the book treats them in some depth, because an American company selling abroad is bound by them and because their approaches often preview what American legislatures will consider. The law described is current to the autumn of 2026. Several matters discussed, including pending lawsuits, draft regulations and bills in state legislatures, were unsettled at the time of writing, and the text says so where that is the case. This book is an educational account of the law and is not legal advice. A final word on vocabulary. The industry uses "knockoff," "copy," "dupe," "fake" and "counterfeit" almost interchangeably. The law does not, and much of what follows depends on keeping them apart. A counterfeit, in the legal sense, bears a spurious mark that is identical with or substantially indistinguishable from a registered one. A knockoff copies a design but carries its own name, and it may or may not infringe depending on what it copies. A dupe, in current usage, is a product marketed precisely as an affordable alternative to a named original, and its legality depends on the same questions as any other knockoff, sharpened by the fact that the seller has invited the comparison. Those distinctions, and the reasons the law draws them where it does, run through every chapter that follows. Chapter 1: The Protection Gap Every fashion lawyer eventually has the same conversation with a designer. The designer has created something new, a cut or a construction or a combination of elements that no one has put together before, and has just seen it reproduced by a larger company at a fraction of the price. The designer wants to know which law has been broken. Very often the honest answer is none. That answer surprises people because the rest of the creative economy is so heavily protected. A novelist owns the novel for life plus seventy years. A songwriter can sue over a few bars of melody. A software company can claim copyright in code that no human will ever read. Fashion, an industry worth hundreds of billions of dollars a year in the United States alone, sits largely outside that system. Understanding why, and understanding the narrow openings that do exist, is the foundation for everything else in the field. Copyright and the useful article The Copyright Act protects "pictorial, graphic, and sculptural works," but it treats the design of a "useful article" differently. A useful article is one with an intrinsic utilitarian function that is not merely to portray its own appearance or convey information. Clothing, shoes and handbags are the textbook examples: they cover the body, protect the feet and carry things. The design of a useful article is protected only to the extent that it incorporates pictorial, graphic or sculptural features that can be identified separately from, and are capable of existing independently of, the article's utilitarian aspects. This rule has a long lineage. In Mazer v. Stein (1954) the Supreme Court held that statuettes used as the bases of table lamps could be copyrighted, because the sculpture remained a work of art even when it was put to use. Congress codified the idea in the 1976 Act, and the House report accompanying it explained that the intent was to protect applied art but not industrial design as such. The shape of an automobile, an airplane, a dress or a food processor, however pleasing, was to remain unprotected unless it contained an element that could be conceptually separated from the utilitarian object. For decades, courts struggled to apply "separability," producing a thicket of competing tests. The Second Circuit's decision in Kieselstein-Cord v. Accessories by Pearl (1980) held that sculptured belt buckles, which buyers wore as jewelry as much as fasteners, contained separable artistic elements. Other courts asked whether the designer's artistic judgment had been exercised independently of functional influences, whether the article would still be marketable as art, or whether the feature could be imagined on a shelf. The results were difficult to predict. The Supreme Court tried to clarify the question in Star Athletica, L.L.C. v. Varsity Brands, Inc. (2017). Varsity Brands held copyright registrations in two-dimensional designs of stripes, chevrons, zigzags and color blocks appearing on cheerleading uniforms, and it sued Star Athletica for selling uniforms bearing similar designs. The Court announced a two-part test. A feature of a useful article is eligible for copyright if it can be perceived as a two- or three-dimensional work of art separate from the useful article, and if it would qualify as a protectable pictorial, graphic or sculptural work, either on its own or in some other medium, if imagined separately. Applying that test, the Court concluded that the surface decorations could be lifted off the uniform and applied to a canvas, where they would be two-dimensional artworks, and so they were eligible for protection. Star Athletica was widely described as a victory for designers, and in one sense it was: it confirmed that surface graphics on garments can be protected and it swept away the more restrictive tests some circuits had used. But the Court was explicit about the limit. Varsity, it said, had no right to prohibit anyone from manufacturing a cheerleading uniform of identical shape, cut and dimensions. The protection extended to the artwork on the uniform, not to the uniform. Imagining the stripes on a canvas did not carry the garment with them. The practical consequence is that American copyright law protects fashion in two main places. The first is textile and print design. An original floral pattern, a graphic, a pictorial print or an artistic arrangement of shapes printed on or woven into fabric is protectable as a two-dimensional work, and fabric-print infringement has long been one of the most active areas of fashion litigation, particularly in the Central District of California, home of the Los Angeles garment trade. The second is ornamental elements that are genuinely separable, such as a sculptural clasp, an appliqué or a piece of jewelry-like hardware. The silhouette, the cut, the drape, the construction and the combination of standard elements into a new look all fall outside the statute. Even within its domain, fabric-print litigation carries its own traps, several of which surfaced in Unicolors, Inc. v. H&M Hennes & Mauritz, L.P. (2022). Unicolors, a Los Angeles textile company, won a jury verdict that H&M had copied one of its designs. H&M argued that Unicolors' copyright registration was invalid because Unicolors had registered thirty-one designs together as a single unit of publication when some had initially been sold only to select customers. The Supreme Court held that the statute's safe harbor for inaccurate information in a registration covers good-faith mistakes of law as well as fact, so that a registrant's misunderstanding of the rules on unit publication does not automatically invalidate the registration. The case matters to fashion because textile houses routinely register collections of designs in batches, and defendants had made a habit of attacking those registrations. Design patents The second federal tool is the design patent. Under the Patent Act, anyone who invents a new, original and ornamental design for an article of manufacture may obtain a patent on it. Unlike copyright, the design patent system was built for the appearance of useful objects, so the useful-article problem does not arise in the same way. A shoe, a handbag or a sunglass frame can all be patented as designs. The design patent has several virtues. Infringement is judged by the "ordinary observer" test, which the Federal Circuit confirmed as the sole test in its en banc decision in Egyptian Goddess, Inc. v. Swisa, Inc. (2008): the patented and accused designs are compared in light of the prior art, and infringement is found if an ordinary observer, familiar with that prior art, would be deceived into thinking the accused design was the same as the patented one. The patent holder need not prove that anyone was confused about source. The remedies are unusually strong. Section 289 of the Patent Act allows the owner to recover the infringer's total profit on the article of manufacture to which the design was applied, although the Supreme Court held in Samsung Electronics Co. v. Apple Inc. (2016) that the relevant "article" may be a component of the product rather than the whole product. The design patent also has serious drawbacks for fashion. It must be applied for within a year of public disclosure, examined and granted, a process that commonly takes well over a year and costs thousands of dollars per design once attorney fees are included. For a garment whose selling life is a single season, that timetable is useless. The patent lasts fifteen years from grant, which is longer than most fashion products need, but the cost and delay mean that brands tend to patent only designs they expect to become perennials: a signature boot, a sneaker sole, a bag closure, a piece of hardware. The design must also be new and non-obvious, and in 2024 the Federal Circuit made the non-obviousness hurdle potentially higher. For decades design patents were assessed under the Rosen-Durling test, which required a single primary prior-art reference with design characteristics "basically the same" as the claimed design before any combination of references could be considered. The test made design patents hard to invalidate for obviousness. In LKQ Corp. v. GM Global Technology Operations LLC (2024), the en banc Federal Circuit overruled Rosen-Durling and held that design patent obviousness should be analyzed under the same flexible framework the Supreme Court has prescribed for utility patents. The decision was about automobile fenders, but its effect on fashion was immediate, because apparel and footwear designs typically draw heavily on earlier designs, and a more flexible obviousness inquiry gives accused copiers a better chance of invalidating the patent. How much better became visible in 2026 in the litigation between Deckers Outdoor Corporation, owner of the UGG brand, and the direct-to-consumer retailer Quince. Deckers asserted both trade dress and a design patent covering a short sheepskin boot. The trade dress claims failed at summary judgment, for reasons examined in Chapter 3. The design patent claim went to a jury in the Northern District of California, which in June 2026 found that Quince's boot infringed the patent but also that the patent was invalid. The court denied Deckers' motion for a new trial in September 2026. A brand that had proven copying was left without a remedy, a result that a design patent holder would have found much harder to suffer a few years earlier. The failed campaign for a fashion design right The protection gap is not an accident that Congress has overlooked. It has been debated for more than a century. Bills to create a registration system for designs were introduced repeatedly in the early twentieth century and in almost every Congress from the 1950s onward. The most recent serious attempt was the Design Piracy Prohibition Act, first introduced in 2006, and its successor, the Innovative Design Protection and Piracy Prevention Act, which the Senate Judiciary Committee approved in 2010 and which was reintroduced in 2012. The later bill would have granted three years of protection to fashion designs that were unique, distinguishable, non-trivial and a non-utilitarian variation over prior designs, with infringement limited to copies "substantially identical" to the protected design. It never reached a floor vote. The opposition was not only from mass-market retailers. A strand of academic argument, most influentially developed by Kal Raustiala and Christopher Sprigman in their 2006 article "The Piracy Paradox," contended that fashion thrives precisely because copying is legal. On this account, copying accelerates the diffusion of new styles, which makes them unfashionable faster among the trend-setting consumers who adopted them first, which drives demand for the next thing. The industry's rapid seasonal cycle, in other words, is fed by the absence of protection. Others, notably C. Scott Hemphill and Jeannie Suk in a 2009 Stanford Law Review article, answered that the argument conflated trends with individual designs, and that close copying of a particular designer's work harms emerging designers who lack the brand recognition to benefit from trademark law. That exchange remains the best short introduction to the policy debate, and neither side has won it decisively. What is clear is that Congress has chosen, by inaction, to leave fashion design in the public domain unless it can find shelter in some other doctrine. The European contrast The European Union made a different choice. Since 2002, the Community Design Regulation has provided two layers of design protection throughout the Union. A registered design, available for up to twenty-five years in five-year renewable terms, protects the appearance of a product, including its lines, contours, colors, shape, texture and materials, if the design is new and has individual character. An unregistered design arises automatically on disclosure in the Union and lasts three years, but protects only against deliberate copying. The unregistered right was designed with the fashion industry in mind: it gives protection for about the length of a product's commercial life without any registration cost at all. The EU overhauled this system in legislation adopted in late 2024, Regulation (EU) 2024/2822 and Directive (EU) 2024/2823. The first stage took effect on 1 May 2025, renaming the "Community design" the "registered EU design" and the "unregistered Community design" the "unregistered EU design," modernizing the definition of a design to cover animation and digital products, and allowing designs to be represented in new file formats. The core fashion-relevant features, novelty, individual character, the unregistered right and the long registered term, remain in place. European copyright is also more generous to applied art. In Cofemel v. G-Star Raw (2019), the Court of Justice of the European Union held that a design may be protected by copyright under EU law if it is an original work reflecting the author's own intellectual creation, and that member states may not impose an additional requirement of artistic or aesthetic value. The case concerned jeans, sweatshirts and T-shirts. The court added that aesthetic effect alone is not enough and that the design must reflect free and creative choices, so not every garment qualifies. But the European starting point is plainly more hospitable than the American one, and a design that is freely copyable in New York may be protected in Milan or Paris. The comparative picture matters in practice. A brand launching a signature design can obtain an EU registered design quickly and cheaply, often within weeks, and rely on the unregistered right in the meantime. That protection is territorial, so it does nothing to stop copying in the United States. But it means that global brands often enforce against copyists in Europe first and treat the American market as a place where they must rely on trade dress, trademark and whatever copyright can be found in the surface of the product. The toolkit, and what each tool costs The legal tools available to protect a fashion product differ in what they cover, how quickly they arise, how long they last and what they demand of the owner. Table 1 sets them side by side for the United States. Table 1. The main US legal tools for protecting fashion products. Tool What it protects How it arises Duration Main obstacle for fashion Copyright Original prints, graphics, separable ornament Automatically on fixation; registration needed to sue Life of author plus 70 years, or 95 years for corporate works Garment shape, cut and construction excluded Design patent New, ornamental, non-obvious product appearance Examination and grant by the USPTO 15 years from grant Cost and delay; invalidity risk after LKQ Trademark Names, logos, monograms, other source identifiers Use in commerce; registration adds presumptions Indefinite while used and distinctive Protects the mark, not the design Trade dress Distinctive, non-functional appearance of product or packaging Use plus proof of secondary meaning for product design Indefinite while used and distinctive Secondary meaning and functionality The pattern in the table tells the story of the field. The tools that protect design as design, copyright and design patent, are either narrow or slow. The tools that last indefinitely, trademark and trade dress, protect design only once the public has come to see it as a signature of a particular source. That is why fashion houses invest so heavily in signature elements, repeated season after season, and why the most important fashion cases of the last thirty years have been trademark and trade dress cases rather than copyright cases. A design that appears once is almost always free to copy. A design that appears for twenty years, in advertising and on the street, may become a brand. The rest of this book follows that logic. Chapters 2 and 3 examine how a design becomes a brand, and the limits the law sets on the process. But the lesson of this chapter should be kept in view throughout. In the United States, the default rule for fashion design is freedom to copy. Every protection is an exception, and each exception has to be earned. Chapter 2: Trade Dress: When a Design Becomes a Brand The idea behind trade dress is simple enough to explain to a child. People recognize things by how they look. A shopper who sees a particular shade of turquoise box knows where the jewelry came from before reading the name on the lid. A person on the street who glimpses a flash of red beneath a black stiletto, or a small leather tab stitched into the back pocket of a pair of jeans, or a curved double line of stitching across the same pocket, forms an instant judgment about the maker. When the appearance of a product or its packaging does that work, it functions like a trademark, and the law will protect it against imitations likely to confuse. What makes trade dress difficult, and what makes it the central doctrine of American fashion law, is the tension between that simple idea and the rule described in Chapter 1: that the design of useful articles is generally free to copy. If any attractive, recognizable look could be claimed as trade dress, trade dress would become a perpetual design right by another name, available without examination, novelty or a time limit. The Supreme Court has spent three decades policing that boundary, and its decisions shape how every fashion house thinks about its signature designs. The statutory basis Trade dress protection rests mainly on Section 43(a) of the Lanham Act, the federal trademark statute. Section 43(a) prohibits the use in commerce of any word, term, name, symbol or device, or any combination of them, that is likely to cause confusion as to the origin, sponsorship or approval of goods. Courts long ago read "symbol or device" broadly enough to include the overall appearance of a product or its packaging. Congress confirmed the point in 1999 when it amended the statute to provide that, in an action for infringement of unregistered trade dress, the person asserting protection bears the burden of proving that the matter sought to be protected is not functional. Trade dress can also be registered on the Principal Register of the United States Patent and Trademark Office, as can shapes and colors. Registration brings procedural advantages, including a presumption of validity, nationwide constructive notice and the ability to record the registration with Customs and Border Protection so that infringing imports can be stopped at the border. Hermès, for example, has registered the shape of its Birkin bag, and Louboutin registered its red sole. But registration does not change the substantive requirements, and many trade dress claims in fashion rest on unregistered rights asserted directly under Section 43(a). To prevail, a trade dress plaintiff must generally establish three things: that the claimed trade dress is distinctive, either inherently or because it has acquired secondary meaning; that it is not functional; and that the defendant's product is likely to cause confusion. This chapter concentrates on the first element and on a preliminary requirement that has become important in its own right, the need to define the claimed trade dress with precision. Functionality and confusion are the subject of Chapter 3. Distinctiveness: from Two Pesos to Wal-Mart The modern law of trade dress distinctiveness was built in three Supreme Court decisions decided between 1992 and 2000. The first, Two Pesos, Inc. v. Taco Cabana, Inc. (1992), involved the décor of a Mexican restaurant chain in Texas, which the Court described as a festive eating atmosphere with interior dining and patio areas decorated with artifacts, bright colors, paintings and murals. A competitor copied it. The question was whether the restaurant owner had to prove secondary meaning, that is, that the public had come to associate the décor with a single source, or whether trade dress, like a word mark, could be inherently distinctive. The Court held that trade dress could be inherently distinctive and protected without proof of secondary meaning. Two Pesos was a victory for the plaintiff, and for several years afterward trade dress claimants in many industries argued that their product designs were inherently distinctive. The second decision, Qualitex Co. v. Jacobson Products Co. (1995), concerned a green-gold color used on dry-cleaning press pads. The Court held that a color alone could serve as a trademark, but only if it had acquired secondary meaning, and only if it was not functional. The Court reasoned that color, unlike a coined word, does not automatically tell a customer that it refers to a brand. It is often used to make a product more attractive, and consumers do not ordinarily assume that a color identifies a source. But over time, customers may come to treat a particular color on a particular product as a signifier of the brand. When that happens, the color does the same work as a word mark and can be protected like one. The third and most important decision for fashion is Wal-Mart Stores, Inc. v. Samara Brothers, Inc. (2000). Samara Brothers designed and manufactured a line of spring and summer one-piece seersucker children's outfits decorated with appliqués of hearts, flowers and fruits. Wal-Mart sent photographs of Samara garments to a supplier, which produced close copies that Wal-Mart sold at lower prices. Samara sued for infringement of unregistered trade dress and won a jury verdict. The Supreme Court reversed unanimously. Writing for the Court, Justice Scalia held that product design, unlike product packaging, can never be inherently distinctive. In an action for infringement of unregistered product-design trade dress, the plaintiff must always prove secondary meaning. The Court's reasoning explains much of what followed. Consumers, it said, are predisposed to regard certain marks, such as a word on a label or a distinctive package, as indications of source. They are not predisposed to regard product design that way, because design is almost always chosen to make the product more useful or more appealing. A rule allowing inherently distinctive product designs would expose competitors to litigation over any attractive design and deprive consumers of the benefits of competition in useful or attractive features. And where it is unclear whether a feature is packaging or design, courts should classify it as design and require secondary meaning. For fashion, Wal-Mart means that the appearance of a garment, shoe or bag cannot be protected under Section 43(a) the moment it is launched. The owner must wait until the public has come to identify the design with the brand, and then prove that identification in court. The designer who creates something striking and is copied in the first season has, in trade dress terms, nothing to assert. Only designs that survive, sell and become famous can be protected, which is one reason trade dress tends to favor established houses over young designers. Proving secondary meaning Secondary meaning exists when, in the minds of the public, the primary significance of a product feature is to identify the source of the product rather than the product itself. Courts consider a cluster of factors that vary slightly by circuit but generally include advertising expenditures, consumer studies linking the design to a source, unsolicited media coverage, sales success, attempts by others to copy the design, and the length and exclusivity of use. Two of those factors deserve particular attention in fashion. The first is "look-for" advertising, meaning promotion that draws consumers' attention to the specific design feature as a mark of the brand. Advertising that simply shows the product being worn is weak evidence, because it may build admiration for the look without teaching consumers that the look means the brand. Advertising that singles out the feature, invites buyers to look for it or presents it as a signature is much stronger. Houses that have succeeded in trade dress litigation have often spent years making their signature element the subject of campaigns, not just a detail within them. The second is exclusivity. If many companies use a similar design, consumers are unlikely to associate it with any one of them. In fashion, where trends spread quickly and many brands work within the same vocabulary, third-party use is often the defense's best evidence. A plaintiff that has tolerated similar designs for years may find that its own inaction has destroyed the secondary meaning it now claims. The Federal Circuit's decision in Converse, Inc. v. International Trade Commission (2018) shows how exclusivity and timing interact. Converse sought to stop imports of sneakers that allegedly copied the midsole design of its Chuck Taylor All Star shoe: the toe cap, the multilayered toe bumper and the stripes on the midsole. The design had been in use for decades, but so had many similar designs from other makers, and Converse had registered the trade dress only in 2013. The Federal Circuit held that where a party claims infringement of both registered and unregistered rights, secondary meaning must be established as of the date the accused infringer first entered the market, and that the presumption of validity from registration does not reach back before the registration date. It also stressed that evidence of third-party use and the relevance of survey evidence must be weighed with those dates in mind. The case was sent back for further proceedings, and it has become a caution to brands that wait until a design has been widely copied before trying to register it. Articulating the trade dress A trade dress claimant must also say precisely what it claims. This requirement sounds procedural, but it frequently decides cases. The leading fashion authority is the Second Circuit's decision in Yurman Design, Inc. v. PAJ, Inc. (2001). David Yurman, the jewelry designer, sued PAJ over jewelry that allegedly copied Yurman's designs, which featured a cable motif. Yurman won a jury verdict on trade dress. The Second Circuit reversed on that claim because Yurman had never articulated the specific elements that made up its trade dress. It had described the claimed dress in terms such as "artistic combination of cable jewelry with other elements," which was too general to identify what exactly the defendant was forbidden to copy. The court explained that a plaintiff seeking to protect a line of products must articulate the specific common elements that make up the trade dress across the line, both so that courts can decide whether those elements are distinctive and non-functional and so that competitors know what they may not do. A plaintiff seeking protection for a whole product line faces a particularly heavy burden, because it must show that the line has a consistent overall look. The practical rule that emerged from Yurman and similar decisions, including the Second Circuit's earlier ruling in Landscape Forms, Inc. v. Columbia Cascade Co. (1997), is that trade dress claims should be pleaded with a list of concrete, visible features. Vague references to a "look," "style" or "aesthetic" invite dismissal. A claim to "the combination of a rectangular leather bag with a trapezoidal flap, two rolled handles, a horizontal strap with a turn-lock closure and a padlock hanging from the strap" is at least testable. A claim to "the elegant design of our handbag" is not. Precision has a price. The more precisely the trade dress is defined, the easier it becomes for a competitor to design around it by changing one or two elements. The more broadly it is defined, the more likely it is to be rejected as vague, generic or functional. Much of the craft of trade dress litigation lies in finding the level of description that captures what consumers actually recognize without claiming a whole category of product. Examples from the field The most successful fashion trade dress claims share a common profile: a limited set of specific visual elements, used consistently for many years, heavily promoted as a mark of the brand, and not required by the function or the ordinary aesthetics of the product. Levi Strauss & Co. has long protected the double arcuate stitching design on the back pockets of its jeans and the small colored tab sewn into the pocket seam. Both are registered, both have been used for generations, and Levi's has litigated extensively against competitors using similar stitching or tabs. The stitching sits on a garment that would work just as well without it, and the company has built consumer recognition over a very long period. Hermès has relied on trade dress protection for the Birkin and Kelly bags, whose silhouettes, closures and hardware have been sold in essentially the same form for decades and are among the most recognized luxury objects in the world. In the litigation against the artist Mason Rothschild over "MetaBirkins" digital tokens, discussed in Chapter 4, the Birkin's fame was central to the jury's finding of likely confusion. Tiffany & Co. has protected the robin's egg blue of its boxes and bags, a color used on packaging rather than product. Because packaging, unlike product design, can be inherently distinctive under Two Pesos and Wal-Mart, Tiffany's position is easier than that of a brand claiming a color on the product itself, although as a single color it still requires secondary meaning under Qualitex. The cases also show what fails. Product-line claims described in general terms fail. Claims to design features that consumers see as ordinary features of the category fail, as UGG's owner discovered when the court in its case against Quince held the claimed boot features generic, a ruling discussed in the next chapter. And claims to features that make the product work better, or that consumers want for their own sake rather than as a signal of source, fail under the functionality doctrine. Those limits are the subject of Chapter 3. Why trade dress does the work It is worth pausing on why trade dress, rather than copyright or design patent, has become the central doctrine for protecting fashion design. It is not because trade dress is well suited to protecting design. In many ways it is poorly suited. It arrives only after the design has become famous, protects only against confusion rather than copying as such, and demands expensive proof of what consumers think. But it has two decisive advantages. It lasts as long as the design continues to signal the brand, which for the most valuable designs means indefinitely, and it requires no application within a fixed period after launch. For a house whose business depends on a few iconic products sold over decades, those features outweigh the rest. The consequence is an industry shaped by the law's incentives. Brands that want legal protection for their designs must repeat them, promote them as signatures and police imitations before they become common. Designs that change every season, which describes most of fashion, remain free to copy. The law does not protect fashion so much as it protects the small part of fashion that has stopped changing. Chapter 3: Functionality, Color and the Limits of Trade Dress A brand that proves its design has acquired secondary meaning has cleared the most visible hurdle in trade dress law, but not the most dangerous one. The functionality doctrine can defeat a claim even when consumers overwhelmingly associate a design with its maker. It exists because trademark law is meant to protect the signals that tell buyers who made a product, not the features that make the product good. If a feature is useful, or if it is the kind of thing competitors need in order to compete, no amount of consumer recognition will make it the property of a single seller. This chapter examines functionality in its two forms, utilitarian and aesthetic, and follows the doctrine into the cases that have defined the outer edge of fashion trade dress: the red sole, the stripes on a sleeve, the shape of a sheepskin boot. It then turns to likelihood of confusion, the element that determines whether a protected design has actually been infringed, and to the distinctive way fashion courts think about confusion that happens after the sale. Utilitarian functionality The classic statement of functionality comes from Inwood Laboratories, Inc. v. Ives Laboratories, Inc. (1982): a product feature is functional if it is essential to the use or purpose of the article or if it affects the cost or quality of the article. The Supreme Court reaffirmed that test in TrafFix Devices, Inc. v. Marketing Displays, Inc. (2001), a case about a dual-spring mechanism that kept road signs upright in the wind. Marketing Displays had held utility patents on the mechanism, and when those expired it tried to protect the visible springs as trade dress. The Court held that a utility patent is strong evidence that the features it claimed are functional, and that once a feature is shown to be functional under the Inwood test, there is no need to consider whether competitors have alternative designs available. A functional feature cannot be protected as trade dress, full stop. In fashion, utilitarian functionality most often appears in footwear and outerwear, where construction choices affect comfort, warmth, grip and durability. A sole pattern that improves traction, a lining that retains heat, a zipper placement that makes a jacket easier to close or a strap configuration that distributes weight are all candidates. The lesson of TrafFix is that brands should be cautious about claiming as trade dress any feature they have also patented as a utility invention or advertised as a performance advantage. A company that spends years telling customers that its sole design prevents slipping will struggle to argue in court that the same design is merely a signature. Aesthetic functionality Fashion's more difficult problem is aesthetic functionality. Some features are valuable not because they make a product work better but because they make it more beautiful, and consumers want them for their own sake. If a single seller could monopolize such a feature, competitors would be excluded from a significant part of the market for reasons that have nothing to do with confusion about source. The doctrine of aesthetic functionality denies protection in those cases. The Supreme Court acknowledged the doctrine in Qualitex and in TrafFix, describing a feature as aesthetically functional when its exclusive use would put competitors at a significant non-reputation-related disadvantage. That qualifier matters. Every successful brand has features that competitors would like to copy because consumers associate them with a prestigious source. The disadvantage a competitor suffers from being unable to copy those features is reputation-related, and it does not make the features functional. A feature is aesthetically functional only when competitors need it for reasons independent of the brand's reputation, because it is a basic element of the product's appeal. Courts have applied the distinction with varying enthusiasm. The Ninth Circuit's decision in Au-Tomotive Gold, Inc. v. Volkswagen of America, Inc. (2006) held that Volkswagen and Audi logos reproduced on key chains and license plate covers were not aesthetically functional merely because consumers wanted the logos for their association with the car brands. That want, the court said, was precisely the source-identifying function trademark law protects. The case confirmed that the doctrine cannot be used to strip protection from famous marks simply because they are desirable. Color and the red sole The most famous fashion trade dress case of the century so far, Christian Louboutin S.A. v. Yves Saint Laurent America Holding, Inc., turned on aesthetic functionality. Christian Louboutin began painting the outsoles of his high-heeled shoes in a glossy red lacquer in the early 1990s. Over the following two decades the red sole became one of the most recognized signatures in luxury fashion, and in 2008 the Patent and Trademark Office registered it as a trademark for "women's high fashion designer footwear." In 2011 Yves Saint Laurent released a collection of monochrome shoes, including an all-red shoe with a red sole. Louboutin sued and sought a preliminary injunction. The district court denied it, reasoning that in the fashion industry color serves an aesthetic function that is essential to competition and that a single-color mark could never be protected in fashion. The Second Circuit, in 2012, rejected that categorical rule. It held that a single color can serve as a trademark in the fashion industry and that the aesthetic functionality defense must be applied case by case. It also found that Louboutin had proven secondary meaning in the red sole. But it then limited the mark to a particular configuration: a red lacquered outsole that contrasts with the color of the shoe's upper. The evidence of secondary meaning, the court reasoned, showed that consumers recognized the contrast between a red sole and a differently colored upper as the Louboutin signature. It did not show that consumers associated a red sole on an entirely red shoe with Louboutin. The court directed the Patent and Trademark Office to modify the registration accordingly, and held that YSL's monochrome red shoe did not use the mark at all. Both sides claimed victory: Louboutin kept its registration, and YSL kept the right to sell a monochrome red shoe. The case illustrates how courts reconcile trade dress with fashion's need for free use of color. Rather than holding color functional in fashion, the Second Circuit narrowed the protected mark to the specific use that consumers actually recognize. The same approach can be seen in the European treatment of the same sole. In Christian Louboutin v. Van Haren Schoenen BV (2018), the Court of Justice of the European Union held that the red sole mark was not a mark consisting exclusively of a "shape," because it claimed a color applied to a specific location on the shoe, so that the EU rule excluding shapes that give substantial value to goods did not apply. The European court did not address whether the mark was valid in every respect, but the ruling kept the red sole on the European register. Stripes, generic features and the exhausted vocabulary Two more recent disputes show how the limits of trade dress bite on the brands with the strongest claims. In adidas America, Inc. v. Thom Browne, Inc., adidas, which owns registrations for its three-stripe mark on apparel and footwear, sued the designer Thom Browne over Browne's "Four-Bar Signature," four parallel horizontal stripes on the sleeve or leg of a garment, and his "Grosgrain Signature," a red, white and blue ribbon. After a trial in the Southern District of New York in January 2023, the jury found that Thom Browne had not infringed or diluted adidas's marks, deliberating for only a few hours. adidas sought a new trial, arguing among other things that the judge's instructions on post-sale confusion were inadequate. The district court denied the motion, and on 3 May 2024 the Second Circuit affirmed, concluding that the instructions given were not meaningfully different from the ones adidas had proposed and that adidas had waived other objections. The case is not a holding that stripes can be freely used. adidas has won many cases against sellers of two- and four-stripe designs on athletic shoes over the years. It is a demonstration that a jury may find that a luxury designer's distinct pattern, sold at very different prices through very different channels and associated with its own designer, is not likely to confuse consumers even when it shares a basic motif with a famous mark. Simple geometric elements, especially stripes, belong to a common vocabulary of design, and the more basic the element, the narrower the protection. The ruling in Deckers Outdoor Corp. v. Last Brand, Inc., the UGG case against Quince, pressed the same principle further. Deckers claimed trade dress in the overall appearance of its Classic Ultra Mini and Tasman sheepskin boots. In October 2025 the district court granted summary judgment to Quince on the trade dress claims, holding the claimed designs generic and therefore ineligible for trade dress protection. Genericness in trade dress means that the design has come to define a category of product rather than identify a particular maker. The shape of a short sheepskin boot with exposed seams and a fleece collar may be closely associated with UGG in the public mind, but if the public understands that shape as what a sheepskin boot looks like, it cannot be claimed by one seller. The ruling was a reminder that even a famous brand can lose trade dress in the form it helped popularize. Together, the Thom Browne and UGG cases suggest a principle that runs through the doctrine. The more a brand's design has shaped the look of an entire category, the harder it becomes to claim that look as trade dress. Success can undermine exclusivity. Likelihood of confusion Trade dress that is distinctive and non-functional is infringed only when the defendant's product is likely to cause confusion. Each circuit applies a multi-factor test. The Second Circuit's version, from Polaroid Corp. v. Polarad Electronics Corp. (1961), considers the strength of the mark, the similarity of the marks, the proximity of the products, the likelihood that the owner will bridge the gap into the defendant's market, evidence of actual confusion, the defendant's good faith, the quality of the defendant's product and the sophistication of buyers. The other circuits use similar lists. In trade dress cases, the similarity inquiry compares overall impressions as encountered in the marketplace, not a side-by-side checklist. The defendant's use of its own brand name matters a great deal. A prominent label on a knockoff can dispel confusion at the point of sale, and many fast-fashion copies escape liability for exactly that reason: the buyer knows perfectly well that the product came from a mass retailer, not a luxury house. Courts have been reluctant to treat labels as a complete defense, however, especially where the label is small or hidden and the design itself is the dominant impression. Post-sale confusion Fashion has pushed the courts to recognize a form of confusion that has little to do with the purchaser. Luxury goods are bought partly to be seen. The buyer of a knockoff may know exactly what it is, but people who see the product being worn or carried may not. If observers take a copy for the genuine article, the brand may suffer harm in two ways: its reputation may suffer if the copy is of poor quality, and the exclusivity that gives a luxury good much of its value may erode as copies proliferate. The Second Circuit recognized post-sale confusion in the 1950s, in a case involving copies of a distinctive clock, and it has applied the doctrine repeatedly to fashion goods. Other circuits have followed, though with varying degrees of enthusiasm. The doctrine has obvious appeal for luxury brands, but it also raises concerns. If post-sale confusion is taken too far, trade dress law begins to protect exclusivity itself rather than consumers' ability to identify sources, and courts have tried to confine the doctrine to cases where the confusion is genuine and where the defendant's product is aimed at capitalizing on it. In the Thom Browne litigation, the adequacy of the post-sale confusion instruction was one of adidas's main grounds for appeal, and the Second Circuit's rejection of that argument confirms that the doctrine, while real, does not guarantee a result. Dilution Famous marks also enjoy protection against dilution under Section 43(c) of the Lanham Act, which does not require confusion. The owner of a famous mark may obtain an injunction against a use that is likely to cause dilution by blurring, which impairs the mark's distinctiveness, or by tarnishment, which harms its reputation. Trade dress can be the basis of a dilution claim, but the statute imposes stringent conditions: for unregistered trade dress, the claimant must prove that the dress, taken as a whole, is not functional and is famous, and that any registered marks it includes are separately famous. Fame means wide recognition by the general consuming public of the United States, not recognition within a niche. Only a handful of fashion designs are likely to qualify, but for those that do, dilution offers a route to relief even against sellers whose products cause no confusion at all. What the limits mean The limits examined in this chapter are not obstacles that a well-advised brand can simply engineer around. They are the structure that keeps trade dress from becoming a design monopoly. Functionality keeps useful and aesthetically necessary features open to competitors. Genericness keeps category-defining shapes open. The confusion requirement means that even a protected design can be copied if the copy is clearly labeled and marketed in a way that makes its source obvious. What remains protected is narrow: a specific, recognized signature, used in a specific configuration, in circumstances where copying would deceive. That narrowness is deliberate, and it is the reason most knockoffs, the subject of the next chapter, are lawful. Chapter 4: Knockoffs, Dupes and the Fast-Fashion Business Model Copying is not a deviation from the fashion business. It is one of its engines. Department stores in the mid-twentieth century sent buyers to the Paris shows and had line-for-line copies of couture gowns made for their customers under license, and less scrupulous firms made the same copies without one. The ready-to-wear industry grew up translating runway designs into affordable garments. What has changed in the last two decades is speed and scale. A fast-fashion retailer can move from spotting a design to selling a close copy in a few weeks, and the ultra-fast-fashion platforms that emerged in the late 2010s claim to add thousands of new styles each day. Social media has made copying visible and has given it a name that consumers use proudly: the dupe. This chapter sorts the legal categories that the industry blurs and applies the doctrines of the previous chapters to the knockoff economy. It then examines the litigation that has tested the boundaries in recent years, from the stripe disputes against Forever 21 to designers' claims against Shein, Lululemon's suit against Costco and the new questions raised by digital goods and parody. Three kinds of copy The law distinguishes three kinds of copy, and the distinctions determine liability, remedies and even whether the government can intervene. Table 2 summarizes them. Table 2. How US law treats different kinds of fashion copy. Category What is copied Typical legal status Main claims available Remedial exposure Counterfeit Registered mark itself, identical or substantially indistinguishable Unlawful Counterfeiting, infringement; criminal liability possible Statutory damages up to $2 million per mark per type of goods if willful; seizure; treble damages Infringing knockoff Design plus protected elements such as trade dress, prints or patented design Unlawful to the extent protected elements are taken Trade dress, copyright, design patent, dilution Actual damages, profits, injunction Lawful knockoff or dupe Unprotected style, silhouette, trend or function Lawful, subject to false advertising and confusion rules Usually none; sometimes false advertising or unfair competition None, if properly labeled and marketed The counterfeit is the easiest case. It carries the brand owner's mark or something indistinguishable from it, and it is sold to make the buyer or the public believe it is genuine. Chapters 5 and 6 deal with it at length. The infringing knockoff does not carry the brand's name but copies something the law protects: a fabric print, a design patent, a trade dress with secondary meaning, or a sufficiently famous design to support a dilution claim. Whether a given knockoff falls into this category depends on the analysis described in Chapters 1 through 3. The lawful knockoff copies the style but none of the protected elements. It carries its own label and makes no claim to be anything other than what it is. As Chapter 1 explained, the default American rule is that copying unprotected design is not only permitted but part of the competition that the law is designed to encourage. The dupe is not a separate legal category but a marketing practice. A dupe is a product sold as an affordable alternative to a named original. Its legality depends on the same questions as any knockoff, with one difference: the seller has chosen to invoke the original's name. Comparative references to a competitor's product are generally lawful under the doctrine of nominative fair use, which allows the use of another's mark to identify that party's product so long as the use is necessary, no more than reasonably needed and does not suggest sponsorship. A retailer can say its leggings are similar to a famous brand's leggings. It cannot suggest that its leggings are made by that brand, and it cannot make false claims about how its product compares. Fast fashion in court: the Forever 21 era For most of the 2000s and 2010s, the retailer most often named in design-copying lawsuits was Forever 21. It was sued dozens of times by designers alleging copied prints and designs, and most of those cases settled without published rulings. The patterns of the litigation illustrate the protection gap in operation. Claims based on fabric prints, where copyright is available, were common and often successful enough to produce settlements. Claims based on garment design alone rarely got far. Several high-profile disputes involved major brands. In 2017 Gucci and Forever 21 sued each other over Forever 21's use of blue-red-blue and green-red-green stripes, which Gucci claimed as trademarks. Forever 21 sought declaratory relief that the stripes were not protectable, and Gucci responded with infringement claims. The dispute was resolved without a decision on the merits. The same year, Puma sued Forever 21, alleging that it copied footwear designed with the singer Rihanna, asserting design patents, trade dress and copyright. That case, too, ended without a trial. adidas also pursued Forever 21 over striped designs. None of these disputes produced a clear precedent, but together they showed that major brands will pursue fast-fashion retailers when the copying reaches their signature elements, and that retailers generally prefer settlement to testing the limits of trade dress in court. Forever 21 filed for bankruptcy in 2019, and its U.S. operating company filed again in 2025 and moved to wind down its stores. Its collapse had many causes, most of them commercial, but it removed a familiar defendant from the landscape just as a new generation of competitors was taking its place. Shein and the algorithmic copy The most significant recent copying litigation has been against Shein, the ultra-fast-fashion platform founded in China, which grew in the late 2010s and early 2020s into one of the world's largest apparel sellers. Shein has been sued by numerous independent designers and artists alleging that their prints and designs appeared on Shein products, and it has settled many such claims. In July 2023 three designers, Krista Perry, Larissa Martinez and Jay Baron, filed Perry v. Shein Distribution Corp. in the Central District of California. The complaint asserted copyright infringement and, unusually, claims under the Racketeer Influenced and Corrupt Organizations Act. The designers alleged that Shein's copying was not a series of isolated mistakes but the product of a business model: an algorithm that identified trending designs online and generated close copies of works Shein did not own, together with a corporate structure that allegedly made liability difficult to pin on any single entity. The RICO claims rested on predicate acts of criminal copyright infringement and wire fraud. On 8 November 2024, Judge Mark Scarsi denied Shein's motion to dismiss the RICO claims, allowing the case to proceed on the theory that systematic infringement could constitute a pattern of racketeering activity. The ruling was a procedural one; it did not find that Shein had done anything, only that the allegations, if proven, could support the claims. The case was later reported to have settled on confidential terms. Its significance lies in the legal theory. A plaintiff that can plausibly allege that copying is embedded in a defendant's operating model may be able to reach the business model itself, with treble damages and the reputational weight of a racketeering claim, rather than litigating one design at a time. The Shein litigation also illustrates the limits of design-by-design enforcement. A small designer whose print is copied can bring a copyright claim, but the costs of federal litigation are high, statutory damages depend on timely registration, and the defendant can replace one infringing product with another before the case is decided. Aggregating claims, whether through RICO, class actions or coordinated litigation by many designers, is one response. Regulation of the platforms themselves, discussed in Chapters 6 and 8, is another. Dupes and the mass retailer: Lululemon and UGG If the Shein litigation concerned copying of small designers' prints, two recent cases concerned copying of major brands' signature products by respectable mass retailers, and both show the limits of trade dress in the dupe economy. In June 2025 Lululemon sued Costco in the Central District of California, alleging that Costco sold products copying the designs of Lululemon's Scuba hoodies, Define jackets and ABC pants, among others, under Costco's own and third-party labels. Lululemon asserted trade dress, trademarks and design patents, and it alleged that some customers believed Costco's products were authentic Lululemon apparel, pointing to the online dupe culture that promoted the Costco items as alternatives. The case attracted attention because it was brought against a mainstream retailer rather than a counterfeiter, and because it squarely raised the question whether the popularity of a dupe could itself be evidence of confusion. In 2026 the parties settled most of the claims, leaving only a dispute over a single men's jacket pending as of mid-2026. The Deckers litigation against Quince, discussed in Chapters 1 and 3, produced a clearer result. Quince openly marketed its products as offering the quality of luxury goods at lower prices. Deckers asserted trade dress and a design patent covering its short sheepskin boots. The court held the trade dress generic, and a jury found the design patent invalid even though it found that Quince's boot infringed it. Before trial, the court also refused to prevent Quince from presenting evidence about the dupe culture in which its products were sold, in effect allowing the defense to argue that consumers who knowingly buy an alternative are not confused about its source. These cases point to a working conclusion about dupes. A dupe that copies an unprotected design and is clearly identified as an alternative is likely to be lawful, and the dupe label may even help the defense by showing that consumers know they are buying something else. A dupe that copies a protected signature, or that is marketed in ways that blur its source, may still infringe. What the "dupe" label does not do is change the underlying analysis. The law asks the same questions it always has: what was copied, is it protected, and are consumers likely to be confused. Parody, art and the First Amendment A separate line of cases concerns products that copy a famous design not to compete with it but to comment on it, and here the Supreme Court's decision in Jack Daniel's Properties, Inc. v. VIP Products LLC (2023) has reshaped the law. For decades, many courts had applied the Rogers v. Grimaldi test, developed by the Second Circuit in 1989, to trademark claims involving expressive works. Under Rogers, the use of a mark in an expressive work was actionable only if it had no artistic relevance to the work or explicitly misled as to source. VIP Products sold a dog toy shaped like a Jack Daniel's whiskey bottle, labeled "Bad Spaniels," and argued that its toy was a parody protected by Rogers. The Supreme Court unanimously held that Rogers does not apply when the defendant uses the plaintiff's mark as a designation of source for its own goods. In those cases, the ordinary likelihood-of-confusion analysis applies, although parody can still be relevant to whether confusion is likely. The Court also held that the Lanham Act's exclusion of noncommercial uses from dilution liability does not protect a parody used as a source identifier. The decision had immediate consequences for fashion. In Vans, Inc. v. MSCHF Product Studio, Inc. (2023), the Second Circuit applied Jack Daniel's to the "Wavy Baby" sneakers produced by the art collective MSCHF, which distorted the design of Vans's Old Skool shoe and its packaging. MSCHF argued that the shoes were artistic commentary on consumer culture. The court held that MSCHF used Vans's trade dress as a source identifier for its own sneakers, so Rogers did not apply, and it affirmed a preliminary injunction for Vans. The parody argument did not disappear, but it became one factor in the confusion analysis rather than a threshold defense. The litigation over Mason Rothschild's "MetaBirkins" shows the other side of the same issue. In 2021 Rothschild created and sold one hundred non-fungible tokens linked to digital images of fur-covered Birkin bags. Hermès sued in the Southern District of New York. The court declined to dismiss the claims and applied Rogers, holding that the question whether the MetaBirkins were artistic works or products that exploited the Birkin mark was for the jury. In February 2023, before the Supreme Court decided Jack Daniel's, the jury found Rothschild liable for trademark infringement, dilution and cybersquatting and awarded Hermès damages of $133,000. The court later entered a permanent injunction, and Rothschild appealed to the Second Circuit. The case was the first major jury verdict on trademarks in digital fashion goods, and it confirmed that an artistic framing does not immunize the sale of products that trade on a brand's name and design. What a designer can do before the copy appears Because the law offers so little after the fact, the most effective protections against copying are arranged before a design reaches the market. Several are routine for established houses and within reach of small designers. The first is to register what can be registered. Original prints and graphics should be registered with the Copyright Office promptly, ideally before publication or within three months after it, because statutory damages and attorney fees are available only for works registered before the infringement began or within that three-month window. Without them, a small designer's recovery is limited to actual damages and the infringer's profits, which may be modest and hard to prove, and litigation rarely makes economic sense. Registration is inexpensive, and group registration options allow designers to register collections of unpublished works together. The second is to use the European design system where the product will be sold. A registered EU design can be obtained quickly and cheaply, and the unregistered EU design arises automatically on first disclosure within the Union. A designer who shows a collection first in Paris or Milan, rather than New York, may acquire three years of protection against deliberate copying across the Union without filing anything, though the rules on where first disclosure occurs deserve careful attention. The third is contractual. A large share of copying originates not with competitors but with suppliers. Factories that produce a design may sell overruns or produce the same design for other customers, and pattern makers and sample rooms can leak designs before launch. Agreements with manufacturers should assign all rights in patterns and samples to the brand, prohibit production for others, require destruction or return of materials at the end of a run and provide for audits. These terms are only as good as the brand's ability to enforce them abroad, but they convert what would otherwise be lawful copying into a breach of contract. The fourth is evidence. Designers who date and preserve sketches, technical packages, sample records and first-sale documentation are in a far better position to prove originality, priority and copying if a dispute arises. In copyright cases, proof of access and similarity often decides the result, and the timeline of a design's development is central to both. None of these steps will stop a determined competitor from copying an unprotected silhouette. They ensure that when copying does take something the law protects, the designer can enforce the right at a cost proportionate to the harm. What the knockoff cases teach The litigation of the past decade supports three conclusions. First, most fashion copying remains lawful, and the growth of fast fashion and dupe culture has not changed that. The law protects signatures, not styles, and a copyist that takes only the style, labels its product honestly and avoids protected prints and patented designs will usually escape liability. Second, the legal risk rises sharply when copying is systematic, when it takes the elements a brand has invested in as signatures, or when it is marketed in ways that confuse consumers about source. Third, courts are increasingly willing to look at the business model rather than the individual product, whether through RICO theories, the treatment of dupe marketing as evidence, or the question whether a supposedly artistic project is in reality a product line. The same shift toward looking at systems rather than individual items is even more pronounced in the fight against counterfeiting, to which the book now turns. Chapter 5: Counterfeiting: The Law's Heaviest Weapons A knockoff borrows a look. A counterfeit steals an identity. That difference explains why American law treats counterfeiting far more harshly than any other form of copying in fashion. The counterfeiter does not merely compete with the brand; it lies to buyers and to the public about who made the product, captures value built by someone else and, in the view of the enforcement agencies, often funds organized crime, exploits labor and evades taxes. The remedies Congress has provided reflect that view. They include statutory damages that can reach millions of dollars without proof of actual harm, ex parte seizures of goods and records, mandatory trebling of damages in many cases and criminal penalties including imprisonment. This chapter sets out the legal framework: what counts as a counterfeit, what remedies are available, and how courts have handled the harder questions of liability for the landlords, marketplaces, payment processors and resellers through which counterfeits reach buyers. The next chapter addresses the enforcement machinery that has grown up around online counterfeiting. What counts as a counterfeit The Lanham Act defines a counterfeit as a spurious mark that is identical with, or substantially indistinguishable from, a registered mark. Two features of the definition matter in practice. First, counterfeiting remedies are available only for marks registered on the Principal Register, and only when the counterfeit is used on the same kind of goods for which the mark is registered. A brand relying solely on unregistered trade dress can sue for infringement under Section 43(a) but cannot invoke the special counterfeiting remedies. This is one reason fashion houses register not only their names and logos but also monograms, patterns, color marks and product shapes. Registration converts what would otherwise be an ordinary infringement case into a counterfeiting case. Second, the standard is visual and demanding. A counterfeit need not be a perfect replica, but it must be substantially indistinguishable from the registered mark. A handbag bearing a monogram with the letters slightly altered, or a pair of sneakers bearing a swoosh facing the wrong way, may infringe without being counterfeit. Courts decide the question by comparing the marks as a consumer would see them on the goods. Genuine goods cannot be counterfeit, even when sold by someone other than the brand. Under the first-sale doctrine, a lawful purchaser may resell genuine branded goods without infringing, so long as the goods are not materially different from what the brand sells. The resale market, now a significant sector of the luxury economy, depends on that doctrine. But the doctrine has limits. Goods that were never authorized for sale, such as factory rejects or items stolen from a supply chain, are not genuine in the relevant sense. Goods that have been altered, repackaged or stripped of quality control mechanisms may be materially different. And resellers who suggest an affiliation with the brand, or who fail to detect counterfeits in their inventory, may be liable. The dispute between Chanel and the resale company What Goes Around Comes Around illustrates the boundaries. Chanel alleged that the reseller had sold counterfeit Chanel goods, sold items that had never been authorized for sale, including handbags whose serial numbers corresponded to stolen or unauthorized units, and used Chanel's marks in advertising in a way that suggested affiliation. In February 2024 a jury in the Southern District of New York found for Chanel and awarded $4 million in statutory damages. The verdict did not question the legitimacy of luxury resale, but it did make clear that resellers who authenticate goods are responsible when their authentication fails, and that they may not borrow a brand's prestige to suggest endorsement. Civil remedies The Lanham Act gives the owner of a registered mark a set of remedies that, taken together, make counterfeiting litigation unusually attractive for plaintiffs. The first is an injunction. As in other trademark cases, courts routinely enjoin counterfeiting, and since the Trademark Modernization Act of 2020 a plaintiff that shows a violation is entitled to a rebuttable presumption of irreparable harm. The second is monetary relief. In an ordinary trademark case, a successful plaintiff may recover the defendant's profits, its own damages and the costs of the action, subject to principles of equity. In a counterfeiting case involving intentional use, the statute directs the court to award treble damages or profits, whichever is greater, together with reasonable attorney fees, unless it finds extenuating circumstances. Alternatively, the plaintiff may elect, at any time before final judgment, to recover statutory damages of not less than $1,000 or more than $200,000 per counterfeit mark per type of goods sold, or, if the court finds that the use was willful, up to $2,000,000 per counterfeit mark per type of goods. Statutory damages are the engine of modern counterfeiting litigation. Counterfeiters rarely keep reliable records, and many disappear before discovery. Statutory damages allow courts to award substantial sums without proof of sales, and courts often treat default by the defendant as evidence of willfulness. Awards in default judgments against online sellers frequently run into the hundreds of thousands or millions of dollars, though collection is another matter. Two recent Supreme Court decisions have shaped the scope of monetary relief in trademark cases generally. In Romag Fasteners, Inc. v. Fossil, Inc. (2020), the Court held that a plaintiff need not show willful infringement as a precondition to recovering the defendant's profits. Romag made magnetic snap fasteners used in Fossil handbags and alleged that Fossil's manufacturer had used counterfeit fasteners. The Court held that willfulness is an important consideration in the equitable decision whether to award profits, but not an absolute prerequisite. In Dewberry Group, Inc. v. Dewberry Engineers Inc. (2025), the Court held that the "defendant's profits" recoverable under the Lanham Act are those of the named defendant, not those of its separately incorporated affiliates, unless the plaintiff can justify disregarding corporate separateness. The case did not involve fashion, but it matters to counterfeiting and infringement suits against groups that sell through affiliated entities, because it requires plaintiffs to name the right defendants or pierce the corporate veil. Ex parte seizure The third civil remedy is unique to counterfeiting. Section 34(d) of the Lanham Act authorizes a court, on an application made without notice to the defendant, to order the seizure of counterfeit goods, the means of making them and records documenting their manufacture and sale. The procedure exists because counterfeiters given notice of a lawsuit tend to move or destroy their inventory. Because ex parte seizure is so intrusive, the statute imposes strict conditions. The applicant must show that an order other than seizure would be inadequate, that the applicant is likely to succeed in showing counterfeiting, that immediate and irreparable injury will occur without seizure, that the harm to the applicant outweighs the harm to the defendant, that the goods are at the location identified, and that the defendant would destroy, move or hide the goods if given notice. The applicant must post security, and the seizure must be carried out by law enforcement officers. A defendant injured by a wrongful seizure may recover damages. Ex parte seizure was designed for physical counterfeiting: warehouses in industrial districts, basement workshops, stalls in street markets. Brands used it heavily in the 1980s and 1990s, often coordinating raids with local police. It remains available but has become less central as counterfeiting has moved online. The modern equivalent, the ex parte temporary restraining order freezing a seller's online assets and suspending its storefronts, borrows the logic of seizure without the statutory machinery, and it is examined in Chapter 6. Criminal liability The Trademark Counterfeiting Act of 1984 made trafficking in counterfeit goods a federal crime. Under 18 U.S.C. § 2320, a person who intentionally traffics in goods and knowingly uses a counterfeit mark on or in connection with them may be imprisoned for up to ten years and fined up to $2 million for a first offense, with higher penalties for repeat offenders, for offenses causing serious bodily injury or death, and for counterfeit military goods and drugs. The statute also authorizes forfeiture and restitution. Criminal prosecutions of fashion counterfeiters are relatively rare compared with the volume of counterfeiting, and they tend to focus on large importers and distributors rather than individual sellers. Federal prosecutors in New York, Los Angeles and other port cities have brought cases against organized groups importing counterfeit luxury goods in shipping containers, sometimes involving hundreds of millions of dollars in claimed retail value. Customs and Border Protection and Homeland Security Investigations, through the National Intellectual Property Rights Coordination Center, lead most of this work. Contributory liability: the landlord and the flea market A brand that sues the individual counterfeiter often finds that the defendant has no assets, no fixed address and a new identity by the time the case is decided. Brands have therefore long sought to hold liable the intermediaries without whom counterfeiters could not operate. The doctrine that makes this possible is contributory trademark infringement, which the Supreme Court recognized in Inwood Laboratories (1982): a party is liable if it intentionally induces another to infringe or if it continues to supply its product or service to one whom it knows or has reason to know is engaging in infringement. The courts first extended Inwood beyond manufacturers to landlords and market operators. In Hard Rock Cafe Licensing Corp. v. Concession Services, Inc. (1992), the Seventh Circuit held that the operator of a flea market could be liable for counterfeiting by vendors if it knew or had reason to know of the infringement, and that willful blindness would count as knowledge, but that it had no affirmative duty to take precautions against counterfeiting it did not know about. The Ninth Circuit reached a similar conclusion in Fonovisa, Inc. v. Cherry Auction, Inc. (1996), a case mainly about copyright. Landlord liability became a staple of fashion enforcement in New York, where Canal Street was for decades the most famous counterfeit market in the country. The Second Circuit's decision in Omega SA v. 375 Canal, LLC (2021) confirmed that a landlord who is aware of specific counterfeiting on its premises and fails to take reasonable steps to stop it may be held contributorily liable. Omega had sent investigators to the building, which had a long history of counterfeiting, and had notified the landlord. The jury found the landlord liable and awarded statutory damages, and the court of appeals affirmed, rejecting the landlord's argument that it lacked knowledge of the specific counterfeiter. Brands have used the threat of landlord liability to push property owners to evict counterfeit sellers, often through settlements requiring landlords to adopt enforcement programs. Online marketplaces and service providers The central question of the internet era has been how contributory liability applies to online marketplaces. The leading case is still Tiffany (NJ) Inc. v. eBay Inc. (2010). Tiffany alleged that eBay was contributorily liable for the sale of counterfeit Tiffany jewelry on its site, pointing to evidence that a large share of "Tiffany" goods on eBay were fake. The Second Circuit held that generalized knowledge that counterfeits were being sold was not enough. To be liable, eBay needed contemporary knowledge of which particular listings were infringing or would infringe in the future. Because eBay promptly removed listings when Tiffany reported them and took other anti-counterfeiting measures, it was not liable. The court left open the possibility that a marketplace could be liable for willful blindness if it deliberately avoided learning of specific infringement. Tiffany v. eBay has shaped online enforcement for more than fifteen years. It effectively established a notice-and-takedown system for trademarks, even though no statute requires one, and it placed the burden of monitoring on brands rather than platforms. Platforms that respond to specific notices and adopt reasonable anti-counterfeiting programs are generally safe. Platforms that ignore notices, or that actively facilitate infringement, are not. The Ninth Circuit's decision in Louis Vuitton Malletier, S.A. v. Akanoc Solutions, Inc. (2011) illustrates the other side of the line. The defendants provided web hosting services to websites selling counterfeit Louis Vuitton goods. Louis Vuitton sent numerous notices, but the defendants did not take down the sites. A jury found the hosts contributorily liable, and the court of appeals affirmed the finding of liability. Hosts that know of specific infringement and continue to provide the service that makes it possible can be liable, even if they did not sell anything themselves. A later dispute showed how marketplaces that take custody of goods may face direct rather than contributory liability. Nike sued the resale platform StockX in the Southern District of New York in 2022, first over digital tokens StockX linked to physical sneakers and later over counterfeit sneakers sold through the platform despite StockX's authentication program. In March 2025 Judge Valerie Caproni granted Nike summary judgment on its counterfeiting claim, finding StockX liable for the sale of thirty-seven pairs of counterfeit sneakers, including pairs purchased by Nike's investigators. The parties settled the remaining claims in August 2025. The ruling underscored a point for any marketplace that authenticates goods and sells them in its own name: it may be treated as a seller, not merely a venue, and its authentication promises may become a basis for liability. Territorial limits Counterfeiting is a global business, but the Lanham Act is a domestic statute. In Abitron Austria GmbH v. Hetronic International, Inc. (2023), the Supreme Court held that the Lanham Act's infringement provisions apply only when the infringing use in commerce occurs in the United States. The case involved radio remote controls for construction equipment, but the rule applies across industries. A brand cannot use the Lanham Act to recover for counterfeits made and sold abroad, even by a defendant subject to American jurisdiction, unless the infringing use occurs domestically. Foreign counterfeiting must be pursued under foreign law, or intercepted when the goods arrive at American borders. The architecture of liability Taken together, these rules produce a layered structure of liability. At the center is the counterfeiter, exposed to crushing statutory damages and criminal prosecution but often beyond practical reach. Around it are intermediaries, landlords, hosts, marketplaces and resellers, who are liable if they know of specific counterfeiting and fail to act, or if they sell counterfeits themselves. And at the outer edge are platforms and services that are generally protected if they respond to notices and take reasonable precautions. Most enforcement in practice consists of moving pressure outward from the unreachable center to the reachable intermediaries, and the next chapter describes the tools brands and governments use to do that at scale. Chapter 6: The Enforcement Machine: Schedule A Suits, Platforms and the Border Counterfeiting law was written for a world of warehouses and street markets. The counterfeiter had a physical location, the goods moved in containers, and a raid could seize both. The counterfeiting that fashion brands face today looks very different. A seller in Guangdong or Karachi opens a storefront on a global marketplace under a nonsense name, lists a few hundred handbags bearing a famous monogram, ships each one by post or air courier directly to a buyer's door, collects payment through a platform's processor, and, when the storefront is shut down, opens another. No single sale is large. No inventory sits in the United States. The seller may never set foot in the country. The legal system has adapted to this model not by rewriting the statutes described in Chapter 5 but by building new enforcement machinery around them. Three parts of that machinery matter most: the mass-defendant lawsuits known as Schedule A cases, the obligations imposed on online marketplaces, and the border controls administered by Customs and Border Protection. Each has changed significantly since 2023. Table 3 compares them. Table 3. The main channels for enforcing against online counterfeit sellers in the United States. Channel Who acts Main legal basis Speed Principal limits Schedule A lawsuit Brand owner in federal court Lanham Act; Rule 65 ex parte relief Days to weeks for a TRO Growing judicial scrutiny of joinder, asset freezes and specificity Platform takedown Marketplace on brand's notice Platform policy shaped by Tiffany v. eBay; INFORM Consumers Act Hours to days Sellers reappear under new identities Customs detention and seizure CBP on recorded marks Tariff Act; Lanham Act; recordation rules At entry Volume of small parcels; limited inspection capacity Criminal prosecution Federal prosecutors 18 U.S.C. § 2320 Months to years Resources; offshore defendants Schedule A litigation Beginning in the early 2010s, lawyers representing brand owners developed a litigation model aimed at online sellers. A single complaint named as defendants dozens or hundreds of online storefronts, identified only by their seller names and listed in an attached schedule, which gave the cases their name. The complaint alleged that the defendants were all selling counterfeits of the plaintiff's products and asked the court for an ex parte temporary restraining order that would suspend the storefronts, freeze the funds in the sellers' accounts at payment processors and marketplaces, and authorize service of process by email or website publication. The complaint and schedule were often filed under seal so that the defendants would not be warned before their assets were frozen. Once the order was entered, the marketplaces and payment processors froze the accounts. Many defendants never appeared, and the plaintiff obtained default judgments with statutory damages, collecting from the frozen funds. Others settled quickly to unfreeze their accounts. The Northern District of Illinois in Chicago became the center of this litigation, handling thousands of Schedule A cases a year at its peak. Brands of every kind used the model, from luxury houses to toy companies to owners of design patents on household products. The model had obvious advantages for brands. It was fast, cheap per defendant, and effective at reaching sellers who were otherwise beyond the reach of American courts. It also had obvious problems. Joining hundreds of unrelated sellers in a single case, on the theory that they were all part of the same series of transactions, stretched the joinder rules. Freezing all of a seller's funds, including proceeds from sales unrelated to the plaintiff's products, was difficult to justify when the plaintiff sought statutory damages, a legal remedy, rather than an equitable accounting of profits. Service by email raised due process and treaty questions when the defendants were located in China, a party to the Hague Service Convention. And sellers of genuine goods, or of products that did not infringe at all, were sometimes swept up with counterfeiters and forced to litigate in a distant forum to recover their funds. Beginning around 2023, judges in the Northern District of Illinois began to push back. Some refused to allow sealing of the defendant schedule, reasoning that the public has a right to know who is being sued. Some severed defendants for misjoinder, requiring the plaintiff to show that the sellers were actually connected rather than merely selling similar products. Some limited asset freezes to the amount of the plaintiff's plausible recovery. And some required individualized evidence of infringement for each defendant before granting relief. The most sweeping criticism came in 2025, in a ruling by Judge John Kness in a case brought by Eicher Motors, the owner of the Royal Enfield motorcycle marks, against a schedule of online sellers. Judge Kness held that the standard Schedule A request failed the requirements of Federal Rule of Civil Procedure 65 because it did not demonstrate harm as to each individual defendant, that a prejudgment asset freeze was not justified when the plaintiff was pursuing statutory damages rather than equitable relief, and that the ex parte procedure made it impossible to weigh the public interest reliably without the defendants' participation. He described the model as creating conditions for abusive and extortionate outcomes. Other judges in the district had imposed narrower limits, and the Seventh Circuit has not yet ruled on the model's validity. Schedule A litigation has not disappeared. Plaintiffs have adapted, filing smaller cases, providing individualized evidence, filing in other districts and seeking narrower relief. But the era in which a brand could freeze the accounts of hundreds of sellers with a single sealed motion and a spreadsheet has largely ended in Chicago. The lesson for brands is that aggressive procedures built for efficiency invite scrutiny when they sweep up the innocent, and that judicial tolerance for such procedures depends on their being used carefully. The platforms' obligations Under Tiffany v. eBay, platforms that respond to notices of specific infringement are generally shielded from contributory liability. Brands have long argued that this standard asks too little of marketplaces that profit from every sale, including sales of counterfeits. Congress has considered imposing stricter obligations but so far has adopted only a narrow transparency measure. The INFORM Consumers Act, enacted in December 2022 and effective in June 2023, requires online marketplaces to collect, verify and in some cases disclose information about high-volume third-party sellers. A high-volume seller is one that has made two hundred or more discrete sales totaling $5,000 or more in gross revenue in a continuous twelve-month period during the previous two years. Marketplaces must collect the seller's bank account information, contact information and tax identification number, verify it, and require sellers to certify annually that it is current. For sellers with $20,000 or more in annual gross revenue on the marketplace, the platform must disclose the seller's name, physical address and contact information to consumers, subject to limited exceptions. Marketplaces must suspend sellers who fail to comply. The Federal Trade Commission and state attorneys general enforce the statute. The INFORM Act does not make platforms liable for counterfeits. It addresses anonymity, which is the counterfeiter's most important asset. By requiring platforms to know who their sellers are, it makes it harder for a seller to disappear after one storefront is shut down and easier for brands and law enforcement to identify repeat offenders. A broader measure, the SHOP SAFE Act, has been introduced in several Congresses. It would make platforms contributorily liable for counterfeits of goods that implicate health and safety unless they adopt specified best practices, including verifying sellers, requiring them to use images of goods they actually own, removing listings promptly on notice and terminating repeat offenders. It has not been enacted. In the meantime, the major marketplaces have adopted voluntary programs that go beyond the Tiffany standard. Amazon, for example, operates brand registry and proactive detection programs and has joined brands as co-plaintiffs in lawsuits against counterfeit sellers. These programs are useful, but they are voluntary and differ from platform to platform, and brands continue to argue that legal obligations are needed. The European Union has gone further. The Digital Services Act, which applies fully since February 2024, requires online marketplaces to verify the traders who use them, design their interfaces so that traders can provide required product information, make reasonable efforts to check whether products are legal, and inform consumers when they have purchased an illegal product. It also creates a system of "trusted flaggers," whose notices must be handled with priority. Very large online platforms face additional risk-assessment and audit obligations. The regime does not impose general monitoring duties, but it moves the default well beyond notice and takedown. Following the money Counterfeiters sell online to be paid, and payment systems have become a further point of pressure. The legal basis for reaching them is again contributory liability. In Gucci America, Inc. v. Frontline Processing Corp. (2010), Gucci sued companies that provided credit card processing services to websites selling counterfeit Gucci goods. The Southern District of New York held that the processors could be contributorily liable if they knew or had reason to know that the websites were selling counterfeits and continued to process their payments, reasoning that without payment processing the websites could not have operated. The court also noted evidence that one processor had charged higher fees to merchants it regarded as high-risk "replica" sellers, which suggested knowledge of what they were doing. The practical consequence has been cooperation rather than litigation. The major card networks and payment platforms maintain procedures under which brand owners can report merchants selling counterfeits, and they can terminate the merchant accounts. Brands have learned that a counterfeit website that cannot take card payments is effectively out of business, and that the payment chain, unlike the seller, is concentrated in a few large, reachable firms with every incentive to avoid association with fraud. Schedule A asset freezes, at their height, worked by the same logic: the plaintiff did not need to find the seller, only the account where the seller's money sat. The border and the end of de minimis Customs enforcement has always been a central part of the fight against counterfeiting. Brand owners who register their marks with the Patent and Trademark Office can record them with Customs and Border Protection, which then has authority to detain and seize imports bearing counterfeit versions of recorded marks. CBP and Homeland Security Investigations seize tens of thousands of shipments of counterfeit goods each year, and apparel, footwear, handbags, watches and jewelry are consistently among the largest categories by value. For much of the past decade, however, the border was undermined by the de minimis rule. Section 321 of the Tariff Act allowed goods valued at $800 or less, imported by one person on one day, to enter the country free of duty and with minimal documentation. Congress raised the threshold from $200 to $800 in 2016. As direct-to-consumer e-commerce boomed, the number of de minimis shipments grew to more than a billion a year. The rule was a foundation of the business model of the ultra-fast-fashion platforms, which shipped individual packages directly from factories to consumers, and it was a gift to counterfeiters, whose goods could arrive in small parcels that CBP had neither the time nor the information to inspect. The de minimis rule has now been dismantled in stages. In 2025 the administration issued a series of executive orders, first suspending de minimis treatment for goods from China and Hong Kong in May 2025 and then, by an order issued on 30 July 2025, suspending it for all countries effective 29 August 2025. Those orders were tied in part to tariffs imposed under the International Emergency Economic Powers Act, and in February 2026, in Learning Resources, Inc. v. Trump, the Supreme Court held that the Act does not authorize the President to impose tariffs. The de minimis suspension did not fall with the tariffs. Congress had already legislated the end of the exemption in the budget reconciliation statute enacted in July 2025, commonly called the One Big Beautiful Bill Act, effective 1 July 2027. And in June 2026 CBP issued an interim final rule, relying on its independent authority under the Tariff Act to protect revenue and prevent unlawful importations, that continues the suspension for non-postal shipments until the statutory repeal takes effect. The end of de minimis matters for fashion in two ways. Commercially, it removes a significant cost advantage from platforms that ship individual parcels from overseas, and several of those platforms have responded by shifting to bulk imports and domestic warehouses. For counterfeiting enforcement, it means that every low-value shipment now requires formal entry and data that allow CBP to target suspicious parcels. Counterfeiters who once relied on the anonymity of the small parcel now face the same scrutiny as commercial importers. It is too early to measure the effect on counterfeit volumes, but the structural change is significant. The European Union handles the small-parcel problem differently. Under its customs enforcement regulation, Regulation (EU) No 608/2013, right holders file applications for action with customs authorities, which may then detain suspected infringing goods. For small consignments, the regulation provides a simplified procedure under which customs can destroy the goods without the right holder's involvement if the declarant or holder does not object within ten working days. The procedure was designed for exactly the flow of individual counterfeit parcels that de minimis encouraged in the United States. The EU has also agreed, as part of a broader customs reform, to remove its own €150 duty exemption for low-value parcels, reflecting the same concern about platforms shipping directly to consumers. A brand operating in both markets therefore needs recorded rights and applications for action on both sides of the Atlantic, and it needs to respond quickly when customs officers ask whether detained goods are genuine. What enforcement at scale requires The enforcement machinery of the 2020s shares a common logic. The counterfeiter is anonymous, offshore and able to reappear instantly. The points at which it can be reached are the platforms on which it sells, the payment systems through which it is paid and the borders through which its goods pass. Every successful enforcement tool works by pressing on one of those points. That logic has a corollary. Each enforcement tool is only as good as the information it rests on. Schedule A cases foundered when plaintiffs could not show which defendants were actually selling counterfeits. Platform takedowns depend on brands identifying specific listings. Customs detention depends on recorded marks, accurate entry data and the ability to distinguish genuine goods from fakes. The brands that enforce most effectively are those that know their own products and supply chains well enough to prove, quickly and specifically, which goods are theirs and which are not. That point connects counterfeiting enforcement to the subject of the remaining chapters. The same traceability systems that allow a brand to authenticate its products at the border are increasingly demanded by laws concerned not with counterfeits but with how genuine products are made. A brand that cannot tell customs officers where its cotton came from will struggle both to prove its goods genuine and to prove them free of forced labor. Chapter 7: Forced Labor, Human Rights and the Law of Provenance On 24 April 2013 the Rana Plaza building in Savar, on the outskirts of Dhaka, collapsed. The eight-story structure housed several garment factories producing clothing for European and American brands. Cracks had appeared in the building the day before, and workers had been ordered back inside. More than eleven hundred people were killed and thousands more injured. It remains the deadliest disaster in the history of the garment industry, and it changed the legal conversation about fashion supply chains. Before Rana Plaza, the prevailing view in the industry and in most legal systems was that brands were buyers, not employers. The factories were independent businesses in other countries, subject to local law, and a brand's responsibility for conditions within them was a matter of voluntary codes of conduct and reputation. After Rana Plaza, that view became harder to maintain. Brands signed the Accord on Fire and Building Safety in Bangladesh, a legally binding agreement with global unions that committed signatories to independent inspections, remediation and, crucially, arbitration enforceable in the courts of the signatories' home countries. The Accord's successor, the International Accord for Health and Safety in the Textile and Garment Industry, took effect in 2021 and extended the model to other countries. The Accord was a private agreement, but it anticipated a broader shift. Over the following decade, governments began to impose legal obligations on brands concerning conditions in their supply chains. This chapter examines those obligations in the United States and the European Union. They fall into four categories: import bans, disclosure requirements, civil liability and mandatory due diligence. Import bans: Section 307 and the UFLPA The oldest and most powerful American tool is Section 307 of the Tariff Act of 1930, which prohibits the importation of goods mined, produced or manufactured wholly or in part in any foreign country by forced labor, including convict labor and indentured labor under penal sanctions. For most of its history the provision was rarely enforced, in part because it contained a "consumptive demand" exception allowing imports of forced-labor goods that were not produced in the United States in sufficient quantities to meet domestic demand. Congress repealed that exception in the Trade Facilitation and Trade Enforcement Act of 2015, and enforcement increased markedly thereafter. CBP enforces Section 307 primarily through withhold release orders, which direct officers to detain shipments of specified goods from specified producers or regions when there is reasonable but not conclusive evidence of forced labor. The importer may obtain release by proving that the goods were not made with forced labor or by re-exporting them. Where CBP has conclusive evidence, it may issue a formal finding, which allows seizure and forfeiture. For the apparel industry, the most important withhold release orders concerned cotton from the Xinjiang Uyghur Autonomous Region of China. Reports by journalists, researchers and governments documented a system of mass detention and state-organized labor transfer programs affecting Uyghurs and other predominantly Muslim minorities in the region. Xinjiang produces the great majority of China's cotton, and China is one of the world's largest cotton producers and by far its largest textile and apparel exporter. In December 2020 CBP issued a withhold release order on cotton and cotton products from the Xinjiang Production and Construction Corps, a paramilitary organization that controls a large share of the region's cotton production. In January 2021 it issued a region-wide order on cotton and tomato products from Xinjiang. Congress then went further. The Uyghur Forced Labor Prevention Act, signed in December 2021 and effective in June 2022, establishes a rebuttable presumption that any goods mined, produced or manufactured wholly or in part in Xinjiang, or by an entity on a list maintained by the Forced Labor Enforcement Task Force, are made with forced labor and are therefore barred from entry. The presumption can be overcome only if the importer shows, by clear and convincing evidence, that the goods were not made with forced labor, and has complied fully with the government's guidance and responded completely to CBP's requests for information. In practice, few importers have succeeded in rebutting it. The UFLPA reversed the ordinary burden. Under Section 307, the government must develop evidence of forced labor. Under the UFLPA, the importer must prove the negative, tracing its supply chain back to the raw material and demonstrating that no stage involved Xinjiang or a listed entity. For apparel, that means tracing cotton from the finished garment through the cut-and-sew factory, the fabric mill, the spinner and the ginner to the farm. Many garments pass through several countries on the way, and cotton from Xinjiang may be spun in China and woven in Vietnam or Bangladesh before reaching an American port. The presumption applies wherever the goods are finished if any input came from Xinjiang. The statute's reach has grown with the entity list. By the law's third anniversary, 144 entities had been added. In early 2025 the Department of Homeland Security added 37 China-based companies in what it described as the largest single expansion of the list, 26 of them in the cotton sector, including Huafu Fashion Co., Ltd., a major yarn producer, and 25 of its subsidiaries. Enforcement data compiled by the Center for Strategic and International Studies show that by August 2025 CBP had detained more than 16,700 shipments valued at nearly $3.7 billion under the statute, and apparel, footwear and textiles have been among the sectors most affected. Enforcement has fluctuated, with notably low monthly detention numbers reported in mid-2025, but the statutory presumption remains in force, and an importer whose shipment is detained bears the full burden of proof. Proving that cotton is not from Xinjiang is not a matter of paperwork alone. Supply chain documents can be falsified, and cotton from different regions is blended in spinning. Brands and CBP have turned to scientific methods, including isotopic testing, which measures ratios of stable isotopes in cotton fibers that reflect the conditions in which the plant grew, and DNA tracing, which uses markers applied to fibers at the gin. These methods are imperfect but increasingly used, and CBP has used isotopic testing in its own enforcement. The practical effect is that brands importing cotton goods must be able to produce not only contracts and invoices but also evidence that can be tested. Disclosure laws A second approach relies on disclosure. The California Transparency in Supply Chains Act, effective in 2012, requires retail sellers and manufacturers doing business in California with annual worldwide gross receipts exceeding $100 million to disclose on their websites the extent, if any, to which they engage in verification of product supply chains, audits of suppliers, certification by suppliers, internal accountability standards and training, all with respect to slavery and human trafficking. The statute requires disclosure of what the company does, not that it does anything. A company may lawfully disclose that it does none of these things. The Act's limits became clear in litigation. Consumers sued several food companies, alleging that they had failed to disclose on product labels that their supply chains included forced or child labor. In Hodsdon v. Mars, Inc. (2018), the Ninth Circuit held that under California consumer protection law, Mars had no duty to disclose on its chocolate labels the risk of forced child labor in its cocoa supply chain, because the information did not concern the product's central function or a safety hazard, and because the legislature had chosen, in the Transparency in Supply Chains Act, to require disclosure through websites rather than labels. The decision confirmed that disclosure statutes, while useful, do not by themselves create liability for what they reveal. The United Kingdom's Modern Slavery Act 2015 and Australia's Modern Slavery Act 2018 impose similar reporting duties, with somewhat more detailed content requirements in the Australian case. Like the California statute, they rely on transparency and reputational pressure rather than direct sanctions for poor practices. Civil liability in American courts A third approach would hold brands liable in court for harm suffered by workers in their supply chains. American law has been largely closed to such claims. The Alien Tort Statute, which gives federal courts jurisdiction over tort suits by aliens for violations of the law of nations, was once seen as a vehicle for such claims. The Supreme Court has progressively narrowed it. In Nestlé USA, Inc. v. Doe (2021), former child slaves from Mali sued American companies that purchased cocoa from Ivorian farms, alleging that the companies aided and abetted forced labor by providing financial and technical assistance to the farms. The Court held that the plaintiffs had not alleged sufficient domestic conduct to overcome the presumption against extraterritoriality, because general corporate activity in the United States, such as making operational decisions, was not enough. After Nestlé, the statute offers little prospect for claims against brands based on conditions in overseas factories. State tort law has fared no better. In Doe v. Wal-Mart Stores, Inc. (2009), workers in factories in China, Bangladesh, Indonesia, Swaziland and Nicaragua sued Wal-Mart, alleging that its supplier code of conduct created duties toward them. The Ninth Circuit held that the code did not make the workers third-party beneficiaries of Wal-Mart's supply contracts, that Wal-Mart was not their employer, and that it owed them no duty of care to monitor the suppliers. The Trafficking Victims Protection Reauthorization Act offers a narrower path. It gives victims of forced labor a civil claim against anyone who knowingly benefits from participation in a venture that the defendant knew or should have known engaged in forced labor. In Ratha v. Phatthana Seafood Co. (2022), the Ninth Circuit held that the statute's civil provision could reach conduct abroad but found insufficient evidence that the American defendant had benefited from or participated in the venture. And in Doe v. Apple Inc. (2024), the D.C. Circuit held that technology companies that purchased cobalt through the global supply chain did not "participate in a venture" with the mines where children worked merely by buying from suppliers. Participation requires more than an ordinary buyer-seller relationship in a global market. For apparel brands, the decision confirms that the purchase of goods from suppliers who use forced labor is not by itself enough for liability, though a brand that exercises close control over a particular supplier may face greater risk. Mandatory due diligence in Europe The European Union has taken a different path: requiring companies to identify and address human rights and environmental harms in their supply chains, with enforcement by regulators and potential liability for failure. Several member states acted first. France's Duty of Vigilance Law of 2017 requires large French companies to establish, publish and implement a vigilance plan to identify and prevent serious violations of human rights and environmental harm resulting from their activities and those of their subcontractors and suppliers. Germany's Supply Chain Due Diligence Act, effective in 2023, imposes similar obligations on large companies operating in Germany, enforced by a federal agency with authority to impose fines. The Netherlands adopted a narrower law on child labor. The EU's Corporate Sustainability Due Diligence Directive, adopted in 2024, was designed to harmonize and extend these national laws. As adopted, it would have applied to EU companies with more than 1,000 employees and a worldwide turnover of more than 450 million euros, and to non-EU companies with EU turnover above the same threshold. It required in-scope companies to identify, prevent, mitigate and remediate adverse human rights and environmental impacts in their own operations, those of their subsidiaries and their "chain of activities," to adopt climate transition plans, and it created a harmonized civil liability regime. The directive was substantially narrowed before it took effect. In February 2025 the European Commission proposed an "Omnibus" simplification package, and after negotiations the amended directive was published in the Official Journal on 26 February 2026. The revised directive applies only to EU companies with more than 5,000 employees and net worldwide turnover exceeding 1.5 billion euros, and to non-EU companies with more than 1.5 billion euros in net EU turnover. The requirement to adopt climate transition plans was removed, the harmonized EU civil liability regime was dropped in favor of national law, and maximum penalties were capped at three percent of worldwide net turnover. Member states have until July 2028 to transpose the directive, and companies must comply from July 2029. The narrowing was a significant retreat, but the directive remains in force, and it will apply to the largest fashion groups selling in the EU, including many American companies. Even companies below the thresholds will feel its effects, because in-scope companies must obtain information and contractual assurances from their business partners. New York's proposal In the United States, the most ambitious proposal specific to fashion is New York's Fashion Sustainability and Social Accountability Act, first introduced in 2021 and reintroduced in later sessions. The 2025 version, amended in April 2025 as the Fashion Environmental Accountability Act, would apply to apparel and footwear sellers with more than $100 million in annual gross receipts doing business in New York. It would require them to map their supply chains across four tiers, conduct environmental due diligence, set science-based greenhouse gas reduction targets and ensure that significant dyeing and finishing suppliers meet wastewater and chemical standards, with penalties of up to $15,000 per violation per day and a remediation fund for affected communities. The bill has not been enacted, and its latest versions have shifted focus from labor to environmental obligations. But it remains the most detailed legislative model for fashion-specific due diligence in the United States, and it illustrates the kind of obligations American brands may eventually face. Provenance as a legal obligation The laws examined in this chapter differ in their mechanisms, but they converge on a single demand: brands must know where their products come from. The UFLPA requires importers to prove the origin of their cotton. The European directive requires the largest companies to map and address harms in their chains of activities. Disclosure laws require companies to describe what they do to verify their supply chains. Even the courts' reluctance to impose tort liability rests on the premise that brands are ordinary buyers, a premise that weakens as brands exercise more control over their suppliers. For most of the industry's history, the supply chain was a source of competitive advantage precisely because it was flexible and opaque. Brands could shift orders between factories and countries in response to price, and they did not need to know where their suppliers sourced their inputs. The new legal regime makes that opacity a liability. The same traceability that allows a brand to authenticate its products against counterfeits, discussed in Chapter 6, is now a legal requirement for importing them at all. Chapter 8: Environmental Mandates: Waste, Chemicals and Green Claims For most of the modern history of the fashion industry, the environmental consequences of clothing were externalities in the economist's sense: real costs borne by someone other than the people who made and sold the product. The water used to grow cotton, the chemicals discharged by dye houses, the emissions from synthetic fiber production and the millions of tons of garments discarded each year were paid for by rivers, communities, municipalities and the atmosphere, not by brands. The legal developments examined in this chapter share a single purpose: to move those costs back onto the companies that put products on the market. The mechanisms vary. Extended producer responsibility makes brands pay for the collection and processing of their products at end of life. Product regulation restricts what brands may do with unsold goods and which chemicals they may use. Consumer protection law requires that environmental claims be true. And France, in the most aggressive measure yet adopted, has targeted the ultra-fast-fashion business model directly. The law in this area is moving faster than in any other part of fashion law, and it is moving fastest in Europe. Table 4 summarizes the principal measures. Table 4. Principal environmental mandates affecting apparel, as of autumn 2026. Measure Jurisdiction Core obligation Status and key dates Responsible Textile Recovery Act (SB 707) California Producers fund collection, repair, reuse and recycling through a producer responsibility organization Enacted 2024; producers had to join the approved organization by 1 July 2026; regulations no earlier than 2028 Revised Waste Framework Directive European Union Mandatory textile EPR schemes with fees modulated by durability and recyclability In force 16 October 2025; national schemes within 30 months Ecodesign for Sustainable Products Regulation European Union Ban on destroying unsold apparel and footwear; disclosure of discarded goods Ban applies to large enterprises from 19 July 2026, medium enterprises from 2030 Empowering Consumers for the Green Transition Directive European Union Bans generic environmental claims and offset-based neutrality claims Applies from 27 September 2026 Anti-fast-fashion law France Eco-penalties and advertising ban for ultra-fast fashion Definitively adopted by Parliament on 29 June 2026; implementation pending PFAS restrictions in textiles (AB 1817) California Bars sale of textiles containing regulated PFAS above set thresholds In effect since 1 January 2025, with a lower threshold from 2027 Extended producer responsibility for textiles Extended producer responsibility, or EPR, is the principle that the producer of a product should bear financial or operational responsibility for its disposal. It has long been used for packaging, electronics, batteries and tires. Its extension to textiles is recent, and it is the single most important structural change in the environmental law of fashion. France was the pioneer. It has required textile producers to participate in an EPR scheme since 2007, financing collection and sorting through an organization now known as Refashion. For many years France was the only EU member state with such a scheme. That changed with the revision of the EU Waste Framework Directive, which entered into force on 16 October 2025. The revised directive requires every member state to establish EPR schemes for textiles, clothing, footwear and certain household textiles within thirty months of its entry into force. Producers, a term that includes brands, importers and in certain circumstances online sellers based outside the member state, must pay fees for each product they place on the market. The fees fund separate collection, sorting, reuse, preparation for reuse and recycling. Crucially, the directive requires fees to be modulated according to sustainability criteria, including durability and recyclability, so that producers of short-lived, hard-to-recycle products pay more. The directive also allows member states to consider ultra-fast-fashion practices in setting fees, a provision aimed squarely at the platforms discussed in Chapter 4. The United States has followed with California's Responsible Textile Recovery Act, enacted as Senate Bill 707 in September 2024. It is the first textile EPR law in the country. The statute covers apparel and textile articles broadly, including clothing, footwear, handbags, towels, bedding and curtains, with exclusions for items such as personal protective equipment and mattresses. It uses a cascading definition of producer: responsibility falls first on the brand owner that manufactures the product, then, if that party is not in California, on the brand licensee, the importer, the distributor and eventually the retailer. Online sellers and out-of-state companies are therefore within its reach. The statute requires producers to join a producer responsibility organization approved by CalRecycle, the state's recycling agency. CalRecycle selected Landbell USA for that role, and producers were required to join by 1 July 2026. The organization must develop a stewardship plan for collection, repair, reuse and recycling throughout the state, funded by producer fees. The timeline is long: CalRecycle's regulations are to take effect no earlier than July 2028, and plans must be approved by 2030. But once the program is operational, retailers and marketplaces may not sell covered products from producers that are not in compliance, and violations carry civil penalties of up to $10,000 per day, or $50,000 per day for knowing violations. EPR changes the economics of fashion in a way that design and marketing choices must now take into account. A brand that makes garments designed to last, to be repaired and to be recycled will pay lower fees than one that makes garments designed to be worn a few times and thrown away. For the first time, the law puts a price on a garment's life after sale. The ban on destroying unsold goods Among the most visible environmental failures of the fashion industry was the routine destruction of unsold goods. Luxury brands burned excess stock to protect exclusivity and prevent discounting. Mass-market brands landfilled or incinerated returns and overstock because it was cheaper than processing them. Media reports of the practice, including disclosures in the late 2010s that a British luxury house had destroyed tens of millions of pounds' worth of goods in a single year, provoked widespread criticism. France banned the destruction of unsold non-food products, including clothing, in its 2020 law on combating waste and the circular economy, effective from 2022, requiring that such goods be reused or recycled. The European Union has now followed. The Ecodesign for Sustainable Products Regulation, adopted in 2024, establishes a framework for setting product-specific sustainability requirements and includes a direct prohibition on destroying unsold apparel, clothing accessories and footwear. The ban applies to large enterprises from 19 July 2026 and to medium-sized enterprises from 19 July 2030. Micro and small enterprises are exempt. The prohibition has exceptions, which the Commission has specified in implementing legislation. Businesses may destroy unsold products that are dangerous, that infringe intellectual property rights, that cannot lawfully be sold, that are damaged or contaminated beyond repair, or for which no recipient can be found after reasonable efforts to donate or sell them over a defined period. The intellectual property exception matters: counterfeits seized or surrendered can still be destroyed, as the enforcement regime described in Chapters 5 and 6 contemplates. In addition to the ban, businesses must disclose annually the number and weight of unsold consumer products they discard, the reasons and the exceptions relied upon, with the first disclosures from large enterprises covering 2026. The ESPR will also bring product-specific ecodesign requirements for textiles, which the Commission has identified as a priority product group. These may include requirements on durability, recycled content and repairability, and a digital product passport carrying information about the product's composition and origin. The details remain to be set in delegated acts, but the direction is clear: in the European market, the design of a garment will increasingly be a regulated matter. Chemicals in clothing Textile production is chemically intensive, and some chemicals used in finishing persist in the product and the environment. The most prominent recent target has been per- and polyfluoroalkyl substances, known as PFAS, a family of synthetic chemicals widely used to make fabrics water-, stain- and oil-repellent. PFAS are extremely persistent, and some have been associated with health risks. California's AB 1817, enacted in 2022, prohibits the manufacture, distribution and sale in the state of new textile articles containing regulated PFAS, defined by reference to total organic fluorine above a threshold. The ban took effect on 1 January 2025 with a threshold of 100 parts per million, falling to 50 parts per million from 1 January 2027, and outdoor apparel for severe wet conditions was given a later compliance date, provided that products carry a disclosure. New York enacted a similar prohibition on PFAS in apparel, and other states have followed with their own restrictions. The European Union has also moved to restrict PFAS, and its chemicals regulation, REACH, already restricts many substances used in textiles. PFAS laws illustrate a pattern common to environmental regulation of fashion: the obligation falls on the brand, but compliance depends on information from suppliers several tiers upstream. A brand cannot know whether its fabric contains PFAS without testing or reliable certification from the mill and the finisher. As with forced labor, legal compliance requires supply chain visibility that many brands have historically lacked. Green claims The final set of environmental rules concerns what brands say about their products. As consumer interest in sustainability has grown, so have claims that garments are "sustainable," "eco-friendly," "conscious," "responsible" or made from "recycled" or "natural" materials. Regulators and courts are increasingly scrutinizing these claims. In the United States, the principal guidance is the Federal Trade Commission's Guides for the Use of Environmental Marketing Claims, known as the Green Guides, last revised in 2012. The Guides are not binding rules, but they describe how the Commission interprets the prohibition on deceptive practices in Section 5 of the FTC Act, and courts and state regulators rely on them. They caution against unqualified general environmental benefit claims such as "eco-friendly," which are difficult to substantiate, and require that claims about recycled content, recyclability and biodegradability be specific and supported by evidence. The Commission began a review of the Guides in December 2022, but revised Guides had not been issued as of the time of writing. The FTC's most prominent fashion enforcement has concerned textiles described as bamboo. Most "bamboo" textiles are in fact rayon made from bamboo through a chemical process, and the Textile Fiber Products Identification Act requires that fibers be identified by their generic names. In 2022, Kohl's and Walmart agreed to pay civil penalties of $2.5 million and $3 million, respectively, to settle FTC charges that they had marketed rayon textiles as bamboo and made related environmental claims. The cases showed that the FTC could obtain penalties for textile misrepresentations under statutes that carry them, even though Section 5 alone does not. Private litigation has produced mixed results. Consumers sued H&M in 2022 over its "Conscious Choice" collection and the environmental "scorecards" it displayed for products, alleging that they were misleading. One case, Lizama v. H&M, was dismissed in 2023 because H&M had not claimed that its products were environmentally friendly, only that they contained more sustainable materials, and had disclosed the details on its website. Another was voluntarily dismissed by the plaintiffs. Other cases against brands over "sustainable" and "recycled" claims have faced similar difficulties in showing that reasonable consumers were misled. American courts have generally been reluctant to find qualified or comparative sustainability claims deceptive. European regulators have been more aggressive. The Norwegian Consumer Authority, the Netherlands Authority for Consumers and Markets and the United Kingdom's Competition and Markets Authority have all challenged fashion brands' sustainability claims. In 2024, the CMA obtained undertakings from ASOS, Boohoo and George at Asda to ensure that their environmental claims were accurate and not misleading, after an investigation into their "eco" and "responsible" product ranges. Those undertakings required the companies to describe the environmental benefits of products specifically, avoid vague terms and ensure that any filters or labels were based on clear criteria. The EU has now legislated. The Empowering Consumers for the Green Transition Directive, adopted in 2024 and applicable from 27 September 2026, adds to the EU's list of commercial practices that are prohibited in all circumstances. It bans generic environmental claims such as "environmentally friendly" or "eco" unless the trader can demonstrate recognized excellent environmental performance relevant to the claim; claims that a product has a neutral, reduced or positive impact on the environment based on greenhouse gas offsetting; sustainability labels not based on a certification scheme or established by public authorities; and claims about the whole product that apply only to part of it. The Commission had also proposed a separate Green Claims Directive requiring prior verification of explicit environmental claims, but in June 2025 it announced its intention to withdraw the proposal, and the negotiations stalled. The ECGT Directive remains, and it is enough on its own to change how brands market sustainability in Europe. The French campaign against ultra-fast fashion France has gone further than any other jurisdiction in targeting the business model of ultra-fast fashion. A bill first introduced in 2023 and passed in different versions by the National Assembly and the Senate was definitively adopted by Parliament on 29 June 2026. The law defines ultra-fast fashion by reference to the volume of new products a company introduces and the incentives it offers for repair. It imposes eco-penalties per item that rise over time and may reach twenty euros per item by 2030, capped at half the product's pre-tax price, with proceeds directed to collection and recycling. It requires ultra-fast-fashion companies to display messages encouraging moderation, reuse and repair, and it bans advertising for them, including through influencers. The law's implementation remains uncertain. The European Commission, notified of the draft, raised concerns about its compatibility with EU law, and the law may face constitutional challenge. But its adoption marks a turning point. For the first time, a major jurisdiction has singled out a particular fashion business model, defined by volume and speed, for adverse legal treatment. Other countries are watching. The cost of the garment's afterlife The measures in this chapter share a logic. They take costs that the fashion industry has historically externalized, the cost of waste, of chemical pollution and of misleading consumers, and they assign those costs to the producers who create them. They do so through fees, bans, disclosure and liability for false claims. And they depend, like the forced-labor laws of Chapter 7, on the producer's ability to know what is in its products, where they were made and what will happen to them. Environmental law has thus joined intellectual property and human rights law in making provenance a legal obligation. Conclusion: The Law of True Signals The three parts of fashion law examined in this book began as separate subjects with separate histories. Trade dress grew out of the law of unfair competition and the Lanham Act. Counterfeiting law grew out of the same statute but developed its own remedies and its own enforcement institutions. Supply chain law grew out of trade law, human rights advocacy and environmental regulation. For most of the twentieth century a fashion company could treat them as the business of different departments: the designers and brand managers, the lawyers who pursued fakes, and the sourcing executives who dealt with factories. The argument of this book has been that those subjects now form a single field, organized around a single idea. Fashion law protects signals rather than designs, and it increasingly demands that those signals be true. Trade dress protects a design only when it has come to tell the public who made a product. Counterfeiting law punishes false signals about origin more harshly than any other form of copying. And the new supply chain laws require that a brand's signals about where and how its products were made, and about their environmental virtues, withstand scrutiny. The unifying question in each area is the same: does this product say something true about where it came from? What follows for the brand Several practical conclusions follow from that argument. The first concerns investment in design. Because American law protects design mainly through trade dress, and trade dress requires secondary meaning, brands that want legal protection for their designs must commit to a small number of signature elements, use them consistently over many years and promote them as marks of origin. The lesson of Wal-Mart, Yurman and the Louboutin case is that protection follows repetition and precise definition. The lesson of the Thom Browne and UGG cases is that even strong signatures lose protection when they become too basic, too common or too closely identified with a whole category of product. A brand that wants to protect a design should register it where it can, define it precisely, advertise it as a signature and police imitations before they become common. A brand whose business depends on constant novelty should accept that most of its designs will be free to copy and compete on speed, quality and brand rather than on legal exclusivity. In Europe, it should use the registered and unregistered design rights that American law does not offer. The second concerns enforcement. The tools against counterfeiting are powerful, but the most powerful of them depend on registration and on information. Statutory damages and customs detention are available only for registered marks. Schedule A cases now require individualized evidence. Platform takedowns require specific listings. Liability of landlords and marketplaces requires proof of specific knowledge. Effective enforcement therefore depends on a brand's ability to identify its own products and distinguish them from fakes quickly and reliably. Brands that invest in authentication, product identifiers and supply chain records enforce more effectively. The third concerns the supply chain itself. The UFLPA, the EU due diligence directive, textile EPR laws, chemical restrictions and green claims rules all require a brand to know what is in its products and where they were made. Brands that do not know will find their goods detained at the border, their fees higher, their claims challenged and their products barred from sale. The historical flexibility of fashion sourcing, the ability to move orders between factories and countries without knowing where inputs come from, is becoming a legal liability. The fourth conclusion follows from the other three. The capabilities that serve one area of fashion law increasingly serve the others. Traceability that allows a brand to prove a garment is genuine also allows it to prove that its cotton did not come from Xinjiang, to substantiate a recycled-content claim and to calculate its EPR fees. A digital product passport, which the EU may require under the ESPR, could carry information useful for authentication as well as for sustainability. Brands that build these capabilities once, and use them across all three areas, will face lower compliance costs and stronger legal positions than brands that treat each area separately. What remains unsettled The field is moving quickly, and several questions remain open. The first is the future of the dupe. The recent cases against Costco and Quince suggest that dupes copying unprotected designs, clearly labeled, are lawful, and that the dupe label may even help defendants. But courts have not yet squarely decided whether marketing a product as a dupe of a named original can itself support liability, whether through confusion, dilution or false advertising. As dupe marketing grows, that question will be litigated. The second is the future of mass enforcement against online sellers. The retreat of Schedule A litigation in Chicago leaves brands without a fast, cheap tool against anonymous offshore sellers. The INFORM Act's transparency requirements, the end of the de minimis exemption and the voluntary programs of the major platforms may fill part of the gap. Whether Congress will impose stronger platform obligations, along the lines of the SHOP SAFE Act or the European Digital Services Act, remains to be seen. The third is the future of supply chain law in the United States. The UFLPA is a powerful tool, but it applies only to one region and a list of entities. The New York Fashion Act has not been enacted. California's textile EPR law is the first of its kind, and its implementation will take years. Federal environmental and human rights due diligence legislation for fashion seems unlikely in the near term. The most important supply chain obligations for American brands may therefore continue to come from abroad, through the EU directives and regulations that apply to companies selling into the European market. The fourth is the future of green claims. The FTC's review of the Green Guides remains pending, and American courts have been reluctant to find qualified sustainability claims deceptive. European law is moving in the opposite direction, banning generic environmental claims from September 2026. Brands that sell in both markets will need to decide whether to adopt the stricter European standard globally or maintain different claims in different markets. The designer's question, answered This book began with a designer asking which law had been broken when her design was copied. The honest answer, in most cases, remains that none has. American law has chosen to leave fashion design largely free to copy, and neither the rise of fast fashion nor the spread of dupe culture has changed that choice. But the designer's question has a second answer that is more encouraging. The law has become increasingly concerned with the truth of what products say about themselves. A copy that carries a false signal about who made it, where it came from or what it is made of is increasingly likely to be unlawful, whether under trademark law, counterfeiting law, customs law, supply chain law or consumer protection law. A designer who builds a distinctive signature, registers it, and can prove the origin and qualities of her products has more legal protection than ever before. The law will not protect her design as such. It will protect the truth of what her products say. Hashtags: #FashionLaw #TradeDress #FashionCounterfeiting #SustainableFashion #FashionIndustry #IntellectualPropertyLaw #TrademarkProtection #FashionDesignRights #DesignPatents #CopyrightLaw #TrademarkInfringement #LuxuryBrandProtection #BrandAuthenticity #AntiCounterfeiting #FashionKnockoffs #DupeCulture #FastFashion #FashionLitigation #LanhamAct #FashionSupplyChains #SustainableSupplyChains #EthicalFashion #SupplyChainTransparency #HumanRightsDueDiligence #ForcedLaborPrevention #TextileSustainability #EnvironmentalCompliance #Greenwashing #FashionRegulations #CorporateSustainability

Download the Book (PDF): Introduction In February 2026, OpenAI published a short post explaining why it would no longer report scores on SWE-bench Verified, a benchmark of real software engineering tasks that had, for eighteen months, been one of the most quoted numbers in the industry. The company had audited a set of problems its own models failed to solve consistently and found that well over half of them had material defects: tests that rejected correct alternative fixes, or tests that checked behaviour the task description never asked for. Worse, every frontier model it probed could reproduce, for some tasks, either the original human-written patch or verbatim details of the problem. The models had, in effect, seen the exam. A number that had been used to rank systems, set research agendas and justify purchasing decisions turned out to be measuring a blend of coding skill, memory of the test, and luck in matching the quirks of a particular grader. Nothing about this was unusual. It was simply a high-profile instance of the most common way that evaluation of language models goes wrong. A score is produced by a procedure; the procedure was designed to stand in for something people actually care about; over time, the link between the procedure and the thing it stood for weakens; and because the score is still a clean number that goes up, people keep reading it as though the link were intact. This booklet is about how to keep that link intact, or at least how to know when it has broken. Its controlling idea is simple to state and demanding to practise: an evaluation is a measurement instrument, and every instrument must be validated against the decision it is meant to inform. Perplexity, BLEU, ROUGE, benchmark accuracy, a model judge's score and a panel of human raters are all instruments of this kind. Each is good at measuring something. None is good at measuring everything. The skill of evaluation lies less in knowing how to compute these numbers, which is mostly easy, than in knowing what each one is actually sensitive to, what it is blind to, and how to combine cheap instruments with expensive ones so that the expensive ones keep the cheap ones honest. Why this is harder for language models than for most software Conventional software has a specification. A function that sorts a list either returns the list sorted or it does not, and a test can say which. Much of what we ask of a language model has no such specification. "Summarise this contract for a non-lawyer" admits thousands of acceptable answers and thousands of unacceptable ones, and the boundary between them depends on accuracy, emphasis, tone, length and the reader's needs. Two careful experts asked to judge the same summary will sometimes disagree, and their disagreement is not noise to be averaged away but information about how underdetermined the task is. Language models also fail in ways that resist simple description. A model can answer a hard chemistry question correctly and then misread a date in the prompt. It can write fluent, confident prose that is wrong in a single clause on which everything else depends. It can behave well on the phrasing used in a test and badly on a paraphrase. And it can change: a new model version, a revised system prompt, or a different sampling temperature can shift behaviour across thousands of cases at once, in directions no one anticipated. Evaluation has to be built to catch this, which means it has to be continuous, versioned and designed around the failures that matter rather than the successes that are easy to count. Finally, the field's public instruments are under constant pressure. Benchmarks that become famous become targets. Their questions leak into training data. Labs tune toward them, sometimes deliberately and sometimes simply by selecting, among many candidate models, the one that scores best. The result is a steady inflation in which headline numbers rise faster than the capabilities they were meant to track. A practitioner who takes public leaderboards at face value will systematically misjudge which model is right for their own problem. What the book covers The chapters move from the narrowest and most mechanical instruments to the broadest and most human, and then to the practice of combining them. The first chapter sets out what an evaluation is for and introduces the vocabulary of measurement, validity and decision that the rest of the book relies on. The second treats perplexity, the oldest and most fundamental number in language modelling, and explains both why it governs training and why it says surprisingly little about whether a model is useful. The third works through BLEU and ROUGE in enough detail that the reader could compute them by hand, which is the surest way to understand what they reward and what they miss. The fourth turns to the metrics that tried to fix those problems: embedding-based scores, learned metrics trained on human judgments, execution-based checks for code, and the now ubiquitous practice of using one language model to grade another. The fifth chapter examines public benchmarks: how they are built, why they saturate, how contamination and label errors distort them, and how to read a leaderboard without being misled by it. The sixth is the most practical in the book. It explains how to design a custom evaluation dataset for a real application, from mining failures to writing rubrics to deciding how many examples are enough to tell two systems apart. The seventh makes the case for human-in-the-loop grading on subjective tasks and, more importantly, describes how to do it well: how to write instructions, choose between absolute and comparative judgments, select and train raters, and measure whether they agree. The eighth assembles all of this into an evaluation programme, a standing system of offline suites, model judges, human audits and production monitoring that lets a team change a system with confidence. A note on what is left out Safety and dangerous-capability evaluations, which ask whether a model can assist with cyberattacks or the synthesis of hazardous materials, are an important specialised field with its own methods and governance; they are mentioned only where their methods illuminate general practice. Fairness and bias auditing receives the same treatment. Both deserve books of their own, and both rest on the measurement principles developed here. The focus throughout is on the evaluator who has to decide, with limited time and money, whether a model or a system built on one is good enough for a particular job, and whether this week's version is better or worse than last week's. The reader does not need a background in statistics or machine learning, although some comfort with probabilities and averages helps. Where formulas appear they are explained in words, and where a worked example would make an idea concrete, one is given. Specific scores and findings are attributed to their sources. Where the field's understanding is genuinely unsettled, as it is on several questions about model judges and benchmark validity, the book says so rather than pretending to a consensus that does not exist. The aim is a reader who, handed any evaluation number, asks the right three questions before believing it: what procedure produced it, what that procedure is actually sensitive to, and whether anyone has checked it against the judgment it is meant to replace. Chapter 1: What an Evaluation Is For Every evaluation exists to support a decision. Sometimes the decision is obvious: ship this model or keep the old one, choose vendor A or vendor B, accept this change to the prompt or roll it back. Sometimes it is buried: a research group deciding which of three training recipes to scale up, or a product manager deciding whether a feature is ready for a pilot with ten customers. But if no decision depends on a number, the number is decoration, and decoration has a way of crowding out measurement. Starting from the decision changes what a good evaluation looks like. A team choosing between two models for a customer-support assistant does not need to know which model is "better" in some general sense. It needs to know which one will resolve more customer issues correctly, in the house tone, without inventing refund policies, at an acceptable cost per conversation. A benchmark of graduate-level physics questions is almost irrelevant to that decision, however impressive the scores. A set of two hundred real support tickets, each with a note from an experienced agent on what a correct resolution looks like, is directly relevant, even though nobody outside the company will ever hear of it. Instruments, constructs and validity It helps to borrow some vocabulary from the older discipline of psychological and educational measurement, which has spent a century thinking about how to measure things that cannot be observed directly. The thing we actually care about is called the construct. "Helpfulness to a customer", "faithfulness of a summary to its source", "ability to write correct code from a description" are constructs. They cannot be read off a model the way temperature is read off a thermometer. What we can do is design a procedure, an instrument, that produces a number we hope tracks the construct: a test set and a scoring rule, a panel of raters with a rubric, an automated metric. Validity is the degree to which the instrument's number actually reflects the construct for the purpose at hand. It is not a property of the instrument alone. BLEU, which compares a system's translation to reference translations by counting shared word sequences, has reasonable validity for comparing two broadly similar machine translation systems on the same test set, and very poor validity for judging whether a single creative rewrite of a paragraph is good. The same instrument, a different use, a different validity. Reliability is the narrower question of whether the instrument gives consistent answers when nothing important has changed. A metric that swings by three points when you rerun the same model with a different random seed, or a rater panel whose verdicts flip when the same pair of responses is shown in the opposite order, is unreliable. Reliability is necessary for validity but not sufficient: a ruler that is consistently two centimetres short is perfectly reliable and consistently wrong. These distinctions sound academic until you meet the failures they describe. Three are worth naming at the outset because they recur throughout the book. The first is construct underrepresentation: the instrument covers only part of what matters. A summarisation metric that rewards overlap with a reference summary says nothing about whether the summary introduced a false claim, so a system can improve on the metric while becoming less trustworthy. The second is construct-irrelevant variance: the instrument is sensitive to something that does not matter. Model judges that prefer longer answers, human raters who reward confident tone regardless of correctness, and multiple-choice benchmarks whose scores shift when the order of the options is shuffled are all examples. The score moves, but not because the construct moved. The third is Goodhart's law, usually paraphrased as "when a measure becomes a target, it ceases to be a good measure." The economist Charles Goodhart made the observation about monetary policy in 1975, and it applies with special force to machine learning, where optimisation is the whole business. Any instrument that is used to select among models will, over time, select for the instrument's quirks as well as for the construct. This is not a moral failing of the people involved; it is a statistical inevitability. The only defence is to keep some instruments out of the optimisation loop and to refresh the ones that are inside it. Three questions an evaluation can answer Evaluations answer broadly three kinds of question, and confusing them is a frequent source of wasted effort. The first is capability: can this model do this kind of thing at all, and how well? Capability evaluations are what public benchmarks mostly measure. They are useful for choosing which models are worth considering and for tracking the state of the field. They are poor guides to performance in a specific application, because applications combine capabilities in particular ways, under particular prompts, on particular data. The second is fitness for purpose: will this system, as configured, do this job acceptably? Here the object of evaluation is not a model but a system: the model plus its prompt, its retrieval pipeline, its tools, its guardrails and the interface through which users reach it. The same model can be excellent in one configuration and unacceptable in another. Fitness-for-purpose evaluations are almost always custom, built from the application's own data and judged against the application's own standards. The third is regression: did this change make things worse? Every time a prompt is edited, a model version is swapped, a retrieval index is rebuilt or a temperature is adjusted, some behaviour changes. Regression evaluations exist to catch the changes no one intended. They need to be fast, cheap and run often, which pushes them toward automated scoring, but they also need to be sensitive to the failures that matter, which pulls them toward careful design. A mature team runs all three, at different cadences and with different instruments. Capability checks happen when new models appear. Fitness-for-purpose evaluations happen before launch and at major milestones. Regression suites run on every change. The mistake is to use one kind for another job: to choose a production model by its leaderboard rank, or to sign off a launch on a regression suite that was designed only to detect breakage. The instrument families The rest of this book examines the main families of instrument in turn. It is worth laying them out side by side before going into each, because their strengths are largely complementary. Table 1 summarises what each family is sensitive to, what it costs to run, and where it tends to mislead. Table 1. The main families of evaluation instrument. Instrument What it is sensitive to Relative cost Typical blind spot Perplexity Fit to a text distribution Very low Usefulness, truth, instruction following Overlap metrics (BLEU, ROUGE) Shared words and phrases with references Very low Paraphrase, factual errors, fluency Learned and embedding metrics Semantic similarity, trained preferences Low Domains unlike their training data Execution and exact checks Verifiable correctness Low to moderate Anything not captured by the check Model-as-judge Rubric-guided quality judgments Moderate Length, position and self-preference biases Human grading The construct itself, as people see it High Rater disagreement, fatigue, drift The pattern is roughly that cost rises as validity rises. Perplexity is nearly free and nearly useless for product decisions. Human grading by trained experts is expensive and slow and remains the closest thing to ground truth for subjective tasks. The art is to build a ladder in which each cheaper rung is periodically calibrated against the more expensive rung above it, so that the cheap instruments can be run constantly while the expensive ones are used sparingly to keep them honest. Evaluations are experiments One further idea underpins everything that follows: an evaluation is an experiment, and should be designed and reported like one. That means being explicit about the population the test set was drawn from, and therefore the population to which its results can be generalised. A test set of five hundred English customer emails says little about performance on Spanish ones. It means controlling the conditions: the same prompt template, the same decoding settings, the same version of the grader, recorded so that the run can be repeated. It means reporting uncertainty: a score of 78 per cent on 200 examples carries a 95 per cent confidence interval of roughly plus or minus six percentage points, which is wider than most of the differences people argue about. And it means guarding against the forking paths of analysis, in which many configurations are tried and only the best is reported. Language model evaluation has been slow to adopt these habits, partly because benchmark culture rewards single headline numbers and partly because the cost of generating outputs made large, repeated runs impractical. Both excuses are weakening. Generating and scoring thousands of outputs is now cheap relative to the cost of shipping a regression to users, and an increasing body of research has shown how often apparent differences between models vanish once uncertainty is taken into account. Evan Miller's 2024 paper "Adding Error Bars to Evals" set out a practical statistical treatment for exactly this setting, including how to account for questions that come in clusters, such as several questions about the same passage, which shrink the effective sample size. The remaining chapters take the instruments one at a time, starting with the oldest. In each case the aim is the same: to understand the mechanism well enough to know what the number means, what it does not mean, and how to tell when it has stopped meaning anything at all. Quality is never the only criterion Starting from the decision has one further consequence that pure quality metrics tend to obscure: the decision almost always involves more than quality. A model that answers support tickets one point more accurately but costs four times as much per conversation, or takes eight seconds to begin responding instead of one, may be the wrong choice even though it wins every quality comparison. Cost, latency, throughput, reliability of the provider, data-handling terms and the effort of migration are all part of fitness for purpose, and an evaluation that ignores them answers a question nobody asked. Consider a concrete decision. A legal-technology company extracts key terms, such as parties, dates, governing law and termination clauses, from commercial contracts. It is choosing between a large frontier model and a smaller, cheaper one. A naive evaluation runs both on a hundred contracts and reports field-level accuracy: perhaps 94 per cent for the large model and 89 per cent for the small one. On that basis the large model wins. A decision-centred evaluation asks different questions. Which fields matter most? An error in the governing-law field may be harmless if a lawyer reviews every extraction anyway; an error in a termination date that triggers an automated reminder may cause real harm. How are errors distributed? If the small model's extra errors are concentrated in one field and can be caught by a simple rule, such as checking that a date falls after the contract's signature date, the effective gap narrows. What happens downstream? If every extraction is reviewed by a paralegal, the relevant measure may be review time per contract rather than raw accuracy, since a model that is slightly less accurate but flags its own uncertainty clearly could save more time than one that is slightly more accurate but uniformly confident. And what does each option cost at the expected volume? With these questions answered, the decision might go either way. The point is that it is made on evidence about the actual consequences, rather than on a single accuracy figure that silently weights every field and every error equally. The habit to cultivate is to write down, before running anything, what result would change the decision. If no plausible result would change it, the evaluation is not worth running. If some result would, the evaluation should be designed to detect exactly that result with enough confidence to act on. This also clarifies how much evaluation is enough. The appropriate investment scales with the cost of being wrong and the reversibility of the decision. Choosing a model for an internal tool that can be swapped next month justifies a day's work. Choosing one for a system that will give medical information to the public justifies months. Treating every evaluation with the same rigour, either uniformly light or uniformly heavy, wastes effort in one direction or invites harm in the other. What a good evaluation report contains Because the rest of the book will refer back to it, it is useful to fix here what a minimally complete evaluation result looks like. It names the system evaluated precisely: model identifier and version, the full prompt or a reference to a versioned copy, decoding parameters, and any retrieval or tool configuration. It names the dataset, its size, how it was sampled and what population it represents. It names the scoring procedure, including the version of any grader model and its prompt, or the rubric and rater pool for human grading. It reports the score with an interval, and, where two systems are compared, the difference with an interval, computed on paired examples where possible. And it records anything unusual: examples excluded and why, failures of the grader, known contamination. None of this is exotic. It is the discipline any experimental science expects. Its absence is the main reason that language model evaluation results so often fail to replicate, and why so many arguments about which model is better are really arguments between two unrecorded configurations. A team that adopts this one habit, a complete record for every number it reports, will find that most of the harder problems discussed later become tractable, because when an evaluation surprises them they will be able to see why. Chapter 2: Perplexity and the Limits of Likelihood Before language models answered questions, wrote code or held conversations, they did one thing: they assigned probabilities to sequences of words. A model that gives high probability to the sentences people actually write, and low probability to the ones they do not, is a good model of the language. Perplexity is the standard way of turning that idea into a number, and it remains the quantity that every large language model is trained to improve. Understanding it properly explains both why it is indispensable during training and why it is almost never the right number for deciding whether a model is fit for a job. What perplexity measures A modern language model reads a sequence of tokens, which are words or fragments of words, and at each position produces a probability distribution over what the next token might be. Given the text "The cat sat on the", a good model might assign 0.4 to "mat", 0.1 to "floor", 0.05 to "sofa" and spread the rest thinly across tens of thousands of other possibilities. To evaluate the model on a passage of held-out text, we walk through the passage token by token and record the probability the model assigned to the token that actually came next. If the passage continues "mat", we record 0.4. Then we move on and record the probability of whatever followed "mat", and so on to the end. These probabilities are multiplied together to give the probability of the whole passage, but a product of thousands of numbers below one quickly becomes too small to work with, and in any case longer passages would always look worse than shorter ones. So instead we take the logarithm of each probability, average those logarithms over all the tokens, and flip the sign. The result is the cross-entropy: the average number of nats (or bits, if base-two logarithms are used) of surprise the model experienced per token. Lower is better; a model that predicted every token with certainty would have a cross-entropy of zero. Perplexity is simply the exponential of the cross-entropy. Its appeal is that it has an intuitive reading. A perplexity of 20 means the model was, on average, as uncertain as if it were choosing uniformly among 20 equally likely options at every step. A perplexity of 1 would mean perfect prediction. Early statistical language models of English news text had perplexities in the hundreds; well-trained modern models on comparable text reach the low single digits or below, depending on the tokenizer and the data. This average-surprise reading also explains why perplexity is the natural training objective. Minimising cross-entropy on a huge corpus is equivalent to maximising the likelihood of that corpus under the model, and it is a smooth, differentiable quantity that can be computed for every token of every document. Every pretraining run of every large language model is, at bottom, an exercise in driving this number down. Why it is indispensable during training Inside a laboratory that trains models, perplexity on held-out data is the heartbeat monitor. It is computed constantly, cheaply and with very low noise because it averages over millions of tokens. It reveals immediately when a training run has diverged, when a data pipeline has started feeding duplicated or corrupted text, or when a learning-rate schedule is misbehaving. It is also the quantity that obeys scaling laws. Work by Jared Kaplan and colleagues at OpenAI in 2020, and the subsequent "Chinchilla" study by Jordan Hoffmann and colleagues at DeepMind in 2022, showed that held-out loss falls as a smooth, predictable power law in model size, data quantity and compute. That predictability is what allows labs to plan large training runs from small ones: train a ladder of small models, fit the curve, and extrapolate the loss the large model will reach. Downstream task scores, by contrast, are far noisier and sometimes jump abruptly, which makes them poor planning tools. Perplexity is also useful for comparing data. Training the same small model on two candidate datasets and comparing held-out perplexity on a common, high-quality validation set is a standard way of deciding which data mixture to use. And perplexity under a reference model is frequently used as a filter: documents that a good model finds extremely surprising are often garbled, machine-generated spam or in a language the pipeline did not expect. Why it does not tell you whether a model is useful The problems begin when perplexity is asked to answer questions it was never designed for. It is not comparable across tokenizers. Perplexity is an average per token, and different models chop text into tokens differently. A model whose tokenizer splits "unbelievable" into three pieces faces three easier predictions where another model faces one harder one. Their per-token perplexities are not on the same scale. The partial remedy is to normalise by something tokenizer-independent, such as bits per byte or bits per character of the original text, which is why research comparing models across families tends to report those units. But even this only fixes the arithmetic, not the deeper problems. It depends entirely on the evaluation text. A model's perplexity on Wikipedia, on legal contracts, on Python code and on Reddit comments can differ enormously. A model that scores well on one distribution may do badly on another, and neither number says anything about the domain you actually care about unless you measure on that domain. Worse, if the evaluation text appeared in the training data, as is likely for anything widely published on the web, perplexity will be flatteringly low because the model is partly remembering rather than predicting. It measures fit to what people wrote, not to what people want. This is the fundamental limitation. The text on the internet contains errors, misconceptions, rudeness and every kind of low-quality writing alongside the good. A model that perfectly mimicked the distribution of web text would reproduce all of it with the right frequencies. Perplexity rewards it for doing so. It does not reward the model for being correct, for following instructions, for declining harmful requests or for writing concisely, except insofar as those behaviours happen to be the most probable continuation. This is why the training of a modern assistant does not stop at pretraining. Instruction tuning and reinforcement learning from human or AI feedback deliberately move the model away from the raw web distribution toward a distribution of helpful, honest responses. One well-documented consequence is that perplexity on general held-out text often gets worse after this alignment stage, even as the model becomes far more useful. The OpenAI InstructGPT paper of 2022 documented a related effect, describing an "alignment tax" in which performance on some standard NLP benchmarks fell after reinforcement learning from human feedback, and mitigating it by mixing pretraining updates back into training. A team that selected among chat models by their perplexity on web text would be choosing, in part, for less successful alignment. It does not see the output a user sees. Perplexity is computed by feeding the model a fixed text and reading off probabilities. It never asks the model to generate anything. But what users experience is generated text, produced by a decoding procedure that samples from or searches through the model's probabilities, often over hundreds of tokens, with the model's own earlier outputs feeding into its later predictions. Errors compound in generation in a way they cannot when the model is always handed the correct previous token. A model with excellent perplexity can still generate repetitive, incoherent or derailed text if decoded badly, and two models with similar perplexity can generate very differently. It is not available for many systems. Many commercial models are accessed through interfaces that return generated text but not the full probability distribution, or return log probabilities only for the top few candidate tokens. Even when log probabilities are exposed, the system the user interacts with may include retrieval, tool use and multiple model calls, none of which has a single perplexity. Uses that remain legitimate outside training None of this means perplexity should be ignored outside the training lab. It has several legitimate uses in applied evaluation, each relying on its narrow meaning rather than stretching it. Domain adaptation checks. If you fine-tune a model on your organisation's documents, perplexity on a held-out set of those documents tells you whether the model has learned their vocabulary and style. It will not tell you whether the fine-tuned model answers questions about them correctly, but a fine-tune that failed to lower in-domain perplexity has almost certainly failed at something. Likelihood-based scoring of fixed options. Many multiple-choice benchmarks are scored not by asking the model to write an answer but by computing the probability it assigns to each candidate answer and picking the highest. This is a form of conditional perplexity, and it is how evaluation suites such as EleutherAI's widely used Language Model Evaluation Harness scored many tasks. It has quirks: longer answer options accumulate lower total probability, so scores are often length-normalised, and different normalisation choices can reorder models. Researchers comparing results across papers have repeatedly found that apparently identical benchmarks give different numbers because of exactly these choices, which is a useful lesson in how much an evaluation result depends on undocumented procedural details. Contamination detection. If a model assigns suspiciously low perplexity to the exact text of a benchmark's test questions, compared with paraphrases of the same questions or with comparable unpublished text, that is evidence the test set was in its training data. Several published contamination detection methods work on this principle, comparing likelihoods on original and perturbed versions of the same items. The method is imperfect, because some text is simply predictable, but it is one of the few tools available to outsiders. Confidence and calibration. The probability a model assigns to its own answer can be informative about whether the answer is right. A model is calibrated if, among all the answers it gives with 80 per cent confidence, about 80 per cent are correct. Calibration was routinely measured for older classifiers and is increasingly measured for language models, both from token probabilities and from verbalised confidence ("I am about 70 per cent sure"). The GPT-4 technical report of 2023 showed that the pretrained model was well calibrated on a multiple-choice benchmark and that post-training reduced that calibration, another instance of alignment moving the model away from the properties that likelihood measures. Likelihood in the long-context era One area where likelihood-based measurement has found a new and instructive role is long context. Models now routinely accept inputs of hundreds of thousands of tokens, and vendors advertise these context lengths prominently. Whether a model can actually use information spread across such a long input is a separate question, and it has proved surprisingly hard to answer. A natural first approach is to measure perplexity as a function of position: does the model predict tokens near the end of a very long document better than tokens near the start, because it can draw on everything that came before? If loss keeps falling as more context is available, the model is presumably using that context. This kind of position-wise loss curve is a standard diagnostic in research on long-context architectures, and it is useful for detecting outright failure, such as a model whose predictions degrade sharply beyond the length it was trained on. But falling loss over long documents can be achieved by using context shallowly, for instance by picking up the document's vocabulary and style, without the model being able to retrieve or reason over a specific fact buried deep inside. So practitioners turned to generation-based tests. The simplest, popularised in 2023 by the developer Greg Kamradt and widely known as the needle in a haystack test, hides a single unrelated sentence somewhere in a long filler document and asks the model to retrieve it. Plotting success against document length and the needle's position gives a quick picture of where retrieval breaks down. Many models soon achieved near-perfect scores on this test, which illustrates the saturation pattern discussed in Chapter 5: the task had become too easy to tell models apart. Harder variants followed. Nelson Liu and colleagues at Stanford, in "Lost in the Middle" (2023, later published in Transactions of the Association for Computational Linguistics), showed that on multi-document question answering, models were often best at using information placed at the beginning or end of the context and noticeably worse at using information in the middle. The RULER benchmark from Cheng-Ping Hsieh and colleagues at NVIDIA in 2024 combined multiple needles, distractors, variable tracking and aggregation tasks, and found that many models' effective context length, the length at which performance stayed above a threshold, was considerably shorter than the length they claimed to support. The sequence is a compact lesson in the theme of this chapter. Perplexity revealed whether a model could process long inputs at all. It could not reveal whether the model could use them for a task, so generation-based tests were built. The first such test was too easy and saturated, so harder ones were built. At each stage the cheaper instrument remained useful for the narrow question it answered, and misleading when stretched beyond it. For an evaluator choosing a model to process long documents, the practical implication is that an advertised context length is a capability claim of the weakest kind, and that the only reliable test is one that places the information the application actually needs in the positions where it actually occurs. A worked contrast A concrete case shows the gap between perplexity and usefulness. Imagine two models evaluated on a held-out set of customer-support transcripts from a software company. Model A is a base model fine-tuned directly on ten thousand historical transcripts. Its perplexity on held-out transcripts is very low: it has learned the product names, the agents' phrasing and the typical flow of a conversation. Model B is a general assistant model, instruction-tuned and aligned, given a carefully written system prompt and access to the company's help-centre articles through retrieval. Its perplexity on the same transcripts is noticeably higher; it does not talk quite like the company's agents, and it does not know which internal ticket codes they use. Now run both on fifty new customer questions and have experienced agents grade the answers. In practice, outcomes like this are common: Model A produces replies that sound exactly like the company's agents, including their habit of saying "let me check that for you" and then inventing a plausible-sounding but wrong answer, because the historical transcripts contained many such moments where the human agent then went and looked something up. Model B sounds less like the house style but grounds its answers in the help-centre articles and declines to answer when they do not cover the question. The agents prefer Model B by a wide margin. Perplexity correctly reported that Model A was a better model of the transcripts. It could not report that the transcripts were not what anyone wanted the model to reproduce. The lesson generalises: likelihood measures fidelity to a distribution, and the moment the goal is anything other than fidelity to a distribution, likelihood becomes at best a weak proxy. Where this leaves perplexity Perplexity is the right instrument for the questions it was built for: how well does this model fit this text, did training go as expected, which data mixture produces a better base model, does this test set appear to be memorised. It is the wrong instrument for questions about usefulness, correctness or safety, and it becomes actively misleading when used to compare models with different tokenizers, different training stages or different data. The next chapter turns to metrics that look at generated text rather than probabilities. They were invented to answer a question perplexity cannot: given that the model produced this output, how good is it? Their answer, which is to compare the output with one written by a person, is both ingenious and deeply limited, and its limitations shaped much of what came after. Chapter 3: BLEU, ROUGE and the Logic of Overlap In 2002 a group of researchers at IBM, Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu, published a paper at the annual meeting of the Association for Computational Linguistics with a modest title: "BLEU: a Method for Automatic Evaluation of Machine Translation." It became one of the most cited papers in the history of natural language processing. Two years later Chin-Yew Lin, then at the University of Southern California's Information Sciences Institute, published ROUGE, a family of related measures for automatic summarisation. Between them, these two metrics shaped how a generation of systems was built and compared, and both are still reported in papers today, including papers about large language models. They deserve close study for two reasons. First, they remain in use, and anyone who reads the literature or inherits an evaluation pipeline will meet them. Second, they are the clearest possible example of what a reference-based metric can and cannot do. Every later metric discussed in this book is, in one way or another, an attempt to fix a weakness that BLEU and ROUGE exposed. The problem BLEU solved In the 1990s, evaluating a machine translation system meant paying bilingual people to read its output and rate adequacy and fluency. It was slow, expensive and impossible to do every time a researcher changed a parameter. Progress was correspondingly slow. What the field needed was a number that could be computed in seconds and that tracked human judgment closely enough to steer day-to-day development. The insight behind BLEU was that good translations tend to share words and phrases with professional human translations of the same source. Not every good translation will match a given reference, because there are many ways to say the same thing, but across a large test set, a system whose outputs share more phrases with the references is, on average, producing better translations than one whose outputs share fewer. Using several reference translations per sentence helps further, because it captures more of the legitimate variation. Computing BLEU by hand BLEU is built from modified n-gram precision. An n-gram is simply a run of n consecutive tokens: "the" is a unigram, "the cat" a bigram, "the cat sat" a trigram. For each n from one to four, BLEU asks: of all the n-grams in the system's output, what fraction also appear in a reference? The word "modified" refers to clipping. Without it, a degenerate output such as "the the the the the the" would score perfectly on unigram precision against any reference containing "the". So each n-gram in the output is credited at most as many times as it appears in the reference. If the reference contains "the" twice, an output containing it six times gets credit for two. These four precisions are combined by taking their geometric mean, which means multiplying them together and taking the fourth root. The geometric mean has an important and often overlooked consequence: if any one of the four precisions is zero, the whole score is zero. Precision alone would reward very short outputs. A system that output only the word "the" for every sentence would have perfect unigram precision. BLEU counters this with a brevity penalty. If the total length of the system's outputs, c, is greater than the total length of the references, r, there is no penalty. If it is shorter, the score is multiplied by e raised to the power of (1 − r/c), which shrinks rapidly as outputs get shorter. There is no corresponding penalty for being too long, because over-long outputs are already punished by lower precision. Consider a single reference, "the cat is sitting on the mat", seven tokens long, and three candidate outputs. The first candidate is "the cat sat on the mat", six tokens, a perfectly good paraphrase. Of its six unigrams, five appear in the reference (only "sat" does not), so unigram precision is 5/6. Of its five bigrams, three appear ("the cat", "on the", "the mat"), giving 3/5. Of its four trigrams, only "on the mat" appears, giving 1/4. Of its three four-grams, none appears in the reference, giving 0/3. The brevity penalty, since six is shorter than seven, is e^(1 − 7/6), about 0.85. Because the four-gram precision is zero, the geometric mean is zero, and the BLEU-4 score of this good translation is zero. The second candidate is "a cat was sitting on the rug", also a reasonable paraphrase though it changes "mat" to "rug". Its unigram precision is 4/7, bigram 2/6, trigram 1/5 ("sitting on the") and four-gram 0/4. Again, BLEU-4 is zero. The third candidate is "the cat is not sitting on the mat", eight tokens, which says the opposite of the reference. Its unigram precision is 7/8, bigram 5/7, trigram 3/6 and four-gram 1/5, since "sitting on the mat" appears in the reference. It is longer than the reference, so there is no brevity penalty. Its BLEU-4 score is the fourth root of (7/8 × 5/7 × 3/6 × 1/5), which is exactly 0.50. Table 2 sets these numbers side by side. Table 2. Sentence-level BLEU-4 for three candidates against the reference "the cat is sitting on the mat" (author's calculation, no smoothing). Candidate 1-gram 2-gram 3-gram 4-gram BLEU-4 the cat sat on the mat 5/6 3/5 1/4 0/3 0.00 a cat was sitting on the rug 4/7 2/6 1/5 0/4 0.00 the cat is not sitting on the mat 7/8 5/7 3/6 1/5 0.50 The example is deliberately extreme, and it is not quite fair to BLEU. The metric was designed to be computed over a whole test corpus, where n-gram counts are pooled across hundreds or thousands of sentences before the precisions are calculated, so zero four-gram precision is rare. When BLEU must be computed per sentence, various smoothing methods add small pseudo-counts to avoid zeros; Boxing Chen and Colin Cherry compared several of these in 2014. And with multiple references, the first two candidates would probably have matched more. But the example captures, in miniature, the two structural weaknesses that no amount of smoothing removes. BLEU cannot recognise a paraphrase that uses different words, and it cannot recognise that a single inserted word has reversed the meaning. What BLEU is good for, and what it is not At the corpus level, comparing two systems of the same general type on the same test set, BLEU correlated reasonably with human judgment in the early statistical machine translation era, which is why it was adopted. Its problems emerged as systems diversified. In 2006, Chris Callison-Burch, Miles Osborne and Philipp Koehn showed that BLEU could badly misjudge systems that were built differently from one another. In a shared task, a rule-based system that human judges ranked highly received a much lower BLEU score than statistical systems, because it chose words that were correct but different from the references. Their paper's title, "Re-evaluating the Role of BLEU in Machine Translation Research", signalled the start of a long reassessment. In 2018, Ehud Reiter published a structured review of studies that had measured BLEU's correlation with human judgment, in the journal Computational Linguistics. He concluded that the evidence supported using BLEU for diagnostic evaluation of machine translation systems during development, but did not support using it to evaluate individual texts, to evaluate systems in other generation tasks, or as the principal evidence in scientific claims about whether a system is better. The most comprehensive test came in 2021 from Tom Kocmi and colleagues at Microsoft, in a paper titled "To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation." They used a very large collection of human judgments, roughly 2.3 million sentence-level judgments covering 4,380 systems, to ask a practical question: when an automatic metric says system A is better than system B, how often do human judges agree? They found that BLEU was among the weaker metrics on this test, that learned metrics such as COMET did substantially better, and argued that the field's reliance on BLEU alone had led to poor deployment decisions and held back better models. A further problem is that "BLEU" is not one number. Scores depend on tokenisation, on whether text is lowercased, on how many references are used and on the smoothing method. In 2018 Matt Post showed that these differences could shift reported scores by enough to swamp the improvements papers were claiming, and released SacreBLEU, a tool that computes BLEU in a standard way and prints a signature recording exactly how. Anyone reporting BLEU today should use it or something like it, and should report the signature. METEOR, chrF and the attempt to patch overlap Before learned metrics arrived, researchers tried to repair BLEU's weaknesses while keeping its transparent, string-based character. Two of these repairs are still in use and illustrate how far patching can go. METEOR, introduced by Satanjeev Banerjee and Alon Lavie at Carnegie Mellon in 2005, addressed the paraphrase problem directly. Rather than counting only exact word matches, it aligns the candidate with the reference in stages: first exact matches, then matches between words that share a stem, such as "sits" and "sitting", and then matches between synonyms drawn from the WordNet lexical database. It computes both precision and recall over these aligned words, weighting recall more heavily on the grounds that missing content is usually worse than adding a little, and it applies a fragmentation penalty that reduces the score when the matched words are scattered in a different order rather than appearing in contiguous chunks. On the cat example above, METEOR scores the paraphrase "the cat sat on the mat" well above zero, because it works from aligned words and their order rather than requiring a matching four-word sequence. Its authors reported better correlation with human judgments at the sentence level than BLEU achieved. METEOR's limitations are instructive. Its synonym matching depends on a lexical resource that exists in good form only for a few languages, and on stemmers that vary in quality. Its design choices, such as the weight on recall and the shape of the fragmentation penalty, are parameters that were tuned against human judgments for particular language pairs, so it carries a small learned component in disguise. And it still cannot tell that "not sitting" contradicts "sitting." chrF, proposed by Maja Popović in 2015, took a different route. Instead of words, it counts overlapping character n-grams, typically up to six characters long, and combines precision and recall into an F-score that weights recall more heavily. Working at the character level has two advantages. It gives partial credit for morphological variants, since "translation" and "translations" share almost all their character sequences, which matters greatly for languages such as Finnish, Turkish or Czech, where a single word can take dozens of inflected forms and exact word matching is harsh. And it needs no tokenizer, sidestepping one of the main sources of irreproducibility in BLEU. In the metrics evaluations run by the annual Conference on Machine Translation, chrF has generally correlated better with human judgment than BLEU, and the Kocmi study discussed above recommended it over BLEU where a string-based metric is needed. These metrics show that thoughtful engineering can recover a good deal of what BLEU loses on surface variation. They also show the ceiling of the approach. However cleverly two strings are compared, the comparison is still between the output and one or a few references, and still blind to whether a small difference in wording changes the meaning. The deeper fixes, discussed in the next chapter, required abandoning the idea that quality could be read off the surface of the text. ROUGE: recall for summaries ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. Where BLEU emphasises precision, asking how much of the output appears in the reference, ROUGE's basic form emphasises recall, asking how much of the reference appears in the output. The logic was that a summary's main job is to cover the important content, and that the important content is what a human summariser chose to include. ROUGE-N counts overlapping n-grams. ROUGE-1 uses unigrams and ROUGE-2 bigrams. In recall form, ROUGE-2 is the number of reference bigrams that appear in the system summary divided by the total number of reference bigrams. Most modern implementations report precision, recall and their harmonic mean, the F-measure, and papers usually quote the F-measure. ROUGE-L uses the longest common subsequence: the longest sequence of words that appears in both texts in the same order, though not necessarily contiguously. If the reference is "police killed the gunman" and the output is "the gunman killed police", the longest common subsequence is two words long ("the gunman"), whereas against an output reading "police kill the gunman" it would be three ("police", "the gunman"). ROUGE-L thus rewards preserving word order without demanding exact phrase matches. A variant called ROUGE-Lsum computes this per sentence and combines the results, which matters for multi-sentence summaries; implementations differ on whether they use ROUGE-L or ROUGE-Lsum, which is another source of incomparable numbers across papers. Take a reference summary of a news story: "The council approved the new budget on Tuesday after a long debate." A system produces: "After lengthy debate, the council passed the budget Tuesday." The reference has twelve words and the system output nine. Ignoring capitalisation and punctuation, the shared unigrams are "after", "debate", "the" (twice in each), "council", "budget" and "tuesday". That is seven of the reference's twelve unigrams, a ROUGE-1 recall of about 0.58, and seven of the system's nine, a precision of about 0.78. The shared bigrams are only "the council"; "the budget" in the system does not match "the new budget". ROUGE-2 recall is therefore 1/11, about 0.09. A competent human summary scores poorly on ROUGE-2 because it chose "passed" instead of "approved" and "lengthy" instead of "long." The deeper problem: faithfulness Summarisation exposes a weakness of overlap metrics that translation partly hides. A summary can share many words with a reference and still contain a false statement. If the system had written "The council rejected the new budget on Tuesday after a long debate", it would score higher on ROUGE than the faithful paraphrase above, because it shares nearly all the reference's words, and it would be completely wrong. This is not a hypothetical concern. When neural abstractive summarisers became common in the late 2010s, researchers began to examine how often their summaries contained information not supported by the source. Joshua Maynez and colleagues at Google, in a 2020 paper "On Faithfulness and Factuality in Abstractive Summarization", had human annotators examine summaries produced by several systems on the XSum dataset and found that a large majority contained hallucinated content, some of it contradicting the source. ROUGE, which compares only against the reference summary and never looks at the source document, could not detect this. Indeed, the XSum reference summaries themselves were found to contain information not present in the article, so a system that learned to imitate them was being trained to hallucinate. The lesson extends well beyond summarisation. Any metric that compares output to a reference measures similarity to that reference, not correctness. When the reference is one of many acceptable answers, similarity is a weak signal of quality. When a small change can flip meaning, similarity is a weak signal of correctness. Both conditions hold for most of what large language models are asked to do. Should you still use them? Despite all this, BLEU and ROUGE have not disappeared, and there are defensible reasons to compute them. They are cheap, deterministic and reproducible, provided a standard implementation is used. They make an adequate regression signal when a system's outputs are expected to stay close to fixed references: if a translation pipeline's corpus BLEU on a stable test set suddenly drops by several points after a code change, something has probably broken. They are useful for tasks where outputs are tightly constrained, such as converting a structured record into a templated sentence, where the space of acceptable answers is small. And they provide continuity with older literature. They are not appropriate as the primary evidence that one large language model is better than another at open-ended generation, as a measure of the quality of individual outputs, or as a measure of factual accuracy. When papers report ROUGE for chat models answering open questions, the number mostly measures how closely the model's style happens to match the reference writer's. The practical rule is to treat overlap metrics as smoke detectors rather than judges. A sudden change means look closer. A small difference means nothing on its own. And whenever an overlap metric is used to support a decision, it should be checked, at least once, against human judgment on the same data. The next chapter describes metrics designed to close the gap that these checks usually reveal. Chapter 4: Beyond Overlap: Learned Metrics, Execution and Model Judges The weaknesses of BLEU and ROUGE pointed in two directions. One was to keep comparing outputs with references but to compare meanings rather than strings. The other was to abandon references where possible and check outputs directly: run the code, verify the arithmetic, test each factual claim against a source, or ask a capable model to read the output and judge it. Both directions produced instruments that are now standard. Both also introduced new and subtler failure modes, which an evaluator has to understand before trusting the numbers. Embedding and learned metrics The first step beyond exact matching was to compare words by meaning. Modern language models represent each token as a vector of numbers, an embedding, in which words used in similar contexts end up close together. "Approved" and "passed", in the context of a budget, have nearby embeddings even though they share no letters. BERTScore, introduced by Tianyi Zhang and colleagues at Cornell and published at ICLR in 2020, exploits this. It passes both the candidate and the reference through a pretrained model such as BERT or RoBERTa, obtaining a contextual embedding for every token. Each candidate token is then matched with the most similar reference token, measured by cosine similarity, and the average of those best-match similarities gives a precision; matching in the other direction gives a recall; the harmonic mean gives an F-score. The council summary from the last chapter, which ROUGE-2 nearly zeroed because it said "passed" instead of "approved", scores well under BERTScore, because the substituted words are close in meaning. BERTScore fixes the paraphrase problem but not the faithfulness problem. "The council rejected the budget" and "the council approved the budget" are still very similar in embedding space, because the sentences share almost everything and "rejected" and "approved" occur in very similar contexts. Embedding similarity measures topical and lexical closeness; it is not a test of logical agreement. The next step was to train a metric directly on human judgments. BLEURT, from Thibault Sellam and colleagues at Google in 2020, and COMET, from Ricardo Rei and colleagues at Unbabel the same year, both take a pretrained language model and fine-tune it to predict the scores human raters gave to machine translations. COMET also reads the source sentence, not only the reference, which lets it notice when a translation diverges from what the original said. These metrics consistently outperform BLEU in the annual metrics shared task at the Conference on Machine Translation, and the Microsoft study discussed in the previous chapter found COMET among the most reliable for predicting which of two systems humans would prefer. Subsequent versions have added reference-free variants, which judge a translation from the source alone, sometimes called quality estimation. Learned metrics inherit the strengths and the blind spots of their training data. A metric trained on human ratings of news translation between major European languages is well validated for exactly that and progressively less validated as one moves to medical text, low-resource languages, or outputs from systems much stronger than those that produced the training data. They can also be fooled. Researchers have repeatedly shown that learned metrics can be insensitive to certain errors, such as wrong numbers or named entities, or can be gamed by outputs optimised against them. A system tuned directly to maximise COMET will learn some of COMET's quirks, which is Goodhart's law again. Checking instead of comparing For some tasks the best metric needs no reference at all, because the output can be checked. Code is the clearest case. In 2021 Mark Chen and colleagues at OpenAI introduced the HumanEval benchmark alongside the Codex model: 164 hand-written Python programming problems, each with a function signature, a docstring describing what the function should do, and a set of unit tests. A solution is correct if it passes the tests. This execution-based evaluation sidesteps the paraphrase problem entirely, since two programs that look nothing alike can both be correct, and it catches subtle bugs that a string comparison would miss. The paper also introduced a careful way of reporting results. Because a model can be sampled many times, the natural question is: if you generate k candidate programs, what is the chance at least one passes? This is pass@k. Naively generating exactly k samples and checking gives a very noisy estimate, so the authors generated n samples, with n larger than k, counted the number c that passed, and computed the unbiased estimate as one minus the probability that a random selection of k samples from the n contains none of the c correct ones. Pass@1 approximates what a user who takes the first answer experiences; pass@10 or pass@100 matters more when a system can generate many candidates and filter them with tests. Execution-based evaluation has its own validity limits. Tests can be incomplete, so a program that passes them can still be wrong for inputs the tests do not cover; a 2023 study by Jiawei Liu and colleagues, which extended HumanEval's tests in a version called HumanEval+, found that this lowered the measured pass rates of many models, meaning some previously "correct" solutions had been wrong all along. Tests can also be over-specific, rejecting correct solutions that differ in some detail the problem did not specify. That was precisely the defect OpenAI's 2026 audit found in SWE-bench Verified: of the problems it examined, a substantial share had tests that rejected correct alternative fixes or checked behaviour the issue description never mentioned. And short, self-contained functions are a narrow slice of real programming, which involves understanding large codebases, which is why repository-level benchmarks emerged. Mathematics with a final numerical answer can be checked by comparing the answer, after normalising formatting. Benchmarks such as GSM8K, a set of grade-school word problems released by OpenAI in 2021, and the MATH dataset of competition problems from Dan Hendrycks and colleagues the same year, are scored this way. The difficulty is extraction: the model's answer must be found in its response, and a parser that fails to find a correct answer written in an unexpected format will score it wrong. More than one published comparison of models has turned out to reflect differences in answer formatting rather than mathematical ability. Factual claims can, with effort, be checked against a source. FActScore, introduced by Sewon Min and colleagues in 2023, breaks a long generated text, such as a biography, into short atomic facts and checks each one against a knowledge source such as Wikipedia, reporting the fraction that are supported. The decomposition and the checking can both be done by a model, which makes it scalable, though each step introduces its own error. Retrieval-augmented systems are commonly evaluated with related checks: is each claim in the answer supported by the retrieved documents, and did the retrieval find the documents needed? A related family of methods checks faithfulness to a source using natural language inference, the task of deciding whether one piece of text entails, contradicts or is neutral toward another. Models trained on inference datasets can be applied to each sentence of a summary, asking whether the source document entails it. This is the logic behind metrics such as SummaC, from Philippe Laban and colleagues in 2022, which split documents and summaries into sentences and aggregate sentence-level entailment scores. A different route, taken by QAGS from Alex Wang, Kyunghyun Cho and Mike Lewis in 2020, generates questions from the summary, answers them once using the summary and once using the source, and treats disagreement between the answers as evidence of unfaithfulness. Unlike ROUGE, both approaches would flag the summary in Chapter 3 that turned "approved" into "rejected", because they compare the output against the source rather than against another person's summary. Their weakness is that the inference or question-answering models they rely on make errors of their own, especially on long documents and on claims that require combining information from several places. Structured outputs can be validated against schemas: is the JSON well formed, are required fields present, are values within allowed ranges? These checks are trivial, which is exactly why they are valuable. A surprising fraction of production failures in systems built on language models are failures of format rather than of reasoning. The general principle is to check what can be checked. Wherever some property of a good output can be verified mechanically, verifying it is cheaper and more reliable than any judgment, human or model. The remaining difficulty is that, for many tasks, the properties that can be checked are not the ones that matter most. Language models as judges For open-ended outputs where no mechanical check captures quality, the field has increasingly turned to using one language model to evaluate another. The approach, often called LLM-as-a-judge, has become ubiquitous because it is cheap, fast and, under the right conditions, agrees with people surprisingly well. The method has several forms. In single-answer grading, the judge is given a question, a response and a rubric, and asked to produce a score, for instance from one to ten. In pairwise comparison, the judge sees two responses to the same question and states which is better, or that they tie. In reference-guided grading, the judge also sees a reference answer, which helps on questions with a correct answer that the judge itself might not know. And in rubric-based or checklist grading, the judge answers a set of specific questions about the response, such as whether it states the refund window or whether it cites a source, each of which is narrower and more reliable than an overall quality score. The case for model judges was made most influentially by Lianmin Zheng and colleagues in 2023, in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." Comparing GPT-4's pairwise judgments with roughly three thousand expert human votes on multi-turn conversations, they found agreement above 80 per cent, about the same as the agreement between different human experts. The same paper also catalogued the problems. Position bias: judges tend to favour whichever response is presented first, or second, depending on the model. The standard mitigation is to run each comparison twice with the order swapped and count only consistent verdicts, treating disagreement as a tie. Verbosity bias: judges tend to prefer longer responses, even when the extra length adds nothing. This was a serious enough problem for the AlpacaEval benchmark, which ranks chat models by how often a model judge prefers their answers to a reference model's, that its authors developed a length-controlled version. Yann Dubois and colleagues reported in 2024 that controlling for length raised the benchmark's rank correlation with human-vote-based Chatbot Arena rankings from 0.94 to 0.98, and made it much harder to inflate a score by telling a model to be more verbose. Self-preference: judges tend to rate outputs from their own model family more highly. Subsequent research has suggested this is linked to the judge recognising text that resembles its own, perhaps because it finds such text more probable. The mitigation is to avoid using a model to judge itself where possible, or to use several judges from different families. Limited competence: a judge cannot reliably grade answers to questions it cannot itself answer. Zheng and colleagues found that judges were often misled by incorrect solutions to mathematics problems, even when the judge could solve the problem on its own. Giving the judge a reference answer, or asking it to solve the problem first, reduces this. Beyond these well-documented biases lie subtler ones. Judges can be swayed by confident tone, by formatting such as headings and bullet points, and by text in the response that addresses the judge directly. Their scores on absolute scales tend to bunch, with most responses receiving a seven or eight out of ten, which limits their ability to discriminate. And their judgments change when the judge model is updated, so a time series of scores from a judge that was silently upgraded halfway through is not one series but two. Making a model judge trustworthy None of these problems is a reason to abandon model judges. They are reasons to treat a model judge as an instrument that needs calibration against the construct, which in practice means calibration against careful human judgment. The procedure is straightforward in outline. Assemble a set of a few hundred outputs that span the range of quality the judge will meet in use, including the failure types that matter most. Have them graded by people who understand the task, using the same rubric the judge will use. Run the judge on the same set. Measure agreement, using a chance-corrected statistic such as Cohen's kappa for categorical verdicts or a rank correlation for scores, and compare it with the agreement between the human graders themselves. Then read the disagreements. They are usually concentrated in a few identifiable categories: the judge misses a particular kind of factual error, or penalises a legitimate refusal, or rewards a format the humans did not care about. Improvements then come from narrowing the judge's task. Overall quality scores on a ten-point scale are the least reliable form. Binary questions about specific properties, such as "Does the response state a specific refund deadline? Answer yes or no", are much more reliable, and several of them can be combined. Asking the judge to explain its reasoning before giving its verdict usually helps, as does providing a reference answer or the relevant source documents. Few-shot examples of graded outputs, drawn from the human-graded set, help the judge learn where the boundaries lie. A concrete example makes the difference between a vague and a narrow judge clear. Suppose the task is answering customer questions about an airline's baggage policy, using retrieved policy text. A vague judge prompt asks the model to "rate the helpfulness and accuracy of this response from 1 to 10." A narrow judge prompt looks more like this: You are checking one answer to a customer question about baggage policy. You are given the question, the policy excerpts the assistant retrieved, and the assistant's answer. Answer each item with YES or NO, then give a one-sentence reason quoting the answer or the policy. 1. Does the answer state an allowance, fee or limit that differs from the policy excerpts? 2. Does the answer include any allowance, fee or limit that does not appear in the policy excerpts at all? 3. If the policy excerpts do not cover the question, does the answer say so rather than guessing? 4. Does the answer address the customer's specific fare class or route, if one was given in the question? Each item corresponds to a failure mode identified by reading real transcripts. Items one and two separate contradiction from unsupported invention, which have different causes and different fixes. Item three tests a behaviour the vague prompt would probably have penalised, since a judge asked about helpfulness tends to prefer an answer over an admission of ignorance. Item four catches a common partial failure: a correct statement of the general rule that ignores the customer's particular circumstances. Requiring a quoted reason makes the judge's output auditable, so that a human reviewer can quickly see whether a NO was justified. Such a judge produces four binary signals per response rather than one fuzzy score, and each can be validated separately against human graders. In practice, some items will reach high agreement quickly, typically the ones that reduce to checking the answer against the provided text, while others, such as whether the answer addressed the specific fare class, may need clearer definitions or examples before the judge and the humans converge. That unevenness is information. It tells the evaluator which parts of the judge can be trusted at scale and which still need human eyes. Once agreement is acceptable, the judge can be used at scale, with a small fraction of its verdicts re-checked by humans on a regular schedule. That ongoing audit matters, because the population of outputs drifts as the system under evaluation changes. A judge validated on last quarter's model may meet new kinds of error in this quarter's. A final caution concerns optimisation. If a model judge's scores are used as the reward signal for training, the model being trained will, given enough pressure, find and exploit the judge's weaknesses. This is a well-known hazard in reinforcement learning from AI feedback, and it is the strongest possible form of Goodhart's law. The judge used to train a system should never be the only judge used to evaluate it. The instruments in this chapter and the last are the building blocks of every evaluation, public or private. The next chapter looks at how they are assembled into the public benchmarks by which the field measures its progress, and why those benchmarks so often say less than their headline numbers suggest. Chapter 5: Public Benchmarks and Their Discontents A benchmark is a fixed dataset, a fixed scoring procedure and, usually, a public table of results. Benchmarks are how a research field agrees on what progress means. They let a group in Beijing and a group in Toronto compare their models without trusting each other's private evaluations, they concentrate effort on problems the community thinks are important, and they provide a historical record. Much of the progress in machine learning over the past two decades has been organised around them, from ImageNet in computer vision to GLUE and SuperGLUE in language understanding. Benchmarks are also among the least reliable guides to whether a particular model suits a particular job. That is not a contradiction. A benchmark answers a narrow, standardised question very well and a broad, application-specific one poorly. The trouble comes when a number that answers the narrow question is read as an answer to the broad one, and when the conditions that made the narrow answer trustworthy quietly erode. A short history of saturation The pattern has repeated often enough to describe as a cycle. A benchmark is released with tasks that current models find hard. Scores climb, slowly at first and then quickly as the benchmark becomes a standard target. Within a few years the best models approach or exceed the estimated human baseline, differences between top models shrink to within noise, and the benchmark stops discriminating. The field moves on to a harder one. GLUE, a suite of language understanding tasks released in 2018, was overtaken by models within about a year, prompting the harder SuperGLUE in 2019, which was itself surpassed by models in less than two years. MMLU, the Massive Multitask Language Understanding benchmark released by Dan Hendrycks and colleagues in 2020, covered 57 subjects from elementary mathematics to professional law with multiple-choice questions, and for several years was the most quoted single number for a model's general knowledge. By 2024 frontier models were scoring close to 90 per cent and the differences between them had become hard to interpret. GPQA, released in 2023 by David Rein and colleagues, was explicitly designed to resist this: graduate-level questions in biology, physics and chemistry, written by domain experts and checked so that skilled non-experts with unrestricted web access could not easily answer them. The authors reported that experts with or pursuing PhDs in the relevant domains reached about 65 per cent accuracy, while highly skilled non-experts managed about 34 per cent. Within two years, frontier models were reported to exceed the expert figure comfortably on the benchmark's widely used Diamond subset. Humanity's Last Exam, organised by the Center for AI Safety and Scale AI and first released in January 2025, was the most ambitious attempt yet to build a benchmark that would last: 2,500 questions at the frontier of human knowledge, contributed by nearly a thousand experts around the world, and filtered so that questions the leading models of the day could answer were removed. At launch, the best models scored in single digits. A peer-reviewed description of the benchmark appeared in Nature in January 2026, by which time the leading score reported on its leaderboard had passed 38 per cent. Scores have continued to climb through 2026. Each new hard benchmark is a genuine achievement and each saturation is genuine evidence of capability. But the speed of the cycle has a consequence for practitioners: any specific benchmark number in a model announcement is likely to describe a test that is either too easy to discriminate among frontier systems or too new to have been examined for flaws. Contamination The most widely discussed threat to benchmark validity is contamination: the presence of test items, or their answers, in a model's training data. Large language models are trained on enormous scrapes of the public web, and benchmark questions are, almost by definition, published on the web, discussed in forums, copied into tutorials and embedded in code repositories. A model that has seen a question and its answer during training may answer it correctly by recall rather than by reasoning. Contamination is hard to rule out from outside. Training data for most frontier models is not disclosed, and even developers acting in good faith struggle to filter every paraphrase and translation of every test item from a corpus of trillions of tokens. The available evidence therefore comes from indirect tests, and several of these have produced striking results. In 2024, Hugh Zhang and colleagues at Scale AI built GSM1k, a new set of grade-school mathematics problems written from scratch to match the style and difficulty of the widely used GSM8K benchmark. Because the new problems had never been published, no model could have trained on them. When they evaluated a range of open and closed models on both sets, they found accuracy drops of up to 8 per cent on the new problems, with several model families showing evidence of systematic overfitting to the original benchmark, while many frontier models showed little or none. The method is a clean one: if a model is genuinely capable at a type of problem, its score should not depend much on whether the specific problems were previously public. OpenAI's February 2026 decision to stop reporting SWE-bench Verified, described in the introduction, combined contamination with a second problem. The company reported that all the frontier models it tested could reproduce, for some tasks, either the original human-written fix or verbatim details of the problem, and that one of its own models had solved tasks that were nearly impossible to solve without such outside information. It recommended SWE-bench Pro, a newer and harder benchmark, in its place. Contamination is also a matter of degree. A model may not have seen a test question verbatim but may have seen a near paraphrase, the same problem with different numbers, or a discussion of the solution method in a forum. These partial exposures inflate scores in ways that are very difficult to measure. Label errors and ambiguous items A benchmark is only as good as its answer key, and answer keys for large benchmarks are written by people under time pressure. In 2024, Aryo Pradipta Gema and colleagues at the University of Edinburgh published "Are We Done with MMLU?", in which they manually re-examined 5,700 questions across all 57 MMLU subjects. They estimated that about 6.5 per cent of MMLU questions contained errors, including wrong answer keys, questions with more than one correct option and questions with no correct option. The errors were not evenly spread. In the virology subset, they found problems with 57 per cent of the questions they analysed. They released a corrected subset, MMLU-Redux, and showed that model rankings changed when evaluated on it. Errors at this rate matter most at the top of the leaderboard. When the best models score around 90 per cent on a benchmark with a 6 per cent error rate, much of the remaining gap to 100 per cent may be unanswerable as keyed, and the differences between the top models may be dominated by which of them happens to agree with the wrong keys. A model that scores higher may be the one that has memorised more of the errors. Sensitivity to format Language models are sensitive to details of presentation that a human test-taker would ignore. Changing the order of the options in a multiple-choice question, the letters used to label them, the instructions given or the number of worked examples in the prompt can move a model's score by several percentage points. Different evaluation frameworks make different choices about these details, and they also differ in whether they score a model by the probability it assigns to each option, as described in Chapter 2, or by parsing the option it writes out. The same model on the same benchmark can thus have several legitimate scores. For agentic benchmarks, where the model must use tools, browse or edit code over many steps, the surrounding infrastructure matters even more. The scaffold, meaning the code that feeds the model observations, executes its actions and decides when to stop, can change scores substantially, as can the time and token budget allowed. A reported score on such a benchmark describes a model and a scaffold together, and comparisons that hold one fixed and vary the other are rarely possible from published numbers alone. Arenas and their pitfalls One response to the limits of static benchmarks has been to rank models by live human preference. The best known example is Chatbot Arena, launched in 2023 by researchers at LMSYS and later operated as LMArena. Users visit the site, type a prompt of their choosing, receive anonymous responses from two models, and vote for the one they prefer. The votes are aggregated with a statistical model of paired comparisons, the Bradley-Terry model, which estimates a strength rating for each model such that the predicted probability of one beating another depends on the difference in their ratings. It is similar in spirit to the Elo system used for chess. The arena approach has real strengths. Prompts come from a broad population of real users rather than from a fixed set that can be memorised. Human preference is the construct many developers actually care about. And the scale is large, with millions of votes collected over time. It also has well-documented weaknesses. The population of arena users is not the population of any particular product's users; it skews toward technically engaged people and toward short, single-turn prompts. Preference votes reward style as well as substance: responses that are longer, more confidently written or more elaborately formatted tend to win more often, which led the arena's operators to introduce a style-controlled ranking that adjusts for such features. And because the ranking is a single public number of great commercial importance, it has become a target. In 2025 a group of researchers led by Shivalika Singh and Sara Hooker, then at Cohere, published "The Leaderboard Illusion," a detailed analysis of arena practices. They reported that some providers had been able to test many private model variants before a public release and publish only the best-scoring one; one major lab, for example, had tested 27 private variants in the run-up to a single model launch. They also estimated that a small number of large providers received a disproportionate share of the arena's battle data, which can itself be used to tune models toward arena preferences, and that proprietary models were sampled more often and removed less often than open ones. The arena's operators disputed parts of the analysis and changed some policies. Whatever the merits of each point, the episode illustrated a general truth: the selection of the best of many attempts against a noisy public measure will inflate that measure, regardless of anyone's intentions. Benchmarks designed to resist decay The threats above have prompted several design responses, and knowing them helps in choosing which public numbers deserve more weight. The first response is temporal holdout: build the test set from material created after the models being tested finished training. LiveCodeBench, introduced by Naman Jain and colleagues in 2024, collects new programming problems from competitive programming sites continuously and tags each with its release date, so that a model can be evaluated only on problems published after its training cut-off. Comparing a model's performance on problems from before and after its cut-off is itself a contamination test: a sharp drop at the cut-off suggests the earlier problems were partly memorised. LiveBench, from Colin White and colleagues the same year, applies a similar idea across mathematics, coding, reasoning, data analysis and language tasks, refreshing questions regularly and using answers that can be scored objectively, to avoid relying on model judges. The second response is private or semi-private test sets. Some benchmark maintainers keep part or all of their test data unpublished and score models submitted to them, so that the questions cannot leak onto the web. Scale AI's SEAL leaderboards and the held-out portion of Humanity's Last Exam follow this approach in different ways. The trade-off is reduced transparency: outsiders must trust the maintainer's procedures, and cannot inspect the items for errors. The third response is breadth with standardisation. The Holistic Evaluation of Language Models project, HELM, launched in 2022 by Percy Liang and colleagues at Stanford's Center for Research on Foundation Models, evaluated many models across many scenarios and several metrics per scenario, including accuracy, calibration, robustness to perturbations and efficiency, all under a single, documented set of prompts and settings. Its central argument is that a single headline number hides the trade-offs that matter, and that comparisons are only meaningful when the evaluation conditions are identical and published. The fourth response is auditing and repair. MMLU-Redux, discussed above, and SWE-bench Verified itself, which was created in 2024 when OpenAI had human engineers screen the original SWE-bench to remove tasks that were underspecified or had unfair tests, are attempts to fix a benchmark's answer key rather than replace the benchmark. The later discovery that even the screened version had substantial problems is a reminder that auditing reduces error but does not eliminate it, and that a benchmark's quality is never settled once and for all. None of these designs makes a benchmark immune to the pressures described in this chapter. Temporal holdouts protect against contamination but not against format sensitivity. Private test sets protect against leakage but not against selection among many submitted variants. Breadth helps against overfitting to a single task but makes results harder to summarise. They do, however, give the reader something to ask about when a new score is announced: was this benchmark built to resist the problems that eroded its predecessors? Reading a benchmark claim Table 3 summarises the main threats to benchmark validity and the questions that expose them. Table 3. Common threats to the validity of a public benchmark score. Threat What happens Question to ask Saturation Top scores cluster near the ceiling Is the gap larger than the noise? Contamination Test items appeared in training data Does the score hold on fresh, matched items? Label error The answer key is wrong or ambiguous Has the benchmark been audited? Format sensitivity Prompt or parsing details move the score Was the setup identical across models? Selective reporting Best of many runs or variants is shown How many attempts preceded this one? Population mismatch Test items differ from real use Do these tasks resemble mine? The practical upshot is a set of habits. Treat a benchmark score as evidence about capability on the benchmark's specific task under its specific setup, and nothing more until shown otherwise. Prefer benchmarks that have been independently audited, that have held-out or regularly refreshed test sets, or whose items were created after the model's training cut-off. Pay attention to confidence intervals, which on benchmarks of a few hundred items are often wider than the differences being advertised. Distrust any single number that a developer has had strong incentives and ample opportunity to optimise. And above all, recognise that the question "which model is best?" has no answer independent of the task. The only benchmark that can answer it for your application is one built from your application, which is the subject of the next chapter. Public benchmarks are not useless. They remain the best available shared language for describing what models can do, and the best of them are built with great care. Their proper role is as a first filter: a way of deciding which handful of models deserve a closer look, not which one to deploy. Chapter 6: Designing Your Own Evaluation Dataset Every serious deployment of a language model eventually needs an evaluation set of its own. Public benchmarks measure what the field cares about in general; a custom evaluation measures what a specific system must do for specific users. It is also the one instrument that no model developer can have trained on, because it does not exist until you build it. Building one well is less a matter of volume than of design. A carefully chosen set of two hundred examples, each with a clear definition of success, usually tells a team more than ten thousand examples scraped from logs and scored by a vague rubric. This chapter works through the design decisions in roughly the order they arise, using a running example: a hypothetical assistant that answers questions from employees of a mid-sized company about its human-resources policies, drawing on the company's handbook and policy documents through retrieval. Start from the decision and the failure modes The first step is not to collect data but to write down what the evaluation is for. For the HR assistant, the immediate decision might be whether it is ready for a pilot with one department. The questions that bear on that decision are concrete: does it answer common questions correctly, does it cite the right policy, does it avoid giving confident answers to questions the documents do not cover, does it handle sensitive topics such as harassment complaints or medical leave with appropriate care, and does it refuse to disclose information about specific named employees? Next, list the ways it could fail that would matter. This list is the backbone of the dataset. A useful practice is to separate failures by severity. Some failures are inconvenient: a correct answer phrased awkwardly. Some are costly: a wrong answer about how many days of parental leave an employee is entitled to, which might lead someone to make a decision they cannot reverse. Some are unacceptable: revealing another employee's salary, or discouraging someone from filing a complaint. The dataset should be weighted toward the costly and unacceptable categories even if they are rare in practice, because those are the failures the evaluation exists to catch. If the system is already running, even as a prototype used by a handful of colleagues, the best source of failure modes is its own transcripts. Read a hundred of them. Note every response that a careful HR professional would object to and write down, in a phrase, what was wrong. After a few dozen, patterns emerge: the system confuses two similar policies, it applies the full-time rule to part-time staff, it answers from general knowledge when the handbook is silent. This open-ended reading, sometimes called error analysis, is the single most valuable evaluation activity, and it is routinely skipped in favour of building automated metrics before anyone knows what they should measure. Sourcing examples Examples can come from several places, and a good dataset usually mixes them. Real usage is the most valid source, because it reflects what people actually ask, in the way they actually ask it. Sampling should be deliberate: a purely random sample of logs will be dominated by the most common, easiest questions. Stratify by topic or by the failure categories identified earlier, and oversample the rare but important ones. Real usage data often contains personal information and must be handled accordingly: anonymised, access-controlled and used within whatever consent users have given. Expert-written examples fill gaps. Before launch there may be no real usage at all, and even afterwards some important cases will be too rare to sample. Ask people who know the domain to write questions that a real employee might ask, including awkward and borderline ones. Encourage variety in phrasing, length and tone; experts writing test cases tend to write clean, well-formed questions, while real users write fragments, typos and questions that bundle three issues together. Adversarial examples probe specific weaknesses: attempts to extract confidential information, questions with false premises ("Since we get 30 days of holiday, can I carry 10 over?" when the policy grants 25), prompts that try to override the system's instructions. These are best written by people who have read the error analysis and are trying to break the system. Synthetic examples generated by a language model can multiply a small seed set, producing paraphrases and variations. They are useful for testing robustness to phrasing and for scaling up coverage of known categories. They should not be the backbone of a dataset, because a model generating test cases tends to produce the kinds of question it finds natural, which are often the kinds it also finds easy. Every synthetic example that enters the final set should be reviewed by a person. Defining what counts as correct For each example, the dataset must say what a good response looks like. This is where most custom evaluations are weakest, and where the most leverage lies. For questions with a definite answer, the simplest form is a reference answer, but as Chapter 3 showed, comparing outputs to a single reference string is a poor test. Better is a set of required elements: the facts or actions the response must contain. For "How much parental leave do I get?", the required elements might be the number of weeks, the eligibility condition, the policy document's name and an indication that the employee should confirm with HR for their specific case. Add prohibited elements: statements that would make the response wrong or harmful, such as stating a different number of weeks or implying eligibility for contractors who are excluded. For questions without a definite answer, write a rubric. A good rubric is specific to the task, describes observable features of the response rather than impressions, and distinguishes levels with examples. "The response is helpful" is not a rubric criterion; "The response identifies the relevant policy by name and states which section applies" is. Rubrics should be tested by having two people apply them independently to the same twenty or thirty responses and comparing results. Where they disagree, the rubric is ambiguous, and it should be revised until they mostly agree. Some examples call for the system to decline or redirect, and the expected behaviour must be specified just as concretely: which questions should be refused, which redirected to a human, and what a good redirection looks like. An evaluation that only rewards answering will push a system toward answering everything. Each example should also carry metadata: its category, its source, its severity tier, the date it was added and, for real-usage examples, enough provenance to trace it. This metadata makes it possible to report results by slice, which is often more informative than the overall score. A system that improves by three points overall while getting worse on the highest-severity category has not improved in the way that matters. How many examples are enough The number of examples needed depends on the size of the difference you need to detect and on how much noise there is. A rough guide comes from the arithmetic of proportions. If a system's true accuracy is around 80 per cent, then on a test set of n independent examples the 95 per cent confidence interval for its measured accuracy has a half-width of about 1.96 times the square root of (0.8 × 0.2 / n). Table 4 shows what this means at several sizes and at two accuracy levels. Table 4. Approximate 95 per cent confidence interval half-width for a measured accuracy, by test set size (normal approximation, independent examples; author's calculation). Examples At 50% accuracy At 80% accuracy At 95% accuracy 50 ±13.9 points ±11.1 points ±6.0 points 100 ±9.8 points ±7.8 points ±4.3 points 200 ±6.9 points ±5.5 points ±3.0 points 500 ±4.4 points ±3.5 points ±1.9 points 1,000 ±3.1 points ±2.5 points ±1.4 points The table explains a great deal of confusion in practice. With 100 examples, a system measured at 80 per cent could plausibly have a true accuracy anywhere from about 72 to 88 per cent. Two systems measured at 78 and 82 per cent on that set are not distinguishable. Teams that make decisions on such differences are, in effect, flipping coins. Two things make the picture better than the table suggests. First, when comparing two systems on the same examples, the right analysis is paired: for each example, does system A succeed where B fails, or vice versa? Examples both systems get right, or both get wrong, carry no information about the difference, and the paired analysis uses this to reduce noise considerably. A test such as McNemar's, which looks only at the discordant examples, is the classical tool. Second, if outputs are sampled with randomness, scoring several samples per example and averaging reduces the noise contributed by sampling, though not the noise contributed by the choice of examples. A worked example shows how much the paired view helps. Suppose two versions of the HR assistant, A and B, are each run on the same 200 examples. A succeeds on 160, an accuracy of 80 per cent; B succeeds on 170, an accuracy of 85 per cent. Treated as two independent samples, the five-point difference is well within the noise: each estimate carries an interval of about five or six points, and the intervals overlap substantially. Now look at the examples individually. Suppose 150 examples are answered correctly by both versions and 20 are failed by both. Those 170 examples say nothing about which version is better. The information lies in the remaining 30: on 20 of them B succeeds where A fails, and on 10 A succeeds where B fails. If the two versions were really equally good, each of these 30 discordant examples would be equally likely to fall either way, like a fair coin. Getting a split of 20 to 10 or more lopsided in either direction from 30 fair coin tosses happens roughly one time in ten. That is suggestive but short of the conventional threshold for statistical significance; a split of 22 to 8 would cross it. McNemar's test formalises exactly this reasoning. The paired view does not make every difference real, but it measures the evidence far more precisely than comparing two overall accuracies, and it shows exactly how many more discordant examples would settle the question. Two further lessons follow from the example. First, the strength of the evidence depends on the discordant examples, not on the overall accuracy. If the same 20-to-10 split had come from 300 examples instead of 200, the evidence about the difference would be no stronger, because the extra examples would all be ones both versions got right or wrong. Second, the discordant examples are where to look next. Reading the twenty cases that B fixed and the ten that it broke shows what the change actually did, and frequently reveals that the ten regressions share a cause that can be fixed without losing the gains. Two things make it worse. When examples come in clusters, such as several questions about the same policy document, they are not independent, and the effective sample size is smaller than the count. And when a grader, human or model, makes errors, those add noise of their own. In practice, a few hundred examples is a reasonable target for a first serious evaluation set, with the most important slices large enough on their own to be measured, perhaps fifty or more examples each. Smaller sets remain useful for qualitative review and for catching outright breakage. Much larger sets help only if their examples are as carefully designed as the small ones. Labelling retrieval as well as answers For a system that answers from retrieved documents, as the HR assistant does, the dataset should record not only what a good answer contains but which documents a good answer needs. This additional labelling roughly doubles the effort per example and is usually worth it, because it separates two failures that look identical from the outside. When the assistant gives a wrong answer about parental leave, either retrieval failed to surface the parental leave policy, or retrieval surfaced it and the model misread or ignored it. The fixes are entirely different. The first calls for better indexing, chunking or query rewriting; the second for a better prompt, a different model or a constraint on answering only from cited text. Without knowing which documents should have been retrieved, the team cannot tell which problem it has, and may spend weeks tuning prompts to fix a search problem. The labelling itself is straightforward but requires domain knowledge. For each example question, a person who knows the policies identifies the document, and ideally the section, that contains the answer. Some questions need more than one document, for instance when a general leave policy is modified by a regional addendum, and those multi-document examples are especially valuable, because they test whether retrieval returns everything needed rather than just the most obvious match. Questions whose answer is in no document should be labelled as such, so that the evaluation can check that the system recognises the gap instead of answering from general knowledge. With these labels, retrieval can be scored on its own with measures such as recall at k, and the answer can be scored conditional on retrieval: how often is the answer correct when the right document was retrieved, and how often when it was not? A system that answers correctly 95 per cent of the time when retrieval succeeds but retrieves the right document only 70 per cent of the time has a retrieval problem, whatever its overall score suggests. Keeping the dataset honest An evaluation set is a living artefact and decays in predictable ways. It leaks into development. Engineers who see the evaluation examples repeatedly start, consciously or not, to tune prompts to them. Over time the system's performance on the set rises faster than its performance in real use. The defence is a split: a development set that the team uses freely for iteration and a held-out test set that is consulted rarely, ideally only at release decisions, and is never used for tuning. When the held-out set has been consulted many times, it should be partly refreshed. It drifts from reality. User behaviour changes, policies are updated, new features open new failure modes. The dataset should be reviewed on a schedule, with stale examples retired or updated and new examples added from recent error analysis. Each production incident should become a test case, so that the same failure cannot recur silently. It accumulates errors. Reference answers go out of date when the underlying policy changes, and a few were wrong from the start. When a system's output is marked wrong, someone should occasionally check whether the system was right and the key was wrong. The MMLU experience described in the previous chapter shows that even carefully built datasets carry error rates high enough to distort conclusions. It must be versioned. Every evaluation result should record which version of the dataset produced it. A change in score caused by a change in the dataset is indistinguishable from a change caused by the system unless the dataset version is recorded. Scoring the custom set With examples and success criteria defined, scoring follows from the instruments of the earlier chapters. Required and prohibited elements can often be checked by a model judge asked a series of narrow yes-or-no questions, validated against human grading as Chapter 4 described. Citations can be checked mechanically against the retrieved documents. Refusal behaviour can be classified. Format can be validated by code. The overall result should be reported as a profile across categories and severity tiers, with confidence intervals, rather than as a single number. For the HR assistant, such a profile might show that factual accuracy on common questions is high, that the system cites the correct policy most of the time, that it correctly declines most questions about named individuals, and that it fails on a specific category, say questions involving part-time staff, at a much higher rate than elsewhere. That last finding is worth more than the overall score, because it tells the team exactly what to fix and gives them the examples to check the fix against. Some of the criteria in a custom evaluation, however, cannot be scored by any automated means with confidence. Whether a reply to an employee asking about bereavement leave strikes the right tone, or whether a summary of a grievance procedure is clear to a non-specialist, are judgments that people make well and machines make unevenly. These are the cases for human grading, which the next chapter takes up. Chapter 7: Human Judgment for Subjective Tasks Every automated instrument in this book is ultimately justified by appeal to people. BLEU was adopted because it correlated with human ratings of translation. COMET was trained to predict them. Model judges are validated by their agreement with human graders. Chatbot Arena ranks models by human votes. Strip away the machinery and the question underneath is always the same: would a competent person, looking at this output for this purpose, consider it good? For tasks with verifiable answers, automated checks can stand in for that person most of the time. For subjective tasks, where quality depends on judgment about tone, clarity, emphasis, appropriateness or usefulness to a particular reader, they cannot, at least not without being calibrated against people first and audited by people continuously. Human evaluation is therefore not an optional extra for well-funded teams. It is the reference standard on which the whole ladder of cheaper instruments rests. If the reference standard is poorly built, everything calibrated against it inherits the flaws. The uncomfortable truth is that human evaluation is easy to do badly, and badly done human evaluation can be worse than none, because it carries an authority it has not earned. What goes wrong with human judgment The research literature on human evaluation of generated text contains a string of sobering findings. In 2021, Elizabeth Clark and colleagues at the University of Washington published "All That's 'Human' Is Not Gold." They asked untrained evaluators to distinguish human-written stories, news articles and recipes from texts generated by GPT-3, and found that the evaluators did so at roughly chance level. Training the evaluators with examples improved accuracy only modestly. The broader point was not about detection but about attention: untrained raters judging fluent text tend to focus on surface features such as grammar and style and miss deeper problems of coherence and fact. The same year, Marzena Karpinska, Nader Akoury and Mohit Iyyer examined the common practice of collecting ratings of generated stories from crowdworkers on Amazon Mechanical Turk. They found that the ratings were often unreliable, with workers in some configurations unable to distinguish model-generated stories from human-written ones consistently, and that many published papers reported too little about their evaluation setup for readers to judge its quality. They recommended, among other things, using workers with relevant expertise and reporting agreement statistics and payment. In machine translation, Markus Freitag and colleagues at Google published "Experts, Errors, and Context" in 2021. They had professional translators annotate translations using the Multidimensional Quality Metrics framework, which requires marking each specific error, classifying its type and assigning it a severity. Compared with the crowd-sourced ratings used in that year's official evaluation campaign, the expert annotations produced substantially different system rankings. In particular, the experts clearly preferred human translations to machine output, whereas the crowd ratings had not, and the experts' judgments aligned much better with the newer learned metrics than with BLEU. The quality of the human evaluation determined which automatic metrics looked good. Most pointedly for language model assistants, Tom Hosking, Phil Blunsom and Max Bartolo published "Human Feedback is not Gold Standard" in 2023. They showed that human preference ratings were influenced by the assertiveness of a response, so that more confident responses were perceived as more factual, and that annotators under-counted factual errors relative to other criteria. Since such preference judgments are used to train models, a systematic tendency to reward confidence over accuracy can be learned and amplified. None of these findings says that human judgment is unreliable in principle. They say that untrained, unmotivated, poorly instructed people, asked vague questions about outputs they do not have the expertise to assess, produce judgments that do not measure what the evaluator hoped. Each finding also points to a remedy. Who should grade The first design decision is who the graders are, and the answer depends on the construct. For tasks where quality depends on domain knowledge, such as whether a medical explanation is accurate, a legal summary misses a material clause or code is idiomatic, graders need that knowledge. A generalist rater can judge whether an answer about tax law sounds confident and well organised; only someone who knows the law can judge whether it is right. This is expensive. It is also unavoidable, because the alternative is to measure confidence and organisation while believing one has measured correctness. For tasks where quality depends on the experience of a particular audience, such as whether an explanation is clear to a first-year student or a reply feels respectful to a customer who has had a bad experience, graders should resemble that audience, or should be experts trained to take its perspective. A panel of engineers is a poor proxy for elderly customers. For tasks that combine both, a common pattern is to split the evaluation: experts judge accuracy, and audience-like raters judge clarity and tone. Splitting also makes each judgment simpler, which improves reliability. Whatever the pool, graders must be paid fairly and given enough time. Rushed raters fall back on surface features. A useful habit is to record how long each judgment takes and to look carefully at raters who are much faster than the median. What to ask The second decision is what form the judgment takes. There are three broad options, and they suit different purposes. Absolute ratings ask a grader to score a single output on a scale, such as one to five for helpfulness. They are simple to collect and give a number for each output, but they suffer from scale drift: different graders use the scale differently, and the same grader's standards shift over a session and between sessions. They are most reliable when each point on the scale is anchored with a concrete description and an example. Pairwise comparisons show two outputs for the same input and ask which is better. People find relative judgments easier and more consistent than absolute ones, which is why preference data for training assistants and arena-style rankings both use them. Their drawback is that they say which is better but not whether either is good enough; two bad answers can be compared as easily as two good ones. They also need care with presentation order, which should be randomised, and with ties, which should be allowed. Error annotation asks graders to find and classify specific problems: a factual error here, an omission there, a tone problem in the closing sentence, each with a severity. The MQM framework used in translation is the best developed example. Error annotation is the slowest form but the most informative, because it says not just how good an output is but what is wrong with it, which is what a team needs to fix it. It is also the most resistant to the surface biases described above, because it forces graders to look for specific faults rather than form an overall impression. Table 5 compares the three on the properties that usually drive the choice. Table 5. Three forms of human judgment compared. Form Consistency across raters Cost per item Diagnostic value Best use Absolute rating Low to moderate Low Low Tracking a single system over time Pairwise comparison Moderate to high Low to moderate Low Choosing between two systems Error annotation High with training High High Diagnosing and calibrating In practice, many good evaluations combine them: a pairwise preference, followed by a short checklist of yes-or-no questions about specific properties, followed by an optional free-text note on anything the checklist missed. Writing instructions that work Instructions to graders are a measurement instrument in their own right, and they deserve the same iterative design as a rubric for a model judge. Several practices consistently help. State the purpose and the audience. Graders judge differently when told that responses are for "a customer who is upset about a late refund" than when told only to rate helpfulness. Give them the context the real user would have, and any source documents the system had. Define each criterion operationally. Instead of "Is the response accurate?", ask "Does the response contain any statement that contradicts the attached policy document? If so, quote it." Instead of "Is the tone appropriate?", describe what inappropriate tone looks like in this setting, with examples. Separate criteria. Asking for a single overall score invites graders to let one dimension, usually fluency, dominate. Asking for accuracy, completeness and tone separately, before any overall judgment, forces attention to each. Provide calibration examples. A set of ten or twenty pre-graded outputs, with explanations of why each received its grade, does more to align graders than pages of abstract guidance. Include borderline cases, because that is where graders diverge. Pilot, measure, revise. Run the instructions with a few graders on a small batch, measure their agreement, discuss the disagreements and revise. Disagreements usually reveal ambiguity in the instructions rather than carelessness in the graders. Two or three rounds of this typically produce much tighter agreement than the first draft. Measuring agreement Agreement between graders is the main evidence that a human evaluation is measuring something stable. Raw percentage agreement is easy to compute but misleading, because two graders who each say "acceptable" to 90 per cent of outputs will agree at least 80 per cent of the time by chance alone. Chance-corrected statistics address this. Cohen's kappa compares two raters' observed agreement with the agreement expected by chance given each rater's overall tendencies; a kappa of zero means agreement no better than chance and one means perfect agreement. Krippendorff's alpha, developed by the communication scholar Klaus Krippendorff, generalises the idea to any number of raters, to missing ratings and to ordinal or interval scales, and is often the more practical choice. A small example shows why the correction matters. Two graders each judge 100 responses as acceptable or unacceptable. Grader one marks 85 acceptable; grader two marks 80 acceptable. They agree on 75 acceptable and 10 unacceptable, so they agree on 85 of the 100, an observed agreement of 85 per cent, which sounds high. But if their judgments were entirely unrelated, with each simply marking responses acceptable at their own overall rate, they would still agree by chance on a good share. The chance that both mark a given response acceptable is 0.85 × 0.80, or 0.68; the chance that both mark it unacceptable is 0.15 × 0.20, or 0.03. Expected chance agreement is therefore 71 per cent. Kappa is the observed agreement beyond chance divided by the maximum possible agreement beyond chance: (0.85 − 0.71) divided by (1 − 0.71), which is about 0.48. On commonly used rules of thumb, such as the scale proposed by J. Richard Landis and Gary Koch in 1977, that counts as only moderate agreement. The 85 per cent figure was mostly a consequence of both graders finding most responses acceptable. The example also shows why agreement statistics should be computed on the items that matter. If the purpose of the evaluation is to catch unacceptable responses, the relevant question is how often the graders agree on which responses those are. Here, of the 15 responses grader one marked unacceptable and the 20 grader two marked unacceptable, they agreed on only 10. That is the number that would worry an evaluator relying on either grader alone to find failures, and it is invisible in the headline percentage. There is no universal threshold for acceptable agreement, but it is worth knowing what is typical. When OpenAI trained InstructGPT in 2022, the paper reported that its training labelers agreed with each other about 73 per cent of the time on pairwise preferences, and that a separate group of held-out labelers agreed with one another about 77 per cent of the time. For subjective preference between two capable outputs, agreement in this range is normal. It means roughly one comparison in four is genuinely contested. That last point deserves emphasis. On subjective tasks, disagreement is partly signal. Some outputs really are borderline, and some criteria really do depend on taste or perspective. A grading process that forces disagreement down to zero, for example by having graders discuss every item until they agree, may simply impose one person's view. The better practice is to measure disagreement, reduce the part that comes from ambiguous instructions, and treat the remainder as a property of the task. Where several graders disagree about an item, that item is probably one where the system's behaviour matters less than the clarity of the policy behind it. When agreement is low and does not improve with revised instructions, the criterion is probably not well defined enough to measure. That is useful knowledge. It means that no automated metric, however sophisticated, can be validated against it either. Managing a grading operation over time A one-off human evaluation is a study. A recurring one is an operation, and operations have failure modes that studies do not. The first is rater drift. Graders' standards shift over weeks and months, often without their noticing. They become more lenient as they grow used to a system's characteristic weaknesses, or harsher after seeing a run of particularly bad outputs. The standard defence is to seed each batch with a small number of gold items, outputs whose correct grades were settled by careful discussion earlier, and to track each grader's agreement with the gold over time. A grader whose agreement falls is not necessarily careless; the gold items may have become outdated, or the grader may have noticed something the original panel missed. Either way, the drop is a prompt for a conversation. The second is fatigue. Judgment quality declines over long sessions, and the decline is concentrated in exactly the subtle cases that most need careful attention. Limiting session length, breaking work into batches with pauses, and randomising the order in which items from different systems appear all help, the last because it prevents fatigue from falling systematically on one system's outputs. The third is leakage between graders and developers. If the people grading outputs know which system produced them, or know which version the team hopes will win, their judgments will lean, however conscientious they are. Blinding graders to the source of each output, and randomising presentation, is as important here as in any clinical trial. So is keeping a degree of separation between the people who build a system and the people who certify it, at least for high-stakes decisions. The fourth is erosion of the guidelines. As new kinds of output appear, graders improvise rulings on cases the instructions do not cover, and different graders improvise differently. A shared log of such rulings, reviewed periodically and folded back into the instructions, keeps the operation consistent. Many of the most useful clarifications in mature grading guidelines began as a single grader's note that the rubric did not say what to do. Human judgment as calibration Human evaluation is expensive, so it has to be used where it adds most. Three uses stand out. The first is calibration: grading a sample of outputs so that cheaper instruments, especially model judges, can be validated against them, as Chapter 4 described. A few hundred carefully graded items, refreshed periodically, can underpin millions of automated judgments. The second is high-stakes decisions: before a launch, before switching to a new model version, or after a significant change to a system's behaviour, a direct human evaluation of the decision-relevant slices provides evidence that no automated proxy can. The third is discovery: having experts read outputs with no fixed rubric, looking for anything that seems wrong. This open-ended reading finds failure modes no one thought to test for, which can then be turned into dataset examples and automated checks. The thread running through all three is that human graders are most valuable when they are treated as experts doing careful work, not as a commodity producing labels. A small number of well-trained, well-paid, well-instructed graders whose agreement is measured and whose disagreements are studied will produce a better reference standard than a large crowd hired by the item. And because every other instrument is ultimately calibrated against that standard, the quality of the human evaluation sets a ceiling on the quality of the whole evaluation programme. The final chapter describes how to build that programme. Chapter 8: Building an Evaluation Programme The previous chapters examined instruments one at a time. In practice, no single instrument is enough, and the value of evaluation comes from how instruments are combined into a standing system that runs alongside development and deployment. A team with such a system can change a prompt on Tuesday, know by Wednesday whether it broke anything, and decide on Thursday whether to ship. A team without one changes things and waits to hear from users. This chapter describes the shape of a working evaluation programme. The details vary enormously with the size of the team and the stakes of the application, but the structure is remarkably consistent across organisations that do this well. The ladder of instruments The organising idea is a ladder. At the bottom are cheap, fast, automated checks that run on every change. At the top are expensive, slow, careful human judgments that run rarely. Each rung is validated against the rung above it, and each is used for the decisions its cost and reliability suit. The lowest rung is deterministic checks: schemas validated, required fields present, forbidden strings absent, responses within length limits, citations pointing to documents that were actually retrieved, code that compiles and passes its tests. These cost almost nothing, never disagree with themselves and catch a surprising share of real breakage. They should run on every commit, like unit tests in conventional software. Above that sits the automated regression suite: the custom evaluation set described in Chapter 6, or its development portion, scored by a combination of checks and validated model judges. This takes minutes to hours and some modest spending on model calls. It should run on every meaningful change to prompts, models, retrieval or tools, and its results should be compared with a stored baseline, with differences reported per category and with confidence intervals. Above that is periodic human review: a sample of outputs graded by trained people against the rubric, both to evaluate the system directly on the hardest criteria and to audit the model judges. This might run weekly or at each release candidate. At the top is deep evaluation for major decisions: a full run on the held-out test set, expert error annotation on the highest-severity categories, perhaps a side-by-side comparison with the incumbent system judged by people who resemble real users. This runs at launches and model migrations. Alongside the ladder runs production monitoring, which is evaluation of a different kind and is discussed below. The essential discipline is calibration between rungs. When the human review disagrees with the model judge, that is not merely a fact about one batch; it is evidence the judge needs adjusting. When a production incident reveals a failure the regression suite did not catch, the suite needs a new example. Information flows up and down the ladder, and the programme improves only if someone is responsible for moving it. Evaluating systems, not models A recurring theme of this book is that the object of evaluation is almost always a system rather than a model. In 2026 that point has become sharper, because many deployed systems are agents: they plan, call tools, read results, and take further steps, sometimes dozens of them, before producing anything a user sees. Evaluating such systems requires instruments at two levels. End-to-end evaluation asks whether the task was accomplished: was the customer's issue resolved, was the right file changed, did the booking go through with the correct details? This is the measure that matters, but when it fails it rarely says why. Component evaluation asks whether each part did its job: did retrieval find the relevant document, did the planner choose a sensible next step, was the tool called with valid arguments, did the final answer reflect what the tools returned? Component metrics are more diagnostic and often easier to automate. Retrieval, for example, can be evaluated with standard information-retrieval measures such as recall at k, meaning the fraction of cases in which the needed document appears among the top k results, provided someone has labelled which documents are relevant for each test query. For agents, the trace of intermediate steps is itself an object of evaluation. Two runs can reach the same correct outcome, one efficiently and one by trial and error that would be expensive or risky in production. Some failures appear only in the trace, as when an agent with access to a payment tool calls it in a context where it should have asked for confirmation, even if the final result happens to be correct. Logging complete traces and building checks that run over them is therefore part of the programme. Agents also complicate reproducibility. Their behaviour depends on external state, such as the contents of a database, the response of an API or the current version of a web page, which changes between runs. Good practice is to evaluate in a controlled environment with fixed, versioned state wherever possible, and to run each task several times to measure variance, since agent outcomes are often far less consistent from run to run than single-turn responses. Comparing versions safely Most evaluation effort goes into one recurring decision: is version B better than version A? Several practices make that comparison trustworthy. Hold everything fixed except the change under test. If a new prompt and a new model are introduced together, their effects cannot be separated. Record the full configuration of both versions so that the comparison can be rerun. Use the same examples for both and analyse the results in pairs, as Chapter 6 described. Report the difference with an interval, not two separate scores. Look at the discordant examples, the ones where one version succeeds and the other fails, because they show what actually changed. Look at slices, not just totals. An improvement in the aggregate that comes from easy categories while the high-severity category gets worse is a regression for most purposes. Define in advance which slices are allowed to get worse, if any, and by how much. Beware the many-attempts problem. If twenty prompt variants are tried and the best one is chosen by its score on the development set, that score is optimistically biased, for the same reason that best-of-many variants inflate public leaderboard scores. Confirm the winner on held-out data before believing its margin. Treat the grader as part of the configuration. If the model judge's underlying model or prompt changes, rerun the baseline with the new grader before comparing. Many apparent improvements have turned out to be changes in grading. Model migrations One kind of comparison deserves separate treatment because it is now among the most frequent and most consequential: moving a working system from one model to another. Model providers release new versions every few months and retire old ones on published schedules, so most teams building on hosted models will migrate several times a year whether they want to or not. A migration is riskier than it looks because a system's prompts, few-shot examples, output parsers and guardrails have usually been tuned, often implicitly, to the quirks of the old model. A newer model that is better on every public benchmark may still format its answers differently, follow some instructions more literally and others less, refuse in different places, or produce longer responses that break a downstream length limit. The gain in general capability is real, but the system around the model was built for its predecessor. A sound migration process starts by running the full regression suite on the new model with the existing prompts unchanged, to see what breaks. The discordant examples, as always, are the most informative: they show where the new model's behaviour differs, and reading them usually reveals a few systematic causes. Prompts are then adapted to the new model on the development set, and the result is confirmed on held-out data. Because both a model and its prompts have now changed, the comparison with the old system measures the combined effect, which is the effect that matters for the decision, but the record should note both changes. Migrations are also where model judges are most likely to mislead. If the judge is itself built on the same provider's models, and the provider's new judge model and the new system model share stylistic preferences, the judge may favour the new system for reasons unrelated to quality. Running the migration comparison past human graders, at least on a sample and on the high-severity slices, is cheap insurance. Finally, migrations are an occasion to check costs and latency, which can move in either direction with a new model and which affect the decision as much as quality does. A model that is more capable but produces twice as many tokens per answer may cost more in practice even at a lower price per token. Watching production However good the offline programme, real users will find behaviours it missed. Production monitoring closes the loop. Some signals are implicit: whether users rephrase and ask again, abandon a conversation, copy a response, escalate to a human agent or accept a suggested code change. These are noisy and context-dependent, since a user who leaves quickly may have been satisfied or frustrated, but trends in them over time and differences between versions in a controlled experiment are informative. Some signals are explicit: thumbs up or down, ratings, complaints. These are sparse and skewed toward the extremes, since most users give no feedback, but negative feedback is a rich source of examples for error analysis. Some signals come from sampled grading: a random fraction of production conversations scored by the validated model judge, and a smaller fraction reviewed by people. This is the most direct way to see whether offline quality holds up in practice, and it is the main defence against distribution shift, the gradual or sudden divergence of real inputs from the ones the evaluation set represents. A spike in a particular kind of question after a policy announcement, or a new population of users after a product expansion, can move real performance in ways the frozen evaluation set cannot see. Where the stakes allow, online experiments settle questions offline evaluation cannot. Routing a small fraction of traffic to a new version and comparing outcomes such as resolution rate or escalation rate, using standard A/B testing methods, measures the construct of real-world usefulness more directly than any offline proxy. The cost is that real users experience the new version before it is fully known to be safe, so online experiments should follow, not replace, offline evaluation. All of this involves user data, and the programme must be designed around privacy from the start: clear policies about what is logged, who may read it, how long it is kept and how it is anonymised before entering evaluation sets. Red-teaming as discovery The instruments described so far measure behaviour on cases someone thought to include. A complete programme also needs a deliberate search for cases no one thought of, and the most effective method for that search is red-teaming: assigning people, sometimes assisted by models, to try to make the system fail. Red-teaming differs from ordinary evaluation in its aim. An evaluation set estimates how often a system fails across a representative population of inputs. A red team does not care about representativeness; it looks for any input that produces an unacceptable output, however unusual. The two answer different questions. A system can fail on one input in ten thousand in normal use and still be trivially breakable by anyone who tries, and for some applications, such as a public-facing assistant whose failures can be screenshotted and shared, the second fact matters as much as the first. Effective red-teaming is structured. Testers are briefed on the system's purpose, its known weaknesses and the categories of failure that matter most. They work in focused sessions on particular themes, such as extracting confidential information, eliciting confident answers to questions the documents do not cover, or provoking an inappropriate tone in a sensitive conversation. Each successful attack is recorded with the exact input and output, classified by severity, and passed to the team that owns the evaluation set. That hand-off is what turns red-teaming from a one-time exercise into part of the programme. Each discovered failure becomes a regression example, often with several paraphrases, so that the fix can be verified and the failure cannot silently return. Over time the evaluation set accumulates the hard cases that red teams found, and the next round of red-teaming has to work harder. Automated red-teaming, in which one model generates attacks against another and a judge scores the results, can extend coverage cheaply, but as with every automated instrument in this book, its findings need human review, and its blind spots are likely to resemble those of the models doing the searching. People and process An evaluation programme fails most often not because of a technical weakness but because nobody owns it. Evaluation sets grow stale, model judges drift unaudited, and dashboards fill with numbers that no one has checked against reality for months. The remedies are organisational. Someone, or some small group, should own the evaluation programme as a product: maintaining datasets, validating graders, reviewing results and deciding when an instrument needs replacing. In small teams this may be a part-time role, but it must be explicit. The people who build the system should read outputs. Dashboards abstract away the texture of failures, and teams that only look at aggregate scores lose the intuition that error analysis builds. A standing habit of reading a sample of real outputs each week, and discussing them, is cheap and consistently valuable. Domain experts should be involved in defining success, not just in grading. The HR professionals who know which mistakes are dangerous, the clinicians who know which omissions matter, the senior engineers who know what idiomatic code looks like, should help write the rubrics and choose the high-severity categories. Their judgment is the construct; the programme exists to approximate it at scale. And the programme should be honest about uncertainty. Reporting confidence intervals, noting known weaknesses of graders, and recording which numbers have and have not been validated against human judgment makes it harder for a misleading number to drive a decision. A culture in which "we don't know yet" is an acceptable answer about whether a change helped is more valuable than any particular metric. A compact example Pulling the threads together, consider again the HR assistant from Chapter 6, now preparing for a company-wide launch. On every change, deterministic checks confirm that each answer cites at least one retrieved policy document, contains no employee names drawn from a protected list, and stays within a length limit. The regression suite of four hundred examples, stratified by policy area and severity, is scored by a model judge asked a handful of narrow questions per example, such as whether the answer states the correct entitlement and whether it contradicts the cited document. The judge was validated against two HR specialists' grading on three hundred examples, reaching agreement close to the specialists' agreement with each other, except on questions about part-time staff, where it was unreliable; those are therefore always routed to human review. Each week, the HR specialists grade fifty randomly sampled production answers and twenty drawn from conversations with negative feedback, using the same rubric. Their disagreements with the model judge are logged and reviewed monthly. Any answer they flag as harmful becomes a new regression example the same day. Before the launch, the held-out set of two hundred examples, untouched during development, is run once, and the specialists annotate every failure. Separately, a small panel of employees from different departments compares answers from the new system and the existing intranet search on forty realistic questions, judging which they would rather receive. The launch decision is made on these results, with the high-severity slice held to a stricter bar than the overall score. Nothing in this design is exotic. Each element is one of the instruments examined in this book, used for the purpose it suits and checked against the one above it. That is what a good evaluation programme looks like: not a single clever metric, but a system of ordinary instruments, each understood and each kept honest. Conclusion The history traced in this book has a recognisable shape. A new instrument appears that makes some part of evaluation cheap: perplexity for language models, BLEU for translation, ROUGE for summaries, static benchmarks for general capability, model judges for open-ended quality. The instrument is adopted because it is cheap, and for a while it tracks what people care about well enough to steer progress. Then the systems being measured change, or the instrument becomes a target, and the link between the number and the construct weakens. Someone eventually checks the number against careful human judgment, finds the gap, and a new instrument is built to close it. It is tempting to read this as a story of failure, in which each metric is eventually exposed. It is better read as a description of how measurement works when the thing being measured is complex and moving. No instrument for judging language will be both cheap and permanently valid, because the systems being judged improve, because the uses to which they are put multiply, and because any instrument used for selection becomes a target. The appropriate response is not to search for the perfect metric but to build practices that expect instruments to decay and catch it when they do. What follows from the argument Several consequences follow from treating evaluation as the validation of instruments against decisions, and they are worth stating plainly because each runs against a common habit. The first is that the leaderboard is not the decision. Public benchmarks and arenas are useful for narrowing the field to a handful of candidates. They cannot choose among those candidates for a particular use, both because their tasks differ from any specific application and because the pressures described in Chapter 5, from contamination to selective reporting, inflate exactly the differences that buyers are tempted to rely on. A team that skips building its own evaluation because a model tops a leaderboard has outsourced its most important judgment to people who did not know its problem. The second is that a small, carefully designed custom evaluation is worth more than a large, careless one. The value lies in the definition of success for each example, the coverage of the failures that matter and the discipline of keeping some data held out, not in the count. But the count must be large enough for the differences one cares about to exceed the noise, and that is usually larger than intuition suggests. Teams that report scores on a hundred examples without intervals are making decisions on noise more often than they realise. The third is that cheap instruments need expensive ones. Model judges, learned metrics and overlap scores are indispensable for running evaluation at the speed of development. None of them is trustworthy without periodic calibration against human judgment of the specific task. The common failure is not using automated graders; it is using them without ever checking them, or checking them once and assuming the result holds as the system changes. The fourth is that human judgment must be engineered, not merely collected. The research reviewed in Chapter 7 shows that the quality of human evaluation varies as much as the quality of any metric, and that poorly designed human evaluation can confidently reward the wrong things, such as assertiveness over accuracy. Expert graders, operational criteria, calibration examples, measured agreement and attention to disagreement are what turn human judgment into a reference standard. The fifth is that evaluation is a continuing activity, not a gate. Systems change, users change and instruments decay. The programme described in Chapter 8, with its ladder of instruments and its monitoring of production, exists because a single evaluation at launch describes a system that will not exist for long. What remains unsettled Several important questions remain open as of late 2026, and a reader should be wary of anyone who claims they are solved. The most pressing is evaluation of systems that are better than their evaluators at some tasks. When a model can solve research-level mathematics or write code across a large repository faster and sometimes better than the people available to grade it, both human grading and model judging face a competence problem. Approaches under active investigation include decomposing tasks into parts that can each be checked, relying on verifiable outcomes such as passing tests or proofs that can be checked mechanically, and having models argue opposing positions before a weaker judge. None is yet a general solution. A second is the validity of benchmarks for agents. Measuring whether an agent can complete a long, multi-step task in a realistic environment is expensive, noisy and sensitive to infrastructure in ways that make published results hard to compare. The field's experience with SWE-bench Verified, where flawed tests and contamination undermined a widely trusted number, suggests that agent benchmarks will need more frequent auditing and refreshing than their predecessors. A third concerns model judges grading models. As judges and the systems they evaluate are increasingly trained on similar data and by similar methods, the risk grows that they share blind spots, so that a judge systematically fails to notice the errors the evaluated model is most prone to. The extent of this problem is not well measured. A fourth is the relationship between human preference and human benefit. Preference judgments, whether from arena voters or trained raters, capture what people prefer to read in the moment. That is not always what serves them. A response that is pleasant, confident and agreeable may be preferred to one that is accurate and appropriately uncertain. Designing evaluations that measure longer-term usefulness, such as whether advice led to good outcomes, is difficult and largely undeveloped. What to do differently For a practitioner, the argument comes down to a few working habits. Before trusting any evaluation number, ask what procedure produced it, what that procedure is sensitive to and whether it has been checked against the judgment it replaces. If the answers are unknown, treat the number as a rumour. Before building any automated metric, read outputs. Do error analysis on real or realistic cases until the failure modes are clear, and let those failure modes define what the metric must detect. Build a custom evaluation set early, keep part of it held out, version it and grow it from every incident. Report intervals and slices, not single numbers. Compare versions on paired examples. Validate every model judge against careful human grading on the task at hand, re-audit it on a schedule, and never use the judge that trains a system as the only judge that evaluates it. Invest in human evaluation as a craft. Pay for expertise where the construct requires it, write instructions with the same care as code, and measure agreement. None of these habits requires exotic tools or large budgets. Each requires only the willingness to treat a number as a claim that needs evidence. That willingness is, in the end, what separates evaluation from scorekeeping, and it is the most valuable thing a team building with language models can have. Hashtags: #LargeLanguageModels #LLMEvaluation #AIModelEvaluation #LanguageModelBenchmarks #EvaluationMetrics #HumanEvaluation #LLMBenchmarks #ArtificialIntelligence #NaturalLanguageProcessing #MachineLearning #ModelPerformance #BenchmarkAccuracy #Perplexity #BLEUScore #ROUGEScore #BERTScore #SemanticSimilarity #LLMAsAJudge #HumanInTheLoop #ModelGrading #EvaluationFrameworks #BenchmarkContamination #EvaluationValidity #ModelReliability #FactualAccuracy #HallucinationDetection #ModelComparison #RegressionTesting #ResponsibleAI #AIQualityAssurance

Download the Book (PDF): Introduction Somewhere on a university campus tonight, a student is lying awake listening to the roommate's phone buzz on the desk for the fortieth time, rehearsing a speech she will never give. Down the hall, a young man is scraping three-day-old rice out of a pot he did not use, wondering whether he is being petty or whether he is the only adult in the apartment. In a shared suite across town, four people have stopped speaking to one another about anything except the Wi-Fi password, and none of them can quite say when it started. None of these students is a villain. That is the first thing worth knowing about roommate trouble, and it is the thing most people get wrong when they are in the middle of it. When you share a small space with someone you did not choose, every irritation arrives with a ready-made explanation: they are inconsiderate, they were raised badly, they do not respect me. Sometimes those explanations are true. Far more often, the conflict is a collision between two reasonable people carrying two different sets of unspoken rules into the same twelve square meters, and neither of them has ever had to say those rules out loud before. This booklet is about saying them out loud—early, clearly and kindly—and about what to do when that is not enough. Why this matters more than it seems It is easy to treat roommate problems as a minor nuisance of student life, the kind of thing people laugh about years later. Some of them are. But anyone who has worked with university students for long knows how much a bad living situation can quietly take from a person. Your room is where you sleep, where you study, where you recover from a hard day and where you are supposed to be able to stop performing. When that space becomes tense, there is nowhere left to rest. Students in chronic roommate conflict often describe the same pattern: they stay out late to avoid going home, study in the library until it closes, sleep badly, lose their appetite for social life and begin to feel that something is wrong with them. Grades slip not because the work is too hard but because the person doing the work never feels settled. The reverse is also true. A good roommate relationship does not need to be a close friendship, but it gives you something valuable: a home base that asks little of you. You can come back from an exam and not brace yourself at the door. You can leave your things out without worrying. You can ask for quiet and get it. That kind of ease is not luck. It is almost always the product of a few honest conversations and a handful of habits that anyone can learn. There is also a longer view. For many students, a dorm room or first shared apartment is the first time they have lived with someone who is not family. The skills involved—negotiating shared resources, raising a problem without starting a war, holding a limit without apology, repairing things after a bad argument—are the same skills that later make or break relationships with partners, housemates, colleagues and one day, perhaps, children. A roommate is a low-stakes place to practice high-stakes skills. It rarely feels low-stakes at the time, which is exactly why the practice is worth having. The approach in this book The framework here rests on three ideas, and each chapter works out one part of them. The first is that most conflict in shared space is a conflict of expectations, not of character. Two people can both be kind, both be responsible and still be miserable together, because one assumes dishes are washed immediately and the other assumes they are washed once a day; one assumes a closed door means "knock" and the other assumes it means nothing at all. The cure for this kind of conflict is not better people. It is making implicit expectations explicit before they harden into grievances. The second is that agreements are living systems, not contracts. Many universities ask new roommates to fill out a roommate agreement in the first weeks of term. That is a good start, but students often treat the form as a one-time ritual and then never look at it again. Arrangements that actually work—chore rotations, quiet hours, guest policies—are maintained the way a garden is maintained: with regular, brief attention and a willingness to change what is not growing. The third is that you can protect the relationship and protect yourself at the same time. Students tend to believe they must choose between being nice and being honest, between keeping the peace and having their needs met. That is a false choice. Boundaries stated clearly and without contempt tend to make relationships more stable, not less, and the skills of respectful confrontation can be learned by anyone, including people who hate conflict. What you will find here The chapters move roughly in the order a roommate relationship unfolds. Chapter 1 explains why shared living is hard even for good people, drawing on what psychology knows about how we explain other people's behavior and how shared resources get neglected. Chapter 2 is about the first few weeks and the conversation that prevents most later fights: building a roommate agreement that means something. Chapter 3 takes on the single most common source of friction—housework—and shows how to design a chore system that survives busy weeks and different standards of clean. Chapter 4 is about boundaries: what they are, what they are not, and how to set them around sleep, study, guests, belongings and privacy. Chapter 5 turns to the skill most people lack: raising a problem early, specifically and in a way the other person can actually hear. Chapter 6 deals with conflict that has already escalated—what goes wrong in heated arguments, how to cool them down and how to repair afterward. Chapter 7 looks at differences that run deeper than habits: culture, class, religion, identity, neurodivergence, mental health and substance use, including what to do when you are worried about a roommate's wellbeing. Chapter 8 addresses the situation no one hopes for, when it is not working: using mediation, knowing when to involve residence staff, recognizing when a situation is unsafe and moving out without burning everything down. A short conclusion draws out what all of this adds up to. Throughout, you will find scripts—actual sentences you can adapt—along with exercises and step-by-step plans. The students who appear in examples are composites, drawn from situations that recur on campus after campus; they are illustrations, not case histories. The research cited is real and well established, and it is cited sparingly, because in this subject the evidence mostly confirms what careful people have long observed: that small, early honesty prevents large, late pain. One practical note before starting. Housing policies, residence life structures and support services differ widely between universities and between countries. Where this book refers to resident assistants, residence directors, housing offices or counseling centers, translate those roles into whatever exists where you live. Almost every institution has someone whose job includes helping residents live together; part of your task is to find out who that person is before you need them. A word about expectations This book will not make every roommate relationship a good one. Some pairings do not work, and some people behave badly no matter how skillfully you approach them. What the book can do is make sure that when things go wrong, it is not because of a misunderstanding that one clear conversation would have prevented, and that when you do need to act, you act from a position of calm clarity rather than accumulated resentment. Aim, at minimum, for a roommate relationship that is respectful, predictable and repairable. Respectful means neither of you treats the other with contempt. Predictable means you both know what to expect about the basics of shared life. Repairable means that when something goes wrong—and something will—there is a way back. Friendship is a welcome bonus when it happens. It is not the goal, and it is not the measure of success. Chapter 1: The Shared Space Problem Consider a first-year student—call her Maya—who arrives at her residence hall with a carefully packed car, a new duvet and high hopes. Her roommate, Jess, arrived the day before and has already claimed the bed by the window, filled two-thirds of the closet and set up a small speaker that plays music whenever she is in the room. Within ten days Maya has a private list of complaints: the music, the late-night calls, the damp towels on the floor, the way Jess's friends sit on Maya's bed without asking. Jess, for her part, thinks things are going fine. She has noticed that Maya seems quiet and a little stiff, but she assumes Maya is shy. By week four, Maya has told three friends and her mother that her roommate is "a nightmare." She has said nothing to Jess. When the explosion finally comes—over something small, a borrowed phone charger—Jess is genuinely stunned. From her side, a normal month has ended in an ambush. This pattern is so common that residence staff can often recite it before a student finishes describing their situation. It is worth understanding why it happens, because once you see the forces at work, you are much less likely to be carried along by them. Proximity without choice Most close relationships are chosen gradually. You meet someone, like them, spend more time together, and slowly learn their habits. By the time you share a space with a friend or a partner, you have already absorbed a great deal about how they live, and you have decided that you can tolerate or even enjoy it. Roommate relationships invert this order. You are assigned to someone, or matched through a brief questionnaire, and then placed immediately into the most intimate kind of proximity: sleeping in the same room, hearing each other breathe, seeing each other at their worst in the morning and at their most tired at night. You learn everything at once, with no gradual on-ramp and often no shared history or friends to buffer the relationship. This matters because intimacy without choice produces a particular kind of stress. The things that would be charming in a friend you see twice a week—her enthusiasm, his spontaneity—can become grating when there is no escape from them. Small habits compound. A pen clicked once is nothing; a pen clicked every evening for three months can feel like a personal attack. Nothing about the pen has changed. What has changed is the accumulated weight of repetition in a space you cannot leave. It also matters because proximity removes the natural cooling-off periods that other relationships have. When a friend annoys you, you go home. When a roommate annoys you, you are home. The irritation stays in the room with you, and unless you have some way of processing it, it grows. The timing makes it harder still. For first-year students especially, the roommate relationship begins at the same moment as everything else: a new academic workload, the loss of familiar friends, homesickness, money worries, and the strain of trying to make a good impression on dozens of strangers at once. People under that kind of load have less patience and less energy for careful communication. They also tend to project their general unease onto the most visible target. A student who is quietly lonely may find her roommate's constant stream of visitors unbearable, not because the visitors are loud but because they are a nightly reminder of what she does not yet have. A student anxious about his grades may experience any noise during study time as sabotage. Part of the skill of shared living is learning to ask, when irritation spikes, how much of it belongs to the roommate and how much belongs to the week you are having. The first practical lesson here is simply to expect friction. It is not a sign that you made a bad match or that something is wrong with either of you. Friction is the normal result of two lives rubbing together in a small space. The question is not whether it will happen but what you will do with it. Everyone brings a rulebook Every household runs on rules, and most of them are never written down. In one family, shoes come off at the door; in another, no one would dream of it. In one home, food in the fridge belongs to whoever is hungry; in another, taking someone's leftovers is a serious offense. In one family, raised voices mean a normal disagreement; in another, they mean something has gone badly wrong. Some people grow up in homes where doors are always open and privacy is a strange idea. Others grow up in homes where a closed door is sacred. You absorbed your household's rules the way you absorbed your first language—without being taught them and without noticing that they were rules at all. They simply felt like the way things are. This is why roommate conflict so often has a moral flavor. When your roommate breaks one of your unspoken rules, it does not feel like a difference in preference. It feels like a failure of basic decency. "Who leaves dishes in the sink overnight?" is not really a question. It is an accusation disguised as one. Your roommate, of course, has a rulebook too, and theirs feels just as obviously correct to them. From their side, you may be the one who is strange: rigid about the dishes, oddly sensitive about sounds, cold about sharing food, too eager to talk about everything. A useful exercise, best done in the first week or at any point when things start to feel tense, is to write down your own household rules as explicitly as you can. Sit with a blank page and complete these sentences for yourself: • In the home I grew up in, a clean kitchen meant... • When someone wanted privacy, they... • Guests were expected to... • Borrowing something without asking was... • When people were upset with each other, they usually... • Quiet time was... The point is not to decide whether your rules are right. It is to notice that they are rules—specific, learned and not universal. Students who do this exercise often have a small moment of recognition: "I didn't realize I expected everyone to clean up immediately. That was just my dad." That recognition does not mean you must abandon your preferences. It means you can present them as preferences, which is far easier for another person to hear than a verdict on their character. Psychologists have long observed that people tend to explain other people's behavior differently from their own. When someone else does something irritating, we are quick to attribute it to who they are—careless, selfish, lazy. When we do the same thing, we are quick to attribute it to circumstances—we were exhausted, we had an exam, we were going to clean it later. The social psychologist Lee Ross named the first half of this tendency the fundamental attribution error in an influential 1977 paper: the habit of overestimating the role of personality and underestimating the role of situation in explaining what others do. Shared living is almost a laboratory for this error. You see your roommate's behavior constantly, but you see very little of the context behind it. You see the dishes left in the sink; you do not see the panic attack they had before their lab report was due. You hear them come in at 2 a.m.; you do not know that their shift at work ran long. Meanwhile, you have full access to your own reasons for everything you do, so your own lapses feel understandable and forgivable. Consider a second-year student—call him Daniel—who shares an apartment with three others. He has become convinced that one of them, Theo, is "just lazy." Theo rarely takes out the trash and leaves his things in the common room. When Daniel finally raises it, he learns that Theo has been working twenty hours a week to cover his tuition and has been leaving his things in the common room because his own room has no desk. Theo still needs to do his share, and the conversation still needs to happen. But the conversation Daniel has after learning this is very different from the one he had been rehearsing in his head, which began, "You're so lazy." A practical habit follows from this. When you notice yourself building a story about your roommate's character, pause and generate two alternative explanations that involve circumstances instead. You do not have to believe them. The goal is to loosen the grip of the first story enough that you can approach the person with curiosity instead of a verdict. "Maybe they don't notice mess the way I do." "Maybe they're overwhelmed right now." "Maybe they think I'm fine with it because I never said anything." This last explanation is, in practice, very often the true one. The shared sink and the silent spiral In 1968 the ecologist Garrett Hardin published an essay in Science called "The Tragedy of the Commons." His argument, in simplified form, was that when a resource is shared by many and owned by none, each individual has an incentive to use a little more than their share and contribute a little less to its upkeep, because the benefit goes to them while the cost is spread across everyone. Left unmanaged, the shared resource degrades. Later scholars, notably the political economist Elinor Ostrom, showed that communities often avoid this outcome by developing clear, locally agreed rules for using and maintaining shared resources—a finding that won her the Nobel Prize in economics in 2009. The shared kitchen, bathroom and living room of a student residence are miniature commons, and they behave exactly as Hardin would predict when no one sets rules. Nobody decides to let the bathroom become disgusting. Each person simply notices that it is a bit dirty, reasons that it is not their mess, or not only their mess, and waits for someone else to deal with it. The person with the lowest tolerance for mess eventually cleans it, resents it, and begins to keep score. The others may not even notice that anything happened. Ostrom's insight is the hopeful half of the story. Shared spaces do not have to degrade. They stay healthy when the people who use them agree on specific rules, can see whether the rules are being followed, and have some low-key way of addressing it when they are not. That is, in essence, what the next two chapters of this book help you build. It is worth noticing what this does to the moral picture. In an unmanaged commons, degradation happens even when everyone is reasonably decent, because the structure rewards small acts of neglect and makes each individual lapse nearly invisible. The fix is not to find better roommates. It is to replace an unmanaged commons with a managed one. Return to Maya and Jess. What turned an ordinary mismatch into an explosion was not the music or the towels. It was the silence. Many students, particularly those who dislike conflict, adopt a strategy of tolerating irritations in the hope that they will go away or that the other person will notice and change. This is understandable. Raising a small issue feels awkward and even petty, and it seems kinder to let it go. The difficulty is that most irritations do not go away, and the other person usually does not notice. So the tolerant roommate keeps tolerating, and something quietly changes inside them. Each new incident is no longer experienced on its own; it is added to a growing file. By the time they finally speak, they are not responding to the phone charger. They are responding to a month of accumulated grievance, and the intensity of their reaction makes no sense to the other person. This is the silence spiral, and it has a cruel logic. The longer you wait, the harder it is to raise an issue, because now you have to explain why you did not say anything before. The harder it is to raise, the longer you wait. Meanwhile, your roommate, receiving no feedback, reasonably assumes that everything is fine and carries on exactly as before. Silence also changes how you see the other person. Grievances that are never spoken are never corrected by new information, so the story you have built about your roommate goes unchallenged. By week four, Maya was not living with Jess. She was living with a character she had constructed out of Jess's worst moments, a character who was selfish and oblivious by design. The real Jess—who would, it turned out, have happily turned the music down if asked—never had a chance. There is a simple principle that interrupts the spiral: raise small things while they are still small. A problem mentioned the first or second time it happens can be mentioned lightly, as information. "Hey, I'm a really light sleeper—any chance the calls could happen in the lounge after eleven?" That sentence is almost impossible to say angrily, and it is almost impossible to hear as an attack. The same request made after thirty nights of silent suffering will be loaded with feeling, and it will land that way. What a healthy roommate relationship actually looks like It helps to have a realistic picture of the goal. Students sometimes arrive at university hoping their roommate will become their best friend, and they feel disappointed or even rejected when the relationship turns out to be cordial but not close. Others assume that as long as there are no open fights, everything is fine, even if they dread coming home. Neither picture is quite right. A healthy roommate relationship has a few recognizable features, and friendship is not required for any of them. Both people can raise a problem without fear. This is perhaps the single best indicator. If you can say "the bathroom's getting bad, can we sort it out this week?" and receive a normal response, most other problems are fixable. The basics are predictable. You both know roughly when the room will be quiet, how guests work, who cleans what and what is shared versus private. Predictability does a great deal of the work that goodwill alone cannot. Differences are treated as differences. You may have very different habits, values or schedules, but neither person treats the other's way of living as a moral failing. Mistakes get repaired. When one of you forgets, snaps or crosses a line, there is some way to acknowledge it and move on, rather than adding it to a permanent record. Each person has a life outside the room. Healthy roommates are not each other's entire social world. When one relies on the other for all their company or support, the relationship tends to become strained. If you have these features, you have a good roommate relationship, even if you rarely eat together or never meet each other's families. If you lack them, the rest of this book offers ways to build them. Three questions are worth sitting with before the next chapter, whether you are about to move in with someone or have been sharing a space for months. First, what are the two or three unspoken rules you hold most strongly about shared living? Write them down, and practice saying them as preferences: "I really like it when..." rather than "People should..." Second, is there something small you have been tolerating that you have not mentioned? If so, it is almost certainly easier to raise now than it will be next week. Third, what story are you currently telling yourself about your roommate's character? Can you think of two other explanations for the same behavior that do not require them to be a bad person? These questions are not about being nicer. They are about being more accurate, and accuracy is where every good roommate relationship begins. Chapter 2: The Expectations Conversation If there is a single intervention that prevents more roommate conflict than any other, it is a proper conversation about expectations in the first week or two of living together. Many universities encourage or require this, usually in the form of a roommate agreement that residents complete with the help of a resident assistant. Students often treat the exercise as a bureaucratic chore, rush through the form in fifteen minutes, write "we'll figure it out" next to half the questions, and file it away. That is a missed opportunity. Done properly, this conversation is the moment when the unspoken rulebooks described in the previous chapter get opened on the table and compared, before anyone has had time to be offended by them. Its value depends on three things: timing, content and upkeep. Timing comes first. The first week of living together has a special property: nobody has done anything wrong yet. That makes it the only time when you can discuss almost any topic in the abstract. "How do we feel about overnight guests?" is a relaxed question in week one. In week six, after your roommate's partner has slept over five nights running, the same question becomes an accusation, and it will be heard as one no matter how carefully you phrase it. Early conversations also benefit from goodwill. Most people arrive at a new living situation wanting it to go well and wanting to be seen as a reasonable person. That motivation is at its peak in the first days, before either of you has built up any grievances. People are unusually willing to make accommodations when they are still forming an impression of each other and when the accommodation costs them nothing yet. Finally, an early conversation sets a precedent that matters more than any individual rule: it establishes that the two of you are people who talk about things. Once you have had one open conversation about shared life, the second one is easier. If your first real conversation about living together happens during a conflict, you will have established the opposite precedent: that talking about the room means someone is in trouble. If you are reading this after the first few weeks have passed, the conversation is still worth having. You can frame it honestly: "We never really sat down and talked about how we want the room to work. Can we do that? I'd rather sort things out now than let little stuff build up." Very few people will refuse that invitation, and it has the advantage of being about the future rather than about anything that has already happened. What to cover A good expectations conversation covers the domains where unspoken rules most often collide. Table 1 sets out the main ones, with sample questions and the kind of hidden assumption that tends to cause trouble in each. You do not need to follow it rigidly, but it is worth making sure you have touched each area at least briefly. Table 1. Key domains for a roommate agreement. Domain Questions to ask Common hidden assumption Sleep When do we each usually go to bed and get up? What helps us sleep? "Everyone sleeps roughly when I do." Study and quiet When do we need quiet? Is music or calling okay while the other works? "Headphones mean I can do anything." Guests How much notice? Overnight guests? How often? "Friends are always welcome." Belongings and food What is shared? What must we ask before using? "Small things don't count as borrowing." Cleaning What does clean mean? Who does what, and how often? "Clean means how my family kept house." Temperature, light, air Window open or shut? Lights on late? Heating? "My comfort is the neutral setting." Communication How do we raise problems? In person or by text? "If something bothers them, they'll say so." Privacy Knocking, changing clothes, personal calls, alone time? "Shared room means shared everything." Several of these deserve a little more comment. Sleep is the domain where mismatches cause the most physical harm. Adults generally need at least seven hours of sleep a night to function well, according to the American Academy of Sleep Medicine and the Sleep Research Society, and university students are notorious for falling short. A roommate relationship where one person's schedule regularly cuts into the other's sleep will degrade quickly, because sleep deprivation shortens everyone's temper. If your schedules differ, do not just note the difference; agree on concrete accommodations, such as a lamp rather than the overhead light after a certain hour, phone calls taken outside, or an alarm that is not snoozed five times. Guests produce more serious disputes than almost any other topic, because they touch on privacy, safety, sexuality and fairness all at once. It is worth being specific here: How much notice do we want before someone comes over? Are there hours when guests are not okay? Are overnight guests okay, and if so, how many nights a week or month? What if one of us wants to use the room for private time with a partner? Many residence halls also have their own guest policies, often limiting the number of consecutive nights a guest may stay; check them, because they give you a neutral reference point that is not "my preference versus yours." Belongings and food are simple to agree and surprisingly easy to fight about. The clearest agreements distinguish three categories: things that are fully shared (perhaps cleaning supplies, bought jointly), things that can be borrowed if you ask first, and things that are off-limits. Food deserves its own line, because some people grew up in households where food was communal and genuinely do not register taking a yogurt as borrowing. Communication is the domain most often left blank, and it is arguably the most important. Discuss how each of you prefers to hear about a problem. Some people would much rather receive a text, which gives them time to think before responding; others find a text about a problem cold and upsetting and would rather be told face to face. Some want to hear about things immediately; others would rather have a weekly check-in. Agreeing on this in advance means that when a problem does arise, the delivery method itself will not become a second grievance. How to run the conversation The way you have this conversation matters as much as what you cover. A few principles make it far more productive. Choose a relaxed moment. Not right before a class, not late at night when one of you is exhausted, and not in front of other people. An unhurried evening in the first week, perhaps over food, works well. If a resident assistant is facilitating, let them set the tone, but do not hand the whole conversation to them; the agreement belongs to you. Go first with your own quirks. The fastest way to make the other person comfortable is to disclose something about yourself that is slightly embarrassing or inconvenient. "Fair warning, I'm a really light sleeper, so noise at night is probably my biggest thing." "I'm kind of messy with clothes, but I'm strict about dishes." This does two things. It shows that you are not setting yourself up as the reasonable one judging the other's flaws, and it gives them permission to be honest about their own quirks in return. Ask before you state. For each domain, try asking the other person what they are used to before sharing your own view. "What was it like at home with guests?" invites a story rather than a position, and stories reveal the unspoken rules far better than preferences do. You may discover that your roommate's apparently casual attitude to overnight guests comes from growing up in a large family where someone was always staying over. Look for the need beneath the preference. When you disagree, try to find out what each of you actually needs, rather than arguing over the specific rule. This idea is at the heart of Getting to Yes, the classic negotiation book by Roger Fisher and William Ury of the Harvard Negotiation Project, which distinguishes between people's positions—what they say they want—and their interests—the reasons they want it. A position might be "No guests after ten." The interest behind it might be "I need to be asleep by eleven on weeknights and I can't sleep with people in the room." Once the interest is on the table, other solutions appear: guests are fine until later on Fridays and Saturdays; on weeknights, guests move to the common room after ten. Arguing over positions tends to produce a winner and a loser. Exploring interests tends to produce arrangements both people can live with. Be concrete. "We'll keep it clean" is not an agreement. "Dishes washed by the end of the day they're used; floor vacuumed every Sunday, alternating weeks" is an agreement. Vague agreements feel friendly in the moment but leave all the actual decisions to be fought over later. Whenever you find yourselves nodding at a general principle, ask, "What would that look like, specifically?" Write it down. It can feel oddly formal to write down an agreement with someone you are trying to get along with, but written agreements protect both people. Memory is selective, especially in conflict, and it is remarkable how often two people remember a spoken agreement in two different ways that each happen to favor themselves. A shared note on your phones, or a page taped inside a closet door, removes that problem. Consider two first-year students—call them Amara and Lucy—who sit down in their first week. Lucy mentions early that she is an early riser who likes to be up at six to go running. Amara, a night owl, admits she usually works best after ten at night. In the abstract, this looks like a disaster. In practice, once they talk it through, they arrive at a workable arrangement: Amara uses a small desk lamp and headphones after Lucy goes to bed at half past ten; Lucy dresses with the lights off and leaves her running clothes out the night before; neither sets an alarm that goes off more than once. They write it down, and they agree to revisit it after midterms. Neither is thrilled by every detail, but neither dreads bedtime. Compare this to the likely outcome if they had never discussed it: weeks of mutual, silent resentment, each convinced the other is inconsiderate. Keeping the agreement alive A roommate agreement is not a treaty to be signed and forgotten. Circumstances change: one of you starts dating someone, takes a part-time job, falls behind on work, or discovers that a rule you both agreed to simply does not work in practice. An agreement that cannot change becomes either a weapon ("but you signed it") or a dead letter that everyone ignores. The simplest way to keep an agreement alive is to schedule a brief check-in. Put it on the calendar for a few weeks after you move in, and then perhaps once or twice a term after that. The check-in does not have to be long or serious. Ten minutes is often enough. Three questions cover most of it: What is working well that we should keep doing? Is there anything that isn't working, or that has changed? Is there anything coming up—exams, visitors, trips—that we should plan for? The advantage of a scheduled check-in is that it gives both people a legitimate, low-pressure occasion to raise minor issues. A student who would never knock on the door of a conflict to say "the towels on the floor bother me" can mention it easily when the question "Is there anything that isn't working?" is already on the table. The check-in makes raising small things normal. It also helps to agree in advance on how changes will be made. The cleanest rule is that either person can ask to revisit any part of the agreement at any time, and neither person changes it unilaterally. That way the agreement stays fair and flexible without becoming a source of new suspicion. A few warning signs suggest that an agreement is no longer serving you. One of you is repeatedly citing it as evidence against the other. One of you has quietly stopped following a part of it without saying anything. Or you both know a rule is being broken and have decided not to mention it. Each of these means a check-in is overdue. The goal is never perfect compliance with a document; it is a shared, current understanding of how you want to live together. When the conversation is hard Not every roommate will engage with this process. Some people find structured conversations awkward and deflect with jokes. Some say "I'm easy, whatever works for you" and then turn out not to be easy at all. Some are simply not very interested in the room; they plan to spend most of their time elsewhere and do not see why they should negotiate anything. With a deflecting roommate, keep the conversation short and specific. Instead of working through every domain, pick the three things that matter most to you and ask about each directly. "Just so I know—what time do you usually go to bed? I'm asking because I'm a light sleeper." Short, practical questions are harder to deflect than a large, formal sit-down. With a roommate who says "whatever works for you," be gently skeptical. That phrase is often sincere at the time, but people rarely know their own preferences until those preferences are violated. It helps to offer concrete options and ask for a choice: "Would you rather we alternate cleaning weeks or split it by area?" or "If I have a friend over, would you want a text first, or is it fine if they just come?" People who are easygoing in principle often have surprisingly specific views when presented with actual choices. With a roommate who is rarely around, agree on the few things that matter even when they are absent: shared costs, belongings, cleaning of shared areas and what happens when they do come back late. The fact that someone is rarely there does not mean the space can go unmanaged; if anything, it makes explicit rules more important, because there will be fewer natural opportunities to talk. A special case deserves mention: moving in with a friend. Students who room with someone they already know often skip the expectations conversation entirely, on the grounds that they know each other well. They usually do not know each other as housemates. Friendship is built in chosen, limited doses—coffee, parties, study sessions—and says very little about how someone behaves at seven in the morning or after three weeks of exams. Friends also tend to find it harder, not easier, to raise complaints, because more feels at stake. If you are living with a friend, have the conversation anyway, and say explicitly that you want to protect the friendship by keeping the practical matters out of it: "I really don't want dishes to be the thing that wrecks us, so can we just agree on it now?" Finally, if a roommate refuses outright to discuss expectations, that is itself useful information. It does not necessarily mean they are difficult; some people simply dislike this kind of conversation. But it tells you early that you may need more support from residence staff if problems arise, and that you should be especially careful to raise things while they are small. Later chapters address what to do when a roommate will not engage at all. The expectations conversation cannot anticipate everything. What it can do is replace a large number of invisible, clashing assumptions with a smaller number of visible, shared ones, and establish from the start that the two of you are people who talk things through. That foundation makes everything in the rest of this book easier. Chapter 3: Chores Without Resentment Ask residence staff what roommates argue about most, and housework will be at or near the top of the list. This is not because students are unusually lazy. It is because housework combines every ingredient of the shared-space problem described in Chapter 1: different household rulebooks, a commons that nobody owns, invisible effort, and a silence spiral that runs on small irritations. Dirty dishes are rarely just dirty dishes. They are a daily, visible question about fairness and respect. The good news is that chore conflict is also one of the most solvable kinds of roommate trouble. Unlike clashes of personality or values, it responds very well to structure. A well-designed chore system takes most of the emotion out of the question, replacing "who is being lazy?" with "whose turn is it?" Building one takes less time than a single serious argument about the sink. Why "we'll just keep it clean" fails Most roommates start with the same informal arrangement: everyone cleans up after themselves, and anything shared gets done by whoever notices it. This arrangement feels friendly and adult. It rarely survives the first month. The first problem is that people notice mess at very different thresholds. One person sees crumbs on the counter and feels a small, genuine discomfort until they are wiped away. Another does not register the crumbs at all until they have become a visible layer. Neither person is choosing their threshold; it is shaped by habit, upbringing and temperament. Under a "whoever notices" system, the person with the lower threshold will always notice first, and therefore always clean first. Over time, they end up doing most of the shared work, while the other person genuinely does not realize anything is unequal. Resentment builds on one side while the other side experiences a perfectly pleasant home. The second problem is that much of the work of running a shared space is invisible. Someone has to notice that the toilet paper is running low, remember to buy more, keep track of when the bins go out, and realize that the fridge has started to smell. The sociologist Allison Daminger, writing in the American Sociological Review in 2019, described this as the cognitive dimension of household labor—anticipating needs, identifying options, deciding and monitoring—and found that it tends to be unevenly shared and easily overlooked, even by the people doing it. In popular writing it is often called the mental load. In roommate life, the person carrying the mental load may do fewer visible chores than it seems and still be carrying far more than their share, because they are the one keeping the whole system in their head. The third problem is that "cleaning up after yourself" cannot cover the shared spaces. Nobody specifically made the bathroom floor dirty or the shower grimy; it happened collectively, over days. Under an informal system, collective mess belongs to no one, and so no one cleans it until the person with the lowest threshold gives in. The lesson is that goodwill is not enough. Good roommates with no system will still slide into inequality, and inequality, left unspoken, becomes resentment. What they need is not more virtue but a better structure. Designing a chore system that works A workable chore system has four parts: a list of tasks, a definition of "done" for each, a way of distributing them, and a rhythm for checking in. It sounds bureaucratic when written out, but most roommates can put one together in half an hour. Step one: list everything. Sit down together and list every recurring task that the shared space requires. Be thorough. The list typically includes washing dishes, wiping counters and the stovetop, cleaning the sink, cleaning the toilet, cleaning the shower or bath, cleaning the bathroom sink and mirror, sweeping or vacuuming floors, mopping, taking out rubbish, recycling and compost, cleaning out the fridge, restocking shared supplies such as toilet paper and dish soap, and dealing with any shared laundry such as towels or bathmats. In a single dorm room, the list is shorter but still exists: floor, bins, surfaces, any shared fridge or microwave. Include the invisible tasks—buying supplies, tracking bin day, noticing when things run out—as tasks in their own right. Naming them makes them visible and ensures that whoever does them gets credit. Step two: define "done." This is the step most roommates skip, and it is where most chore disputes actually come from. "Clean the bathroom" means very different things to different people. For one, it means a quick wipe of the sink. For another, it means scrubbing the toilet, the shower, the floor and the mirror. If you do not agree on the standard, the person with the higher one will always feel that the other person's turn did not count. Eve Rodsky's book Fair Play, written for couples dividing household work, calls this agreeing on a "minimum standard of care," and the idea translates well to roommates. For each task, agree briefly on what a completed job looks like and how often it needs doing. "Dishes: washed, dried or on the rack, and the sink rinsed, by the end of the day they were used." "Bathroom: toilet, sink and shower wiped with cleaner, floor swept, bin emptied, once a week." You do not need a manual. A single line per task is enough to prevent most disagreements. Rodsky also argues that each task has three phases—conceiving that it needs doing, planning how, and executing it—and that fairness comes from giving one person ownership of all three, rather than having one person notice and nag while another carries out instructions. For roommates, this suggests a useful rule: when a task is yours, it is fully yours. You notice when it needs doing, you get the supplies, and you do it. Nobody should have to remind you. Step three: distribute the work. There are several ways to divide chores, and the best one depends on your schedules, your preferences and how many of you there are. Table 2 compares the most common models. Table 2. Common chore-sharing models compared. Model How it works Best for Weak point Fixed ownership Each person permanently owns certain tasks People with strong task preferences Can feel unequal if tasks differ in effort Weekly rotation Tasks rotate on a set schedule Groups wanting visible fairness Breaks down if one person skips a turn Zones Each person owns an area, not a task Suites and apartments Some zones are dirtier than others Clean-as-you-go plus weekly blitz Daily self-cleaning, one shared session a week Close roommates with overlapping time Needs everyone present at once Shared cleaner, split cost Hire help and divide the bill Groups with money but no time Cost; still requires daily tidying Fixed ownership works well when people have genuine preferences. Some people do not mind cleaning toilets but hate washing dishes; others feel the reverse. If you can match tasks to preferences, you may find that each person does the jobs they mind least, and everyone is happier. The risk is that tasks differ in effort and frequency, so you need to be honest about balancing the load. A common solution is to give the heavier or more unpleasant jobs a higher weighting and make sure each person's total is roughly equal. Rotation is the most visibly fair system and works well for groups of three or more, especially when nobody has strong preferences. Its weakness is that a missed turn causes a pile-up, and the next person inherits a worse job. It needs a clear rule for what happens if someone misses their week. Zones work well in apartments where each person can take responsibility for a room—one person has the kitchen, another the bathroom, another the living room and hallways. Rotating zones every few weeks keeps the burden fair, since the kitchen and bathroom typically need far more attention than a hallway. The "clean as you go plus weekly blitz" model suits roommates who get on well and enjoy some shared time. Everyone keeps their own mess under control day to day, and once a week, perhaps on a Sunday afternoon with music on, everyone cleans together for an hour. It can build goodwill, but it depends on schedules lining up. Hiring a cleaner is not an option for most students, and it does not eliminate the need for daily tidying. But in some shared houses it resolves long-running conflicts, and if it is feasible, it is worth considering without embarrassment. Whatever model you choose, make it visible. A chart on the fridge or a shared digital list works far better than a spoken agreement, because it removes the need for anyone to remember or remind. When the chart says it is Priya's week for the bathroom, nobody has to raise it as a personal request. The system does the reminding. Shared supplies deserve their own arrangement, because they generate a steady trickle of small unfairness. One person always seems to be the one buying dish soap, toilet paper and bin bags, and is too embarrassed to ask for a few coins back each time. A simple fix is a shared kitty: everyone contributes the same modest amount at the start of each month, and whoever does the supply run pays from it. Alternatively, add "restocking" to the rotation as a task in its own right, with costs recorded and settled monthly. Either way, the goal is that nobody quietly subsidizes the household. Step four: build in a rhythm. Chore systems decay. Schedules change, people discover they hate a job, exams make everyone slack off. A brief check-in every few weeks, perhaps as part of the roommate check-in described in Chapter 2, keeps the system working. Ask whether the load feels fair, whether any task needs a different standard, and whether anything is coming up that will disrupt the routine. Agree in advance what happens during exam periods: some roommates suspend the rotation and do only the essentials; others trade tasks so that the person with the heaviest exam load does less that week and makes it up later. Different standards of clean Even with a good system, roommates with very different standards of cleanliness face a real problem. The person with high standards feels they are living in squalor; the person with lower standards feels they are being policed. Both experiences are genuine. The first thing to recognize is that neither standard is objectively correct above a basic level of hygiene. There is a floor below which things become a genuine health problem—food left to rot, mold, pests, unwashed dishes piling up for days—and it is fair for either roommate to insist that the shared space stays above that floor. Above that floor, though, standards are preferences. A roommate who wants the counters wiped after every use and the floor vacuumed daily has a legitimate preference, not a moral high ground. The fairest arrangement usually settles somewhere between the two standards for shared spaces, while leaving each person free to keep their own private area as they like. A higher-standard roommate might accept that the floor will be vacuumed twice a week rather than daily; a lower-standard roommate might accept that dishes must be washed the same day rather than whenever the pile becomes inconvenient. It is also reasonable for a person with a much higher standard to do some extra work themselves, if they want the space cleaner than the agreed minimum. This can be a relief rather than a burden, as long as it is openly acknowledged. "I know I like the kitchen cleaner than you do, so I'll do an extra wipe-down when I want it—I just need you to keep to what we agreed" is a mature arrangement. What causes resentment is not doing extra work; it is doing extra work silently while believing you are being taken advantage of. Consider a pair of roommates—call them Sam and Kofi—sharing a small apartment. Sam grew up in a house where the kitchen was cleaned every night after dinner and the counters were always clear. Kofi grew up in a busy household where the kitchen was cleaned once a day, usually in the morning, and nobody minded a few dishes overnight. After a month of silent tension, they sit down and discuss it. They agree that dishes will be washed before bed, not immediately after eating; that counters will be cleared at night; and that Sam, if he wants the stovetop spotless more often than that, will do it himself without comment. Neither has the kitchen exactly as he would like it. Both can live with it, and neither feels judged. That is the realistic goal. When someone isn't doing their share Sometimes a roommate simply stops doing their part. The chart says it is their week, and the bathroom has not been cleaned. Their dishes sit in the sink for three days. This is the moment when many students either explode or quietly start doing the work themselves while keeping a mental tally. Both responses make things worse. A better approach begins with a light, specific reminder, as early as possible. "Hey, it's your bathroom week—any chance you can get to it by tomorrow?" Most of the time, that is all it takes. People forget, get busy, lose track of the chart. A reminder delivered without edge is usually received without defensiveness. If a reminder does not work, or the pattern keeps recurring, it is time for a direct conversation, using the skills described in Chapter 5. Focus on the specific behavior and its effect, not on the person's character. "The last three weeks, the bathroom hasn't been done on your weeks, and I've ended up doing it because I couldn't stand it any longer. I'm starting to feel resentful, and I don't want that. What's going on?" Then listen. There may be a reason—an overwhelming workload, a mental health struggle, a genuine dislike of that task—that suggests a better arrangement. Avoid two common traps. The first is passive-aggressive notes. A note on the fridge reading "SOME PEOPLE need to learn to wash their dishes!!" almost never produces clean dishes; it produces embarrassment and anger, and it makes a direct conversation much harder. If you must leave a note, make it neutral and specific—"Reminder: bins go out tonight! —A"—and never use it as a substitute for a real conversation about a pattern. The second trap is the silent tally: doing the work yourself while privately keeping a list of every time you covered for someone. The tally feels like protection, but it is really a store of ammunition, and when it is eventually fired, it will be out of all proportion to whatever set it off. If you are covering for someone, say so, early, and ask for it to be made up. If a roommate consistently refuses to do their share despite direct conversation, the problem has moved beyond chores, and the steps described in Chapter 8—including involving residence staff—become appropriate. In many residence halls, maintaining a minimum standard of cleanliness in shared spaces is part of the housing agreement, which gives staff a clear basis for intervening. A final point about chores: they are one of the few areas of roommate life where you can generate goodwill cheaply. Doing your share reliably and without being reminded, occasionally doing a small extra task without comment, and thanking your roommates when they do theirs costs very little and builds a store of trust that pays off when harder conversations come. The roommate who always does their dishes has far more credibility when they need to raise something difficult. Chapter 4: Boundaries in a Small Space The word boundaries has become so common that it risks meaning almost nothing. People use it to describe everything from a reasonable request to an ultimatum, from self-care to cutting someone off. In shared living, it helps to be more precise, because a clear understanding of what a boundary is—and what it is not—makes the difference between limits that stabilize a relationship and "boundaries" that simply start new fights. Consider a second-year student—call him Arjun—whose roommate, Ben, has fallen into the habit of using Arjun's desk whenever Arjun is out, because it is closer to the window and the outlet. Arjun comes back to find his notes shuffled, crumbs on his keyboard and his chair at the wrong height. He tells a friend he needs to "set a boundary." When he tries, he says to Ben: "You need to respect my boundaries and stop being so entitled." Ben, who had no idea the desk mattered, feels accused of something vague and serious. He apologizes stiffly, but the room feels cold for a week, and Arjun is left wondering whether boundaries even work. What went wrong was not that Arjun set a boundary. It was that he did not actually state one. He delivered a verdict on Ben's character and left Ben to guess what was being asked. What a boundary is and is not A boundary, in the sense that matters for shared living, is a clear statement of what you need, what you are willing to do, and what you will do if a limit is crossed. It describes your side of the relationship. The therapist Nedra Glover Tawwab, whose book Set Boundaries, Find Peace is one of the more practical treatments of the subject, frames boundaries as the expectations and needs that help you feel safe and comfortable in a relationship, communicated to the people involved. The emphasis is on communication. A boundary nobody has heard is not yet a boundary; it is a private rule. It helps to distinguish boundaries from three things they are often confused with. A boundary is not a demand that another person change who they are. "Stop being so loud" is not a boundary; it is a request about someone else's general behavior, and it is too vague to act on. "I need the room quiet after eleven on weeknights so I can sleep" is a boundary. It names a specific need, it is limited in time and scope, and it gives the other person something concrete to do. A boundary is not a punishment. When people are angry, they sometimes announce "boundaries" that are really retaliation: "Fine, from now on you're not allowed to touch anything of mine." Boundaries set in anger tend to be broader than the need behind them and are experienced by the other person as hostility. If you notice that the limit you want to set is bigger than the problem that caused it, it may be worth waiting until you are calmer. A boundary is not a wall. Some people, especially after a bad experience, try to protect themselves by withdrawing completely—never speaking to their roommate, never sharing anything, treating the room as enemy territory. That is sometimes necessary for safety, but as a general strategy it tends to produce exactly the tense, joyless atmosphere it was meant to escape. Good boundaries make connection possible by making it safe. They let you say yes to some things precisely because you can say no to others. It also helps to understand that you cannot, strictly speaking, control whether another person respects your boundary. You can state it clearly, ask for it to be honored and decide what you will do if it is not. The last part is often overlooked. "Please don't borrow my clothes without asking" is a request. "Please don't borrow my clothes without asking. If it keeps happening, I'm going to start keeping them locked in my wardrobe" is a boundary with a consequence you can actually carry out. The consequence is not a threat or a punishment; it is simply what you will do to protect the need, and it is entirely within your control. The main boundary areas in shared living Shared living generates boundary questions in a handful of predictable areas. Some were touched on in Chapter 2 as part of the expectations conversation; here the focus is on what to do when a limit needs setting or resetting. Physical space and belongings. In a shared room, the few things that are truly yours—your bed, your desk, your wardrobe, your shelf—take on outsized importance. They are the only private territory you have. Many people are relaxed about others sitting on their bed or using their desk; others find it deeply uncomfortable. Both are legitimate. The same goes for belongings: some people happily share clothes, chargers and food, while others want to be asked every time. A useful script is simple and specific: "I've realized I'm pretty particular about my desk. Could you use your own, or ask me first if you need mine?" If the boundary is about borrowing, make the rule clear: "Chargers and snacks, go ahead. Clothes and my laptop, please ask first." Sleep, study and quiet. These are the boundaries that most directly affect wellbeing and academic performance. It is entirely reasonable to insist on protected sleep hours and a way of studying without constant interruption. The script should name the need and the specific ask: "I've got an eight o'clock class every day this term, so I really need the lights off and the room quiet by eleven on weeknights. Could calls and videos move to the lounge after that?" If your roommate's schedule genuinely conflicts with yours, look for accommodations rather than trying to win: eye masks, earplugs, a desk lamp, agreed times when one person uses the room for study and the other goes elsewhere. Guests and partners. This is the most sensitive area, because it involves a third person, and often sex, privacy and safety. Several issues come up repeatedly. How often can guests visit, and do they need to leave by a certain time? Are overnight guests acceptable, and how often? What happens when one person wants privacy with a partner? The last question has produced its own slang on some campuses—being "sexiled," locked out of your own room so that your roommate can have sex. A single, pre-arranged request is one thing. Being regularly pushed out of the room where you live and sleep is not a reasonable arrangement, and you have every right to say so. Useful scripts here are direct and non-judgmental about the relationship itself: "I'm glad you're seeing someone, and I want to work this out. But I've been out of the room three nights this week, and I need to be able to sleep in my own bed. Can we agree on a limit—say, one night a week with at least a day's notice?" For guests generally: "I'm fine with friends coming over in the afternoon, but on weeknights after ten I really need the room to wind down. Could they head out by then?" Many residence halls have formal guest policies, which typically cap how many consecutive nights a guest can stay and require roommate consent for overnight visitors. If your roommate's guests are becoming a problem, check the policy. It often gives you a neutral basis for the conversation, and if things continue, it gives residence staff a clear reason to step in. Guests also raise boundary questions of their own. Your roommate's friends are not party to your agreement, and they may sit on your bed, eat from your shelf or linger long after you want to sleep simply because nobody has told them otherwise. The cleanest approach is to hold your roommate responsible for passing on the house rules to their guests, rather than confronting the guests yourself: "Could you let your friends know my bed and desk are off-limits? I don't want to be the one policing it." Privacy. Sharing a room means living with very little privacy, which makes the privacy you do have more valuable. Common boundary areas include knocking before entering, especially if the room has been shut; being able to change clothes without being watched; being able to take personal phone calls without being listened to; and having some time alone in the room. A fair arrangement often involves agreed "alone time," perhaps a few hours a week when each person can have the room to themselves. Scripts can be light: "I know it sounds weird, but I really need a bit of time alone in the room each week to recharge. Could we work out a couple of slots where one of us is out?" Digital life. Shared living now includes a digital layer that earlier generations never had to consider. Photos and videos taken in the room can end up on social media; a roommate's phone calls, group chats or streams can broadcast your space to strangers. It is reasonable to set a clear boundary: "Please don't post photos or videos of me, or of my side of the room, without asking first." Recording someone without their knowledge can also raise legal and conduct issues, depending on where you live, and it is worth raising promptly if it happens. Emotional labor. This is the boundary students find hardest to name. Sometimes a roommate begins to rely on you for constant emotional support: talking through every problem late into the night, needing reassurance repeatedly, treating you as their main confidant. At first, this can feel like closeness. Over time, it can become exhausting, especially if you are managing your own stress. Supporting a friend is part of life, but you are not a counselor, and it is not unkind to limit how much support you can give. "I care about you, and I want to help. But I'm getting really stretched, and I think you deserve more support than I can give. Have you thought about talking to someone at the counseling center?" Chapter 7 discusses what to do if you are seriously worried about a roommate's mental health. How to state a boundary Stating a boundary well follows a consistent pattern, whatever the subject. Four elements tend to make it land. Be specific. Name the behavior and the situation, not the person's character. "When you use my desk while I'm out" is specific. "You're always taking over my space" is not. Name your need briefly. You do not need to justify a boundary at length, and over-explaining tends to invite argument. "I need to be able to find my notes where I left them" is enough. A long, defensive explanation implies that the boundary is only valid if the other person accepts your reasons. Make a clear request. Say what you would like to happen instead. "Could you ask first if you need my desk?" or "Could guests head out by ten on weeknights?" Keep a consequence in reserve. You do not always need to state a consequence the first time. Most boundaries are respected once they are clearly stated. But know what you will do if the boundary is repeatedly ignored, and be prepared to say it calmly: "If it keeps happening, I'll start locking my drawer," or "If this doesn't change, I'll need to talk to our RA about it." Returning to Arjun: a clearer version of his conversation might sound like this. "Hey, I've noticed you've been using my desk when I'm out. I know the outlet's better there, but I'm pretty particular about my stuff, and it throws me off when things get moved. Could you ask me first if you want to use it? And if the outlet's the problem, maybe we can get an extension cord for your side." Ben can say yes to that easily. There is nothing in it he needs to defend. A few further points help with the practice of setting boundaries. Tone matters. A boundary delivered with warmth is far more likely to be respected than one delivered with frostiness. You can be completely firm about the content and still be friendly in the delivery. "Clear is kind," as the researcher and writer Brené Brown puts it in Dare to Lead: vague hints and unspoken expectations feel gentle but are ultimately unkind, because they leave the other person guessing. Guilt is normal. Many people, especially those raised to put others first, feel guilty after setting a boundary. The guilt does not mean you did something wrong. It usually means you did something unfamiliar. If you stated a reasonable need clearly and without contempt, you have done nothing that requires an apology. You can acknowledge the other person's inconvenience—"I know this is a change"—without taking the boundary back. Expect some pushback. People sometimes test a new boundary, especially if it changes a habit they have become comfortable with. That is not necessarily malicious; habits are hard to change, and people forget. A calm, consistent restatement is usually enough: "Remember, we agreed on asking first for the desk." If the testing continues, it is time to use the consequence you held in reserve. Respecting your roommate's boundaries Boundaries run in both directions, and the student who is quick to set limits is not always as quick to honor them. Being a good roommate means taking the other person's boundaries as seriously as your own, even when they seem odd to you. When a roommate sets a boundary with you, the most useful response is simple: listen, ask a clarifying question if you need to, and say yes if you can. "Got it—sorry, I didn't realize. So ask before using the desk, but the lamp's fine?" If you cannot fully agree, say so honestly and look for a compromise, rather than agreeing and then quietly ignoring it. "I get that you need it quiet at eleven. The trouble is my shift ends at ten-thirty, and I get back around ten to eleven. Can I have until eleven-fifteen to get ready for bed, as long as I keep it quiet?" It helps to remember that a boundary someone sets with you is a form of trust. They could have stayed silent and resented you, or complained to their friends. Instead they have told you something about what they need, which gives you a chance to be a good roommate. Receiving boundaries graciously is one of the most effective ways to make the other person feel safe raising things in the future. Watch for your own defensive reactions. It is natural to feel stung when someone says that something you do bothers them. You may want to explain, justify or point out something they do that bothers you. Resist that impulse in the moment. You can raise your own concerns later, in a separate conversation. Responding to a boundary with a counter-complaint almost always turns a simple request into a fight about who is the worse roommate. Finally, notice the boundaries your roommate has not stated. People from some cultures and families find it very hard to say no directly or to ask for things. A roommate who goes quiet, withdraws, or seems uncomfortable when you have guests over may be setting a boundary in the only way they know how. It is generous to check in: "I've had people round a lot this week—is that working for you? Honestly, it's fine to tell me if it isn't." Making it safe for a quieter roommate to speak is one of the most valuable things you can do for a shared home. Boundaries in a small space are never perfect. There is not enough room for either person to have everything exactly as they would like it. But clear, specific, warmly stated limits turn a constant low-level negotiation into a predictable arrangement, and predictability, as Chapter 1 argued, is one of the foundations of any healthy roommate relationship. Chapter 5: Speaking Up Most roommate problems could be solved by one conversation that never happens. The previous chapters have described the forces that keep people silent: the awkwardness of raising something small, the fear of seeming petty, the hope that things will improve on their own. The conversation itself, when it finally happens, can be done well or badly: early or late, specific or sweeping, in a form the other person can hear or one they can only defend against. It is a learnable skill. People who seem naturally good at it have usually just practiced more. And the stakes of doing it well are high, because the way a difficult conversation begins shapes almost everything that follows. The first three minutes The psychologist John Gottman spent decades observing couples as they discussed disagreements, and one of his most frequently cited findings is that the way a conflict conversation starts strongly predicts how it will end. He called a hostile or accusing opening a harsh startup, and in The Seven Principles for Making Marriage Work, written with Nan Silver, he reports that conversations which begin harshly tend to end badly, often within the first few minutes. A gentler opening—what he called a softened startup—makes a constructive outcome much more likely. Gottman's research was on romantic couples, and roommates are not couples. But the underlying dynamic is familiar to anyone who has ever had an argument. When a conversation opens with an attack, the other person's first move is to defend themselves, and from that moment on, both people are fighting about the attack rather than solving the problem. When it opens with a clear, non-blaming description of a concern, the other person has room to listen. A harsh startup often includes one or more of a few common features: a sweeping generalization ("you always," "you never"), a verdict on the person's character ("you're so inconsiderate"), sarcasm, or a question that is really an accusation ("do you even know how to use a sponge?"). A soft startup describes a specific situation, says how it affects you, and makes a clear request. Table 3 shows the difference in some common roommate situations. Table 3. Harsh and soft startups compared. Situation Harsh startup Soft startup Dishes "You never wash anything. It's disgusting." "There are dishes from the weekend in the sink. Can you get to them today?" Late calls "Do you ever think about anyone but yourself?" "The calls after midnight are waking me up. Could you take them in the lounge?" Borrowing "You just take whatever you want." "My charger went missing again. Could you ask before borrowing it?" Guests "Your boyfriend basically lives here now." "He's stayed over most nights this week. Can we talk about a limit?" Mess "Your side is a pigsty." "Your clothes are spreading onto my side. Could you keep them on yours?" Notice that the soft versions are not weaker. They are, if anything, clearer. They name the exact problem and the exact request, which the harsh versions do not. A roommate who hears "you're so inconsiderate" has to guess what you want. A roommate who hears "could you take the calls in the lounge after midnight?" knows exactly what to do. A useful structure for a soft startup comes from Marshall Rosenberg's Nonviolent Communication, which breaks a clear request into four parts: an observation, a feeling, a need, and a request. The observation describes what actually happened, without evaluation, as a camera would record it. The feeling says how you feel about it. The need names what matters to you. The request says what you would like the other person to do. In practice, it sounds something like this: "When the music's on while I'm studying"—observation—"I get really distracted and stressed"—feeling—"because I need quiet to focus on this reading"—need. "Would you be willing to use headphones when I'm at my desk?"—request. Written out in full, the structure can sound stilted, and you do not need to use every part every time. Its real value is as a check. Before you raise something, ask: have I described what happened without judging it? Have I said how it affects me? Have I made a specific request? If the answer to all three is yes, your opening is probably soft enough. Two cautions about the structure. First, "I feel that you're being selfish" is not a feeling; it is a judgment wearing the grammar of a feeling. Real feelings are words like stressed, frustrated, tired, anxious, embarrassed, hurt. Second, a request is not a demand. A request leaves room for the other person to say no, suggest something else, or explain a constraint you did not know about. If you are not prepared to hear anything but yes, you are making a demand, and the other person will probably sense it. Preparing, and choosing the moment For small matters, a single sentence delivered in passing is often enough: "Hey, could you turn the music down a bit? I'm trying to read." For anything bigger or more sensitive—a recurring problem, a sensitive topic like guests or hygiene, or an issue you have been sitting on for a while—a little preparation makes a real difference. The authors of Difficult Conversations, Douglas Stone, Bruce Patton and Sheila Heen of the Harvard Negotiation Project, argue that every difficult conversation is really three conversations happening at once. There is the "what happened" conversation, about the facts and who is to blame. There is the feelings conversation, about the emotions that both people are having whether or not they are named. And there is the identity conversation, about what the situation says about each person—am I a good roommate, am I being treated with respect, am I the kind of person who makes a fuss? Preparing well means thinking about all three. A practical preparation routine might look like this. Clarify the problem. Write down, in one or two sentences, the specific behavior that is bothering you and its effect on you. If you cannot state it specifically, you are not yet ready to raise it. "I'm upset about everything" is not a conversation anyone can have. "The bathroom hasn't been cleaned on her weeks for a month, and I've been doing it" is. Separate impact from intent. Stone, Patton and Heen point out that we tend to assume that because we were hurt, the other person meant to hurt us. Usually they did not. You know the impact of your roommate's behavior on you; you do not know their intent. Plan to describe the impact and ask about the intent, rather than assuming it. Consider your contribution. This is not the same as blaming yourself. It means asking whether anything you have done, including staying silent, has contributed to the situation. "I should have said something weeks ago" is a powerful sentence to say in a difficult conversation. It takes some of the heat out of the discussion and makes it easier for the other person to acknowledge their part. Decide what you actually want. What would a good outcome look like? A specific change in behavior? An apology? Simply to be understood? Knowing what you want helps you recognize when you have got it, and stops the conversation from drifting into a general airing of grievances. Anticipate their side. Try, honestly, to imagine how the situation looks from your roommate's perspective. What might they say? What constraints might they be under? What might they be feeling? You do not need to agree with their likely view, but thinking about it in advance will make you calmer and more curious when you hear it. When and how you raise something matters nearly as much as what you say. Choose a moment when both of you are reasonably calm and not rushed. Not when one of you is heading out the door, not late at night when you are both exhausted, and not in the heat of the moment, while you are still furious about the thing that just happened. If you are very angry, wait. Anger narrows attention and makes a harsh startup almost inevitable. A few hours, or a night's sleep, often changes what you want to say. It is usually better to ask for a conversation than to launch into one. "Hey, have you got a few minutes later? There's something about the room I'd like to talk about" gives the other person a chance to be ready and signals that you take the issue seriously but are not ambushing them. Keep the preview neutral and brief so that they do not spend the afternoon dreading it. Choose a private place. Raising a problem in front of other roommates or friends almost always makes the other person more defensive, because now they are also managing embarrassment. If the issue involves several roommates, a shared conversation may be appropriate, but give the person most involved a chance to hear about it privately first where possible. The question of text versus face-to-face is one students often ask about, and the answer depends on the situation. Text has real advantages: it gives the other person time to think before responding, it avoids the heat of a live confrontation, and it can feel less threatening to someone who dislikes conflict. For small, practical reminders—"bins go out tonight"—it is ideal. For anything emotionally significant, though, text has serious drawbacks. It strips out tone, so a neutral message can be read as cold or angry. It invites people to respond quickly and defensively, sometimes in front of a group chat audience. And it makes it easy to escalate, because people say things in writing they would never say to someone's face. A reasonable rule is to use text for logistics and short, friendly reminders, and to talk in person for anything involving feelings, patterns or repeated problems. If you do use text for something sensitive—perhaps because you have agreed that your roommate prefers it—read your message aloud before sending it, cut anything that sounds like sarcasm or a verdict, and consider ending with an invitation to talk: "Happy to chat about it in person if that's easier." Never use a group chat to criticize one person. Even if the complaint is justified, public criticism reliably produces humiliation and defensiveness rather than change. If you must address the whole group about a shared problem, make it about the problem, not about any individual. During and after the conversation Consider a third-year student—call her Leila—who has been bothered for several weeks by her roommate, Chloe, coming back late and turning on the overhead light. Leila has been lying awake afterward and feels exhausted. She decides to raise it. She waits for a relaxed afternoon when they are both in the room, and asks if Chloe has a few minutes. "I wanted to mention something about the evenings," Leila begins. "When you get back late and the big light goes on, it wakes me up, and I've been having a hard time getting back to sleep. I've got early classes this term and I'm getting pretty worn down. I should probably have said something sooner. Would you be up for using the desk lamp or your phone light when you come in after I'm asleep?" Chloe's first response is slightly defensive: "I didn't know it was that bad. I try to be quiet." This is the moment when many conversations go wrong. Leila could respond by listing all the nights she was woken, proving that it really is that bad. Instead, she acknowledges Chloe's point. "I know you do—you're really quiet, honestly. It's just the light. It hits me straight in the face." That acknowledgment takes the heat out of the moment. Chloe laughs, says she had no idea, and suggests she could get a small lamp for her side. The conversation lasts five minutes. Several things made this work. Leila opened softly and specifically. She took a little responsibility for waiting. She made a clear request. And when Chloe got defensive, Leila did not argue about the facts; she acknowledged what was true in Chloe's response and repeated the core issue. Some general principles for the conversation itself: Listen more than you think you need to. After you have stated the issue, stop talking and let the other person respond. People rarely change their behavior until they feel their side has been heard. If they explain their situation, reflect it back: "So your shift runs late and you're just trying to get to bed quickly. That makes sense." Expect some defensiveness, and do not match it. Almost everyone feels a flash of defensiveness when told that something they do is causing a problem. It is not necessarily a sign that the conversation is failing. Respond calmly, acknowledge anything reasonable in what they say, and return to the specific issue. Stay on one topic. It is tempting, once the conversation has begun, to raise everything else that has been bothering you. Resist it. Adding a second and third complaint turns a solvable problem into an indictment. If there are other issues, raise them separately, another day. End with something concrete. Before the conversation finishes, make sure you both know what has been agreed. "So—lamp instead of the big light after eleven. Thanks, that'd really help." If nothing specific has been agreed, the conversation has probably not solved the problem. Follow up with appreciation. When your roommate changes their behavior, notice and acknowledge it. "Thanks for the lamp thing—I've been sleeping so much better." Appreciation reinforces the change, and it shows that raising problems in your room leads to good outcomes rather than lasting tension. That makes the next conversation easier for both of you. If you have already got it wrong—if you exploded before you had a chance to prepare, or opened with the harsh version of Table 3—the situation is recoverable. Go back, once you are calm, and separate the delivery from the substance: "I'm sorry about how I raised that yesterday. I was really fed up, and it came out as an attack. The actual issue still matters to me, though, so can we talk about it properly?" Most people respond well to this, and it often produces a better conversation than a perfect first attempt would have. When you are on the receiving end Sooner or later, your roommate will raise something about you. How you respond matters as much as how you raise things yourself, because it determines whether they will feel safe doing it again. The first instinct, for almost everyone, is to defend: to explain why the behavior was reasonable, or to point out something the other person does that is just as bad. Both responses are understandable, and both are almost always counterproductive. They tell the other person that raising problems leads to arguments, and they teach them to stay silent next time—which, as Chapter 1 showed, is where serious conflict begins. A better response has three parts. First, thank them for telling you, even if it stings. "Thanks for saying something—I'd rather know." Second, make sure you understand. Ask a question if you need to, and summarize what you have heard: "So it's the light, not the noise?" Third, respond to the request. If you can do what they ask, say so. If you cannot, explain your constraint and offer an alternative. You can raise your own concerns too, but not in the same breath. If you have a grievance of your own, wait a day and raise it separately, using the same skills. "Hey, I've been thinking about our conversation yesterday. There's something I wanted to mention too." Separating the conversations keeps each one focused and prevents the exchange from turning into a scoring contest. Being able to receive feedback gracefully is, in the end, one of the most important roommate skills there is. A roommate who can hear "that bothers me" and respond with "thanks for telling me" is someone almost anyone can live with. Chapter 6: When Conflict Escalates Even with good agreements, clear boundaries and careful conversations, some roommate disputes will escalate. Someone will be tired, or stressed, or hurt, and a conversation about the dishes will turn into shouting about respect. A disagreement over a guest will become a week of silence. Two roommates in a four-person apartment will stop speaking, and the other two will be pulled into choosing sides. Escalation is not a sign that you have failed. It is a normal feature of close living, and nearly every roommate relationship goes through it at least once. What distinguishes relationships that survive from those that break down is not whether conflict gets heated but what happens when it does: whether both people can recognize the danger signs, step back before lasting damage is done, and repair things afterward. The four patterns and the flood John Gottman's research on couples, introduced in the previous chapter, identified four patterns of communication that were especially strong predictors of relationships breaking down. He called them, memorably, the Four Horsemen: criticism, contempt, defensiveness and stonewalling. Roommate relationships are not marriages, but these four patterns appear in any close relationship under strain, and learning to recognize them is one of the most useful things you can do. Criticism means attacking the person rather than addressing the behavior. It is the difference between a complaint—"the dishes weren't done this week"—and a criticism—"you're so lazy and selfish." Complaints are a normal and healthy part of living together. Criticism turns a problem into a character judgment, which the other person has to defend against rather than solve. Contempt is criticism laced with disdain: sarcasm, mockery, eye-rolling, sneering, name-calling, or a tone that communicates "I am better than you." Gottman identified contempt as the most destructive of the four. It tells the other person that they are not just wrong but beneath you, and it is extremely hard to recover from. In roommate life, contempt often shows up in subtle forms: a mocking impression of a roommate's accent or habits shared with friends, a sigh and an eye-roll when they walk in, a sarcastic "oh, great, you're home." It can also show up in group chats, where roommates privately joke about one member of the household. Even when the person never sees those messages, the attitude behind them leaks out. Defensiveness is the reflexive response to feeling attacked: making excuses, denying responsibility, or immediately counter-attacking with a complaint of your own. "Well, you never clean the bathroom either." Defensiveness is understandable—it is a natural response to criticism—but it tends to escalate conflict, because it tells the other person that their concern has not been heard. Stonewalling is withdrawal: going silent, refusing to engage, leaving the room, or answering in monosyllables. In roommate life, it often takes the form of the silent treatment, which can last for days or weeks. People often stonewall because they feel overwhelmed and do not know how else to protect themselves. But to the other person, stonewalling feels like rejection, and it usually makes them pursue harder, which makes the stonewaller withdraw further. Each of the Four Horsemen has an antidote, and Gottman's work describes them in some detail. The antidote to criticism is the soft startup described in Chapter 5: a specific complaint about behavior, without a verdict on character. The antidote to contempt is building a culture of appreciation and respect—noticing and saying what the other person does well, and refusing to mock them, even privately. The antidote to defensiveness is taking responsibility for even a small part of the problem: "You're right, I did leave them there." And the antidote to stonewalling is learning to calm down and then come back, which deserves a closer look, because it is the skill that makes every other one possible. A useful exercise is to review your last few arguments with a roommate and ask honestly which of the Four Horsemen appeared, and from whom. Most people find that they have a signature pattern—some criticize, some get defensive, some stonewall. Knowing your own tendency lets you catch it in the moment. One of Gottman's more practical findings concerns what he called flooding: a state of physiological overwhelm in which the body's stress response takes over during conflict. When you are flooded, your heart rate rises, your muscles tense and your ability to think clearly and listen drops sharply. You may feel an overwhelming urge either to attack or to escape. In this state, productive conversation is almost impossible. People say things they later regret, hear threats that were not intended, and lose track of what the argument was even about. The signs of flooding are recognizable if you learn to watch for them: a pounding heart, a hot face, a clenched jaw, a sense of tunnel vision, a feeling that you have to win or get out. When you notice them, the most useful thing you can do is stop the conversation and take a break. A good break has three features. First, it is announced, not simply taken. Walking out without a word is stonewalling; it leaves the other person feeling abandoned or dismissed. Saying "I'm getting too wound up to talk about this well. I need to take a break and come back to it" is a very different act. Second, it has a return time. "Can we pick this up after dinner?" or "Let's talk again tomorrow afternoon" reassures the other person that you are not avoiding the issue. Gottman suggests that the body needs at least about twenty minutes to calm down after flooding, so the break should be at least that long. Third, it is used for calming down, not rehearsing. If you spend the break replaying the argument and preparing devastating replies, you will return just as flooded as you left. Go for a walk, listen to music, do something absorbing. The goal is to let your body settle. Consider how this plays out between two roommates—call them Nadia and Grace—who have been arguing on and off about Grace's boyfriend staying over. One evening Nadia raises it again, and within a minute Grace is saying, "You've hated him from day one," and Nadia is saying, "This isn't about him, it's about you never asking." Both voices are rising. Nadia notices that her hands are shaking and that she has started composing insults in her head. She stops. "I'm too worked up to do this properly right now. I don't want to say something I'll regret. Can we talk tomorrow after lunch?" Grace, still angry, says fine and puts her headphones on. Nadia goes for a long walk. The next afternoon, the conversation is still uncomfortable, but it is a conversation rather than a fight: Grace admits the overnight stays had become more frequent than she realized, Nadia admits she had been cold to the boyfriend, and they agree on a limit of two nights a week with a text in advance. Nothing about the underlying disagreement changed overnight. What changed was that both of them were capable of thinking again. Roommates can agree on a break signal in advance, ideally during a calm check-in. "If either of us says we need a break, we stop, and we come back to it later" is a simple rule that can prevent a lot of damage. It helps to agree that asking for a break is always legitimate and never a sign of weakness or avoidance, as long as you come back. Understanding conflict styles People handle conflict in characteristically different ways, and roommates whose styles clash can escalate simply because each reacts badly to the other's approach. One widely used framework, the Thomas-Kilmann Conflict Mode Instrument developed by Kenneth Thomas and Ralph Kilmann in the 1970s, describes five broad approaches to conflict, organized by how much a person asserts their own concerns and how much they cooperate to meet the other person's. Table 4 summarizes them. Table 4. Five conflict styles and their costs. Style What it looks like When it helps Cost if overused Competing Pushing for your own position Safety issues, clear rule violations Damages trust; others stop cooperating Accommodating Giving the other person what they want When the issue matters more to them Resentment builds; your needs go unmet Avoiding Sidestepping or postponing Trivial issues, or when you need to cool off Problems grow in silence Compromising Each gives up something Time pressure, equal stakes Nobody fully satisfied; creativity lost Collaborating Working together to meet both sets of needs Important, ongoing issues Takes time and energy None of these styles is simply good or bad. Each has situations where it fits. If a roommate is endangering your safety, competing—firmly insisting on a change—is appropriate. If an issue matters much more to them than to you, accommodating is generous and sensible. If something is genuinely trivial, avoiding it may be fine. The problem is that most people default to one or two styles regardless of the situation. In roommate conflict, two combinations cause particular trouble. The first is an avoider living with a competitor. The competitor raises issues forcefully; the avoider withdraws; the competitor, frustrated, pushes harder; the avoider withdraws further. This is the pursue-withdraw pattern, and it can escalate until the avoider eventually explodes or moves out. The second is two accommodators, each quietly giving way to the other, each silently resenting it, until the relationship is full of unspoken grievances and neither knows why. For most ongoing roommate issues, the most useful style is collaborating—treating the problem as a shared puzzle rather than a contest. That is the approach behind the position-and-interest distinction from Getting to Yes, discussed in Chapter 2. Consider two roommates who are fighting about temperature. One keeps opening the window; the other keeps closing it. Stated as positions, the conflict is unsolvable: window open versus window shut. Explored as interests, it opens up. One needs fresh air because the room feels stuffy and gives her headaches. The other needs warmth because he is always cold and cannot sleep. Solutions appear once the interests are on the table: the window opens for half an hour in the late afternoon when he is out; he gets an extra blanket; they ask maintenance to check whether the radiator is working. Neither gets their position, and both get what they actually needed. If you know your default style, you can make deliberate choices. A habitual avoider can practice raising things earlier; a habitual competitor can practice asking questions before stating positions; a habitual accommodator can practice noticing when giving way is breeding resentment. Repair Every roommate relationship will have moments that go badly: an argument that gets too heated, a cutting remark, a door slammed. Gottman found that what distinguished stable relationships from unstable ones was not the absence of such moments but the presence of what he called repair attempts—any statement or action that stops negativity from spiralling out of control. A joke that breaks the tension, a touch on the arm, a "let's start over," a simple "I'm sorry, that came out wrong." In healthy relationships, these attempts are made often and received well. For roommates, repair comes in two forms: repair during a conflict, and repair afterward. During a conflict, repair can be as simple as naming what is happening. "Hang on, we're both getting really defensive." "I don't want to fight about this—can we try again?" "That came out harsher than I meant." Such statements do not settle the underlying issue, but they stop the escalation and create room to address it. They work best when they are received generously. If your roommate makes a clumsy attempt at repair, meet it halfway, even if you are still annoyed. After a conflict, repair usually requires a genuine apology, and it is worth knowing what makes one work. A good apology names the specific thing you did, acknowledges its effect on the other person, takes responsibility without excuses, and says what you will do differently. "I'm sorry I snapped at you about the dishes last night. That was out of proportion, and it wasn't fair to you. I was stressed about my exam, but that's not your problem. Next time I'll try to say something earlier instead of letting it build up." A poor apology, by contrast, is vague ("sorry if anything I said upset you"), shifts responsibility ("I'm sorry, but you started it"), or comes with conditions ("I'll apologize if you do"). Such apologies tend to make things worse, because the other person hears that you are not really sorry. Apologizing does not mean conceding everything. You can apologize for how you handled something while still holding to the underlying point. "I'm sorry I shouted. I still think we need to sort out the guest thing, but I shouldn't have said it like that." Separating the manner from the substance makes it easier to repair the relationship without abandoning your needs. Receiving an apology well also matters. If a roommate apologizes, accept it, even if you are still somewhat hurt. "Thanks. I appreciate that." You do not have to pretend everything is fine, but meeting an apology with further criticism tells the other person that repair is pointless. Conflict in larger groups Many students live not with one roommate but with three, four or more, in suites or shared houses. Group living brings its own escalation patterns. The most common is triangulation: instead of raising a problem directly with the person involved, you complain about them to a third roommate. It is an understandable impulse—venting feels better than confronting—but it has predictable effects. The third person is put in an awkward position, the complaint spreads, and the original person eventually senses that they are being talked about. Factions form. Group chats split. What started as a problem between two people becomes a problem for the whole household. The antidote is a simple household norm: problems are raised with the person directly. If you are the third party and a roommate starts complaining to you about someone else, it is helpful to listen briefly and then redirect: "That sounds frustrating. Have you talked to her about it?" You can offer to help think through how to raise it, but avoid becoming the messenger or the judge. For problems that genuinely affect everyone—a shared kitchen that is always a mess, a household that cannot agree on quiet hours—a house meeting is often the best approach. A few principles make meetings work. Hold them regularly rather than only in crisis, so that a meeting is not a sign that something has gone wrong. Keep them short, with a simple agenda. Focus on systems and agreements rather than individuals. Make sure every person gets a chance to speak, especially quieter members, who may need to be directly invited. And end with specific decisions, written down and shared. Watch out for majority rule becoming a tool for isolating one person. In a house of four, three people agreeing on something does not automatically make it fair to the fourth, especially if the decision affects that person disproportionately. The goal is not to win votes but to find arrangements that everyone can live with. When conflict in a group becomes entrenched—factions that no longer talk, one person frozen out, meetings that turn into blame sessions—it is usually time to bring in outside help, such as a resident assistant or residence director trained in mediation. Chapter 8 explains how that process works. Seeking help at that point is not a failure; it is often the fastest route back to a household where everyone can come home without dread. Chapter 7: Differences That Run Deeper Most of this book has dealt with differences in habit: when people sleep, how they clean, how often they have guests. Those differences are real, but they are relatively easy to negotiate, because nobody's identity depends on whether dishes are washed at night or in the morning. Some roommate differences go deeper. They involve culture, faith, money, sexuality, gender, race, disability, neurodivergence and mental health—parts of a person's life that are bound up with who they are. When these differences cause friction, the ordinary tools still apply, but they need to be used with more care, and it helps to understand some of the particular ways these situations go wrong. University is, for many students, the first time they have lived closely with someone from a very different background. That is one of its great opportunities. It is also a source of genuine difficulty, and pretending otherwise does not help anyone. Culture, faith and money Cultural differences often show up not in dramatic clashes of values but in small matters of style. One of the most consequential is how directly people communicate. The anthropologist Edward T. Hall, in books such as Beyond Culture, distinguished between low-context communication, in which meaning is carried mainly by explicit words, and high-context communication, in which much of the meaning is carried by shared understanding, tone, silence and relationship. The distinction is a simplification, and individuals vary enormously within any culture, but it captures something that roommates encounter constantly. A roommate from a more direct communication background may say exactly what they think—"your music is too loud"—and consider this perfectly polite. A roommate from a more indirect background may find the same sentence shockingly rude, and may themselves signal discomfort through hints, silence or a change in manner that the direct roommate never notices. Each can come to see the other as either aggressive or evasive. Neither is right; they are speaking slightly different dialects of politeness. Consider two first-year roommates—call them Mei and Hannah. Mei grew up in a family where openly refusing a request from a peer felt disrespectful, and where discomfort was usually conveyed by a slight withdrawal that everyone understood. Hannah grew up in a family that argued cheerfully at dinner and said exactly what it meant. When Hannah's friends began gathering in the room most evenings, Mei started staying late at the library and answering Hannah's questions with a brief "it's fine." Hannah took her at her word. After six weeks Mei asked housing for a room change, and Hannah, blindsided, told everyone she had been abandoned for no reason. Neither of them had behaved badly. Each had sent signals in a language the other did not read. A single conversation about how each of them preferred to raise problems, early in the term, would very likely have kept them together. The practical implication is to discuss communication style explicitly, as suggested in Chapter 2, and to make allowances in both directions. If you are a direct communicator living with someone more indirect, soften your delivery, check in more often and watch for non-verbal signs that something is wrong: "You seemed a bit quiet when my friends were here. Was that too much?" If you are an indirect communicator living with someone more direct, try to say what you need in plain words, even if it feels uncomfortable, and try not to hear their directness as hostility. Faith and religious practice raise specific practical questions. A roommate may pray at particular times, observe dietary laws, fast during certain periods, keep a sabbath or avoid alcohol. These practices are not preferences in the same sense as a taste in music; for the people who observe them, they are obligations. Good roommates learn what they involve and make reasonable room for them. That might mean keeping the room quiet during prayer times, labeling food carefully in a shared fridge, not bringing alcohol into a shared room without discussion, or being mindful of eating in front of someone who is fasting. It also works the other way: a religious roommate living with someone who does not share their faith is entitled to their practice but not to impose it on the other person or to treat their roommate's different choices as wrong. Money is one of the least discussed and most corrosive sources of roommate tension. Students come to university from vastly different financial backgrounds, and those differences show up in everyday choices: who can afford takeaway, who buys the cleaning supplies, whether splitting costs evenly is fair, whether it is reasonable to suggest a shared purchase. A student working long hours to pay rent may feel invisible and resentful living with someone who never worries about money; a wealthier student may feel awkward and not know how to raise the subject without seeming condescending. A few practices help. Agree explicitly on what is shared and how costs are divided before anything is bought, rather than presenting a roommate with a bill after the fact. Do not assume that everyone can afford the same things, and do not make suggestions that require others to spend money without giving them an easy way to decline. Keep a simple, transparent record of shared expenses—many roommates use a shared note or a bill-splitting app—so that nobody has to raise the subject awkwardly. And if you are the roommate with more financial freedom, be careful that your choices do not become the household's default. Suggesting a weekly shared takeaway can be a friendly gesture to one person and a real burden to another who feels unable to refuse. Identity and discrimination Roommates who differ in race, ethnicity, sexuality, gender identity, disability or nationality face the ordinary challenges of shared living, and sometimes more. It is worth being clear about a distinction here. There is a difference between friction arising from difference—unfamiliarity, misunderstanding, clumsy questions—and discrimination or harassment, which is conduct that demeans, excludes or targets someone because of who they are. Friction arising from difference is normal and often resolvable. A roommate may ask questions about your hair, your food, your accent, your faith or your pronouns that feel intrusive or tiring. Often they come from genuine curiosity and a lack of experience rather than malice. You are not obliged to educate anyone, and it is entirely reasonable to say, "I'd rather not get into that right now." But many students find that a patient answer, where they have the energy for one, can turn an awkward moment into a better understanding. If you are the one asking, it helps to remember that the other person has probably been asked the same question many times, to ask in a way that makes declining easy, and to do some of your own learning rather than relying on your roommate to be your teacher. It is also important to take responsibility for your own mistakes. If you use the wrong pronoun for a roommate, make an assumption based on their background or tell a joke that lands badly, the most useful response is a brief, genuine correction and apology—"sorry, they"—and then to move on and do better. Long, anguished apologies tend to make the other person manage your discomfort, which is its own burden. Discrimination and harassment are different. Slurs, mockery of someone's identity, deliberately misgendering, excluding someone from shared spaces because of who they are, threatening or intimidating them, or sharing demeaning content about them are not differences of habit to be negotiated. They are serious, they are typically prohibited by university conduct codes, and in many places they may also violate the law. If you are experiencing this kind of treatment, it is not your job to fix it with good communication skills. Document what happens—dates, times, what was said or done, any witnesses, screenshots of messages—and report it to residence staff or to your university's equity, diversity or student conduct office. Many universities also have specific support services for students experiencing identity-based harassment. You are entitled to feel safe where you live. If you witness a roommate being treated this way by others in your household, your response matters. Simply saying "that's not okay" in the moment, checking in privately with the person targeted and supporting them if they choose to report can make an enormous difference. Silence from bystanders is often experienced as agreement. Neurodivergence, disability and different needs Some roommate friction stems from differences in how people's brains and bodies work. Roommates who are autistic, have ADHD, have sensory processing differences, live with chronic illness or have other disabilities may have needs that differ from those of the people they live with—and may not always want to disclose why. Neurodivergent students often experience shared living in particular ways. An autistic student may find unpredictable noise, light or visitors genuinely overwhelming rather than merely annoying, and may rely on routine and clear expectations in a way that roommates can misread as rigidity. A student with ADHD may genuinely struggle to remember chores or notice mess, not from indifference but because of differences in attention and executive function. Neither of these is an excuse for ignoring shared responsibilities, but both suggest different strategies. Several of the tools already described in this book happen to be especially helpful here. Explicit, written agreements work well for people who find unspoken social rules confusing, as many autistic people report. Clear chore charts with specific definitions of "done," visible reminders and a fixed schedule suit many people with ADHD far better than vague expectations. Agreed quiet hours and advance notice of guests help anyone with sensory sensitivities. In a sense, a well-run roommate agreement is simply good accessible design for a shared home. Consider a second-year student—call him Luca—who has ADHD and shares a flat with two others. For months, his roommates have been frustrated that his dishes pile up and that he forgets his bathroom weeks, and Luca has been frustrated with himself, feeling lectured and ashamed. When the three of them finally talk about it, the conversation shifts from blame to design. Luca explains that he does not see the pile until it is enormous and that verbal reminders evaporate by the evening. They agree that he will take the jobs that happen on a fixed day—bins and recycling—rather than the ones that require noticing, that a shared calendar will send an alert on his cleaning mornings, and that he will wash his dishes immediately after dinner as a routine rather than "when they need doing." His roommates still sometimes have to nudge him, but they no longer read his lapses as contempt, and he no longer dreads coming into the kitchen. A student with a disability or health condition that affects their housing needs—a need for a single room, for particular equipment, for a quieter environment or for accommodations around a service or assistance animal—should contact their university's disability services office. Many institutions have formal processes for housing accommodations, and it is often better to go through that process than to rely solely on a roommate's goodwill. If your roommate has formal accommodations, respect them, even if you do not know the reason for them. It is also worth remembering that disclosure is a personal choice. A roommate is not obliged to explain why they need the light off, why they find guests difficult or why they take particular medication. You can ask what would help without asking for a diagnosis: "Is there anything about the room that would make it easier for you?" When you are worried about your roommate Sometimes the difficulty is not a conflict at all but a growing sense that your roommate is not okay. You notice that they have stopped going to class, that they sleep most of the day, that they have stopped eating or are eating in secret, that they are drinking heavily or using drugs, that they cry often, that they talk about feeling hopeless, or that they have made comments about not wanting to be alive. Living in close proximity, roommates are often the first to notice these signs. This is one of the hardest situations in shared living, because it pulls in two directions at once. You care about your roommate and want to help. You are also a student with your own life and your own limits, and you are not a trained professional. If you are worried, it is usually right to say something. A gentle, private check-in is a good start: "I've noticed you've seemed really low lately, and you haven't been going to class. I'm not trying to pry, but I wanted to check how you're doing." Listen without rushing to fix things. Let them know that support exists, and offer to help them access it: "Would it help if I looked up the counseling center's hours with you?" or "I could walk over there with you if you want." If you are concerned that your roommate may be thinking about suicide, it is appropriate to ask them directly. Asking someone whether they are having thoughts of suicide does not put the idea in their head, a point emphasized by major suicide prevention organizations; it often brings relief, because it tells the person that they can talk about what they are feeling. If they say yes, or if you are worried for any reason, help them connect with support right away. In the United States, the 988 Suicide & Crisis Lifeline can be reached by calling or texting 988, and it offers support to people in crisis and to those worried about someone else. Other countries have their own crisis lines, and most universities publicize local numbers and after-hours support. If someone is in immediate danger—they have a plan and the means, they have harmed themselves, or you believe they may do so imminently—contact local emergency services straight away. Do not leave them alone if it is safe to stay, and tell residence staff. Alcohol and other drugs call for a specific kind of vigilance. If a roommate has been drinking heavily and shows signs of alcohol poisoning—confusion, vomiting, seizures, slow or irregular breathing, cold or clammy or bluish skin, or being hard to wake up—call emergency services. Do not assume they will sleep it off, and do not leave them lying on their back, where they could choke if they vomit. Many universities have medical amnesty or "good Samaritan" policies designed to encourage students to call for help in such situations without fear of disciplinary consequences, but these policies vary, and in an emergency the priority is always to get help. It is important to be honest about confidentiality. If a roommate confides something serious and asks you to keep it secret, you may find yourself in a painful position. You can promise to respect their privacy about most things, but you cannot keep a secret that puts their life or safety at risk, and you should not promise to. Resident assistants and many other university staff are typically required to pass on certain kinds of information, particularly about safety, so it is worth knowing that before you share something with them, and being honest with your roommate about it too. Counseling services generally offer a higher level of confidentiality, though even there, limits usually apply when someone is at serious risk. Finally, look after yourself. Supporting a roommate who is struggling can be exhausting and frightening, especially if it goes on for weeks. It is not selfish to set limits on how much you can do, to talk to a counselor yourself, or to ask residence staff for help. You do not have to carry this alone, and you should not be your roommate's only support. In many situations, the most caring thing you can do is to connect them with people who are trained to help, and then to continue being a decent, ordinary presence in their life—a roommate, not a therapist. Chapter 8: When It Isn't Working Sometimes, despite everyone's best efforts, a roommate relationship does not work. The conversations happen and nothing changes. The agreement is ignored. Boundaries are crossed again and again. The atmosphere in the room becomes so tense that one or both people dread coming home. Occasionally, a situation moves beyond ordinary incompatibility into something that is genuinely unsafe. It is important to be able to recognize these situations and to know what to do about them. Students often stay in bad living situations far longer than they should, out of guilt, a sense that they have not tried hard enough, uncertainty about whether their problem is serious enough to justify involving anyone else, or simple ignorance of what options exist. Equally, some students jump straight to demanding a room change after a single bad week, without trying the steps that might have resolved things more easily. The skill lies in matching the response to the problem. The escalation ladder It helps to think of the responses available to you as a ladder, climbed one rung at a time. Most problems are resolved on the lower rungs. You move higher only if the lower steps have not worked, or if the situation is serious enough to skip them. Table 5 sets out the rungs. Table 5. An escalation ladder for roommate problems. Step When to use it What it looks like 1. Direct conversation Most problems, as early as possible A private, specific, soft-startup conversation 2. Revisit the agreement Repeated or pattern problems A formal check-in to renegotiate specific terms 3. Informal staff advice Direct efforts not working Asking an RA for advice on handling it yourself 4. Facilitated mediation Stalemate or entrenched conflict A staff member runs a structured conversation 5. Formal involvement Policy violations, ongoing refusal to engage Housing or conduct processes 6. Room change or move Irreconcilable situations or safety concerns Relocation through housing, or leaving a lease The first two rungs are the subject of most of this book. If you have raised an issue directly and clearly, perhaps more than once, and revisited the roommate agreement without success, it is reasonable to move to the next step. Moving up the ladder too quickly carries its own costs. A roommate who first learns that something is wrong when a staff member calls them in for mediation will usually feel ambushed and aggrieved, and may be far less willing to cooperate than if you had raised it with them directly. Unless there is a safety concern, give the lower rungs a fair chance, which usually means at least one clear, direct conversation and a reasonable period to see whether anything changes. A useful test before climbing is whether you could honestly say to a staff member, "I have told them specifically what the problem is and what I need, and it hasn't changed." If you cannot yet say that, the direct conversation is probably still ahead of you. The third rung—seeking informal advice—is underused. Many students imagine that talking to a resident assistant means "reporting" their roommate or getting them in trouble. Usually it does not. Resident assistants and similar residence staff spend a great deal of their time helping students think through roommate problems, and a quiet conversation with one can be enormously useful. They can help you plan what to say, suggest approaches you had not considered, tell you what the relevant residence policies are and give you a realistic sense of how serious your situation is. Many students find that after one such conversation, they go back and resolve the problem themselves. When you seek advice, it helps to be clear about what you want. "I don't want to make this formal yet, I just want advice on how to handle it" is a reasonable request, and most staff will respect it for ordinary roommate problems. Be aware, though, that staff are generally required to act on certain kinds of information, especially involving safety, harassment, sexual misconduct or serious policy violations, so you should not assume that everything you say will stay informal. Mediation If direct efforts fail, facilitated mediation is often the most effective next step. Most residence life programs offer some form of it, usually led by a resident assistant or a more senior staff member such as a residence director. The details vary between institutions, but the process usually looks similar. A mediator typically meets with each roommate separately first, to hear their perspective and understand what they hope to get out of the process. Then the roommates meet together, with the mediator present. The mediator sets ground rules—each person speaks without interruption, no name-calling, focus on specific behaviors—and guides the conversation. Each person describes the situation as they see it, and the mediator helps them identify the underlying interests, clarify misunderstandings and generate possible solutions. The session usually ends with a revised roommate agreement, written down and signed by both people, and sometimes a follow-up meeting to check how things are going. Mediation works for several reasons. The presence of a neutral third party tends to make both people behave better, reducing the likelihood of the Four Horsemen described in Chapter 6. The mediator can slow the conversation down, reframe accusations as concerns and make sure that each person actually hears the other. And the structure itself—separate meetings, ground rules, a written outcome—gives the process a seriousness that roommates' own conversations sometimes lack. To get the most out of mediation, prepare in advance. Write down the two or three specific issues that matter most to you, with concrete examples. Think about what you actually want to change, and what you would be willing to offer in return. Consider your own contribution to the problem honestly; mediators notice when one person arrives ready to take some responsibility, and it tends to make the other person more willing to do the same. Go in expecting to listen as well as to speak. The goal is not to win the case against your roommate but to leave with an arrangement that makes the room livable. Consider two roommates—call them Tom and Marcus—whose relationship has deteriorated over a term. Tom thinks Marcus is inconsiderate: late nights, loud gaming, friends over constantly. Marcus thinks Tom is uptight and passive-aggressive: sighing, slamming drawers, leaving pointed notes. They have barely spoken in weeks. In separate meetings, a residence director hears both stories and notices that each of them is describing the other's reactions to his own behavior. In the joint session, she asks each to describe what a normal weeknight looks like for him. Marcus explains that he plays online games with his friends from home, who live in a different time zone, which is why he is up late. Tom explains that he has an early lab every day and has been getting about five hours of sleep a night, which is why he has become snappish. Neither knew this about the other. They agree on a set of changes: Marcus will use a headset and game in the lounge after midnight on weeknights; Tom will stop leaving notes and will raise issues directly; they will check in with the residence director in three weeks. The follow-up meeting is brief. Things are not perfect, but they are working. It is also possible that you will be on the other side of this process: a resident assistant tells you that your roommate has asked for mediation, or that a complaint has been made about you. That is an uncomfortable moment, and the natural reactions are hurt, embarrassment and a strong urge to defend yourself or to retaliate with complaints of your own. It helps to remember that a request for mediation is not a verdict. It usually means that your roommate has been struggling with something and has not found a way to raise it directly, perhaps because they find confrontation frightening, perhaps because earlier attempts did not go well. Go into the process with the same preparation you would bring if you had requested it. Ask yourself honestly what your roommate might be experiencing, and whether any of it could be fair. Listen to the specific concerns before responding, and resist the temptation to answer each one with a counter-accusation. You can and should describe your own difficulties, but students who arrive curious rather than defensive tend to leave mediation with arrangements they can live with, and often with a better relationship than they had before it began. Some students hesitate to involve staff at all because it feels like "telling on" a roommate, a betrayal of the unspoken code that residents sort things out among themselves. That hesitation is understandable, but it rests on a misreading of what residence staff are for. Helping residents live together is a central part of their job, and asking for help with a stuck relationship is no more a betrayal than asking a tutor for help with a stuck essay. What matters is how you do it. Try direct conversation first where it is safe to do so, tell your roommate honestly if you intend to seek mediation—"I don't think we're getting anywhere on our own, and I'd like to ask the RA to help us talk it through"—and frame the request as a search for a solution rather than a complaint. Handled that way, involving a third party is an act of good faith, not an escalation. Mediation does not always succeed. Sometimes one person refuses to participate, or agrees to changes and then ignores them. Sometimes the differences are simply too great. But even a failed mediation is useful, because it creates a record that you have made a good-faith effort, which matters if you later need to request a room change or involve formal processes. When to skip the ladder The ladder is designed for ordinary conflict: different habits, clashing preferences, poor communication, repeated disregard for agreements. It is not designed for situations involving danger or serious misconduct, and in those situations you should not work through the steps in order. Skip straight to seeking help if a roommate: • threatens you, physically intimidates you or is violent towards you; • sexually harasses or assaults you, or pressures you sexually; • steals from you, or damages your belongings deliberately; • harasses you because of your identity, as discussed in Chapter 7; • stalks you, monitors your communications, or records or photographs you without consent in private situations; • brings weapons, or illegal drugs in quantities or ways that put you at risk, into the room; • makes you feel unsafe to sleep in your own room. In any situation involving immediate danger, contact local emergency services first. Then contact campus security and residence staff. For sexual harassment or assault, most universities have a specialized office—in the United States, this is typically the Title IX office—as well as confidential advocates or counseling services who can explain your options, including options that do not require making a formal report. You are entitled to support whatever you decide about reporting. If you are unsure how serious a situation is, it is always reasonable to ask residence staff or a counselor for help in thinking it through. It is worth keeping a simple record in any serious situation: dates, times, what happened, any witnesses and any messages. You may never need it, but if you do, it will be far more useful than a memory reconstructed weeks later under stress. Changing rooms, moving out and moving on Sometimes the right outcome is for one or both roommates to move. This is not a failure. Some combinations of people simply do not work, and continuing to live together can do real damage to both people's wellbeing and studies. In university residence halls, room changes are usually managed through the housing office, and policies vary a great deal. Some institutions hold a room-change period early in each term; others handle requests individually. Some require mediation to have been tried first, except in cases involving safety. Availability often depends on how full the residence is. It helps to find out your institution's policy early—before you need it—so that you know what the options are and what steps you would need to take. When you request a change, be specific and factual. Describe the problem, the steps you have taken to resolve it and why you believe a change is necessary. If you have tried direct conversation and mediation, say so. If there are safety concerns, make them clear, since they may change how quickly the request is handled. Residence staff are generally trying to balance the needs of many students, and a clear, calm request supported by evidence of good-faith effort is more likely to be granted than an angry complaint. Off-campus housing is more complicated, because it usually involves a legal agreement with a landlord. In many places, roommates who sign a joint lease can be held responsible for the whole rent, not just their own share, if a co-tenant leaves or stops paying. Leaving a lease early can carry significant financial consequences. Before signing any shared lease, it is worth reading it carefully and understanding what happens if someone needs to leave. If you are already in a difficult off-campus situation, seek advice from your university's off-campus housing office, student legal services or a local tenants' advice organization before making decisions. Tenancy law varies considerably between countries, states and cities, so general advice is no substitute for advice about your specific situation. However the move happens, try to leave well. That does not mean pretending everything was fine, but it does mean settling shared costs fairly, returning borrowed items, cleaning your part of the space and avoiding final confrontations or parting shots on social media. You are likely to see this person again on campus, perhaps in classes or among mutual friends, and a civil departure makes those encounters much easier. It is also better for you. Leaving with dignity, even when you have been treated badly, lets you put the situation behind you rather than carrying it forward. A difficult roommate situation can leave a mark. Students who have been through one sometimes approach their next living arrangement with anxiety, expecting the worst, scanning for signs of trouble, or keeping their new roommate at a distance to avoid being hurt again. Others blame themselves, wondering whether they caused the problem or should have handled it better. It is worth reflecting honestly on what happened, not to assign blame but to learn. Were there early signs that you missed or ignored? Did you stay silent for too long? Were your expectations unrealistic? Were there things you could have done differently? Equally, were there things that were simply outside your control? Most bad roommate experiences involve a mixture, and seeing that clearly is more useful than either complete self-blame or complete blame of the other person. Carry the lessons forward into your next arrangement. If you stayed silent too long last time, commit to raising small things early. If you never had an expectations conversation, have a thorough one this time. If you let a roommate's needs dominate your own, practice setting clear boundaries from the start. Your next roommate is a different person, and they deserve a fresh start, but you can bring what you have learned to the relationship. If a roommate situation has left you feeling anxious, low or unable to settle, talking to a counselor can help. University counseling services see students dealing with the aftermath of difficult living situations all the time, and there is no need to wait until things feel like a crisis. A bad roommate experience can feel very isolating, and it helps to talk it through with someone who has seen many similar situations and can help you make sense of it. Conclusion It is natural to assume that a good roommate relationship depends mainly on luck: get matched with someone compatible, and everything will be fine; get matched with someone difficult, and nothing will help. The chapters of this book point to a different conclusion. Compatibility matters less than most students think, and maintenance matters more. Look back at the composite students who have appeared along the way. Maya and Jess were not incompatible; they were silent. Amara and Lucy had schedules that looked impossible on paper and made them work with one conversation and a desk lamp. Sam and Kofi had different standards of clean and settled on a kitchen neither loved and both could live with. Tom and Marcus, weeks into a cold war, discovered in a single mediated session that each had been reacting to a situation he did not understand. Mei and Hannah, who genuinely liked each other, parted ways because nobody had asked how either of them preferred to raise a problem. Again and again, the difference between a livable arrangement and a miserable one was not who the people were. It was whether the unspoken had been spoken, whether agreements were kept current, and whether there was a way back after things went wrong. What follows from this If maintenance matters more than compatibility, several things follow that are worth stating plainly. First, the most useful question to ask about a potential roommate is not "are we alike?" but "can we talk?" Students choosing housemates for a second or third year often look for people who share their tastes, schedules or social lives. Those things help. But a friend with similar habits who cannot hear criticism will be harder to live with than an acquaintance with different habits who can. If you have a choice, pay attention to how a prospective roommate responds when you raise something small. That response predicts a great deal. Second, the first two weeks are worth more than the next twenty. Nearly every tool in this book is easier to use early: the expectations conversation, the chore system, the statement of a boundary, the soft request made before resentment has accumulated. The cost of doing these things early is a few awkward conversations. The cost of not doing them is often a term or a year of low-level misery. If you take one practical step away from this book, make it this: have the conversation before anyone has done anything wrong. Third, structure is kinder than goodwill. It is tempting to believe that decent people do not need chore charts, written agreements or scheduled check-ins, and that relying on such things implies a lack of trust. The reverse is closer to the truth. Structures take the burden of noticing, reminding and complaining off individuals, which is precisely what keeps decent people from quietly turning into resentful ones. The chart on the fridge does the nagging so that nobody has to. The monthly check-in makes raising a small problem ordinary instead of dramatic. Good roommates build systems not because they distrust each other but because they want to keep liking each other. Fourth, repair matters more than prevention. No agreement will prevent every conflict. People will have bad weeks, say sharp things and let each other down. A relationship that can absorb those moments—where an apology is made and accepted, where "let's start again" is possible—is far more robust than one that depends on nothing ever going wrong. If you are going to invest in one skill, invest in the ability to say "I handled that badly" and mean it. What remains unsettled Some questions do not have clean answers, and it would be dishonest to pretend otherwise. The first is how much to accommodate. Throughout this book, the advice has been to look for arrangements both people can live with, to treat differences as differences, to respect a roommate's needs even when they seem odd. But accommodation has limits, and they are not always obvious. A student who is endlessly flexible can end up with no space of her own. A student who never bends can make a home unbearable for everyone else. There is no formula for where the line falls. The best guide is probably to notice resentment. Accommodation that you can make freely, without keeping score, is generosity. Accommodation that leaves you quietly furious is a boundary that has not yet been stated. The second is where difference ends and harm begins. Chapter 7 drew a line between friction arising from difference and discrimination or harassment, and that line is real. But in practice, situations are often ambiguous. A roommate's repeated jokes about your accent might be clumsy affection or might be a slow drip of contempt. A roommate's constant partying might be an ordinary difference in lifestyle or might be a pattern that makes it impossible for you to sleep, study or feel safe. When you are unsure, it is legitimate to ask for help in deciding. Residence staff and counselors see a great many of these situations, and part of their value is helping students calibrate. The third is the tension between respecting a roommate's privacy and acting on concern for their wellbeing. A roommate has a right to their own life, their own choices and their own struggles. But you may also be the person best placed to notice that something is seriously wrong. There is no rule that resolves every case. What can be said is that when safety is at stake, the balance tips firmly toward acting, and that you are never obliged to hold a secret that puts someone's life at risk. What to do differently If you are about to move in with someone new, begin with the expectations conversation. Go first with your own quirks. Ask before you state. Be concrete. Write it down, and put a check-in on the calendar. If you are already living with someone and things are fine, protect what you have. Keep doing your share without being asked. Say thank you when your roommate does theirs. Raise small things while they are still small. And make sure your roommate knows that they can raise things with you, by receiving the first one well. If things are tense, look honestly at the pattern. Is there something you have not said? Is there a story about your roommate's character that has grown up in the silence? Are the Four Horsemen showing up in your conversations? Pick one issue, prepare for it and raise it with a soft startup. If that does not work, ask for help sooner rather than later. If things are unsafe, do not wait and do not try to handle it alone. Contact the people whose job it is to help. The student who learns to do these things will find that the skills travel. The expectations conversation becomes the conversation you have before moving in with a partner. The chore system becomes the way you share work in a household of your own. The soft startup becomes the way you raise a problem with a colleague. The ability to repair after a bad argument becomes the foundation of every lasting relationship you will have. A shared room is a small space, but the habits you build in it are not small at all. A good roommate relationship, in the end, asks for three things: respect, predictability and the possibility of repair. None requires friendship, though friendship sometimes grows from them. All of them can be built, deliberately, by ordinary people willing to say what they need and to listen when the other person does the same. Hashtags: #HealthyRoommateRelationship #RoommateRelationships #SharedLiving #RoommateConflict #ConflictResolution #HealthyBoundaries #PersonalBoundaries #RoommateCommunication #RespectfulCommunication #StudentWellness #CollegeRoommates #DormLife #CampusLiving #SharedSpaces #RoommateAgreement #HealthyRelationships #EmotionalIntelligence #InterpersonalSkills #ConflictManagement #SettingBoundaries #MutualRespect #HouseholdHarmony #SharedResponsibilities #ChoreManagement #EffectiveCommunication #DifficultConversations #RoommateMediation #RelationshipBuilding #StudentMentalHealth #PeacefulCoexistence

Download the Book (PDF): Introduction At six-forty on a weekday morning, a second-year nursing student stands on a train platform with a backpack, a laptop bag and a canvas tote holding yesterday's leftover rice in a plastic tub. She left the house before anyone else was awake. She will sit through a lecture at eight, a lab at ten, and a two-hour gap in the afternoon that she will spend in the library because there is nowhere else to go. She will get home a little after seven in the evening, eat whatever is easiest, study until her eyes blur, and do it again tomorrow. By Thursday she is tired in a way that sleep on Saturday does not quite fix. By the middle of the semester she is buying lunch from a vending machine three days a week and wondering why she feels worse than her friends who live on campus, even though she is not doing anything obviously wrong. Her situation is not unusual. A large share of university students around the world do not live where they study. They live with parents to save money, or in cheaper housing farther out, or with partners and children, or in the town where their job is. They take buses, trains, trams and subways; they drive in traffic and hunt for parking; some of them cycle or walk long distances. What they share is a fixed block of time at each end of the day that belongs neither to home nor to campus, and a day that stretches from an early departure to a late return with no home base in between. This book is about the health of that student. It deals with three things the description on its cover promises: packing food that actually carries you through a long day, managing the particular tiredness that daily travel produces, and staying physically active when so much of your time is spent sitting in a vehicle, a lecture hall and a library chair. But it does not treat these as three separate topics with three separate lists of tips. It treats them as parts of one problem. The argument of this book The central claim is simple, and everything else follows from it. The commute is not dead time to be endured between the parts of your day that matter. It is a fixed, predictable part of your day, and most of the damage it does comes not from the distance itself but from the friction around it: short sleep forced by early departures, long stretches without food or with the wrong food, hours of unbroken stillness, and the mental load of a day with no place to rest. Those frictions are largely predictable, and what is predictable can be planned for. That shift in view matters because most commuting students treat their travel as an interruption. They plan their studying, perhaps their exercise, sometimes their meals, and then treat the ninety minutes a day on the road as something that happens to them. The result is a day built around a hole. Food decisions get made at the worst possible moment, in a station or a campus shop, hungry and rushed. Sleep is squeezed from both ends because the schedule was never built around the real departure time. Movement disappears because there is no obvious slot for it. And the sense of never quite belonging on campus, of always arriving and always leaving, wears on mood in ways that are easy to misread as personal failing. A student who treats the commute as a designed part of the day asks different questions. What time do I actually need to be asleep, given when the bus leaves? What will I eat at eleven, at two, and at five, and how will it stay safe and appetising in my bag until then? Where in this route can I add ten minutes of walking? What is the commute for: reading, recovery, calling home, or simply looking out of the window? None of these questions is hard. Together they change the day. Who this book is for The reader I have in mind is a university student, undergraduate or graduate, who travels to campus most days and whose round trip is long enough to shape the rest of the day. For some that means twenty minutes each way on a crowded bus; for others it means two hours on a regional train or a long drive on a motorway. The advice is written to work across modes, and where driving, public transport, cycling and walking differ in important ways, the chapters say so. Many readers will be balancing more than study. Commuting students are more likely than residential students to work paid jobs, to care for family members, and to be older than the traditional entering age. The chapters do not assume a large budget, a car, a gym membership, or a kitchen full of equipment. They do assume a reader who is capable of making sensible decisions once they understand what is going on, and who would rather have the reasoning than a list of commandments. Some details, particularly those about campus services, health insurance and transport systems, vary a great deal between countries and institutions. Where a point is specific to the United States, I say so. Food safety temperatures are given in both Fahrenheit and Celsius. What the chapters do The first chapter looks honestly at what the evidence says about commuting and wellbeing, and at the particular position of the commuting student: why long travel tends to cost more than people expect, and why the costs are not evenly spread. Chapter 2 takes apart the idea of "transit fatigue." It is not one thing. It is a combination of sleep loss, circadian mistiming, the odd drowsiness produced by sitting still with little to do, sensory overload in crowded vehicles, motion sickness, and the steady drain of small decisions. Each has a different remedy, and confusing them is why so many standard tips fail. Chapter 3 deals with sleep, because it is where most commuter fatigue begins. It covers how to build a sleep schedule backwards from your departure time, how to use caffeine without borrowing from tonight's sleep, how to nap on transit sensibly, and why drowsy driving deserves to be taken as seriously as drink driving. Chapters 4 and 5 are about food. The first covers what to eat: how to build a packed day of meals and snacks that keeps energy and concentration steady from an early breakfast to a late dinner, on a student budget and with limited time. The second covers how to carry it: food safety, containers, cold chains, what travels well and what does not, and how to make the whole system efficient enough that you will actually keep it up. Chapter 6 covers the things you drink and the things you grab: water, coffee, energy drinks, and the vending machine and convenience-store food that fills the gaps when planning fails. Chapter 7 is about movement. It explains why long sitting matters in its own right, separate from how much exercise you get, and shows how to fold activity into a commuting day through active travel, short bouts between classes and a realistic approach to exercise that does not depend on free evenings. Chapter 8 addresses the mind: the commute as a transition between roles, the stress of unreliable travel, loneliness and belonging on campus, safety when travelling late, and how to decide what the commute itself is for. The conclusion draws these together into a way of thinking about the commuting day that is only possible once the pieces have been laid out, and it names the questions that remain genuinely open. A note on evidence and on medical advice Where this book cites research, the studies are real and are listed in the Notes at the back. The evidence on commuting and health is uneven. Some of it, like the research on food safety temperatures or on the effects of sleep loss, is solid and has been replicated many times. Some of it, like the research on how commuting affects happiness, comes mostly from observational studies that cannot fully separate the commute from everything else in a person's life. I try to say which is which. Where the evidence is thin, I say so rather than dress up a guess as a finding. The student scenarios in this book are illustrative composites, not real people. This book is educational and is not a substitute for individual medical advice. If you have a health condition such as diabetes, an eating disorder, a heart or lung condition, a sleep disorder, or a pregnancy, the general principles here may need adjusting, and a clinician who knows your situation is the right person to help. Most universities have a health service and a counselling service that commuting students are entitled to use, even if they rarely think of them as theirs. Part of the purpose of this book is to encourage you to use them. The nursing student on the platform does not need a different life. She needs a day that has been designed with the commute in it rather than around it. That is what the rest of this book sets out to help you build. Chapter 1: The Hidden Cost of the Daily Journey Ask a commuting student how long their commute is and you will usually get a single number: forty minutes, an hour, "about ninety minutes on a good day." The number is almost always wrong, and not because the student is careless. It is wrong because it measures only the time in motion. It leaves out the fifteen minutes of getting ready earlier than you otherwise would, the wait on the platform, the walk from the car park, the time spent killing because the next train would make you late, and the evening lost to recovery when you get home too tired to do anything useful. The real cost of a commute is not its length. It is the shape it gives to the whole day. This chapter looks at what we know about that cost, why it tends to be underestimated, and why commuting students carry a version of it that differs from that of commuting workers. What the research says about commuting and wellbeing The most striking early finding came not from medicine but from economics and psychology. In 2004 a team led by the psychologist Daniel Kahneman asked about nine hundred working women in Texas to reconstruct the previous day episode by episode and to rate how they felt during each activity. The method, called the Day Reconstruction Method, was designed to capture ordinary experience rather than general impressions. Among all the activities of the day, the morning commute came out at the bottom of the ranking for positive feeling, below work itself and below housework. The evening commute was not much better. A few years later, the economists Alois Stutzer and Bruno Frey examined large-scale German household survey data and described what they called "the commuting paradox." Standard economic reasoning says that people should only accept a long commute if they are compensated for it, through a higher wage, a cheaper or better home, or some other benefit, so that on balance they are no worse off. Stutzer and Frey found that people with longer commutes reported systematically lower satisfaction with their lives, and that the compensation standard theory predicts did not appear to be there. People seemed to be choosing long commutes that left them worse off. Since then, a substantial literature has grown up. A review by Kiron Chatterjee and colleagues, published in Transport Reviews in 2020, pulled together the evidence on commuting and wellbeing and reached conclusions that are useful precisely because they are cautious. Longer commutes are generally associated with lower satisfaction, more stress and less time for health-promoting activities such as exercise and sleep. But the size of the effect depends heavily on how people travel, how reliable the journey is, and how much control they have over it. Active commuters, those who walk or cycle, tend to report more positive commutes than drivers or public transport users, even over comparable times. Unpredictable journeys, with delays you cannot plan around, are consistently worse than long but reliable ones. Two cautions belong here. First, almost all of this evidence is observational. People who commute long distances differ from those who do not in many ways: income, housing, family circumstances, the kind of work or study they do. Researchers try to adjust for these, but no adjustment is perfect. Second, most of it concerns working adults, not students. We should not assume the numbers transfer directly. What does transfer, I think, is the shape of the finding: that long travel costs more than people predict, that unreliability and lack of control are especially corrosive, and that how you travel matters alongside how far. Why the cost is underestimated The commuting paradox has several plausible explanations, and each one points to something a student can do. The first is that people are poor at predicting how much they will mind a repeated minor unpleasantness. When choosing where to live, a forty-minute bus ride looks like a small price for cheaper rent or staying at home. The rent saving is concrete and visible every month; the commute is a vague future inconvenience. But the commute is paid every single day, twice, and small daily costs accumulate in a way that single large costs do not. Students who choose to live at home are often making a sound financial decision. The mistake is not the decision itself but assuming it comes without a health bill. The second is that the commute displaces other things without anyone deciding that it should. Nobody sits down and chooses to give up exercise, cooking or sleep to make room for travel. Those things simply get squeezed because the commute is fixed and they are not. Research on time use among commuters repeatedly shows this displacement: longer commutes go with less time spent on physical activity, food preparation and sleep. The losses are small on any given day and easy to overlook. The third is that the commute changes the edges of the day more than the middle. A student with an eight o'clock lecture and a seventy-minute commute has to leave by around a quarter to seven, which means waking around six, which means that a reasonable seven or eight hours of sleep requires going to bed between ten and eleven. For many young adults, whose body clocks naturally run later than those of older adults, that bedtime is hard to reach. The commute has not just taken seventy minutes; it has shifted the entire day earlier, against the grain of their biology. The fourth is that the costs of commuting are partly social and emotional, and those are the hardest to see in yourself. It is easy to notice that you are tired. It is harder to notice that you have stopped staying for the evening talk, that you do not know many people in your programme, that you feel faintly like a visitor on your own campus. Those effects are real, and later chapters return to them. The commuting student's particular position A commuting worker and a commuting student face the same traffic, but not the same day. Several features make the student's situation distinctive. There is no base. A worker usually arrives at a desk, a locker, a staff room, a fridge and a microwave. Many commuting students arrive at a campus where they have no fixed place of their own. Residential students can go back to their rooms between classes to eat, rest, change or drop books. Commuters carry everything all day. This single fact explains much of what follows in this book: why packed food matters so much, why food safety becomes a real issue, why back and shoulder strain from heavy bags is common, and why long gaps between classes are so draining. The day is fragmented. Timetables are rarely built around commuters. A student may have a class at nine, another at one, and a third at four, with gaps that are too short to go home and too long to use well. The day stretches from early morning to evening even if the actual teaching is only four hours. Workers generally have a continuous day; students often have a patchwork one, and the patchwork is tiring in its own way. The schedule changes every term. Workers usually settle into a routine. Students get a new timetable every semester, sometimes with different days on campus, different start times and different gaps. Every change means rebuilding the system for sleep, food and movement. This is one reason why habits that seemed solid in one term collapse in the next. Money is tight. Commuting students often choose to commute precisely because they cannot afford to live closer. That means the obvious fixes, like buying a proper lunch on campus every day or paying for a gym near the university, may not be available. The advice in this book is built with that constraint in mind. Many are carrying other roles. The literature on commuter students, going back to Barbara Jacoby's work on institutional responses to commuting students in the late 1980s, has consistently noted that they are more likely to have jobs, family responsibilities and, often, a life centred on their home community rather than the campus. This is not a deficit. It does mean that the commuting student's time is more contested, and that advice which assumes a free evening is useless. The four frictions If we look closely at how a commuting day goes wrong, the problems cluster into four areas. It is worth naming them here because the rest of the book is organised around them. The first friction is sleep. Early departures and late returns compress the night. Travel itself does not produce much restful sleep, even when students doze on a train. Most of the fatigue that commuting students attribute to "the commute" is in fact accumulated sleep debt that the commute has created. The second is fuel. A day that runs from six in the morning to seven at night with no kitchen and no base is a day in which eating well requires deliberate preparation. Without it, students skip meals, rely on whatever is sold near the station or on campus, and experience the energy swings and poor concentration that come from long gaps followed by large, poorly balanced meals. The third is stillness. Commuting adds an hour or more of sitting to days already full of sitting in lectures and libraries. Beyond the question of how much exercise someone gets, long unbroken sitting has effects of its own on metabolism, on musculoskeletal comfort and on alertness. The fourth is load: the mental and emotional weight of a day with no place to rest, of unreliable journeys, of crowds and noise, of the sense of always being in transit. This friction is the least tangible and the easiest to dismiss, but it shapes the others. A student who feels depleted by the journey is less likely to prepare food the night before or to walk the extra stop. These frictions interact. Short sleep increases appetite for sugary, high-fat food and makes planning harder. Poor fuelling makes the afternoon slump worse and the evening commute feel longer. Long sitting worsens the drowsiness that short sleep produces. Mental load makes all of it harder to fix. The interactions are why piecemeal tips so often fail: a student who tries to fix lunch without fixing sleep will find the lunch plan collapsing by Wednesday. Measuring your real commute Before changing anything, it helps to know what you are dealing with. The number you give when someone asks how long your commute is will not do. What you need is a picture of the door-to-door day, and the simplest way to get it is a one-week audit. For five ordinary days, write down a few plain facts in a notebook or a phone note. What time did you wake? What time did you leave home, and what time did you arrive at the place you actually needed to be, whether that was the lecture hall or the library? How long did you wait at stops, platforms or in traffic, roughly? What did you eat and drink, and when? When did you last stand up and walk for more than five minutes? What time did you leave campus, arrive home, and fall asleep? And at two points in the day, perhaps mid-morning and mid-afternoon, how alert did you feel on a simple scale from one to five? Most students who do this find three things. The first is that the true door-to-door time is longer than their estimate, often by a third or more, once the waiting and walking are counted. The second is that the variation matters more than the average: the day the train was cancelled, or the car park was full, costs far more than its extra minutes, because it throws off everything that follows. The third is that the worst moments of the day, the ones scored one or two for alertness, cluster at predictable times and are usually linked to something other than travel: a missed meal, a short night, three hours without moving. The audit is not a diary to keep forever. A week is enough. Its purpose is to replace a vague sense that "the commute is exhausting" with specific, fixable observations: I have not eaten between seven and two on Tuesdays and Thursdays; I sleep under six hours on the nights before my early lecture; I sit for nearly five hours without a break on Wednesday afternoons. Those are problems with answers, and the chapters that follow supply them. The audit also helps with a decision that many commuting students face at some point: whether the commute is sustainable at all. Occasionally the honest answer is no. A student who is travelling three hours a day, working twenty hours a week and consistently sleeping under six hours is not going to fix that with a better lunchbox. For that student, the right conversation may be with an academic adviser about a lighter course load, with the financial aid office about housing support, or with an employer about hours. The audit gives that conversation facts to rest on. For most students, though, the commute itself is workable once the day around it is rebuilt. What changes when you see the whole day Consider a first-year engineering student who drives fifty minutes each way from his parents' house. In the first month he feels fine. By the sixth week he is irritable, sleeping through his alarm on two days a week, and eating a large fast-food meal at a drive-through on the way home most evenings because he is starving by then. He assumes the problem is the drive. He considers moving closer but cannot afford it. When he maps out an ordinary Tuesday, the drive turns out to be the least of it. He wakes at six-fifteen, skips breakfast because there is no time, drinks a large coffee in the car, buys a pastry between classes, has nothing between noon and six because his lab runs through lunch, and gets home at seven having eaten perhaps a third of what he needs. He has sat for about ten hours. He goes to bed after midnight because the evening is the only time that feels like his own. None of these problems is the drive itself. All of them follow from the way the drive shapes the day. His solutions are modest. He moves his bedtime earlier by degrees, as Chapter 3 describes. He eats a real breakfast, even a simple one. He packs a lunch and two substantial snacks, including one to eat before the drive home so he is not driving hungry. He parks in the far car park and walks eight minutes each way. None of this requires moving house. Within a few weeks the drive feels shorter, though its length has not changed. That is the pattern this book is built on. The commute is fixed; the day around it is not. Understanding the commute's real cost does not require you to stop commuting. It requires you to take the commute seriously enough to plan for it. The next chapter begins with the complaint every commuting student knows best: the tiredness. Chapter 2: What Transit Fatigue Really Is "The commute wipes me out" is one of the most common things commuting students say, and it is almost always true. But it is also imprecise in a way that matters. Tiredness after travel is not a single condition with a single cause. It is at least six different things that happen to feel similar from the inside. A student who feels drained after a train journey might be short of sleep, might be running low on food, might be experiencing the peculiar drowsiness of sitting still with nothing to do, might be overloaded by noise and crowds, might be mildly motion sick, or might simply be mentally exhausted by a day of small decisions and minor frustrations. Usually it is several of these at once. The distinction matters because the remedies are different. Coffee helps with some kinds of fatigue and makes others worse. A nap on the train helps with sleep debt but not with sensory overload. Reading on the bus is restful for some people and nauseating for others. This chapter takes the tiredness apart so that the solutions in later chapters can be aimed accurately. Six sources of commuter tiredness Sleep debt. The first and largest contributor is simply not enough sleep. Early departures push wake times earlier, while evenings, social life and study pull bedtimes later. The result is chronic partial sleep restriction: not a single all-nighter, but night after night of six hours or less. The effects of this kind of restriction are well documented. In laboratory studies, people restricted to around six hours a night for two weeks show steadily worsening performance on attention tasks, eventually reaching levels comparable to those seen after a full night without sleep, while their own sense of how sleepy they are levels off. In other words, they get worse but stop noticing. For commuting students this is the key trap: after a few weeks of short nights, the tiredness feels normal, and it gets blamed on the journey rather than on the nights. Circadian mistiming. Separate from how much you sleep is when you sleep. The body's internal clock regulates alertness across the twenty-four hours, and in adolescents and young adults it tends to run later than in older adults. A student whose body would naturally sleep from one in the morning to nine is going to feel poor at seven, even after seven hours in bed, because they are awake during their biological night. Early commutes force exactly this. The effect is strongest on Monday mornings after weekends in which the student slept to their natural rhythm, a pattern sometimes described as social jetlag. Passive fatigue and underload. A curious finding from research on drivers is that monotony itself produces tiredness. Researchers studying driver fatigue, notably Paula Desmond and Peter Hancock, distinguished between active fatigue, caused by sustained demanding work, and passive fatigue, caused by long periods of monitoring with little to do. Long, uneventful drives on straight roads are the classic example. The same principle applies, in a milder form, to sitting on a train or bus with nothing engaging to attend to: the combination of low stimulation, warmth, steady vibration and a seated posture pulls the body toward sleep. This is why so many people doze on public transport even when they are not especially sleep-deprived, and why it is dangerous for drivers. Sensory and social overload. At the other extreme, a packed rush-hour bus or subway car is loud, bright, crowded, often hot, and full of strangers standing too close. Standing for long periods, holding on as the vehicle lurches, managing bags and personal space, listening for announcements: all of this demands low-level vigilance for the entire journey. Many people arrive at campus already depleted in a way that has nothing to do with sleep and that coffee does not touch. Students who are sensitive to noise or crowds, including many with anxiety or with sensory processing differences, feel this particularly strongly. Motion sickness. Some students feel queasy, headachy or simply washed out after travel without ever recognising it as motion sickness, because they are not actually vomiting. The mainstream explanation, the sensory conflict theory set out by James Reason and J. J. Brand in the 1970s, holds that motion sickness arises when the signals from the eyes, the balance organs of the inner ear and the body's sense of position disagree. Reading on a moving bus is the textbook case: the eyes report a still page while the inner ear reports acceleration and turning. Low-grade motion sickness produces drowsiness, headache, yawning and a general sense of malaise, a cluster sometimes called the sopite syndrome, which can last for some time after the journey ends. Decision fatigue and cognitive load. Finally, there is the steady drain of small decisions and minor frustrations: checking whether the train is on time, deciding whether to wait for the next bus, finding a seat, planning where to eat, working out where to spend a three-hour gap. The popular idea that willpower is a finite resource that runs out has been challenged by later research, and the original studies have not replicated reliably, so I do not rely on it. But the everyday observation that a day full of small unresolved choices feels tiring is widely shared, and the practical point holds regardless of the mechanism: decisions made in advance, once, are easier than decisions made on the platform, repeatedly. How the sources differ by mode of travel The mix of these sources depends heavily on how you get to campus. Table 1 sets out the typical pattern for the main modes. It is a guide to where to look, not a precise measurement; any individual journey can depart from it. Table 1. Main sources of commuter fatigue by mode of travel. Mode Most prominent fatigue sources Main risk First things to try Driving Sleep debt, passive fatigue, traffic stress Drowsy driving Protect sleep; never drive drowsy; short breaks on long drives Bus or tram Motion sickness, crowding, unreliability Arriving drained or nauseated Face forward; look outside; plan for delays Train or subway Crowding, passive drowsiness, sleep debt Dozing and missing stops; overload Use time deliberately; set an alarm if napping Cycling Physical effort, traffic vigilance, weather Collisions; arriving hungry Eat beforehand; safe routes and visibility Walking Physical effort, weather, time cost Heavy bags; poor footwear Lighten the load; good shoes; rain gear Two points stand out. The first is that active commuters face a different kind of tiredness from passive ones. A cyclist arriving after thirty minutes of pedalling is physically tired but usually mentally alert; the effort has woken them up. A train passenger arriving after thirty minutes of sitting may be physically fresh but mentally dull. This is part of why, in the wellbeing research described in Chapter 1, active commuters tend to report more positive journeys. The second is that driving stands apart because its fatigue carries a direct safety risk to the driver and to others. For passengers, dozing is an inconvenience. For drivers, it can be fatal. Chapter 3 returns to this in detail. Reading your own fatigue Because the sources feel similar, it is worth learning to tell them apart in yourself. A few questions help. When does it start? Tiredness that is present before you leave home, and on the days when you do not travel, points to sleep debt or circadian mistiming. Tiredness that appears during the journey and lifts within half an hour of arrival suggests passive fatigue or overload. Tiredness with a queasy or headachy quality that starts after reading or looking at a phone on a moving vehicle suggests motion sickness. What does coffee do? Caffeine is fairly effective against sleepiness caused by sleep debt and passive fatigue, at least in the short term. It does little for sensory overload and can make the jittery, overstimulated quality of it worse. If a strong coffee makes your journey feel better, sleepiness was probably the problem. If it makes you feel more frazzled, the problem was probably load. What does movement do? A brisk five-minute walk on arrival typically clears passive drowsiness quickly. It does less for true sleep debt, which returns within the hour. If you feel fine after walking from the station but crash again by mid-morning, the underlying problem is the night, not the journey. What does food do? If the tiredness improves markedly after eating, and especially if it came with irritability, difficulty concentrating or a shaky feeling, you had gone too long without food. This is extremely common among commuting students who leave early without breakfast, and it is easily mistaken for travel fatigue. Does it follow a weekly pattern? Fatigue that is worst on Monday and improves through the week suggests social jetlag from shifting sleep at weekends. Fatigue that builds from Monday to Friday suggests accumulating sleep debt that is only partly repaid at the weekend. Consider a third-year history student who takes two buses, about seventy minutes each way. She describes herself as "exhausted by the commute." When she pays attention for a week, she notices that on the one day she gets a seat on the first bus and looks out of the window instead of reading, she arrives feeling fine. On the other days, standing in a crowd and reading lecture notes on her phone, she arrives with a mild headache and a heavy, dull feeling that lasts most of the morning. Her problem is not primarily sleep. It is a combination of low-grade motion sickness and overload. Her solutions are different from those of a sleep-deprived student: she catches an earlier, emptier bus on two days a week, stops reading on the second, twistier leg, and listens to recorded lectures or podcasts instead. The commute has not become shorter, but it has stopped costing her the morning. Practical responses to each source Later chapters deal with sleep, food and movement at length. For the other sources, a few practical measures cover most of what helps. For motion sickness, sit facing forward and, on buses, near the front, where movement is smoother and the view of the road ahead is best. Look at the horizon or out of the window rather than at a page or screen, particularly on winding stretches. If you want to use travel time for learning, audio is far better tolerated than reading. Fresh air helps some people. Avoid travelling on a completely empty stomach or immediately after a heavy, greasy meal. Some people find that motion sickness decreases over weeks of regular travel as the brain adapts. If it remains a significant problem, a pharmacist or doctor can advise on medications; several older options cause drowsiness, which matters if you need to be alert on arrival or if you drive. For crowding and overload, the most effective single step is often changing when you travel. Even fifteen minutes earlier or later can make the difference between standing crushed and sitting comfortably. If your timetable allows, travelling outside the peak on some days gives you a recovery rhythm across the week. Noise-cancelling headphones or simple earplugs substantially reduce the sensory burden for many people. It helps to give yourself a few minutes after arrival before your first demand: a short walk, a drink of water, a quiet seat, rather than rushing straight from the platform into a lecture. For unreliability, build the delay into the plan rather than treating it as an emergency. Know the alternative route. Know what you will do if you are going to be late: whom you will email, where the lecture recordings are. Carry a snack that can stand in for a meal if a delay pushes lunch back. The stress of unreliable travel comes as much from the sense of helplessness as from the lost time, and a pre-decided plan restores some control. For passive drowsiness, give the journey a purpose that engages you enough to stay awake if you need to be awake, or accept a short nap if you do not. Standing for part of a train journey, if it is safe and possible, can clear drowsiness. For drivers, the standard advice applies: if you feel drowsy, stop somewhere safe. Opening the window and turning up the radio do not reliably work. For decision load, move decisions out of the commute. Decide the night before what you are wearing, what you are eating, which bag you are taking, and what you will do in each gap between classes. This sounds trivial. It is one of the most consistently useful things commuting students report doing, because it turns a stream of small choices at the worst possible moments into a single short planning session at a calm one. When the tiredness is not about the commute There is one more possibility that commuting students should keep in mind, precisely because the commute is such a convenient explanation. Sometimes persistent tiredness has a medical cause, and blaming the journey delays finding it. Several conditions that are common in people of university age produce fatigue as a main symptom. Iron deficiency, with or without anaemia, is one of the most frequent, particularly among people who menstruate, people who eat little or no meat without planning their iron sources, and endurance athletes. Depression very often presents as tiredness, low motivation and poor concentration before it presents as sadness, and students frequently read these signs as laziness or as the cost of a hard schedule. An underactive thyroid can produce slow, heavy fatigue along with feeling cold and gaining weight. Obstructive sleep apnoea, in which breathing repeatedly stops during sleep, is not only a disease of older, heavier adults; a person who snores loudly, wakes unrefreshed despite enough hours in bed, or is told they stop breathing at night should mention it to a doctor. Lingering fatigue after a viral illness is also common, including after infections such as glandular fever (infectious mononucleosis), which is frequent in this age group. A few signals suggest the tiredness deserves a clinical conversation rather than another adjustment to your routine. The first is that it persists even when you have slept well for a week or more, for instance during a holiday when you are not commuting. The second is that it comes with other changes: breathlessness on stairs that you did not used to have, unusual paleness, weight change you did not intend, persistent low mood or loss of interest in things you used to enjoy, very heavy periods, or loud snoring with pauses. The third is that it is getting steadily worse rather than rising and falling with your timetable. None of this is a reason for alarm. Most commuter fatigue has the ordinary causes described earlier in this chapter. But the campus health service exists partly for this kind of question, a simple blood test can rule out the common causes, and it is better to find out than to spend a year assuming the bus is to blame. If low mood is part of the picture, the counselling service is an equally legitimate place to start. The underlying principle The common thread in all of this is that transit fatigue is less a property of the journey than a property of the fit between the journey and the person. The same seventy-minute bus ride can be restful for a student who has slept, eaten, found a seat and is listening to something they enjoy, and draining for one who has done none of those things. The distance is fixed. The fit is not. That is also why the largest single lever, for most commuting students, is not the commute at all. It is the sleep that comes before it. The next chapter deals with that directly. Chapter 3: Sleep Built Backwards from the Departure Time If you remember one practical idea from this book, it should be this one: for a commuting student, the sleep schedule has to be built backwards from the time you leave the house, not forwards from the time you happen to feel tired. Almost every other piece of advice in these pages works better when this is in place, and very little works well when it is not. The reason is simple arithmetic combined with an awkward fact about young adult biology. The arithmetic is that an early departure fixes the wake time, and the wake time fixes the latest bedtime compatible with enough sleep. The awkward fact is that many people in their late teens and twenties have body clocks that make falling asleep early genuinely difficult. The commuting student is caught between a fixed morning and a biology that resists the evening. This chapter is about managing that tension. How much sleep, and when The American Academy of Sleep Medicine and the Sleep Research Society, in a joint consensus statement published in 2015, recommended that adults sleep seven or more hours per night on a regular basis to promote optimal health, and noted that more than nine hours may be appropriate for young adults, people recovering from sleep debt, and people who are ill. Individual needs vary, but for most university students seven hours is a floor, not a target, and eight or so is closer to what many young adults need to feel and perform at their best. The research on chronic sleep restriction is sobering precisely because it looks so ordinary. In a well-known experiment published in 2003, Hans Van Dongen, David Dinges and colleagues restricted healthy adults to four, six or eight hours in bed per night for two weeks. The six-hour group showed steadily accumulating deficits in attention and working memory, until by the end of the fortnight their performance resembled that of people who had gone a night or more without any sleep. Their subjective sleepiness, however, rose only modestly and then flattened. They felt somewhat tired. They were, measurably, a great deal more impaired than they felt. For a commuting student, six hours a night is an easy place to end up. It is what happens when you wake at six to catch a seven o'clock train and go to bed at midnight because the evening is the only time you have. The study suggests that after a couple of weeks of this, you will be operating well below your capacity while believing that you are merely a bit tired. It also suggests why the commute takes the blame: the tiredness becomes most noticeable during the dull, passive stretch of the journey, so the journey seems to be causing it. Timing matters as much as quantity. During adolescence the body clock shifts later, and it gradually shifts back through the twenties. A typical nineteen-year-old finds it easier to fall asleep at midnight and wake at eight than to fall asleep at ten and wake at six, even though the total is the same. Early commutes run against this tendency. That does not make them impossible; it means the evening needs deliberate structure to pull the clock earlier, rather than relying on tiredness to arrive on time. Building the schedule backwards The method is straightforward. Start from the time you must leave home on your earliest regular day. Subtract the time you genuinely need to get ready, including eating breakfast, which Chapter 4 argues you should not skip. That gives your wake time. Subtract the amount of sleep you need, and then subtract a further twenty to thirty minutes, because almost nobody falls asleep the instant they lie down. That gives the time you need to be in bed with the light off. Take a student who must leave at six-fifty for an eight o'clock lecture. She needs forty minutes to wash, dress and eat, so she wakes at six-ten. She aims for seven and a half hours of sleep, which puts sleep onset at ten-forty, and lights out at about ten-fifteen. Written down, this is often a shock. Many students discover that the bedtime their commute demands is one they have not reached since secondary school. The next question is what to do with the days when you do not have to leave early. The research on circadian rhythms suggests that a consistent wake time across the week is one of the most effective ways to stabilise the body clock and make earlier sleep easier. Sleeping in until eleven on Saturday and Sunday lets the clock drift later, which makes Sunday night's sleep harder and Monday morning brutal: the social jetlag described in Chapter 2. You do not need to wake at six-ten every day of the week. But keeping weekend and late-start wake times within about an hour or two of your early-day wake time makes a large difference to how the early days feel. This is where timetables help or hurt. If you have any choice in your course scheduling, and many students have more than they use, consider consistency as a factor alongside convenience. A timetable with four starts at nine is often easier to live with than one with two starts at eight and two at eleven, even though the latter gives you two lie-ins, because it lets you settle into one rhythm. For some students, concentrating all teaching into three long days on campus and studying from home on the other two is the best option. For others, spreading the load is better. The audit described in Chapter 1 will tell you which kind of student you are. Moving bedtime earlier Knowing you need to be asleep by ten-forty does not make you sleepy at ten-forty. Moving an entrenched late schedule earlier takes some deliberate work, and the evidence on body clocks gives a clear direction. Move gradually. Shift your bedtime and wake time earlier by fifteen to thirty minutes every few days rather than trying to jump two hours in one night. Sudden large shifts produce a night of lying awake, which tends to discourage the whole attempt. Use light deliberately. Light is the strongest signal the body clock receives. Bright light in the morning pulls the clock earlier; bright light late in the evening pushes it later. For a commuting student, the morning journey is an opportunity: walking to the station in daylight, sitting by a window, standing on the platform rather than waiting inside. In winter, when mornings are dark, this matters even more, and getting outside during the brightest part of the day helps. In the evening, dimming lights over the last hour or so and reducing bright screen use helps the opposite way. The evidence on the specific effect of phone screens is more mixed than popular writing suggests, and part of the problem with late-night phone use is simply that it is engaging and keeps you up rather than the light itself. Either way, the practical advice is the same. Protect the last hour. The commuting student's evening is compressed: they get home late, need to eat, may have family obligations, and have studying to do. The temptation is to use every minute and then fall into bed. A short routine that signals the end of the day, even twenty minutes of something undemanding, makes sleep onset quicker and more reliable. It is worth setting an alarm for the start of this wind-down period rather than for bedtime itself. Stop studying at a fixed time. This is hard advice to follow in examination periods, and there will be exceptions. But studying until the moment of lights out usually leaves the mind racing. A fixed stopping time, with a quick note of what you will do tomorrow so you are not holding it in your head, tends to improve both sleep and the next day's work. Using caffeine without borrowing from tonight Caffeine is the commuting student's most widely used tool against fatigue, and it works. It blocks the action of adenosine, a chemical that builds up in the brain during wakefulness and produces the sense of sleep pressure. Used well, it is a reasonable way to sharpen a morning. Used badly, it quietly damages the sleep that the next morning depends on. The key facts are about timing. Caffeine's half-life in the body, the time it takes for the level to fall by half, averages around five hours in healthy adults but varies widely between individuals, from a couple of hours to considerably longer. It is prolonged by some medications, including some hormonal contraceptives, and during pregnancy. That means an afternoon coffee may still be substantially active at bedtime. A study by Christopher Drake and colleagues, published in 2013, gave participants a dose of caffeine equivalent to a large coffee at zero, three or six hours before their normal bedtime. Even the dose taken six hours before bed measurably reduced total sleep, by roughly an hour on objective measurement, although participants did not necessarily notice the effect themselves. For a commuting student who needs to be asleep by eleven, a six-hour window points to a practical cut-off somewhere around the middle of the afternoon, and earlier is better for people who find they are sensitive. The late-afternoon coffee on the way to the train home is precisely the one to reconsider. If you find yourself needing it to stay awake on the journey, that is a signal of sleep debt, not a caffeine deficiency. On quantity, the US Food and Drug Administration has cited 400 milligrams a day as an amount not generally associated with dangerous negative effects in healthy adults, while noting that sensitivity varies widely. Pregnant people are generally advised to keep intake substantially lower, and people with heart rhythm problems, anxiety disorders or certain other conditions may need to limit it further; this is a question for your own clinician. Chapter 6 discusses energy drinks and the practical differences between caffeine sources. Napping on the commute Many commuting students nap on trains and buses, and some feel guilty about it. There is no need. A short nap can partly offset sleep debt and improve alertness for a while afterwards. The research on naps suggests that brief ones, around ten to twenty minutes, tend to give the best balance: they improve alertness quickly without the grogginess, known as sleep inertia, that often follows longer naps in which you drop into deeper sleep. A study by Amber Brooks and Leon Lack comparing naps of different lengths found that a ten-minute nap produced prompt improvements in alertness and performance that lasted well afterwards, while longer naps initially produced more grogginess. A few practical points make commute naps work. Set a phone alarm with vibration, both to limit the nap and so you do not miss your stop. Nap on the outward journey rather than late on the return journey if the late nap makes it hard to sleep at night; for most people, a short nap before mid-afternoon has little effect on night-time sleep, but a long one at six in the evening may. Keep your bag secured, with straps around a leg or arm, and valuables out of sight. And recognise that a nap on a vibrating, noisy vehicle is lighter and less restorative than sleep in bed. It is a patch, not a replacement. Some students deliberately combine a small coffee with a short nap: drinking it just before dozing, so that the caffeine begins to take effect around the time they wake. There is some experimental support for this in studies of drivers, and it can be a sensible tool for an occasional hard morning. It is not a strategy for living. Drowsy driving Everything above about napping applies to passengers. For drivers, the rules are different, and this section deserves to be read carefully. Drowsy driving is dangerous in a way that most people underestimate. A frequently cited study by Ann Williamson and Anne-Marie Feyer, published in 2000, found that after seventeen to nineteen hours without sleep, people's performance on several tests was equivalent to or worse than that seen at a blood alcohol concentration of 0.05 per cent, and after longer periods it reached levels equivalent to 0.1 per cent. Chronic partial sleep restriction, the kind described earlier in this chapter, produces impairment of a similar kind. A driver who has slept six hours a night for a fortnight is not operating at full capacity, whatever they feel. Drowsy driving also produces a specific danger that alcohol does not: the microsleep, a brief lapse into sleep lasting a few seconds, during which the driver is effectively absent. At motorway speeds, a few seconds covers a long distance. Drivers frequently have microsleeps without being aware of them. The warning signs include repeated yawning, heavy eyelids, difficulty remembering the last few kilometres, drifting out of your lane or onto the rumble strip, missing exits, and a sense that your head is nodding. If you notice any of these, the only reliable response is to stop somewhere safe. Winding down the window, turning up the music and slapping your face do not reliably restore alertness. A short nap of fifteen to twenty minutes in a safe place, perhaps combined with a coffee, can help enough to get you home; continuing to drive in the hope that it will pass cannot. For a commuting student who drives, several practical rules follow. Do not drive home after an all-nighter, even a short distance. If you have been awake since early morning and are driving home after an evening class, treat that drive with particular care, and have a plan for what you will do if you are too tired: a friend's sofa, a campus quiet room, a family member who can collect you, public transport home and back for the car tomorrow. On days when you know you slept badly, consider taking public transport if you have the option. If you ever experience a crash, near miss or microsleep caused by tiredness, treat that as a serious warning about your sleep, and consider talking to a doctor, particularly if you also snore heavily or wake unrefreshed. If anyone is injured in a road collision, call the emergency number for your country: 911 in the United States, 112 across the European Union, 999 in the United Kingdom, and other numbers elsewhere. When sleep problems need help Most commuting students' sleep problems are schedule problems, and the measures in this chapter will address them. But some are not. If you regularly take more than half an hour to fall asleep, wake for long periods in the night, or wake too early and cannot return to sleep, for at least three nights a week over a period of months, you may have insomnia, which is common and treatable. The recommended first-line treatment for chronic insomnia in adults is a structured psychological approach called cognitive behavioural therapy for insomnia, rather than sleeping medication, and many university health and counselling services either offer it or can point you to it, including through validated digital programmes. Loud snoring with pauses in breathing, persistent unrefreshing sleep, or overwhelming daytime sleepiness despite adequate time in bed are also reasons to see a clinician. For everyone else, the core of this chapter is short. Fix your wake time from your departure time. Protect a bedtime that gives you at least seven hours. Keep weekend wake times close. Use morning light and evening dimness to help your clock along. Stop caffeine by mid-afternoon. Nap briefly if you are a passenger; stop the car if you are a driver. Once sleep is in place, the rest of the commuting day becomes far easier to fix, beginning with what you eat. Chapter 4: Eating for a Twelve-Hour Day The residential student who skips breakfast can grab something from the dining hall at ten. The one who misses lunch can go back to their room at two and make toast. The commuting student has neither option. If they leave home at seven and return at seven, whatever they eat in those twelve hours is either carried with them or bought on the way, and the second option is expensive, often poor, and dependent on being in the right place at the right time. That makes food the area where planning pays off most visibly for commuters. It is also the area where students most often give up, because the usual advice about packed lunches assumes a single meal in the middle of an ordinary working day. The commuting student's problem is different. It is not lunch. It is the whole day. This chapter is about what to eat. The next deals with how to carry it safely and efficiently. The shape of the commuting day Think of the commuting day in terms of fuelling points rather than meals. A typical long day has five: breakfast before leaving, a mid-morning point, a midday point, a mid-afternoon point, and a point just before the journey home. Not every one needs a full meal. But each one is a moment when, without food, energy and concentration tend to fall and the next decision about food becomes worse. The last of these, the pre-journey snack, is the one most students miss and the one that makes the biggest difference to the evening. A student who leaves campus at five-thirty having last eaten at twelve arrives home at seven ravenous. That is the moment when the fast-food stop, the entire packet of biscuits and the enormous late dinner happen, which then interferes with sleep, which then makes the next morning harder. A modest snack before the journey breaks that chain. For drivers, it also matters for safety: hunger compounds tiredness and irritability behind the wheel. The goal across the day is not to eat constantly. It is to avoid long gaps, roughly anything over four or five hours while you are awake and active, and to make sure each eating point contains enough substance to carry you to the next. What "substance" means The Harvard T.H. Chan School of Public Health's Healthy Eating Plate is a useful, evidence-based guide that translates easily into a lunchbox. Its basic proportions are about half the plate vegetables and fruit, about a quarter whole grains, and about a quarter healthy protein, with healthy oils in moderation and water as the main drink. Applied to a commuting day, these proportions do not have to appear in every container; they can be spread across the day's food. Within that frame, a few components deserve particular attention for commuters because of what they do for satiety and steady energy. Protein is the most satiating of the macronutrients, meaning that for a given amount of energy it tends to keep people feeling full longer. It also supports the maintenance of muscle, which matters more than most students think if they are sitting for much of the day. Good portable sources include eggs, beans, lentils and chickpeas, tofu, tinned fish, cooked chicken, yoghurt, cheese, nuts and seeds. A lunch that is mostly bread with a thin layer of filling, or a bowl of plain pasta, will usually leave you hungry within a couple of hours; adding a substantial protein source changes that. Fibre slows digestion and helps smooth the rise in blood sugar after a meal. It comes from whole grains, beans and lentils, vegetables, fruit, nuts and seeds. Most adults in Western countries eat considerably less fibre than dietary guidelines recommend, and students living on white bread, instant noodles and pastries are typically well short. Whole grains rather than refined ones provide fibre and a slower release of energy. Brown rice, whole-wheat pasta, oats, wholegrain bread, barley and quinoa all travel well. Fats from sources like nuts, seeds, olive oil, avocado and oily fish help with satiety and make food more satisfying. There is no need to avoid them; a lunch with no fat at all is often unsatisfying and leads to snacking later. Fruit and vegetables add bulk, fibre, water and a wide range of micronutrients. Many are easy to carry. Whole fruit like apples, bananas, satsumas and pears needs no container at all. The practical test for any packed meal is a single question: will this carry me for three or four hours? A salad of leaves and a few tomatoes will not, however healthy it looks. The same salad with a cup of chickpeas, some feta, a handful of seeds and a slice of wholegrain bread will. Building the packed day Table 2 sets out how the fuelling points of a long commuting day might be filled. It is a framework, not a menu, and the examples are just that; the right choices depend on taste, culture, budget, dietary needs and what you have in your kitchen. Table 2. A framework for fuelling a long commuting day. Fuelling point Purpose What it needs Examples Breakfast at home Break the overnight fast before travel Protein plus whole grain or fruit Oats with milk and nuts; eggs on wholegrain toast Mid-morning Bridge to lunch after an early start Small, portable, some protein Yoghurt; nuts and fruit; cheese and crackers Midday meal Main meal of the campus day Protein, whole grain, vegetables Grain bowl with beans; wrap with chicken and salad Mid-afternoon Hold concentration through late classes Fibre plus protein or fat Hummus and carrots; peanut butter sandwich Before the journey home Prevent arriving home ravenous Modest and filling Banana and nuts; wholegrain bar; leftover half of lunch Several patterns in this table are worth drawing out. Breakfast is not optional for commuters. The research on breakfast and health outcomes is genuinely mixed; for adults with flexible days, skipping it is not clearly harmful, and some people do well without it. But the commuting student's situation is specific. Leaving home at seven and not eating until noon means five or more hours of travel, walking and concentration on an empty stomach, and it guarantees a large, hungry decision at a campus food outlet. If you cannot face food early, something small and easy counts: a glass of milk and a banana, a yoghurt, a slice of toast with peanut butter. Or pack it and eat it on the train in the first half hour. What matters is that the first fuelling point exists. The midday meal does not have to be large. Many students find that a moderate lunch plus substantial snacks works better than a big lunch, because a large meal in the early afternoon tends to deepen the natural post-lunch dip in alertness, which is part of our circadian pattern regardless of food. If your hardest class is at two, a lighter lunch and a mid-afternoon snack may serve you better. Leftovers are the most efficient packed food there is. Cooking a slightly larger dinner and packing the extra portion for tomorrow's lunch costs almost no additional time. Many of the most practical commuter meals are simply yesterday's dinner in a box. Splitting a meal works. A student with a long, fragmented day can pack one substantial meal and eat half at noon and half at four. This removes the need to plan two separate items. Staying on a student budget Money is the constraint most commuting students name first, and the good news is that packing food is almost always cheaper than buying it. The challenge is doing so without the monotony that makes people give up. The cheapest nutritious foods are well known and have been for a long time: dried or tinned beans and lentils, oats, rice and pasta bought in larger packets, eggs, frozen vegetables, seasonal and less glamorous fresh vegetables like cabbage, carrots and onions, bananas and apples, tinned fish, peanut butter, and plain yoghurt bought in large tubs and portioned out. Frozen vegetables are nutritionally comparable to fresh and do not spoil. Tinned tomatoes, spices, stock cubes and a good sauce or two turn these basics into food you want to eat. A few habits stretch money further. Plan the week's lunches before you shop, so you buy what you will use. Buy snacks in bulk and portion them yourself rather than buying individual packs: a large bag of nuts divided into small containers costs a fraction of the equivalent in single-serving packets. Keep a small emergency stock at home of things that make a lunch in two minutes: a tin of beans, a pouch of pre-cooked rice, a jar of peanut butter, some crackers. And recognise that the most expensive lunch is the one you bought because you forgot to pack one. Some students experience genuine food insecurity, not simply a tight budget. If you regularly cannot afford enough food, or are skipping meals because of cost rather than time, that is not a personal failing and it is more common among students than most people realise. Many universities now run food pantries, emergency grants, meal vouchers or subsidised meal schemes, and these are usually open to commuting students as much as residential ones. In the United States, some students are eligible for federal food assistance, although student eligibility rules are complicated and change; the financial aid office or dean of students is the place to ask. Arrangements in other countries differ, but the student services office is the right starting point everywhere. Making it sustainable The difference between a student who packs good food for three weeks and one who does it for three years is rarely knowledge. It is usually system. Several habits make the difference. Pick a small repertoire. You do not need thirty lunch ideas. You need four or five that you like, that travel well, and that you can make without thinking. Rotate them. Add something new occasionally when you are bored. Cook once, eat several times. A large pot of lentil soup, a tray of roasted vegetables and chickpeas, a batch of rice and a pan of chilli made on Sunday can supply the base of four or five lunches with different additions. Chapter 5 covers how long cooked food keeps safely. Pack the night before. The morning of an early commute is the worst possible time to assemble a lunch. Packing in the evening, ideally as part of the post-dinner routine, moves that decision to a calmer moment. If you share a kitchen, a labelled shelf in the fridge helps. Keep the kit ready. A clean lunchbox, a set of containers with matching lids, an insulated bag and ice packs, all in one place, lower the friction enough to make packing automatic. The next chapter goes into this in detail. Make it enjoyable. Food that you look forward to gets eaten; food you pack out of duty gets left in the bag while you buy chips. Use sauces, spices, pickles and seasonings. Pack foods from your own culture and family cooking; they often travel very well and are frequently more nutritious than the Western lunch staples. A cold lunch is not a punishment. A week in practice It helps to see how this works across a real week rather than a single day. Consider a second-year psychology student who takes a train and a bus to campus four days a week, with long days on Monday and Wednesday and shorter ones on Tuesday and Thursday. She works a Saturday shift and has about ninety minutes on Sunday afternoon to prepare food. Her budget is tight. On Sunday she cooks two things at once. A large pot of red lentil and vegetable soup, made with onions, carrots, a tin of tomatoes, stock and spices, simmers on one burner. A tray of chickpeas, sweet potato and peppers roasts in the oven. While they cook, she boils six eggs and makes a pan of brown rice. When everything has cooled quickly, as the next chapter explains, she portions the soup into three insulated-flask-sized servings, freezing one, and puts the roast vegetables, rice and eggs into separate containers in the fridge. She also divides a large bag of mixed nuts into small tubs and washes a bag of apples. On Monday, her longest day, she eats oats with milk and a spoonful of peanut butter before she leaves. She packs a grain bowl of rice, roast vegetables, chickpeas and a boiled egg with a small pot of yoghurt dressing; an apple and a tub of nuts for mid-morning; and two slices of wholegrain bread with cheese for the late afternoon, eaten on the platform before the train home. On Tuesday she takes a flask of hot soup and a bread roll, plus a banana. On Wednesday she repeats Monday's bowl with a different sauce. On Thursday she takes soup again. On Friday, at home, she eats what is left. Nothing in this plan is sophisticated. It costs less than buying two campus lunches. It needs one cooking session and ten minutes each evening to pack. And it covers every fuelling point on her long days. When she gets bored after a few weeks, she swaps the lentil soup for a bean chilli and the roast vegetables for a pasta salad with tuna, keeping the same structure. The general lesson is that the structure matters more than the recipes. One batch-cooked base, one or two flexible components, portable protein, and ready-portioned snacks will carry most commuting students through most weeks. Eating with particular needs The framework in this chapter is general. Some students need more specific approaches. Students with diabetes need to coordinate eating, medication and activity, and the long unpredictable commuting day complicates that. Carrying fast-acting glucose for hypoglycaemia, and a snack that can cover a travel delay, is essential. The diabetes care team is the right source of an individual plan. Students who follow vegetarian or vegan diets can eat very well from packed food, since beans, lentils, tofu, nuts and seeds all travel well. Vegans in particular should make sure they have a reliable source of vitamin B12, typically a fortified food or supplement, and should think about iron, iodine, calcium and omega-3 fats; a dietitian can help. Students who fast for religious reasons, for example during Ramadan, face long days of travel and study without food or water during daylight hours. Planning the pre-dawn and evening meals for slow-releasing energy, protein and fluids, and reducing unnecessary exertion during the day, is the practical core. Many universities make arrangements such as quiet spaces or flexibility, and it is reasonable to ask. Students with a history of disordered eating may find detailed food planning either helpful or unhelpful, and the structure described here should be adapted rather than applied rigidly. If you notice that thinking about food, weight or eating is taking up a large amount of your attention, or that you are restricting, bingeing or compensating, the campus health or counselling service is a good first step, and specialist support is effective. The case of the student who "just isn't hungry" A final pattern deserves mention because it is common and easy to misread. Some commuting students say they are simply not hungry during the day. They eat little until evening, then eat heavily. They are not trying to diet; food just does not appeal on campus. Often, when this is examined, several factors lie behind it. Caffeine suppresses appetite for a while. Stress and rushing do the same. The food available on campus may be unappealing or unaffordable, which dulls interest. And habit plays a large role: a body used to not eating in the day stops signalling hunger strongly at those times. The signs that the pattern is costing something are the ones described in Chapter 2: afternoon irritability, difficulty concentrating, a slump in the late afternoon, and an evening of overeating. If this describes you, start small. Eat something modest at two of the five fuelling points, even without much appetite, and see how the afternoon feels after a fortnight. Most students who try it report that concentration improves and evening overeating decreases. Appetite tends to follow the pattern, not lead it. Knowing what to eat is half the problem. The other half is getting it from your kitchen to a lecture hall six hours later without it becoming unsafe or unappealing. That is the subject of the next chapter. Chapter 5: Carrying Food Safely Through a Long Day A packed lunch that sits in a warm bag for six hours is not the same food it was when you packed it. Most of the time the change is harmless and merely makes the food less pleasant: soggy bread, wilted leaves, a sandwich that tastes of its container. Occasionally it is not harmless. Bacteria that cause food poisoning grow quickly at the temperatures of a crowded bus, a heated lecture hall or a car parked in the sun, and a commuting student's food spends far longer at those temperatures than an office worker's lunch that goes straight into a fridge. This chapter deals with the practical business of getting food from your kitchen to your mouth, many hours later, in a condition that is safe and still worth eating. It is less glamorous than nutrition, but it is where many good intentions fail. The temperature rules Food safety authorities in many countries base their advice on the same basic principle: bacteria that cause foodborne illness multiply most rapidly within a middle band of temperatures, and the aim is to keep perishable food either cold enough or hot enough to stay out of that band for as long as it matters. The US Department of Agriculture calls this band the "danger zone" and defines it as between 40°F and 140°F, which is roughly 4°C to 60°C. Its guidance is that perishable food should not be left in this range for more than two hours in total, or more than one hour when the surrounding temperature is above 90°F (about 32°C), as it might be in a hot car or on a summer platform. Other food safety agencies use slightly different figures, typically keeping chilled food at or below about 5°C and hot food at or above about 63°C, but the principle is the same. "Perishable" here means foods that support bacterial growth: cooked meat, poultry and fish; cooked rice, pasta and other grains; dairy products such as milk, yoghurt and soft cheeses; eggs and egg dishes; cooked vegetables and beans; cut fruit and prepared salads; and anything made with mayonnaise or cream. Whole uncut fruit, bread, crackers, nuts, dry cereals, hard cheeses for a few hours, peanut butter and unopened shelf-stable products are much less of a concern. For a commuting student this means a simple rule of thumb. Any perishable food that will not be eaten within about two hours of leaving the fridge needs a way to stay cold or hot. Since most commuting students eat lunch four or five hours after leaving home, that applies to most packed lunches. Two further points matter. First, the two-hour limit is cumulative. Food that sat on the kitchen counter for forty minutes while you got ready has already used some of its allowance. Second, you cannot tell whether food is unsafe by smell, taste or appearance. The bacteria that cause food poisoning do not necessarily cause spoilage. A lunch that has been warm for five hours may taste perfectly fine. Keeping cold food cold An insulated lunch bag with ice packs is the basic tool, and it works well when used properly. The USDA recommends using at least two cold sources, such as two ice packs or one ice pack and a frozen drink, and placing perishable foods between them. A single small ice pack at the bottom of a large bag does very little. A few practical details make a difference: · Start cold. Pack food straight from the fridge and put it in the bag as late as possible. Food that is already cold stays cold far longer than food that has warmed on the counter. · Freeze what you can. A frozen bottle of water acts as an ice pack and becomes a cold drink by lunchtime. Some foods, such as a portion of frozen soup in a sealed container to be heated later, or a frozen sandwich made with suitable fillings, can go in frozen and thaw during the morning. Leafy salad, mayonnaise, raw vegetables and boiled eggs do not freeze well. · Choose the right bag. A well-insulated, zipped bag of the right size for its contents keeps temperatures steadier than a large, half-empty one. Soft insulated bags are easier to carry than hard coolers. · Keep it out of heat. Do not leave the bag in direct sun, on a radiator, or in a car in warm weather. A car interior can become very hot very quickly, well beyond what any lunch bag can resist. · Use a fridge if you can. If your department has a communal fridge, find out whether commuting students can use it. Many can but do not ask. Some universities provide fridges in commuter lounges. On arrival, the question to ask is whether the ice packs are still at least partly frozen at lunchtime. If they are, the food has probably stayed cold. If they are completely melted and warm, the food has been in the danger zone for some time, and perishable items are not safe to rely on. Keeping hot food hot A vacuum-insulated flask, sometimes called a food jar or thermos, lets you carry hot soup, stew, chilli, curry or porridge and eat it hot at lunchtime without a microwave. For commuting students who have no reliable access to a microwave, it is one of the most useful pieces of kit available. Flasks work well only if you use them correctly. Pre-heat the flask by filling it with boiling water and leaving it for several minutes, then pour that water away. Heat the food until it is steaming hot throughout; USDA guidance for reheating leftovers is to reach 165°F, or about 74°C. Fill the flask as full as possible, since air space lets heat escape, and seal it immediately. Do not open it until you are ready to eat. A good flask handled this way will keep food well above the danger zone for several hours. The product instructions will usually give an indication of how long. If you open it at lunchtime and the food is only lukewarm, it has probably been in the danger zone for too long, and it is safer to reheat it thoroughly if you can or not to eat it. Reheating on campus Many campuses have microwaves available to students, in cafeterias, student unions, commuter lounges or departmental kitchens. They are extremely useful, but they have two drawbacks. One is queues: at lunchtime there may be a line of students waiting, which eats into a short break. The other is that microwaves heat unevenly. Food that is scalding at the edges can be cold in the middle, and cold spots allow bacteria to survive. To reheat safely, cover the container loosely to hold in steam, stir halfway through, and let the food stand for a minute or two afterwards so the heat evens out. Food should be steaming hot throughout. Microwave only in containers labelled as microwave-safe; some plastics are not designed for heating, and ordinary takeaway containers may warp or leach. Glass containers with removable lids are a good choice for food you intend to reheat. Rice deserves a specific mention. Uncooked rice can contain spores of a bacterium called Bacillus cereus, which survive cooking. If cooked rice is left at room temperature, the spores can germinate and produce toxins that reheating does not destroy. The UK National Health Service advises cooling cooked rice quickly, refrigerating it promptly, eating it within a day of cooking, and not reheating it more than once. Rice is safe and very useful in packed lunches, but it should be treated as a perishable food, not a dry one. What travels well Not all foods are equally suited to a long day in a bag. Table 3 compares some common categories by how well they tolerate travel, what they need, and what tends to go wrong. It is a practical guide rather than a scientific classification. Table 3. How common packed foods tolerate a long commuting day. Food type Temperature needs Travels well? Common problem Tip Whole fruit, nuts, crackers None Very well Bruising; crushing Hard-sided container for soft fruit Sandwiches and wraps Cold if filling is perishable Moderately Soggy bread Pack wet fillings separately Grain and bean salads Cold Very well Dressing soaks in Add dressing at lunchtime Leafy salads Cold Poorly Wilting Pack leaves separately; sturdier greens Soups, stews, curries Hot or cold, not warm Well in a flask Leaks; lukewarm by noon Pre-heat flask; fill to the top Yoghurt, soft cheese Cold Moderately Warm by afternoon Eat early in the day A few general principles emerge. Sturdy food travels better than delicate food. Grain salads with beans, roasted vegetables, cabbage and carrot slaws, and hearty soups tolerate a long day far better than delicate leaves or cream-based dishes. Keep wet and dry apart. Most sogginess comes from dressing, sauce or juicy fillings soaking into bread or leaves over several hours. Small containers for dressings and sauces solve most of this. Eat the most perishable items first. If your day includes yoghurt and a bean salad, eat the yoghurt at mid-morning while it is cold and the salad at lunch. The kit and the routine The right equipment makes packing easier and more reliable. It need not be expensive. A basic commuter kit might include an insulated bag, two reusable ice packs, a food flask, three or four containers of different sizes with leak-proof lids, a couple of small pots for dressings and sauces, a reusable water bottle, and a fork and spoon that live in the bag. Some students add a small cloth napkin and a zipped pouch for rubbish. Leaks are one of the most common reasons people abandon packed food. A lid that fails and spreads soup over a laptop does more damage to the habit than any number of boring lunches. It is worth testing containers at home, upside down over the sink, before trusting them in a bag with books and electronics. Keeping food in a separate bag, or at least a separate compartment, from electronics is a sensible precaution. The routine matters as much as the kit. Wash containers when you get home, not the next morning. Put the ice packs back in the freezer as soon as you unpack. Pack the next day's food in the evening. These are small steps, but they are the difference between a system that runs almost automatically and one that depends on a burst of effort each morning. Batch cooking and storage Chapter 4 argued that cooking once and eating several times is the most efficient way to feed a long commuting week. That only works if the food is stored safely. Cooked food should be cooled quickly and refrigerated within two hours of cooking, and sooner is better. Large pots of soup or stew cool slowly in the centre; dividing them into shallow containers speeds this up considerably. It is safe to put warm food in the fridge; the old advice to wait until it is completely cold is not current guidance, although very large volumes of hot food can warm a small fridge and are better divided first. The USDA advises that most cooked leftovers kept in the fridge should be eaten within three to four days. That fits a week of commuting only if you plan for it: cook on Sunday for Monday to Wednesday, and either cook again midweek or freeze the portions for Thursday and Friday. Freezing is the commuting student's friend. A freezer drawer with a few portions of soup, chilli, curry or cooked grains means that a week in which you had no time to cook does not become a week of vending machine lunches. Frozen food should be thawed in the fridge overnight or in the microwave, not left on the counter. Label containers with the date they were cooked. It sounds fussy, but in a shared household fridge it is often the only way to know whether the curry at the back is from this week or last. When the day does not go to plan Food safety rules are easy to follow on an ordinary day. They are tested on the days when something changes: the train breaks down and you sit on a stalled carriage for two hours, a seminar is cancelled and you go home early with an uneaten lunch, you leave the lunch bag in a warm car all afternoon, or you stay on campus unexpectedly into the evening. A few decisions, made in advance, cover most of these situations. If your lunch has been warm for a long time, do not eat the perishable parts. A chicken wrap that has been in a hot bag since seven in the morning, with ice packs long melted, belongs in the bin, however it smells. The whole fruit, crackers and nuts in the same bag are fine. This is one reason to pack perishable and non-perishable items so that the non-perishable ones can stand alone as a small meal. Do not bring perishable food home to eat later if it has been out of the cold all day. It is tempting not to waste food, and throwing away a lunch you paid for and prepared feels bad. But food that has been in the danger zone for five or six hours does not become safe when you put it back in the fridge. If you often bring lunch home uneaten, the better fix is to pack less or to pack foods that do not need cold. Food that stayed cold can be kept. If your ice packs are still frozen when you unexpectedly go home early, the food has stayed cold and can go straight back in the fridge for tomorrow. Plan for long days in warm weather. On hot days, or on days when you will be out for twelve hours or more, lean toward foods from the first row of Table 3 and toward a flask of hot food rather than a cold perishable lunch, or eat the perishable items early and rely on sturdy, non-perishable foods for the afternoon. Consider a student who drives to campus and leaves her lunch bag in the car on a warm September day, meaning to collect it at lunchtime. A lab overruns and she does not get back to the car until four. The bag has two ice packs, but the car has been hot since mid-morning. She eats the apple and the oatcakes from the bag and her emergency stash, and throws away the pasta salad with tuna. It is annoying, but it is the right call. The following week she starts carrying the lunch bag into the building with her, and asks her department whether commuting students can use the staff kitchen fridge. The answer is yes. When food has gone wrong Most food poisoning is unpleasant but short-lived, and resolves without treatment beyond rest and fluids. The main risk is dehydration, particularly with repeated vomiting or diarrhoea. Drink small amounts of fluid often; oral rehydration solutions from a pharmacy are useful. Seek medical advice promptly if you cannot keep fluids down, if symptoms last more than a couple of days, if you see blood in your stool, if you have a high fever, if you show signs of dehydration such as very little urine, dizziness on standing or extreme thirst, or if you are pregnant, have a weakened immune system or a significant underlying illness. In an emergency, such as severe dehydration, confusion or collapse, call your country's emergency number. One more practical point: if you have been sick with vomiting or diarrhoea, particularly if it may have been caused by a virus such as norovirus, staying home until you have been free of symptoms for at least forty-eight hours is commonly advised, both for your recovery and to avoid spreading it on crowded transport and in shared campus spaces. That is a good reason to know in advance how you will catch up on missed teaching. Carrying food safely is not complicated. It comes down to a handful of habits: cold food cold, hot food hot, two cold sources, pre-heated flasks, shallow containers for cooling, three or four days for leftovers, and a freezer as backup. With those in place, the packed lunch stops being a gamble and becomes simply the way you eat. The next chapter turns to the things that fill the gaps: drinks, caffeine and the food you buy on the go. Chapter 6: Water, Caffeine and the Food You Grab However well you plan, some days go wrong. The train is cancelled and you spend an extra hour on a replacement bus. The lab overruns into lunch. You forgot the lunch bag on the kitchen table. On those days, what you drink and what you buy on the move decide whether the day merely wobbles or collapses. And on ordinary days, drinks are a larger part of the commuting student's intake than most people realise, from the morning coffee to the afternoon energy drink to the bottle of fizzy drink with a vending machine snack. This chapter covers the gaps: hydration, caffeine, sugary drinks, energy drinks, and how to make better choices from what is actually on sale in stations, convenience stores and campus machines. Staying hydrated when you are away all day Hydration advice attracts more myth than most areas of nutrition. The popular rule of eight glasses of water a day is not based on solid evidence, and much of the water people need comes from food and from other drinks, including tea and coffee. The European Food Safety Authority, in its reference values published in 2010, set adequate total water intakes for adults of about 2.0 litres a day for women and 2.5 litres for men, from all sources including food, under moderate conditions. Needs rise with heat, physical activity, illness and pregnancy. For most healthy people, thirst is a reasonable guide, and pale yellow urine is a reasonable sign that intake is adequate. The problem for commuting students is not that they need unusual amounts of water. It is that the conditions of a commuting day tend to suppress drinking. Some students limit fluids deliberately because toilets are not available on the bus or are unpleasant on the train. Some go hours in back-to-back classes without a break. Some simply do not carry water and do not want to buy it. Mild dehydration is associated in several studies with headache, fatigue, reduced concentration and worse mood, though the size of these effects varies and the research is not all of high quality. For a commuting student who already has several sources of fatigue, adding mild dehydration is an unnecessary burden. The practical solution is simple. Carry a reusable water bottle and refill it on campus. Drink a glass of water with breakfast before you leave. Drink during the day rather than all at once in the evening. If toilet access on your route is a genuine problem, drink more at the beginning of the campus day and after the journey, and less in the hour before a long stretch of travel, rather than cutting down overall. On hot days, and if you cycle or walk a long way, increase intake accordingly. Some students also find that the pattern of their drinking matters for sleep. A large volume of fluid late in the evening, especially if it includes caffeine or alcohol, is a common cause of waking in the night to use the toilet. Front-loading fluids earlier in the day helps with this too. Caffeine, revisited Chapter 3 dealt with caffeine's effects on sleep. Here the question is how to use it well during the commuting day. Caffeine is effective. It improves alertness, reaction time and attention, particularly when you are tired. It is also easy to overuse, because the amounts in common drinks vary so widely and because tolerance develops with regular use. A small filter coffee, a large coffee-shop coffee, a can of cola and a large energy drink can differ several-fold in caffeine content, and the exact figures depend on the product, the brew and the serving size. Reading labels, where caffeine content is given, is more useful than relying on general estimates. As mentioned earlier, the US Food and Drug Administration has cited 400 milligrams a day as an amount not generally associated with dangerous effects in healthy adults. That is a ceiling, not a recommendation. Many people feel best at considerably less, and sensitivity varies. Signs that you are having too much include jitteriness, anxiety, a racing heart, stomach upset, headaches and, most commonly, difficulty sleeping. Three patterns are worth watching for in commuting students. Caffeine as a substitute for breakfast. A large coffee on the way to the station suppresses appetite, provides a quick lift and costs no time. It is a very common replacement for breakfast. The result is often a mid-morning crash, when the caffeine wears off and the absence of food becomes obvious. Coffee with breakfast is fine. Coffee instead of breakfast tends to backfire. Escalating doses. Regular use leads to tolerance, so the same amount produces less effect, and people drift toward larger or more frequent drinks. If you find you need more caffeine to achieve the same alertness, that is usually a sign that the underlying problem, most often sleep debt, has grown. Withdrawal on non-commuting days. People who drink a lot of caffeine on weekdays and much less at weekends often get headaches, tiredness and irritability on Saturday. This is caffeine withdrawal, and it is common. Keeping intake more consistent across the week, at a moderate level, avoids it. If you want to cut back, doing so gradually over a week or two reduces withdrawal symptoms. Switching some drinks to decaffeinated or half-caffeinated versions is a simple way to do it. Energy drinks Energy drinks deserve particular attention because they are widely used by students and because they differ from coffee in ways that matter. Many energy drinks contain substantial amounts of caffeine, sometimes combined with other stimulants such as guarana, which is itself a source of caffeine, along with large amounts of sugar. Some cans are large enough that a single one contributes a large share of a day's sensible caffeine intake, and drinking two or three in a day can take a person well past the level at which side effects become common. The sugar content of full-sugar versions is often high. The World Health Organization recommends that free sugars make up less than 10 per cent of total daily energy intake, and suggests a further reduction to below 5 per cent would bring additional benefits. A single large full-sugar energy drink can take up much of that allowance. There are specific risks to be aware of. Energy drinks combined with alcohol are a particular concern: the stimulant can mask the feeling of drunkenness without reducing the actual impairment, which may lead people to drink more and take more risks. The US Centers for Disease Control and Prevention and other health authorities have warned against mixing them. People with heart conditions, high blood pressure, anxiety disorders, or who are pregnant should be especially cautious with high-caffeine products, and anyone who experiences chest pain, palpitations, fainting or severe anxiety after consuming them should seek medical advice promptly. In an emergency, call your country's emergency number. None of this means an occasional energy drink is dangerous for a healthy adult. It means that energy drinks are not an efficient solution to commuter fatigue. They deliver a large dose of stimulant and sugar that produces a sharp lift and often a sharp fall. If you find yourself relying on them to get through the commuting day, that is a strong signal to look at sleep and eating patterns first. Sugary drinks and the afternoon slump Sugar-sweetened drinks, including fizzy drinks, sweetened iced teas, many flavoured coffees and fruit drinks, are among the largest sources of free sugars in many people's diets. They also do little for fullness; people generally do not eat less food to compensate for the energy they drink. For commuting students, the pattern that matters most is the afternoon one. A student who has eaten little since breakfast, feels a slump at three, and buys a large sugary drink gets a quick burst of energy, often followed by a further slump an hour later, just in time for the journey home. A snack with some protein and fibre, and a glass of water, usually gives a steadier result. This is not a case for never drinking anything sweet. It is a case for knowing why you are reaching for it. If it is because you are hungry, food will help more. If it is because you are tired, the solutions are in Chapters 2 and 3. If it is simply because you enjoy it, then enjoy it, but not as your main source of fuel. Buying food on the move Every commuting student will sometimes buy food in a station, a convenience store, a campus shop or from a vending machine. The goal is not to avoid this entirely but to do it well when it happens. The principles from Chapter 4 still apply: look for protein and fibre, and ask whether what you are buying will carry you for a few hours. In most convenience settings, some choices consistently do better than others. Better choices typically include plain nuts and seeds, trail mix without heavy coatings, fresh fruit, yoghurt, cheese portions, boiled eggs where sold, sandwiches or wraps with a substantial protein filling and wholegrain bread, bean or grain salads, hummus with vegetable sticks or crackers, oatcakes, plain popcorn, and milk or plain soy drinks. Weaker choices, not forbidden but unlikely to carry you, include pastries, sweets and chocolate bars eaten on their own, crisps on their own, sugary cereal bars, and large sugary drinks. When the options are limited, combining helps. A packet of crisps on its own will not hold you; a packet of crisps with a pot of yoghurt and a piece of fruit is a reasonable stopgap. A chocolate bar with a bag of peanuts is better than two chocolate bars. The idea is to add protein and fibre to whatever you have. Reading nutrition labels quickly is a useful skill. On a pre-packed sandwich or salad, a glance at protein and fibre, as well as sugar and salt, tells you most of what you need. Many countries use front-of-pack labelling schemes, such as the UK's traffic-light system, that make this faster; the details differ between countries. Eating a bought meal well Sometimes the choice is not between a packed lunch and a vending machine but between a packed lunch and a proper bought meal in a campus cafeteria or food court. Many commuting students buy lunch on campus one or two days a week, for convenience, for company, or because they ran out of time to prepare. That is a perfectly reasonable part of a commuting life, and there are ways to make it work well. The Healthy Eating Plate proportions from Chapter 4 transfer directly to a cafeteria tray. Look for a meal that includes a clear source of protein, a portion of vegetables that is more than a garnish, and, where possible, a whole-grain or bean-based starch rather than only chips or white bread. Many campus food outlets now offer grain bowls, salad bars with protein options, soups with bread, and hot meals with vegetables; these usually beat pizza, fried food and pastries for staying power through the afternoon. A few habits help. Look at all the options before choosing, since the first thing you see is often the least balanced. If the main meal is light on vegetables, add a side salad or a piece of fruit. Watch portion size with very large, starchy meals if you have a demanding class soon after, because a heavy lunch deepens the early-afternoon dip. Choose water rather than a sugary drink by default. And if buying lunch is regularly straining your budget, that is a signal to shift one or two more days a week to packed food, not to skip meals. Buying lunch can also be one of the most social parts of the day. Eating with classmates is a way of building the connections on campus that Chapter 8 discusses, and for some commuting students that value is worth the cost on one day a week. There is no need to treat every bought lunch as a failure of planning. What matters is that it is a choice rather than an emergency. Consider two students who both spend about the same on food each week. One packs lunch four days and buys lunch with friends on Friday. The other packs nothing, buys a pastry and coffee every morning and a sandwich every afternoon, and ends up at a drive-through on the way home twice a week. The first has a steadier day, better food, and a social lunch to look forward to. The second has spent the same money on three rushed decisions a day. The difference is not willpower. It is that one of them decided in advance. The emergency stash The most effective single defence against bad gap-filling is to have something better with you already. A small emergency stash, kept permanently in your bag, car or locker, covers the days when plans fail. A good stash is non-perishable, compact, and filling enough to stand in for a small meal. Examples include a small bag of nuts or trail mix, a couple of oatcakes or wholegrain crackers, a sachet of nut butter, a wholegrain cereal bar with reasonable protein and fibre, and a small carton of long-life milk or soy drink. Dried fruit adds energy but is best combined with something else. Replace items as you use them, and check the stash every few weeks so it does not become stale. For drivers, a stash in the car also serves a safety purpose. Driving hungry, particularly late in a long day, adds to tiredness and irritability. A snack before a long drive home is part of safe driving, not just good nutrition. Consider a student who works an evening shift at a supermarket after classes twice a week and drives between campus and work. On those days, he used to eat nothing between a late lunch and a meal at ten at night. He now keeps a stash of nuts, crackers and a cereal bar in the glovebox, and eats a proper snack in the car park before his shift. His shifts feel shorter, he no longer buys the pastries that the supermarket's bakery offers at the end of the night, and he is less irritable on the drive home. The change cost almost nothing and took a single trip to the shops to set up. Alcohol and the commuting student Finally, a word about alcohol, because it intersects with commuting in specific ways. Commuting students often have to plan around whether they will drive home, which can be protective: many drive and therefore do not drink at campus events. Others use public transport and may drink and then face a late, long journey home. Both situations call for planning. If you drive, the only safe plan is not to drink before driving. Legal limits vary between countries and are sometimes zero for new drivers, and impairment begins below them. If you plan to drink at an evening event, arrange in advance to get home another way, or to stay somewhere nearby. If you travel home by public transport after drinking, plan the route before you start, know when the last service runs, keep your phone charged, travel with friends where possible, and avoid falling asleep alone on a late train or bus. Alcohol also harms sleep quality, even when it makes falling asleep easier, which matters for the next morning's early start. If someone is very drunk and cannot be woken, is breathing slowly or irregularly, is vomiting while not fully conscious, has a seizure, or has cold, clammy or bluish skin, these can be signs of alcohol poisoning. Call your country's emergency number, such as 911 in the United States, stay with the person, keep them on their side, and do not leave them to sleep it off. What you drink and grab in the gaps can support a well-planned day or undo it. With a water bottle, a sensible caffeine pattern, and an emergency stash, most of the damage is avoided. The next chapter turns to the other side of the commuting student's health: the problem of spending so much of the day sitting still. Chapter 7: Moving Through a Sitting Day Count the hours a commuting student spends sitting on an ordinary day. Breakfast at the table: fifteen minutes. The bus or train or car: an hour, perhaps two. Lectures and seminars: three or four hours. The library in the gaps: two or three more. The journey home. Dinner. An evening of study at a desk. For many students the total comfortably exceeds ten hours, and it may be the most sedentary period of their lives, more so than school, where there were at least breaks and physical education, and more so than many jobs. It is easy to assume that this is only a problem if you do not exercise. The evidence suggests something more interesting: long, unbroken sitting has effects of its own, separate from how much deliberate exercise a person does, and small amounts of movement spread through the day can offset a surprising share of them. For commuting students, whose evenings are short and whose days are long, that is good news. It means the solution does not depend on finding a free hour for the gym. Why sitting matters separately from exercise For a long time, physical activity research focused on exercise: how many minutes of moderate or vigorous activity a person did each week. More recently, researchers have paid attention to sedentary behaviour, meaning time spent sitting or lying with low energy expenditure while awake, as a separate factor. The current World Health Organization guidelines on physical activity and sedentary behaviour, published in 2020, recommend that adults do at least 150 to 300 minutes of moderate-intensity aerobic activity per week, or 75 to 150 minutes of vigorous activity, or an equivalent combination, along with muscle-strengthening activities on two or more days a week. They also recommend that adults limit the amount of time spent being sedentary, and state that replacing sedentary time with physical activity of any intensity, including light intensity, provides health benefits. The most informative study on how sitting and exercise interact is a large harmonised meta-analysis led by Ulf Ekelund and published in The Lancet in 2016. It pooled data on more than a million adults. People who sat for many hours a day and were physically inactive had a higher risk of death during follow-up than those who sat less. But among those who did high levels of moderate activity, around sixty to seventy-five minutes a day, high sitting time was no longer associated with higher mortality. The authors concluded that this level of activity seemed to eliminate the increased risk associated with high sitting time, though it only reduced, rather than eliminated, the risk associated with very high amounts of television viewing. Two lessons follow. The first is that sitting and inactivity together are the problem; a person who sits a great deal and also moves a great deal is in a different position from one who only sits. The second is that the amount of activity needed to fully offset very long sitting is substantial, more than most students do. That makes reducing and breaking up sitting, not only adding exercise, a sensible part of the strategy. There is also evidence at a shorter timescale. In a laboratory study published in 2012, David Dunstan and colleagues had overweight and obese adults either sit continuously for several hours after a sugary drink or interrupt their sitting every twenty minutes with two minutes of light or moderate-intensity walking. The interruptions substantially lowered the rise in blood sugar and insulin after the drink. The participants were not students, and one study cannot tell us everything, but it supports a simple and practical idea: frequent short breaks in sitting do something useful. For commuting students, there is a more immediate reason as well. Long sitting worsens the passive drowsiness described in Chapter 2, stiffens the back, neck and hips, and makes the afternoon slump deeper. A few minutes of walking between classes is one of the most reliable ways to restore alertness. Active commuting The most efficient way to add movement to a commuting day is to make part of the commute itself active. A student who walks or cycles all or part of the way to campus accumulates activity without needing any extra time, because the travel had to happen anyway. The evidence that active commuting is associated with better health is reasonably consistent, though again largely observational. A meta-analysis by Mark Hamer and Yoichi Chida, published in 2008, found that active commuting was associated with a modest reduction in cardiovascular risk. A large analysis of UK Biobank data by Carlos Celis-Morales and colleagues, published in the BMJ in 2017, followed more than a quarter of a million commuters and found that cycle commuting was associated with lower risks of cardiovascular disease, cancer and death from all causes, and that walking to work was associated with lower cardiovascular risk, compared with non-active commuting. Because these are observational studies, they cannot prove that commuting mode caused the differences. But they are consistent with everything else we know about regular moderate activity. Most commuting students cannot walk or cycle the whole way. Many can do part of it. Table 4 sets out some of the options by type of commute. None of them is compulsory; the aim is to find one or two that fit your route. Table 4. Ways to add movement to different kinds of commute. Commute type Low-effort option Moderate option Practical cautions Driving Park at the far end of the car park Park off campus and walk the last stretch Check parking rules and personal safety Bus or tram Walk to a further stop Get off one or two stops early Allow extra time; plan for weather Train or subway Take stairs at stations Walk or cycle to and from the station Bike storage; crowded stairs Cycling Ride part way; use transit for the rest Ride the whole route on some days Lights, visibility, safe routes Walking Take a slightly longer, pleasanter route Brisk pace for part of the walk Footwear; lighten bag Several practical questions come up whenever students try active commuting. Time. Walking an extra stop or parking farther away typically adds five to fifteen minutes to a journey. That is real time, but it is often recovered: many students find they arrive more alert and need less time to get going. On days when time is tight, skip it; this is a flexible tool, not a rule. Sweat and changing. For walking at a normal pace, this is rarely an issue. For cycling or brisk walking in warm weather, it can be. Riding at a relaxed pace, carrying a change of shirt, and finding out whether your campus has showers, often in sports facilities, all help. Some universities provide lockers or changing facilities specifically for active commuters. Weather. Rain gear, layered clothing and good footwear make walking viable in most weather. In extreme heat, cold, ice or storms, it is reasonable to fall back on the whole transit journey. Safety. Cycling safety deserves particular care. Use lights and reflective clothing in low light; wear a helmet, which is legally required for some riders in some places and advised in many others; plan routes that use bike lanes and quieter streets where possible; and take extra care at junctions, where many collisions occur. For walking, choosing well-lit, well-used routes after dark matters, and Chapter 8 discusses personal safety on late journeys more fully. Air pollution. Walkers and cyclists on busy roads are exposed to traffic pollution, which is a legitimate concern. Choosing quieter routes, even slightly longer ones, reduces exposure. Most researchers who have studied this conclude that for most people in most cities, the benefits of active travel outweigh the risks from pollution, though this may not hold in the most polluted cities or for people with significant lung conditions. Movement across the campus day Even if the commute itself cannot be made active, the campus day offers many chances to move. The idea is to replace some sitting with standing and walking, and to break up long stretches of sitting, rather than to add a formal workout. Walk in the gaps. A student with a two-hour gap between classes can spend twenty minutes of it walking, perhaps to a different library, a café across campus, or simply around the grounds. Walking with a friend makes it social; walking alone with a podcast or lecture recording makes it productive. Stand when you can. Some libraries and study spaces have standing desks. Standing for part of a study session, especially when reading, breaks up sitting without requiring any extra time. Standing is not exercise, and standing all day has its own drawbacks, but alternating is helpful. Take the stairs. A few flights of stairs a few times a day adds up. In many campus buildings, the stairs are faster than waiting for lifts anyway. Move every half hour or so in long study sessions. A simple rule is to stand up and move for a minute or two roughly every thirty to sixty minutes of sitting. Refill your water bottle, walk to the end of the corridor, stretch. This fits naturally with study techniques that use short, timed work intervals. Use the commute's waiting time. Standing rather than sitting on the platform, walking up and down while you wait for a bus, and taking the stairs rather than the escalator at stations all count. Consider a master's student in public policy with a ninety-minute train commute and long days of seminars. She does not have time for the gym and has given up trying. She starts with two changes: walking the fifteen minutes from the station to campus instead of taking a connecting bus, and taking a twenty-minute walk during her lunch break. Together these give her about fifty minutes of walking on campus days. She adds a stretching routine of five minutes after long seminars. None of this feels like exercise, but within a month she notices that her back hurts less, the afternoon slump is milder, and she sleeps better. Finding time for real exercise Light movement and active commuting help a great deal, but they do not fully replace more vigorous activity or muscle strengthening. The WHO's recommendations for moderate-to-vigorous activity and for strength work on two or more days a week still apply to commuting students. The challenge is finding time in a day with no free evenings. A few approaches work for many commuting students. Use the campus gym in a gap. Many universities have sports facilities available to students at low or no cost. A three-hour gap between classes can accommodate a forty-minute workout and a shower. This depends on facilities being close and on having somewhere to store gym clothes, so it is worth asking about lockers. Train at home in short sessions. Muscle strengthening does not require a gym. Bodyweight exercises, such as squats, lunges, push-ups against a wall or on the floor, planks and step-ups, can be done at home in twenty minutes with no equipment. Two or three short sessions a week meet the strengthening recommendation. Use the day you do not commute. Many commuting students have one or two days a week off campus. These are natural days for a longer walk, a run, a swim, a class or a sport. Combine exercise with social life. Joining a campus sports club, running group or dance class gives exercise a social reason to happen, and it also helps with the sense of belonging on campus that Chapter 8 discusses. Many clubs have casual or beginner levels. Accept imperfect weeks. A commuting student in examination season is unlikely to meet every recommendation, and that is fine. What matters is the pattern across the term and the year. Looking after the body while you sit Some sitting is unavoidable. A ninety-minute train ride or a three-hour seminar cannot be turned into a walk. But the way you sit, and what you do with your body while sitting, makes a difference to comfort and alertness. On transit, many students spend the journey hunched over a phone with the head bent forward and the shoulders rounded. Sustained forward head posture is a common contributor to neck and upper back discomfort. The evidence that any particular posture causes lasting damage is weaker than popular accounts suggest, and the most useful general principle is that no single posture is ideal for hours; change is what matters. Holding the phone higher, looking up and out of the window from time to time, and shifting position every so often all help. Several simple movements can be done in a seat without drawing attention: rolling the shoulders backwards a few times, gently turning the head from side to side, pressing the shoulder blades together, straightening and flexing the legs, circling the ankles, and pressing the feet into the floor and releasing. They take a minute and are particularly useful on long journeys and in long seminars. At a desk in the library, the basics of setting up a workstation help: a chair height that lets your feet rest flat, a screen roughly at eye level, which usually means raising a laptop on a stand or a stack of books and using a separate keyboard, and elbows at roughly a right angle. Many students work for hours on a laptop flat on a table, looking down. For occasional use this is fine; for long study sessions it is worth improving. On very long journeys, of several hours, standing and moving periodically where it is safe to do so is also sensible. People with a history of blood clots, or other risk factors such as pregnancy or recent surgery, should ask their clinician about precautions for long periods of sitting during travel. Finally, pay attention to your eyes. Long stretches of close work on a screen, on a moving vehicle and then in the library, can leave the eyes tired and dry. A simple habit, widely recommended by eye care professionals, is to look at something in the distance for a short time at regular intervals. The commute itself offers a natural chance to do this: looking out of the window on part of the journey is good for the eyes as well as for motion sickness. Carrying the load A final physical issue deserves attention because it is specific to students who have no base on campus: carrying everything all day. A laptop, books, food, a water bottle, a flask, a change of clothes and a gym kit can add up to a substantial weight, carried for many hours. There is no universally agreed safe weight for adult backpacks, and the evidence linking backpack weight to lasting back problems in adults is limited. But heavy, badly carried bags commonly cause shoulder, neck and back discomfort, and that discomfort makes a long day harder. Practical steps help: use a backpack with two padded straps and wear both, adjust it so it sits close to the back rather than hanging low, pack heavier items close to the back, and remove anything you do not need that day. If your campus offers lockers, including day lockers, use them for things you do not need to carry between classes. Some students keep a set of textbooks at home and use digital versions on campus. Rolling bags suit some routes but not crowded trains and stairs. If you have persistent back, neck or shoulder pain, or any numbness, tingling or weakness, that is a reason to see a clinician or physiotherapist rather than to keep adjusting straps. Movement for the commuting student is less about adding a new block of exercise to an overfull day than about changing the texture of the day itself: a little walking in the commute, a little in the gaps, a little standing, a short strength session two or three times a week. The body responds to that texture. So does the mind, which is the subject of the final chapter. Chapter 8: The Commute and the Mind Sleep, food and movement are the physical frictions of the commuting day. This chapter deals with the fourth, which shapes all the others: the mental and emotional load of a life lived partly in transit. It is the least tangible of the four, and commuting students often dismiss it, as if the difficulty of being always on the way somewhere were a matter of attitude. It is not. It has identifiable parts, and most of them respond to practical changes. The chapter covers four themes: what the commute can be for, the stress of unreliable and crowded travel, belonging and connection on a campus where you do not live, and safety on late journeys. It ends with mental health and where to find help. Deciding what the commute is for Most students default to one of two approaches to travel time. Some treat it as study time, trying to read, revise or answer email throughout. Others treat it as dead time to be endured, scrolling a phone without much purpose. Both can work, but neither is the only option, and deciding deliberately what the journey is for tends to make it feel better. There is interesting evidence that a commute can serve as a psychological transition between roles. Research by Jon Jachimowicz and colleagues, published in 2021, studied employees and found that those who used the morning commute to think ahead about their work day, a process the authors called work-related prospection, experienced less of the negative effect of long commutes on their job satisfaction. The participants were workers rather than students, and the findings should not be stretched too far. But the idea translates naturally: the morning journey can be a time to shift from the role of family member, flatmate or employee into the role of student, and the evening journey a time to shift back. In practice, several uses of commute time work well for students, and it is reasonable to vary them by day and by mood. Preparation. Spend ten minutes of the morning journey looking at what the day holds: which classes, what you want to get from them, what you need to do in each gap. This is not the same as studying; it is orienting. Many students find it reduces the scattered feeling of a fragmented day. Light learning. Reviewing flashcards, listening to recorded lectures or podcasts related to your subject, or reading a chapter all work for some students on some routes. They work best when the journey is smooth and seated, and poorly on crowded, twisting routes where motion sickness is a risk. Audio is almost always easier on the body than reading. Recovery. On the evening journey especially, doing nothing demanding is a legitimate choice. Listening to music, looking out of the window, or simply resting is not wasted time if it means you arrive home with something left. Connection. Calling a family member or friend, or messaging people you have not had time to talk to, turns the commute into social time. For students who feel isolated on campus, this can be an important link. Closing the day. On the way home, taking a few minutes to note what you finished and what is left for tomorrow can help separate study from the evening, so that you are not carrying it all into the house. The common thread is intention. A student who has decided that the morning train is for flashcards and the evening train is for music experiences the same journey differently from one who drifts through it. And it is perfectly sound to decide that some journeys are for nothing at all. Stress, control and unreliable travel As Chapter 1 noted, unpredictable journeys are consistently experienced as more stressful than long but reliable ones. The stress comes less from the lost minutes than from the loss of control: standing on a platform watching the departure board, unable to do anything, knowing that you will miss the start of a seminar. Several measures restore some sense of control. Build in slack. Plan to arrive fifteen or twenty minutes before your first commitment rather than just in time. On good days, use that slack for a coffee, a walk or a look over your notes. On bad days, it absorbs the delay without panic. Know your alternatives. Learn the second route before you need it. Save the transit agency's service updates in your phone. Know whether the lecture is recorded, whether slides are posted, and whom to email. Tell your instructors. Many teaching staff are sympathetic to commuting students but do not know who they are. A short, early note explaining that you commute a long distance and may occasionally be delayed by transit problems is professional and often appreciated. It also makes the conversation easier if something goes wrong. Accept what you cannot control. There is a point at which the train is late and nothing you do will change it. Recognising that point, and deciding to use the waiting time for something else rather than repeatedly checking the board, is not passive resignation. It is a practical way to limit the stress to the delay itself. Crowding is a related source of stress. The measures in Chapter 2, travelling slightly outside peak times where possible, using headphones or earplugs, and allowing a few minutes of recovery on arrival, all help. For students with anxiety, sensory sensitivities or conditions that make crowded travel particularly difficult, the university's disability or accessibility service may be able to support adjustments such as flexibility on attendance at early classes. Rules and eligibility vary by institution and, in the United States, are shaped by disability law; it is worth asking. Talking to strangers, and why it may help One of the more surprising findings in the psychology of commuting comes from research by Nicholas Epley and Juliana Schroeder, published in 2014. They asked commuters on trains and buses in the Chicago area to either strike up a conversation with a stranger, keep to themselves, or commute as normal. Participants predicted that talking to a stranger would be the least pleasant option. In fact, those who talked to a stranger reported a more pleasant commute than those who kept to themselves, and did not report being less productive. People also underestimated how interested others would be in talking to them. This does not mean every commuting student should start conversations every day, and there are good reasons, including personal safety and cultural norms, to be selective. But the finding points to something important. People tend to underestimate the value of small social contact, and the commute, often experienced as isolating, can occasionally be the opposite. Some students on regular routes find they come to recognise and talk to other students making the same journey. Some universities encourage this deliberately, with commuter student networks and car-sharing schemes. Belonging on a campus you do not live on The deepest part of the mental load for many commuting students is not the journey but what happens at either end. A residential student lives inside the university: they meet people in halls, eat together, stay for evening events and gradually build a network. A commuting student arrives for class, has gaps with nowhere in particular to be, and leaves when teaching ends, often before the social life of the campus begins. Many describe feeling like a visitor. The research literature on commuter students has noted this for decades. Commuting students are often less involved in campus activities than residential students, although this varies greatly and many are deeply involved in communities at home. The issue is not that one kind of student life is better. It is that feeling disconnected from the place where you study can affect motivation, persistence and wellbeing, and commuting students are more exposed to that risk. Several practical steps help build connection within the constraints of a commuting schedule. Find a base. Many universities have commuter lounges, student union spaces or departmental common rooms. Having a regular place to be between classes, where you see the same people, makes a large difference over time. If your institution has a commuter student office or society, it is worth seeking out. Use the gaps socially. A gap spent in the library alone every week could sometimes be spent in a study group, a club meeting held at lunchtime, or simply eating with classmates. Choose one or two commitments that fit. You probably cannot join everything. One society, one sports club or one volunteer role that meets at a time compatible with your commute will do more for your sense of belonging than many that you cannot attend. Plan a late day occasionally. Some of the most important parts of university life happen in the evening. Arranging one evening every few weeks when you stay late, perhaps planning to eat on campus and take a later train, lets you take part without making it a daily cost. If you drive, plan in advance how you will manage fatigue on the late drive home, as discussed in Chapter 3. Talk to other commuters. Commuting students often assume they are unusual. On many campuses they are the majority. Finding each other, whether through a society, a course group chat or conversations on a shared route, helps. Consider a first-generation student who lives with her family and commutes an hour each way. In her first semester she comes to campus, attends class and goes home, and by November she is wondering whether she belongs at university at all. In the second semester she joins a lunchtime society related to her major, starts sitting in the same area of the student union during her Tuesday gap, and discovers several other students who take the same train line. None of this changes her commute, but it changes the day. The campus becomes a place where she knows people rather than a place she passes through. Commuting alongside work and family Many commuting students are not only students. They work paid jobs, sometimes many hours a week; they care for children, siblings or parents; they have responsibilities in their home communities. For these students, the commute is not simply the link between home and campus. It is one leg of a triangle between home, campus and work, or a thread between study and caring. The mental load in these lives comes less from any single demand than from the constant switching between them. A student who leaves a lecture, drives to a shift, and then goes home to help a younger sibling with homework is changing roles several times a day, often with no time in between. The idea from earlier in this chapter, that the journey can serve as a transition between roles, is especially useful here. Even five minutes at the end of a drive, sitting in the car before going in, or a few stops of a bus ride with no screen, can mark the change from one role to another and reduce the feeling of carrying everything at once. Several practical measures help students with heavy outside commitments. Map the whole week, not just the timetable. The audit from Chapter 1 should include work shifts, caring responsibilities and travel between them, not only classes. It is common to discover that the real pressure point is not the campus commute at all but a particular day when study, work and family collide. Communicate across your roles. Employers are often more flexible about shifts than students assume, especially around examinations, if asked early. Instructors are often more flexible about deadlines for students with caring responsibilities if they know about them. Many universities have formal policies for student parents and carers; it is worth finding out what yours are. Protect one fixed recovery point. When every day is full, it is easy to lose any time that is truly your own. One protected hour a week, used for something restorative, is modest but real. It is also a signal to yourself that your wellbeing counts among your responsibilities. Use the services built for you. Some universities provide childcare, emergency funds, flexible study options, or advice specifically for mature students, student parents or working students. They are often underused because the students who need them are least likely to have time to look for them. A single visit to student services, once a year, to ask what is available, can be one of the most productive hours in a busy student's calendar. These students often bring considerable strengths to university: organisation, resilience and clear purpose. The aim is not to reduce their commitments, which may not be optional, but to reduce the unnecessary friction around them. Safety on late and early journeys Commuting students travel at the edges of the day, sometimes in the dark, sometimes on quiet platforms or in empty car parks. Most journeys are uneventful. It still makes sense to plan for safety, without letting it become a source of constant anxiety. Plan your route home in advance for late evenings, and know when the last services run. Wait in well-lit, populated areas, near staff or cameras where possible. Keep your phone charged, and carry a small battery pack if your phone tends to run low. Share your journey with a friend or family member, many phones and apps make this easy. Sit near the driver on buses or in busier carriages on trains. Stay alert: late at night is not the time for both earphones and deep absorption in a screen. Many universities offer safety escort services, late-night shuttles or safe-walk schemes, and these are available to commuting students; find out what yours offers. For drivers, park in well-lit areas close to building entrances when you know you will leave late, have your keys ready as you approach your car, and follow the drowsy-driving guidance in Chapter 3. If you are ever in immediate danger, call the emergency number for your country: 911 in the United States, 112 across the European Union, 999 in the United Kingdom, and other numbers elsewhere. Report harassment or unsafe incidents on public transport to the transit operator or police; many transit systems have dedicated reporting lines or text services. Mental health and where to find help University is a period of life in which many mental health problems first appear or become worse, and commuting students are not exempt. Some aspects of commuting, such as isolation from campus, compressed sleep and the stress of a stretched day, may add to the risk. At the same time, commuting students often have strong support at home, which can be protective. It is worth knowing the signs that stress has become something more. Persistent low mood or loss of interest in things you used to enjoy, lasting more than a couple of weeks; anxiety that interferes with daily life, including travel; changes in sleep and appetite beyond what your schedule explains; withdrawing from people; difficulty concentrating to the point that studying becomes very hard; and using alcohol or other substances to cope are all reasons to talk to someone. University counselling services are available to commuting students on the same terms as residential ones, although commuters use them less, sometimes because appointments are hard to fit around travel. Many services now offer telephone or video appointments, evening slots, or online programmes that reduce this barrier. It is reasonable to ask for an appointment at a time that fits your commute. A family doctor or local mental health service near home is another route. If you are having thoughts of suicide or self-harm, or someone you know is, this is an emergency. In the United States, you can call or text 988 to reach the 988 Suicide & Crisis Lifeline. In other countries, crisis lines and emergency numbers differ; your local emergency number is always an option, and your university's counselling service or student services office can tell you about local crisis support. If someone is in immediate danger, call emergency services. The mental load of commuting is real, but it is not fixed. Giving the journey a purpose, building slack against unreliability, finding a base and a few people on campus, travelling safely, and knowing where help is available together make the commuting life lighter. They also make it easier to keep up the sleep, food and movement habits that the earlier chapters describe, because a student who feels they belong somewhere is more likely to take care of themselves there. Conclusion: Designing the Day Around the Journey The argument of this book began with a claim: the commute is a fixed, predictable part of the day, and most of its cost comes not from the distance but from the friction around it. Having worked through sleep, fatigue, food, drink, movement and the mind, it is now possible to say something more specific than that, and to say it in a way that could not have been said at the start. The frictions have an order The first thing the chapters show is that the four frictions are not equal partners. They have an order, and the order matters for anyone trying to change their commuting life. Sleep comes first because it sits upstream of everything else. A student who is chronically short of sleep will find the commute exhausting whatever they eat, will reach for sugar and caffeine to compensate, will be too tired to walk the extra stop, and will experience delays and crowds as more stressful than they are. Chapter 3's method of building the night backwards from the departure time is not one tip among many. It is the foundation, and students who try to fix lunch or exercise before fixing sleep usually find that the new habit collapses within weeks. Fuel comes second, because it is the next most direct determinant of how the day feels and because it is the area where preparation pays off most visibly. Movement comes third, not because it matters less over a lifetime, but because it is easier to add once the student is sleeping and eating well, and because much of it can be folded into the commute itself. The mental load runs through all three, eased by each of them and, in turn, making each easier to sustain. This ordering gives a practical answer to the question students most often ask: where do I start? Start with the night before. The unit of planning is the term, not the day The second conclusion concerns time. Most health advice is framed at the level of the day: eat this for lunch, walk this much, sleep this long. The commuting student's life shows why that framing is incomplete. The commuting day is shaped by a timetable, and the timetable changes every term. A system for sleep, food and movement that works beautifully in autumn can fail completely in spring because the early lecture moved from Monday to Thursday, the long gap disappeared, or a new job took two evenings. What follows is that the most important planning moment is not each evening but the first week of each term. That is when a commuting student should sit down with the new timetable and rebuild the system: the earliest departure and the bedtime it requires; which days are long and need the full five fuelling points; where the gaps are and what they are for; where movement fits; which evening, if any, will be a late one on campus. It takes an hour. It prevents the slow collapse that otherwise sets in by the fifth or sixth week, and which students tend to blame on themselves. It also suggests that timetable choices deserve more weight than students usually give them. Where there is any choice of sections or class times, consistency and the shape of the commuting day are legitimate considerations alongside convenience and preference. Responsibility is shared The third conclusion is one that a book addressed to students could easily neglect. Much of what makes commuting hard is not within the individual student's control. Whether there are fridges and microwaves that commuters can use, lockers for bags and gym kit, commuter lounges, showers for cyclists, recorded lectures for days when trains fail, affordable healthy food on campus, safe late-night transport, and timetables that avoid scattered single classes across a long day: these are institutional decisions. Students can and should use the facilities that exist, and a surprising number of commuting students do not know what is available. But they can also ask for what is not. Student unions, commuter societies and student representatives on university committees are legitimate channels for pointing out that a campus built around residential students leaves many of its students without a base. The individual habits in this book make a hard situation easier; the institutional ones can make it less hard in the first place. What remains unsettled Honesty requires acknowledging what we do not know. Most of the research on commuting and wellbeing concerns working adults, not students, and almost all of it is observational. We know that long and unreliable commutes are associated with lower wellbeing, but we know less about how much of that is caused by the commute itself, and very little about what specifically helps students. Several practical questions remain open. Whether studying on transit is an efficient use of time or a false economy that produces poor learning and more fatigue is not well studied, and probably depends on the student, the route and the task. How the growth of hybrid and online teaching will change the commuting student's life is uncertain: fewer days on campus may reduce the total burden, or may make the remaining days longer and more crowded and further weaken the sense of belonging. And the research on sedentary behaviour, though growing, has not yet established precisely how much breaking up sitting matters for young, otherwise healthy adults, as opposed to older or less healthy groups. These gaps do not undermine the practical advice, which rests on well-established principles about sleep, nutrition, food safety and activity. They do mean that commuting students should treat this book as a starting framework and pay attention to their own experience, which is the best evidence they will have about their own journeys. What to do differently If the chapters of this book reduce to a single change in behaviour, it is this: stop treating the commute as something that happens between the parts of your day that matter, and start designing the day with the commute inside it. Concretely, that means a handful of actions. Do the one-week audit from Chapter 1, so you know your real day. Set your bedtime from your departure time and keep your wake time steady across the week. Plan five fuelling points for long days, pack the night before, and keep an emergency stash. Carry food cold or hot, not warm. Make part of the journey active, and move in the gaps. Decide what each journey is for. Find a base on campus and one or two people or activities that make it yours. Rebuild the system at the start of every term. Use the services your university provides, and ask for the ones it does not. The nursing student on the platform at six-forty in the morning, whom we met at the start of this book, still has a long day ahead of her. The train is no faster. But if she slept seven and a half hours, ate breakfast, has a lunch that will last, a snack for the journey home, a plan for her afternoon gap and a reason to look forward to part of the day, she is starting a different day from the one she used to have. The commute has not changed. She has changed what surrounds it, and that turns out to be most of what mattered. Hashtags: #CommuterStudentHealth #StudentWellness #TransitFatigue #HealthyCommuting #CommuterStudentLife #HealthyPackedLunches #MealPrepForStudents #StudentNutrition #BudgetFriendlyMeals #HealthyMealPlanning #PortableHealthyMeals #FoodSafety #BalancedNutrition #HealthySnacking #EnergyManagement #SleepHygiene #SleepHealth #CircadianRhythm #FatigueManagement #DrowsyDrivingPrevention #ActiveCommuting #PhysicalActivity #SedentaryLifestyle #StressManagement #MentalWellbeing #CampusWellness #HealthyHydration #TimeManagementForStudents #HealthyStudentHabits #StudentHealthGuide

Download the Book (PDF): Introduction A carton of oncology drugs leaves a packaging line in Ireland. Over the following three weeks it will be aggregated onto a pallet, sold to a wholesaler, cross a border, sit in a bonded warehouse, be broken down into cases, travel to a regional distributor, and finally reach a hospital pharmacy in Ohio. At each of those steps somebody records something: a serial number scanned, a shipment notice sent, a customs declaration filed, a temperature logger read. Almost none of those records live in the same place. Each company keeps its own, in its own system, under its own rules, and shares fragments with its immediate neighbours in the chain when contracts or regulators require it. When something goes wrong — a recall, a suspected counterfeit, a container that spent eleven hours on a hot quay — reconstructing what actually happened means stitching those fragments together by email, spreadsheet and telephone, often under time pressure and often with parties who have good reasons not to show their whole hand. That is the traceability problem, and it is not a technology problem in the first instance. It is a problem of trust and incentives among organisations that must cooperate but do not fully trust one another, that compete in some respects while depending on each other in others, and that are each accountable to regulators who increasingly expect them to answer questions about goods they no longer hold. This booklet is about one specific answer to that problem: permissioned blockchain, and in particular Hyperledger Fabric, the open-source platform that has become the most widely deployed enterprise ledger for business-to-business consortia. It is written for people who need to understand how Fabric actually works well enough to make decisions about it — architects, technical leads, supply chain and compliance managers, and the executives who sign off on consortium agreements — and who want something more precise than vendor brochures and less exhausting than the full reference documentation. The argument The argument of the book is simple to state and takes the rest of the pages to earn. A permissioned ledger is valuable in a supply chain not because it is "decentralised" in some abstract sense, and certainly not because it makes data true. It is valuable because it lets a group of organisations write their agreements about who may record what, who must vouch for it, and who may see it into machinery that every member runs and no single member controls. Fabric's distinctive design choices — its execute-order-validate transaction flow, its endorsement policies, its channels and private data collections, its chaincode lifecycle that requires organisations to approve code before it runs — are best understood as a toolkit for encoding those agreements. Projects that treat Fabric as a database with extra steps tend to fail. Projects that treat it as a governance instrument, and design their endorsement policies, privacy boundaries and data models as deliberately as they design their consortium contracts, have a much better chance. The corollary is uncomfortable for enthusiasts. The ledger is rarely the hard part. The hard parts are getting competitors to agree on a data model, getting physical events captured accurately at the moment they happen, deciding who pays, and keeping a network of independently operated nodes upgraded and healthy for a decade. The most famous Fabric supply chain network, the TradeLens shipping platform built by IBM and Maersk, was technically capable and commercially unsuccessful; its closure is one of the most instructive episodes in the field, and it recurs in these pages for that reason. What the book covers and what it leaves out The first chapter makes the case for permissioned ledgers in business consortia, and is honest about when a shared database or a regulator-run repository is the better answer. The second explains Fabric's architecture from the ground up: identities and membership services, peers and orderers, the ledger and world state, and the three-phase transaction flow that sets Fabric apart from public blockchains. The third and fourth chapters are the technical heart of the book. Chapter 3 treats endorsement policies — the rules that decide which organisations must sign off on a transaction before the network accepts it — as the primary place where a consortium's trust model becomes code. Chapter 4 does the same for privacy, working through channels, private data collections and the off-chain patterns that complement them. Chapter 5 turns to chaincode, Fabric's term for smart contracts, and to the specific discipline of modelling provenance: how to represent products, lots, serial numbers, shipments and events so that the ledger can answer the questions people will actually ask of it. The next two chapters apply all of this to the two domains named in the title. Chapter 6 deals with pharmaceutical provenance, under the United States Drug Supply Chain Security Act and the European Union's Falsified Medicines Directive, two regimes that have taken strikingly different architectural paths. Chapter 7 deals with international shipping manifests and trade documents, where the problem is less about serial numbers and more about the dozens of parties who handle the paperwork for a single container. Chapter 8 addresses governance and operations: onboarding, upgrades, ordering service control, cost sharing, and the reasons consortia fall apart. The conclusion draws out what follows for anyone deciding whether, and how, to build. Some things are deliberately left out. There is no survey of every enterprise blockchain platform; Corda, Quorum, Besu and others are mentioned only where a comparison sharpens a point about Fabric. There is no discussion of tokens, cryptocurrencies or decentralised finance, which belong to a different conversation. Code appears only where a short snippet makes a mechanism clearer than prose would; this is not a programming tutorial, and readers who want one should go to the official Fabric documentation and samples, which are good. Version details reflect the state of the platform in 2026: the long-term-support 2.5 line remains the workhorse of production networks, while the 3.x line, first released in 2024, adds a Byzantine fault tolerant ordering option. Where the difference matters, the text says so. A note on vocabulary Enterprise blockchain has accumulated a thicket of jargon, some of it Fabric-specific and some borrowed loosely from public blockchains. This book defines terms where they first appear and tries to use them consistently. A few distinctions are worth flagging now because they are commonly blurred. A consortium here means a group of independent organisations that jointly operate a network under some agreement. A network is the technical infrastructure they run. A channel is a Fabric-specific construct: a private sub-ledger shared by a subset of the network's members. Chaincode is the program that defines the business logic; Fabric's newer documentation also uses smart contract for the individual contracts packaged inside a chaincode, and this book uses the two terms with that relationship in mind. Traceability means the ability to follow an item forward and backward through the chain; provenance means the documented history of where an item came from and who has held it. The two overlap heavily, and in practice the same records serve both. Finally, trust. The word is used constantly and loosely in this field, often to suggest that blockchain makes trust unnecessary. It does not. What a well-designed permissioned network does is make trust specific: instead of trusting a partner in general, members trust a defined set of signatures, a defined piece of code, and a defined set of operators, each of which can be inspected and argued about. That shift, from diffuse trust to specified trust, is the real product. Everything that follows is about how to specify it well. Chapter 1: The Consortium Problem Every supply chain is a chain of custody, and every chain of custody is a chain of records. The goods move; the records are supposed to move with them. In practice they move separately, partially, and late. A manufacturer knows what it shipped. A distributor knows what it received and what it sent on. A carrier knows which container was loaded on which vessel. A customs authority knows what was declared. A pharmacy knows what it dispensed. Each of these parties holds a true but incomplete account, and the account that matters most — the continuous history of a particular item from origin to end use — is held by nobody. This chapter is about why that is so hard to fix, why the obvious fixes have repeatedly failed, and where a permissioned ledger fits among the alternatives. It is also about where it does not fit, because a great deal of money was spent between 2016 and 2022 on blockchain projects that should have been databases, and the lessons from that period are the best guide to building something that lasts. Why nobody owns the record The simplest solution to fragmented records is a central system that everyone writes to. It is also the solution that has been tried most often, and it fails for reasons that have nothing to do with technology. Consider who would operate it. If the largest manufacturer runs the central traceability database, its competitors are being asked to hand commercially sensitive data — volumes, customers, prices implied by shipment patterns — to a rival. If the largest distributor runs it, manufacturers face the same problem in reverse, and the distributor gains a structural advantage over smaller wholesalers who must plug into its platform on its terms. If a technology vendor runs it, every participant becomes dependent on that vendor's pricing, roadmap and survival. If a trade association runs it, someone must fund it, staff it and govern it, and associations are not usually equipped to run critical infrastructure with the reliability that operational supply chains demand. Each of these objections is a version of the same thing: whoever controls the central record gains power over everyone else who depends on it. The power can be used to raise prices, to extract data, to favour some participants, to change the rules, or simply to fail through neglect. Organisations that understand this are reluctant to join, and a traceability system that key participants refuse to join does not trace anything. The second obstacle is that the data really is sensitive. A pharmaceutical wholesaler's shipment records reveal which pharmacies it serves and in what volumes; a manufacturer would love to see that, and the wholesaler's competitors would love it even more. A freight forwarder's bookings reveal its customers. A shipper's commercial invoice reveals what it pays its suppliers. Traceability requires sharing some of this information with some parties, but almost never all of it with everyone. Any system that cannot express fine-grained rules about who sees what will either leak information participants will not tolerate, or will be configured so conservatively that it shares nothing useful. The third obstacle is accountability. When a regulator asks a company to demonstrate that a product it sold was authentic, the company needs records that it can stand behind. A record in someone else's database, which that someone else could alter, is weak evidence. A record in your own database, which you could alter, is weak evidence to anyone else. What participants want is a record that each of them holds, that none of them can quietly change, and that carries the attestations of the parties who created it. Those three needs — no single controller, fine-grained confidentiality, and tamper-evident shared records with clear attribution — are the needs that permissioned blockchains were designed to meet. Whether they meet them better than the alternatives depends on the specific consortium, and it is worth being clear about the alternatives before going further. The alternatives, honestly assessed There are broadly four ways to share traceability data among organisations that do not fully trust one another, and each has real deployments behind it. The first is point-to-point messaging. Each party sends standardised messages to its immediate trading partners: advance shipping notices, electronic data interchange transactions, or event messages in the GS1 Electronic Product Code Information Services standard, known as EPCIS. This is how most of the world's supply chains actually work today, and how the United States pharmaceutical industry has chosen to meet most of its federal traceability obligations. Its strength is that nobody has to agree on a shared platform; each party only needs to exchange data with the parties it already does business with. Its weakness is that the history of an item is scattered across every system it has passed through. Answering a question about an item's full journey means querying each holder in turn, and relies on each one answering promptly and truthfully. The second is a central repository operated by a neutral or regulated body. The European Union took this route for medicines: under the Falsified Medicines Directive, manufacturers upload serial numbers to a hub run by a non-profit stakeholder organisation, the data flows to national systems, and pharmacies verify and decommission packs against them before dispensing. The model works because the operator is legally mandated, governed by the stakeholders, and funded by them. Where such a body can be created, it is often the simplest answer. Its weaknesses are the ones described above — concentration of power and data in one place — mitigated by law and governance rather than by architecture. The third is a shared database with cryptographic audit features. Several cloud providers offer ledger databases that maintain an append-only, cryptographically verifiable journal; Amazon's Quantum Ledger Database was the best known, though Amazon announced in 2024 that it would end the service in 2025, a reminder that dependence on a single vendor is itself a risk. Such databases can prove that records have not been altered since they were written, but they are still operated by one party, and the other participants must trust that party not to refuse service, censor writes or rewrite history before it is journalled. The fourth is a permissioned distributed ledger: a network in which each participating organisation runs its own nodes, holds its own copy of the shared records, and participates in the rules by which new records are accepted. No single participant controls the record. Each participant can verify independently that the record has not been tampered with. The rules for writing, the code that enforces them and the membership itself are governed collectively. Hyperledger Fabric, R3's Corda and the enterprise variants of Ethereum such as Hyperledger Besu are the main platforms of this kind. The honest summary is that the fourth option is the most powerful and the most expensive. It costs more to build, more to operate and, above all, more to govern, because every participant runs infrastructure and every rule change requires agreement. It is justified when the first three options fail for a specific reason: when no neutral operator exists or can be created, when the parties will not accept any single operator, when the data must be verifiable by each party independently, and when the rules about who may write and see what are complex enough to need enforcing by code rather than by contract alone. Regulation is raising the stakes The question of how to share traceability data would be an interesting management puzzle if it were left to commercial choice. It is not. Over the past decade regulators in several sectors have moved from asking companies to keep their own records towards asking them to produce, quickly and electronically, records that span the chain. Pharmaceuticals led the way, and Chapter 6 examines the American and European regimes in detail. Food is following. Under the Food Safety Modernization Act, the US Food and Drug Administration finalised a food traceability rule in 2022, usually called the Section 204 rule, which requires companies that manufacture, process, pack or hold foods on a designated Food Traceability List — leafy greens, fresh-cut fruit, shell eggs, certain fish and cheeses among them — to record specified key data elements at defined critical tracking events such as harvesting, cooling, initial packing, shipping, receiving and transformation. Firms must be able to provide the FDA with an electronic, sortable spreadsheet of the relevant records within twenty-four hours of a request. The original compliance date was January 2026; in 2025 the FDA moved to push it back to mid-2028, giving the industry more time, but not changing what will be required. The structure of that rule is telling. It does not require a shared platform. It requires each firm to keep its own records and to pass a defined set of data elements to the next firm in the chain. That is the point-to-point model, and it is what most firms will use. But it also creates exactly the situation in which a shared ledger becomes attractive for some groups of participants: when an investigation needs to join records from a grower, a packer, a distributor and a retailer within a day, a network on which all four have already recorded the same events in the same format turns a scramble into a query. Customs and trade regimes are moving in the same direction. Advance cargo information requirements, discussed in Chapter 7, oblige carriers and increasingly other parties to send detailed data about goods to customs authorities before they arrive. Sustainability and forced-labour regulations — the European Union's deforestation regulation, for instance, and the United States' Uyghur Forced Labor Prevention Act — require importers to demonstrate where goods and their inputs came from, sometimes several tiers back in the supply chain. Each of these regimes increases the value of a shared, attributable record of provenance, and each increases the cost of not having one. Why public blockchains do not fit Before turning to Fabric itself, it is worth addressing a question that newcomers often ask: why not simply use a public blockchain such as Ethereum, which already exists, is already decentralised, and costs nothing to join? The objections are practical rather than ideological. Public blockchains are transparent by default; every transaction is visible to anyone. Techniques exist to add confidentiality, including zero-knowledge proofs and layer-two networks, and they have matured considerably, but they add complexity and they do not change the fact that the base layer is designed for openness. Supply chain participants are generally unwilling to publish even encrypted or hashed records of their commercial activity to the world, partly because patterns of activity can be revealing even when contents are hidden. Public blockchains also have unknown participants. That is their point: anyone can run a node, anyone can transact, and the system remains secure through economic incentives even if many participants are hostile. Business consortia do not need that property, because their participants are known legal entities bound by contracts and subject to courts. The cost of providing security against anonymous adversaries — in throughput, latency, energy or transaction fees — buys nothing a consortium needs. Finally, regulated industries need to know who is accountable. A pharmaceutical regulator wants to know which licensed wholesaler made a particular record, and needs that wholesaler's identity to be bound to a real-world legal entity with a licence number. Public blockchains identify participants by keys, and mapping keys to legal identities is an add-on rather than a foundation. In a permissioned ledger, identity is the foundation: every participant, every node and every transaction is signed by a certificate issued under the consortium's rules. None of this means public blockchains have no role in supply chains. Some projects anchor periodic hashes of private ledger state onto a public chain, to give an external timestamp that no consortium member could later dispute. Some consumer-facing provenance schemes, particularly for luxury goods, use public tokens to represent ownership. But for the core business-to-business problem of sharing operational records among known parties under confidentiality rules, permissioned platforms have become the default choice, and Fabric the most widely used of them. Where Fabric came from Hyperledger was launched by the Linux Foundation at the end of 2015 as an umbrella for open-source enterprise blockchain projects. Fabric grew out of code contributed by IBM, with further contributions from Digital Asset Holdings, and it became the first Hyperledger project to graduate to production status. Version 1.0 was released in July 2017. Version 2.0, released in January 2020, introduced the decentralised chaincode lifecycle, under which organisations must each approve a chaincode definition before it can run — a change whose governance significance is discussed in Chapter 3. The 2.5 release became the long-term support line on which most production networks run. Version 3.0, released in 2024, added an ordering service based on a Byzantine fault tolerant protocol called SmartBFT, so that the network can tolerate ordering nodes that behave maliciously rather than merely crashing. In the same year the Hyperledger Foundation was folded into a new Linux Foundation umbrella, LF Decentralized Trust, which now hosts Fabric alongside related projects. The design is laid out in an academic paper by Elli Androulaki and colleagues at IBM Research, published at the EuroSys conference in 2018, which described Fabric as "a distributed operating system for permissioned blockchains". The phrase is apt. Fabric is less a single blockchain than a framework for building them: the consensus mechanism is pluggable, the identity provider is pluggable, chaincode can be written in general-purpose languages such as Go, Java and JavaScript, and the rules for validating transactions are configured per application rather than fixed by the platform. That flexibility is why Fabric attracted supply chain projects. A traceability consortium can define its own membership, its own endorsement rules and its own privacy partitions without forking the platform. The best-known early deployment, IBM Food Trust, launched in 2018 after pilots with Walmart. In one widely reported test, Walmart's food safety team found that tracing a package of sliced mangoes back to its farm took nearly seven days using conventional records and about two seconds using the ledger. The figure is often quoted out of context; the speed came less from the ledger itself than from the fact that every participant had agreed to record the same events in the same format at the time they happened. But it illustrates exactly what a shared record makes possible once the hard work of agreeing on that record has been done. The lesson of the boom years Between roughly 2016 and 2020, enterprise blockchain went through a hype cycle of unusual intensity. Consultancies published forecasts of trillions of dollars in value. Banks, shippers, food companies and pharmaceutical firms announced pilots by the hundred. A large proportion of those pilots never reached production, and several of the consortia that did reach production later shut down. The most significant closure for this book's subject is TradeLens. Announced by Maersk and IBM in 2018, it was a Fabric-based platform for sharing shipping documents and container events among carriers, ports, terminals, customs authorities, freight forwarders and shippers. It signed up several of the world's largest ocean carriers and a long list of ports and customs agencies. In November 2022 the two companies announced that it would be wound down, citing the failure to reach full commercial viability, and the platform closed in early 2023. Around the same period the bank-led trade finance network we.trade closed, and Marco Polo, another trade finance network built on Corda, entered insolvency. None of these failures was caused by the ledger technology failing to work. TradeLens processed large volumes of real shipping events. What failed was the consortium: competing carriers were wary of a platform majority-associated with their largest rival, the commercial model never attracted enough paying participants, and the value of the shared record depended on network effects that never fully materialised. Chapter 8 returns to this case in detail. For now the point is the one stated in the introduction: in a business consortium, the ledger is the easy part. A booklet about Fabric for traceability therefore has to be about two things at once. It has to explain the machinery precisely, because consortia that do not understand their own machinery make bad design decisions that are expensive to reverse. And it has to keep returning to the question the machinery is supposed to answer: what agreement among these organisations is being encoded here, and is it an agreement they will still want to honour in five years? The next chapter begins with the machinery. Chapter 2: How Fabric Works Hyperledger Fabric's design makes most sense when read against the problem described in the previous chapter: a group of known organisations who need to share a record without handing control of it to any one of them. Every major component maps onto part of that problem. Identity answers the question of who is who. Peers answer the question of who holds the record and runs the logic. The ordering service answers the question of what happened in what sequence. The transaction flow answers the question of how a change becomes accepted. This chapter takes each in turn, and then follows a single transaction from start to finish. Identity first On a public blockchain, anyone can generate a key pair and start transacting. On Fabric, nothing happens without a certificate. Every actor in the network — every peer node, every ordering node, every administrator, every client application, every user — holds an X.509 certificate issued by a certificate authority that the network recognises. Every message they send is signed with the corresponding private key. This is the bedrock on which everything else rests, and it is what makes Fabric "permissioned". The component that turns certificates into organisational identity is the Membership Service Provider, or MSP. An MSP is, in essence, a definition of an organisation: the root and intermediate certificate authorities it trusts, the certificates of its administrators, any revocation lists, and rules for classifying identities into roles. When the network configuration says that an organisation called PharmaCoMSP is a member, it is saying that any certificate chaining to PharmaCo's certificate authorities belongs to PharmaCo. Fabric's MSPs support a feature called node organisational units, which classify identities as client, peer, admin or orderer according to a field in the certificate. This matters a great deal for policies. A rule that says "PharmaCo's peer must sign" can be distinguished from a rule that says "any PharmaCo identity may submit", and an attacker who obtains an ordinary client certificate cannot impersonate a peer. Each organisation typically runs its own certificate authority. Fabric ships with one, called Fabric CA, but organisations are free to use their existing enterprise public key infrastructure instead. The consequence is that no central body issues identities for the whole network. Each member vouches for its own people and machines, and the network trusts each member's vouching only for that member's identities. If a consortium later wants to remove a member, it updates the channel configuration to drop that member's MSP, and every certificate that member issued ceases to be recognised. It is worth dwelling on how unusual this is. The network has no single root of trust. A consortium of a manufacturer, three distributors and a logistics provider has five roots of trust, each authoritative for one organisation. That is exactly the shape of the trust relationships in the real world, where each company is responsible for its own employees and systems. Peers, ledgers and world state The peer is the workhorse of a Fabric network. Each organisation runs one or more peers, and each peer does three things: it holds a copy of the ledger for every channel it has joined, it executes chaincode when asked to endorse a transaction, and it validates and commits blocks delivered by the ordering service. The ledger itself has two parts, and the distinction between them is fundamental to how Fabric applications are designed. The first part is the blockchain: an append-only sequence of blocks, each containing a batch of transactions and a cryptographic hash linking it to the previous block. The blockchain records every transaction ever submitted, including those that were later found invalid, along with a flag marking which ones were valid. It is the immutable history. The second part is the world state: a key-value database holding the current value of every key. If the blockchain is the bank statement, the world state is the balance. When a valid transaction commits, its writes are applied to the world state. Chaincode reads from and writes to the world state; it does not scan the blockchain. The world state can be rebuilt at any time by replaying the blockchain from the start. Fabric offers two world state databases. LevelDB, embedded in the peer, is fast and simple, and supports lookups by key and by key range. CouchDB, run as a separate process alongside the peer, stores values as JSON documents and supports rich queries over their contents — for example, "every shipment whose destination is Rotterdam and whose status is in transit". Rich queries are attractive for traceability applications, which tend to ask questions of that kind, but they come with a caveat discussed in Chapter 5: results of rich queries are not re-checked at validation time, so they should not be used to make decisions that determine what a transaction writes. Peers come in several roles. An endorsing peer is one that has chaincode installed and can execute it to produce endorsements. A committing peer validates and commits blocks; every peer does this. An anchor peer is one advertised in the channel configuration so that peers from other organisations can discover it and begin exchanging information through Fabric's gossip protocol. A peer can, and usually does, play several of these roles at once. The ordering service The ordering service is the component that puts transactions into a single agreed sequence and packages them into blocks. It is the closest thing Fabric has to a "consensus mechanism" in the public blockchain sense, but its job is narrower. Orderers do not execute chaincode, do not check whether transactions are valid against the world state, and in normal configurations do not need to see transaction contents in the clear. They receive endorsed transactions, agree on an order, cut blocks and deliver them to peers. Since version 2.0, the recommended ordering protocol has been Raft, a crash fault tolerant consensus algorithm. A Raft ordering service consists of several ordering nodes, ideally operated by different organisations, which elect a leader and replicate a log. A cluster of five nodes can tolerate two failing. Raft assumes that nodes may crash or become unreachable, but not that they will lie. If one consortium member's orderer were compromised and began sending conflicting messages, Raft would not protect against it. Version 3.0 added SmartBFT, a Byzantine fault tolerant protocol in which ordering nodes check one another's work and the service remains correct as long as fewer than a third of nodes are faulty in any way, including maliciously. A BFT cluster needs more nodes for the same resilience — four to tolerate one Byzantine fault, seven to tolerate two — and imposes more message overhead. For a consortium of mutually suspicious competitors, it closes a gap that Raft leaves open. Older protocols, Solo for development and Kafka for production, were deprecated in the 2.x line and removed in 3.0. Who runs the orderers is one of the most consequential governance decisions a consortium makes. If a single member or a vendor runs all of them, that party can refuse to order transactions, delay them, or — under Raft — reorder them within the constraints of the protocol. It cannot forge endorsements or alter committed blocks, because peers verify signatures and hashes independently, but it can censor. Distributing ordering nodes across several members removes that power from any one of them. Chapter 8 returns to this. The components introduced so far are summarised in Table 1, which shows what each does, who usually operates it, and what it is trusted for. Table 1. Core Fabric components and their roles. Component Main job Usually operated by Trusted to Certificate authority Issue identities for one organisation Each member organisation Vouch only for its own members Membership Service Provider Define an organisation and its roles Defined in channel configuration Map certificates to organisations Peer Hold ledger, run chaincode, validate blocks Each member organisation Execute and validate honestly for its owner Ordering node Sequence transactions into blocks Several members, or a neutral operator Order fairly; cannot forge or alter Chaincode Encode business rules Approved by members, run on peers Behave identically on every endorsing peer Client application Propose and submit transactions Each member, or its users Act for one identity Execute, order, validate The single most important thing to understand about Fabric is its transaction flow, because it differs fundamentally from that of most other blockchains and because nearly every design decision in a Fabric application follows from it. On Ethereum and most of its descendants, transactions are first ordered and then executed: every node receives the same sequence of transactions and runs each one in turn, updating its state. This is called order-execute. It is simple and it guarantees that every node reaches the same result, but it requires that every node execute every transaction, that execution be strictly deterministic, and that contracts be written in languages restricted enough to guarantee that determinism. Fabric inverts the order. Its flow is execute-order-validate, and it proceeds in three phases. In the execute phase, a client application builds a transaction proposal — essentially, "please call function ShipLot with these arguments" — signs it, and sends it to the endorsing peers required by the endorsement policy. Each of those peers runs the chaincode against its current copy of the world state. The chaincode does not change anything yet. Instead, the peer records what the chaincode read and what it would write: a list of keys read along with the version of each, and a list of keys to write with their new values. This pair is the read-write set. The peer signs it and returns it to the client as an endorsement. The client collects endorsements and checks that they agree. In the order phase, the client packages the proposal, the read-write set and the collected endorsements into a transaction and sends it to the ordering service. The orderers place it in sequence with other transactions, cut a block, and deliver the block to every peer on the channel. In the validate phase, every peer, independently, examines every transaction in the block. It checks two things. First, it checks the endorsements against the endorsement policy: did the right organisations sign, and are the signatures valid? Second, it checks the read set against its current world state: are the versions of the keys the chaincode read still the same as when it read them? If another transaction earlier in this block, or in a previous block, changed one of those keys in the meantime, the read set is stale and the transaction is marked invalid. This is called a multi-version concurrency control conflict, or MVCC conflict. Only transactions that pass both checks have their writes applied to the world state. Invalid ones are still recorded in the block, flagged as invalid, so the history is complete. Why the inversion matters Execute-order-validate has several consequences that shape every traceability application built on Fabric. The first is that only the endorsing peers run the chaincode. If the endorsement policy for a pharmaceutical application requires the manufacturer and the distributor to endorse a transfer, only their peers execute it. Other organisations on the channel validate the result by checking signatures and versions, but they never run the business logic. This improves scalability and, as Chapter 4 explains, makes some kinds of privacy possible. The second is that chaincode does not have to be written in a special language. Because results are checked by comparing endorsements rather than by re-executing on every node, chaincode can be written in Go, Java or JavaScript, using ordinary libraries. Determinism still matters — if two endorsing peers produce different read-write sets, their endorsements will not match and the transaction will fail — but the burden of guaranteeing it falls on the chaincode author and the endorsement check rather than on the language. The third is that trust is expressed in the endorsement policy. The question "how do we know this transaction is legitimate?" is answered not by "every node ran it and got the same answer" but by "the organisations named in the policy each ran it and signed the same answer". Who those organisations are, and how many of them are required, is a business decision encoded in configuration. That is the subject of the next chapter. The fourth is the concurrency model. Because transactions are executed against a snapshot and validated later, two transactions that touch the same key at nearly the same time will conflict, and one will fail. Applications that funnel many updates through a single key — a global counter, a single inventory record for a busy warehouse — will see high failure rates under load. Traceability data models therefore tend to favour many small keys, each updated rarely, over a few large ones updated constantly. Chapter 5 works through what this means in practice. Following one transaction It helps to watch the whole process once with concrete actors. Suppose a channel is shared by a drug manufacturer, PharmaCo, and a wholesale distributor, MedDist. The chaincode includes a function that records the transfer of ownership of a sealed case of medicine, identified by its serial number. The endorsement policy for that function requires signatures from both organisations. A warehouse clerk at PharmaCo scans the case as it is loaded onto a MedDist truck. PharmaCo's warehouse system, acting as a Fabric client with a certificate issued by PharmaCo's certificate authority, builds a proposal: call TransferCase with the case serial number, the new owner MedDist and a timestamp. It sends the proposal to one PharmaCo peer and one MedDist peer. Each peer runs TransferCase. The chaincode reads the case record, checks that its current owner is PharmaCo and its status is not "recalled" or "quarantined", and computes a new record with MedDist as the owner and the status "in transit". Each peer returns a signed endorsement containing the read set — the case key at version 41 — and the write set — the case key with the new value. The client checks that the two endorsements match, wraps them into a transaction, and submits it to the ordering service. A few hundred milliseconds later, the transaction appears in a block delivered to every peer on the channel. Each peer checks that the transaction carries valid signatures from a PharmaCo peer and a MedDist peer, satisfying the policy. It checks that the case key is still at version 41. If, in the meantime, a recall had been recorded against that case — changing its version — the transfer would fail validation, and the clerk's system would be notified. Otherwise, the write is applied, the case key moves to version 42, and both companies now hold an identical, signed, timestamped record that ownership passed from PharmaCo to MedDist. Notice what neither company had to do. Neither had to trust the other's database. Neither had to trust the operator of the ordering service with the business logic. Neither could later alter the record without the other noticing, because each holds the hash-linked chain of blocks. And a regulator who later asks either party for the case's history can be shown a record that both parties signed. Notice also what the ledger did not do. It did not verify that the clerk actually scanned the right case, or that the case contained what its label said. It recorded faithfully what two organisations agreed had happened. Everything Fabric guarantees is about the integrity of agreement, not the truth of the physical world. That gap, between what is recorded and what is real, is a theme that will return in every chapter that follows. Chapter 3: Endorsement Policies as a Trust Model The previous chapter ended with a transaction that both PharmaCo and MedDist had to sign before the network would accept it. That requirement did not come from Fabric. It came from a line of configuration that the consortium wrote. Change the line, and the same chaincode running on the same peers would accept transfers signed by PharmaCo alone, or by any two of five distributors, or by a distributor plus a regulator's observer node. The rule that decides which signatures make a transaction valid is the endorsement policy, and it is where a consortium's theory of who should be believed about what becomes machinery. This chapter explains how endorsement policies are written, where they live, how they interact with the chaincode lifecycle and with individual records, and how to design them. It also explains what they do not do, because the most common security mistakes in Fabric applications come from assuming that an endorsement policy guarantees more than it does. What an endorsement means Start with precision about what an endorsement is. When a peer endorses a transaction, it signs a statement that says, in effect: "I, a peer belonging to organisation X, executed this chaincode function with these inputs against my copy of the world state, and the result was this read-write set." The signature binds the result to organisation X. An endorsement policy is a rule over these signatures. It says which combinations of organisations' endorsements are sufficient for the network to treat a result as authoritative. When every peer validates a block, it checks each transaction's endorsements against the applicable policy. If the policy is not satisfied, the transaction is invalid, regardless of what it contains. Two consequences follow immediately. First, the policy is about organisations, not about the code. A transaction endorsed by PharmaCo's peer is trusted because PharmaCo is trusted, not because anyone has verified that PharmaCo's peer ran the agreed code. In Fabric 2.x, each organisation installs its own chaincode package on its own peers, and the platform only requires that the installed package match the agreed chaincode definition — its name, version, sequence number and policies. An organisation is free to install a different implementation, for example one that adds extra internal checks. It is equally free, in principle, to install one that cheats. The endorsement policy is the defence against that: if the policy requires two independent organisations, a single dishonest one cannot produce an accepted transaction alone. Second, the policy is about the peers that endorsed, not about the client that submitted. If the policy requires endorsement by PharmaCo and MedDist, then any client — including one belonging to a third organisation on the channel — can ask PharmaCo's and MedDist's peers to execute a transfer, and if both peers produce matching endorsements the transaction will be valid. The policy has said nothing about who is allowed to request a transfer. That check has to be written into the chaincode itself, which must inspect the identity of the submitting client and refuse to proceed if it lacks authority. Forgetting this is the classic Fabric vulnerability, and it is discussed further below. The policy language Fabric endorsement policies are written in a small expression language built from three operators and a set of principals. A principal names an organisation and a role, in the form 'PharmaCoMSP.peer', 'MedDistMSP.member' or 'RegulatorMSP.admin'. The role narrows which identities from that organisation count; for endorsement, peer is the usual choice, since endorsements come from peers. The operators are AND, OR and OutOf. AND('PharmaCoMSP.peer', 'MedDistMSP.peer') requires both. OR('PharmaCoMSP.peer', 'MedDistMSP.peer') requires either. OutOf(2, 'DistAMSP.peer', 'DistBMSP.peer', 'DistCMSP.peer') requires any two of the three. Expressions nest, so a policy can say "the manufacturer, plus any one of the three distributors": AND('PharmaCoMSP.peer', OR('DistAMSP.peer', 'DistBMSP.peer', 'DistCMSP.peer')) Policies written this way are called signature policies. Fabric also has implicit meta policies, which do not name organisations but refer to the policies of the organisations on the channel collectively. The most important is MAJORITY Endorsement, which is satisfied when a majority of the channel's member organisations have satisfied their own Endorsement sub-policy. Others include ANY and ALL over the same sub-policies. The distinction matters more than it first appears. A signature policy is fixed: it names specific organisations and keeps naming them until someone changes it. An implicit meta policy floats with membership: if a channel grows from three organisations to seven, MAJORITY Endorsement silently changes from "two of three" to "four of seven". Sometimes that is exactly what the consortium wants. Sometimes it produces surprises, such as a network whose throughput collapses because every transaction now needs four organisations' peers to be online, or a policy that newly admits a combination of junior members who were never meant to be able to act together. Where policies live Endorsement policies can be set at three levels, and a well-designed traceability application usually uses more than one. At the chaincode level, the endorsement policy is part of the chaincode definition. When a consortium agrees to deploy or upgrade a chaincode, each organisation's administrator runs an approve step for a definition that includes the policy, and once enough organisations have approved, one of them commits the definition to the channel. Every transaction that calls that chaincode is then checked against that policy, unless something more specific overrides it. If no policy is specified, the default is the channel's /Channel/Application/Endorsement policy, which in the standard configuration is MAJORITY Endorsement. At the collection level, private data collections — described fully in the next chapter — can carry their own endorsement policies. A collection holding confidential pricing between a manufacturer and one distributor might require only those two to endorse writes to it, regardless of the chaincode-wide policy. At the key level, Fabric supports what it calls state-based endorsement. Chaincode can attach an endorsement policy to an individual key, using the SetStateValidationParameter function. From then on, any transaction that writes that key must satisfy the key's policy, in addition to — or rather in place of — the chaincode-level one for that key. This is one of the most powerful and least used features in Fabric, and it fits traceability unusually well. Consider a serialised case of medicine. When the manufacturer creates its record, the right policy might be "the manufacturer alone", since nobody else knows about it yet. When the case is sold to Distributor A, the right policy for subsequent updates might become "Distributor A, plus the manufacturer for recall operations". When Distributor A sells it on to a pharmacy chain, the policy shifts again. State-based endorsement lets the chaincode change the policy on the case's key each time ownership changes, so that the rules for altering a record always track the organisation currently responsible for the physical item. Without it, the only options are a chaincode-wide policy that requires signatures from parties who have no involvement in a given case, or one so permissive that any single party can alter any case. Who gets asked: discovery and endorser selection A policy says which organisations must endorse. It does not, by itself, tell the client which peers to send a proposal to. Early Fabric applications hard-coded lists of endorsing peers, which broke whenever a peer was added, moved or taken down for maintenance. Fabric now provides a discovery service, run by the peers themselves, which a client can ask: for this chaincode on this channel, given its policies and any collections the function will touch, which sets of peers would satisfy the requirements? The client libraries and the Fabric Gateway use it automatically, choosing peers that are up and that have the relevant chaincode installed. Discovery has a subtle interaction with state-based endorsement. The discovery service knows the chaincode-level and collection-level policies, but it cannot know in advance which keys a function will write, and therefore cannot know which key-level policies will apply. If a pack's key carries a policy requiring Distributor B, and the client sends the proposal only to the peers that the chaincode-level policy requires, the endorsements will be insufficient and the transaction will fail at validation. Applications that use key-level policies must therefore tell the gateway which organisations to include for a given transaction, or must read the relevant key's current custodian before submitting. This is a modest burden, but it is one that teams adopting state-based endorsement routinely discover late. There is also a strategic dimension to endorser selection. Because the client chooses which peers to ask, a client facing an OutOf(2, ...) policy across five organisations can pick the two most likely to agree with it. If one organisation's peers are known to apply stricter internal checks — because, as noted above, each organisation may run its own implementation of the agreed chaincode — a client can simply avoid them. Threshold policies protect against a minority of dishonest endorsers; they do not ensure that any particular honest endorser is consulted. Where the consortium wants a specific party to see and vouch for every transaction of a certain kind, the policy must name that party with AND, not merely include it in a pool. The lifecycle is a policy too The chaincode lifecycle introduced in Fabric 2.0 deserves separate treatment, because it is an endorsement policy over code rather than over transactions, and it is one of Fabric's genuinely distinctive contributions to consortium governance. Before 2.0, a single organisation's administrator could instantiate or upgrade chaincode on a channel. That gave whoever held the administrator role enormous power: they could change the rules of the shared application unilaterally. In 2.0 and later, the process has four steps. Each organisation packages the chaincode and installs it on its own peers. Each organisation's administrator approves a chaincode definition — name, version, sequence number, endorsement policy, collection configuration and initialisation requirement. When enough approvals exist to satisfy the channel's LifecycleEndorsement policy, which defaults to MAJORITY Endorsement, any organisation can commit the definition, and the chaincode becomes active. The practical effect is that changing the business logic of a shared traceability application requires the same kind of collective agreement as changing a contract. A manufacturer cannot quietly deploy a new version of the transfer function that, say, skips the check for recalled cases. The new definition must be approved by a majority of members — or whatever the consortium has set as its lifecycle policy — and each of those members' administrators can inspect the code before approving. The lifecycle also governs the endorsement policy itself. Because the policy is part of the definition, tightening or loosening it is an upgrade that requires the same approvals. A consortium that wants to move from "manufacturer plus distributor" to "manufacturer plus distributor plus logistics provider" must collectively agree. Policy is not a setting an administrator toggles; it is a jointly held parameter of the shared system. Patterns that work Across supply chain deployments, a small number of endorsement patterns recur. Table 2 sets out the most common, with what each protects against and what it costs. Table 2. Common endorsement policy patterns in traceability networks. Pattern Example expression Protects against Cost or risk Single owner 'OwnerMSP.peer' Nothing beyond owner honesty One party can write anything it owns Bilateral AND(seller, buyer) Unilateral claims of transfer Both must be online for every transfer Owner plus witness AND(owner, OR(auditors...)) Owner collusion with itself Witnesses must run peers and stay available Threshold OutOf(3, five members) Any two colluding members Higher latency; complex to reason about Majority meta MAJORITY Endorsement Minority collusion Shifts silently as membership changes Key-level owner Set per record as custody moves Writes by non-custodians Chaincode must manage policies correctly The bilateral pattern is the natural fit for custody transfers. A transfer of goods from one party to another is, in the physical world, an event both parties experience: one hands over, the other receives. Requiring both to endorse the ledger record mirrors that. Neither can claim a transfer the other disputes. The cost is availability: if the buyer's peers are down, the seller cannot record the handover. In practice, many networks split the transfer into two transactions — a "ship" recorded by the sender, and a "receive" recorded by the recipient — each endorsed by its own author, with chaincode logic that will not let the item progress until both halves exist. That pattern preserves the evidential value of two-party confirmation while removing the requirement that both be online at the same moment. The owner plus witness pattern adds an independent party — an auditor, a trade association node, or in some proposals a regulator — whose peer must endorse alongside the owner. It protects against an owner fabricating records about its own goods, which is a real risk in sectors such as food and pharmaceuticals where an owner may have an incentive to conceal problems. Its cost is that the witness must run reliable infrastructure and must be trusted by the others not to censor. The threshold pattern is common for shared reference data that is not owned by any one party: a list of recognised product codes, a registry of licensed trading partners, a set of approved temperature profiles. For that kind of data, no single member should be able to change it, and the network should keep working if one or two members are temporarily absent. OutOf(3, ...) across five members provides both. Mistakes that recur Three mistakes account for most of the endorsement-related problems seen in real Fabric deployments. The first, already mentioned, is confusing endorsement with authorisation. A policy that says AND('PharmaCoMSP.peer', 'MedDistMSP.peer') tells the network which peers must sign. It does not say which clients may invoke the function. If the chaincode for TransferCase does not check that the submitting client belongs to the case's current owner, any channel member can ask both peers to execute a transfer of a case they do not own, and both peers will dutifully compute and sign the result. The fix is to inspect the client's identity inside chaincode. Fabric provides a client identity library for this; the Go version exposes functions such as GetMSPID and GetAttributeValue, which return the submitter's organisation and any attributes embedded in their certificate. Every state-changing function should begin by checking that the caller is entitled to make the change, in the same way that a web application checks that a logged-in user may edit a particular record. The second mistake is choosing a policy for its security properties alone and ignoring liveness. A policy of AND over all six members of a consortium is very hard to subvert. It also means that any one member's peer outage halts every transaction for everyone, and that any one member can veto any transaction simply by refusing to endorse. In a supply chain, where goods physically move whether or not the ledger records them, a network that cannot record events for six hours while one member patches its servers will quickly be abandoned in favour of spreadsheets. Policies should be chosen with an explicit view of which parties must be available for which operations, and with an explicit plan for what happens when they are not. The third mistake is leaving the defaults in place. The default chaincode endorsement policy is MAJORITY Endorsement and the default lifecycle policy is the same. For a three-member pilot, that means any two members can write any record and approve any code change. That may be acceptable; it should at least be a decision. Many projects discover only during a security review, or worse, during a dispute, that the policy they believed protected them was the default nobody had looked at. Endorsement policies as negotiated text It is useful to think of the set of endorsement policies in a Fabric network as a negotiated document, in the same category as the consortium's legal agreement. Each policy is a clause: "transfers of custody shall be recorded only with the confirmation of both parties"; "reference data shall be amended only with the agreement of three of the five founding members"; "recall notices may be issued by the manufacturer alone and shall take effect immediately". The lifecycle policy is the amendment clause: "these rules may be changed only with the approval of a majority". Seen this way, the work of designing endorsement policies is not primarily technical. It requires the people who negotiate the consortium agreement to understand what the policies mean, and the people who write the policies to understand what the agreement intends. When those two groups work separately — lawyers drafting a governance charter, engineers configuring a network — the two documents drift apart, and the network ends up enforcing rules nobody agreed to while failing to enforce the ones they did. The most successful consortia treat the policy file as an annex to the contract, reviewed by both sides, versioned alongside it, and changed only through the same process. Endorsement policies answer the question of who must vouch for a record. They say nothing about who may see it. On a Fabric channel, by default, every member sees every transaction. For a supply chain of competitors, that is rarely acceptable, and the next chapter turns to the mechanisms Fabric provides for keeping data confidential. Chapter 4: Channel Privacy and Private Data A traceability network is a peculiar thing: a system built to share information among organisations whose commercial interests depend on not sharing too much of it. A manufacturer wants its distributors to see the provenance of the goods they buy, but not to see what its other distributors buy. A freight forwarder wants a carrier to see the booking details it needs to load a container, but not the commercial invoice that shows the shipper's margin. A pharmacy chain wants a regulator to be able to verify that its stock is legitimate, but not to hand its dispensing volumes to the wholesaler down the road. Fabric's answer to this is a layered set of privacy mechanisms. Channels partition the network into separate ledgers. Private data collections keep particular data among particular organisations within a channel, leaving only a hash on the shared ledger. Off-chain storage, anchored by hashes, keeps bulky or highly sensitive documents out of the ledger altogether. Each mechanism has a different cost and a different failure mode, and choosing among them is one of the core design tasks in any Fabric project. Channels: separate ledgers A channel is the coarsest privacy boundary in Fabric. Each channel has its own ledger, its own world state, its own membership list, its own policies and its own chaincode definitions. Peers join channels individually, and a peer that has not joined a channel receives nothing from it: not the transactions, not the blocks, not even the knowledge of how many transactions there are. Two organisations that share a channel with each other and also share a different channel with a third party are, for practical purposes, running two independent ledgers that happen to use the same peer software. The channel's membership and rules are held in a special configuration block. It lists the organisations on the channel with their MSP definitions, the ordering nodes, the anchor peers, and the policies that govern who may read, who may write and who may change the configuration itself. Changing a channel's membership — adding a new distributor, removing a carrier that has left the consortium — means submitting a configuration update signed by enough administrators to satisfy the channel's Admins policy, which by default is a majority of member organisations. Channels give strong isolation. Their weakness is that they are coarse and they do not compose. Information on one channel cannot be read by chaincode on another channel within a transaction that writes, and there is no atomic transaction spanning two channels. If a manufacturer creates a separate channel for each of its twenty distributors, it gets excellent bilateral privacy, but its products' provenance is now spread across twenty ledgers. Tracing a case that moved from Distributor A to Distributor B — which happens all the time in pharmaceutical secondary markets — requires a process outside the ledger to join records from two channels. And each channel has to be created, configured, monitored and kept consistent. Networks that began with a channel per relationship have often found the operational cost unsustainable. A further point about channels is easily overlooked: the ordering service sees every transaction on every channel it serves. The orderers do not execute chaincode, but they receive the full transaction, including its read-write set. If a channel's transactions write a shipment's contents in plain text, the organisations operating the ordering nodes can read them. For a consortium whose ordering nodes are spread among its own members, that may be acceptable. For one that uses a vendor-run or a shared ordering service across several consortia, it may not be. Since version 2.3, Fabric has supported creating channels without a system channel, using a channel participation interface on each orderer; in 3.0 the system channel was removed entirely. This simplifies operations and means that an ordering service can participate in one channel without knowing about the others, which modestly reduces the information orderers hold. It does not change the basic fact that orderers see the transactions they order. Private data collections The finer-grained mechanism, introduced in Fabric 1.2 and considerably extended in 2.0, is the private data collection. A collection is a named subset of a channel's member organisations, defined as part of a chaincode's definition. Chaincode can write data to a collection instead of to the channel's public world state. When it does, the actual data travels only to the peers of the organisations in the collection, directly from peer to peer through Fabric's gossip protocol. What goes into the transaction — and hence to the orderers and to every other peer on the channel — is only a cryptographic hash of the keys and values. The result is a transaction that every channel member can see happened, and can verify was correctly endorsed, but whose contents only the collection's members can read. Those members store the private data in a separate private state database on their peers. Any member can later prove what the data was by revealing it to a third party, who can hash it and check the hash against the ledger. A collection is defined in a JSON configuration with a handful of fields. The most important are these. The policy lists the organisations whose peers may hold the data. requiredPeerCount sets how many other peers the endorsing peer must successfully send the data to before it returns an endorsement; setting it above zero ensures the data is not lost if the endorsing peer fails before the transaction commits. maxPeerCount limits how widely the endorser will push it. blockToLive sets how many blocks the data is kept before it is automatically purged from private state, leaving only the hash; zero means indefinitely. memberOnlyRead and memberOnlyWrite restrict chaincode access to clients from member organisations. And an endorsementPolicy can override the chaincode-level policy for writes to this collection. A short example shows the shape: { "name": "pricingPharmaCoDistA", "policy": "OR('PharmaCoMSP.member','DistAMSP.member')", "requiredPeerCount": 1, "maxPeerCount": 3, "blockToLive": 0, "memberOnlyRead": true, "memberOnlyWrite": true, "endorsementPolicy": { "signaturePolicy": "AND('PharmaCoMSP.peer','DistAMSP.peer')" } } Version 2.0 also introduced implicit collections: one per organisation, created automatically, named in the form implicitorg_ followed by the organisation's MSP identifier. Each holds data visible only to that organisation's peers. Implicit collections are useful for patterns where each party records its own confidential version of a shared event — for example, each side of a sale recording its own copy of the agreed price — and they avoid the need to define a collection for every pair of organisations. Using private data correctly Private data collections are powerful and they are also easy to misuse. Four details matter in nearly every traceability design. The first concerns transaction inputs. When a client sends a proposal to an endorsing peer, the function arguments are included in the proposal, and the proposal is included in the final transaction that goes to the orderers and to every peer. If a client calls RecordPrice with the price as an ordinary argument, the price ends up on the shared ledger in plain text, even though the chaincode writes it only to a private collection. The correct approach is to pass private inputs in the transient field of the proposal, which is available to the chaincode during execution but is not included in the transaction. Fabric's samples and documentation stress this point, but it remains a common error. The second concerns the strength of the hash. The hash that every channel member can see is a SHA-256 of the private value. That is cryptographically sound, but only if the value has enough unpredictability. If the private data is a temperature reading between two and eight degrees, a status that is one of five strings, or a price that falls within a narrow and known range, a curious channel member can simply hash every plausible value and see which one matches. This is a pre-image attack by enumeration, and it is trivially easy for low-entropy data. The fix is to add a random salt to the private value before writing it, and to share the salt only with collection members, so that the hash on the ledger cannot be reversed by guessing. The third concerns dissemination and availability. Private data travels by gossip to the peers of member organisations. If a member's peer is offline when the transaction is endorsed, it will try to fetch the missing private data from other members when it next sees the block that references it. If all the peers that held the data have since purged it, or are themselves unavailable, the data may be unrecoverable for that member. Setting requiredPeerCount to at least one, and ensuring that each member organisation runs more than one peer, reduces this risk. For data that must be retained for regulatory periods — and pharmaceutical traceability records under the United States regime must be kept for six years — the collection's blockToLive must be set accordingly, and members should have their own archiving arrangements rather than relying solely on peer storage. The fourth concerns deletion. For a long time, private data could be removed only by the automatic expiry set through blockToLive. Version 2.5 added an explicit purge function, PurgePrivateData, which removes a key's private value and its history from the private state of every peer that holds it, leaving only the hash on the ledger. This was added in part to address data protection requirements, such as the right to erasure under the EU's General Data Protection Regulation, that sit uneasily with immutable ledgers. It is an important capability for networks that hold personal data, but it should be noted what it does and does not achieve: the hash remains, and so does any copy a member has already exported outside the peer. Off-chain data and hashes Some data should not go on a ledger at all, even in a private collection. Scanned documents, photographs, sensor logs at one-minute resolution, full electronic data interchange messages — these are bulky, they would slow every peer that stores them, and they often carry legal or personal data that the parties want to manage under separate controls. The standard pattern is to store such data off-chain, in a document store, an object store or a member's own system, and to record on the ledger a hash of it together with a pointer to where it can be obtained and metadata about who created it and when. Anyone who later obtains the document can verify that it is identical to the one referenced on the ledger, and the ledger's timestamp and endorsements establish that the document existed in that form at that time and was vouched for by the endorsing organisations. This is sometimes called anchoring. Off-chain storage moves several problems out of Fabric rather than solving them. Access control for the documents must be managed separately. Availability depends on whoever hosts them. And if the host deletes or loses a document, the hash on the ledger proves only that something once existed. But for large, sensitive artefacts, it is usually the right trade-off, and it composes cleanly with both channels and private data: the hash itself can be written to a private collection if even the fact of the document's existence is sensitive. Choosing among the mechanisms The three mechanisms differ along the same few dimensions, and Table 3 sets these out side by side. Table 3. Privacy mechanisms compared. Property Separate channel Private data collection Off-chain with hash Who sees the data Channel members only Collection members only Whoever the host permits Who sees that a transaction occurred Channel members All channel members Depends where hash is written Orderers see contents Yes No, hash only No Cross-record logic in chaincode Within channel only Yes, for members No; data not in chaincode Operational cost High: per-channel setup Moderate: config per collection Separate storage to run Typical use Distinct business networks Prices, quantities, parties Documents, images, sensor logs Several principles emerge from experience with real networks. Use channels to separate genuinely distinct business networks — a consortium's production traceability network from its test network, or a pharmaceutical traceability application from a completely unrelated logistics one among the same members. Avoid using channels to give each pair of trading partners privacy, because the number of channels grows with the square of the number of members and because provenance that crosses pairs becomes fragmented. Use private data collections for the commercially sensitive attributes of otherwise shared events. A common design records a custody transfer on the public channel state — this case moved from manufacturer to distributor A at this time — while writing the price, the quantity of the overall order and the commercial terms to a collection containing only the two parties. Every channel member can then see that the case's chain of custody is unbroken, without seeing the commercial terms of any sale they were not part of. Consider carefully whether even the fact of an event is sensitive. A public record that "this case moved from PharmaCo to DistA" tells every channel member, including DistA's competitors, that DistA buys this product from PharmaCo. In some markets that is common knowledge; in others it is a trade secret. If it is a secret, the custody record itself may need to go into a collection, with only an opaque identifier on the public state, and the chaincode logic for verifying an unbroken chain must then run on the peers of parties who can see the relevant records. Use off-chain storage for documents and large data, always with a hash and metadata on the ledger, and choose where to write that hash according to who should know the document exists. Metadata leaks and identity Even a carefully designed collection leaks something. Every transaction on a channel carries the identity of the client that submitted it and the identities of the peers that endorsed it, and every channel member can see those identities. A transaction that writes only hashes to the public state, with all the content in a collection shared by PharmaCo and DistA, still tells every other member that PharmaCo and DistA transacted, at what time, through which chaincode function. Over months, the timing and frequency of those transactions can reveal a good deal: seasonal volumes, the onboarding of a new customer, the loss of an old one. Several partial remedies exist. Function names can be made generic, so that the name alone does not reveal what kind of event was recorded. A neutral party, such as a consortium-operated service, can submit transactions on members' behalf, so that the submitter's identity on the ledger is the service rather than the member — at the cost of trusting that service. Fabric also supports an identity scheme called Identity Mixer, based on anonymous credentials, which allows a client to prove that it belongs to an organisation, or holds an attribute, without revealing which specific identity it is and without making its transactions linkable to one another. Identity Mixer has been part of Fabric for years but remains little used in production, partly because of limitations in tooling and partly because most supply chain applications actually want transactions to be attributable to a named party. Its main practical use is for end users — patients, consumers, individual drivers — rather than for organisations. The honest position is that a Fabric channel reveals patterns of activity among its members to all of them, and a consortium should decide explicitly whether that is acceptable. If it is not, the relevant activity belongs on a separate channel or off the ledger. Privacy and verifiability pull in opposite directions There is an underlying tension in all of this that no mechanism removes. The value of a shared ledger comes from many parties being able to verify the same record. Privacy comes from fewer parties seeing it. Every time data moves from the public channel state into a private collection, fewer organisations can independently verify it, and the network's assurance about that data rests more heavily on the honesty of the collection's members. In the extreme case, a network that puts everything into bilateral collections offers little more than a set of point-to-point messages with timestamps — which may be enough, but should be recognised for what it is. In the other extreme, a network that puts everything on public channel state will not be joined by anyone with commercial secrets. The design task is to place each piece of data at the point on that spectrum where the consortium's need for shared verification and each member's need for confidentiality are both acceptably met, and to write those placements down as explicitly as the endorsement policies. The mechanisms of this chapter and the previous one — who must sign, who may see — come together in chaincode, where they are applied to specific records about specific products. The next chapter is about designing those records. Chapter 5: Chaincode and the Data Model of Provenance Endorsement policies decide who must vouch for a record and privacy mechanisms decide who may see it. Neither decides what the record is. That is the job of chaincode, and in a traceability network the most important decisions in chaincode are not about algorithms but about data: what entities exist, how they are identified, what events change them, and how the history can be queried afterwards. A traceability ledger with a poor data model can be perfectly secure and perfectly useless, recording thousands of transactions a day that cannot answer the question a regulator asks during a recall. This chapter treats chaincode design for provenance as a modelling discipline first and a programming task second. It covers the identifiers and event vocabulary that supply chains already use, how to map them onto Fabric's key-value world state, the determinism and concurrency rules that Fabric's transaction flow imposes, and the patterns for querying history that make the ledger useful once it is full. Start with the vocabulary the industry already has The single most consequential decision in a traceability data model is whether to invent one or to adopt an existing standard. The answer, almost always, is to adopt one, and for physical goods the standard is GS1. GS1 is the global, not-for-profit standards body behind the barcode. Its identifiers are already printed on most traded goods and already used in most enterprise systems. Four matter for traceability. The Global Trade Item Number, or GTIN, identifies a product type at a particular packaging level — a box of thirty tablets of a given strength, say. A GTIN combined with a serial number, called a serialised GTIN or SGTIN, identifies one specific pack. The Serial Shipping Container Code, or SSCC, identifies a logistics unit such as a pallet or a case assembled for shipment. The Global Location Number, or GLN, identifies a party or a physical location: a factory, a warehouse dock, a pharmacy. Alongside identifiers, GS1 publishes EPCIS, the event standard mentioned in Chapter 1, and its companion Core Business Vocabulary. EPCIS describes supply chain activity as a stream of events, each answering four questions: what (which identifiers were involved), when (the time and time zone), where (the location where the event occurred and, separately, where the goods are now), and why (the business step, such as shipping, receiving or packing, and the disposition, such as in transit, active or recalled). The Core Business Vocabulary supplies standard values for the business steps and dispositions so that "shipping" means the same thing to every participant. EPCIS defines a handful of event types. An object event records something happening to identified objects: they were commissioned, observed, shipped, received or destroyed. An aggregation event records objects being packed into or unpacked from a container: these forty-eight packs went into this case. A transformation event records inputs consumed to produce outputs, as when bulk product is packaged or raw materials are processed. A transaction event associates objects with a business transaction such as a purchase order. EPCIS 2.0, ratified by GS1 in 2022, added an association event for linking assets such as sensors to objects, native support for sensor readings, and a JSON-LD representation that is far easier to work with in modern software than the original XML. Adopting this vocabulary has three benefits. Participants' existing systems already generate much of the data, so integration is cheaper. The data model has been argued over by industry for two decades and handles edge cases — split shipments, returns, rework, destruction — that a freshly invented model will miss. And regulators in both the United States and Europe already refer to GS1 identifiers, so records mapped onto them are easier to use in compliance. Mapping events onto world state Fabric's world state is a key-value store, and EPCIS is a stream of events. The two do not map onto each other automatically, and how the mapping is done determines the performance, privacy and queryability of the whole application. The tempting approach is to store each event as a key and do nothing else. It preserves history perfectly and never produces concurrency conflicts, because each event is a new key. But it pushes all the work of answering "where is this pack now and who owns it?" onto the reader, who must find and replay every event touching the pack. It also gives chaincode nothing to check new events against: the chaincode cannot refuse to record a "shipped" event for a pack that has already been destroyed if it does not maintain the pack's current status. The more useful approach maintains two kinds of record. Each identified item — each SGTIN, each SSCC — has a state record holding its current owner, location, status and parent container if any. Each event is also recorded, either as its own key or simply as the transaction itself, with the state records it changed. Chaincode functions then follow a pattern: read the current state of the items involved, check that the proposed event is allowed given that state and the caller's identity, write the event, and update the state. The blockchain preserves every version of every state record, so history is never lost, and the world state always holds the current position. Keys need careful design. Fabric provides composite keys, built from an object type and a list of attributes, which allow range queries over a prefix. A pack's state might be stored under a composite key of type pack with attributes GTIN and serial number. An event might be stored under type event with attributes for the GTIN, serial and event timestamp, so that all events for a given pack can be retrieved with a single partial-key query. In Go, the relevant functions are CreateCompositeKey and GetStateByPartialCompositeKey. Range queries over composite keys are re-checked at validation time for phantom reads, which makes them safe to use in logic that determines what a transaction writes. The aggregation problem Aggregation deserves special attention, because it is where naive traceability data models most often break. A typical pharmaceutical pallet holds dozens of cases, each holding dozens or hundreds of packs. A single pallet may thus contain several thousand serialised items. When the pallet is shipped, every item on it is shipped. If the data model requires updating the state record of every pack each time the pallet moves, a single shipment becomes a transaction writing thousands of keys. That is slow, it creates large blocks, and it multiplies the chance of a concurrency conflict with any other transaction touching any of those packs. The standard solution is the same one EPCIS uses: record containment once, and let movement apply to the outermost container. When packs are aggregated into a case, the chaincode records, for each pack, that its parent is the case. When cases are aggregated into a pallet, it records that their parent is the pallet. When the pallet ships, only the pallet's state record is updated. To find where a pack is, the reader walks up from the pack to its case to its pallet and reads the pallet's location. When the pallet is broken down at a distribution centre, a disaggregation event clears the parent links and the cases begin moving independently. This design reduces a shipment from thousands of writes to one, at the cost of slightly more complex reads. It also mirrors the physical world, where the pallet is the thing that is scanned on the loading dock. It does require discipline: every disaggregation must be recorded, or packs will appear to be in a pallet that has long since been broken up. Missing disaggregation events are one of the most common data quality problems in real traceability systems, on and off the ledger, and chaincode can help by refusing to accept an event about a child item whose recorded parent is somewhere else. Determinism and the transaction flow Chapter 2 explained that Fabric endorsing peers execute chaincode independently and their results must match. That requirement imposes a set of rules on chaincode that are easy to state and easy to break. Chaincode must not read the system clock. Two peers will read slightly different times, their write sets will differ, and the endorsements will not match. The correct source of a timestamp is the transaction's own timestamp, set by the client in the proposal and available through GetTxTimestamp. The event time recorded in EPCIS terms — when the physical event happened — is a separate value supplied by the client as an argument, and the chaincode can check it for plausibility against the transaction timestamp. Chaincode must not generate random numbers, call external services, or read files that might differ between peers. A function that calls a currency-conversion service or a product master database will see different answers on different peers at different moments. If external data is needed, it should be supplied by the client as an input — ideally signed by its source so the chaincode can verify it — or written to the ledger by a prior transaction. Chaincode must not depend on the iteration order of unordered collections. In Go, iterating over a map yields keys in a randomised order, and a function that builds its output by iterating a map can produce different serialisations on different peers. JSON serialisation of the same struct can also differ between language libraries; peers running chaincode written in different languages, or the same language with different library versions, should be tested together before deployment. Finally, chaincode must be written with MVCC conflicts in mind. If two transactions in the same block read and write the same key, the second will fail. In traceability this is usually manageable, because most items are touched rarely. The danger lies in shared counters and aggregate records. A key holding "total cases shipped today" that every shipment increments will produce a conflict on nearly every concurrent shipment. The general rule is to avoid shared mutable keys on hot paths: compute aggregates off-chain from the event stream, or use one key per contributor and sum them at read time. A short Go fragment shows the shape of a well-behaved state-changing function: func (c *Contract) ShipCase(ctx contractapi.TransactionContextInterface, sscc string, toGLN string, eventTime string) error { caller, _ := ctx.GetClientIdentity().GetMSPID() key, _ := ctx.GetStub().CreateCompositeKey("unit", []string{sscc}) raw, err := ctx.GetStub().GetState(key) if err != nil || raw == nil { return fmt.Errorf("unknown unit %s", sscc) } var u Unit json.Unmarshal(raw, &u) if u.OwnerMSP != caller { return fmt.Errorf("caller does not own unit") } if u.Status == "recalled" || u.ParentSSCC != "" { return fmt.Errorf("unit cannot ship in current state") } u.Status = "in_transit" u.Destination = toGLN u.LastEventTime = eventTime out, _ := json.Marshal(u) return ctx.GetStub().PutState(key, out) } The function checks the caller's organisation against the recorded owner, refuses to ship a recalled unit or one still packed inside a larger container, uses a client-supplied event time rather than the clock, and writes a single key. A production version would also validate the inputs, record the event itself, emit a chaincode event and handle errors from unmarshalling, but the essential pattern — identify the caller, read the state, check the transition, write the new state — is all there. State machines, not free-form updates The pattern in that fragment generalises. The safest way to write traceability chaincode is to model each kind of item as a state machine with explicit allowed transitions. A pack may go from commissioned to packed to shipped to received to dispensed or destroyed. It may go to recalled or quarantined from almost any state, but only on the instruction of specific parties. It may not go from dispensed back to shipped. Every chaincode function corresponds to one or more transitions, and each function checks that the current state permits the transition and that the caller is entitled to make it. This approach turns the chaincode into an enforceable specification of the consortium's business process. It catches a large class of errors and frauds at the moment they are attempted: a pack that has already been dispensed cannot be "received" by a second pharmacy, which is precisely the signature of a diverted or counterfeit pack carrying a copied serial number. And it makes the chaincode reviewable by non-programmers, because the transitions can be written out as a table in the consortium's operating rules and checked line by line against the code. Generic "update any field" functions, by contrast, are the source of many traceability failures. They are easy to write and easy to integrate, but they let any authorised caller put any item into any state, reducing the ledger to a shared spreadsheet with signatures. Reading the history back A traceability ledger earns its keep at the moment someone asks a question: where did this pack come from, where are the other packs from the same lot, which customers received product from this contaminated batch? Fabric offers several ways to answer such questions, with different properties. History queries, through GetHistoryForKey, return every committed version of a single key along with the transaction identifier and timestamp of each change. For a state record, this gives the full sequence of owners, locations and statuses. It is the simplest way to show a pack's provenance and requires no extra modelling. It must be enabled on the peer, which it is by default. Partial composite key queries return all keys sharing a prefix, as described above. They are efficient, safe for use in transaction logic, and well suited to "all events for this item" or "all items in this lot" if the keys were designed for it. Rich queries, available only with CouchDB as the state database, allow selection on the contents of JSON values: all packs with a given lot number and a status of shipped. They are flexible and convenient, but rich query results are not re-validated at commit time. A transaction that uses a rich query to decide what to write can be affected by a phantom read: another transaction may have added a matching record between execution and validation, and Fabric will not detect it. Rich queries are therefore appropriate for read-only queries issued by applications, not for logic inside state-changing transactions. Rich queries over large datasets also require CouchDB indexes, which are packaged with the chaincode. For anything beyond simple lookups — a recall that must find every downstream recipient of a lot across thousands of packs, or analytics across months of events — the standard pattern is to stream ledger events off-chain. Chaincode can emit events with SetEvent, and client applications can subscribe to block and chaincode events through the Fabric Gateway, the client-facing service introduced in version 2.4. A member's own indexer listens to committed blocks, decodes them and writes the data into a database designed for querying, such as a relational store or a graph database. Because each member builds its indexer from the blocks on its own peer, each member's query database reflects exactly the data that member is entitled to see, and each can verify its indexer against the ledger. Deployment and maintenance Two practical points complete the picture. First, Fabric supports running chaincode as an external service, known as chaincode as a service, rather than having the peer build and launch it in a container. Most production networks, especially those running on Kubernetes, now use this approach because it fits standard deployment pipelines and separates the chaincode's runtime from the peer's. Second, chaincode upgrades go through the lifecycle described in Chapter 3. Every change to business logic requires a new definition approved by the consortium, so the data model should be designed for forward compatibility from the start: versioned record formats, optional fields that older code ignores, and migration functions that can be run once after an upgrade. A traceability ledger is meant to hold records for years. Its data model will outlive several generations of the code that writes it. With the machinery and the modelling discipline in place, the book turns to the two domains it set out to address. The first is the movement of medicines. Chapter 6: Tracking Pharmaceutical Provenance Medicines are the clearest case for supply chain traceability anywhere in the economy. The goods are small, valuable and easy to counterfeit. The consequences of a fake are not a disappointed customer but a patient who receives no active ingredient, the wrong one, or a toxic contaminant. The supply chains are long and international, with active ingredients often made in one country, finished doses in another and packaging in a third, and with a secondary wholesale market in which product may change hands several times before it reaches a pharmacy. And regulators in the major markets have, over the past fifteen years, imposed legal obligations that require serialisation of individual packs and, in varying forms, the ability to verify and trace them. This makes pharmaceuticals the domain in which blockchain traceability has been most seriously studied, piloted and argued over. It is also the domain in which the case for a shared ledger has been tested most directly against the alternatives, and in which the alternatives have, for the core regulatory obligations, mostly won. Understanding why, and where Fabric nevertheless fits, is the best possible lesson in when a permissioned ledger is the right tool. Two regulatory regimes, two architectures The two largest regulated markets, the United States and the European Union, have both required pack-level serialisation, but they have built strikingly different systems on top of it. The United States Drug Supply Chain Security Act, or DSCSA, was enacted in November 2013 as Title II of the Drug Quality and Security Act. It set a ten-year schedule for building an electronic, interoperable system to identify and trace prescription drugs at the package level as they move through the supply chain. Manufacturers were required to print a product identifier on each package — a two-dimensional Data Matrix barcode encoding the National Drug Code, a serial number, the lot number and the expiry date — and trading partners were required to exchange transaction information and transaction statements with each sale, verify product identifiers when investigating suspect product, and verify returned product before resale. The final stage, the enhanced drug distribution security requirements, took effect on 27 November 2023: from then on, the transaction information was to be exchanged electronically, at the package level, in an interoperable way. The industry was not fully ready, and the Food and Drug Administration announced a one-year stabilisation period to November 2024, followed by exemptions staggered by trading partner type. Manufacturers and repackagers were given until May 2025, wholesale distributors until August 2025, larger dispensers until November 2025, and small dispensers — pharmacies with twenty-five or fewer full-time employees — longer still; in 2025 the FDA extended the small dispenser exemption to November 2027. Crucially, DSCSA does not prescribe an architecture. It requires interoperable electronic exchange, and leaves the industry to decide how. The industry has largely settled on a distributed, point-to-point model: each trading partner sends EPCIS events describing a shipment to its customer at or before the time of the shipment, and each holds its own records. Verification requests — "is this serial number one you actually made?" — are routed to the manufacturer through a lookup directory and a set of verification router services operated by solution providers. A body called the Partnership for DSCSA Governance was formed in 2020 to develop shared standards and governance for the interoperable system, including the credentials trading partners use to prove to one another that they are authorised to trade. The European Union's Falsified Medicines Directive, adopted in 2011, and its Delegated Regulation of 2016 took the opposite path. From 9 February 2019, most prescription medicines sold in the EU had to carry a unique identifier and an anti-tampering device. Manufacturers upload each pack's identifier to a central hub operated by the European Medicines Verification Organisation, a non-profit body run by the stakeholder associations, which distributes the data to national repositories run by national medicines verification organisations. At the point of dispensing, the pharmacy scans the pack, the national system verifies that the identifier is genuine and has not already been dispensed, and the pack is decommissioned. Wholesalers verify packs in certain higher-risk situations, such as returns and purchases from non-manufacturer sources. The EU system is an end-to-end verification model: it checks authenticity at the start and the end, and largely ignores the path in between. The US system is a track-and-trace model: it records each change of ownership along the path. Table 4 compares the two. Table 4. Pharmaceutical serialisation regimes in the United States and European Union. Feature United States (DSCSA) European Union (FMD) Legal basis Drug Quality and Security Act, 2013 Directive 2011/62/EU; Regulation 2016/161 Core model Track ownership changes along the chain Verify at origin and at dispensing Data location Each trading partner's own systems Central EU hub and national repositories Exchange format EPCIS events between trading partners Uploads and queries to repositories Operator No single operator; many providers Stakeholder-run non-profit bodies Path between endpoints Recorded at every sale Largely not recorded Where the ledger was supposed to fit During the long build-up to the 2023 DSCSA deadline, permissioned blockchains were widely proposed as the interoperable system the law required. The argument was intuitive. The law demanded a shared, trustworthy, electronic record of each package's movement among trading partners who compete with one another, and no neutral operator existed. That is the situation Chapter 1 described as the case for a permissioned ledger. The FDA ran a pilot programme in 2019 to explore technologies for the enhanced requirements, and several of the pilots used blockchain. The best known involved IBM, KPMG, Merck and Walmart, and used a Fabric-based network to share package-level data among participants; its participants reported in 2020 that a permissioned network of this kind could support the interoperable, package-level exchange the law envisaged, while acknowledging that governance and broad industry participation remained open questions. Other industry groups, such as the MediLedger project convened by the technology company Chronicled with a number of large manufacturers and wholesalers, built blockchain-based networks for specific DSCSA functions, including verification of returned product and credentialing of trading partners. Yet the industry's main interoperability path turned out not to be a shared ledger. There were several reasons, and each is instructive. The first was that the data was too sensitive to share broadly, even in a permissioned network. A wholesaler's DSCSA transaction information reveals its customers and volumes. Under the point-to-point model, it goes only to the immediate customer, which already knows it. Under a shared ledger, even with private data collections, the wholesaler would need to trust the design, the configuration and the other operators to keep it contained. Many decided the risk was not worth it. The second was that the law required exchange, not a shared record. DSCSA obliges each partner to receive and keep transaction information about the products it handles, and to provide it on request. It does not require anyone to maintain a shared, global history of each package. A point-to-point model with good record-keeping meets the obligation; a shared ledger exceeds it, at greater cost. The third was the size and diversity of the population. The US pharmaceutical supply chain includes a few large manufacturers and wholesalers and tens of thousands of pharmacies, many of them small independent businesses with no IT staff. Asking each to run, or even to subscribe to, nodes on a permissioned ledger was a significant burden. Asking each to receive EPCIS files through a service provider they already used was much easier. The fourth was that point-to-point messaging already existed. Large wholesalers and manufacturers had been exchanging advance shipping notices and EPCIS data for years. The incremental path from those systems to DSCSA compliance was shorter than the path to a new shared platform. None of this means the ledger approach was wrong in principle. It means that for the specific obligations DSCSA imposed, among the specific population it covered, the alternatives were cheaper and good enough. That is precisely the test Chapter 1 proposed. Where Fabric does fit in pharmaceuticals The more promising uses of permissioned ledgers in pharmaceuticals are those where the alternatives genuinely fall short: narrower networks with a smaller number of more capable participants, more demanding requirements than the law's minimum, and a need for multi-party records rather than bilateral ones. High-value and specialty products are the clearest example. Cell and gene therapies, for instance, involve collecting cells from a specific patient, shipping them to a manufacturing site, modifying them and shipping them back for infusion into the same patient. The chain of identity — the guarantee that the cells returned are that patient's cells — and the chain of custody, including tight temperature control throughout, involve the hospital, couriers, the manufacturer and sometimes third-party laboratories. Every party needs to see the same record, in real time, with clear attribution. That is a small, capable, multi-party network with no natural central operator, and it fits the permissioned ledger model well. Cold chain assurance for temperature-sensitive biologics and vaccines is another. A shipment's temperature history may be recorded by loggers belonging to the shipper, the carrier and the consignee, and disputes about who was responsible for an excursion are common and expensive. A ledger on which each party's logger readings, or hashes of them, are endorsed by the party responsible at that point in the journey gives a shared, attributable record that all can rely on in a dispute. Recall coordination across a network is a third. When a manufacturer recalls a lot, it needs to reach every party that holds packs from that lot, and those parties need to confirm what they hold and what they have done with it. In a point-to-point system, the manufacturer knows its direct customers, who must in turn notify theirs. In a shared network, the recall can be recorded once, every holder of an affected pack can be identified from the custody records, and the response from each — quarantined, returned, destroyed — can be tracked on the same ledger. Saleable returns verification and trading partner credentialing have both been addressed by ledger-based services. The verification of a returned pack requires a wholesaler to confirm with the manufacturer that the serial number is genuine; credentialing requires each party to prove to others that it holds a valid licence. Both are problems of shared reference data among known parties, which fits the threshold-endorsed pattern described in Chapter 3. A design for a specialty pharmaceutical network To make this concrete, consider how a consortium for a specialty product line might use Fabric. The participants are three manufacturers, two specialty distributors, a specialist logistics provider, and a group of hospital pharmacies. The goals are to record pack-level custody among them, to support rapid recall, to share cold chain records, and to provide a single place where any participant can verify that a pack is genuine and has not already been dispensed. The network uses one channel for all participants, because provenance crosses every pair of members and because a single channel keeps recall logic simple. Each member runs at least two peers. The ordering service has five Raft nodes, one at each manufacturer and one at each distributor, so no single member controls ordering. The public channel state holds, for each pack and each logistics unit, a minimal state record: the identifier, the current custodian's organisation, the status in the state machine described in Chapter 5, and the parent container. Serial numbers are stored in the clear on the public state because the consortium has decided that knowing which pack is where is the point of the network, but commercial terms are not. Transaction information for each sale — price, order quantity, the full DSCSA transaction statement — is written to the implicit collections of the seller and buyer, so that each holds its own copy and no one else sees it. A hash of the transaction information is written to the public state alongside the custody transfer, so that either party can later prove what was agreed. The endorsement policies follow the custody model. Commissioning a pack, which brings it into existence on the ledger, requires the manufacturer's endorsement alone, and chaincode checks that the caller's MSP matches the manufacturer named in the product master data. Shipping requires the endorsement of the current custodian. Receiving requires the endorsement of the recipient, and chaincode refuses to record a receipt for a pack that has not been shipped to that recipient. Through state-based endorsement, each pack's key carries a policy naming its current custodian, updated on each receipt, so that only the custodian's peer can endorse changes to it. Recall can be recorded by the manufacturer alone, overriding the custodian policy through a separate key-level policy reserved for the manufacturer, because the consortium has agreed that a manufacturer must be able to stop distribution of its product immediately. Dispensing or decommissioning requires the endorsement of the dispensing pharmacy. Cold chain data from loggers is stored off-chain by the logistics provider, with hashes and summary flags — "excursion detected" or "within range" — written to the public state for each leg of the journey, endorsed by the provider and the custodian of that leg. Each member runs an indexer that reads committed blocks into its own query database. When a recall occurs, the manufacturer's indexer identifies every pack from the affected lot and its current custodian; the recall transaction updates the status of each affected lot's packs; each custodian's application is notified through chaincode events and responds by recording quarantine or return. This design does not replace the members' DSCSA obligations. It supplements them, and it can generate the EPCIS messages each member must send under the point-to-point model from the events on the ledger. What it adds is a single, shared, attributable history that every member can verify, which is what the specialty context requires. The physical-digital gap, again All of this depends on something no ledger can guarantee: that the serial number on the screen corresponds to the pack in the hand. A counterfeiter who copies a genuine serial number onto a fake pack will produce a pack that verifies as genuine — until the real one is also scanned, at which point one of them will be flagged as a duplicate. The state machine described in Chapter 5, which refuses to dispense a pack already dispensed or receive one already received elsewhere, catches exactly this. But it catches it only if every legitimate event is recorded, promptly and accurately. That is why the operational discipline of scanning, aggregation and disaggregation matters more than the choice of ledger. The common experience of serialisation programmes on both sides of the Atlantic is that data quality — missing aggregation records, mismatched serial numbers between shipment notices and physical stock, delayed uploads — is the dominant cause of exceptions. A ledger makes inconsistencies more visible, because every party sees the same state, and that visibility is valuable. It does not make them disappear. The next chapter turns to a domain where the documents, rather than the goods, are the centre of the problem. Chapter 7: International Shipping Manifests and Trade Documents The pharmaceutical case is about objects: billions of individually serialised packs whose identity and custody must be tracked. International shipping turns the problem inside out. The objects — containers — are already uniquely identified, carry their numbers painted on their sides, and are tracked fairly well by the carriers and terminals that handle them. What is hard to track is the paper. A single container crossing an ocean generates a thicket of documents: bookings, shipping instructions, bills of lading, commercial invoices, packing lists, certificates of origin, phytosanitary and veterinary certificates, dangerous goods declarations, customs declarations at both ends, arrival notices, delivery orders. Each is created by a different party, needed by several others, and copied, re-keyed, amended and forwarded many times before the box is released to its consignee. In 2017, when they announced their joint venture, Maersk and IBM described a study in which they had followed a single refrigerated shipment from East Africa to Europe and found that it involved nearly thirty organisations and more than two hundred separate communications. They estimated that the cost of processing trade documentation could approach a fifth of the cost of the physical transport. Whatever the precise figure for a given lane, every practitioner recognises the picture: an industry that moves goods with great efficiency and moves information about them with remarkable friction. This chapter considers how a permissioned ledger can address that friction, what has been tried, and why the most important recent progress has come from law and standards rather than from technology. The documents and their peculiarities Three documents sit at the centre of the problem, and each has properties that matter for ledger design. The bill of lading is the oldest and the strangest. Issued by the carrier when it takes the goods into its charge, it does three things at once. It is a receipt, recording what the carrier received and in what apparent condition. It is evidence of the contract of carriage. And, in its negotiable form, it is a document of title: whoever holds the original paper document is entitled to take delivery of the goods at the destination, and transferring the paper by endorsement and delivery transfers that right. Banks financing trade under letters of credit rely on this, taking the bill of lading as security until the buyer pays. Because possession of a unique original is what confers the right, a bill of lading cannot simply be emailed. A scanned copy is a picture of a document of title, not a document of title. The cargo manifest is the carrier's list of everything loaded on a vessel, filed with customs authorities in advance of arrival so they can assess risk. Customs authorities have steadily tightened the timing and detail they require. The United States requires ocean carriers to file manifest data twenty-four hours before loading at the foreign port, and importers to file additional security data. The European Union's Import Control System 2, rolled out in phases since 2021, requires detailed advance safety and security data from carriers and, in many cases, from other parties in the chain such as freight forwarders, so that information the carrier does not hold — the actual seller and buyer, for example — reaches customs directly. A manifest is thus increasingly a multi-party document assembled from data held by different actors. The commercial invoice and packing list, created by the seller, describe the goods, their value and their packing. They are needed by customs for valuation and classification, by banks for trade finance, and by the buyer, and they contain information — prices, margins, supplier identities — that the seller does not want everyone else in the chain to see. Around these sit dozens of other documents, each with its own issuer and audience. Table 5 summarises the main ones and their implications for a ledger design. Table 5. Principal shipping documents and their treatment on a shared ledger. Document Issued by Needed by Sensitivity Ledger treatment Booking confirmation Carrier Shipper, forwarder Moderate Event on channel; details private Bill of lading Carrier Holder, banks, consignee High; title Unique record with controlled transfer Commercial invoice Seller Buyer, customs, bank High Off-chain; hash in private collection Cargo manifest data Carrier, forwarder Customs authorities Moderate Assembled from events; shared with customs Certificate of origin Chamber or authority Customs, buyer Low to moderate Issued by authority node; hash anchored Container events Terminal, carrier Most parties Low Public channel state What TradeLens built TradeLens remains the most ambitious attempt to put this information onto a shared ledger, and its architecture is worth understanding in some detail. The platform was built on Hyperledger Fabric and operated by IBM and Maersk. Participants included ocean carriers, ports and terminal operators, inland transport providers, freight forwarders, customs brokers, shippers and customs authorities. At its peak it reported participation by several of the world's largest container lines, which together accounted for a substantial share of global container capacity, as well as more than a hundred ports and terminals. Its data fell into two broad categories. Shipment events — gate in, loaded, departed, arrived, discharged, gate out — were published by the parties that observed them, and were visible to the parties with an interest in the shipment. These events were valuable even without any document exchange, because they gave shippers and forwarders a consolidated view of where their containers were without having to query each carrier and terminal separately. Documents were exchanged through a document-sharing service with permissions set by the document's owner, with hashes and metadata recorded on the ledger so that the version history was tamper-evident. TradeLens later added an electronic bill of lading capability. The ledger's role was narrower than its publicity suggested. Much of the data lived in conventional databases operated by the platform, with the blockchain used to record events, hashes and permission changes and to give participants who chose to run peer nodes an independently verifiable copy of the records relevant to them. Carriers could run their own nodes; many participants simply used the platform's interfaces. This hybrid design was a pragmatic response to the reality that most of the parties in a shipping chain had neither the capability nor the inclination to operate blockchain infrastructure. What doomed TradeLens was not its architecture. Other carriers were reluctant to participate fully in a platform so closely associated with the largest of their competitors, and the platform's operators were never able to persuade them that it was genuinely neutral. Without the full participation of the carriers, the network effects on which its value depended never materialised, and without them, shippers and forwarders had insufficient reason to pay. Chapter 8 returns to this governance lesson. For the purposes of this chapter, the important point is that the technical patterns TradeLens used — events visible to interested parties, documents off-chain with hashes on-chain, permissions controlled by document owners — remain sound. The electronic bill of lading problem The electronic bill of lading illustrates a point that applies well beyond shipping: a ledger cannot create legal effect that the law does not recognise. Electronic bills of lading have existed for decades. Bolero launched a rulebook-based system in the late 1990s, and several other platforms followed. They worked by having all participants sign up to a contractual framework under which an electronic record would be treated as equivalent to a paper bill. That works among members, but it does not bind third parties who have not signed, and courts in many jurisdictions did not recognise the electronic record as a document of title in its own right. Adoption remained small: industry estimates have generally put electronic bills at a low single-digit percentage of the total. The legal picture has begun to change. In 2017 the United Nations Commission on International Trade Law adopted the Model Law on Electronic Transferable Records, known as MLETR, which sets out the conditions under which an electronic record can perform the function of a transferable document such as a bill of lading. Its central requirement is that a reliable method be used to identify the record as the one that is capable of being subject to control, to render it capable of being subject to control throughout its life, and to retain its integrity. Control, in the electronic world, replaces possession. A growing number of jurisdictions have adopted laws based on MLETR. Among the most significant for shipping, the United Kingdom's Electronic Trade Documents Act 2023, which came into force in September 2023, gave electronic trade documents meeting its criteria the same legal effect as paper ones under English law, which governs a large proportion of international trade contracts. In parallel, the Digital Container Shipping Association, an industry body formed in 2019 by major ocean carriers, published standards for electronic bills of lading and their exchange between platforms, and its members committed in 2023 to issuing all their bills electronically by 2030. Modelling title on a Fabric ledger The MLETR requirements map naturally onto Fabric's capabilities, which is why permissioned ledgers remain an attractive substrate for electronic bills even though most platforms do not use Fabric specifically. A bill of lading on a Fabric ledger is a single key — identified by the bill's unique number, perhaps prefixed by the issuing carrier's code — whose value records the bill's essential terms, a hash of the full document, its current status and its current holder. Because the key is unique on the channel and Fabric's validation rules ensure that each version of a key is written by exactly one valid transaction, there can be only one current holder at a time. That is the property MLETR calls exclusive control. The chaincode implements a state machine for the bill's life: issued by the carrier to the shipper, transferred by endorsement from one holder to another, surrendered by the final holder to the carrier in exchange for delivery, and accomplished once the goods are released. Amendments, splitting a bill into several, or switching it to paper for a counterparty that cannot accept electronic records, are further transitions. Endorsement policies follow the holder. Through state-based endorsement, the bill's key carries a policy requiring endorsement by the current holder's organisation; a transfer function checks that the caller belongs to that organisation, writes the new holder and changes the key's policy to name the new holder. The issuing carrier retains a separate right, encoded in the chaincode logic and its own endorsement path, to record surrender and accomplishment, since only the carrier can release the goods. Privacy needs careful thought. The existence of a bill and its current holder may be sensitive, since the chain of holders reveals who is financing and buying the goods. A design might keep only an opaque bill identifier and status on public channel state, with the holder history and the terms in a private collection whose membership is the carrier, the current holder and previous holders. Alternatively, each transfer could be recorded in the implicit collections of the transferor and transferee, with hashes on public state that let any member verify that the chain of transfers is unbroken. Banks taking the bill as security need assurance that it has not been transferred elsewhere, and the uniqueness of the key gives them exactly that. A ledger-based electronic bill solves the technical problem neatly. It leaves the legal and commercial problems where they were. The parties must operate under a jurisdiction that recognises the record, or under a contractual framework that binds them all. Banks must accept it under their letter of credit practices; the International Chamber of Commerce's rules for electronic presentation under documentary credits address part of this. And the record is only as good as the carrier's discipline in issuing it accurately and the integrity of the identity system that determines who the holder is. One container, end to end It helps to follow a single shipment through a ledger designed along these lines. A furniture manufacturer in Vietnam sells a container of flat-packed tables to a retailer in Germany on terms under which the retailer's bank issues a letter of credit. The parties on the channel include the ocean carrier, a freight forwarder acting for the seller, the terminals at Ho Chi Minh City and Hamburg, the seller's and buyer's banks, and the buyer's customs broker. The forwarder books space with the carrier. The booking confirmation is recorded as an event on public channel state — booking reference, carrier, vessel and voyage, port pair — with the commercial details in a collection shared by the carrier and the forwarder. The carrier releases an empty container; the depot records a gate out empty event against the container number. The seller loads the container and seals it, and the forwarder records the seal number and the declared contents, with a hash of the packing list. The container is trucked to the terminal, which records gate in and, later, loaded on the vessel. Each of these events is endorsed by the party that observed it, and each is visible to every party with an interest in this shipment. When the vessel sails, the carrier issues an electronic bill of lading to the seller. Its key is created with the seller as holder and a key-level policy naming the seller. The seller presents the bill, along with the commercial invoice and certificate of origin, to its bank; on the ledger, this is a transfer of the bill to the bank, recorded by the seller and, through the policy change, now controllable only by the bank. The documents themselves are off-chain, in a document service that the seller has granted the banks access to; their hashes sit in a collection shared by the seller and the two banks. The seller's bank checks the documents against the letter of credit and forwards them to the buyer's bank, transferring the bill again. When the buyer pays, its bank transfers the bill to the buyer. Meanwhile, advance cargo data has flowed to the destination customs authority. The carrier's transport data and the forwarder's goods data, each endorsed by its author, have been made available through a defined interface. When the vessel arrives in Hamburg, the terminal records discharged. The buyer, now holder of the bill, surrenders it to the carrier; the carrier records the surrender and issues a release; the customs broker records the customs clearance; the terminal records gate out full when the haulier collects the container. At no point did anyone courier a paper bill across the world, and at no point could two parties both believe they held title. Every event is attributable. If the retailer later claims the tables arrived damaged, the record shows who held the container at each stage and when. If the bank disputes a document, it can verify the hash against the ledger. The weak points are the familiar ones: whether the seal number recorded is the seal actually applied, whether the goods description is honest, and whether every party in the chain — including the haulier, who may be a one-truck business — has recorded what it was supposed to record. Manifests and customs The second major opportunity is the cargo manifest and its relationship with customs. As advance filing regimes such as the EU's Import Control System 2 increasingly require data from several parties in the chain, customs authorities are effectively asking the supply chain to assemble a multi-party record before the goods arrive. A shared ledger can serve this directly. If the shipper or forwarder records the commercial details — seller, buyer, goods description, classification codes — in a private collection shared with the relevant customs authority, and the carrier records the transport details on the same channel, the authority receives a consistent, attributable dataset in which each element is signed by the party responsible for it. If the forwarder's goods description and the carrier's container weight are inconsistent, the discrepancy is visible to customs at once, rather than surfacing at the port. Customs authorities have taken part in blockchain pilots — several participated in TradeLens — but their use as full network members raises its own issues. A customs authority cannot be bound by a private consortium's governance in the way a commercial member can, and it is subject to public law obligations about record retention, access and procurement. In practice, customs authorities are more likely to consume data from such networks through defined interfaces than to run endorsing peers. Where they do join, a design in which the authority's peer participates in a collection containing the data it is legally entitled to see, with read-only rights, is the natural model. The broader point is that the physical-digital gap discussed in earlier chapters takes a specific form in shipping. Misdeclared cargo — goods described as one thing but actually another, often to avoid dangerous goods handling rules or tariffs — is a persistent cause of fires at sea and of customs fraud. A ledger records what the shipper declared and who declared it. It cannot open the container. What it can do is make the declaration indelible and attributable, so that when an inspection or an incident reveals the truth, responsibility is clear. Shipping, like pharmaceuticals, thus offers a clear technical fit for a permissioned ledger and a much harder commercial and governance problem. The final chapter turns to that problem directly. Chapter 8: Governance, Operations and Why Consortia Fail The previous chapters have described what a Fabric network can do. This chapter is about what it takes to keep one running, and about the ways in which consortia that built technically sound networks nonetheless failed. The two subjects belong together, because almost every serious failure of an enterprise blockchain consortium has been a failure of governance or economics rather than of software, and because the operational decisions a network makes — who runs the orderers, who can add members, who pays for upgrades — are themselves governance decisions in technical form. The founder's dilemma Almost every consortium begins with a founder: one company, or a small group, that sees the opportunity, funds the initial development and recruits the first members. This is unavoidable, since nobody else will pay for a network that does not yet exist. It is also the origin of the problem that has most often sunk consortia. The founder designs the network to serve its needs, chooses the technology partner, writes the initial chaincode and endorsement policies, often operates the ordering service and much of the infrastructure, and sets the commercial terms. All of this gives the founder a position of advantage. Prospective members, particularly the founder's competitors, look at the network and ask the question Chapter 1 identified as the root of the consortium problem: whoever controls this shared system gains power over everyone who depends on it. If the answer is the founder, they stay out. TradeLens is the clearest illustration. Maersk was the world's largest container carrier when the platform launched, and the platform was a joint venture between Maersk and IBM. Rival carriers were concerned that their commercial data would be visible to Maersk, or that the platform would evolve in Maersk's interest. The founders attempted to address this, establishing an advisory board, emphasising data permissions and allowing carriers to run their own nodes. Several large carriers did join. But the platform never shed its association with its founder, its commercial model required participants to pay for a service whose value depended on others also paying, and in November 2022 Maersk and IBM announced its closure, saying it had not reached the level of commercial viability needed to continue. The lesson is not that founders should not found. It is that a consortium must, from early on, give non-founding members real control: over the rules, over the code, over the infrastructure and over the money. Fabric provides the technical means — distributed ordering, lifecycle approval by majority, configurable admin policies — but using them in ways that genuinely dilute the founder's control is a decision the founder has to make, often against its short-term interest. Governance structures that work Successful consortia have converged on a small number of structural choices. The first is a separate legal entity to own and operate the shared elements of the network: the governance documents, the shared code base, any shared infrastructure, and the contracts with technology providers. The entity might be a non-profit, a cooperative, or a company jointly owned by the members, and its board represents the members rather than the founder. This entity becomes the counterparty for new members, which removes the problem of joining a competitor's platform. The second is a written operating agreement that sets out, in terms the members' lawyers and engineers both understand, the rules the network enforces: who may join, on what terms, who may write which records, who may see what, how disputes are resolved, how the rules change, and what happens when a member leaves. As argued in Chapter 3, the endorsement policies, collection definitions and channel policies should be annexes to this agreement and should change only through its amendment process. The third is distributed infrastructure. Ordering nodes should be operated by several members, not one, and ideally by members whose interests differ — a manufacturer, a distributor and a logistics provider, rather than three manufacturers. The channel's Admins policy, which governs configuration changes such as adding members, should require approval from a majority of members or a defined set of founding members, so that no single administrator can alter the network's composition. The fourth is a clear economic model. Members need to know what they will pay, for what, and how costs will change as the network grows. Common models include a membership fee covering shared costs, transaction-based fees, or a model in which the parties that benefit most — often those at the end of the chain who receive the provenance data — pay most. Networks that leave this question vague, or that expect early participants to bear costs in anticipation of network effects that may never arrive, struggle to retain members. What a network actually costs Discussions of enterprise blockchain often focus on transaction costs, which for a permissioned network are negligible: there are no fees in the public blockchain sense, and the computing cost of endorsing and validating a transaction is small. The real costs lie elsewhere, and they are borne by every member. Each member must run infrastructure: at least two peers for resilience, their state databases, monitoring, backups, and, if it hosts an orderer, that too. Whether on its own hardware, in a cloud account or through a managed service, this is a continuing operational commitment with people attached to it. Each member must integrate the network with its own systems — the warehouse management system that knows when a pallet ships, the enterprise resource planning system that holds the purchase order, the customs filing software — and integration is usually the single largest cost of joining. Each member must maintain its identity infrastructure, its certificate lifecycle and its security controls. And each member must participate in governance: reviewing chaincode, approving upgrades, voting on new members, attending working groups. For large members, these costs are manageable. For small ones — independent pharmacies, small hauliers, small growers — they can be prohibitive, and a network that requires every participant to run a node will never include them. Most traceability networks therefore have a tiered structure: larger members run peers and participate fully in endorsement and governance, while smaller participants connect through a gateway service operated by a larger member, the consortium or a vendor. Those smaller participants then depend on whoever operates the gateway, which reintroduces some of the trust problems the ledger was meant to solve. It is a sensible compromise, but the consortium should acknowledge it as one. The trade finance network we.trade, founded by a group of European banks and built on Fabric, offers a cautionary example of costs outrunning value. It launched in 2018, attracted a substantial number of member banks across Europe, and processed live transactions. But it never achieved the volume of business needed to cover its costs, and in 2022 it ceased operations after failing to secure further funding. As with TradeLens, the technology worked; the economics did not. Onboarding and identity Adding a new member to a Fabric network is a sequence of technical steps, each of which has a governance dimension. The new organisation sets up its certificate authority, or arranges to use an existing enterprise one, and creates its MSP definition. An existing channel administrator prepares a configuration update adding the MSP to the channel. The update is circulated to the other members' administrators, who each sign it; once enough signatures exist to satisfy the Admins policy, the update is submitted to the ordering service and the new organisation becomes a channel member. The new organisation joins its peers to the channel, installs the chaincode, approves the current chaincode definition, and, if the definition's policies use implicit meta policies such as MAJORITY Endorsement, the effective endorsement requirements for the whole network change at that moment. Several governance questions are buried in that sequence. Who decides whether a prospective member is admitted? On what criteria — a licence, a credit check, a security audit of its infrastructure? Who verifies that the MSP definition the newcomer presents genuinely belongs to the legal entity it claims to represent? In regulated sectors, the answer to that last question matters a great deal. A pharmaceutical network must be confident that an organisation recorded as a licensed wholesaler is in fact licensed, and must be able to remove it promptly if its licence is withdrawn. Some networks bind the MSP definition to verifiable credentials issued by regulators or industry bodies; others rely on manual checks at onboarding and periodic review. Leaving is as important as joining. When a member exits — through choice, insolvency, acquisition or expulsion — its MSP must be removed from the channel configuration, and its peers will stop receiving new blocks. But the departing member retains its copy of the ledger up to that point, including any private data it was entitled to see. The operating agreement must address what happens to that data, and the members must accept that some of it cannot be recalled. Upgrades and long-term operation A traceability network is meant to run for years, and Fabric itself does not stand still. The platform has moved from 1.x to 2.x to 3.x in under a decade, with each major version changing aspects of the lifecycle, ordering and configuration. Long-term support releases ease this, but networks must still plan for upgrades of Fabric itself, of the chaincode, of the certificate infrastructure and of the underlying operating systems and container platforms. Upgrades in a consortium are harder than in a single enterprise because every member runs its own nodes and must upgrade on a coordinated schedule. Fabric's capability levels help: each channel declares the capability level it uses, which determines the features available and the validation rules applied. Members upgrade their binaries first, while the channel continues to run at the old capability level; once every member has upgraded, administrators update the channel's capability level, and the new features become active. A member that lags behind can stall the whole process. Certificates expire, and expired certificates break networks. Every peer, orderer, administrator and client identity has a certificate with a validity period, and an expired certificate on an ordering node can halt a channel. Mature networks automate certificate renewal and monitor expiry dates across members, but early networks have repeatedly been surprised by expiries a year or two after launch. Chaincode upgrades go through the lifecycle described in Chapter 3. Because each member must approve the new definition, upgrades are a governance event as well as a deployment. Members need time to review the code, which means the consortium needs a release process with defined review periods, test environments in which members can run the new code against copies of production data, and a policy for emergency fixes that may need faster approval. Finally, there is the question of who operates the nodes. Many members of a supply chain consortium do not want to run blockchain infrastructure themselves. Managed services from cloud providers and specialist vendors exist for this, and their use is often sensible. But a member whose peers are run by a vendor is trusting that vendor with its keys, and if several members use the same vendor, the vendor may become a single point of control — and failure — that the network's distributed design was meant to avoid. Some consortia require members to hold their own signing keys in hardware security modules even when the nodes are hosted elsewhere, which preserves the member's control over what its organisation endorses. The data quality problem Every chapter of this book has returned, sooner or later, to the gap between what the ledger records and what is physically true. In governance terms, that gap becomes a question of accountability: who is responsible for the accuracy of what they write, and what happens when they write something wrong? A ledger cannot prevent a member from recording a false event. It can make the false event indelible and attributable, so that when it is discovered, the member responsible can be identified. That is a strong deterrent, but only if the consortium's operating agreement says what consequences follow — financial liability, suspension, expulsion — and only if the members are willing to enforce them against one another. More common than deliberate falsification is simple error: a scan missed, an aggregation not recorded, a timestamp from a misconfigured device, a status left unchanged. In a traceability network, errors propagate. A missing disaggregation event leaves packs apparently inside a pallet that no longer exists; the next custodian cannot record receiving them because, according to the ledger, they are somewhere else. The network must have a process for correcting errors, and since the ledger is append-only, correction means recording a further event, with its own endorsement, that sets the record straight. The chaincode should support these correction events explicitly, with appropriate authorisation, rather than leaving members to work around the state machine. Networks that take data quality seriously typically measure it: the proportion of shipments for which every expected event was recorded, the time between a physical event and its recording, the number of exceptions raised by the chaincode's state checks. They publish these metrics to members and use them in governance. Those that do not tend to accumulate inconsistencies until users stop trusting the network and revert to email. Why consortia fail, and why some survive Drawing together the experience of the past decade, a handful of reasons account for most consortium failures. Founder dominance, as described above, deters competitors from joining, leaving the network without the participation that would make it valuable. Unclear value for the parties who must do the work. In a traceability network, the parties who must record events — manufacturers, carriers, terminals — are often not the parties who benefit most from the shared record, who may be regulators, consumers or downstream buyers. Unless the economic model compensates the contributors, they record the minimum. Pilots that never become products. Many networks were built as proofs of concept with test data and a handful of enthusiastic participants. Moving to production requires integration with members' operational systems, which is expensive, and a governance and commercial framework, which is slow to negotiate. Many pilots stalled at that transition. Solving a problem the alternatives already solve. As Chapter 6 described for pharmaceutical compliance, a permissioned ledger is sometimes a more expensive way to achieve what point-to-point messaging or a regulated repository achieves adequately. The consortia that have survived share some features in turn. They solve a specific problem that participants feel acutely, rather than a general aspiration to transparency. Their governance gives members real control. Their economic model works for the parties who must contribute. They integrate with participants' existing systems rather than requiring new ones. And they are honest about what the ledger guarantees, investing as much in data capture and quality as in the ledger itself. A checklist for the decision For a consortium considering a Fabric network for traceability, the arguments of this book reduce to a short set of questions that should be answered, in writing, before the first line of chaincode. What specific question will the network answer that members cannot answer today, and who will ask it? Who will record the events that answer it, and why will they bother? Is there a neutral operator who could run a central repository, and if so, why is a distributed ledger better? Who will run the ordering service, and how will no single member control it? What are the endorsement policies for each kind of record, and do they reflect what the members have agreed about who must vouch for what? What data will be visible to whom, and has each member's legal and commercial team reviewed that? What happens when a member leaves? Who pays, how much, and how does that change as the network grows? How will the network measure and enforce data quality? Who will own the network in five years, and what will make the members still want to belong? A consortium that can answer those questions has done the hard work. Building the ledger is then a well-understood engineering task. A consortium that cannot answer them should not start building. Conclusion The case for Hyperledger Fabric in supply chain traceability, stripped of the enthusiasm of its boom years, comes down to one claim. Where a group of organisations must share a record of goods moving among them, where none of them will accept any single party as the keeper of that record, and where the rules about who may write and see what are too intricate to be left to contracts alone, a permissioned ledger lets them write those rules into machinery that each runs and none controls. Fabric's particular contribution is to make the rules configurable at every level that matters: identity per organisation, endorsement per chaincode, per collection and per key, visibility per channel and per collection, and changes to the code itself subject to collective approval. That claim is narrower than the one made for blockchain a decade ago, and it is more defensible. It does not say that a ledger makes data true. It does not say that decentralisation is valuable for its own sake. It does not say that every supply chain needs a shared ledger, or that a ledger is better than a well-governed central repository or a disciplined exchange of standard messages. It says that in a specific kind of situation — among known parties, with real conflicts of interest, needing shared verifiable records — the ledger provides something the alternatives do not: specified trust, enforced by code, distributed across the participants. What follows from the argument Several practical conclusions follow from the preceding chapters, and they are worth stating plainly because they cut against how many projects have been run. Design the trust model before the technology. The most important design documents for a Fabric traceability network are not the architecture diagrams but the endorsement policies and the privacy map: for each kind of record, who must vouch for it, and who may see it. Those documents are statements of the consortium's agreement about trust, and they should be negotiated by the people who understand the business relationships, written in language both lawyers and engineers can read, and kept in step with the operating agreement. A network whose policies were left at their defaults has no trust model; it has an accident waiting to be discovered. Treat chaincode as a specification of the business process. State machines with explicit transitions, authorisation checks on every state-changing function, and standard identifiers and event vocabularies make the chaincode reviewable by the people whose agreement it encodes. Generic update functions and invented data models turn the ledger into a shared spreadsheet with extra cost. Use the privacy mechanisms deliberately, and accept their trade-off. Every piece of data moved from the shared channel state into a private collection or off the ledger altogether is verifiable by fewer parties. That is often the right choice, but it is a choice, and the network's assurance about that data rests correspondingly on fewer shoulders. Invest more in data capture than in the ledger. The ledger records faithfully what members agree happened. Whether that corresponds to the physical world depends on scanners, sensors, warehouse processes, aggregation discipline and the incentives of the people doing the recording. In both of the domains this book has examined, pharmaceuticals and shipping, the dominant source of problems is not tampering with records but failure to create accurate ones in the first place. Govern for the members who are not yet in the room. The founder's dilemma is the single most common cause of consortium failure. A network that wants its founder's competitors to join must give them real control over rules, code, infrastructure and money, and must do so before they are asked to commit. What remains unsettled Some questions remain genuinely open, and anyone building in this space should watch them. The first is the role of Byzantine fault tolerance. Fabric's SmartBFT ordering service, added in version 3.0, closes a gap that Raft left open: a consortium can now tolerate an ordering node that behaves maliciously, not merely one that fails. Whether consortia will adopt it widely, accepting its greater node count and overhead in exchange for stronger guarantees, will say a good deal about how much members of these networks actually distrust one another. The second is the interaction between ledgers and the emerging digital trade law. Laws based on the UNCITRAL Model Law on Electronic Transferable Records, such as the United Kingdom's Electronic Trade Documents Act 2023, give electronic records of title legal effect if they meet criteria of control and integrity. Permissioned ledgers can meet those criteria, but so can other systems. Whether the ledger becomes the preferred substrate for electronic bills of lading, or one option among several connected by interoperability standards, is not yet clear. The third is interoperability among networks. The pharmaceutical experience suggests that industry-wide traceability will be built from many systems exchanging standard messages rather than one shared ledger. If so, the most valuable Fabric networks may be those that serve a specific, demanding segment — specialty therapies, high-value cold chain, particular trade lanes — and exchange data with the wider ecosystem through standards such as EPCIS and the Digital Container Shipping Association's interfaces. Fabric's design does not prevent this, but the tooling for connecting Fabric networks to one another and to other systems remains less mature than the core platform. The fourth is what happens to the data at the end of a network's life. TradeLens's closure raised a question few consortia had planned for: when a network shuts down, what happens to the records? Each member that ran a peer retains its copy of the ledger; members that relied on the platform's interfaces may retain nothing. Networks built today should decide, at the outset, how the records will be preserved and made available if the network ends, since regulatory retention obligations do not lapse when a consortium dissolves. The decision in front of the reader For a reader deciding whether to build, join or recommend a Fabric network for traceability, the argument of this book suggests a straightforward test. Can the consortium state, specifically, what question the network will answer that cannot be answered today? Can it name the parties who will record the events, and explain why they will? Can it explain why a central repository or point-to-point exchange would not do? Has it agreed, in writing, on who must vouch for each kind of record and who may see it? Has it decided who runs the infrastructure, who pays, and how no single member controls the whole? If the answers are yes, Fabric offers a mature, well-documented and widely deployed platform for encoding that agreement, and the engineering work, while substantial, is well understood. If the answers are no, the ledger will not supply them. The hardest and most valuable work in enterprise blockchain has always been the agreement among the organisations. The ledger's job is to hold them to it. Hashtags: #EnterpriseBlockchain #HyperledgerFabric #BlockchainTechnology #PermissionedBlockchain #SupplyChainTraceability #SupplyChainManagement #BlockchainForSupplyChain #DistributedLedgerTechnology #EnterpriseBlockchainSolutions #SupplyChainTransparency #ProductTraceability #ProductProvenance #ChainOfCustody #SmartContracts #HyperledgerChaincode #EndorsementPolicies #BlockchainGovernance #BlockchainPrivacy #PrivateDataCollections #BlockchainConsortium #SupplyChainSecurity #PharmaceuticalTraceability #DrugSupplyChainSecurity #ColdChainManagement #DigitalTradeDocuments #ElectronicBillOfLading #GS1Standards #SupplyChainDigitalization #BlockchainInteroperability #BlockchainInnovation

Download the Book (PDF): Introduction For most of the history of commercial software, a database was a single machine. You bought the biggest server the budget allowed, put a relational database on it, and when it filled up you bought a bigger one. The model was beautiful in its simplicity. A transaction either happened or it did not. Two customers could not both buy the last seat on a flight. A bank balance was a number, not a negotiation. The price of that simplicity was a ceiling: one machine, one failure domain, one location on the planet. The internet broke the ceiling, and for about a decade the industry's answer was to give up the guarantees. Engineers sharded MySQL by hand, writing routing logic into every application and accepting that a transaction spanning two shards was simply not possible. Others adopted the new NoSQL systems, which scaled horizontally by relaxing consistency: a write acknowledged in one data center might not be visible in another for some time, and the application had to cope. Amazon's Dynamo paper of 2007 made eventual consistency respectable, and Cassandra, Riak, and their relatives carried it into production at enormous scale. The trade was explicit. You could have scale and availability, or you could have the comfort of ACID transactions, and anyone who wanted both was told to read about the CAP theorem. Distributed SQL is the claim that the trade was not as absolute as it looked. A distributed SQL database stores data across many machines, often across many regions, and still presents a single logical relational database with serializable or near-serializable transactions, secondary indexes, joins, and a SQL interface. It scales by adding nodes rather than by buying bigger ones. It survives the loss of a machine, a rack, an availability zone, and, if configured to, an entire region, without losing acknowledged writes and without an operator promoting a replica at three in the morning. Three systems define the category, and this booklet treats them together because they share an architecture while differing in instructive ways. Google Spanner came first. Described in a paper at the USENIX OSDI conference in 2012, Spanner was built to replace a sharded MySQL deployment behind Google's advertising business. Its most famous innovation is TrueTime, a clock API backed by GPS receivers and atomic clocks in Google's data centers that returns not a single timestamp but an interval guaranteed to contain the true time. Spanner uses that bounded uncertainty to give every transaction a timestamp that respects real-time order across the globe. Google later offered Spanner as a managed service on Google Cloud, and in April 2026 announced Spanner Omni, a downloadable version for other clouds and private data centers, in preview. CockroachDB, built by Cockroach Labs and first released in 2015, set out to bring Spanner's design to environments without atomic clocks. It runs on ordinary hardware, on any cloud or on premises, synchronizes clocks with NTP, and speaks the PostgreSQL wire protocol. Its designers described the system in detail at ACM SIGMOD in 2020. It replaced Spanner's Paxos with Raft, replaced TrueTime with hybrid logical clocks and uncertainty intervals, and over time added a declarative multi-region SQL vocabulary that is probably the clearest expression anywhere of how to design for regional survivability. YugabyteDB, from Yugabyte, took a different approach to the SQL layer. Rather than writing a SQL engine from scratch, it reused PostgreSQL's own query layer, rewiring its storage underneath onto a distributed, Raft-replicated document store called DocDB. The result is unusually faithful PostgreSQL compatibility, including stored procedures, triggers, and many extensions, on top of a storage layer with a strong family resemblance to Spanner and CockroachDB. The three share a skeleton. Each splits tables into contiguous chunks of key space, called splits in Spanner, ranges in CockroachDB, and tablets in YugabyteDB. Each replicates every chunk to several machines and keeps the replicas in agreement with a consensus protocol, Paxos in Spanner and Raft in the other two. Each assigns transactions timestamps and keeps multiple versions of every row, so that reads can proceed without blocking writes. Each implements distributed transactions with a variant of two-phase commit layered on top of consensus, so that the coordinator's own state is replicated and cannot be lost with a single machine. The controlling idea The argument of this booklet is simple to state and takes the rest of the book to earn. Distributed SQL does not repeal physics; it prices it. Every guarantee these systems offer is paid for in a specific currency: round trips between replicas, and uncertainty about what time it is. A write is durable when a majority of replicas have it, which means the write cannot complete faster than the round trip to the nearest majority. A read is strongly consistent when it is served by a replica that knows it holds the latest committed value, which means either a lease, a round trip, or a wait. A transaction that spans regions must, at some point, hear from those regions. No configuration, vendor, or clever protocol removes these costs. What a good distributed SQL system does is make the costs explicit, keep them as low as the physics allows, and give the designer controls over where they are paid. It follows that the most consequential decisions in a distributed SQL deployment are not made by the database. They are made by whoever chooses the primary keys, decides which tables are global and which are regional, sets the survival goal, and places replicas and leaseholders. A schema that ignores locality will be correct and slow. A schema that ignores hotspots will be correct and fall over under load. A topology chosen without arithmetic will survive the wrong failures. The database will enforce ACID regardless, which is the point of the category; whether it does so in four milliseconds or four hundred depends almost entirely on design. What this booklet covers The first chapter sets out the problem distributed SQL solves, the history that produced it, and the vocabulary of consistency you need to read anything else on the subject: linearizability, serializability, external consistency, and the CAP and PACELC framings. The second chapter covers how data is cut into ranges and tablets, how those pieces split and move, how a SQL row becomes a key, and why the choice of primary key decides whether your workload spreads across a cluster or piles onto one machine. The third chapter is a full treatment of Raft, the consensus algorithm described by Diego Ongaro and John Ousterhout in 2014: leader election, log replication, the safety argument, membership changes, and the practical extensions such as pre-vote and leases that production systems add. The fourth chapter explains time in Spanner: TrueTime, commit wait, and how bounded clock uncertainty turns into external consistency and lock-free snapshot reads. The fifth chapter explains how CockroachDB and YugabyteDB live without atomic clocks, using hybrid logical clocks, maximum clock offsets, and uncertainty intervals, and what happens to correctness and performance when clocks misbehave. The sixth chapter takes apart distributed transactions: write intents, transaction records, parallel commits, conflict resolution, isolation levels, and the retry errors that applications must handle. The seventh chapter turns to topology. It works through the latency arithmetic of quorums across zones and regions, explains survival goals, and shows how replica and leaseholder placement determine what failures a cluster can absorb. The eighth chapter is about schema design for multi-region active-active systems: regional tables, regional-by-row tables, global tables, geo-partitioning in YugabyteDB, interleaving and locality in Spanner, and the patterns that keep most transactions local while preserving a single logical database. The ninth chapter covers operations and selection: migration, testing with fault injection, observability, upgrades, licensing and cost, and how to decide which of the three systems, if any, fits a given workload. The conclusion draws out what the preceding chapters imply for how teams should approach these systems, and what remains genuinely unsettled. The booklet assumes you are comfortable with relational databases and SQL, and that you know roughly what a transaction is. It does not assume any background in distributed systems theory. Every term is defined when it first appears, and the theory is introduced only when it explains a practical decision. Where specific behavior differs by version, the booklet describes the systems as they stood in late 2026: CockroachDB in its v26 series, YugabyteDB in its 2025.2 long-term-support and 2026.1 series, and Spanner as offered on Google Cloud in its Standard, Enterprise, and Enterprise Plus editions, with Spanner Omni in preview. Chapter 1: The Problem Distributed SQL Solves Every distributed database begins as an answer to a specific pain. Spanner's pain was Google's advertising system, which by the late 2000s ran on a manually sharded MySQL deployment. The F1 paper, published at VLDB in 2013 by the team that moved advertising onto Spanner, describes the cost bluntly: the last resharding of that MySQL fleet took over two years of intense effort, involving coordination across dozens of teams. Every time the data outgrew its shards, engineers had to split them, move data, update routing, and pray nothing broke. Cross-shard transactions did not exist. Cross-shard queries were the application's problem. That story is not unique to Google. It describes nearly every large relational deployment of the 2000s and early 2010s, and many that still run today. Understanding why it happens, and why the first generation of fixes made different trade-offs, is the clearest way into what distributed SQL actually changes. Three ways to outgrow a database A single-node relational database runs out of room in one of three ways. It runs out of storage, because the data no longer fits on the disks attached to one machine. It runs out of throughput, because one machine's CPUs and I/O cannot handle the write rate. Or it runs out of geography, because users on another continent experience a hundred or more milliseconds of round-trip latency to every query, and because one location is one failure domain. The traditional responses each address one of these and leave the others. Vertical scaling buys a bigger machine. It is the right first answer for most workloads, and modern servers with hundreds of cores and many terabytes of memory push the ceiling very high. But it keeps one failure domain and one location, and the ceiling, though high, is real. Read replicas copy the primary's write-ahead log to secondary machines that serve reads. This scales read throughput and can put read copies near distant users. It does nothing for write throughput, since every write still goes to one primary. And replicas lag: a user who writes a comment and then reloads the page may read from a replica that has not received the write yet. Applications learn to route "read your own writes" traffic back to the primary, which erodes the benefit. Failover adds a standby that takes over when the primary fails. With asynchronous replication the standby may be missing the last few seconds of acknowledged writes, so failover loses data. With synchronous replication to a single standby, the primary cannot commit when the standby is unreachable, so either availability or durability suffers. Automated failover systems have to decide whether the primary is really dead or merely slow, and a wrong guess produces two primaries accepting conflicting writes, the condition known as split brain. GitHub published a detailed account of exactly such an incident in October 2018, when a 43-second network partition between its US East Coast data center and its primary database cluster caused an automated failover that left writes split across two sites and degraded the service for more than 24 hours while engineers reconciled the data. Manual sharding splits the data across many independent databases by some key, typically customer or tenant identifier. It scales storage and write throughput nearly without limit. It also pushes enormous complexity into the application. Every query must know which shard to visit. Transactions across shards are either impossible or implemented ad hoc. Rebalancing requires moving data between live databases. Secondary indexes on anything other than the shard key require querying every shard. Schema changes must be rolled out to hundreds of databases in a coordinated way. Tooling such as Vitess, originally built at YouTube, automates much of this for MySQL, and it remains a strong choice for some workloads, but the fundamental limitation remains: the shard is the unit of transactional consistency. The NoSQL detour Between roughly 2007 and 2014, a different answer gained ground. Amazon's Dynamo paper, published at SOSP in 2007, described a key-value store designed for the shopping cart: always writable, even during network partitions, with conflicts resolved later. Google's Bigtable paper, at OSDI in 2006, described a sparse, distributed, sorted map that scaled to petabytes with single-row atomicity only. Apache Cassandra combined Dynamo's replication model with Bigtable's data model. MongoDB offered flexible documents with, in its early years, very weak durability defaults. These systems scaled horizontally and survived failures gracefully. They achieved it by narrowing the contract. Transactions, where they existed, covered a single row or document. Secondary indexes were eventually consistent or absent. Replicas could disagree, and applications had to tolerate reading stale data or reconcile conflicting versions. Google's own experience is instructive. Bigtable was enormously successful, but teams kept asking for more. Megastore, described at CIDR in 2011, layered synchronous Paxos replication and ACID transactions on top of Bigtable, within small "entity groups." The Spanner paper notes that Megastore was used by many of Google's applications, including Gmail, Picasa, Calendar, Android Market, and App Engine, despite its relatively poor write throughput, precisely because developers valued its semi-relational schema and synchronous replication. The lesson Google drew was that programmers would rather deal with performance problems caused by the overuse of transactions than always code around the lack of transactions. That sentence, close to a direct paraphrase of the Spanner paper, is the founding argument of distributed SQL. The vocabulary of consistency Discussions of distributed databases go wrong more often over vocabulary than over substance. Four terms need precise meanings. Serializability is a property of transactions. A set of concurrent transactions is serializable if the result is the same as if they had run one at a time in some order. Serializability says nothing about which order; it only requires that some serial order exist. A database that reorders two transactions submitted an hour apart is still serializable, although no user would accept it. Linearizability is a property of single operations on single objects. An operation is linearizable if it appears to take effect instantaneously at some point between its start and its completion, and that point respects real time: if operation A completes before operation B begins, then B observes A's effect. Maurice Herlihy and Jeannette Wing formalized the idea in 1990. A linearizable register behaves like a single variable in memory, even though it is replicated. Strict serializability combines the two: transactions are serializable, and the serial order respects real time. If transaction T1 commits before T2 starts, T2 is ordered after T1. Spanner's paper calls this external consistency, a term borrowed from earlier work by David Gifford, and it is the strongest isolation guarantee a database can reasonably offer. A user who receives a confirmation that a payment went through, and then tells a colleague who queries the account from the other side of the world, is guaranteed the colleague will see the payment. Snapshot isolation is a weaker guarantee, widely implemented. Each transaction reads from a consistent snapshot taken at its start and commits only if no other transaction wrote the same rows in the meantime. It prevents most anomalies but permits one called write skew: two transactions each read an overlapping set of rows, each write a different row based on what they read, and both commit, producing a state no serial order would allow. The canonical example is two on-call doctors each checking that the other is on call before signing off, and both succeeding. With those definitions in hand, the three systems in this booklet can be placed. Spanner provides external consistency for read-write transactions by default. CockroachDB defaults to serializable isolation, and its SIGMOD paper is careful to claim less than Spanner on real-time order: transactions are serializable, operations on any single key are linearizable, and reads are never stale, but two transactions touching disjoint keys are not guaranteed to be ordered by real time. The nuance is the subject of Chapter 5. CockroachDB also offers read committed isolation, introduced in v23.2 and generally available since v24.1. YugabyteDB's YSQL API supports serializable, snapshot (which it maps to PostgreSQL's repeatable read), and read committed isolation, defaulting to snapshot. CAP, and why it is the wrong question The CAP theorem, conjectured by Eric Brewer in 2000 and proved by Seth Gilbert and Nancy Lynch in 2002, states that a distributed system cannot simultaneously guarantee consistency (in the linearizable sense), availability (every request to a non-failed node receives a response), and partition tolerance. Since networks do partition, the practical reading is that during a partition, a system must choose between consistency and availability. Distributed SQL systems choose consistency. When a network partition separates a minority of replicas from the majority, the minority side stops serving strongly consistent reads and writes for the data it cannot reach a quorum for. In CAP terms, they are CP systems. That label hides more than it reveals, for two reasons. First, CAP availability is an extreme definition: every non-failed node must answer. A system in which the majority side of a partition keeps serving everyone it can reach is "unavailable" by CAP's definition but highly available by any user's. Brewer himself made this argument in a 2017 Google paper on Spanner and CAP. He wrote that Spanner is technically a CP system but in effect a CA system, because partitions are rare enough, and the network redundancy deep enough, that Spanner was not a significant contributor to application downtime. He reported that Spanner achieved more than five nines of availability, and that network problems contributed less than ten percent of Spanner's already rare outages. Second, CAP says nothing about the normal case, which is where systems spend nearly all their time. Daniel Abadi's PACELC framing, published in IEEE Computer in 2012, extends CAP: if there is a Partition, choose Availability or Consistency; Else, choose Latency or Consistency. The "else" clause is where distributed SQL lives day to day. Every strongly consistent write must reach a quorum, and every quorum has a round-trip time. The more widely the quorum is spread for survivability, the slower each write. That is the real trade-off, and it is the subject of most of this booklet. Three origin stories The three systems examined in this booklet were built by overlapping groups of people reacting to the same problem from different positions, and their origins explain many of their differences. Spanner was an internal Google system for years before it was a product. The OSDI paper of 2012 described a database already running in production behind F1, the advertising backend. Google opened Cloud Spanner to external customers in a beta in February 2017 and made it generally available a few months later. Because it was built inside one company, on that company's network, with that company's hardware, Spanner could assume things no outside system could: a private global backbone with redundant paths, and a time infrastructure built specifically for the database. Its early external API reflected its internal heritage, with its own SQL dialect and client libraries, and PostgreSQL compatibility came later as an additional interface. CockroachDB was founded in 2015 by Spencer Kimball, Peter Mattis, and Ben Darnell, all former Google engineers who had worked with Google's internal infrastructure and wanted something like it outside Google. Their central design question was how much of Spanner could be rebuilt without TrueTime and without Google's network. The first production release, version 1.0, came in 2017. The team chose Go as its implementation language, the PostgreSQL wire protocol as its interface so that existing drivers and tools would work, and Raft as its consensus algorithm, initially through the etcd project's Raft library. Over time CockroachDB grew a distinctive emphasis on multi-region operation, culminating in the declarative multi-region SQL described in Chapter 8. YugabyteDB was founded in 2016 by Kannan Muthukkaruppan, Karthik Ranganathan, and Mikhail Bautin, who had built and operated large data systems at Facebook, including work on Apache HBase and Cassandra. Their starting point was a storage layer: a distributed, Raft-replicated document store, DocDB, written in C++ and built on a heavily modified RocksDB, with a Raft implementation derived from Apache Kudu's. On top of it they placed two APIs. YCQL is compatible with the Cassandra Query Language, for teams coming from that world. YSQL reuses PostgreSQL's own parser, planner, and executor, with storage redirected to DocDB. The decision to reuse PostgreSQL code rather than reimplement SQL is YugabyteDB's defining bet: it trades some freedom to optimize the SQL layer for distribution in exchange for compatibility that is difficult to achieve any other way. The three are often compared on benchmarks, and vendors publish many, each tending to favor its author's system. This booklet does not adjudicate those claims. Performance depends overwhelmingly on schema design, topology, and workload, the subjects of later chapters, and a well-designed schema on any of the three will outperform a poorly designed one on another. The shared architecture Spanner, CockroachDB, and YugabyteDB arrived at strikingly similar designs. Table 1 summarizes where they align and where they differ; the details of each row are developed in the chapters that follow. Table 1. The three systems at a glance (as of late 2026). Property Spanner CockroachDB YugabyteDB Unit of data Split (within a Paxos group) Range Tablet Consensus Paxos with leader leases Raft with leader leases Raft with leader leases Clock TrueTime (GPS and atomic clocks) Hybrid logical clock over NTP Hybrid logical clock over NTP SQL interface GoogleSQL and PostgreSQL dialects PostgreSQL wire protocol, own engine PostgreSQL 15 query layer (YSQL), plus YCQL Deployment Google Cloud; Spanner Omni in preview Self-hosted or managed, any cloud Self-hosted or managed, any cloud License Proprietary service CockroachDB Software License (source available) Apache 2.0 core The layering is common to all three. At the top sits a SQL layer that parses queries, plans them, and translates rows into key-value operations. Beneath it, a transaction layer assigns timestamps, tracks locks or intents, and coordinates commits that span multiple pieces of data. Beneath that, a distribution layer maps key ranges to the machines holding them and routes requests. Beneath that, a replication layer keeps several copies of each range in agreement using consensus. At the bottom, a storage engine on each node persists data to local disk: Pebble, a Go reimplementation of RocksDB's design, in CockroachDB; a heavily modified RocksDB inside DocDB in YugabyteDB; and Google's internal Colossus-backed storage in Spanner. The essential move, shared by all three, is to make the consensus group the unit of replication and the transaction protocol the unit of atomicity across groups. A single range or tablet is a small replicated state machine that never loses an acknowledged write as long as a majority of its replicas survive. A transaction that touches many ranges commits atomically by writing a transaction record, itself stored in a replicated range, so that the coordinator's decision survives the coordinator's death. Classic two-phase commit had a notorious weakness: if the coordinator failed after participants voted yes, they could be blocked indefinitely. Building two-phase commit on top of consensus removes that weakness, because there is no single coordinator machine whose loss matters. What changes for the application For an application developer, a distributed SQL database looks like PostgreSQL or a close relative. You connect with a standard driver, create tables, run joins, and wrap work in transactions. That familiarity is the selling point and also the trap. Three things change, and teams that do not internalize them have a difficult first year. First, latency becomes a function of data placement. On a single node, every row is equally close. In a distributed database, a row's leaseholder may be in the same rack or on another continent. Two queries that look identical in SQL can differ by a factor of fifty in latency depending on which rows they touch. Second, contention becomes visible as retries. Optimistic and timestamp-based concurrency control means that conflicting transactions are often aborted and must be retried, rather than simply waiting on a lock. Applications must be written to retry transactions on specific serialization errors, which most code written for single-node databases does not do. Third, primary key design becomes a performance decision. A monotonically increasing key, the default in much single-node schema design, funnels every insert into the last range of the table and onto one machine, no matter how many nodes the cluster has. The rest of this booklet explains why each of these is true, starting with how data is divided. Chapter 2: Ranges, Tablets, and Splits A distributed database must answer one question before any other: where does a given row live? The answer determines which machines participate in each query, how load spreads across the cluster, and how the system grows when nodes are added. Spanner, CockroachDB, and YugabyteDB answer it in similar ways, and the similarities run deep enough that understanding one illuminates the others. The differences, mostly in how keys are distributed across the key space, explain some of the most common performance problems teams encounter. From rows to keys Underneath the SQL, all three systems are built on a sorted key-value store. Every table row, and every entry in every secondary index, is encoded as one or more key-value pairs. The encoding is designed so that the byte-wise sort order of the keys matches a useful logical order, usually the order of the primary key. CockroachDB makes this concrete. A row in a table is stored under a key that begins with the table's numeric identifier, then the index identifier (the primary index is index 1), then the encoded values of the primary key columns, then a column family identifier. A row in a table with ID 104 and primary key 42 lives under a key that a debugging tool would print as /Table/104/1/42/0. A secondary index entry on that table lives under /Table/104/2//42, ending with the primary key so that the index entry can point back to the row. Because keys sort lexicographically, all the rows of a table sit together, in primary key order, and all the entries of each index sit together in index order. Column families let a table store groups of columns under separate keys. A table with a large, rarely updated document column and a small, frequently updated counter column can put them in different families, so that updating the counter does not rewrite the document. By default all columns share one family, which is usually right. YugabyteDB's DocDB encodes rows similarly but with an important addition at the front of the key. For a hash-sharded table, which is the default in YSQL, the key begins with a 16-bit hash of the hash-partitioning columns, followed by those column values, then any range columns, then a column identifier and a hybrid timestamp. The hash prefix means that rows with adjacent primary key values are scattered across the key space rather than stored together. That choice has large consequences, discussed below. Spanner's data model, described in its 2012 paper and refined in later work, also maps rows to sorted keys, with one distinctive feature: tables can be declared as interleaved in a parent table, so that child rows are stored physically adjacent to their parent row. A Singers table with an interleaved Albums table stores each singer followed immediately by that singer's albums, because the album's key begins with the singer's key. The original paper called a set of contiguous keys sharing a common prefix a directory, and described directories as the unit of data placement and movement. Cutting the key space Once every piece of data has a key, the system divides the sorted key space into contiguous chunks. CockroachDB calls each chunk a range, YugabyteDB calls it a tablet, and Spanner calls it a split. Each chunk is replicated, typically three or five times, and each set of replicas forms its own consensus group. A large cluster runs many thousands of independent consensus groups simultaneously. A small metadata index maps key spans to the chunks that hold them. In CockroachDB this index is itself stored in special ranges, organized in two levels, called meta1 and meta2, so that the location of any key can be found in at most a few lookups, and nodes cache the results aggressively. YugabyteDB keeps tablet metadata in its master service, a separate small Raft group that also manages the catalog, and clients and tablet servers cache tablet locations. The size of each chunk is a balancing act. Small chunks spread load finely and move quickly when rebalancing, but each one carries overhead: a consensus group, heartbeats, metadata, and memory. Large chunks reduce overhead but make rebalancing slow and concentrate load. CockroachDB's default maximum range size has been 512 MiB since version 20.1; when a range grows beyond it, the range splits in two. YugabyteDB splits tablets automatically when they cross size thresholds that vary with the number of tablets a table already has, so that a new table splits aggressively at first and less often as it grows. Size is not the only reason to split. Load-based splitting divides a chunk that is receiving a disproportionate share of traffic, even if it is small. CockroachDB monitors each range's request rate or CPU usage and, when it exceeds a threshold, picks a split key that divides the load roughly in half. Spanner does the same, and its documentation describes load-based splitting as a core mechanism by which a database adapts to its workload. Load-based splitting is powerful, but it has a hard limit: it can only divide traffic that is spread across many keys. If all the traffic goes to one key, or to keys that always fall at the same end of a range, no split helps. Moving data around Splitting creates chunks; something must also decide where they live. Each system runs a background process that continually rebalances replicas across nodes. CockroachDB's allocator considers disk usage, replica counts, load, and the placement constraints the operator has configured, and moves replicas by adding a new replica on the target node, waiting for it to catch up through a snapshot and the Raft log, and then removing the old one. YugabyteDB's load balancer, running in the master leader, does the equivalent for tablets and also balances Raft leaders so that no single tablet server leads a disproportionate number of tablets. Rebalancing is what makes horizontal scaling real. Add a node to a CockroachDB or YugabyteDB cluster and, over the following minutes to hours depending on data volume, replicas migrate to it until it carries its share. No application change, no resharding project, no maintenance window. This is the capability that the two-year MySQL resharding effort described in Chapter 1 lacked. Rebalancing also has costs that operators must plan for. Moving a replica means copying its data across the network, and a cluster that has just lost a node must re-replicate every range that node held, which can saturate disks and network links. CockroachDB and YugabyteDB both throttle snapshot transfer rates for this reason, and a common operational surprise is how long a large cluster takes to restore full replication after losing a node holding several terabytes. Hash versus range distribution The most consequential design choice in this chapter is how primary keys map onto the key space, and here the systems diverge. Range distribution stores rows in primary key order. CockroachDB and Spanner use it by default; YugabyteDB uses it when the primary key is declared with ASC or DESC. Its great advantage is efficient range scans: a query for all orders between two dates, or all rows for a customer when the key begins with customer ID, reads one or a few contiguous chunks. Its great danger is sequential keys. If the primary key increases monotonically, every new row sorts after every existing row, and every insert lands in the last range of the table. That range's leaseholder handles every write, no matter how large the cluster. Load-based splitting cannot help, because each split leaves the hot tail on one side. Hash distribution stores rows by a hash of the key. YugabyteDB uses it by default for YSQL tables: the first primary key column is hashed unless declared otherwise. Sequential keys scatter uniformly across all tablets, so inserts spread across the cluster. The cost is that range scans over the hashed column become scatter-gather queries across every tablet. A query for "all orders with ID between 1000 and 2000" in a hash-sharded table must ask every tablet. YugabyteDB's syntax makes the choice explicit in the primary key declaration: -- Hash-distributed on tenant_id; rows for one tenant -- are sorted by created_at within the tenant's tablet. CREATE TABLE events ( tenant_id UUID, created_at TIMESTAMPTZ, event_id UUID, payload JSONB, PRIMARY KEY ((tenant_id) HASH, created_at ASC, event_id) ); This composite is a common and effective pattern. The hash component spreads tenants across the cluster; the range component keeps each tenant's events in time order, so the query "the last hundred events for this tenant" is a short scan on one tablet. Avoiding hotspots Because a hotspot on one range caps the throughput of an entire workload, every system documents ways to avoid them, and every team migrating from a single-node database needs to know them. The first rule is to avoid monotonically increasing primary keys in range-distributed tables. That rules out auto-increment integers and plain timestamps as leading key columns. Random UUIDs (version 4) distribute writes evenly and are the default recommendation in CockroachDB and Spanner. CockroachDB's gen_random_uuid() function generates them. Spanner's documentation, in addition to recommending UUIDs, offers bit-reversed sequences, which generate unique integers whose bits are reversed before use so that consecutive values land far apart in the key space. CockroachDB's handling of the SERIAL type deserves a specific warning. By default, CockroachDB maps SERIAL to its unique_rowid() function, which produces 64-bit integers built from a timestamp and a node identifier. These are unique without coordination, but they are roughly time-ordered, so inserts on a table keyed by them concentrate at the tail of the key space. They are far better than a true global sequence, which requires a round trip to a single range for every value, but they are not uniform. The second rule applies when the access pattern genuinely needs sequential ordering, as with a time-series table that is queried by time. CockroachDB's hash-sharded indexes handle this case. Declaring an index USING HASH adds a hidden computed column holding a hash of the indexed columns modulo a bucket count, sixteen by default, and prefixes the index with it. Writes spread across the buckets; a range scan on the original columns becomes a fixed number of parallel scans, one per bucket. CREATE TABLE readings ( sensor_id UUID, ts TIMESTAMPTZ, value FLOAT8, PRIMARY KEY (ts, sensor_id) USING HASH ); The third rule is to watch for hotspots that no key choice can prevent, because they are in the workload itself. A single counter row updated by every request, a "global settings" row read on every request and occasionally written, a queue table where every worker polls the same head row: these are single-key hotspots. The fixes are application-level. Counters can be sharded into many rows that are summed on read. Frequently read, rarely written rows can be placed in tables optimized for reads everywhere, discussed in Chapter 8. Queues can be partitioned by worker or replaced with a purpose-built queue. Spanner provides a tool for finding hotspots after the fact, called Key Visualizer, which renders access patterns across the key space over time. CockroachDB's DB Console offers a hot ranges page and, in recent versions, a key visualizer of its own. YugabyteDB exposes per-tablet metrics through its web UI and Prometheus endpoints. In every case the signal is the same: one range or tablet consuming far more CPU or receiving far more requests than its peers. The hidden hotspot in secondary indexes Teams that carefully choose random primary keys are often surprised to find a hotspot anyway. The cause is almost always a secondary index. Recall that every secondary index is stored as its own sorted set of keys, beginning with the indexed value. Consider an orders table with a random UUID primary key, which distributes rows evenly, and a secondary index on created_at, added so that the operations team can query recent orders. Every new order has a created_at value later than every existing one. The index entries therefore sort to the tail of the index's key space, and every insert writes one entry into the last range of the index. The table's primary data is spread across the cluster, but every insert also performs a write to one hot range on one machine. Throughput is capped by that range just as surely as if the primary key had been sequential. The cap is quantifiable in principle. A range's throughput is bounded by what its leaseholder can process and replicate through Raft, a figure that depends on hardware, row size, and replication topology. Whatever that figure is, a workload whose every insert touches the same range cannot exceed it, no matter how many nodes are added. A team that load-tests on a three-node cluster, sees acceptable throughput, and then provisions thirty nodes for production expecting ten times the capacity will be disappointed, because the bottleneck never moved. The fixes mirror those for primary keys. A hash-sharded index on created_at spreads entries across buckets. Alternatively, the index can lead with a column of higher cardinality that spreads writes, such as a store or tenant identifier, followed by the timestamp, which serves queries of the form "recent orders for this store" efficiently and spreads writes across stores. What the team should not do is assume that an index is a free addition. In a distributed database, every index is another table, with its own key distribution and its own potential hotspots, and each one adds a write to a possibly different range for every insert, which also widens the set of ranges a transaction must touch. A related design tool reduces how often indexes require extra work on reads. A covering index stores additional columns alongside the index key, so that a query satisfied by the index never needs to look up the primary row. CockroachDB expresses this with a STORING clause, and YugabyteDB and Spanner's PostgreSQL dialect accept PostgreSQL's INCLUDE. Spanner's GoogleSQL dialect uses STORING as well. On a single-node database, the extra lookup is a cheap local read. In a distributed database, the primary row may live in a different range on a different machine, so a covering index can remove a network round trip per row from a frequent query. Colocation and interleaving The opposite problem, too many tiny chunks, is real as well. A schema with hundreds of small reference tables, each in its own ranges or tablets, multiplies consensus groups and metadata for little benefit, and a join across many small tables can touch many machines. YugabyteDB addresses this with colocation. A database created with colocation enabled stores all of its tables, unless a table opts out, in a single tablet. Small tables share one Raft group, joins among them stay local, and only tables that grow large enough are given their own tablets. This is particularly useful for applications migrated from PostgreSQL with many small tables. Spanner's interleaving serves a related purpose from the other direction. By physically storing child rows with their parent, interleaving makes the common query "fetch this customer and all their orders" a read of one contiguous key span, typically in one split, and makes transactions that touch a parent and its children single-split transactions. Spanner's documentation is careful to note that interleaving is a performance hint: interleaved tables still behave as separate tables in SQL, and Spanner can split between a parent row and its children when an interleaved hierarchy grows large. CockroachDB once supported interleaved tables but deprecated and removed them, finding that the maintenance cost exceeded the benefit for most workloads given its other optimizations; teams wanting the same locality in CockroachDB achieve it by sharing a primary key prefix between parent and child tables. What the partitioning layer means for design Three practical lessons follow from this chapter and recur throughout the booklet. First, the primary key is a physical placement decision. Choose it for distribution and locality first, and for convenience second. In CockroachDB and Spanner, avoid sequential leading columns. In YugabyteDB, decide deliberately between hash and range for each table based on whether range scans or write distribution matter more. Second, locality of related data reduces cross-range work. When rows that are read or written together share a key prefix, they tend to live in the same range or tablet, which turns distributed operations into local ones. This idea scales up from a single node to an entire region, and it becomes the basis for multi-region schema design in Chapter 8. Third, splitting and rebalancing are automatic but not magic. They distribute keys and load that are already spread across the key space; they cannot spread load that the schema concentrates. With data partitioned and placed, the next question is how each range keeps its replicas in agreement, which is the job of consensus. Chapter 3: Consensus with Raft Every range in CockroachDB and every tablet in YugabyteDB is a small replicated state machine. Several machines each hold a copy of the data and apply the same sequence of commands in the same order, so that all copies stay identical. The hard part is agreeing on that sequence when machines crash, messages are delayed or lost, and networks partition. That agreement problem is called consensus, and the algorithm both systems use to solve it is Raft. Raft was published by Diego Ongaro and John Ousterhout at the USENIX Annual Technical Conference in 2014, in a paper titled "In Search of an Understandable Consensus Algorithm," which won the conference's best paper award. Ongaro's Stanford doctoral dissertation, completed the same year, expanded it with material on membership changes, log compaction, client interaction, and performance. The algorithm's stated design goal was understandability. Its authors had found Leslie Lamport's Paxos, the dominant consensus algorithm since the late 1980s, notoriously difficult to teach and, more importantly, difficult to turn into a complete working system, because the published descriptions left many practical details unspecified. Raft decomposes consensus into relatively independent subproblems and makes deliberate choices that reduce the number of states a system can be in. The paper included a user study with 43 students at Stanford and Berkeley, in which 33 of them scored higher on a quiz about Raft than on an equivalent quiz about Paxos. Spanner uses Paxos, not Raft. But Spanner's variant uses long-lived leaders and leader leases, and in practice behaves very much like Raft. Understanding Raft in depth is the most direct path to understanding all three systems' replication layers. The replicated log Raft maintains a log: an ordered list of entries, each containing a command for the state machine. In a database range, a command is something like "write these key-value pairs at this timestamp" or "resolve this transaction's intents." Each replica stores the log on disk and applies committed entries to its state machine, which is the actual key-value data, in log order. The goal is that every replica's log eventually contains the same entries in the same order, and that once an entry is committed, it will never be lost or changed. An entry is committed once it is stored on a majority of replicas. With three replicas, a majority is two; with five, it is three. A majority is the key to everything that follows, because any two majorities of the same group must overlap in at least one member. Whatever one majority agreed to, at least one member of any future majority remembers. Raft is a leader-based algorithm. At any moment, at most one replica is the leader, and all others are followers. Clients send commands to the leader. The leader appends each command to its own log, sends it to the followers, and once a majority, counting itself, have stored it, marks it committed, applies it, and responds to the client. Followers are passive: they accept entries from the leader and respond to its messages. This strong-leader design is one of Raft's simplifications relative to classic Paxos, where any replica can propose values and conflicts are resolved through a more complex protocol. Terms and leader election Raft divides time into terms, numbered with consecutive integers. Each term begins with an election. If a candidate wins, it serves as leader for the rest of the term. If the election splits and no one wins, the term ends with no leader and a new term begins. Terms act as a logical clock: every message carries the sender's term, and a replica that sees a higher term than its own immediately adopts it and, if it was a leader or candidate, steps down to follower. A replica that receives a message from an older term rejects it. This single rule cleanly handles a leader that was partitioned away and returns believing it is still in charge: its first message to anyone carries a stale term, and the reply tells it to step down. Leaders maintain authority by sending periodic heartbeats, which are simply append messages with no new entries. A follower that hears nothing for an election timeout assumes the leader has failed. It increments its term, becomes a candidate, votes for itself, and asks every other replica for its vote. Each replica votes for at most one candidate per term, on a first-come basis, subject to one crucial restriction discussed below. A candidate that collects votes from a majority becomes leader and immediately sends heartbeats to establish its authority. If two candidates split the vote, both time out and try again in a new term. The paper's solution to repeated split votes is randomization. Each replica chooses its election timeout randomly from a range; the paper suggests 150 to 300 milliseconds. Usually one replica times out first, becomes a candidate, and wins before anyone else wakes up. The paper's experiments showed that this simple mechanism elected leaders quickly and reliably. Production systems tune the range to their environment. CockroachDB's documentation describes a randomized Raft election timeout of two to four seconds, considerably longer than the paper's example, because its clusters span wide-area networks and must avoid unnecessary elections caused by transient slowness. Log replication and the matching property When the leader receives a command, it appends a new entry to its log, tagged with the current term and its index, and sends an AppendEntries message to each follower. The message includes not only the new entries but also the index and term of the entry immediately preceding them. A follower accepts the new entries only if its own log contains an entry at that preceding index with that term. If not, it rejects the message, and the leader retries with an earlier preceding index, walking backward until it finds the point where the follower's log agrees with its own. The follower then deletes any conflicting entries after that point and appends the leader's. This consistency check produces what the paper calls the Log Matching Property: if two logs contain an entry with the same index and term, the logs are identical in all entries up through that index. The property holds by induction. The leader creates at most one entry per index in a given term, and every append is checked against the entry before it. The consequence is that the leader never needs to compare whole logs; matching one entry implies matching everything before it. The design has a clear direction of authority. The leader's log is always right, and followers are overwritten to match it. Leaders never delete or overwrite their own entries. That simplicity is only safe if the elected leader is guaranteed to have every committed entry, which brings us to the safety argument. Why Raft is safe The central danger in any consensus algorithm is that a new leader might lack an entry that was already committed, then overwrite it on followers, losing data that a client was told was durable. Raft prevents this with the election restriction. When a candidate requests a vote, it includes the index and term of its last log entry. A voter refuses to vote for any candidate whose log is less up to date than its own, where "up to date" means the last entry has a higher term, or the same term and a higher or equal index. Here is why that suffices. A committed entry is stored on a majority. A winning candidate received votes from a majority. Those two majorities overlap in at least one replica, which holds the committed entry. That replica voted for the candidate only if the candidate's log was at least as up to date as its own. The paper shows that this implies the candidate's log contains the committed entry. The guarantee is called Leader Completeness: any entry committed in a given term is present in the logs of the leaders for all higher terms. There is one subtle case the paper treats carefully. A leader cannot conclude that an entry from a previous term is committed merely because it now sits on a majority of replicas. The paper constructs a sequence of leader changes in which an old-term entry reaches a majority, the leader crashes, and a different replica with a competing entry from a later term wins and overwrites it. Raft's fix is a rule about when the leader counts replicas: a leader only commits entries from its own current term by counting replicas. Once an entry from its current term is committed, all earlier entries in its log are committed indirectly, by the Log Matching Property. In practice, a newly elected leader commits a no-op entry at the start of its term to flush out any uncommitted entries from predecessors. Together, these rules give Raft's main guarantee, State Machine Safety: if any replica has applied an entry at a given index, no other replica will ever apply a different entry at that index. Every replica's state machine passes through the same sequence of states. The paper backs this with a proof, and the authors also produced a formal specification in TLA+, Lamport's specification language, which others have used to model-check the algorithm. Reads, and the problem of the deposed leader Writes in Raft are straightforward: commit through the log. Reads are subtler. A leader might naively answer reads from its own state machine. But a leader that has been partitioned away from the majority might not know it has been replaced. It would happily serve stale data while a new leader, elected by the majority on the other side, accepts writes it never sees. That violates linearizability. The dissertation describes several fixes. The safest is to send reads through the log like writes, which costs a full consensus round per read. A cheaper alternative is for the leader, before answering a read, to confirm that it is still leader by exchanging heartbeats with a majority; this costs a round trip but no disk writes. The cheapest is a lease: the leader relies on the fact that followers promise not to elect a new leader for some period after hearing from it, so for that period it can answer reads locally with no communication at all. Leases depend on bounded clock drift, since the leader and followers must agree on how long the period lasts. Production distributed databases use leases, because local reads are essential for performance. Spanner's paper describes Paxos leader leases that default to ten seconds, renewed by successful writes. CockroachDB long separated the concept into a leaseholder, the replica permitted to serve reads and coordinate writes for a range, and the Raft leader, and worked to keep the two on the same replica. Starting in v25.2, CockroachDB made a new mechanism called leader leases the default. Leader leases merge the two roles, so the Raft leader is always the leaseholder except briefly during transfers. They are built on a decentralized store liveness mechanism and on a strengthened form of Raft leadership the CockroachDB team calls fortification, in which followers promise support to a leader and will not campaign while that support holds. The CockroachDB documentation states that with leader leases, network partitions heal in under twenty seconds and liveness failures cause outages of under one second, and that the change removed the previous design's dependency on a single node liveness range. YugabyteDB similarly ties leader leases to Raft leadership, so that the tablet leader serves consistent reads locally. Membership changes Clusters change. Nodes are added, removed, and replaced, and replicas move between nodes during rebalancing. Changing the set of replicas in a consensus group is dangerous, because during the transition the old and new configurations might each elect a leader with their own separate majority. The Raft paper's solution is joint consensus. To move from configuration C-old to C-new, the leader first commits a transitional configuration in which decisions require separate majorities from both C-old and C-new. Once that is committed, it commits C-new alone. At no point can two independent majorities form. Ongaro's dissertation later described a simpler alternative that changes membership one server at a time, since adding or removing a single server cannot create disjoint majorities. Single-server changes are easier to implement, but a subtle bug in the original description of the approach was found and corrected after publication, a reminder that even small consensus details deserve formal scrutiny. CockroachDB uses joint consensus for replica changes so that it can atomically swap one replica for another, for instance moving a replica from a failing node to a healthy one in a single step, without passing through a configuration with fewer replicas than intended. Extensions production systems need The core algorithm is not the end of the story. Running thousands of Raft groups on real networks reveals failure modes that the paper's model treats as edge cases. Pre-vote addresses a disruptive replica. A replica separated from the cluster by a partition will repeatedly time out, increment its term, and campaign without success. When the partition heals, its inflated term forces the healthy leader to step down, causing an unnecessary election. With pre-vote, described in the dissertation, a would-be candidate first asks whether it could win an election before incrementing its term. Replicas that have heard from a current leader recently say no, and the disruption never happens. Check quorum addresses the opposite problem: a leader that has lost contact with a majority but still believes itself leader. With check quorum, a leader that has not heard from a majority within an election timeout steps down voluntarily. Partial partitions, in which some links fail but not others, can otherwise cause leadership to flap between replicas indefinitely. Cloudflare published a detailed account of a November 2020 incident in which a switch entered a partially working state for about six minutes, dropping some traffic while its control protocols looked healthy. Members of an etcd cluster, which uses Raft, ended up with conflicting views of who could reach whom; elections tied repeatedly without producing a leader every member could reach, and the cluster went read-only. Knock-on effects stretched the incident to more than six hours. Partial connectivity is exactly the condition that the textbook model of "up or down" nodes does not cover, and it is where these extensions earn their keep. Multi-Raft optimizations address scale. A node in a large CockroachDB cluster may host replicas of tens of thousands of ranges. Sending separate heartbeats for each would overwhelm the network. CockroachDB coalesces heartbeats between each pair of nodes into a single message, and quiesces idle ranges entirely, stopping their heartbeats until new traffic arrives. YugabyteDB, whose Raft implementation descends from the one in Apache Kudu, applies similar batching. Log compaction keeps the log from growing forever. Once entries are applied and durable in the state machine, the log can be truncated. A follower that falls too far behind, or a new replica, receives a snapshot of the state machine instead of the full log. In a database, the snapshot is a copy of the range's data, which is why replica movement and recovery involve significant data transfer, as noted in Chapter 2. What consensus costs Raft's guarantees come with a fixed price per write: the leader must persist the entry and hear back from enough followers to form a majority. Followers can respond in parallel, so the latency is roughly one round trip from the leader to the nearest majority of followers, plus disk write time. With three replicas in one region spread across availability zones a millisecond or two apart, that is a few milliseconds. With five replicas spread across regions, it is the round trip to the second-closest remote region, often tens of milliseconds. Leases make the read side cheap, but only at the leaseholder. A read served by a follower requires extra machinery to be consistent, discussed in Chapters 4, 5, and 7. And consensus provides agreement within one range only. A transaction touching three ranges needs three consensus groups to commit consistently with one another, and they need a shared notion of when things happened. That shared notion is time, the subject of the next two chapters. Chapter 4: TrueTime and External Consistency Consensus solves agreement within a single range. It does not, by itself, tell you how transactions across different ranges relate to one another in time. Suppose a user in Tokyo commits a transaction that writes to a range whose leader is in Japan, and a moment later a user in Belgium, who has been told about the first transaction by phone, starts a transaction that reads that row and writes to a range whose leader is in Europe. Both ranges run their own consensus. Nothing in Raft or Paxos tells the European range that the Japanese transaction happened first. If the database assigns the second transaction an earlier timestamp than the first, a snapshot read could see the second transaction's effects without the first's, an outcome no human observer would accept. The obvious solution is a global ordering service: a single machine, or a single consensus group, that hands out increasing timestamps. Several systems do exactly this. Google's Percolator, described at OSDI in 2010, used a centralized timestamp oracle, and TiDB, a distributed SQL system not covered in depth here, uses a similar placement driver. A central oracle is simple and correct, but every transaction must contact it, which adds a round trip to wherever it lives. For a database spanning continents, that round trip can be a hundred milliseconds or more, and the oracle becomes a scaling and availability concern in its own right. Spanner's designers took a different path. Instead of asking one place what time it is, they made every machine's clock trustworthy enough that each could pick timestamps locally, and then paid a small, bounded wait to cover the remaining uncertainty. That idea, and the hardware investment behind it, is TrueTime. Clocks you can trust, within bounds Every computer has a clock, and every clock is wrong. Quartz oscillators drift by parts per million, which accumulates to many milliseconds per day. Network Time Protocol corrects drift by polling time servers, but NTP over a general-purpose network typically keeps clocks within some milliseconds of each other at best, with no hard guarantee: a congested link, an asymmetric route, or a misconfigured server can push a clock tens or hundreds of milliseconds off without anyone noticing. The Spanner paper's key observation is that the problem is not that clocks are imprecise but that their imprecision is unknown. If a system knew, at every moment, a bound on how wrong its clock could be, it could reason about ordering safely. TrueTime exposes exactly that. Its central call, TT.now(), does not return a timestamp. It returns an interval, [earliest, latest], with a guarantee that the true absolute time at the moment of the call lies within the interval. Two helper calls follow: TT.after(t) returns true if t has definitely passed, and TT.before(t) returns true if t has definitely not yet arrived. The half-width of the interval is called epsilon. Making the bound real required hardware. The paper describes a set of time master machines in every data center. Most are equipped with GPS receivers with dedicated antennas. The rest, which the paper calls Armageddon masters, carry atomic clocks. The two references were chosen because they fail differently: GPS can be lost to antenna problems, radio interference, or system-wide issues, while atomic clocks drift slowly and independently of any external signal. Masters compare their time against each other, and a master whose local time diverges significantly from the others evicts itself. Every other machine runs a timeslave daemon that polls a variety of masters, both in its own data center and farther away, and both GPS and atomic. It applies a variant of Marzullo's algorithm, a classic method for combining interval estimates from several sources, to detect and reject masters that disagree with the consensus, then synchronizes the local clock. Between polls, the daemon widens its uncertainty to account for possible drift, assuming a conservative worst-case drift rate of 200 microseconds per second. The paper reports a poll interval of 30 seconds, so epsilon follows a sawtooth pattern, rising from about 1 millisecond just after a poll to about 7 milliseconds just before the next, and averaging around 4 milliseconds. Machines whose local clocks drift faster than the assumed bound are detected and removed. The important property is not that epsilon is small, although it is. It is that epsilon is known. The rest of Spanner's design depends on that knowledge. Commit wait Armed with TrueTime, Spanner assigns each read-write transaction a commit timestamp and enforces a simple rule that guarantees external consistency. The paper states the invariant precisely: if the commit of transaction T2 starts after the commit of T1 finishes, then T2's commit timestamp is greater than T1's. Two rules achieve it. The first, which the paper calls the start rule, says that the leader coordinating a transaction's commit chooses a commit timestamp s no less than TT.now().latest at the time it decides to commit. Because latest is an upper bound on the current true time, s is at least the true time at that moment, and possibly a little later. The second, the commit wait rule, says that the coordinator does not let anyone see the transaction's effects, either by releasing its locks or by acknowledging the client, until TT.after(s) is true. That is, it waits until the timestamp it chose is definitely in the past. Together the rules guarantee the invariant. When T1's client is told the commit succeeded, the true time is already past T1's timestamp. Any transaction T2 that starts after that point picks a timestamp at least as large as the true time when it decides to commit, which is later still. So T2's timestamp is larger than T1's. No coordination between the two transactions' leaders is required. They might be on different continents, in different consensus groups, with no shared state at all; the bounded clock does the work that a central oracle would otherwise do. The cost is the wait. In the paper's analysis, the expected commit wait is at least twice the average epsilon, since the coordinator picks a timestamp at the top of the uncertainty interval and waits until the bottom of a later interval passes it. With an epsilon around 4 milliseconds, that is roughly 8 milliseconds or so. Crucially, the wait overlaps with other work. The coordinator is simultaneously waiting for Paxos to replicate the commit record to a majority, which in a multi-region configuration often takes longer than the commit wait itself. In practice, commit wait mostly hides behind replication latency. The design creates an incentive that Google pursued for years after the paper: every reduction in epsilon directly reduces commit latency. The paper itself mentions that improvements to the time infrastructure were under way to reduce the tail of epsilon values, and the system has been refined considerably since. Commit wait, step by step A concrete timeline makes the mechanism clearer. Suppose epsilon is currently 4 milliseconds everywhere, and use a simple clock in which true time is measured in milliseconds. At true time 1000, the coordinator leader for transaction T1 is ready to commit. It calls TT.now() and receives the interval from 996 to 1004. By the start rule, it picks commit timestamp 1004, the top of the interval. It immediately begins replicating the commit record through Paxos. By the commit wait rule, it cannot release locks or acknowledge the client until TT.after(1004) is true, which means until the bottom of the current interval exceeds 1004. That happens at true time 1008, when TT.now() returns the interval from 1004 to 1012, and a moment later. Suppose Paxos replication finishes at true time 1006. The coordinator still waits until just past 1008, then acknowledges T1's client. Now a user who received that acknowledgment starts transaction T2, at true time 1009 or later, on a completely different machine in a different Paxos group. When T2 commits, its coordinator calls TT.now() at some true time of at least 1009, receives an interval whose top is at least 1009, and picks a commit timestamp at least that large, which is greater than 1004. The ordering is guaranteed without the two coordinators ever communicating. The example also shows why the wait is often hidden. If the Paxos round had taken 30 milliseconds, as it might with replicas in another region, replication would have finished at true time 1030, long after commit wait completed at 1008. The wait would have cost nothing. Commit wait dominates only when replication is very fast, as in a single-region configuration with replicas a millisecond apart, and even there it adds only a few milliseconds. What if the bound is wrong? Everything depends on TrueTime's interval actually containing the true time. If a machine's clock drifted faster than the assumed bound without detection, it could produce intervals that do not contain the true time, and a transaction could pick a timestamp that is too low, breaking external consistency. The Spanner paper addresses this directly. It argues that clocks are among the more reliable components in a machine, reporting that Google's machine statistics showed bad CPUs to be six times more likely than bad clocks. The timeslave daemon checks the local oscillator's behavior against the masters, and machines whose drift exceeds the bound are evicted. The time masters check each other. The worst-case drift assumption of 200 microseconds per second is deliberately conservative relative to the behavior of real oscillators. The design is not that failures cannot happen, but that clock failures are detected and handled like any other hardware fault, by taking the faulty machine out of service. This is an important point for anyone tempted to emulate TrueTime with less rigorous infrastructure. The value of TrueTime lies entirely in the reliability of its bound. An interval that is usually correct but occasionally wrong provides no guarantee at all. The engineering effort behind TrueTime went less into making clocks accurate than into making the reported uncertainty trustworthy. Transactions across Paxos groups Read-write transactions that touch a single Paxos group commit through that group's leader. Transactions touching several groups use two-phase commit, with Paxos making each participant reliable. Spanner's read-write transactions use pessimistic locking. As the transaction reads, it acquires read locks at the leaders of the relevant groups. Writes are buffered at the client until commit. To avoid deadlock, Spanner uses the wound-wait scheme: when a transaction requests a lock held by a younger transaction, it forces the younger one to abort; when it requests a lock held by an older transaction, it waits. Older transactions therefore always make progress. At commit time, the client chooses one participant group as the coordinator and sends its buffered writes to every participant leader. Each non-coordinator participant acquires write locks, chooses a prepare timestamp greater than any timestamp it has previously assigned, logs a prepare record through Paxos, and tells the coordinator its prepare timestamp. The coordinator acquires its own write locks, waits for all participants, and chooses a commit timestamp no less than every prepare timestamp, greater than TT.now().latest, and greater than any timestamp it has previously assigned. It logs the commit record through Paxos, performs commit wait, and then tells the client and all participants the outcome. Participants log the outcome through Paxos, apply the writes at the commit timestamp, and release locks. Every step that matters is recorded in a Paxos group, which is what turns two-phase commit from a blocking protocol into a fault-tolerant one. If the coordinator leader crashes, its Paxos group elects a new leader that has the coordinator's state. The participants never wait on a single dead machine. Lock-free reads at a timestamp TrueTime's second benefit is arguably more important in practice than external consistency for writes: it makes consistent reads cheap. Spanner stores multiple versions of every row, each tagged with the commit timestamp of the transaction that wrote it. A read at timestamp t returns, for each row, the latest version with a timestamp no greater than t. Such a read sees a consistent snapshot of the entire database as of t, and it needs no locks at all, because old versions are immutable. The paper defines read-only transactions, which Spanner executes as snapshot reads. For a read-only transaction to be externally consistent, Spanner assigns it a timestamp at least TT.now().latest, so it sees every transaction committed before it started. Any replica can serve the read, not only the leader, provided the replica is sufficiently up to date. Each replica tracks a safe time, the maximum timestamp at which it knows it has every committed write. Safe time is the minimum of two values: the timestamp of the latest Paxos write the replica has applied, and a bound from the transaction manager that accounts for transactions that have prepared but not yet committed. If a read's timestamp is at or below a replica's safe time, the replica can answer immediately. Otherwise it waits, or asks the leader to catch it up. Spanner also offers stale reads, in which the client asks for data as of a timestamp in the past, either an exact timestamp or a bounded staleness such as "no more than fifteen seconds old." Stale reads can be served by any replica whose safe time has passed the requested timestamp, which is usually the nearest one, with no communication with the leader. Google's documentation describes stale reads as offering performance similar to eventual consistency with much stronger guarantees: the data may be slightly old, but it is a consistent snapshot, never a mixture of states. For read-heavy, latency-sensitive workloads in multi-region configurations, stale reads are the single most effective tool Spanner offers. Timestamps also enable atomic schema changes. Spanner can assign a schema change a timestamp in the future, and transactions with timestamps before it use the old schema while transactions after it use the new one. A schema change across thousands of servers becomes a single atomic event at a known instant, without blocking ongoing work. Spanner as a product For most of its public life, Spanner has been available only as a managed service on Google Cloud. It offers two SQL dialects, GoogleSQL and a PostgreSQL dialect with its own interface, and three editions. According to Google's documentation, the Standard edition supports regional configurations with a 99.99 percent availability SLA; the Enterprise edition adds features such as Spanner Graph, full-text and vector search, managed autoscaling, and a columnar engine; and the Enterprise Plus edition adds dual-region and multi-region configurations, geo-partitioning, and an SLA of up to 99.999 percent. Serializable isolation with external consistency is the default, and Spanner has added a repeatable read isolation level, implemented as snapshot isolation, for workloads that can tolerate its anomalies in exchange for fewer aborts. TrueTime itself is not directly exposed as a customer-facing clock API. Applications benefit from it indirectly, through commit timestamps and consistent reads. In April 2026 Google announced Spanner Omni, a downloadable version of Spanner that runs on premises, in other clouds, on Kubernetes, and even on a single laptop, with a free developer edition in preview and a commercial edition in early access. Omni is revealing for this chapter's argument. Outside Google's data centers there are no GPS-and-atomic-clock time masters, so Google built what it describes as a software-based TrueTime alternative that provides error-bounded time synchronization across the servers of a deployment. The design commitment that survives is not the hardware but the interface: a clock that reports its own uncertainty, which the database then pays for explicitly. What TrueTime teaches TrueTime is often described as Spanner's secret weapon, which makes it sound like magic that only Google can afford. The more useful framing is that TrueTime converts clock uncertainty from an unknown risk into a known, bounded latency cost. Spanner pays epsilon, twice, on every read-write transaction, and in return eliminates coordination between leaders and gets lock-free consistent reads from any sufficiently fresh replica. That framing makes the next chapter's question precise. What happens when you build the same architecture on clocks whose uncertainty is not tightly bounded by dedicated hardware but loosely bounded by configuration? The answer, in CockroachDB and YugabyteDB, is that the cost moves from a commit wait that every write pays to an uncertainty penalty that some reads pay, and the guarantees shift accordingly. Chapter 5: Living Without Atomic Clocks When Cockroach Labs began building CockroachDB in 2014 and 2015, its founders faced a constraint that Google did not: their customers would run the database on ordinary servers, in ordinary clouds and data centers, with clocks synchronized by NTP and nothing more. YugabyteDB, founded shortly after by engineers who had built Apache HBase and Cassandra infrastructure at Facebook, faced the same constraint. Both wanted Spanner's architecture. Neither could rely on Spanner's clocks. Their solution, which both arrived at independently and which differs in details, combines three ideas: a hybrid logical clock that never runs backward and carries causality between nodes, a configured maximum clock offset that the whole cluster promises to respect, and an uncertainty interval that turns that promise into correct reads. The result preserves serializability on commodity hardware, at the cost of occasional read restarts and a weaker real-time guarantee than Spanner's. Understanding exactly what is given up, and when it matters, is the main business of this chapter. Hybrid logical clocks There are two classic ways to timestamp events in a distributed system. Physical clocks read the machine's wall clock. Their timestamps are meaningful to humans and roughly comparable across machines, but they can drift, jump, or disagree. Logical clocks, introduced by Leslie Lamport in 1978, are counters that each node increments on every event and advances past any timestamp it receives in a message. Lamport clocks capture causality perfectly, in the sense that if one event could have influenced another, its timestamp is smaller, but the numbers bear no relation to real time. A hybrid logical clock, or HLC, combines them. The idea was described by Sandeep Kulkarni, Murat Demirbas, and colleagues in a 2014 paper titled "Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases." An HLC timestamp has two parts: a physical component, close to the node's wall clock, and a logical counter used to break ties and to preserve causality when the physical component cannot advance. When a node generates a timestamp, it takes the maximum of its wall clock and the last HLC timestamp it issued or received; if the physical part did not advance, it increments the logical counter. When a node receives a message carrying an HLC timestamp ahead of its own, it advances its clock to at least that value. Two properties follow. HLC timestamps never go backward on a node, even if its wall clock jumps back. And causality is carried by messages: if node A sends anything to node B, B's subsequent timestamps exceed the timestamp A attached. Because the physical component tracks the wall clock, HLC timestamps also stay close to real time, typically within the clock offset between nodes. CockroachDB's HLC piggybacks timestamps on every message between nodes, so the clocks of nodes that communicate are continually pulled forward to match the fastest among them. YugabyteDB's equivalent, called hybrid time, works the same way: its timestamps combine microsecond physical time with a logical component, and are propagated on every RPC. The maximum offset and the uncertainty interval HLCs preserve causality for events connected by messages. They do not solve the problem that motivated TrueTime: two transactions with no communication between them, whose order in real time must still be respected. Suppose transaction T1 commits on node A, whose clock runs slightly fast, at timestamp 100. Moments later, in real time, transaction T2 starts on node B, whose clock runs slightly slow, and picks timestamp 95. If T2 reads the row T1 wrote, a naive multi-version store would show it the version before T1, because 95 is less than 100. T2 would miss a write that, in real time, had already committed when T2 began. That is a stale read. CockroachDB and YugabyteDB prevent stale reads with a cluster-wide promise and a check. The promise is the maximum clock offset: an upper bound on how far any node's clock may be from any other's. The default in both systems is 500 milliseconds. CockroachDB's --max-offset flag and YugabyteDB's max_clock_skew_usec flag configure it. The check is the uncertainty interval. When a transaction starts with timestamp t, it also records an upper bound t + max_offset. Any value it encounters with a timestamp in the interval between t and that upper bound is uncertain: it might have been written by a transaction that committed before this one started in real time, on a node whose clock was ahead. The transaction cannot ignore an uncertain value, since that could be a stale read. It cannot simply read it either, since the value might genuinely be from the future. So it moves its own timestamp forward past the uncertain value, which guarantees the value is visible, and then deals with the consequences. If the transaction has not yet read anything else, moving its timestamp is free. If it has, the earlier reads were performed at the old timestamp and might now be inconsistent with the new one. CockroachDB attempts a read refresh: it re-checks the spans it previously read to see whether anything changed between the old and new timestamps. If nothing did, the transaction continues at the new timestamp with no visible effect. If something did, the transaction must restart. YugabyteDB surfaces the equivalent situation as a read restart, which it retries transparently in many cases, especially for the first statement of a transaction, and otherwise returns to the client as a "restart read required" error. Because every transaction has an uncertainty interval, a transaction never misses a write that committed before it started, as long as the maximum offset holds. The cost is borne by reads that happen to encounter recently written data, which may need to bump their timestamps or restart. Narrowing the interval A 500-millisecond uncertainty window is large. A table receiving many writes per second will often have values in any reader's uncertainty interval, causing frequent timestamp bumps and refreshes. Both systems use several techniques to narrow the window in practice. The most important in CockroachDB is the observed timestamp. When a transaction first visits a node, it records that node's current clock reading. Any value on that node with a timestamp above the observed reading must have been written after the transaction first arrived, since the node's own clock had not yet reached it, and therefore cannot have committed before the transaction started in real time. So for all subsequent reads on that node, the transaction's uncertainty upper bound is reduced to the observed timestamp. In practice, this means a transaction's uncertainty only matters for values written in the brief window before it first reached each node, and the 500-millisecond worst case is rarely realized. The second technique is better clocks. The maximum offset is a configuration, not a law. Operators who can guarantee tighter synchronization can reduce it, and Cockroach Labs has written that it is generally safe to reduce the maximum offset to 250 milliseconds with properly configured clock synchronization. Lowering it shrinks every uncertainty interval proportionally. Cloud providers have made much better clocks available. In November 2023, AWS announced that its Time Sync Service, using a precision hardware clock exposed through the Nitro system on supported EC2 instances, delivers clock accuracy in the low double-digit microseconds, and released an open-source daemon called ClockBound that reports a computed error bound with each reading, much as TrueTime does. YugabyteDB integrated ClockBound so that, when enabled, each node's uncertainty interval is derived from the measured error bound rather than the fixed 500-millisecond configuration. Yugabyte's own demonstration showed read restart errors disappearing for a workload that triggered them under the default setting. CockroachDB supports reading time from a PTP hardware clock device via its --clock-device flag. These developments substantially close the gap between commodity deployments and Spanner's dedicated time infrastructure, at least in clouds that offer them. An uncertainty restart, step by step A concrete sequence shows how these pieces interact. Take a CockroachDB cluster with the default 500-millisecond maximum offset. Node A's clock runs 100 milliseconds ahead of true time; node B's runs accurately. Express timestamps in milliseconds of each node's clock. At true time 10,000, a transaction T1 coordinated through node A writes a new value for key K, whose leaseholder is on node A. T1 receives timestamp 10,100 from A's fast clock and commits. At true time 10,050, after T1's client has been told of the commit, transaction T2 starts through node B and receives timestamp 10,050 from B's accurate clock. Its uncertainty upper bound is 10,550. T2 reads K. The leaseholder finds T1's committed value at timestamp 10,100, which is above T2's timestamp of 10,050. A naive snapshot read at 10,050 would skip it and return the older value, a stale read, since T1 committed before T2 began. But 10,100 lies inside T2's uncertainty interval, between 10,050 and 10,550, so T2 cannot ignore it. T2 advances its read timestamp to just above 10,100 and reads T1's value. If K was T2's first read, nothing else is affected, and the client never knows anything happened. If T2 had already read other keys at 10,050, it refreshes those reads at the new timestamp. If none of them changed between 10,050 and 10,100, it continues. If one did, T2 restarts with a new timestamp, and the client may see a retryable error, as described in the next chapter. Now suppose the same scenario, but T2 had already visited node A earlier in its life, at true time 10,020, when A's clock read 10,120. T2's observed timestamp for node A is therefore 10,120. That does not help here, since T1's value at 10,100 is below 10,120 and could have been written before T2 started. But any value on node A with a timestamp above 10,120 is known to have been written after T2 first arrived there, and T2 can safely ignore it without bumping its timestamp. The effective uncertainty window on node A shrinks from 500 milliseconds to the brief interval between T2's start and its first visit. The example also shows what happens if the promise is broken. Had node A's clock been 700 milliseconds ahead instead of 100, T1's timestamp would have been 10,700, outside T2's uncertainty window. T2 would have treated the value as being from its future, ignored it, and returned the old value: a stale read that violates serializability. That is why exceeding the maximum offset must never be allowed to persist, and why nodes are shut down when it is detected. Table 2 summarizes how the three systems handle clock uncertainty; figures are from the Spanner OSDI paper, CockroachDB and YugabyteDB documentation, and AWS's 2023 Time Sync announcement. Table 2. How each system copes with clock uncertainty. Aspect Spanner CockroachDB YugabyteDB Time source GPS and atomic clock masters NTP, optionally PTP device NTP, optionally AWS ClockBound Uncertainty bound Measured epsilon, about 1 to 7 ms (2012) Configured, 500 ms default Configured 500 ms, or measured via ClockBound Who pays Every read-write commit (commit wait) Reads meeting recent writes Reads meeting recent writes How paid Wait about 2 epsilon Timestamp bump, refresh, or restart Read restart, often automatic Real-time order Strict, all transactions Per key, plus no stale reads Per key, plus no stale reads What happens when the clock lies The uncertainty interval protects correctness only if the maximum offset actually holds. If a node's clock drifts beyond it, a transaction on that node could pick a timestamp so far behind another node's writes that the writes fall outside its uncertainty interval, and the transaction would read stale data, violating serializability. Both systems therefore treat clock offset as a hard safety boundary and enforce it actively. CockroachDB nodes continuously measure their offset from peers using round trips of their regular communication. According to Cockroach Labs, if a node detects that its clock is more than 80 percent of the maximum offset away from at least half of the other nodes, which is 400 milliseconds under the default, it shuts itself down rather than risk serving inconsistent data. The error message operators then see in logs is unmistakable, reporting that the node is too far from the known nodes. YugabyteDB similarly tracks skew and rejects or crashes on unsafe offsets. This turns a potential correctness problem into an availability problem, which is the right trade. But it means that time synchronization becomes a production dependency of the same order as disks and networks. Several failure modes recur in practice. Virtual machines that are paused, live-migrated, or starved of CPU can see their clocks jump. A misconfigured NTP setup that points all nodes at a single upstream server will let them drift together, then diverge if that server fails. Leap seconds, when they occurred, caused jumps in some configurations; Cockroach Labs recommended smearing the leap second gradually rather than stepping the clock. An operator who disables NTP on a host to debug an unrelated problem can take a node out of the cluster without realizing why. The operational rules are straightforward. Use a reliable, redundant time source, ideally the cloud provider's own time service, which is designed for the purpose. Monitor clock offset as a first-class metric and alert well before the self-termination threshold. Configure consistent leap-second handling across all nodes. Do not reduce the maximum offset below what your synchronization can actually guarantee under adverse conditions, because the consequence of being wrong is stale reads rather than a mere warning. The real-time guarantee, precisely Spanner's commit wait gives external consistency: if T1 commits before T2 starts, T2's timestamp is higher, regardless of which data either touches. CockroachDB and YugabyteDB do not perform commit wait by default, so they cannot give quite that guarantee. What they guarantee instead is that reads are never stale and that operations on any single key are linearizable. If T1 writes key K and commits, and T2 later reads K, T2 sees T1's write, because either T1's timestamp is below T2's or T1's value falls in T2's uncertainty interval and T2 moves forward. The ordering of transactions that touch overlapping data respects real time. The gap concerns transactions on disjoint keys. Suppose T1 writes key A on a node with a fast clock, receiving timestamp 100, and commits. Then, strictly later in real time, T2 writes key B on a node with a slow clock and receives timestamp 90. T1 and T2 touch no common data, so neither saw the other. A third transaction reading both A and B at timestamp 95 would see T2's write but not T1's, implying T2 happened first even though T1 committed earlier in real time. Cockroach Labs' early engineering writing called this the causal reverse anomaly. Whether this matters depends on whether the application relies on real-time order between unrelated writes communicated outside the database. The classic risk scenario involves a side channel: a system writes a record, then sends a message through an external queue to another service, which writes an unrelated record that is supposed to be causally after the first. A later reader might observe them out of order. In most applications the anomaly is invisible, because causally related writes either touch common data or flow through the same client session, which carries its HLC timestamp forward. CockroachDB has long offered an option to perform a Spanner-style commit wait of the full maximum offset, recovering strict serializability at the cost of adding up to the maximum offset to every commit, a price few deployments accept with NTP-grade clocks. As clock bounds shrink toward microseconds, that calculation changes, and the boundary between these systems and Spanner becomes a matter of degree. Where the costs land The comparison with Spanner comes down to who pays for clock uncertainty and how often. Spanner charges every read-write transaction a commit wait of roughly twice epsilon, which is small because its clocks are excellent, and which mostly hides behind replication latency. In return, every read at any replica can be served without uncertainty. CockroachDB and YugabyteDB charge nothing on commit. They charge readers that encounter recent writes in their uncertainty window, in the form of timestamp bumps, refreshes, and occasional restarts. On workloads with little read-write contention on hot rows, the charge is negligible. On workloads where readers constantly scan recently written data, it can be significant, and it shows up in metrics as read restarts or uncertainty errors. Reducing the maximum offset, using precise clock sources, and avoiding hot-spot reads of just-written data all reduce it. Both designs are coherent, and neither is free. The rest of the transactional machinery, how these timestamps are used to commit writes across many ranges atomically, is the subject of the next chapter. Chapter 6: Distributed Transactions in Practice A consensus group guarantees that one range's writes are durable and ordered. A clock, whether TrueTime or a hybrid logical clock, gives every transaction a timestamp. What remains is the protocol that ties them together: how a transaction that writes rows in a dozen ranges, on a dozen different machines, commits all of them atomically, isolates itself from concurrent transactions, and recovers when its coordinator dies halfway through. This chapter takes CockroachDB's transaction protocol as the main example, because its SIGMOD 2020 paper and documentation describe it in unusual detail, then compares YugabyteDB's and Spanner's choices. The protocols differ in their mechanics but face identical problems, and the problems are what an application designer needs to understand. Write intents and the transaction record When a CockroachDB transaction writes a row, it does not write a final value. It writes a write intent: a provisional value at the transaction's current timestamp, combined with an exclusive lock on the key and a pointer to the transaction's transaction record. The transaction record is a small piece of state stored in one of the ranges the transaction writes to, typically the range of its first write. It holds the transaction's status, which is one of PENDING, STAGING, COMMITTED, or ABORTED. The record is the single source of truth about whether the transaction committed. A reader that encounters an intent does not know, from the intent alone, whether to see it. It follows the pointer to the record. If the record says COMMITTED, the intent is treated as a committed value at the transaction's commit timestamp. If ABORTED, the intent is ignored. If PENDING, the reader is in conflict with a live transaction and must decide what to do, described below. Because the transaction record lives in a Raft-replicated range, its state survives the failure of any single machine. That is the property that makes this a non-blocking form of two-phase commit. In classic two-phase commit, a participant that has voted yes and lost contact with the coordinator must wait, possibly forever, because only the coordinator knows the outcome. Here, any participant or reader can look up the outcome directly in the record. Once a transaction commits, its intents still need to be converted into ordinary committed values, a process called intent resolution. CockroachDB does this asynchronously after acknowledging the client, so resolution does not add to commit latency. Readers that encounter an intent before it is resolved simply consult the record and resolve it themselves if they find it committed. A transaction coordinator periodically heartbeats its record to show it is still alive. If a coordinator crashes, its heartbeats stop, and any other transaction that encounters its intents and finds the record stale can mark it ABORTED and clean up. No intent is left locked forever because a client disappeared. Making commit cheap The straightforward version of this protocol requires two rounds of consensus to commit. First, all the transaction's writes must be replicated. Then the record must be flipped to COMMITTED, which is another Raft write. In a multi-region cluster where each consensus round costs tens of milliseconds, doubling it matters a great deal. CockroachDB's parallel commits protocol, introduced in version 19.2, removes the second round from the client's critical path. When the coordinator is ready to commit, it writes the record with status STAGING along with a list of all keys the transaction has written, its in-flight writes, in parallel with the final batch of writes. Once all of those writes and the staging record have replicated, the transaction is implicitly committed: anyone examining the record can verify, by checking that every listed write succeeded at the right timestamp, that the transaction must commit. The coordinator acknowledges the client at that moment, then asynchronously flips the record to COMMITTED, making the commit explicit. If the coordinator dies after acknowledging but before the flip, the next reader that finds the STAGING record runs a recovery procedure that checks the in-flight writes and finishes the commit, or aborts if some write is missing and can be prevented from ever succeeding. Write pipelining, introduced shortly before, attacks the latency of the writes themselves. Without it, each statement in a transaction waits for its writes to replicate through Raft before the next statement runs. With pipelining, the coordinator sends writes and proceeds immediately, tracking which writes are still in flight and verifying them before commit or before any later read of the same key. A transaction with ten writes no longer pays ten sequential consensus rounds. The SIGMOD paper reports the combined effect: with parallel commits and pipelining, a transaction's consensus cost approaches a single round, regardless of how many statements it contains. For multi-region deployments, this is the difference between transactions that are practical and transactions that are not. Conflicts and how they resolve Concurrency control in CockroachDB combines timestamp ordering with locks. Three kinds of conflict arise. Read meets write intent. A reader at timestamp t encounters a pending intent from another transaction at a timestamp below t. It cannot see the provisional value, and it cannot ignore it, because the other transaction might commit. The reader waits in a queue for that key's lock. Since version 20.1, CockroachDB has organized this waiting through a concurrency manager with a per-range lock table, so waiting transactions queue in order rather than repeatedly polling. If the lock holder appears dead because its heartbeats have stopped, the reader can abort it. If the intent's timestamp is above the reader's, the reader simply ignores it, because it lies in the reader's future. Write meets later read. A writer tries to write a key that another transaction has already read at a higher timestamp. If the write were allowed at its original timestamp, it would retroactively change what the reader saw, breaking serializability. CockroachDB tracks the highest timestamp at which each key has been read in a per-range timestamp cache, and when a write arrives below that high-water mark, it pushes the writer's timestamp above it. The writer's transaction now has a later timestamp than when it started, so its earlier reads may be stale. As with uncertainty restarts, it attempts a read refresh: verify that none of the keys it read have changed between the old and new timestamps. If none have, it commits at the new timestamp. If something has, it must restart. Write meets write. A writer encounters another transaction's intent on the same key. It waits for the lock, or, if the holder is abandoned, aborts it. If it finds a committed value at a higher timestamp, it must move its own timestamp past that value, with the same refresh-or-restart consequences. Deadlocks are possible when transactions wait on each other's locks. CockroachDB detects them through its transaction push mechanism, which tracks wait dependencies and aborts one transaction in any cycle. YugabyteDB's mechanics differ in detail. Its provisional writes are stored in a separate intents store within each tablet, and transaction status is held in dedicated transaction status tablets, which are Raft-replicated like any other. YugabyteDB offers two conflict-handling policies. Under fail-on-conflict, which its documentation describes as the default, conflicting transactions are assigned random priorities and the lower-priority one is aborted. Under wait-on-conflict, enabled with the tablet server flag enable_wait_queues=true, conflicting transactions wait in queues as they would in PostgreSQL, and a distributed deadlock detector runs in the background. Teams migrating PostgreSQL applications that rely on blocking behavior, such as SELECT ... FOR UPDATE used as a queue lock, generally find the wait-on-conflict policy closer to what they expect. Spanner, as Chapter 4 described, uses pessimistic locking with wound-wait for read-write transactions under its default serializable isolation, and a separate snapshot-based repeatable read level where reads take no locks. The retry contract Every distributed SQL system in this booklet, under its strongest isolation level, will sometimes abort a transaction that did nothing wrong except run concurrently with another. The application must be prepared to retry. This is the single largest behavioral difference from a single-node database for most teams, and the most common source of production bugs during migration. In CockroachDB, retryable errors carry the PostgreSQL SQLSTATE code 40001, serialization failure, with a message indicating the cause, such as RETRY_SERIALIZABLE, RETRY_WRITE_TOO_OLD, or ReadWithinUncertaintyIntervalError. CockroachDB retries automatically on the server when it can. Single-statement implicit transactions are retried transparently, and for explicit transactions, the server can retry as long as it has not yet sent any results back to the client, since it can then replay the work without the client noticing. Once results have been returned, only the client knows what logic depended on them, so the client must retry. A correct client-side retry loop is short, but it must retry the entire transaction, including any application logic that depended on what was read: def transfer(conn, src, dst, amount, max_retries=5): for attempt in range(max_retries): try: with conn.transaction(): bal = conn.execute( "SELECT balance FROM accounts WHERE id = %s", (src,)).fetchone()[0] if bal < amount: raise InsufficientFunds() conn.execute("UPDATE accounts SET balance = balance - %s " "WHERE id = %s", (amount, src)) conn.execute("UPDATE accounts SET balance = balance + %s " "WHERE id = %s", (amount, dst)) return except SerializationFailure: time.sleep(0.05 2 * attempt) raise TooManyRetries() Many ORMs and drivers now include retry helpers for these databases. The essential points are to retry the whole unit of work, to use bounded exponential backoff, and to make any side effects outside the database, such as sending an email, happen only after the commit succeeds. Isolation levels and their trade-offs Under serializable isolation, every anomaly is prevented, and the price is retries under contention. Weaker isolation levels reduce retries by permitting some anomalies. CockroachDB's read committed level, generally available since version 24.1, takes a new snapshot for each statement rather than for the whole transaction, and retries individual statements on the server rather than whole transactions. This dramatically reduces client-visible retries for applications written for PostgreSQL's default read committed level. It permits the anomalies that read committed always permits, including non-repeatable reads and lost updates if the application reads a value and writes back a computed result without locking. Applications using it should use SELECT ... FOR UPDATE where they read-then-write. YugabyteDB's YSQL defaults to snapshot isolation, which it maps to PostgreSQL's repeatable read, and also supports serializable and read committed. Snapshot isolation prevents most anomalies but permits write skew, described in Chapter 1. Spanner's repeatable read level is likewise snapshot-based and documented as permitting anomalies that serializable prevents. The on-call example from Chapter 1 shows concretely how the levels differ. A hospital requires at least one doctor on call per shift. Two doctors, Alice and Bob, are both on call, and each, at the same moment, opens a transaction that counts the doctors on call, sees two, concludes it is safe to go off call, and updates their own row. BEGIN; SELECT count(*) FROM on_call WHERE shift = 42 AND active; -- returns 2 UPDATE on_call SET active = false WHERE shift = 42 AND doctor = 'alice'; COMMIT; Under snapshot isolation, as in YugabyteDB's default or Spanner's repeatable read, both transactions read from snapshots in which both doctors are active. They write different rows, so there is no write-write conflict, and both commit. The shift is left with no one on call. Under read committed, the outcome is the same. Under serializable isolation, the database detects the conflict. In CockroachDB, one transaction's write to its row is ordered after the other's read of that row, which pushes its timestamp; when it attempts to refresh its reads at the new timestamp, it finds that the count has changed, and it must restart, at which point it sees only one doctor on call and the application declines the request. The fix under weaker isolation is explicit locking: selecting the shift's rows FOR UPDATE before counting forces the two transactions to serialize on those locks. The example generalizes. Any invariant that spans multiple rows, and is maintained by reading some rows and writing others, is vulnerable to write skew below serializable isolation. Such invariants are common in scheduling, inventory, financial limits, and quota enforcement. What parallel commits buy, in numbers The latency effect of parallel commits and pipelining is easiest to see in a multi-region cluster. Suppose a CockroachDB database with a region survival goal, so each write's quorum requires a round trip to the nearest other region, and suppose that round trip is 20 milliseconds. A transaction writes three rows in three different ranges, all with leaseholders in the local region, then commits. Without pipelining or parallel commits, each of the three writes waits for its own consensus round before the next statement proceeds: 60 milliseconds. Then the transaction record is written as committed: another 20 milliseconds. Total consensus latency is about 80 milliseconds, plus local processing. With pipelining, the three writes are sent without waiting, and their consensus rounds overlap. With parallel commits, the staging record is written in parallel with the final verification of those writes. The client is acknowledged once all have replicated, roughly one consensus round after the last write was issued. Total consensus latency falls to about 20 milliseconds. The four-to-one improvement in this example grows with the number of statements per transaction, which is why these optimizations make interactive multi-statement transactions practical in geo-distributed deployments. The choice depends on the workload. A financial ledger with invariants that span rows, such as "the sum of these accounts must not go negative," needs serializable, or explicit locking that achieves the same effect. A content management system where conflicting edits are rare and last-writer-wins is acceptable can comfortably use read committed. The mistake to avoid is choosing a weaker level for performance without auditing which invariants the application relies on. Designing transactions for a distributed engine A handful of practices consistently reduce latency and contention in all three systems. Keep transactions short. Every millisecond a transaction holds intents or locks is a millisecond during which conflicting transactions wait or abort. Do not hold a transaction open across user think time or slow external calls. Touch fewer ranges. A transaction whose writes all land in one range commits in a single consensus round with no distributed protocol. CockroachDB optimizes this case into a one-phase commit, and YugabyteDB treats single-shard transactions as a fast path that skips the transaction status tablet entirely. Schema designs that keep related rows together, as discussed in Chapter 2, make this case common. Batch writes. Sending many rows in one statement lets the database group them by range and replicate them in parallel, rather than paying a round trip per row. Avoid read-modify-write on hot rows. A counter incremented by every request will serialize every request behind one lock and generate retries. Sharded counters, append-only event tables aggregated later, or a dedicated system for counting are all better. Read at a fixed past timestamp when freshness permits. CockroachDB's AS OF SYSTEM TIME, YugabyteDB's follower reads, and Spanner's stale reads all let a read see a consistent snapshot slightly in the past. Such reads never conflict with ongoing writes, never trigger uncertainty restarts, and, as the next chapter shows, can often be served by a nearby replica instead of a distant leaseholder. For dashboards, reports, search indexing, and many user-facing reads, a few seconds of staleness is invisible, and the benefits are large. The common thread is that transactional guarantees are cheapest when transactions are local, short, and unconflicted. Where data lives relative to the transactions that touch it is the subject of the next two chapters, beginning with the topology of replicas across zones and regions. Chapter 7: The Topology of Survival The previous chapters described mechanisms: ranges, consensus, clocks, transactions. This chapter is about arithmetic. Given a set of failure domains and the network distances between them, where should replicas go, how many should there be, and where should the leaseholder sit? The answers determine both what a cluster can survive and how long every read and write takes. They are not matters of taste. They follow from the majority rule and the speed of light, and they can be worked out on paper before a single node is provisioned. The speed of light is a design input Light in optical fiber travels at roughly two-thirds of its speed in a vacuum, about 200,000 kilometers per second. That works out to roughly one millisecond of round-trip time for every hundred kilometers of fiber path, before any switching, routing, or queuing delay. Fiber paths are longer than straight lines, so real round trips are larger still. The consequence is a set of latency tiers that every distributed database designer should carry in their head. Machines in the same data center or availability zone communicate in well under a millisecond. Availability zones within one cloud region are typically separated by tens of kilometers and see round trips of one to a few milliseconds. Regions on the same continent see round trips of tens of milliseconds: US East to US West is commonly in the range of sixty to seventy milliseconds. Intercontinental round trips are larger again: US East to Western Europe commonly runs around eighty to a hundred milliseconds, and US East to East Asia commonly well over a hundred and fifty. Exact figures vary by provider, route, and time of day, and every cloud provider publishes inter-region latency measurements worth checking for the specific regions under consideration. These numbers are not overheads to optimize away. They are floors. No protocol can commit a write to replicas in Virginia and Frankfurt faster than the round trip between them. Quorums and the latency of a write Recall from Chapter 3 that a write commits when the leader has heard from a majority of replicas, counting itself. Followers respond in parallel, so the leader waits for the fastest followers needed to complete a majority. With three replicas, the leader needs one follower. With five, it needs two. The latency of a write is therefore approximately the round trip from the leader to the follower that completes the majority, the median follower rather than the farthest. This simple rule explains most topology decisions. Consider four placements of a range's replicas. Three replicas in three zones of one region. The leader needs one follower, a zone away: a write costs one to a few milliseconds of replication. The range survives the loss of any one zone. It does not survive the loss of the region. Three replicas in three regions. The leader needs one follower in the nearest other region: a write costs tens of milliseconds. The range survives the loss of any one region. But reads from the leaseholder are only fast near the leaseholder's region, and losing the leaseholder's region forces the lease to move, potentially to a distant region. Five replicas across three regions, two plus two plus one. Place two replicas in the home region, two in a nearby region, and one in a third. The leader, in the home region, needs two followers: the other home-region replica, a zone away, and one replica in the nearest remote region. A write costs one round trip to the nearest remote region. The range survives the loss of any region, including the home region, because the remaining three replicas still form a majority. This is the layout CockroachDB uses for databases with a region survival goal, as described below. Five replicas in five regions. Every write needs two remote followers, so it waits for the second-nearest remote region. This survives two region failures, at a higher latency cost, and is appropriate only for workloads where that level of survivability justifies it. The principle that falls out is that surviving the loss of a failure domain requires a write quorum that spans failure domains, and therefore costs the round trip across them. There is no configuration that survives region failure with zone-level write latency. Any system claiming to offer both is either replicating asynchronously, and will lose recent writes on region failure, or redefining what survival means. Survival goals as a first-class setting CockroachDB turns this arithmetic into a declarative setting. A multi-region database has a primary region, a set of additional regions, and a survival goal. The default goal, SURVIVE ZONE FAILURE, places voting replicas for each table's data in its home region, across zones, so that writes commit at zone latency. To make reads in other regions possible, CockroachDB also places non-voting replicas in the other database regions. Non-voting replicas, introduced in version 21.1, receive the Raft log and can serve follower reads, but do not participate in quorum, so they add no write latency. The cost of this goal is that losing the home region makes the data in that region unavailable until the region returns. The stronger goal, SURVIVE REGION FAILURE, requires at least three database regions and raises the replication factor to five, placed two, two, and one across three regions, with the leaseholder preferring the home region. CockroachDB's documentation states the trade-off plainly: write latency increases by at least the round-trip time to the nearest region, and read performance is unaffected, since reads are still served by the leaseholder in the home region. ALTER DATABASE bank SET PRIMARY REGION "us-east1"; ALTER DATABASE bank ADD REGION "us-central1"; ALTER DATABASE bank ADD REGION "us-west1"; ALTER DATABASE bank SURVIVE REGION FAILURE; These four statements replace what would otherwise be a page of zone configuration specifying replica counts, constraints, voter placement, and lease preferences per table. They also make the decision legible: anyone reading the schema can see what the database is designed to survive. Table 3 sets out the general relationship between placement, survivability, and latency for a single range. The latency entries are expressed as round trips rather than milliseconds because they depend on the specific regions chosen. Table 3. Replica placement, what it survives, and what writes cost. Placement Survives Write latency (approx.) Typical use 3 replicas, 3 zones, 1 region One zone Zone round trip Single-region apps 3 voters in home region plus non-voters elsewhere One zone; remote stale reads Zone round trip CockroachDB zone survival 5 replicas, 2+2+1 over 3 regions One region Nearest-region round trip CockroachDB region survival 5 replicas, 5 regions Two regions Second-nearest-region round trip Extreme survivability Spanner's replica types Spanner expresses the same trade-offs through instance configurations and replica types. Google's documentation describes three kinds of replica. Read-write replicas hold full copies of the data, vote in write quorums, and can become leaders. Read-only replicas hold full copies and serve reads, but do not vote, so they add read capacity in distant locations without slowing writes. Witness replicas vote in write quorums and leader elections but do not maintain a full copy of the data and cannot serve reads. Witnesses are a clever answer to the cost of the two-plus-two-plus-one layout: the third region contributes a vote for survivability without the storage and compute cost of a full replica. A regional Spanner configuration places three read-write replicas in three zones of one region. Multi-region configurations, available in the Enterprise Plus edition, typically place two read-write replicas in each of two read-write regions, one of which is the default leader region, and a witness in a third region, optionally with read-only replicas elsewhere. Google's nam3 configuration, for example, uses Northern Virginia and South Carolina as its read-write regions. Spanner's dual-region configurations, added more recently for countries where only two cloud regions exist, place six replicas, three per region, with two read-write replicas and a witness in each. The default leader region matters for the same reason the leaseholder region matters in CockroachDB: it is where strong reads and write coordination are cheapest. Spanner lets you choose it, and Google's guidance is to place it near the bulk of write traffic. Reading near the user The leaseholder or leader serves strongly consistent reads locally. Clients in other regions pay a round trip to reach it. For globally distributed users, that round trip is often the dominant latency, and all three systems provide ways to serve reads from nearby replicas instead. The underlying mechanism is the same everywhere: a replica can serve a consistent read at timestamp t if it knows it has every write with a timestamp at or below t. In Spanner, as Chapter 4 explained, that knowledge is the replica's safe time. In CockroachDB, it is the closed timestamp: each leaseholder continuously promises that it will accept no more writes below a timestamp that trails the present by a target interval, and propagates that promise to followers through Raft. A follower that has applied everything up to the closed timestamp can serve reads at or below it. CockroachDB's follower_read_timestamp() function returns a timestamp old enough to be served by followers under default settings, a few seconds in the past. SELECT * FROM products AS OF SYSTEM TIME follower_read_timestamp() WHERE category = 'books'; Bounded staleness reads take this further: CockroachDB's with_max_staleness and with_min_timestamp functions let a single-row read accept any timestamp within a staleness bound, so the nearest replica answers with the freshest data it can guarantee, and the read remains available even when the leaseholder is unreachable. YugabyteDB offers follower reads through session settings that allow reads from followers with a configurable staleness, and read replicas, asynchronous observer copies in other regions that serve timeline-consistent reads without participating in the primary cluster's quorum. Spanner offers stale reads, with exact or bounded staleness. Follower reads cost freshness, typically a few seconds, and nothing else. They do not slow writes, do not conflict with writers, and do not need the leaseholder to be reachable. For a remarkable share of real read traffic, including product catalogs, user profiles viewed by others, configuration, and reporting, a few seconds of staleness is entirely acceptable, and the latency saved is the full intercontinental round trip. Asynchronous replication between clusters Everything so far describes synchronous replication within one cluster, where survivability comes from quorums. Some deployments instead run separate clusters in separate regions and replicate between them asynchronously. YugabyteDB's xCluster replication streams changes between independent universes, as Yugabyte calls clusters, in one or both directions. In a two-region active-active deployment, each region has its own cluster with local-latency writes, and changes flow asynchronously to the other. CockroachDB offers physical cluster replication, which maintains a standby cluster for failover, and logical data replication, introduced in version 24.3, which replicates tables between active clusters, optionally in both directions. These arrangements are valuable when only two regions are available, which makes quorum-based region survival impossible, or when write latency in each region must stay local. But they reintroduce the trade-offs of the pre-distributed-SQL world. Replication lag means a region failure loses the most recent writes that had not yet crossed. Bidirectional replication means the same row can be written in both regions concurrently, so conflicts must be resolved, typically by last-writer-wins on timestamps, which silently discards one write. Uniqueness constraints and multi-row invariants cannot be enforced across clusters. These are acceptable for some data and unacceptable for others; the important point is to know which guarantees apply where, rather than assuming that the word "active-active" implies the guarantees of a single synchronous cluster. Walking through a region failure Survival goals are promises about what happens during a failure, and it is worth tracing one in detail. Consider a CockroachDB database with three regions, call them East, Central, and West, a region survival goal, and East as the primary region. A range of data homed in East has five replicas: two in East, two in Central, and one in West. Its leaseholder, and in current versions its Raft leader, is in East. During normal operation, a write from a client in East reaches the leaseholder locally, which replicates to the other East replica and to the Central replicas. The quorum of three, two in East and the first Central replica to respond, completes after one East-to-Central round trip. A strong read from East is served by the leaseholder with no replication at all. A strong read from West crosses to East and back. Now East goes dark, all at once, as when a regional network failure isolates it. The two East replicas are gone. The three survivors, two in Central and one in West, still form a majority of five. After the election timeout, a few seconds under CockroachDB's documented settings, one of the survivors campaigns, wins, and becomes leader. Under leader leases, it acquires the lease once it is established as a fortified leader. Clients that were connected to East nodes lose their connections and must reconnect, through a load balancer or driver, to nodes in Central or West. From that point, writes commit with a quorum of three among the survivors, and with only three survivors, all of them must acknowledge each write: the leader in Central, the other Central replica, and the replica in West. Write latency rises, because the quorum now spans Central and West, and the range has no remaining tolerance for another failure until the allocator can place replacement replicas or East returns. Strong reads are served from the new leaseholder in Central. No acknowledged write is lost, because every acknowledged write was on at least three of five replicas, and at most two were in East. Availability is interrupted for the few seconds of detection and election, which is the practical meaning of "survives region failure": not zero disruption, but recovery within seconds, without operator action and without data loss. When East returns, its replicas rejoin, catch up through the Raft log or a snapshot, and the allocator transfers the lease back to East according to the configured preferences. Throughout, the database remains a single cluster with a single consistent state; there is nothing to reconcile. The same walkthrough under a zone survival goal shows the difference. The range's three voters are all in East. When East fails, the range loses all its voters. The non-voting replicas in Central and West hold copies of the data and can serve follower reads at timestamps up to their last closed timestamp, but they cannot form a quorum or accept writes. The data homed in East is unavailable for writes until East returns, or until an operator performs a recovery procedure that accepts the risk of losing the most recent writes. That is precisely the trade the zone survival goal makes: lower write latency in exchange for exposure to region failure. Partial failures and gray failures Clean failures, in which a region disappears entirely, are the easy case. The hard cases are partial. A region may remain reachable from some regions but not others. A node may stay up but become very slow because of a failing disk. Network packet loss may rise without links going down entirely. Consensus handles some of these well. A slow follower simply does not contribute to quorum, and the leader proceeds with faster ones. A slow leader is more damaging, because every write and strong read waits on it, and a leader that is slow but still heartbeating may not trigger an election. CockroachDB's leader lease redesign, described in Chapter 3, was motivated in part by such cases, and its documentation cites much faster recovery from partial network partitions under the new mechanism than under the previous one. Operators should still monitor for leaseholders on degraded nodes and be prepared to drain those nodes, which moves their leases elsewhere gracefully. Topology decisions also influence how gray failures spread. A cluster in which every range's quorum depends on one particular inter-region link is vulnerable to that link's degradation in a way that a cluster with more diverse quorums is not. Where possible, choose regions whose interconnections are independent, and test what happens when one link degrades rather than fails. Choosing a topology Topology decisions should start from a written statement of what failures the system must survive and what latency each class of operation must achieve, then follow the arithmetic. If the requirement is surviving zone failures with the lowest possible latency, a single region with three zones is correct and simple. Most applications that believe they need multi-region do not, if they are honest about their availability targets and their users' locations. If the requirement is surviving region failure without losing acknowledged writes, the system needs at least three regions, a quorum spanning them, and acceptance of at least a nearest-region round trip on every write. The design work is then to choose regions close enough together that this round trip is tolerable, often three regions on one continent, and to place leaseholders near the writers. If users are spread globally and reads dominate, follower reads, read-only replicas, and non-voting replicas bring reads close to users without affecting writes. If writes come from everywhere and each must be local, the data itself must be partitioned so that each row's quorum and leaseholder sit near the users who write it. That is a schema problem rather than a topology problem, and it is the subject of the next chapter. Chapter 8: Designing Schema for Multi-Region Active-Active "Active-active" is one of the most abused phrases in infrastructure. In vendor marketing it often means little more than "both regions are running." For the purposes of this chapter, it has a precise meaning: users in every region can read and write with local latency, the system as a whole remains a single logical database with transactional guarantees, and the loss of any one region loses no acknowledged writes and leaves the surviving regions fully operational. Chapter 7 established that those goals are in tension. Surviving a region requires a write quorum spanning regions, which costs a cross-region round trip. Local-latency writes everywhere require each write's quorum to be nearby. The resolution is not a cleverer protocol. It is to recognize that different data has different access patterns, and to give each piece of data a placement that matches who uses it. A user's profile is written almost entirely by that user, from one region. A product catalog is read by everyone and written rarely. An order is written once in one region and read occasionally elsewhere. A distributed SQL schema designed for active-active operation encodes these patterns directly, so that the common case is local and the uncommon case is correct. Three kinds of data Almost every table in a multi-region application falls into one of three categories, and the three systems in this booklet all provide a placement strategy for each. Data with a single home region. Some tables are accessed mostly from one place: an internal back-office table used by a team in one office, a table of US tax records, a regional inventory system. These should simply live, with their leaseholders and voting replicas, in that region. Data partitioned by home region at the row level. Most user-generated data has a natural home: the region of the user, tenant, account, or store it belongs to. A table of user accounts, orders, or messages should store each row near the user who owns it, so that the overwhelming majority of reads and writes are local, while the table remains one logical table that can be queried, joined, and constrained as a whole. Data read everywhere and written rarely. Reference data, such as currency codes, product catalogs, feature flags, permission definitions, and configuration, is read by every request in every region and changed occasionally. It should be readable locally everywhere, even at the cost of slower writes. CockroachDB names these three categories directly as table localities, which makes its vocabulary the clearest way to discuss them. Table 4 summarizes them, drawing on CockroachDB's documentation; the same patterns appear under other names in YugabyteDB and Spanner. Table 4. CockroachDB table localities and their trade-offs. Locality Data placed Local reads Local writes Fits REGIONAL BY TABLE Whole table in one home region In home region In home region Single-region data REGIONAL BY ROW Each row in its own home region In row's region In row's region User, tenant, account data GLOBAL Readable replicas in every region Everywhere No; slower everywhere Read-mostly reference data Regional-by-row tables The regional-by-row pattern is the heart of active-active schema design, and it is worth examining in detail in each system. In CockroachDB, declaring a table REGIONAL BY ROW adds a hidden column named crdb_region of the database's region enum type. Under the hood, the table and each of its indexes are partitioned by this column, and each partition's replicas and leaseholder are placed in the corresponding region according to the database's survival goal. By default, a new row's home region is the region of the gateway node that received the insert, so users writing through a local endpoint automatically create local rows. CREATE TABLE users ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), email STRING NOT NULL UNIQUE, name STRING, created_at TIMESTAMPTZ DEFAULT now() ) LOCALITY REGIONAL BY ROW; Queries that filter on crdb_region, or that the optimizer can confine to one region, read only that region's partition. For queries that do not specify the region, such as a lookup by primary key, CockroachDB uses locality optimized search: it searches the local region's partition first and only fans out to remote regions if the row is not found locally. Because most users access their own data from their own region, most lookups hit locally on the first try. Two refinements matter in practice. First, users move. A user who signs up in Europe and relocates to the United States would have every request cross the Atlantic. CockroachDB supports auto-rehoming, which, when enabled, updates a row's home region to the region of the gateway that most recently updated it. Second, applications often know the right region better than the gateway does. A computed column can derive crdb_region from an application attribute, such as a country code, making placement a deterministic function of the data rather than of which endpoint happened to receive the insert. CREATE TABLE accounts ( id UUID NOT NULL DEFAULT gen_random_uuid(), country STRING NOT NULL, balance DECIMAL NOT NULL, crdb_region crdb_internal_region NOT VISIBLE NOT NULL AS ( CASE WHEN country IN ('DE','FR','NL') THEN 'europe-west1' WHEN country IN ('JP','KR') THEN 'asia-northeast1' ELSE 'us-east1' END) STORED, PRIMARY KEY (id) ) LOCALITY REGIONAL BY ROW AS crdb_region; The uniqueness problem Partitioning a table by region creates a subtle problem with unique constraints. In the users table above, email is unique across the whole table. When a user in Tokyo inserts a row with a given email, the database must guarantee that no row anywhere has the same email, including rows homed in Europe and the United States. That check is a cross-region read, and it adds a cross-region round trip to every insert or update of a unique column. There are three ways to manage this cost. The first is to accept it for columns where global uniqueness is genuinely required and writes are infrequent, as with email addresses at signup. The second is to scope uniqueness to the region, by including the region column in the constraint, when the business rule permits it, as with a username that need only be unique within a country. The third is to rely on keys whose uniqueness is guaranteed by construction. CockroachDB recognizes that a column defaulting to gen_random_uuid() is unique with overwhelming probability and, by default, skips the cross-region uniqueness check for it, which is one more reason to prefer random UUIDs as primary keys in regional-by-row tables. Foreign keys raise similar questions. A foreign key from a regional-by-row orders table to a regional-by-row users table requires checking the referenced user on every order insert. If orders and users share a home region, as they will if both are homed by the same user attribute, the check is local. If they can differ, it is a cross-region read. Designing the home region of related tables to align is the multi-region analogue of keeping related rows together within one range, described in Chapter 2. Global tables Reference data presents the opposite challenge: every region needs to read it with local latency, but it rarely changes. Follower reads, described in Chapter 7, can serve slightly stale reads locally. But many uses of reference data need current values, often inside read-write transactions. A transaction that checks a user's permission against a permissions table and then writes an order needs the permission check to be strongly consistent, and it should not cross an ocean to perform it. CockroachDB's global tables solve this with what its designers call a non-blocking transaction protocol. Writes to a global table are assigned timestamps in the future, far enough ahead that by the time the timestamp arrives, the write has replicated to every region and every replica's closed timestamp has passed it. A reader in any region at the present time therefore always finds that its local replica is up to date and can serve a strongly consistent read locally without contacting the leaseholder. The writer pays for this: it must wait, in a commit-wait step reminiscent of Spanner's, until its future timestamp has passed, which makes writes to global tables slow, typically hundreds of milliseconds, and dependent on the cluster's maximum clock offset. CockroachDB's documentation is explicit that global tables are optimized for low-latency reads from every region at the cost of higher write latency from any region. CREATE TABLE currencies ( code STRING PRIMARY KEY, name STRING NOT NULL, digits INT NOT NULL ) LOCALITY GLOBAL; The design rule is simple. Use global tables for data that is read on the hot path in many regions and written rarely, perhaps a handful of times per minute at most, and never for data written on user request paths. Geo-partitioning in YugabyteDB YugabyteDB achieves the regional-by-row pattern with PostgreSQL's own features. It uses declarative list partitioning to divide a table by a region column, and tablespaces with replica placement policies to pin each partition to a region. CREATE TABLESPACE eu_ts WITH (replica_placement = '{"num_replicas": 3, "placement_blocks": [ {"cloud":"aws","region":"eu-west-1","zone":"eu-west-1a","min_num_replicas":1}, {"cloud":"aws","region":"eu-west-1","zone":"eu-west-1b","min_num_replicas":1}, {"cloud":"aws","region":"eu-west-1","zone":"eu-west-1c","min_num_replicas":1}]}'); CREATE TABLE orders ( id UUID DEFAULT gen_random_uuid(), geo TEXT NOT NULL, total NUMERIC, PRIMARY KEY (id, geo) ) PARTITION BY LIST (geo); CREATE TABLE orders_eu PARTITION OF orders FOR VALUES IN ('EU') TABLESPACE eu_ts; Each partition is a separate set of tablets with replicas confined to the tablespace's placement, so EU orders are stored and replicated only in the EU region, and their Raft leaders are local. The parent table remains a single logical table for queries that span regions. Placement blocks can also express leader preferences, so a tablespace can spread replicas across regions for survivability while preferring leaders in one region for latency, the two-plus-two-plus-one layout of Chapter 7 expressed in JSON. The approach is more verbose than CockroachDB's declarative localities and requires the application to supply the partition column. Its advantages are that it uses standard PostgreSQL constructs that many teams already understand, and that it gives direct, explicit control over every partition's placement. For reference data, YugabyteDB relies on follower reads and read replicas rather than an equivalent of global tables, and the documentation's design patterns describe placing leaders of reference tables in a central region with followers everywhere. Spanner: interleaving and geo-partitioning Spanner's tools for multi-region locality have historically been coarser. A multi-region instance configuration places all data with the same leader region, so writes from far regions cross to it. Spanner's long-standing locality tool is interleaving, described in Chapter 2, which keeps related rows together within one split so that transactions across a parent and its children stay within one Paxos group. That improves efficiency within the leader region but does not move data closer to distant users. Spanner's newer geo-partitioning feature addresses row-level placement. According to Google's documentation, it lets an instance contain additional instance partitions, each with its own instance configuration and nodes, and lets a database define placements mapped to those partitions. Tables declare a placement key column, and each row is stored in the partition named by its placement key value. The feature requires the Enterprise Plus edition, and as of late 2026 Google's documentation still labels it a Preview, with notable limitations, including a cap on instance partitions per instance and incompatibility with some features such as change streams and incremental backups. Teams designing on Spanner today should treat row-level geo-placement as an emerging capability and verify its status for their configuration. Data residency Multi-region placement controls are also compliance controls. Regulations such as the European Union's GDPR, and data localization laws in other jurisdictions, may require that certain personal data be stored only within specific borders. Regional-by-row and geo-partitioned tables can satisfy such requirements, but only if every copy of the data is constrained. A regional-by-row table under CockroachDB's default zone survival goal places voting replicas in the home region but non-voting replicas in other regions for follower reads, which would place copies of European rows outside Europe. CockroachDB provides a PLACEMENT RESTRICTED option that confines all replicas to the home region, and super regions, which group regions so that data can survive the loss of one region while never leaving the group, for example keeping EU data within a set of EU regions. YugabyteDB's tablespace placements are explicit by construction. In every system, backups, change data capture streams, logs, and query results sent to application servers are also copies of the data, and a residency design that constrains replicas but not backups is not a residency design. Anti-patterns The mechanisms above are powerful enough that misusing them is easy. Several mistakes recur often enough to name. Global tables for frequently written data. A table of user sessions, shopping carts, or rate-limit counters is read everywhere, which makes global locality look attractive. But each of these is written on request paths, and every write to a global table pays the commit-wait cost of the non-blocking protocol. A login flow that writes a session row to a global table will be slow in every region. Data that is written on user request paths should be regional by row, homed with the user. Regional-by-table for user data. Placing a users table in one region, perhaps because it was the simplest way to get started, makes every user outside that region pay a cross-region round trip on every request. This is the most common reason a multi-region deployment turns out slower than the single-region system it replaced. Misaligned home regions. If orders are homed by the region where the order was placed, and users are homed by the region of their billing address, then a traveling user's orders and profile live in different regions, and every operation that touches both is a cross-region transaction. Home region should be derived from a single, stable attribute shared across related tables wherever possible. Unbounded cross-region scans. A query without a region filter on a regional-by-row table, such as a report over all orders in the past day, fans out to every region. That is correct and sometimes necessary, but it should be deliberate, run against follower replicas at a past timestamp where possible, and kept off latency-sensitive paths. Ignoring the leaseholder of reference data in single-region designs. Even in a cluster that is not configured for multi-region, a small, hot reference table can end up with its leaseholder far from most clients or concentrated on one node. Global locality, or at least explicit lease preferences, should be considered for any table read on nearly every request. A worked design Consider a payments application with customers in North America, Europe, and Asia, running on CockroachDB in three regions with a region survival goal. The customers, accounts, and transactions tables are regional-by-row, with the home region computed from the customer's country, so a European customer's accounts and transactions live in Europe. Primary keys are random UUIDs. Account and transaction rows include the customer's region, so that foreign key checks between them are local. Email uniqueness on customers is enforced globally and accepted as a cross-region cost at signup. The currencies, fee_schedules, and country_rules tables are global, read on every payment in every region, and updated by an operations tool a few times a day. A reconciliation_jobs table used by a back-office team in one location is regional by table in that region. A typical payment, a European customer paying from their account, reads global reference tables locally, reads and writes regional rows homed in Europe, and commits with a quorum that includes a replica in the nearest other region, as the survival goal requires. The latency is one round trip to that region. A cross-border transfer between a European and an Asian customer touches rows homed in two regions, and pays for it in a distributed commit spanning both. That is correct: the transfer genuinely involves data in two places, and the database charges for exactly that, while the common case stays fast. This is what designing for active-active survivability means in practice. The database supplies the mechanisms and enforces correctness. The schema decides where each row lives, and therefore what each operation costs. Chapter 9: Operating, Migrating, and Choosing The earlier chapters describe how distributed SQL databases work and how to design for them. This chapter is about living with one: getting data in, proving it behaves as claimed, watching it in production, upgrading it, paying for it, and deciding in the first place whether to adopt one at all and which. These are the questions that determine whether an adoption succeeds, and they are often given less attention than consensus algorithms. Migrating from a single-node database Most distributed SQL deployments begin as migrations from PostgreSQL, MySQL, Oracle, or SQL Server. Each vendor provides tooling. Cockroach Labs offers MOLT, short for Migrate Off Legacy Technology, a set of tools that includes schema conversion, bulk data loading with MOLT Fetch, and row-level verification with MOLT Verify. Yugabyte offers YugabyteDB Voyager, which assesses a source database, converts schema, migrates data, and supports live migration with change capture from the source. Google provides the Spanner Migration Tool, formerly called HarbourBridge, for schema and data migration from MySQL and PostgreSQL. Tooling moves bytes. The harder work is in four areas the tools cannot fully automate. Compatibility gaps. Wire compatibility with PostgreSQL does not mean feature parity. YugabyteDB, because it reuses PostgreSQL's query layer, supports most PostgreSQL features, including stored procedures, triggers, and many extensions, though not all extensions work with a distributed storage layer. CockroachDB implements its own SQL engine and has steadily added PostgreSQL features, including PL/pgSQL functions, stored procedures, and triggers in recent versions, but some behaviors, system catalogs, and extensions differ or are absent. Spanner's PostgreSQL dialect supports a defined subset of PostgreSQL syntax and functions. Every migration should begin with a systematic assessment of the application's SQL, not a sample. Primary keys. Schemas from single-node databases are full of auto-incrementing integer keys. Chapter 2 explained why these concentrate writes. Changing primary keys is invasive, because foreign keys, application code, and external systems refer to them. Teams often keep existing integer IDs as a secondary unique column for compatibility and introduce UUID primary keys, or use hash-sharded indexes where the integer must remain the key. Transaction retries. Applications written for single-node databases rarely retry serialization failures, because under read committed on PostgreSQL they rarely occur. Under serializable isolation on a distributed database, they will. Before migration, every transactional code path needs a retry strategy, as described in Chapter 6, or the team should deliberately choose read committed isolation where its semantics are acceptable. Query patterns. Queries that were cheap on a single node can become expensive when distributed. A query that scans a large table without a selective index now scans many ranges across many machines. A join between two large tables keyed differently may shuffle data across the network. Chatty application patterns, issuing hundreds of small queries per request, multiply network round trips. The remedy is the same discipline that good single-node performance requires, applied more strictly: selective indexes, batched statements, and fewer round trips. A safe migration typically runs the new database in parallel, replicating changes from the source, verifying data continuously, and shifting read traffic, then write traffic, in stages with a rollback path. The cutover itself should be the least eventful step. Testing what you were promised A distributed database's guarantees are only as good as its implementation, and implementations have bugs. The most rigorous public evidence on this point comes from Jepsen, Kyle Kingsbury's independent testing project, which subjects databases to network partitions, clock skew, and process crashes while checking whether the resulting histories are consistent with the claimed isolation level. Jepsen's 2017 analysis of CockroachDB's beta releases found two serializability violations: one in which the timestamp cache failed to distinguish transactions with identical timestamps, and one in which retries after lost network responses could apply a transaction twice. Cockroach Labs fixed both before the 1.0 release, the second by introducing an explicit ambiguous result error rather than transparently retrying operations whose outcome was unknown. Jepsen's 2019 analysis of YugabyteDB 1.3.1 found a serializability violation, a G2-item anti-dependency cycle, that could occur when master nodes crashed or paused, fixed in a subsequent patch release, along with problems with non-transactional schema changes, including columns with default values sometimes initialized to null. Two lessons follow. First, these systems were improved substantially by independent adversarial testing, and their vendors deserve credit for engaging with it publicly. Second, the classes of bugs Jepsen found are instructive: they cluster around edge cases such as identical timestamps, ambiguous network outcomes, schema changes concurrent with transactions, and control-plane failures. These are exactly the conditions an application team should exercise in its own pre-production testing. Kill nodes during load tests. Partition the network between regions. Skew clocks within the configured maximum offset. Run schema migrations under write load. Verify that the application's retry logic and the database's guarantees together produce correct results, not merely that the system stays up. Observability Distributed SQL databases expose a great deal of telemetry, and most of it is noise until something goes wrong. A smaller set of signals deserves continuous attention. Latency by statement fingerprint and by region. Aggregate latency hides the pattern that matters most in a distributed database: specific queries that cross regions or scan too many ranges. CockroachDB's SQL activity pages, YugabyteDB's pg_stat_statements support, and Spanner's query statistics tables all group statements by normalized fingerprint. Retry and restart rates. Rising transaction retries indicate growing contention, often on a single hot row. Rising uncertainty restarts indicate readers of freshly written data or clock offset pressure. Hot ranges and tablets. One range or tablet consuming far more CPU or receiving far more requests than its peers is the signature of a hotspot, discussed in Chapter 2. Clock offset. As Chapter 5 explained, a node whose clock approaches the maximum offset will be shut down. Alert well before that. Replication health. Under-replicated and unavailable ranges indicate that some data has fewer replicas than its configuration requires, or has lost quorum. Under-replication is a warning; unavailability is an outage for the affected data. Leaseholder and leader placement. In multi-region clusters, leases drifting away from their preferred regions silently add cross-region round trips to every strong read. Storage engine health. LSM-tree storage engines such as Pebble and RocksDB accumulate files that must be compacted. When compaction falls behind, read amplification rises and performance degrades nonlinearly. CockroachDB exposes this as LSM health and admission control metrics; its admission control system throttles work to protect nodes whose storage is falling behind. Backups and recovery from mistakes Replication protects against hardware and infrastructure failure. It does nothing to protect against an engineer who runs a DELETE without a WHERE clause, an application bug that corrupts rows, or a schema migration that drops the wrong column. Those changes are replicated faithfully to every replica within milliseconds. Distributed SQL does not reduce the need for backups; if anything, the confidence that replication inspires makes teams more likely to neglect them. All three systems provide backup mechanisms designed for their architecture. CockroachDB's BACKUP statement writes consistent, distributed backups of a cluster, database, or table directly to cloud object storage, supports incremental backups, and can write locality-aware backups so that each region's data is backed up to storage in that region, which matters for data residency. Its RESTORE statement recovers from them, and revision history in backups allows restoring to a point in time. YugabyteDB provides distributed snapshots and point-in-time recovery, along with backup to external storage. Spanner offers managed backups and point-in-time recovery within a configurable version retention period of up to seven days, and its documentation lists scheduled and incremental backups among edition features. Multi-version storage provides a further, often overlooked tool. Because these databases keep old versions of rows for a retention window, it is often possible to read data as it was before a mistake. In CockroachDB, AS OF SYSTEM TIME with a timestamp shortly before an erroneous change can retrieve the previous values, provided they are within the garbage collection window, which defaults to a few hours in recent versions. Spanner's stale reads within its version retention period serve the same purpose. This is not a substitute for backups, but it can turn a disaster into a quick repair. A backup that has never been restored is a hope rather than a plan. Restoring to a separate cluster on a regular schedule, and timing the restore, is the only way to know the recovery time objective is achievable. Capacity and headroom Capacity planning for a distributed database must account for failure, not just steady-state load. A three-region cluster designed to survive a region failure must be able to carry the full workload on the surviving two regions, including the extra load of re-replicating data that lost replicas. A cluster sized so that each region runs at seventy percent of capacity in normal operation will be overloaded when a region fails and its traffic redistributes. The arithmetic is unforgiving: surviving the loss of one of three regions requires that the surviving two thirds of capacity carry everything. Disk headroom matters similarly. When a node fails, its replicas are rebuilt on other nodes, which must have room. LSM storage engines also need free space for compaction. Vendors publish specific guidance on maximum disk utilization, and their production checklists are worth following closely. Finally, balance matters. Placement constraints, leaseholder preferences, and regional-by-row homing can concentrate load in the region where most users are. A cluster with identical node counts in each region may run hot in one and idle in the others. Provision by measured load per region, not by symmetry. Upgrades and release cadence All three systems support rolling upgrades without downtime, one node at a time, with the cluster operating in a mixed-version state until finalization. The operational practice is the same as with any distributed system: upgrade one node, watch it, continue, and do not finalize until confident, because finalization typically enables new on-disk formats and makes downgrade impossible. The release models differ. CockroachDB moved to a quarterly release cadence that alternates between regular releases, with longer support windows, and innovation releases, with shorter windows and the latest features that can be skipped; as of late 2026, its documentation lists v26.2 as the current regular release and v26.3 as an innovation release, with v26.4 expected in the fourth quarter. YugabyteDB distinguishes long-term support series from standard-term support series; its 2025.2 series is the current LTS and 2026.1 the current STS, and the 2025 releases moved the query layer from PostgreSQL 11 to PostgreSQL 15, with in-place upgrade support. Spanner, as a managed service, is upgraded by Google continuously and invisibly. Licensing and cost Licensing has shifted repeatedly in this category, and it now differs sharply among the three. CockroachDB was released under the Apache 2.0 license, moved most of its code to the Business Source License in 2019, and in 2024 made a larger change. Cockroach Labs announced in August 2024 that it would retire the free CockroachDB Core edition and offer a single CockroachDB Enterprise edition under its own source-available CockroachDB Software License, effective November 18, 2024, beginning with version 24.3. Enterprise is free under an Enterprise Free tier for individuals, students, and businesses with less than ten million dollars in annual revenue, which also gains features such as backups and disaster recovery that were previously paid. The free tier requires sending telemetry to Cockroach Labs, except for short-lived clusters, and larger businesses receive a thirty-day trial before needing a paid license. The managed CockroachDB Cloud service continued separately. The code remains source available but is not open source under any OSI-approved license. YugabyteDB has been fully open source under Apache 2.0 since 2019, when Yugabyte relicensed previously proprietary enterprise features. The company's commercial offerings are its managed service and support and management tooling. Spanner is a proprietary managed service priced by compute capacity, in nodes or smaller processing units, and storage, with edition determining features and SLA. The Spanner Omni preview announced in April 2026 opens a path to running Spanner outside Google Cloud, with a free developer edition for non-production use and a commercial edition in early access. Cost comparisons are workload-specific and should be measured rather than estimated. Two general points hold. First, replication multiplies storage and compute: a five-replica, region-survivable deployment stores five copies of every byte and runs consensus traffic across regions, and cross-region network transfer is billed by every major cloud. Second, the operational cost of self-hosting a distributed database, including on-call expertise, upgrades, capacity planning, and incident response, is substantial and often underestimated relative to a managed service's price. Choosing, or not choosing The first question is whether a distributed SQL database is needed at all. A single PostgreSQL instance on modern hardware, with synchronous replication to a standby in another zone and asynchronous replicas elsewhere, handles very large workloads with mature tooling and a vast ecosystem. If the requirements are a few terabytes, write rates a single machine can absorb, and zone-level availability with a short failover, a distributed database adds latency, cost, and operational novelty without a corresponding benefit. Distributed SQL earns its place when at least one of these holds: write throughput or data volume exceeds what one machine can comfortably handle with headroom; the system must survive the loss of a zone or region without operator intervention and without losing acknowledged writes; users in several regions need local-latency access to a single logical database; or data residency requires row-level placement within one database. If none applies, the honest recommendation is to stay with a single-node database and revisit later. When one does apply, the choice among the three systems follows a few clear lines. Spanner suits teams committed to Google Cloud that want the strongest consistency guarantees, external consistency for all transactions, with no database operations work. Its constraints are platform lock-in, at least until Spanner Omni matures, a PostgreSQL interface that is a dialect rather than full compatibility, and row-level geo-partitioning that is still in preview. CockroachDB suits teams that need to run across clouds or on premises, that value its declarative multi-region SQL, which remains the most complete expression of the patterns in Chapter 8, and that are comfortable with its license terms. Its constraints are a SQL engine that is PostgreSQL-compatible but not PostgreSQL, and, for larger companies self-hosting, a commercial license. YugabyteDB suits teams migrating PostgreSQL applications that rely on deep PostgreSQL features, such as stored procedures, triggers, and extensions, and teams that require an Apache-licensed core. Its constraints are a multi-region configuration model that is more manual, built from tablespaces and partitions, and a smaller ecosystem of multi-region abstractions. The category also has newer entrants. Amazon Aurora DSQL, generally available since May 2025, is a serverless, PostgreSQL-compatible distributed database offering strongly consistent multi-region active-active operation, with AWS citing 99.99 percent single-region and 99.999 percent multi-region availability. It uses optimistic concurrency and supports a narrower PostgreSQL feature set than the three systems examined here. TiDB, built around a central timestamp oracle and MySQL compatibility, is another significant option. The principles in this booklet apply to all of them, because they all pay the same physical costs: quorum round trips and clock uncertainty, settled in whatever currency each design chooses. A decision checklist Before committing to any distributed SQL database, a team should be able to answer the following in writing. 1. Which failures must the system survive without losing acknowledged writes: node, zone, or region? 2. What latency budget does each critical operation have, and does the required quorum fit within it, given measured round trips between the chosen regions? 3. Where are the system's writers and readers, and does each table's placement put its quorum and leaseholder near its writers? 4. Which tables are regional, which are partitioned by row, and which are read-mostly reference data? 5. Which primary keys are sequential, and how will they be changed or sharded? 6. Which code paths need retry handling, and which isolation level does each require? 7. How will the team test partitions, node loss, clock skew, and schema changes under load before production? 8. Who operates the database, at what cost, and under what license? Teams that can answer these clearly tend to succeed regardless of which system they choose. Teams that cannot tend to discover the answers in production. Conclusion The promise of distributed SQL is usually stated as a reconciliation: the scale and resilience of NoSQL with the guarantees of a relational database. The chapters of this booklet suggest a more precise statement, and the precision changes how a team should approach these systems. Distributed SQL did not reconcile the old trade-off by finding a loophole. It reconciled it by changing what the trade-off is about. In the NoSQL era, the choice was made once, at the level of the whole database: you chose a system that was consistent or one that was available and fast, and every piece of data inherited that choice. Distributed SQL moves the choice down to the level of the individual table, and often the individual row. A payments table can require a cross-region quorum while a product catalog is readable locally everywhere and a user's profile lives only in the user's region, all within one database, one transaction model, and one SQL interface. The costs are still there. Every strongly consistent write still pays the round trip to a majority, and every timestamp still carries some uncertainty that someone must pay for. What changed is that those costs became addressable, piece by piece, by whoever designs the schema. That shift has a consequence that is easy to state and hard to absorb: the database administrator's most important work has moved into the data model. In a single-node database, schema design was mostly about correctness and query efficiency. In a distributed SQL database, schema design is also topology design. Choosing a primary key decides whether writes spread across the cluster or pile onto one machine. Choosing a table's locality decides which region pays for each write. Choosing a survival goal decides how many milliseconds every transaction costs, forever. Organizations adopting these systems often treat them as drop-in replacements for PostgreSQL and staff accordingly. The ones that succeed treat schema review as an architectural activity, with latency arithmetic on the whiteboard, because a mistake in key design or placement cannot be fixed by adding nodes. The clock gap is closing The second conclusion concerns time. When Spanner's paper appeared in 2012, TrueTime looked like an unrepeatable advantage, a hardware investment only Google could make, and the rest of the industry built around its absence. CockroachDB and YugabyteDB accepted a configured 500-millisecond uncertainty bound, paid for it with read restarts, and settled for a real-time guarantee weaker than Spanner's for transactions on disjoint keys. That gap is narrowing from both directions. Cloud providers now offer clock synchronization with microsecond-level error bounds and, crucially, software that reports the bound with every reading, which is the essential property of TrueTime. YugabyteDB already consumes those bounds. CockroachDB can read a precision hardware clock. And Google itself, in building Spanner Omni to run outside its own data centers, built a software-based TrueTime alternative. The lesson of TrueTime turns out not to be about atomic clocks. It is about an interface: a clock that admits its own uncertainty, so that a database can pay for that uncertainty explicitly rather than hope it is small. As bounds shrink toward microseconds, the commit wait that strict serializability requires shrinks with them, until it becomes negligible relative to replication latency. The likely result over the next several years is convergence: commodity-hardware distributed databases able to offer Spanner's external consistency at a cost too small to argue about, at least in clouds with good time infrastructure. Teams choosing a system today should weigh the real-time guarantee less heavily than they would have five years ago, because the difference is becoming a configuration detail rather than an architectural one. They should, however, invest now in the time infrastructure itself, because it is a correctness dependency whose importance will only grow. What remains unsettled Several questions remain open, and a careful team should treat them as such rather than accept any vendor's framing. The first is how much cross-region coordination applications can avoid. Regional-by-row tables keep most operations local, but global uniqueness constraints, foreign keys between rows homed in different regions, and transactions that genuinely span regions still pay cross-region round trips. No system has a general solution, because none is possible: a constraint that spans the planet must, at some point, consult the planet. The design patterns for minimizing these costs, such as aligning home regions across related tables, scoping uniqueness to a region, and preferring keys that are unique by construction, are still matters of craft rather than settled practice. The second is the relationship between synchronous and asynchronous multi-region designs. Quorum-based survival requires at least three regions and costs a round trip on every write. Asynchronous replication between independent clusters, offered by YugabyteDB's xCluster and CockroachDB's logical data replication, provides local writes in only two regions but reintroduces data loss on failover and last-writer-wins conflict resolution. Many real deployments will mix them, using synchronous replication for data whose loss is unacceptable and asynchronous replication for data whose conflicts are tolerable. The tooling to express and verify such mixed guarantees is immature, and the risk is that teams describe a mixed design as "active-active" and assume guarantees it does not provide. The third is row-level geo-placement in Spanner, still labeled a preview as of late 2026, and the broader question of whether Spanner's guarantees remain distinctive as its competitors close the clock gap and as Spanner itself leaves Google Cloud through Omni. The fourth is commercial. Licensing in this category has changed repeatedly, from Apache 2.0 to the Business Source License to a proprietary source-available license with a revenue threshold in CockroachDB's case. A database chosen today will likely run for a decade or more. Teams should plan for the possibility that terms will change again, and should weigh an open-source license or a managed service with a clear contract accordingly. What to do differently The practical implications can be put as a short set of changes in habit for teams coming from single-node databases. Write down the survival requirement before choosing anything, and be honest about it. Region survival costs a cross-region round trip on every write, forever. Many applications that ask for it would be better served by zone survival in one well-chosen region, plus follower reads and a tested disaster recovery plan. Choosing the weaker goal deliberately is better engineering than choosing the stronger one by default. Measure round-trip times between candidate regions and do the quorum arithmetic before provisioning. A topology whose write latency exceeds the application's budget will not be rescued by tuning. Make follower reads and bounded-staleness reads the default for any read that can tolerate a few seconds of staleness, and require a reason for strong reads from distant regions rather than the reverse. This single habit removes more cross-region latency than any other. Treat primary keys as placement decisions. Ban sequential leading keys in range-distributed tables, and decide hash versus range distribution deliberately per table. Assume transactions will be retried, and build the retry loop before migration, not after the first incident. Choose isolation levels by auditing invariants, not by measuring throughput. Test failure, not just load. Partition regions, kill leaseholders, skew clocks within the configured bound, and run schema changes under write load, then verify not only that the system stayed up but that the data is correct. Distributed SQL databases are among the most sophisticated software systems in production use, and their designers have done remarkable work to make that sophistication invisible behind a familiar SQL interface. The final lesson of this booklet is that the invisibility is partial by necessity. The physics is still there. A team that understands where the costs fall can place them where they hurt least, and will get what the category promises: a single relational database that scales, survives, and stays correct. A team that does not will get the same correctness, at a latency it cannot explain. Hashtags: #DistributedSQL #DistributedDatabases #SQLDatabases #GoogleSpanner #CockroachDB #YugabyteDB #DatabaseArchitecture #DistributedSystems #CloudDatabases #NewSQL #HorizontalScaling #DatabaseReplication #RaftConsensus #PaxosConsensus #TrueTime #HybridLogicalClocks #ACIDTransactions #DistributedTransactions #Serializability #StrongConsistency #MultiRegionDatabases #HighAvailability #FaultTolerance #DataPartitioning #DatabaseSharding #MVCC #GeoPartitioning #DatabasePerformance #DatabaseScalability #CloudNativeDatabases

Download the Book (PDF): Introduction On the afternoon of 10 March 2016, in the second game of a five-game match in Seoul, a computer program placed a black stone on the fifth line of the right side of a Go board. The commentators paused. Michael Redmond, a nine-dan professional providing English commentary, picked up the stone, put it on his demonstration board, took it off again, and wondered aloud whether it had been entered by mistake. Lee Sedol, the opponent and one of the strongest players of his generation, stood up and left the room for nearly a quarter of an hour. The move, which history now calls Move 37, was one the program itself estimated a human professional would play with a probability of roughly one in ten thousand. It turned out to be the decisive idea of the game. AlphaGo won the match four games to one. The move is often told as a story about creativity, and it is that. But it is more usefully read as a story about where knowledge comes from. AlphaGo had learned its first instincts from human games, so it knew what a professional would usually do. What it played instead came from something it had built for itself: an evaluation of positions refined by millions of games played against copies of itself, and a search procedure that tested candidate moves by looking ahead. The program did not choose Move 37 because it had seen it. It chose it because, after search, the move looked better than the moves it had seen. Eighteen months later, the same team removed the human games altogether. AlphaGo Zero started from random play, knowing only the rules, and within three days of training was beating the version that had defeated Lee Sedol in every game of a hundred-game match. A year after that, AlphaZero did the same for chess and shogi with a single algorithm. Two years later, MuZero did it without even being told the rules: it learned its own model of how the game worked and planned inside that model. Along a parallel track, policy-gradient methods culminating in proximal policy optimization, PPO, carried reinforcement learning into games too large and too messy for tree search, such as Dota 2, and then out of games altogether, into the training of the language models that many people now use daily. This book is about that sequence of ideas, and about the question it raises for anyone who wants to use them: what, exactly, transfers from games to the world? The single idea underneath The answer this book argues for is simple to state, and the chapters are the work of earning it. The systems that master games without human data all run on one loop. A search procedure, looking ahead from the current situation, produces better decisions than the system's raw instincts. Those better decisions are then used as training targets to improve the instincts. Improved instincts make the next search sharper, which produces still better targets. Reinforcement learning researchers call this policy improvement and policy evaluation; the AlphaGo Zero paper described the search as a powerful policy improvement operator. Stripped of terminology, it is a machine that bootstraps its own teacher. That loop is extraordinarily powerful, but it has preconditions, and they are where the hard thinking lies. The loop needs a way to generate experience cheaply and faithfully: a simulator, a set of rules, or at minimum a learned model good enough to plan in. It needs a score that actually measures what you want: a win, a compression ratio, a verified proof, a protein that folds. And it needs enough repetition for errors to wash out. Board games provide all three in their purest form, which is why they were the proving ground. The real world provides them only in patches. The transfer of game-playing algorithms to real problems succeeds precisely to the extent that a problem can be recast so that the loop has something to run on, and fails, sometimes quietly, where it cannot. Read in that light, the field's history stops looking like a list of trophies and starts looking like a sustained campaign to relax those preconditions one at a time. AlphaGo needed human data to get started; AlphaGo Zero showed that the loop could start from nothing if the rules were known. MuZero relaxed the need for known rules by learning a model that was only accurate about the things that matter for planning. PPO and its descendants relaxed the need for search entirely, at the cost of far more experience, which made them the tool of choice where simulators are fast but tree search is impractical. Systems for poker and StarCraft relaxed the assumption of perfect information. And the newest work in mathematics and chemistry relaxes the assumption that the environment must be a game at all, by turning a proof checker or a physics model into the thing that keeps score. What this book covers, and what it does not The first chapter sets out the basic vocabulary of reinforcement learning, but only the parts that bear weight later: states and actions, rewards and returns, value functions and policies, and the credit assignment problem that makes all of it difficult. It does so through the two landmark systems that preceded AlphaGo, Gerald Tesauro's backgammon program and DeepMind's Atari player. The second chapter is a close reading of Monte Carlo Tree Search, the algorithm that rescued computer Go in the mid-2000s and remains the search engine inside every Alpha system. The third and fourth chapters follow AlphaGo, then AlphaGo Zero and AlphaZero, with attention to the specific engineering choices that made the self-improvement loop stable. The fifth explains MuZero, the step from planning with a rulebook to planning with a learned model, and the successors that made it cheaper and more general. The sixth chapter changes register to cover the other great family of methods: policy gradients, actor-critic algorithms, and PPO, the workhorse that trained OpenAI Five and, in a modified form, the instruction-following language models of the 2020s. The seventh deals with games that break the assumptions of board games, namely StarCraft, poker and Diplomacy, and with the uncomfortable discovery that superhuman Go programs can be beaten by simple adversarial tricks. The eighth and ninth chapters take the algorithms out of the arcade: first into logistics, scheduling, chip design, compression and algorithm discovery, then into proteins, where the honest story is more interesting than the popular one. AlphaFold, the system most people associate with protein folding, is largely not a reinforcement learning system at all; but search-and-self-play methods descended from AlphaZero are now designing proteins that have been built and tested in the laboratory. The book assumes a reader who is comfortable with the idea of a neural network as a trainable function approximator and who is willing to follow a mechanism step by step. It does not require calculus, though a few ideas are stated precisely where precision helps. Some short code sketches appear where a few lines say more clearly than a paragraph what an update rule does. There are no derivations, and nothing here is a substitute for the original papers, which are listed in the notes and the further reading at the end and are, in most cases, unusually readable. Several neighbouring subjects are left aside or touched only where they bear on the argument. Multi-agent theory, robotics, offline reinforcement learning and the safety literature on reward hacking each deserve books of their own. They appear in this one only where a game-playing system ran into them. Why it matters now It is fair to ask why a book about game-playing algorithms belongs in 2026, when the public conversation about artificial intelligence is dominated by large language models. The answer is that the two stories have converged. The step that turned pretrained language models into useful assistants was reinforcement learning from human feedback, typically run with PPO. The step that produced the reasoning models of 2024 and 2025, systems that work through mathematics and programming problems at length before answering, was large-scale reinforcement learning against verifiable rewards, using PPO variants such as group relative policy optimization. DeepMind's AlphaProof, which reached silver-medal standard at the 2024 International Mathematical Olympiad and was described in Nature in 2025, is AlphaZero's self-play loop wrapped around a formal proof checker. The ideas worked out on Go boards are now load-bearing parts of the most consequential software being built. They are also, increasingly, part of the infrastructure of ordinary industry. A MuZero agent has been used to control rate decisions in the VP9 video codec on YouTube traffic. A tree-search agent found faster sorting routines that are now part of the LLVM standard C++ library. Ride-hailing dispatch, warehouse inventory, data-centre cooling and chip floorplanning have all seen serious deployments or serious claims, and some of those claims have been seriously contested. A reader who understands the self-improvement loop, and its preconditions, is in a much better position to tell which claims are likely to hold. That is the practical promise of this book. By the end, the reader should be able to look at a new problem, whether a delivery network, a molecule or a scheduling system, and ask the right three questions: Where does the experience come from? What keeps the score? And what plays the role of search? Those questions do not have to be answered with AlphaZero. But the systems in these chapters are the clearest demonstration we have of what becomes possible when all three have good answers. CHAPTER 1 Learning from Reward Every system in this book learns from a signal that is thinner than the one most machine learning relies on. A supervised image classifier is told, for every training example, the right answer. A reinforcement learning agent is told only how well things went, often long after the decisions that mattered, and never which decision deserves the credit or the blame. A Go program that wins after 250 moves receives a single bit of information: it won. Somewhere in those 250 moves were the good ideas and somewhere were the mistakes the opponent failed to punish. Working out which was which, from that single bit, repeated over millions of games, is the core difficulty the field exists to solve. This chapter sets out the small amount of machinery needed to talk about that difficulty precisely, and then follows the two systems that proved it could be solved at scale before AlphaGo: Gerald Tesauro's TD-Gammon, which taught itself backgammon in the early 1990s, and DeepMind's Deep Q-Network, which learned to play dozens of Atari games from raw pixels in 2015. Both are worth understanding in detail, because between them they establish nearly every pattern, and every failure mode, that the later systems inherit. States, actions, rewards and the problem of delay The standard framing is the Markov decision process. At each step an agent observes a state, chooses an action, receives a numerical reward and moves to a new state. The process is called Markov because the next state and reward are assumed to depend only on the current state and action, not on the path that led there. A chess position is Markov in this sense, give or take the details of castling rights and repetition: the board tells you everything that matters about the future. A hand of poker is not, since what matters includes cards you cannot see, a complication the later chapters return to. The agent's goal is to maximise not the next reward but the return, the total reward accumulated over time. In many formulations future rewards are discounted by a factor slightly less than one per step, so that a reward a hundred steps away counts for somewhat less than one received now. Discounting has a mathematical purpose, keeping infinite sums finite, and a practical one, expressing a preference for sooner over later. In board games the discount is usually set to one and the only reward is the result at the end: plus one for a win, minus one for a loss, zero for a draw. Two functions organise almost everything that follows. A policy is a rule for choosing actions, often a probability distribution over moves given the current state. A value function estimates how good it is to be in a state, meaning the expected return from that state if the agent follows some given policy afterwards. A close relative, the action-value function, usually written Q, estimates the expected return from taking a particular action in a particular state and then following the policy. If you knew the true action-values for the best possible policy, acting optimally would be trivial: in every state, pick the action with the highest Q. All the difficulty lies in learning those values, or learning a good policy directly, from experience. The relationship that makes learning possible is the Bellman equation, named after Richard Bellman, who developed dynamic programming in the 1950s. It says that the value of a state equals the immediate reward plus the value of the state that follows, averaged over whatever randomness is involved. This is almost a tautology, but it is a very useful one, because it turns a question about the whole future into a question about one step. If your estimate of a state's value disagrees with the reward you received plus your estimate of the next state's value, one of those estimates is wrong, and you can nudge the first towards the second. That nudge is temporal-difference learning, introduced in its modern form by Richard Sutton in 1988. The difference between the old estimate and the new one-step target is the temporal-difference error, and learning consists of shrinking it. The idea is so central that it is worth seeing in the few lines it actually occupies. # One step of tabular TD(0) value learning def td_update(V, state, reward, next_state, alpha=0.1, gamma=0.99, done=False): target = reward if done else reward + gamma * V[next_state] td_error = target - V[state] V[state] += alpha * td_error # move part-way to target return td_error What makes this remarkable is that it learns from guesses. The target contains the agent's own estimate of the next state, which may be badly wrong. Yet under reasonable conditions the process converges, because the one true piece of information, the actual reward, keeps flowing back through the chain of estimates. Sutton and Andrew Barto, whose textbook Reinforcement Learning: An Introduction remains the standard reference, call this bootstrapping. Every later system in this book bootstraps in some form. AlphaGo Zero's value network learns to predict game outcomes; MuZero's learned model bootstraps value estimates several steps ahead; PPO's critic is a temporal-difference learner. The credit assignment problem has two dimensions. The temporal dimension is the one already described: which of many past actions earned a delayed reward? Temporal-difference learning attacks it by propagating value backwards one step at a time, which is efficient but slow to carry information across long gaps. Monte Carlo methods attack it from the other end, waiting until the game is over and crediting every move with the final result, which is unbiased but noisy, since a good move in a lost game is punished along with the bad ones. Most practical algorithms blend the two. The structural dimension is subtler: when an agent's behaviour is produced by a network with millions of parameters, which parameters should change? That is the problem gradient descent solves, and it is why deep reinforcement learning only became possible once deep learning worked. The final ingredient is exploration. An agent that always takes the action it currently believes best will never discover that some other action was better. An agent that explores too much wastes its experience on actions it already knows to be bad. Simple schemes take a random action some fraction of the time; more principled ones, which Chapter 2 examines in the context of tree search, add a bonus to actions that have been tried less often, so that uncertainty itself becomes a reason to look. Exploration is the problem that most clearly separates reinforcement learning from supervised learning, because the agent's choices determine which data it ever sees. Two further distinctions organise the families of methods that the later chapters describe, and it helps to fix them now. The first is between model-free and model-based learning. A model-free agent learns values or a policy directly from experience, without ever representing how the environment works; it knows that pressing a button in a certain situation tends to pay off, but not why. A model-based agent has, or learns, a model that predicts what will happen after each action, and can use it to plan by imagining consequences before committing. The rules of chess are a perfect model, and the AlphaZero family exploits them. MuZero learns a model. DQN and PPO, in their standard forms, use none. The second distinction is between on-policy and off-policy learning. An on-policy method learns about the policy it is currently following, from data that policy generated; once the policy changes, old data describes a different agent and must largely be discarded. An off-policy method can learn about one policy, typically the best one it can identify, from data generated by another, such as an older version of itself or a more exploratory variant. Off-policy learning can reuse experience many times, which makes it more economical, but it is also one corner of the deadly triad described below, and much of the engineering in the field is about making it stable. PPO, as Chapter 6 explains, sits near the on-policy end and pays for its stability with a heavy appetite for fresh data; DQN and MuZero's reanalysis sit at the off-policy end and work hard to keep old data useful. TD-Gammon: the first self-taught expert Gerald Tesauro, working at IBM's Thomas J. Watson Research Center, published the first version of TD-Gammon in 1992. It was a neural network, tiny by modern standards, with a single hidden layer of a few dozen units in its early versions, that took a backgammon position as input and output an estimate of the probability of winning. It chose moves by evaluating each legal successor position and picking the one it rated best. And it learned that evaluation by playing against itself, adjusting its weights after every move with a temporal-difference rule called TD(lambda) that blends one-step and longer-horizon targets. Tesauro had previously built a backgammon program, Neurogammon, by training a network on positions labelled by human experts. Neurogammon was good for its time. TD-Gammon, given no expert labels at all, overtook it. The version Tesauro described in his 1995 article in Communications of the ACM, trained on about 1.5 million self-play games and equipped with a shallow search, played at a level close to the best human players in the world. More striking than its strength was its influence. Its assessments of certain opening plays, notably how to handle opening rolls such as 2-1, 4-1 and 5-1, where established human practice favoured one style and the program consistently preferred another, led strong players to re-examine and in several cases change their own play. It was the first clear instance of a self-taught program revising expert human opinion, a pattern AlphaGo would repeat on a much larger stage two decades later. Why did it work? Part of the answer is luck in the choice of game. Backgammon has dice, so the positions encountered in self-play are naturally varied: the randomness forces the agent to explore without any explicit exploration scheme, and the value function is relatively smooth, with similar positions having similar winning chances. Attempts to replicate TD-Gammon's success in chess and Go during the 1990s largely failed. In deterministic games, self-play tends to collapse into narrow repeated lines, and the value of a position can swing violently with a single stone or piece. Researchers drew the reasonable conclusion that backgammon was special. They were half right. Backgammon was special in the sense that the dice did some of the work that, in deterministic games, would later have to be done by search. TD-Gammon also displayed, in miniature, the core loop that this book is about. The policy was implicit, choosing the move leading to the best-valued position, so improving the value function automatically improved the policy. Improved play generated more informative games, which improved the value function further. Tesauro's program did one-step or two-step lookahead at best. The systems of the next decades would put a much stronger search at the centre of the same loop. Deep Q-Networks and the Atari benchmark For roughly twenty years after TD-Gammon, reinforcement learning with neural networks had a reputation for instability. The difficulty was well understood in theory. Combining three things, function approximation by a network, bootstrapping from the network's own estimates, and learning from data generated by a policy different from the one being evaluated, could cause value estimates to diverge to infinity. Sutton and Barto later named this combination the deadly triad. Most successful applications either avoided neural networks or used carefully hand-designed features. In 2013 a small London company called DeepMind posted a paper showing a single network learning to play several Atari 2600 games directly from screen pixels. The expanded version, published in Nature in February 2015 by Volodymyr Mnih and colleagues, tested the Deep Q-Network, or DQN, on 49 games using the same architecture, the same hyperparameters and the same learning procedure for all of them. The agent saw the same thing a human player sees, the raw screen, plus the score, and could press the same buttons. On 29 of the 49 games it reached at least 75 per cent of the score of a professional human games tester, and on many it far exceeded it. DQN learned the action-value function Q with a convolutional network, essentially the image-recognition architectures that had just transformed computer vision. Its two key stabilising tricks are worth naming, because versions of both recur throughout this book. The first was experience replay: rather than learning from each transition as it happened, the agent stored its recent experience in a large buffer and trained on random samples from it. This broke the strong correlations between consecutive frames, which otherwise cause a network to overfit to whatever it happens to be doing now, and allowed each experience to be reused many times. The second was a target network: the bootstrapped targets were computed using a frozen copy of the network, updated only periodically, so that the agent was not chasing a target that moved every time it took a step. The results were uneven in an instructive way. DQN was superb at games where rewards came quickly and frequently, such as Breakout and Pong, where every hit or miss produces immediate feedback. In Breakout it famously learned to tunnel through the side of the wall of bricks and send the ball bouncing along the top, a strategy its designers had not anticipated. It was hopeless at games where reward was sparse and required long sequences of purposeful action. In Montezuma's Revenge, which requires navigating rooms, collecting keys and avoiding hazards before any points arrive, DQN scored essentially nothing. Random exploration almost never stumbles on a reward, and without reward there is nothing to propagate. The six years that followed produced a long series of improvements, among them double Q-learning to correct systematic overestimation, prioritised replay, distributional value estimates, and exploration bonuses based on novelty. DeepMind's Agent57, reported in 2020, was the first agent to exceed the human benchmark on all 57 games of the standard Atari suite, including Montezuma's Revenge and the equally notorious Pitfall. It needed enormous amounts of experience to do so, billions of frames for the hardest games, a point that MuZero and its successors would later take up. What the early systems established TD-Gammon and DQN together leave behind four lessons that frame the rest of this book. The first is that learning from self-generated experience can exceed learning from human labels, but only when the experience is rich enough. TD-Gammon beat Neurogammon because self-play generated an endless supply of positions whose labels, final outcomes, were exactly the thing that mattered. The quality of the experience depended on the dice keeping play varied. Where experience is narrow or uninformative, as in deterministic self-play without search or in sparse-reward Atari games, learning stalls. The second is that value estimates are only as good as the states they are trained on. A network that has never seen a certain kind of position will evaluate it unreliably, and an agent that acts on unreliable evaluations may steer itself into exactly those positions. This distribution problem returns in Chapter 7, in the discovery that superhuman Go programs can be led into board configurations they misjudge. The third is that the reward defines the task, often more literally than its designers intend. Atari agents learned to exploit quirks of game physics and scoring. In a widely cited OpenAI demonstration from 2016 involving a boat-racing game called CoastRunners, an agent discovered that it could score more points by circling endlessly through a lagoon collecting regenerating targets, catching fire and crashing, than by finishing the race. Games make reward easy to specify, which is one reason they were the proving ground. Real-world problems rarely do, which is one reason transfer is hard. The fourth lesson is the one most directly relevant to what came next. Neither TD-Gammon nor DQN did much planning. TD-Gammon looked a move or two ahead; DQN looked nowhere at all, acting on its instantaneous estimate of Q. This works when the value function is smooth and accurate enough that one glance suffices. It does not work in Go, where a single stone changes the life or death of a large group and the consequences of a move may not become visible for dozens of turns. For Go, the field needed a way to look ahead that did not require an accurate evaluation function to begin with, since nobody knew how to write one. That method arrived, not from deep learning at all, but from a simple idea about random games, and it is the subject of the next chapter. CHAPTER 2 Monte Carlo Tree Search When IBM's Deep Blue defeated Garry Kasparov in 1997, it did so by searching. Its custom chips evaluated around two hundred million chess positions per second, pruning the game tree with the alpha-beta algorithm and scoring the positions at the leaves with an evaluation function written by people who understood chess. The evaluation function was the indispensable part. Alpha-beta search tells you how to look ahead efficiently; it does not tell you what a position is worth when you stop looking. In chess, decades of human knowledge, about material, mobility, king safety and pawn structure, could be compressed into a function that was crude but serviceable. In Go, nobody could write that function. For the next fifteen years, that single fact was the main reason computers played Go poorly. This chapter explains how a different kind of search, built on random games rather than expert judgement, broke the deadlock, and why that search, Monte Carlo Tree Search, became the engine inside AlphaGo and every system descended from it. Why Go resisted brute force Go is played on a 19-by-19 grid. Players alternately place stones on empty points, and stones or groups of stones are captured when they are surrounded. The goal is to control more territory than the opponent. The rules fit on a page. The consequences do not fit anywhere: John Tromp and colleagues computed in 2016 that the number of legal positions on a full board is about 2.08 times ten to the power of 170. The size of the state space is not, by itself, the problem. Chess also has an astronomically large state space. The problem is the combination of two properties. The first is branching: a typical Go position offers around 250 legal moves, against roughly 35 in chess, and games last around 150 moves. A full-width search a few moves deep is already hopeless, and alpha-beta pruning, which in good conditions roughly halves the effective depth of exponential growth, cannot rescue it. The second, more fundamental property is that Go positions resist static evaluation. In chess, counting material gives a rough but meaningful estimate of who is winning, and most of a position's value is visible locally. In Go, the status of a group of stones, whether it will live or die, can depend on reading sequences many moves long, on distant stones that interact with it through a ladder or a shared liberty, and on judgements about thickness and influence that strong players can feel but struggle to articulate. Programs that tried to encode this knowledge by hand, the strongest of the 1990s and early 2000s such as GNU Go and The Many Faces of Go, reached the level of a moderate club player and stalled. Simple tactical errors could be corrected one at a time; strategic judgement could not. What the field needed was a way to estimate the value of a position without understanding it. The idea that supplied it was, at first sight, absurd. Random games as an evaluation function In 1993 the physicist Bernd Brügmann proposed evaluating Go moves by playing out many random games from each candidate move to the end and recording how often each led to a win. A random game of Go is nonsense: neither side plays sensibly, groups live and die by accident. But averaged over thousands of such games, the win rate after a given move carries real information. A move that solidly secures territory tends to win more random continuations than a move that does not, because the random play that follows is equally bad for both sides and the solid move's advantages survive the noise. Brügmann's program was weak, and for a decade the idea lay mostly dormant. Its revival in the mid-2000s came from combining random playouts with a tree. Rather than spreading playouts evenly across all candidate moves, the program builds a search tree incrementally, concentrating effort on moves that look promising, and uses the random playouts only to evaluate the leaves. In 2006 Rémi Coulom described this combination in a paper on selectivity and backup operators in Monte Carlo tree search, coining the name, and his program Crazy Stone won the 9-by-9 Go tournament at that year's Computer Olympiad. In the same year Levente Kocsis and Csaba Szepesvári published UCT, short for upper confidence bounds applied to trees, which gave the selection rule a principled foundation and became the canonical form of the algorithm. Within a year, programs using the approach, notably Sylvain Gelly and colleagues' MoGo, had overtaken every hand-crafted Go program. The algorithm repeats four steps many thousands of times per move. In the selection step, the program starts at the root, the current position, and descends through the existing tree, at each node choosing which child to follow according to a rule that balances how good the child has looked so far against how little it has been explored. In the expansion step, when the descent reaches a position that is not yet in the tree, that position is added as a new leaf. In the simulation step, often called the rollout, the program plays a fast game from the new leaf to the end, choosing moves randomly or by a cheap heuristic. In the backup step, the result of that game, a win or a loss, is passed back up the path that was followed, and every node along it updates its visit count and its running average of results. After the time budget is spent, the program plays the move at the root that was visited most often. Using the visit count rather than the average value is a deliberate choice: a move with a high average from very few visits may be a statistical fluke, while a move that the selection rule kept returning to has survived repeated scrutiny. Several properties make this procedure unusually well suited to hard games. It is anytime: it can be stopped after any number of iterations and will return its best current guess, with quality improving smoothly with more time. It is asymmetric: the tree grows deep along promising lines and stays shallow elsewhere, which is exactly how strong human players read. It requires no domain knowledge beyond the rules, since the random playouts need only to know which moves are legal and who won. And it converges: given unlimited time, UCT's estimates approach the true minimax values of the game. The exploration bonus The heart of the algorithm is the selection rule, and it repays close attention because a descendant of it sits inside AlphaGo, AlphaZero and MuZero. The problem it solves is a version of the multi-armed bandit, named after a gambler facing a row of slot machines with unknown payout rates. Each pull reveals a little about one machine. Pulling only the machine that has paid best so far risks never discovering a better one; pulling machines at random wastes money on known losers. In 2002 Peter Auer, Nicolò Cesa-Bianchi and Paul Fischer analysed a rule called UCB1, which scores each machine by its average payout plus a bonus that grows with the logarithm of the total number of pulls and shrinks with the number of times that machine has been pulled. The bonus expresses optimism in the face of uncertainty: an arm that has been tried rarely might be better than it looks, and the rule gives it the benefit of the doubt until evidence accumulates. UCB1 provably keeps the cost of exploration, the regret, growing only logarithmically with time. Kocsis and Szepesvári's insight was to treat every node of the search tree as its own bandit problem, applying UCB1 to choose among children. Each descent through the tree is then a sequence of bandit decisions, and the backed-up playout result is the payout. The interaction is subtle, because the payouts at a node shift as the subtree below it improves, but the method works and comes with convergence guarantees. AlphaGo replaced UCB1 with a variant, introduced by Christopher Rosin in 2011 and known as PUCT, that adds a prior. Each child receives an exploration bonus proportional to a prior probability, supplied in AlphaGo's case by a neural network trained to predict good moves, multiplied by the square root of the parent's visit count and divided by one plus the child's own visit count. A move the network considers likely gets explored early and often. A move the network considers implausible still gets explored, but only after the plausible ones have been examined enough that their bonuses have shrunk. The prior does not dictate the search; it shapes where the search looks first. That distinction turns out to be what allowed AlphaGo to play Move 37, a move its prior rated as very unlikely but its search eventually found to be strong. The formula looks compact, but the consequences of its constants are large. A high exploration weight makes the search broad and cautious; a low one makes it deep and confident. Much of the engineering in the later systems consists of tuning this balance, and Chapter 5 describes a 2022 refinement, Gumbel MuZero, that replaced parts of the rule with a more principled procedure precisely because the classic form behaves poorly when only a handful of simulations are affordable. One search, followed closely The mechanics become clearer when followed through a small case. Imagine a position in which the program has three candidate moves, which we can call A, B and C, and a budget of a few hundred simulations. At the start, the tree contains only the root. The first three iterations each expand one of the three moves, since a move with no visits has an unbounded exploration bonus and is always selected before any move that has been tried. Each is evaluated by a single random playout. Suppose A's playout wins and B's and C's lose. On the fourth iteration, the selection rule compares the three. A has an average of one win from one visit; B and C have zero from one. Their exploration bonuses are equal, since each has been visited once, so A is chosen. The search descends to A's position, expands one of the opponent's replies to A, plays out from there, and backs up the result. If it loses, A's average falls to one half, and on the next iteration the comparison is closer. Over the following dozens of iterations a pattern emerges. If A really is the best move, its average will settle at a higher level than the others', and the selection rule will return to it more and more, building a deep subtree beneath it in which the opponent's best replies are themselves being identified and explored. B and C will continue to receive occasional visits, because as A's visit count grows, the exploration bonuses of the neglected moves slowly rise relative to A's. If one of them was unlucky in its early playouts, these occasional revisits give it the chance to recover. Something subtle happens inside A's subtree. At the nodes where it is the opponent's turn, the selection rule picks the reply that is best for the opponent, so the statistics backed up through those nodes increasingly reflect the opponent's strongest replies rather than an average over all replies. As the subtree deepens, the average value at A therefore drifts from "how good is A against random play" towards "how good is A against the best reply." This is how Monte Carlo Tree Search approximates minimax search without ever performing an explicit minimisation: the asymmetric allocation of visits does the work. It is also why its estimates converge slowly; the averages at the root include many early simulations made before the search had found the best replies, and that early noise is diluted only gradually. At the end, the program plays whichever move has the most visits. In our example, A might have received two hundred of three hundred simulations, B seventy and C thirty. That distribution is itself informative. It tells us not only which move the search prefers, but how confident it is, and in which other moves it saw merit. That richer output will matter a great deal when, in Chapter 4, visit distributions become training targets. What pure Monte Carlo search could and could not do Between 2006 and 2015, Monte Carlo Tree Search took computer Go from weak amateur to strong amateur. Programs improved their rollouts with patterns learned from expert games, so that the simulated games were less random and the evaluations less noisy. A technique called RAVE, rapid action value estimation, developed by Gelly and David Silver in 2007, shared statistics between moves played at different times in the playouts, on the reasonable assumption that a move good now is often good a few moves later too, which made the early stages of search much more efficient. On the small 9-by-9 board, programs became competitive with professionals. On the full board, the best programs, Crazy Stone and Zen among them, reached around five or six dan on online amateur servers. In 2013 Crazy Stone defeated the professional Ishida Yoshio in a game where it received a four-stone handicap, a real achievement and also a precise measure of the gap that remained. The limitation was the evaluation. Random or lightly guided playouts are biased estimators of a position's value. They systematically misjudge positions whose outcome depends on precise sequences, such as a semeai, a capturing race between two groups, where a random player will blunder the race half the time regardless of who should win it. No amount of additional search corrects a systematic bias in the leaf evaluation; it only makes the program more confident in a wrong answer. Researchers could improve the rollout policies, but every improvement was hand-engineered and incremental. The tree also had a different problem: breadth. Even with the exploration bonus, the search spends simulations on many moves that any competent player would dismiss instantly. A prior that knew which moves were worth considering would make the same number of simulations go much further. By around 2014 the shape of the solution was visible to several groups at once. Deep convolutional networks had just transformed image recognition, and a Go board is, among other things, a 19-by-19 image. In late 2014, Chris Maddison, Aja Huang, Ilya Sutskever and David Silver, and separately Christopher Clark and Amos Storkey, showed that a convolutional network trained to predict expert moves could, without any search at all, play at the level of a decent Monte Carlo program. Facebook's Darkforest program combined such a network with tree search in 2015 and reached the level of the best existing engines. The architecture that would win, then, was coming into focus. Use a network trained on expert games as the prior, so that search looks at the right moves. Replace or supplement the random rollouts with a network that evaluates positions directly, so that leaf evaluations stop being systematically naive. Keep Monte Carlo Tree Search as the glue, because it combines prior and evaluation in a way that is robust to the errors of each. That is a fair description of AlphaGo. Search as an amplifier Before following that story, it is worth stating plainly what Monte Carlo Tree Search contributes, because the point becomes the backbone of everything that follows. Search is an amplifier of whatever knowledge it is given. Given only the rules and random playouts, it turns that minimal knowledge into strong amateur play. Given a good prior and a good evaluator, it turns them into something better than either alone, because it checks the prior's suggestions against the evaluator's judgements, several moves deep, and discards the ones that do not hold up. The output of search is a set of visit counts at the root, which constitute a sharper, better-informed policy than the prior that guided it. That last observation is the hinge on which this book turns. If search reliably produces a better policy than the network that guided it, then the network can be trained to imitate the search. The improved network then guides a better search, which produces a still better target. Monte Carlo Tree Search, in other words, is not only a way to choose a move during a game. It is a way to generate training data that is better than anything the learner currently knows. AlphaGo used it mostly in the first sense. AlphaGo Zero would use it in the second, and that change is what removed the need for human games. CHAPTER 3 AlphaGo and the Marriage of Search and Learning In October 2015, in a closed room at DeepMind's London office, a program called AlphaGo played five formal games against Fan Hui, the reigning European Go champion and a professional of second-dan rank. It won all five. The result was kept quiet until the publication of the paper by David Silver, Aja Huang and colleagues in Nature on 27 January 2016, which reported it alongside a challenge: in March, AlphaGo would play Lee Sedol, holder of eighteen international titles and widely regarded as one of the strongest players of the previous decade, for a prize of one million dollars. Most professional players, and many computer Go researchers, expected Lee to win. The gap between Fan Hui and Lee Sedol was large, and the conventional estimate was that computers were still a decade from professional strength. The expectation was reasonable on the evidence available and wrong because of something the evidence did not show: AlphaGo was not a better version of the existing programs. It was a different kind of system, in which three neural networks and a tree search each covered for the others' weaknesses. This chapter takes that system apart. Three networks and a search AlphaGo's pipeline had four stages, and the order matters, because each stage built on the previous one. The first stage was imitation. A deep convolutional network, the supervised-learning policy network, was trained to predict the move a strong human would play in a given position. Its training data was about 30 million positions drawn from games played on the KGS Go server by players of high amateur rank. The input was the board encoded as a stack of 19-by-19 feature planes, recording stone positions, liberties, recent moves and a few simple tactical features, and the output was a probability distribution over the 361 points. The network, thirteen layers deep, predicted the human expert's move 57 per cent of the time, a substantial jump over the previous best result of around 44 per cent. On its own, without any search, it played a respectable game. Alongside it, the team trained a much smaller and cruder rollout policy, which predicted expert moves only about a quarter of the time but could choose a move in roughly two microseconds instead of three milliseconds. The fast policy existed for one purpose: to play out simulated games to the end inside the tree search, where speed mattered more than quality. The second stage was self-improvement. A copy of the supervised policy network was trained further by reinforcement learning, playing games against randomly chosen earlier versions of itself and adjusting its weights by a policy-gradient rule, a method Chapter 6 explains in detail, to make moves that led to wins more likely and moves that led to losses less likely. Sampling opponents from a pool of past versions, rather than always playing the latest, was a deliberate guard against a familiar failure of self-play, in which a policy overfits to beating its current self and forgets how to handle anything else. The resulting reinforcement-learning policy network won about 80 per cent of its games against the supervised network it started from, and, playing without any search, won 85 per cent against Pachi, then one of the strongest open-source Monte Carlo programs. The third stage was evaluation. A value network, architecturally similar to the policy networks but with a single output, was trained to predict the winner of a game from a given position, assuming both sides played according to the reinforcement-learning policy. This was the network that addressed the problem that had held computer Go back for decades: the absence of a good evaluation function. The team's first attempt, training on positions from complete human games, failed in an instructive way. Successive positions in the same game are nearly identical but share the same outcome, so the network memorised games rather than learning to evaluate positions. The fix was to generate 30 million positions each from a separate self-play game, so that every training example was effectively independent. The resulting value network evaluated a position in a single forward pass with accuracy approaching that of hundreds of rollouts with the reinforcement-learning policy, at a tiny fraction of the cost. The fourth stage was the search that brought these components together during play. AlphaGo ran Monte Carlo Tree Search with the PUCT selection rule described in Chapter 2. When the search added a new position to its tree, the supervised policy network supplied the prior probabilities that shaped which children would be explored. When the search reached a leaf, it evaluated the leaf in two ways and averaged them: the value network's estimate, and the result of a fast rollout to the end of the game using the cheap rollout policy. The paper reported that the mixture played better than either method alone. The value network captured strategic judgement; the rollouts, crude as they were, remained sensitive to the precise tactical outcome of the specific position. One detail of this design is quietly revealing. The policy used as the search prior was the supervised network, not the stronger reinforcement-learning network. The authors found that the supervised policy worked better in search, presumably because it spread its probability across a more diverse set of plausible human moves, while the reinforcement-learning policy concentrated narrowly on the single move it liked best. A prior that is too confident starves the search of alternatives. The stronger player alone was the weaker guide for search, and the point anticipates a theme of the next chapter: what makes a good policy network for the purposes of search is not quite the same as what makes a good player. The search itself was expensive. The distributed version that played Fan Hui ran on 1,202 CPUs and 176 GPUs. The version that played Lee Sedol ran on Google's tensor processing units, chips whose existence Google announced publicly two months after the match. Seoul, March 2016 The match was played at the Four Seasons Hotel in Seoul between 9 and 15 March 2016, with each side given two hours of main time followed by byo-yomi overtime periods. Lee Sedol lost the first game after a complex middle game in which AlphaGo's play was solid rather than spectacular. The second game produced Move 37. The move was a shoulder hit on the fifth line, played on the right side of the board at a stage of the game when conventional wisdom held that such a high move ceded too much territory to the opponent. AlphaGo's own policy network rated the probability of a human playing it at about one in ten thousand. The search, examining the move's consequences, concluded it was strong anyway. Over the following hours of play, the stone's influence connected with AlphaGo's position across the board and contributed to a win that professional commentators, after initial bewilderment, came to describe as a demonstration of a style of play that humans had not considered. AlphaGo won the third game as well, taking the match. Lee won the fourth, and his winning move has its own name. Move 78 was a wedge in the centre of the board that, like Move 37, AlphaGo's policy network had rated as extremely unlikely. This time the low prior was a failure rather than an opportunity. Because the network rated the move improbable, the search had spent few simulations exploring it, and AlphaGo's evaluation did not register the danger until several moves later, after which its play deteriorated into a series of moves that looked to professionals like errors of a weak program. Lee resigned the fifth game after another close contest. The final score was four games to one. The two moves are mirror images, and together they show precisely how the search-and-prior architecture behaves. Search can overrule the prior, which is what produced Move 37: given enough simulations, a low-probability move that turns out to be strong will be found and chosen. But search can also be misled by the prior, which is what happened with Move 78: a strong move by the opponent that the prior considers unlikely may receive too few simulations for its strength to become visible before it is too late. The exploration bonus in the PUCT rule is a trade-off, not a guarantee, and the strength of the overall system depends on the prior being right often enough that its blind spots rarely matter. Playing to win, not to win by much One feature of AlphaGo's play puzzled professionals throughout the match and is worth understanding, because it follows directly from how the system was built and recurs in every descendant. Human professionals, when ahead, generally try to increase their lead or at least keep it, since a larger margin provides insurance against later mistakes. AlphaGo frequently did something else. In positions where it judged itself comfortably ahead, it played moves that conceded points, sometimes many points, in exchange for simplifying the position. Commentators initially called these moves slack or even mistakes. The games kept ending in AlphaGo victories by small margins. The explanation is that AlphaGo's value network did not estimate the score. It estimated the probability of winning. A move that gives up ten points but removes all remaining complications might raise the estimated winning probability from 90 to 95 per cent, while a move that gains ten points but leaves a complicated fight might lower it. For a system optimising the probability of winning, the first move is better, and the final margin is irrelevant. Human professionals, who also care about winning rather than margin, adopt this logic in principle; what they lack is the confidence to know that a half-point win is secure. AlphaGo's evaluations, refined through millions of self-play games, gave it that confidence, and its play reflected it. The same feature explains the strange sequence that followed Lee Sedol's Move 78 in the fourth game. Once AlphaGo's search finally recognised that its position had deteriorated, its estimated winning probability fell steeply. When every available move leads to a likely loss, the differences between moves in winning probability become small, and the choice between them becomes erratic: the system may prefer a move with a tiny chance of an opponent blunder over a move that loses more gracefully, because a gamble is all that remains. Professionals watching saw a sequence of moves that looked like those of a weak program. In a sense they were seeing the value function's indifference among losing positions. Later Go programs, including KataGo, address this by also predicting the score, so that when the outcome is no longer in doubt, the program can still prefer moves that keep the margin respectable, which makes them far more useful as teaching tools for human players. The lesson extends beyond Go and anticipates a theme of the later chapters. An optimiser does exactly what its objective specifies, and nothing that the objective omits. AlphaGo's objective said nothing about margin, so it was willing to trade margin freely, which was correct for its purpose but alarming to observers who assumed it shared their sense of what counted as playing well. When these methods are applied to real problems, the gap between what the objective specifies and what the people commissioning the system actually care about is often wider, and the consequences of the optimiser exploiting it can be less benign than a narrow win. AlphaGo Master and the human aftermath After Seoul, DeepMind continued to develop AlphaGo. Around the turn of 2017, an anonymous account named Master played sixty fast games online against many of the world's top professionals, winning all sixty; DeepMind then confirmed it was a new version of AlphaGo. In May 2017, at the Future of Go Summit in Wuzhen, China, AlphaGo Master defeated Ke Jie, then ranked first in the world, three games to nothing, and DeepMind announced that AlphaGo would retire from competitive play. The Master version differed from the Lee Sedol version in ways the team described more fully in the AlphaGo Zero paper later that year. According to that paper, Master used the same network architecture, reinforcement learning algorithm and search procedure as AlphaGo Zero, but it kept the handcrafted input features and the rollouts of the Lee Sedol version, and its training was still initialised by supervised learning from human games. It was, in hindsight, a transitional system. The human consequences were mixed and real. Lee Sedol retired from professional play in November 2019, telling the South Korean news agency Yonhap that even if he became number one, there was an entity that could not be defeated. Professional players began studying with AI, adopting openings that AlphaGo and its open-source successors favoured, including early invasions at the 3-3 point that human theory had long regarded as premature. A 2023 study in the Proceedings of the National Academy of Sciences by Minkyu Shin, Jin Kim, Bas van Opheusden and Thomas Griffiths analysed more than 5.8 million moves by professional players over 71 years, using a superhuman program to estimate the quality of each decision. They found that the quality of human decisions began to improve significantly after the advent of superhuman AI, and that novel moves, previously unobserved patterns, became both more frequent and more strongly associated with good outcomes. The program had not simply beaten the humans. It had, by their own measured play, made them better. What AlphaGo proved, and what it left open AlphaGo's achievement is sometimes summarised as deep learning beating Go, which misses the point. Deep networks alone did not beat Go; neither did search alone. AlphaGo worked because each component compensated for the specific failure of another. The policy network made search tractable by focusing it on a few dozen plausible moves instead of hundreds. The value network made search accurate by replacing biased random playouts with learned judgement. The search made the networks reliable by checking their suggestions several moves deep, catching the tactical errors that a single forward pass inevitably makes. The overall system was stronger than any of its parts by a margin that was the whole difference between a strong amateur and a world champion. What remained open was the role of the human data. AlphaGo's networks began by imitating amateur experts on a Go server. The reinforcement-learning stage improved the policy, but in a way that was bounded by its starting point: self-play policy gradients improve a player gradually, and a player that begins with human habits will tend to explore around those habits. The value network was trained on games played by that policy. Every component carried human assumptions, including whatever mistakes human players systematically make. There was a subtler mismatch inside the pipeline as well. The value network learned to predict the outcome of games played by the reinforcement-learning policy network acting alone, without search. But during a real match, AlphaGo did not play that way; it played with search, which made it considerably stronger than the raw policy. The value network was therefore estimating the value of positions under a weaker player than the one that would actually continue the game. In a position where the raw policy tended to blunder but the search would find the right continuation, the value network would be too pessimistic, and in a position where the raw policy happened to play well, its estimate might be too optimistic for reasons that search would not reproduce. The rollouts mixed into the leaf evaluation partly compensated, but the fundamental issue remained: the evaluator was trained on one player and used to guide another. The cure for this mismatch is conceptually simple. Train the value network on games played by the full system, search included, so that it learns to evaluate positions as the system will actually play them. Then train the policy network to predict what the search would choose, not what humans or the raw policy would choose. If both networks are trained on the outputs of search, and search uses both networks, then every component is aligned with every other, and improvements in any part propagate to the rest. That is exactly the design of AlphaGo Zero. There was also the matter of complexity. AlphaGo's training pipeline had several separately trained networks, a handcrafted rollout policy, a set of carefully engineered input features encoding Go-specific concepts such as liberties and ladders, and a mixing constant between value network and rollouts that had to be tuned. Each component was justified, but the whole was an elaborate machine built for one game. The question DeepMind's team set themselves next was whether all of that was necessary. Could a single network, given only the raw board and the rules, learn from nothing but its own play, using the search not merely to choose moves but to teach itself? The answer, published in October 2017, removed the human data, the rollouts, the handcrafted features and the second network in one step, and produced a stronger player than any version of AlphaGo. The next chapter explains why that simplification worked, and why it was not obvious that it would. CHAPTER 4 Tabula Rasa: AlphaGo Zero and AlphaZero The paper that DeepMind published in Nature on 19 October 2017 had a title that doubled as its thesis: "Mastering the game of Go without human knowledge." Its system, AlphaGo Zero, began from completely random play. It was given the rules of Go, the board as raw stone positions with a short history of recent moves, and nothing else: no human games, no handcrafted features describing liberties or ladders, no rollout policy. After three days of training it defeated the version of AlphaGo that had beaten Lee Sedol by a hundred games to none. After forty days it defeated AlphaGo Master, the version that had swept the world's top professionals, by 89 games to 11. A year later, in Science, the same team reported AlphaZero, which applied an almost identical algorithm, unchanged except for the rules and board representation, to chess and the Japanese game shogi as well as Go, and surpassed the strongest existing programs in all three. This chapter explains why removing human knowledge made the system stronger rather than weaker, and identifies the specific design decisions that let a self-teaching loop run stably for millions of games without collapsing. The loop, stated plainly The central idea of AlphaGo Zero is that Monte Carlo Tree Search, guided by a neural network, produces a better policy than the network on its own, and that this better policy can serve as the network's next training target. The loop has three parts that run concurrently. In self-play, the current best network plays games against itself. At every move, it runs a tree search, 1,600 simulations per move in the Go version, using the network to provide both the prior probabilities over moves and the value estimate at each leaf. There are no rollouts; the value network is trusted entirely for leaf evaluation. The search produces visit counts over the moves at the root, and these, normalised, form a probability distribution that the paper calls the search probabilities. A move is chosen in proportion to these probabilities, with some randomness early in the game, and the position, the search probabilities and eventually the final game result are recorded. In training, the network's parameters are updated on positions sampled from recent self-play games. For each position the network is trained to do two things at once: make its policy output match the search probabilities, and make its value output match the eventual game result. The loss function is the sum of a squared error between predicted value and outcome, a cross-entropy between the network's policy and the search probabilities, and a small regularisation term. # The AlphaGo Zero objective for one training position def loss(net, state, search_probs, outcome, c=1e-4): policy_logits, value = net(state) value_loss = (outcome - value) ** 2 policy_loss = -(search_probs * log_softmax(policy_logits)).sum() return value_loss + policy_loss + c * l2_norm(net.parameters()) In evaluation, the version used in the Go paper, newly trained checkpoints played 400 games against the current best network, and replaced it as the self-play generator only if they won at least 55 per cent. AlphaZero later dropped this gate and simply used the latest network for self-play, which turned out to work just as well. The logic of the loop is the logic of policy iteration, one of the oldest ideas in dynamic programming. Policy iteration alternates between evaluating a policy, working out how good each state is under it, and improving the policy, making it greedier with respect to those evaluations. Classical policy iteration does both exactly, over every state. AlphaGo Zero does both approximately, using sampling and a neural network. The value head performs policy evaluation: it learns to predict the outcome of games played by the current search-guided policy. The search performs policy improvement: it takes the network's policy and value, looks ahead, and produces a sharper policy. Training the policy head on search outputs distils that improvement back into the network, so that next time the improvement starts from a higher base. Two features make this better than the self-play of AlphaGo's original reinforcement-learning stage. First, the training target for the policy is a full distribution over moves, the visit counts, rather than a single bit about whether the chosen move eventually won. This is a far richer and less noisy signal. Every position yields information about the relative merits of every candidate move the search considered, not just the one that was played. Second, the target is always somewhat better than the network, because search improves on the network's own judgement. The learner is always chasing a teacher that is a step ahead of it, and the teacher improves whenever the learner does. The details that made it stable The idea of training a network to imitate its own search was not new in 2017. A closely related method, expert iteration, was developed independently by Thomas Anthony, Zheng Tian and David Barber and published in the same year, with the game Hex as its test bed. What AlphaGo Zero demonstrated was that the idea could be made to work stably at the scale of Go, and several specific engineering choices were responsible. The first was the network architecture. AlphaGo Zero used a single residual network, the architecture that had won the 2015 ImageNet competition, with two output heads, one for policy and one for value, sharing a common trunk. The paper's own ablations found that combining the two networks into one, and switching from ordinary convolutional layers to residual ones, each improved playing strength substantially, together by around 600 Elo points. Sharing a trunk forces the network to learn representations useful for both predicting good moves and evaluating positions, and each task regularises the other. The second was the decision to drop rollouts entirely. In AlphaGo, rollouts had been a hedge against an unreliable value network. In AlphaGo Zero, the value network is trained only on positions from strong, search-guided self-play, and it becomes reliable enough that the rollouts would only add noise and cost. Removing them also made each simulation much cheaper, allowing more simulations in the same time. The third was exploration. A self-play system playing deterministically against itself will tend to repeat the same games and stop learning. AlphaGo Zero introduced two forms of randomness. In the first thirty moves of each game, moves were sampled in proportion to the visit counts rather than chosen greedily, so the openings varied. And at the root of every search, the network's prior probabilities were mixed with noise drawn from a Dirichlet distribution, so that the search always gave some attention to moves the network currently undervalued. Without the noise, a move the network had wrongly learned to dismiss might never be searched, and so could never be rehabilitated. The fourth was the replay of recent experience. Training sampled positions from the most recent half-million games, so that the network learned from a mixture of its recent selves rather than only the latest. This damped oscillations in which a network learns to counter its immediate predecessor's habits and then forgets how to handle the habits it has just abandoned. What the system learned, and in what order, was itself a striking result. AlphaGo Zero's progression through Go knowledge recapitulated, in compressed and sometimes altered form, the human history of the game. It learned to capture stones first, then simple life and death, then standard corner sequences, joseki, that human players had developed over centuries. Some of these it later abandoned in favour of variants humans had not used. Notably, it learned the concept of a ladder, a simple capturing pattern that beginning human players are taught in their first weeks, relatively late, because recognising a ladder requires reading a long, precise diagonal sequence across the board, which is exactly the kind of thing a convolutional network finds hard and search must supply. AlphaZero: one algorithm, three games In December 2017, DeepMind posted a preprint describing AlphaZero, and the full version appeared in Science in December 2018. AlphaZero removed the last Go-specific elements of AlphaGo Zero. It did not exploit the board's rotational and reflectional symmetry, which is valid in Go but not in chess or shogi. It allowed for draws by predicting expected outcome rather than win probability. It dropped the evaluation gate. Otherwise it was the same algorithm, run separately for each game, with the same hyperparameters, apart from a noise setting scaled to the typical number of legal moves. The single game-dependent setting is revealing in its own right. The Dirichlet noise added at the root of each search was controlled by a parameter that determines how concentrated the noise is: small values put most of the noise on a few randomly chosen moves, large values spread it evenly. AlphaZero set it to 0.3 for chess, 0.15 for shogi and 0.03 for Go, scaled inversely to the typical number of legal moves in each game, while in all three games a quarter of the prior was replaced by noise. The logic is that exploration should nudge the search towards a few unconsidered moves rather than dissolving the prior entirely, and the number of moves that counts as a few depends on how many there are to choose from. That one scaling rule, together with the board representation and the rules, was the only place where knowledge of the specific game entered the algorithm. Training was correspondingly uniform. In each game, AlphaZero used 800 simulations per move during self-play, half the AlphaGo Zero figure, and the same network design, training schedule and optimiser. That the same settings worked in three games of very different character, Go strategic and territorial, chess tactical and concrete, shogi dynamic because captured pieces can be returned to the board, was perhaps the paper's most important empirical claim. It suggested that the loop was not a collection of tricks tuned for one game but a general method whose behaviour depended mostly on the simulator and the score it was given. The results were decisive. In chess, the final evaluation against Stockfish 8, the 2016 Top Chess Engine Championship winner, comprised a thousand games under tournament time controls, of which AlphaZero won 155, lost 6, and drew the rest. In shogi, it defeated the 2017 computer champion Elmo, winning 91.2 per cent of games. In Go, it defeated the three-day version of AlphaGo Zero, winning 61 per cent. Training used 5,000 first-generation TPUs to generate self-play games and 64 second-generation TPUs to train the networks. In chess, AlphaZero overtook Stockfish's playing strength within a few hours of starting from random play. The chess result was the most startling to outsiders, because chess engines had been superhuman for twenty years and their designers had refined alpha-beta search and handcrafted evaluation with extraordinary care. The 2017 preprint drew fair criticism of its match conditions, including a fixed one-minute-per-move time control and a modest hash-table size for Stockfish. The Science paper addressed these with longer matches under tournament conditions, against a stronger Stockfish configuration, and including games from popular human openings, with AlphaZero still clearly ahead. The contrast between the two engines illuminated what the search-and-learning loop actually contributes. Stockfish searched around 60 million positions per second; AlphaZero searched around 60 thousand. AlphaZero compensated for a thousandfold disadvantage in search speed with a vastly better evaluation and a vastly better sense of which moves deserved attention. Its play, as the grandmaster Matthew Sadler and the mathematician Natasha Regan documented in their 2019 book Game Changer, favoured long-term piece activity and initiative over material, readily sacrificing pawns for positional pressure in a manner that strong humans found more recognisably human than traditional engine play, but more ruthless. Table 1 sets out how the systems in this lineage differed in what they were given and what they achieved. Table 1. The AlphaGo lineage: what each system was given and what it achieved. System Year published Human game data Rollouts in search Headline result AlphaGo Fan 2016 Yes, KGS games Yes Beat Fan Hui 5–0 AlphaGo Lee 2016 match Yes Yes Beat Lee Sedol 4–1 AlphaGo Master 2017 Yes Yes Beat Ke Jie 3–0 AlphaGo Zero 2017 None None Beat AlphaGo Lee 100–0 AlphaZero 2018 None None Chess: +155 −6 vs Stockfish 8 Source: Silver et al., Nature 2016 and 2017; Silver et al., Science 2018. The open-source afterlife AlphaZero's code was never released, but the paper described the method in enough detail that others could reproduce it, and they did. Leela Zero, organised by the Belgian programmer Gian-Carlo Pascutto, reproduced AlphaGo Zero using computing time donated by volunteers, and reached superhuman strength. Leela Chess Zero did the same for chess and won the Top Chess Engine Championship in 2019. David Wu's KataGo, described in a 2019 paper, made a series of improvements, among them auxiliary training targets that predicted territory ownership and final score as well as the winner, which let the network learn far more from each game. Wu reported that KataGo reached the strength of the original ELF OpenGo reproduction with roughly fifty times less computation. KataGo became the standard analysis engine for professional Go players. The traditional engines responded by borrowing. In 2020 Stockfish adopted a small neural evaluation network of a type called NNUE, originally developed by Yu Nasu for computer shogi, which runs efficiently on ordinary processors within an alpha-beta search. The resulting hybrid regained the top position in computer chess. The lesson was not that alpha-beta search was obsolete but that learned evaluation was indispensable. The two families have since converged, with each borrowing the other's strengths. Researchers also turned to understanding what AlphaZero had learned. A 2022 study in the Proceedings of the National Academy of Sciences by Thomas McGrath and colleagues, including the former world chess champion Vladimir Kramnik, probed AlphaZero's network for human chess concepts such as material balance, king safety and mobility, and found that many of them were represented and could be located in the network's layers, emerging at different points in training. Kramnik had earlier collaborated with DeepMind researchers, in a 2020 study led by Nenad Tomašev, on using AlphaZero to explore chess variants with altered rules, such as chess without castling, assessing their balance and character in days rather than the decades human play would require. That is an early and modest instance of a theme of the later chapters: once a system can teach itself any game with known rules, the game becomes a design variable. Why removing the humans helped It is natural to assume that human data should help, since it encodes centuries of accumulated understanding. AlphaGo Zero's own experiments complicated that intuition. The paper compared the tabula rasa system with an identical network trained by supervised learning on human games. The supervised network was initially better at predicting human moves, as one would expect, and initially stronger. But the self-taught network overtook it in playing strength within about a day of training, despite remaining worse at predicting what humans would play. The explanation follows from the loop. Human data provides a good starting point, but it is also a ceiling on diversity. A network trained to imitate humans inherits their blind spots and their stylistic preferences, and the search it guides will explore around those preferences. A network trained only on its own search outputs has no such anchor. Its early play is terrible, but the training signal is always tailored to its current state, and its knowledge is built entirely from what actually wins under precise reading, rather than from what humans habitually do. The deeper lesson concerns the preconditions identified in the introduction. AlphaGo Zero and AlphaZero did not need human knowledge because they had something better: a perfect simulator. The rules of Go, chess and shogi specify exactly what happens after every move, cost almost nothing to compute, and produce an unambiguous score at the end. That simulator is what allows millions of games of self-play and what makes the tree search reliable, since every position the search examines is exactly the position that would really arise. Most problems people care about do not come with such a simulator. A robot does not know the physics of every object it might touch; an economy does not come with a rulebook; even a video game, from the point of view of an agent that sees only pixels, has unknown dynamics. The next step was therefore to ask whether the self-improvement loop could run with a model of the world that the agent had learned for itself. That is MuZero. CHAPTER 5 MuZero and Planning with a Learned Model AlphaZero's search depended on a simple service that the rules of chess provide: given a position and a move, return the next position. Every simulation in the tree search called on this service dozens of times. For board games it is free and exact. For almost everything else it is unavailable. An agent playing an Atari game sees pixels, not the internal state of the console. A system controlling a video encoder, a warehouse or a power grid has no rulebook it can consult to compute precisely what happens next. The obvious remedy is to learn a model of the environment, a network that predicts the next observation from the current one and an action, and to plan inside that. The obvious remedy had a poor record. Models that tried to predict future observations in full, every pixel of the next Atari frame, spent their capacity on details irrelevant to the task, and their small errors compounded rapidly when the planner chained predictions together over many steps. Before 2019, the strongest Atari agents were model-free, and the strongest board-game agents were given perfect models. Nothing did both. MuZero, described by Julian Schrittwieser and colleagues in a preprint in November 2019 and in Nature in December 2020, did both. It matched AlphaZero in Go, chess and shogi without being told the rules, and set a new state of the art across the 57 games of the Atari benchmark. The key was a change in what the model was asked to predict. Predict only what planning needs MuZero learns three functions, implemented as parts of one network. The representation function takes the observations available at the current moment, such as the board or the last few Atari frames, and encodes them into a hidden state, an internal vector with no prescribed meaning. The dynamics function takes a hidden state and an action and produces the next hidden state together with a predicted immediate reward. The prediction function takes a hidden state and outputs a policy, a distribution over actions, and a value, an estimate of future return. Planning works exactly as in AlphaZero, with one substitution. The tree search starts from the hidden state produced by the representation function. Whenever the search wants to know what happens after an action, it calls the learned dynamics function instead of the real rules. Whenever it reaches a leaf, it calls the prediction function for a prior and a value. All of the search happens inside the network's own internal space. The crucial design choice is in how the model is trained. The hidden state is never asked to reconstruct the observation. There is no loss that compares predicted pixels with real pixels, or a predicted board with the real board. Instead, the training procedure takes a real trajectory from experience, encodes the first observation, and then unrolls the dynamics function for several steps, typically five, along the actions that were actually taken. At every step of the unroll, the predicted policy is trained to match the search probabilities that the agent actually computed at that point, the predicted value to match the observed return, and the predicted reward to match the reward actually received. That is all. The effect is that the model learns only what matters for choosing actions. If the background colour of an Atari game never affects reward or value, the model has no reason to represent it. If two board positions differ in a way that does not change the best move or the outcome, the model may treat them as identical. Researchers later named this property value equivalence: a model need not be faithful to the environment in every respect, only in the respects that determine values and good policies. Christopher Grimm and colleagues formalised the principle in 2020, and it is the conceptual heart of MuZero. A faithful model of the world is expensive and fragile; a model faithful to the decisions you need to make is cheaper and more robust, because errors in irrelevant details simply never arise. MuZero kept a few connections to reality. In board games, the search was told which moves were legal at the root, though not deeper in the tree, where the network had to learn for itself that illegal moves led nowhere useful. The rewards in Atari were the real game scores. And the search, as in AlphaZero, used 800 simulations per move in board games, though only 50 in Atari, where decisions are made many times a second. The training procedure is easiest to grasp as code. The sketch below omits the search itself and many details, but it shows where each prediction gets its target. # One MuZero training example: unroll the learned model K steps def muzero_loss(net, obs, actions, search_policies, returns, rewards, K=5): state = net.represent(obs) # h: observation -> hidden state loss = 0.0 for k in range(K + 1): policy, value = net.predict(state) # f: hidden state -> policy, value loss += cross_entropy(search_policies[k], policy) loss += mse(returns[k], value) if k < K: state, reward = net.dynamics(state, actions[k]) # g: imagined step loss += mse(rewards[k], reward) return loss Two features of this loop deserve emphasis. The first is that the gradient flows back through the entire chain of imagined steps, so the representation function is shaped by what the dynamics function needs and the dynamics function by what the prediction function needs. Nothing forces the hidden state after three imagined steps to resemble the hidden state the representation function would produce from the real observation three steps later; the two are linked only through the requirement that both lead to correct predictions of policy, value and reward. The second is that the targets themselves come from search and from real experience, so the model is anchored to reality at every step of the unroll, even though it never sees reality directly after the first observation. A natural objection is that a model trained this way is opaque. AlphaZero's search tree contains real board positions that a human can inspect; MuZero's contains vectors whose meaning is defined only by their usefulness. When MuZero plays a move, there is no straightforward way to ask what it imagined would happen. An empirical study by Jinke He, Thomas Moerland, Frans Oliehoek and colleagues, presented at the European Conference on Artificial Intelligence in 2024, examined what MuZero's model actually learns. It found that the model's value equivalence held reasonably well for the policies that generated its training data but generalised poorly to evaluating policies it had not seen, and that MuZero's search works in part because the learned policy prior steers it towards actions where the model is accurate. The model, in other words, is trustworthy close to the agent's own behaviour and less so away from it, which is the pattern the Go adversaries of Chapter 7 exploit in a different form. For applications where an operator needs to understand and audit why a controller acted as it did, this opacity is a real cost, and it is one reason why deployed systems built on MuZero-like methods are usually surrounded by constraints and monitoring that do not depend on the model's internal states. What MuZero achieved, and where it was expensive In Go, MuZero slightly exceeded AlphaZero's performance despite having to learn the game's dynamics. In chess and shogi it matched AlphaZero. These are the results that most clearly demonstrate the point of value equivalence: a system that never received the rules, and never reconstructed a board, nonetheless planned as effectively as one that did. In Atari, MuZero achieved higher mean and median human-normalised scores across the 57 games than the strongest previous agent, R2D2, a sophisticated model-free method, establishing that planning with a learned model could beat the best model-free learning on its home ground. The cost was data. The headline Atari results used tens of billions of environment frames. A variant called MuZero Reanalyze reduced this considerably by revisiting old trajectories and rerunning the search on them with the latest network, producing fresher training targets from stale experience. Reanalysis turns out to be one of the most valuable techniques in the family. It recognises that search is not only a decision procedure but a label generator, and that old data can be relabelled as the labeller improves. The original MuZero also made assumptions that limited its reach. It assumed a discrete, modest set of actions, since the tree search enumerates every action at every node. It assumed deterministic dynamics, since the learned model produced a single next state. And it assumed enough computation for dozens or hundreds of simulations per decision. Each of these limitations was taken up by a successor. The successors: continuous, stochastic, cheap, sample-efficient Sampled MuZero, published by Thomas Hubert and colleagues in 2021, extended the method to large and continuous action spaces, such as the joint torques of a simulated robot, where enumerating every action is impossible. The search considers only a small sample of actions at each node, drawn from the network's policy, and the paper showed how to correct the resulting improvement step so that learning remains sound. The method performed well on standard continuous-control benchmarks and on Go with sampled moves, demonstrating that the planning-and-distillation loop did not depend on being able to list every option. Stochastic MuZero, presented by Ioannis Antonoglou and colleagues at the International Conference on Learning Representations in 2022, addressed randomness. A deterministic model cannot properly represent a dice roll: if it tries, it predicts an average over outcomes, which corresponds to no real state. Stochastic MuZero splits each transition into two stages, first an afterstate representing the result of the agent's action, then a chance outcome drawn from a learned discrete distribution of possible random events. The search treats chance nodes differently from decision nodes, averaging rather than maximising. The method matched or exceeded strong existing programs in backgammon, the game where TD-Gammon began this story, and in the sliding-tile puzzle 2048. Gumbel MuZero, by Ivo Danihelka and colleagues at the same conference, tackled the cost of search. The classic PUCT selection rule, inherited from AlphaGo, behaves poorly when the simulation budget is small. With only a handful of simulations, it may fail to consider some actions at all, and the resulting visit-count policy can actually be worse than the network's prior, so that training on it makes the network worse. Danihelka's team replaced the root selection with a procedure based on sampling actions without replacement using the Gumbel-max trick, then allocating simulations by sequential halving, a bandit algorithm that repeatedly discards the weaker half of candidates. They proved that the resulting policy improvement step is guaranteed to improve on the prior in expectation, and showed that in Go and chess the method continued to learn effectively with as few as two simulations per move, where the original algorithm broke down. Gumbel-based search has since become a common default in open-source reimplementations precisely because most practical applications cannot afford hundreds of simulations per decision. The most dramatic improvement in sample efficiency came from outside DeepMind. EfficientZero, by Weirui Ye and colleagues at Tsinghua University and the University of California, Berkeley, presented at NeurIPS in 2021, targeted the Atari 100k benchmark, which allows only 100,000 agent steps per game, roughly two hours of real-time play. EfficientZero added three modifications to MuZero: a self-supervised consistency loss encouraging the predicted hidden state after an action to match the encoding of the actual next observation, a model that predicts the sum of rewards over a short horizon rather than individual rewards, sidestepping the difficulty of pinning down exactly when a reward will arrive, and corrections for off-policy targets in reanalysis. It reported a mean human-normalised score of 194.3 per cent and a median of 109 per cent on the benchmark, exceeding human-level performance with the same two hours of experience a human tester would have. The consistency loss is a partial retreat from MuZero's purist position, reintroducing some grounding in observations, and its success suggests that pure value equivalence is the right objective but not always the most efficient way to learn when data is scarce. Planning with learned models was also pursued along a different branch. Danijar Hafner's Dreamer family learns a world model that does reconstruct observations and trains a policy entirely inside imagined trajectories, without tree search. DreamerV3, published in Nature in 2025, used a single configuration across a wide range of domains and was the first algorithm to collect diamonds in Minecraft from scratch, without human data or curricula. The difference between MuZero and Dreamer is instructive. MuZero uses its model for lookahead search at decision time and distils the search; Dreamer uses its model to generate synthetic experience on which to train a policy. Both demonstrate that learned models had become good enough to plan in, but they place the model in different parts of the self-improvement loop. MuZero leaves the arcade The first production use of MuZero was announced by DeepMind in February 2022. YouTube encodes uploaded videos with codecs, among them the open-source VP9, that must decide how many bits to spend on each frame. Spending more on a frame improves its quality but leaves fewer bits for the rest of the video, and frames that later frames refer to deserve more bits than frames that nothing references. The codec's rate controller makes this allocation with handcrafted heuristics refined over many years. DeepMind's team, in work described in a 2022 preprint by Amol Mandhane and colleagues, framed rate control as a sequential decision problem in which the agent chooses a quantisation parameter for each frame, subject to a target total size. The difficulty is that there is no natural reward: the goal is to match a bitrate budget while maximising quality, and these trade off in ways that vary across videos. The team's solution was a mechanism they called self-competition, in which the agent's reward for a video is defined by whether it beats its own historical performance on that video, converting a constrained optimisation into something closer to a game with a win condition. MuZero then learned to play that game. On a large, diverse set of videos, it achieved an average bitrate reduction of 4 per cent at equal quality, a significant figure at YouTube's scale, and the system was deployed on a portion of YouTube's live traffic. The episode is worth noticing for what it reveals about transfer. The codec was a perfect simulator in the sense that mattered: given a video and a sequence of rate decisions, it produces an exact encoded output and an exact quality measure. The hard part was not the model but the score. The team had to invent a reward that made the problem behave like a game, and self-competition did so by turning an absolute target into a relative one. That pattern, finding a way to give a real problem the scoring structure of a game, will recur in every application chapter that follows. Self-competition also handled a second difficulty: videos differ enormously in how hard they are to compress, so an absolute quality score would reward the agent mainly for being given easy videos. Measuring each episode against the agent's own past performance on the same video removes that variation, in the same way that a baseline in policy-gradient methods removes variation that has nothing to do with the action taken. MuZero also completed the conceptual arc that began with TD-Gammon. Tesauro's program had a fixed model, the dice and the rules, and a learned value. AlphaGo Zero had a fixed model and learned both policy and value, using search to improve them. MuZero learned the model as well, using exactly the training signals that drive search. All three ingredients of planning were now learned from experience, and the only things the agent needed from outside were observations, actions and a reward. What it still needed, in quantity, was search, and search requires a decision tree small enough to explore. For games whose action spaces are vast and whose episodes span tens of thousands of decisions, such as real-time strategy and multiplayer online battle games, tree search was impractical even with a learned model. Those games were conquered by the other great tradition of reinforcement learning, which improves policies directly from experience without looking ahead. Its most important algorithm is PPO. CHAPTER 6 Policy Gradients and Proximal Policy Optimization The systems of the previous three chapters improve their behaviour by looking ahead. At every decision they build a tree, and the tree's verdict is both the action and the lesson. That approach is extraordinarily data-efficient per decision, since every search squeezes a refined target out of a single position, but it presupposes three things: a small enough set of actions to enumerate, a short enough horizon to search meaningfully, and the time to run the search before acting. Many problems offer none of these. In Dota 2, a five-on-five online battle game, each player controls a hero in continuous time, choosing among thousands of possible action combinations every fraction of a second, over games lasting forty-five minutes or more. A robot hand rotating a cube chooses joint torques from a continuum, many times a second. A language model writing an answer chooses each word from a vocabulary of tens of thousands of tokens, across responses hundreds or thousands of tokens long. Tree search over any of these is either impossible or hopelessly shallow. For such problems, the field relies on a different family of methods, which improve the policy directly from sampled experience, without lookahead. This chapter explains how they work, why the most important of them, proximal policy optimization, became the default algorithm of applied reinforcement learning, and how it moved from games into the training of language models. The policy gradient The idea behind policy gradient methods is the most direct possible response to reward. Represent the policy as a network that outputs a probability for each action. Let the agent act. If an episode turns out well, increase the probability of the actions taken in it; if it turns out badly, decrease them. Do this with gradient ascent on expected return. Ronald Williams gave this idea its classic form in 1992, in an algorithm he named REINFORCE. Its update multiplies the gradient of the log-probability of each chosen action by the return that followed it. Actions followed by high return are reinforced; actions followed by low return are suppressed. The policy gradient theorem, proved in general form by Richard Sutton and colleagues in 2000, established that this is an unbiased estimate of the true gradient of expected return, even though the agent never differentiates through the environment. It need not know how the world works, only what it did and how things turned out. REINFORCE is correct and almost useless in its raw form, because its gradient estimates are extremely noisy. Every action in a winning episode is reinforced, including the bad ones that the agent happened to survive. The variance of the estimate grows with the length of the episode, and for long tasks the signal is drowned out. Two refinements address this, and every modern method uses both. The first is a baseline. Instead of reinforcing an action by the raw return, reinforce it by how much better the return was than expected. Subtracting an expectation leaves the gradient unbiased but can reduce its variance enormously, since an action that led to an ordinary outcome now receives almost no push in either direction. The natural baseline is a learned estimate of the state's value. The difference between the actual return and that value is called the advantage, and it answers exactly the right question: was this action better or worse than what the policy usually does here? The second refinement is to replace the actual return with a bootstrapped estimate, using the value function to stand in for the uncertain future, just as temporal-difference learning does. The resulting methods have two components: an actor, the policy, and a critic, the value function that judges the actor's choices. The two are often parts of one network. Actor-critic methods trade a little bias, since the critic may be wrong, for a large reduction in variance. John Schulman and colleagues' generalised advantage estimation, published in 2016, provided a single parameter that slides smoothly between pure Monte Carlo returns and one-step bootstrapped estimates, letting practitioners choose where on that trade-off to sit. In 2016 DeepMind's A3C, the asynchronous advantage actor-critic of Volodymyr Mnih and colleagues, showed that many actor-learners running in parallel on ordinary processors could train strong Atari agents without a replay buffer, making actor-critic methods practical at scale. Trust regions and the clipping trick Actor-critic methods remained temperamental. The core problem is the size of the update. A policy gradient tells you which direction improves the policy locally, but says nothing about how far you can safely move. Too small a step wastes data. Too large a step can change the policy so much that it collapses: it stops visiting the states where it was competent, the data it collects becomes useless, and performance falls off a cliff from which it may never recover. Because the data in reinforcement learning is generated by the policy itself, a bad update corrupts the very data that would correct it. Schulman, Sergey Levine, Pieter Abbeel and colleagues proposed trust region policy optimization, TRPO, in 2015. TRPO maximises a surrogate objective, an estimate of how much the new policy would improve on the old using data collected under the old one, subject to a hard constraint on how different the new policy may be, measured by the Kullback-Leibler divergence between the two action distributions. Within that trust region, the surrogate is a reliable guide; outside it, it is not. TRPO came with a theoretical guarantee of monotonic improvement under idealised conditions, and it worked well in practice, but it required second-order optimisation machinery that was complicated to implement and awkward to combine with standard network architectures. In July 2017, Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford and Oleg Klimov at OpenAI posted a short paper titled "Proximal Policy Optimization Algorithms." PPO keeps TRPO's insight, that the new policy should stay close to the old one, but enforces it with a far simpler device. For each action in the collected data, compute the ratio between its probability under the new policy and under the old. Multiply that ratio by the action's advantage to get the surrogate objective. Then clip the ratio to a narrow band around one, typically between 0.8 and 1.2, and take the smaller of the clipped and unclipped values. # PPO's clipped surrogate loss (to be minimised) def ppo_policy_loss(new_logp, old_logp, advantages, eps=0.2): ratio = torch.exp(new_logp - old_logp) unclipped = ratio * advantages clipped = torch.clamp(ratio, 1 - eps, 1 + eps) * advantages return -torch.min(unclipped, clipped).mean() The effect of taking the minimum is that the objective stops rewarding further movement once the policy has moved far enough in the favourable direction. If an action had positive advantage, the incentive to increase its probability vanishes once the ratio exceeds 1.2. If it had negative advantage, the incentive to decrease it vanishes once the ratio drops below 0.8. But movements in the unfavourable direction are never clipped, so the objective still penalises the policy for making a good action less likely or a bad action more likely. The result is a pessimistic bound on improvement that removes the incentive to take large steps, without any constraint optimisation at all. Because the clipping keeps updates safe, PPO can reuse each batch of experience for several epochs of minibatch gradient descent, extracting more learning from the same data than a single gradient step would. It runs with any network architecture and any first-order optimiser. On the continuous-control and Atari benchmarks of its day, it matched or beat TRPO and A3C while being dramatically easier to implement and tune. It was, in Schulman's framing, an algorithm that traded a little theoretical purity for a large gain in practicality, and practitioners took the trade. The relationship between PPO's description and its behaviour is less tidy than its elegance suggests. A 2020 study by Logan Engstrom and colleagues found that much of PPO's advantage over TRPO in standard benchmarks came from code-level optimisations, among them reward normalisation, value-function clipping, learning-rate annealing and particular network initialisations, rather than from the clipped objective itself. Shengyi Huang and colleagues later catalogued thirty-seven implementation details that materially affect PPO's performance. The algorithm is robust in the sense that a careful implementation works across a wide range of problems; it is not robust in the sense that any implementation will. Table 2 places PPO among the main policy-gradient methods that preceded and followed it. Table 2. Major policy-gradient algorithms compared. Algorithm Year Variance reduction How update size is controlled Notable use REINFORCE 1992 Optional baseline Learning rate only Teaching and theory TRPO 2015 Learned value critic Hard KL-divergence constraint Continuous control A3C 2016 Learned value critic Learning rate only Parallel Atari training PPO 2017 Critic with GAE Clipped probability ratio OpenAI Five, RLHF GRPO 2024 Group mean of sampled rewards Clipped ratio plus KL penalty DeepSeek reasoning models Source: Williams 1992; Schulman et al. 2015 and 2017; Mnih et al. 2016; Shao et al. 2024. OpenAI Five: scale as a substitute for search The most ambitious demonstration of PPO in games was OpenAI Five, a team of five neural networks, one per hero, that played Dota 2. Dota is a hard environment in almost every dimension. It is played in continuous time, with games lasting tens of thousands of decision steps at the agent's sampling rate. Each player sees only part of the map. The observation at each step comprised roughly sixteen thousand values describing units, buildings and the map, and the space of possible actions at each step was in the thousands, depending on how they are counted. Five agents had to coordinate, and the reward of a win arrived at the very end. OpenAI's approach, described in the 2019 paper by Christopher Berner and colleagues, was in algorithmic terms almost conservative. Each hero was controlled by a large recurrent network, with a long short-term memory layer of 4,096 units at its core, trained by PPO with generalised advantage estimation. There was no tree search and no learned model. The agents learned through self-play, with about 80 per cent of games played against the current version and 20 per cent against earlier versions to prevent strategy collapse. Rewards were shaped with intermediate signals for things like kills, gold and destroyed buildings, to give the policy gradient something to work with long before the end of a game, and a "team spirit" parameter gradually shifted each agent's reward from its own performance towards the team's. What the system had instead of search was scale. Over ten months of training, OpenAI Five played the equivalent of about 45,000 years of Dota against itself, running on thousands of GPUs and tens of thousands of processor cores. The team also developed a technique it called surgery, for transferring trained parameters into a modified network when the architecture or game version changed, which allowed training to continue across Dota updates without restarting from scratch. In April 2019 OpenAI Five defeated OG, the reigning world champion team, two games to nothing. In a subsequent public online event it played more than seven thousand games against human teams and won 99.4 per cent of them. The system had real constraints: it played with a restricted pool of 17 of the game's more than one hundred heroes, and some game mechanics were simplified or disabled. Within those limits, the result showed that a simple policy-gradient algorithm, applied at sufficient scale with careful engineering and shaped rewards, could reach superhuman play in a domain where tree search could not even be attempted. The price was a thousandfold or more increase in experience relative to what a search-based method would need in a domain where search was possible. PPO buys generality with data. From games to language The most consequential use of PPO turned out not to be in games at all. In 2017 Paul Christiano and colleagues at OpenAI and DeepMind showed that an agent could learn complex behaviours, such as performing a backflip in simulation, from human judgements comparing pairs of short video clips. The comparisons trained a reward model, a network predicting which behaviour a human would prefer, and the agent was then trained by reinforcement learning to maximise the learned reward. Subsequent work applied the same recipe to language, first to stylistic continuation and then, in Nisan Stiennon and colleagues' 2020 study, to summarisation, where the method produced summaries that human raters preferred to those of much larger models trained only by supervised learning. The 2022 InstructGPT paper by Long Ouyang and colleagues turned the recipe into the standard method for making language models follow instructions. A pretrained model was first fine-tuned on human-written demonstrations, then a reward model was trained on human rankings of the model's outputs, and finally the model was optimised with PPO against the reward model, with a penalty on the divergence from the original model to prevent it from drifting into text that fooled the reward model without being good. Human evaluators preferred the outputs of a 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3 it started from. This reinforcement learning from human feedback, RLHF, became the final stage of training for ChatGPT and, in various forms, for most widely used assistants. In this setting, the game is a single response. The state is the prompt and the text so far; an action is the next token; the episode ends when the response does; the reward is the reward model's score. The KL penalty plays a role very similar to PPO's clipping, keeping the policy close to a trusted reference. The analogy is close enough that the same algorithm transferred with little modification, and the difficulties transferred too. A learned reward model is a fallible score, and policies optimised hard against it learn to exploit its errors, producing responses that are longer, more confident or more flattering than they should be. Leo Gao and colleagues at OpenAI documented in 2022 how optimising against a proxy reward model first improves and then degrades true quality as optimisation pressure increases, a quantitative version of the boat-racing agent from Chapter 1. Partly for this reason, and partly for cost, simpler alternatives emerged, notably direct preference optimization, introduced by Rafael Rafailov and colleagues in 2023, which fits the preference data directly without a separate reward model or reinforcement learning loop. Reinforcement learning returned in force in 2024 and 2025, with a different kind of score. OpenAI's o1 models, announced in September 2024, were trained with large-scale reinforcement learning to produce long chains of reasoning before answering. The details were not published. DeepSeek's work, which was, made the method public. Its 2024 DeepSeekMath paper by Zhihong Shao and colleagues introduced group relative policy optimization, GRPO, a PPO variant that dispenses with the critic. For each prompt the model samples a group of responses, scores them, and uses each response's reward relative to the group's mean as its advantage. That substitutes a cheap empirical baseline for an expensive learned one, which matters when the policy is a model with tens or hundreds of billions of parameters. The DeepSeek-R1 report of January 2025, published in revised form in Nature in September 2025, applied GRPO with rewards from automatic verifiers, checking whether a mathematical answer matched the known solution or whether code passed its tests. In the R1-Zero experiment, reinforcement learning applied directly to a base model without any supervised fine-tuning raised its pass rate on the 2024 American Invitational Mathematics Examination from 15.6 per cent to 71.0 per cent, and the model spontaneously learned to produce longer reasoning, check its own work and revise mistaken approaches. This is the game-playing paradigm recognisably at work. Verifiable rewards play the role that the rules of Go played for AlphaGo Zero: an exact, cheap, unambiguous score that cannot be fooled by fluent-looking nonsense. Where such a score exists, as in mathematics with known answers and programming with test suites, reinforcement learning has produced large and rapid gains. Where it does not, as in open-ended writing or advice, the method remains dependent on learned reward models with all their exploitable flaws. The boundary between those two kinds of task is, at present, one of the most important lines in applied machine learning, and it is exactly the line this book has been drawing since its introduction: the self-improvement loop needs a trustworthy score. PPO and its variants occupy a particular niche in that picture. They need no model and no search, only the ability to sample experience and score it. In exchange, they need far more experience than search-based methods, and they improve only gradually, by small safe steps. They suit problems where simulation is cheap and plentiful but the space of actions is too large or too continuous to search. The next chapter turns to games that strain both families of methods, because they hide information, involve many interacting players, or reward exploiting the opponent's weaknesses. CHAPTER 7 Hidden Information, Many Players and Brittle Champions Go, chess and shogi share three properties that made them ideal for the self-improvement loop. Both players see everything. There are exactly two players, and one's gain is the other's loss. And the best move in a position does not depend on who the opponent is: a perfect player could simply play the theoretically correct move every time, without modelling anyone. Under those conditions, beating your previous self reliably means getting better in an absolute sense, and tree search over the true game state is sound. Most strategic situations of real interest violate at least one of these. In poker, cards are hidden. In StarCraft, much of the map is hidden and there is no clean notion of a best move, only strategies that beat some opponents and lose to others. In Diplomacy, seven players negotiate in natural language, form alliances and betray them. Each of these breaks the loop in a different place, and the systems that conquered them, AlphaStar, Libratus and Pluribus, and Cicero, each had to rebuild part of it. This chapter follows those repairs, and then turns to a discovery from 2022 and 2023 that bears on every system in this book: that superhuman Go programs, the purest products of the loop, could be beaten consistently by an adversary exploiting a blind spot that no amount of self-play had found. Poker: when the state is a belief The difficulty with hidden information is not merely that the agent lacks some facts. It is that the value of an action depends on what the opponent believes, which in turn depends on the agent's own strategy. In chess, if a move is good, it is good regardless of how often you would play it in that position. In poker, a bet with a weak hand, a bluff, is only profitable if you sometimes bet with strong hands in the same way, so the opponent cannot tell them apart. If you always bluff, you lose; if you never bluff, your strong bets are too readable to be paid off. The right action is therefore not a single move but a probability distribution, and the value of a particular situation cannot be computed in isolation from the strategy that led there. This breaks Monte Carlo Tree Search in a fundamental way. The search assumes that each node has a well-defined value independent of how the rest of the tree is played. In poker, a node corresponds not to a single state but to an information set, the collection of all states the player cannot distinguish, such as all the possible hands the opponent might hold, weighted by how likely each is given the betting so far. Those weights depend on the opponent's strategy, which depends on yours. Searching a subgame in isolation, as chess engines do, can produce strategies that are easily exploited. The method that cracked poker comes from game theory rather than from search. Counterfactual regret minimisation, introduced by Martin Zinkevich, Michael Bowling and colleagues at the University of Alberta in 2007, has an agent play the game against itself repeatedly and track, at every information set, how much it regrets not having played each action, meaning how much better it would have done had it chosen that action there. The policy is then shifted towards actions with positive regret. The average strategy over all iterations provably converges towards a Nash equilibrium, a strategy pair in which neither player can improve by deviating. In 2015 Bowling's group used a refined version to essentially solve heads-up limit Texas hold'em, producing a strategy so close to equilibrium that a human lifetime of play could not distinguish it from perfect. No-limit hold'em, in which players may bet any amount, has a vastly larger game tree. In January 2017, Libratus, built by Noam Brown and Tuomas Sandholm at Carnegie Mellon University, played 120,000 hands of heads-up no-limit hold'em against four top professionals over twenty days and defeated them by a wide, statistically significant margin. The same year, DeepStack, developed by Matej Moravčík, Bowling and colleagues, reported a similar result against professional players. Both combined a precomputed coarse strategy with real-time search that solved subgames as they arose, using techniques that kept the refined subgame strategy consistent with the overall equilibrium so that it could not be exploited. DeepStack additionally used deep networks to estimate the values of subgames, a role analogous to AlphaGo's value network. In 2019 Brown and Sandholm's Pluribus took on six-player no-limit hold'em, where game theory's guarantees no longer apply cleanly, since in games with more than two players a Nash equilibrium need not be unbeatable and may not even be well defined as a target. Pluribus computed a blueprint strategy by self-play with a form of counterfactual regret minimisation in eight days on a single 64-core server, at a cloud-computing cost the authors put at around 150 dollars, and refined it during play with depth-limited search that assumed opponents might switch among several different continuation strategies. Over 10,000 hands against elite professionals it won at a rate of about 48 milli-big-blinds per game, a decisive margin by the standards of professional poker. The poker programs did not grow out of AlphaZero, and their mathematical foundations differ. But they contain the same loop in a different guise: self-play generates experience, an improvement operator, in this case regret minimisation rather than tree search, makes the strategy better, and real-time search refines the stored strategy at decision time. Later work explicitly unified the two traditions. Brown's ReBeL, in 2020, recast imperfect-information games so that the state became a public belief state, a probability distribution over hidden information that both players share, and then ran an AlphaZero-like loop of search and learning over those belief states. DeepMind's Student of Games, published in 2023, applied a single algorithm combining guided search, self-play and game-theoretic reasoning to chess, Go, poker and the hidden-information board game Scotland Yard. DeepNash, also from DeepMind and published in Science in 2022, reached the top three of an online leaderboard of human experts in Stratego, a board game with hidden piece identities and a game tree too large for search, using a model-free reinforcement learning method designed to converge to equilibrium rather than to chase the current opponent. StarCraft and the league StarCraft II, a real-time strategy game, combines hidden information with a scale of action and time that rules out tree search. Players gather resources, build bases and armies, and fight across a map most of which they cannot see, over games lasting tens of minutes with hundreds of actions per minute. There is no dominant strategy: an early aggressive rush beats a greedy economic build, the greedy build beats a defensive turtle, and the turtle beats the rush. Games of this kind are intransitive, and that is what breaks naive self-play. An agent trained only against its latest self tends to chase its own tail, learning to beat the current strategy, then the counter to that, then the counter to the counter, and forgetting earlier lessons along the way. DeepMind's AlphaStar, described in Nature in October 2019 by Oriol Vinyals and colleagues, addressed intransitivity with a structure it called the league. Rather than a single agent playing itself, the league maintained a population of agents with different roles. Main agents aimed to become strong against everything, playing against the whole league with emphasis on opponents they found difficult, a method the team called prioritised fictitious self-play. Main exploiters trained specifically to find and punish the weaknesses of the current main agents. League exploiters searched for strategies that beat the population as a whole. Exploiters were periodically reset, and snapshots of all agents were frozen and kept as opponents. The effect was to maintain diversity, so that the main agents were forced to become robust to a wide range of strategies rather than overfitting to one. AlphaStar departed from the tabula rasa approach in an important respect: its agents were first trained by supervised learning on human replays, and the human data continued to shape training through a penalty that discouraged drifting too far from human statistics. Without it, the team found, exploration in a game this large did not discover the basic strategies at all. The networks combined a transformer over game units, a deep recurrent core and an autoregressive action head that built each action in stages: what to do, which units to command, where to target. Policy learning used an off-policy actor-critic method with several corrections suited to learning from a league of other agents' experience. An earlier version had made its public debut in January 2019, in a series of games against two professional players, Dario "TLO" Wünsch and Grzegorz "MaNa" Komincz, winning ten games to nothing. That version could observe the whole visible map at once rather than moving a camera as humans must. In a final live exhibition game, a new version restricted to a camera view lost to MaNa, who found a way to exploit its handling of a repeated harassment manoeuvre. The episode was a small preview of the adversarial findings later in this chapter: a system with superhuman average play could still be led into a pattern of behaviour it did not handle. The constraints were chosen to make comparison with humans fair. AlphaStar observed the game through a camera-like view rather than the whole map, and its rate of actions was limited, with delays modelled on human reaction times. Under these conditions, playing anonymously on the official European Battle.net ladder, it reached Grandmaster level for all three of the game's races and ranked above 99.8 per cent of officially ranked human players. The constraints were contested by some professional players, who argued that its precise control of individual units in bursts still exceeded what humans could do, a reminder that fairness in such comparisons is a design choice. Diplomacy: games where you must talk Diplomacy is a board game for seven players set in pre-1914 Europe. Its mechanics are simple: all players write orders for their armies and fleets, which resolve simultaneously. Its substance is negotiation. No single player can win without alliances, and alliances must be built through private conversation, maintained by trust, and occasionally broken. An agent that plays the board perfectly but cannot persuade anyone will lose. Meta's Cicero, described in Science in November 2022 by the Fundamental AI Research Diplomacy team, played the full natural-language version. It coupled a strategic reasoning module with a 2.7-billion-parameter language model. The strategic module predicted what each other player was likely to do, planned its own moves, and decided what intentions to communicate; the language model generated messages grounded in those intentions, and a series of filters screened out messages that were nonsensical or inconsistent with its plans. The strategic reasoning illustrates a refinement of self-play that is essential when cooperating with humans. Pure self-play in Diplomacy produces agents that play well with copies of themselves but adopt conventions humans do not share, which makes them poor allies. Cicero's planning used a method its developers called piKL, which searches for a strategy that improves on a model of human play while penalising divergence from it, a regularisation closely analogous to the KL penalty in RLHF. The agent learns to play better than humans without becoming unrecognisable to them. In 40 games on the webDiplomacy.net platform, played anonymously against humans, Cicero achieved more than double the average human score and ranked in the top 10 per cent of participants who played more than one game. The Cicero team reported that it was designed to be largely honest, sending messages that reflected its actual intentions, partly because consistent honesty made it a more trusted and therefore more effective ally. Later analysis of game transcripts by other researchers documented instances in which it nonetheless broke agreements, a sign of how hard it is to pin down behaviour in a system optimised for winning in a social setting. Table 3 summarises the main systems in this chapter and the device each used to cope with its game's departure from the board-game ideal. Table 3. Systems for games beyond two-player perfect information. System Game Main obstacle Key device Reported result Libratus (2017) Two-player poker Hidden cards Safe subgame solving Beat four top pros AlphaStar (2019) StarCraft II Intransitive play League of exploiters Above 99.8% of players Pluribus (2019) Six-player poker Many players Depth-limited search 48 mbb/game vs pros DeepNash (2022) Stratego Hidden pieces Equilibrium-seeking RL Top three on Gravon Cicero (2022) Diplomacy Negotiation Human-anchored planning Top 10% of players Source: Brown and Sandholm, Science 2018 and 2019; Vinyals et al., Nature 2019; Perolat et al., Science 2022; Meta FAIR, Science 2022. Superhuman, and still beatable If any game-playing systems could be called solved, it would be the Go engines descended from AlphaGo Zero. KataGo, the strongest freely available engine, is far beyond the best human professionals, who now use it as a teacher. In late 2022, Tony Wang, Adam Gleave and colleagues at the research group FAR AI and several universities reported, in a paper later presented at the International Conference on Machine Learning in 2023, that they had trained adversarial agents that beat KataGo consistently. The adversaries were not strong Go players. A competent amateur could beat them easily. What they had learned was a specific trap. The most effective, which the researchers called the cyclic attack, slowly builds a loop-shaped group of the victim's stones surrounded by the adversary's stones, in a configuration that KataGo's network misjudges, failing to see that its own group is in danger until it is captured. Against KataGo at a level of search the researchers described as superhuman, the adversary won 96 per cent of games; against KataGo with ten million search visits per move, far beyond normal settings, it still won 72 per cent. Training the adversary took about 14 per cent of the computation used to train KataGo. And the attack was learnable by humans: Kellin Pelrine, an author of the study and a strong amateur rather than a professional, used the strategy to beat superhuman versions of KataGo and Leela Zero repeatedly, without computer assistance during the games. A follow-up study by Tom Tseng and colleagues, published at the AAAI conference in 2025, tested three natural defences: adversarial training on hand-constructed positions containing the cyclic pattern, iterated adversarial training in which the victim and fresh adversaries take turns, and switching the network to a vision-transformer architecture. Some of these defences protected against the attacks already discovered. None withstood a freshly trained adversary, and most of the successful new attacks were variations on the same cyclic theme. The result illuminates the loop's central limitation. Self-play learns to beat the opponents self-play produces. Positions that self-play never generates, because neither copy of the agent would ever steer towards them, are positions the value network has never been trained on, and its judgements there may be arbitrarily wrong. Search corrects many errors of the network but not this kind, because search uses the network's evaluations at its leaves; if the network is confidently wrong about a whole class of positions, deeper search simply explores more positions it is wrong about. The cyclic group is exactly such a class: a configuration that strong play never produces, that a deliberately adversarial opponent can produce, and that the network has no reason to understand. This is not a quirk of Go. It is the shape of the problem whenever a system trained in simulation meets an environment that includes adversaries, or simply circumstances that its training distribution did not cover. AlphaStar's league was in part an answer to it, and the poker programs' equilibrium reasoning was another, but neither scales to arbitrary settings. When the next two chapters carry these methods into logistics and biology, the question of what lies outside the training distribution is never far away. A protein designed by search is judged by a model that may be wrong about unusual sequences in exactly the way KataGo is wrong about unusual groups, and the laboratory is the only test that cannot be fooled. CHAPTER 8 From Boards to Logistics: Optimisation as a Game A large share of the hard problems in industry are puzzles in disguise. Routing a fleet of delivery vans is a matter of choosing, one stop at a time, an order that minimises total distance subject to time windows and capacities. Placing components on a chip means choosing, one block at a time, positions that minimise wire length and congestion. Writing a fast sorting routine means choosing, one instruction at a time, a program that is correct and as short as possible. Each of these is a sequence of discrete choices, built step by step, judged at the end by a number. Each, in other words, has the formal structure of a single-player game. This observation is old. Operations research has studied these problems for seventy years and developed formidable tools for them: integer programming, branch and bound, local search, simulated annealing, and specialised heuristics refined over decades. What changed after AlphaZero was the possibility that a learned search could discover, from scratch, solutions or solution strategies that those tools had missed. This chapter examines where that possibility has been realised, where it has been claimed and contested, and what distinguishes the two. The pattern that emerges is consistent with the argument of this book. The successes come where the problem can be turned into a game with an exact, cheap simulator and an unambiguous score, and where the space of solutions has not already been exhaustively mined by human experts with better tools. The disappointments come where the score is a proxy for what really matters, where the benchmark comparisons are against weak baselines, or where decades of specialised algorithms already sit close to the frontier. Algorithm discovery: AlphaTensor and AlphaDev The cleanest examples of game-playing methods producing genuinely new results come from mathematics and computer science, where the score can be computed exactly and correctness can be verified. Matrix multiplication is the arithmetic core of nearly all scientific computing and machine learning. The schoolbook method for multiplying two n-by-n matrices uses n cubed scalar multiplications, so 8 for two-by-two matrices. In 1969 Volker Strassen found a way to do the two-by-two case with 7, and because the trick applies recursively to blocks of larger matrices, it reduced the asymptotic cost of matrix multiplication for the first time. For more than fifty years afterwards, the best known method for four-by-four matrices was to apply Strassen's algorithm twice, using 49 multiplications. Every such algorithm corresponds to a way of decomposing a particular three-dimensional array of numbers, the matrix multiplication tensor, into a sum of simple pieces, and the number of pieces equals the number of multiplications. AlphaTensor, described by Alhussein Fawzi and colleagues at DeepMind in Nature in October 2022, recast that decomposition as a single-player game it called TensorGame. The state is the tensor remaining to be decomposed. Each move subtracts one simple piece, a rank-one tensor, from it. The game ends when the remainder reaches zero, and the player is penalised for every move taken, so the shortest successful game corresponds to the algorithm with fewest multiplications. The game is extraordinarily hard: the number of possible moves at each step is larger than ten to the power of twelve for interesting cases, far beyond Go. AlphaTensor handled this with an AlphaZero-style loop using sampled actions, in the manner of Sampled MuZero, a transformer-based network with symmetries built into its architecture, and a supply of synthetic training data generated by constructing random decompositions and asking the network to recover them. It found an algorithm for four-by-four matrices in arithmetic modulo two, the arithmetic used in some coding and cryptography applications, with 47 multiplications, the first improvement on Strassen's two-level method in that setting. In standard arithmetic it improved the best known counts for several rectangular sizes, for example reducing the multiplication of four-by-five by five-by-five matrices from 80 multiplications to 76. By including measured running time on specific hardware in the reward, it also found algorithms that ran faster in practice on particular GPUs and TPUs than standard methods for large matrices. The results quickly prompted human follow-up of exactly the kind that game-playing systems had earlier prompted in Go. Within days of publication, the mathematicians Manuel Kauers and Jakob Moosbauer used a different, non-learned search method to improve one of AlphaTensor's records, reducing five-by-five multiplication modulo two from 96 to 95. In May 2025 DeepMind's AlphaEvolve, a different kind of system that uses large language models to propose and evolve programs scored by automated evaluators, found a way to multiply four-by-four complex-valued matrices with 48 multiplications. The frontier moved because the problem had been made into a game that many players, human and machine, could now play. AlphaDev, published by Daniel Mankowitz and colleagues in Nature in June 2023, applied the same approach one level lower. Its game, AssemblyGame, has the agent write a program one assembly instruction at a time. The reward combines correctness, checked by running the program on test inputs, and speed, measured either by instruction count or by actual latency on hardware. The targets were small routines that sort three, four or five elements, called billions of times inside larger sorting routines. These had been hand-optimised by expert programmers for years, and the space of short assembly programs had been searched by superoptimisers. AlphaDev nonetheless found shorter sequences, including what its authors called the swap move and the copy move, small instruction-saving tricks that, in context, eliminated a step. The new routines were incorporated into the LLVM project's libc++ standard C++ sorting library, with improvements of up to 70 per cent for short sequences and about 1.7 per cent for sequences of more than 250,000 elements. A hashing routine AlphaDev discovered was added to Google's open-source Abseil library, improving speed by up to 30 per cent on certain input sizes. Both systems succeeded for the same reasons. The game had an exact simulator, since a tensor decomposition or a program can be checked precisely. The score could not be gamed, since an incorrect algorithm simply fails. The solutions were short enough for search to construct them step by step, and the problems were narrow enough that a small improvement had real value. Neither system was a general-purpose optimiser; each was a careful translation of a specific problem into a specific game. Placement, routing and dispatch The application that drew the most attention, and the most dispute, was chip floorplanning. In a paper published in Nature in June 2021, Azalia Mirhoseini, Anna Goldie and colleagues at Google described a method for macro placement, deciding where on a chip to put the large blocks, such as memory arrays, whose positions largely determine the quality of the rest of the layout. They framed it as a game in which an agent places one macro per move, with a graph neural network encoding the circuit's connectivity and a policy trained by PPO. The reward was a weighted combination of proxy measures computed at the end, estimated wire length, congestion and density, since a full evaluation of power, performance and area takes hours or days per layout. The paper reported that the method produced placements in under six hours that were comparable or superior to those of human experts, and that it had been used in the design of a generation of Google's TPU chips. The claims were challenged. In 2023 a team led by Chung-Kuan Cheng and Andrew Kahng at the University of California, San Diego, published an independent assessment using Google's released code, and found that established methods, including simulated annealing and commercial placement tools, produced better results than the reinforcement learning approach on the benchmarks they tested, and that the method's advantage depended on pre-training and initial placement details not fully disclosed in the original paper. Other researchers, including Igor Markov, published critiques of the experimental methodology. Nature added an editor's note to the paper in 2023 indicating that its claims were being examined. In September 2024 the Google team published an addendum, named the method AlphaChip, released a pre-trained checkpoint, stated that it had been used in three generations of TPU, and in a subsequent paper argued that the independent assessments had not reproduced the method faithfully, among other things omitting pre-training and using much less compute. As of 2026 the dispute has not been settled in a way that commands general agreement among researchers in electronic design automation. The substance of the dispute is instructive regardless of who is right. Macro placement is a domain where classical methods are mature and strong, where the score used for training is a proxy for the real objective, and where reported improvements depend heavily on baselines, benchmarks and computational budgets. Those are exactly the conditions under which a claim of superiority is hard to establish and easy to contest. The AlphaTensor and AlphaDev results did not provoke comparable controversy because their outputs could be checked by anyone: an algorithm either uses 47 multiplications or it does not. Vehicle routing presents a similar picture with less heat. Oriol Vinyals, Meire Fortunato and Navdeep Jaitly's pointer networks of 2015 and Irwan Bello and colleagues' 2016 work on neural combinatorial optimisation showed that networks trained by supervised or reinforcement learning could construct reasonable tours for the travelling salesman problem. Wouter Kool, Herke van Hoof and Max Welling's 2019 attention model, trained with REINFORCE and a baseline derived from a greedy rollout of the current policy, produced near-optimal solutions quickly for instances of up to a hundred cities and several vehicle-routing variants. A substantial research literature has since followed. But mature solvers, such as Concorde for the travelling salesman problem and the LKH heuristic for larger instances and many variants, remain better on solution quality for large real-world problems. The learned methods' main advantages are speed once trained and the ability to adapt to problem variants for which no specialised solver exists. Practitioners in logistics mostly use learned methods, if at all, in combination with classical solvers, for example to choose which neighbourhood a local search should explore next, rather than as replacements for them. The most successful deployments in logistics have been those that use reinforcement learning to supply something classical methods lacked, usually a long-horizon value estimate. The Chinese ride-hailing company DiDi described in a 2020 paper by Zhiwei Qin and colleagues, which won the INFORMS Wagner Prize in 2019, a dispatching system in which each driver-order match is still solved as a combinatorial assignment at each moment, but the assignment's objective incorporates a value function learned by temporal-difference methods from historical trips. The value function estimates how good it is for a driver to end up in a given place at a given time, so the dispatcher prefers matches that leave drivers well positioned for future demand rather than only those that are best right now. This is the credit assignment problem of Chapter 1 in commercial form: a match's true value includes its consequences for every subsequent trip. Amazon's supply-chain group published a similar hybrid in 2022, by Dhruv Madeka and colleagues, describing a deep reinforcement learning system for inventory replenishment that they described as deployed end to end. Their key move was to build a differentiable simulator from historical data, correcting for demand that went unobserved when products were out of stock, and to optimise the ordering policy by gradients through that simulator. The approach depends on the same precondition as everything else in this book: the simulator had to be faithful enough, in the respects that mattered for ordering decisions, that a policy optimised inside it would perform well outside it. Control in the physical world Systems that control physical plant face the hardest form of the transfer problem, because the cost of a bad action is real and the simulator is always an approximation. In 2016 DeepMind and Google reported that a system built on neural networks trained on historical sensor data from Google's data centres, used to recommend settings to human operators, had reduced the energy used for cooling by up to 40 per cent. In 2018 they described a successor that controlled cooling directly, in which predicted actions were checked against operator-defined safety constraints before being applied, operators could intervene at any time, and the system fell back to conventional control if confidence dropped. The companies reported that the autonomous system delivered energy savings averaging around 30 per cent after several months of operation. Technically, this was predictive control using learned models of the plant, closer to the model-based branch of the field than to AlphaZero, but it illustrates the engineering that any deployed learning controller needs: a learned model to plan in, an explicit set of constraints that the planner cannot violate, and a human or conventional fallback when the model wanders outside its experience. The same pattern of guarded deployment appeared in the MuZero rate controller described in Chapter 5, and in later work on applying reinforcement learning to plasma control in fusion reactors. A 2022 Nature paper by Jonas Degrave and colleagues at DeepMind and the Swiss Plasma Center at EPFL trained a controller entirely in simulation to shape and hold the plasma inside the TCV tokamak, then deployed it on the real machine, where it produced a variety of plasma configurations, including ones that had not previously been achieved with that device's conventional controllers. The simulator was not perfect; the controller succeeded because it was trained across variations in the simulated physics and constrained by the operating limits of the device. Table 4 summarises the applications discussed in this chapter, organised around the three questions from the introduction: where the experience comes from, what keeps the score, and how the result has been received. Table 4. Game-playing methods applied to real optimisation and control problems. Application Experience source Score Reported outcome Status AlphaTensor (2022) Exact tensor algebra Multiplications used 47 for 4×4 mod 2 Verified, extended by others AlphaDev (2023) Executing assembly code Correctness and latency Up to 70% faster short sorts Merged into LLVM libc++ MuZero rate control (2022) Real VP9 encoder Self-competition 4% average bitrate saving Used on YouTube traffic Chip placement (2021) Proxy cost model Wire length, congestion Comparable or better than experts Disputed DiDi dispatch (2020) Historical trip data Long-run trip value Deployed at scale Prize-winning case study TCV tokamak (2022) Plasma simulator Shape-tracking accuracy Novel configurations held Experimental demonstration Source: Fawzi et al. 2022; Mankowitz et al. 2023; Mandhane et al. 2022; Mirhoseini et al. 2021 and Cheng et al. 2023; Qin et al. 2020; Degrave et al. 2022. What makes the transfer work Read across, the cases sort themselves by a few properties that can be checked in advance. The first is the fidelity of the simulator. AlphaTensor and AlphaDev had perfect simulators; their results held in reality by construction. MuZero's video codec was the real codec. The tokamak controller and the dispatch system had imperfect simulators and needed robustness training, constraints and fallbacks. The chip placer optimised a proxy for the real objective, and much of the controversy concerns whether gains on the proxy translate into gains on the real metric. The second is the honesty of the score. Where correctness can be checked exactly, search cannot cheat. Where the score is a learned or simplified proxy, search will find its weaknesses as surely as an Atari agent finds a scoring bug, and whatever it finds will look like progress until measured against the real objective. The third is the strength of the incumbent. Game-playing methods produced new results in matrix multiplication partly because the relevant search space was small enough to explore, but also because human mathematicians had not searched it exhaustively with computers. In mature domains, such as the travelling salesman problem, the burden of proof is high and the gains, where they exist, are modest. The fourth is whether the method contributes something that classical tools lack. The deployments that have lasted, in dispatch and inventory and cooling, typically use learning for a specific missing ingredient, a long-horizon value function or a learned model of plant dynamics, inside a framework of classical optimisation and engineering constraints. That is less dramatic than replacing the operations research department with AlphaZero. It is also how the methods actually earn their keep. The next chapter turns to the domain the popular account most often associates with game-playing AI: protein folding. There, the relationship between the algorithms of this book and the most famous result is less direct than it first appears, and the most direct applications are in a related problem, protein design, where all four of these properties are tested at once. CHAPTER 9 Proteins, Molecules and Proofs In November 2020, the organisers of the fourteenth Critical Assessment of protein Structure Prediction, CASP14, announced that a system called AlphaFold 2 had predicted the three-dimensional structures of proteins with a median accuracy that placed most of its predictions in the range of experimental methods. John Moult, who had co-founded the competition in 1994, said that in an important sense the problem had been solved. Four years later, Demis Hassabis and John Jumper of Google DeepMind shared half of the 2024 Nobel Prize in Chemistry for the work. The other half went to David Baker of the University of Washington, for computational protein design. Because AlphaFold came from the laboratory that built AlphaGo, it is commonly described as a game-playing algorithm turned to biology. That description is mostly wrong, and the ways in which it is wrong are illuminating. AlphaFold 2 is a supervised learning system, trained to map amino acid sequences to known structures; it has no reward, no self-play and no tree search. Yet the problems adjacent to structure prediction, designing new proteins, planning the synthesis of molecules, and proving mathematical theorems, are being attacked with exactly the search-and-learning loop of the earlier chapters, and some of those attacks have produced proteins that have been built, imaged and tested in animals. This chapter separates the two stories and argues that the second matters more for the future of these methods than the first. What AlphaFold is, and what it borrowed A protein is a chain of amino acids, typically hundreds long, drawn from an alphabet of twenty. In water, most proteins fold spontaneously into a specific three-dimensional shape, and that shape determines what the protein does. Christian Anfinsen's experiments in the late 1950s and early 1960s, for which he shared the 1972 Nobel Prize, showed that the sequence alone contains the information needed to specify the shape. Determining the shape experimentally, by X-ray crystallography, nuclear magnetic resonance or, more recently, cryo-electron microscopy, can take months or years per protein. Predicting it from sequence was one of the central open problems of molecular biology. For decades the strongest computational approaches treated folding as a search problem. Baker's Rosetta software, developed from the late 1990s, assembled candidate structures from fragments of known proteins and refined them by Monte Carlo sampling, accepting or rejecting random moves according to a physics-inspired energy function. In 2008 Baker's group, with the computer scientists Seth Cooper, Zoran Popović and colleagues at the University of Washington, released Foldit, a video game in which players manipulated protein structures by hand, scored by Rosetta's energy function. A 2010 paper in Nature reported that top Foldit players could outperform Rosetta's automated search on some problems, particularly those that required large rearrangements that the automated search was unlikely to attempt. In 2011 Foldit players produced a model of a retroviral protease from a monkey virus, a structure that had resisted solution for years, accurate enough to let crystallographers determine the real structure. Folding was, quite literally, made into a game, scored by an energy function, and human players acted as the search policy. The Foldit researchers also recorded the sequences of moves that successful players used and distilled them into reusable recipes, some of which turned out to resemble improved versions of Rosetta's own automated strategies. That is the AlphaGo Zero pattern with humans in place of the network: good search behaviour, discovered by playing, captured and fed back into the next generation of searchers. AlphaFold changed the method rather than the game. The first version, which won CASP13 in 2018, used a network to predict the distribution of distances between pairs of amino acids and then found a structure satisfying those predictions by gradient descent. AlphaFold 2, described by Jumper and colleagues in Nature in 2021, replaced that pipeline with a network that predicts the structure directly. Its core, the Evoformer, processes a multiple sequence alignment, a set of evolutionarily related sequences from other organisms, together with a representation of pairwise relationships between residues, using attention mechanisms that pass information between the two. A structure module then produces three-dimensional coordinates. The system was trained on structures from the Protein Data Bank, the public repository built up over half a century by experimental structural biologists. At CASP14 its median accuracy, measured by the competition's global distance test, was about 92 on a scale of 100, far ahead of every other entrant. The connections to the game-playing work are real but indirect. AlphaFold 2 iterates its own predictions through the network several times, a process called recycling, which plays a role loosely analogous to search: repeated refinement of an estimate using the same learned function. More directly, it used self-distillation. After initial training, the team used the model to predict structures for hundreds of thousands of sequences without known structures, kept the predictions the model was confident in, and retrained on the original data plus those predictions. This is the self-training idea that runs through AlphaGo Zero, in which a model's own best outputs become training data for its successor, applied in a supervised setting. And the organisational lessons carried over: a large team, a clear benchmark acting as a scoreboard, massive engineering investment, and a willingness to discard intermediate systems when a better approach appeared. The AlphaFold Protein Structure Database, built with the European Bioinformatics Institute, expanded in 2022 to more than 200 million predicted structures, covering nearly every catalogued protein. AlphaFold 3, described by Josh Abramson and colleagues in Nature in May 2024, extended prediction to complexes of proteins with DNA, RNA, small molecules and ions, using a diffusion-based architecture that generates atomic coordinates by iteratively removing noise. Its limits are also well documented. Predicted structures are static snapshots, while many proteins work by changing shape; regions that are intrinsically disordered, with no fixed structure, are predicted with low confidence, as they should be; and accuracy falls for proteins with few evolutionary relatives in the sequence databases, since the multiple sequence alignment is a major source of the system's information. AlphaFold's own confidence measures, which estimate per-residue and pairwise accuracy, are among its most useful outputs precisely because they tell a biologist when not to trust the prediction. None of this is reinforcement learning. Structure prediction had a vast supply of labelled examples, experimentally determined structures, and supervised learning is the right tool when the answers are known. The self-improvement loop is for problems where they are not. Design as a game Protein design is the inverse problem: instead of asking what shape a given sequence will fold into, ask what sequence will fold into a shape that does something useful, binding a virus, catalysing a reaction, assembling into a cage. No database contains the answers, because the proteins do not yet exist. There is, however, a score, or several: physics-based energy functions, structure predictors such as AlphaFold that can check whether a proposed sequence is predicted to fold as intended, and ultimately the laboratory. That combination, no labelled answers but a computable score, is exactly the setting in which game-playing methods earn their place. The most striking demonstration came from Baker's own group. In a paper published in Science in April 2023, Isaac Lutz, Shunzhi Wang and colleagues described what they called top-down design of protein architectures with reinforcement learning. The usual approach to designing protein assemblies works bottom-up: design a monomer that folds well, then search for ways to arrange copies of it into a larger structure. Lutz's team inverted this. They specified the overall architecture first, for example a hollow icosahedral shell made of 60 identical subunits, or a disc with a central pore, and then used Monte Carlo Tree Search to build the protein backbone fragment by fragment, with each move adding a structural segment and the reward measuring how well the growing structure satisfied the architectural constraints: fitting the symmetry, packing against neighbouring copies, avoiding clashes, and remaining designable. Sequences were then designed for the resulting backbones, and the proteins were made in the laboratory. The results were tested in the way the earlier chapters' results were not, by physical reality. Cryo-electron microscopy structures of the designed nanopores and icosahedral cages closely matched the computational models. The cages, smaller and more compact than those produced by earlier methods, allowed very dense display of molecules on their surfaces. When they were decorated with the influenza haemagglutinin antigen and used to immunise mice, they produced stronger antibody responses than a vaccine candidate already in clinical development. Cages displaying a signalling protein, angiopoietin, potently activated the corresponding pathway in human endothelial cells. Search over structure, scored by geometric and physical criteria, had produced objects that worked as intended in biological systems. A different group, Yi Wang and colleagues in China, published EvoPlay in Nature Machine Intelligence in 2023, applying an AlphaZero-style self-play loop to protein engineering. Each move mutates a single amino acid; a policy-value network guides a tree search over sequences of mutations; the reward comes from a surrogate model of fitness, a learned predictor of how well a variant will perform. The team tested the method on several tasks, including the design of peptide binders and the engineering of a luciferase, the enzyme that makes fireflies glow. The best engineered luciferase variants showed a 7.8-fold improvement in bioluminescence over the wild type in laboratory tests. Both results highlight the precondition that matters most in design: the score. In Lutz's work, the reward measured geometric properties that could be computed precisely, and the downstream sequences were checked with established tools before synthesis. In EvoPlay, the reward came from a learned surrogate, which will be wrong in regions of sequence space far from its training data. A search optimising hard against a learned surrogate faces the same danger as the policies optimised against reward models in Chapter 6 and the Go engines exploited in Chapter 7: it may find sequences that score well because the surrogate is wrong about them. Protein designers have learned this lesson from experience. Sequences generated by early approaches that optimised directly against structure predictors were sometimes adversarial in exactly this sense, predicted to fold beautifully and failing to fold at all in the laboratory. The field's response has been to use several independent scores, to keep proposals close to the distribution of natural proteins, and above all to test. The dominant design tools of the mid-2020s, such as the diffusion model RFdiffusion, described by Joseph Watson and colleagues from Baker's group in Nature in 2023, are generative models rather than search agents. But the search-and-score pattern sits around them: generative models propose, predictors and physics filter, and laboratory results feed back into training. Reinforcement learning enters increasingly as a way to fine-tune generative models towards properties that the training data does not directly encode, an approach that mirrors the use of PPO and GRPO to steer language models. Molecules and proofs: when a verifier is the rulebook Two neighbouring fields show the loop at work where the score is especially trustworthy. Chemical synthesis is a planning problem. A chemist who wants to make a target molecule works backwards, a process called retrosynthesis, asking which simpler molecules could be combined to make it, then which still simpler molecules could make those, until reaching compounds that can be bought. Each step is a move; the game is won when every branch ends in a purchasable starting material. In a 2018 Nature paper, Marwin Segler, Mike Preuss and Mark Waller combined Monte Carlo Tree Search with a policy network, trained on essentially all published organic reactions, to propose retrosynthetic steps, and a second network to filter out implausible ones. Their system solved almost twice as many molecules as the traditional rule-based computer search, thirty times faster, and in a double-blind test, chemists on average considered its routes equivalent to those reported in the literature. The architecture is AlphaGo's, applied to chemistry: a learned prior to propose moves, a search to look ahead, and a filter acting as a crude value function. Retrosynthesis planning tools built on this approach are now in routine use in pharmaceutical research. Mathematics offers the purest verifier of all. In the Lean proof assistant, a proof is a program that a small, trusted kernel checks mechanically; a proof either type-checks or it does not. DeepMind's AlphaProof, which took part in the 2024 International Mathematical Olympiad and was described by Thomas Hubert and colleagues in Nature in 2025, used this checker as the rules of a game. A language model translated around a million informal competition problems into formal Lean statements, producing a very large training set of problems without known proofs. AlphaZero-style reinforcement learning then trained a prover network, with tree search over proof steps and the Lean kernel deciding victory. For the hardest problems, the system used what the authors call test-time reinforcement learning, generating and training on millions of variants of the target problem during the competition itself. At the 2024 Olympiad, AlphaProof solved three of the five non-geometry problems, including the hardest problem of the competition, which only five human contestants solved; together with AlphaGeometry 2, which solved the geometry problem, the combined system scored at the level of a silver medal. The proofs took far longer to find than the time allowed to human contestants, a significant caveat, but every one of them was certified correct by the checker. The following year, systems working in natural language rather than formal proofs, among them a version of Google's Gemini with an extended reasoning mode and an experimental OpenAI model, reached gold-medal standard on the 2025 Olympiad problems, with Google's solutions graded by the competition's coordinators and OpenAI's by former medallists it engaged. Those systems were trained with large-scale reinforcement learning against verifiable rewards of the kind described in Chapter 6. The formal and informal approaches represent two ways of supplying a score. Lean supplies an incorruptible one at the cost of having to formalise everything; natural-language grading is more flexible but relies on human or model judgement that is harder to scale and easier to fool. Where the loop runs, and where it stops The protein, chemistry and mathematics cases sharpen the pattern established in the logistics chapter. Where the answers are known and plentiful, supervised learning is usually the better tool, and AlphaFold is the proof. Its triumph came from fifty years of experimentally determined structures, curated in a public database, and from network architectures well matched to the problem. No reward or self-play could have substituted for that data. Where the answers are not known but can be checked, the search-and-learning loop comes into its own. Lean checks proofs; reaction databases and chemists check synthetic routes; geometric criteria and, ultimately, cryo-electron microscopy check designed assemblies. In each case, a system can generate candidates far beyond human throughput, keep what passes, and learn from the results. Where the check is itself a learned model, the loop needs an external anchor. Surrogate fitness models, structure predictors used as oracles and learned reward models can all be exploited by sufficiently strong search. The anchor in biology is the laboratory, and the most credible results, Lutz's cages, EvoPlay's luciferase, are those that went through it. What has not happened, despite much speculation, is an AlphaZero for biology in the full sense, a system that learns from its own laboratory experiments in a closed loop the way AlphaGo Zero learned from self-play. Automated laboratories exist and are improving, and closed-loop design-build-test cycles are an active area of research. But laboratory experiments are slow and expensive compared with simulated games, millions of times slower per sample, which means that the loop must run mostly in simulation and touch reality rarely. The quality of the simulation, and the trustworthiness of the scores it produces, therefore determines how far the loop can go. That was true of Go, where the simulation was perfect. It is the central question everywhere else. Conclusion The history traced in this book is usually told as a story about algorithms: Monte Carlo Tree Search, then deep networks, then self-play, then learned models, then PPO at scale. Each step was real, and each paper repays reading. But laid side by side, the chapters suggest a different conclusion about where the difficulty now lies. Over a decade, the algorithms became general. AlphaZero's loop runs unchanged on any game with known rules; MuZero's runs without the rules; Gumbel search makes it affordable with a handful of simulations; PPO and GRPO run on almost anything that can be sampled and scored. What did not become general, and cannot, is the thing each algorithm runs on. The binding constraint on deep reinforcement learning is no longer the learner. It is the simulator and the score. That claim can be checked against every case in the book. AlphaGo Zero surpassed human knowledge because the rules of Go are a perfect, free simulator with an incorruptible score. AlphaTensor and AlphaDev produced results that entered textbooks and standard libraries because tensor algebra and executable code are perfect simulators and correctness cannot be faked. MuZero's first product used a real codec as its environment, and its hard problem was inventing a score, which self-competition solved. AlphaProof reached Olympiad standard because the Lean kernel is an incorruptible referee. The reasoning models of 2025 improved fastest in exactly the domains, mathematics with known answers and code with tests, where verifiers exist. By contrast, the contested cases share the opposite feature. Chip placement was trained on a proxy cost, and the argument about it is, at bottom, an argument about whether improving the proxy improves the chip. Reward models in RLHF are exploited by the policies optimised against them. Superhuman Go engines fail on positions their self-generated experience never contained, and protein designs optimised against learned predictors fail to fold when the predictor was wrong. What follows If the constraint has moved, effort should move with it. For anyone hoping to apply these methods, the most valuable work is often not choosing between PPO and MuZero but building the environment: a simulator that is faithful in the respects that matter for decisions, and a score that measures the real objective rather than a convenient stand-in. This is unglamorous work. It means writing verifiers, assembling test suites, calibrating physical models against measurements, and defining success precisely enough that a tireless optimiser cannot satisfy the letter of the definition while violating its intent. It is also where advantage now accumulates, because the algorithms are published and the environments are not. The three questions from the introduction become, in practice, a sequence of decisions. First, where does experience come from? If the answer is a perfect simulator, the AlphaZero family is likely to work and search is worth its cost. If it is an imperfect simulator, the system will need training across simulated variation, explicit constraints and a fallback for when reality departs from the model, as the cooling and tokamak controllers did. If the answer is historical data alone, the first task is to build a model from it, as Amazon's inventory team did, and to be honest about where that model is blind. Second, what keeps the score? If it is exact, optimise hard. If it is learned or proxied, optimise gently, regularise towards known-good behaviour in the manner of Cicero's piKL and the KL penalty of RLHF, and validate against the real objective often. Third, what plays the role of search? If decisions are few and consequential, spend computation looking ahead. If they are many and fast, distil the lookahead into a policy, or do without it and pay in data. These questions also equip a reader to judge claims. A report that a learned system beats established methods deserves the most trust when its outputs can be checked independently, as AlphaTensor's could, and the least when its gains are measured on a proxy, against baselines its authors chose, with a compute budget others cannot match. That is not cynicism about the field; it is the field's own experience. What remains unsettled Three questions are genuinely open, and each will shape the next decade. The first is robustness. The adversarial attacks on KataGo showed that superhuman average performance can coexist with simple, exploitable failures, and the follow-up work showed that obvious defences do not close the gap. Go is the friendliest possible setting for robustness: a perfect simulator, a narrow domain, and an enormous margin of skill. If robustness is hard there, it will be harder in systems deployed among people, some of whom will be looking for the cyclic group. Whether population-based training, adversarial training at scale or some new idea can produce systems whose worst case approaches their average case is unknown. The second is what reinforcement learning actually adds to large pretrained models. The reasoning models' gains are large and real, but researchers disagree about their source. Some studies, such as a 2025 analysis by Yang Yue and colleagues, have argued that reinforcement learning with verifiable rewards mostly makes a model more reliably produce solutions it could already sometimes find, rather than teaching it genuinely new capabilities; others point to behaviours, like self-correction, that appear to emerge during training. In AlphaGo Zero there was no doubt: the system began from nothing and everything it knew came from the loop. In language models, the loop begins from a model that has read much of the written record, and disentangling what the loop discovered from what pretraining supplied is difficult. The answer matters for how far scaling reinforcement learning on top of pretrained models can go. The third is whether the loop can run with reality in it. Every major success in this book ran mostly in simulation. Closed-loop laboratories, in which a learning system proposes experiments, robots perform them and results flow back, would let the loop touch the world directly. The obstacle is speed: an assay takes hours where a game of Go takes a fraction of a second. Learned models, in the spirit of MuZero, may bridge part of that gap by letting each real experiment train a model that supports many simulated ones. How much of it they can bridge is an empirical question that biology and materials science will answer over the coming years. What to do differently The practical change this book argues for is a change of emphasis. When a problem looks like a candidate for reinforcement learning, the first deliverable should not be an agent but an environment and a scoring function, built carefully enough that a superhuman optimiser could be trusted with them. If that environment cannot be built, the problem is probably not ready for these methods, and classical optimisation, supervised learning or human judgement will serve better. If it can, the algorithms are largely ready, openly described and increasingly cheap to run. Move 37 is remembered as a moment of machine creativity. It is better remembered as a moment when a system was given a world it could rehearse in endlessly and a score it could not cheat, and used them to find something no human had found. The systems will keep improving. Whether they find more such moves in medicine, logistics or mathematics depends less on them than on whether we can build those worlds and those scores for the problems that matter. Hashtags: #DeepReinforcementLearning #ReinforcementLearning #ArtificialIntelligence #MachineLearning #DeepLearning #AlphaGo #AlphaGoZero #AlphaZero #MuZero #GameAI #SelfPlayLearning #MonteCarloTreeSearch #MCTS #NeuralNetworks #PolicyOptimization #PolicyGradients #ProximalPolicyOptimization #PPO #DeepQNetworks #DQN #ModelBasedReinforcementLearning #ModelFreeReinforcementLearning #RewardOptimization #ValueFunctions #ActorCriticMethods #ExplorationVsExploitation #AIPlanning #GameTheory #AutonomousAgents #AIInnovation

Download the Book (PDF): Introduction: The Mirror and the Choice In October 2019 a team of researchers led by Ziad Obermeyer published a short paper in Science about an algorithm that almost nobody outside American hospital administration had heard of. The algorithm was a commercial risk-prediction tool, used by health systems to decide which patients should be enrolled in "high-risk care management" programmes: extra nurses, closer monitoring, priority appointments. Tools of this kind, the authors estimated, were being applied to around two hundred million people in the United States each year.1 The algorithm did not use race as an input. Its designers had not set out to treat anyone unfairly. It was accurate, by the standard it had been built to meet. And yet, when the researchers looked at patients who received the same risk score, they found that the Black patients among them were considerably sicker than the white patients: they had more uncontrolled chronic illness, worse blood pressure, worse kidney function. At the threshold the hospital used to auto-enrol people in the programme, correcting the disparity would have raised the share of Black patients receiving extra help from 17.7 percent to 46.5 percent. The cause was a single design decision. The tool had been trained to predict how much a patient would cost the health system in the coming year, on the reasonable-sounding assumption that sicker people cost more. But in the United States, Black patients with the same burden of disease have historically incurred lower medical costs than white patients, because of unequal access to care, unequal insurance coverage, distance from providers, and a well-founded distrust of medical institutions. The algorithm learned that relationship faithfully. It predicted costs extremely well. In doing so it reproduced, with the authority of a number, a pattern of unequal treatment that had been built over generations. This booklet is about that pattern and what can be done about it. Its argument can be stated briefly. Machine learning systems are, at bottom, instruments for compressing historical records into rules that can be applied to the future. When the historical record contains inequality, as every record of human institutions does, the system will learn the inequality along with everything else, and it will apply it at scale, consistently, and with an air of objectivity that makes it harder to challenge than the human decisions it replaced. There is no setting in which this happens only by accident, and there is no dataset in which fairness is simply waiting to be found. It follows that fairness in an algorithmic system is not a property that can be discovered. It has to be chosen. It has to be specified precisely enough to be measured, and the measurement has to be defended against alternatives that are equally precise and that point in different directions. The mathematics of fairness, which has matured remarkably over the last decade, does not remove that choice. It sharpens it, by showing exactly which reasonable goals cannot be achieved at the same time. And because the choice is made, in practice, by the people who frame the problem, assemble the data, select the objective and sign off on the evaluation, the responsibility for it falls heavily on data scientists and machine learning engineers, whether or not their job descriptions say so. What the booklet covers The first two chapters examine where bias comes from. Chapter 1 treats training data as an archive: a record of past decisions about lending, hiring, policing, medicine and language, carrying the imprint of the institutions that produced it. Chapter 2 follows harm through the stages of a machine learning pipeline, from the choice of what to predict to the way predictions are used, and shows that the data is only one of several points of entry. Chapters 3 and 4 turn to measurement. Chapter 3 sets out the main statistical definitions of group fairness, beginning with demographic parity and moving through equalised odds, equal opportunity, predictive parity and calibration, with worked arithmetic so that the definitions are concrete rather than symbolic. Chapter 4 explains why these definitions conflict, using the debate over the COMPAS recidivism tool as its central case, and offers a way of choosing among them that starts from the harm in question rather than from the mathematics. Chapters 5 and 6 go inside the models. Modern systems do not work directly on raw inputs; they work on learned representations, vectors in high-dimensional "latent spaces" where meaning is encoded as geometry. Chapter 5 shows how social bias takes the form of directions and clusters in those spaces, from word embeddings trained on news text to the image-and-text representations behind current generative models. Chapter 6 examines the techniques developed to remove that bias, from projecting out a "gender direction" to adversarial training and closed-form concept erasure, and is candid about how much they actually achieve. Chapter 7 widens the frame to interventions across the whole pipeline: reweighting data, constraining training, adjusting decision thresholds, and, most powerfully and least often, changing what the system is asked to predict. Chapter 8 deals with auditing systems once they are in the world, including the legal frameworks that now require it. Chapter 9 addresses the people who build these systems directly: what they owe the people their work affects, and what that obligation looks like on an ordinary working day. What the booklet does not cover Some adjacent questions are deliberately left aside. The booklet does not attempt a general survey of AI ethics; privacy, surveillance, labour displacement, autonomous weapons and the long-run risks of advanced AI each deserve their own treatment and are not argued here. Nor is it a software manual. Where open-source tools exist for the techniques described, they are named, but the emphasis is on understanding what a method does and when it fails, which is what a practitioner needs before reaching for any library. The examples are drawn largely from the United States and Europe, because that is where the best-documented cases and the most developed legal frameworks are found. The underlying dynamics are not regional. Wherever a system learns from records of unequal treatment, the same mechanisms apply. A note on vocabulary The word "bias" carries at least three meanings in this field, and confusion between them causes real trouble. In statistics, bias is the systematic difference between an estimator's expected value and the true value of the thing being estimated. In machine learning theory, it is the error introduced by a model's simplifying assumptions, traded against variance. In ordinary and legal usage, it means unfair treatment of people on the basis of who they are. This booklet is concerned with the third meaning, but the first two are not irrelevant, because statistical bias is often the mechanism through which unfair treatment arrives. When the word appears without qualification, it means a systematic pattern in a system's behaviour that disadvantages people because of their membership in a group, typically a group defined by race, sex, age, disability, religion, national origin or another characteristic that law or ethics treats as protected. "Discrimination" refers to the effect of such patterns on decisions about people. "Fairness" is used as the name for whatever property a system is supposed to have instead, with the understanding, developed at length in the chapters that follow, that there are several incompatible candidates for that property and that choosing among them is part of the work. The Obermeyer case ended, unusually, with a partial repair. The researchers contacted the company that made the tool, and together they found that replacing the cost label with a combination of cost and direct measures of health reduced the bias in their tests by 84 percent. The fix did not require a new class of algorithm or a breakthrough in mathematics. It required someone to ask what the system was really predicting, and whether that was what anyone actually wanted. That question, asked early and asked honestly, is where fair systems begin. CHAPTER 1 The Archive of Inequality Every dataset is a record of decisions. A table of loan applications records which applicants a bank chose to approve, at which rates, under which policies, in which neighbourhoods. A collection of hiring outcomes records whom recruiters chose to interview and whom managers chose to promote. Arrest records describe where police chose to patrol and whom they chose to stop. Even a corpus of text scraped from the web records which people had the time, access and confidence to write in public, and what they chose to say. This seems obvious once stated, but the habits of data science make it easy to forget. Data arrives as rows and columns, stripped of the institutional processes that produced it, and the training procedure treats it as a sample from some fixed underlying reality. The model's job, in the standard framing, is to learn the function that maps inputs to outputs in that reality. If the outputs in the data are themselves the product of unequal treatment, the "function" the model learns includes the unequal treatment. It has no way to tell the difference, because nobody has told it there is one. The phrase that best captures this is the one used by the legal scholars Solon Barocas and Andrew Selbst in their influential 2016 article on big data and disparate impact: data mining can "reproduce existing patterns of discrimination, inherit the prejudice of prior decision makers, or simply reflect the widespread biases that persist in society."1 The model is not inventing prejudice. It is inheriting it. The long memory of a map Consider credit and housing in the United States, where the documentary trail is unusually clear. In the 1930s the federal Home Owners' Loan Corporation produced "residential security" maps for more than two hundred American cities, grading neighbourhoods from A ("best") to D ("hazardous"). The D neighbourhoods, coloured red, were very often those with Black residents, and the explanatory notes made the reasoning explicit: the presence of Black families, or of immigrants from disfavoured countries, was treated as a cause of declining property values. Whether these particular maps were used directly by lenders is still debated by historians, but the practice they codified, the systematic denial of mortgage credit to Black neighbourhoods, became pervasive, was reinforced by federal mortgage insurance underwriting rules, and persisted for decades. The digitised maps, now freely available through the Mapping Inequality project at the University of Richmond, make it possible to compare the 1930s grades with present-day outcomes, and the correspondence in many cities is striking: lower home values, lower rates of homeownership, poorer health indicators. Explicit redlining was outlawed by the Fair Housing Act of 1968 and the Equal Credit Opportunity Act of 1974. But its consequences are embedded in exactly the variables a modern credit model would use. Wealth gaps between Black and white American households are large and persistent, and much of household wealth is home equity. Credit histories are shorter for people whose parents could not obtain mortgages and so could not pass on property or co-sign loans. Neighbourhood, measured through postcode, carries a strong statistical association with race because of the residential segregation that redlining helped to entrench. A model trained on the outcomes of past lending, using inputs like these, does not need a race variable to learn the pattern. The pattern is already distributed across the variables it has. There is evidence that this happens. In 2021 the investigative newsroom The Markup analysed more than two million conventional mortgage applications from the federal Home Mortgage Disclosure Act data for 2019, controlling for the financial variables available in that dataset. It found that lenders were substantially more likely to deny Black applicants than comparable white applicants, with the gap reported at 80 percent nationally, and that much of the underwriting was driven by automated systems mandated by the government-sponsored enterprises Fannie Mae and Freddie Mac.2 The industry objected that the public data omitted credit scores and other key variables, which is true; the analysis cannot establish that an algorithm was responsible. But the episode illustrates the central difficulty. When the inputs themselves have been shaped by history, controlling for them does not neutralise history. It hides it inside the control variables. An academic study by Robert Bartlett, Adair Morse, Richard Stanton and Nancy Wallace, published in the Journal of Financial Economics in 2022, offers a more nuanced picture. Using a richer dataset of mortgages guaranteed by the government-sponsored enterprises, they found that Latinx and African-American borrowers paid higher interest rates than otherwise similar borrowers, and that this held for algorithmic lenders as well as for face-to-face lenders, though the gap was somewhat smaller for the algorithms.3 Automation, on this evidence, can reduce some of the discretionary discrimination of human loan officers while preserving the structural kind. That is a common pattern, and one worth keeping in mind whenever a vendor claims that a model is fairer than the people it replaced. It may be. It may also be unfair in a quieter, more durable way. Hiring, and the problem of the successful employee The most frequently cited example of a model learning historical bias from its labels is a recruiting tool that never went into full use. In October 2018 Reuters reported that Amazon had spent several years, from around 2014, building an automated system to rate job applicants' résumés on a scale of one to five stars.4 The system was trained on résumés submitted to the company over a ten-year period, with the implicit target of resembling the people the company had hired. Because the technology workforce over that period was predominantly male, the model learned to penalise résumés that included the word "women's", as in "women's chess club captain", and, according to the report, downgraded graduates of two all-women's colleges. Engineers edited the system to neutralise those particular terms, but could not be confident that it would not find other proxies, and the project was eventually abandoned. The Amazon case is often told as a story of corporate embarrassment, but its technical lesson is more general and more uncomfortable. In hiring, the label "good employee" is almost never observed directly. What is observed is who was hired, who was promoted, who received strong performance reviews, who stayed. Each of those is the outcome of human judgements made inside a particular organisation with a particular history. If women were held to higher standards for promotion, or left because the culture was hostile, the data will report that women are worse bets. If people from certain universities were favoured because the managers had attended them, the data will report that those universities produce better employees. A model trained to predict these outcomes is trained to reproduce the institution's past judgements, including its mistakes, and the model's accuracy is measured by how well it does so. This is the core reason why "the algorithm just reflects the data" is not a defence. It is a description of the problem. Policing and the data that observes itself Criminal justice data adds a further twist: the records are not merely shaped by past decisions, they are generated by the very activity a predictive system is meant to guide. Arrest data records where police were present and whom they chose to stop, search and charge. For offences that are rarely reported by victims, such as drug possession, the arrest record is almost entirely a function of enforcement intensity. Kristian Lum and William Isaac made this concrete in a 2016 article in Significance.5 They took a widely used predictive policing algorithm and applied it to drug arrest data from Oakland, California. They compared the resulting predictions with an independent estimate of where drug use actually occurred, derived from a national public health survey combined with census demographics. Drug use, according to the survey-based estimate, was spread fairly evenly across the city. The arrests, and therefore the algorithm's predictions, were concentrated in a small number of predominantly Black and low-income neighbourhoods. Had police followed the predictions, they would have been sent back to the same places, made more arrests there, and generated more data confirming that those were the places where drug crime happened. The structure of this loop, in which the system's outputs determine its future inputs, is examined in the next chapter. What matters here is the status of the data. A record of arrests is not a record of crime. It is a record of policing, and to treat it as a measurement of crime is to assume away the very thing in question. Language as an archive The archive metaphor applies with special force to language. Large language models and the word representations that preceded them are trained on text produced by enormous numbers of people over long periods, and that text carries the associations of its authors. In 2017 Aylin Caliskan, Joanna Bryson and Arvind Narayanan published a paper in Science with the plain title "Semantics derived automatically from language corpora contain human-like biases."6 They took the Implicit Association Test, a psychological instrument that measures how quickly people associate pairs of concepts, and built an analogue for word embeddings, which they called the Word Embedding Association Test. Using embeddings trained on a large web crawl, they reproduced a long list of findings from human studies. Flowers were more associated with pleasant words than insects were. European-American first names were more associated with pleasant words than African-American first names. Female names were more associated with family and male names with career. Most strikingly, the strength of the association between an occupation word and female terms in the embedding correlated closely with the actual proportion of women in that occupation according to US labour statistics. That last result captures the double nature of the problem. In one sense the embeddings were accurate: they had correctly learned how language is used, and language reflects the world, including its occupational segregation. In another sense they were dangerous: a system that uses these representations to, say, rank candidates for a nursing or engineering job will carry the segregation forward, treating a statistical fact about the past as a signal about the suitability of an individual in the future. The embedding cannot distinguish between "this association exists in the world" and "this association is a good basis for decisions." Only the people building the system can draw that line. Text archives are also skewed in who they represent. Much of the training data for large models comes from web crawls in which English dominates, certain countries and demographic groups are heavily overrepresented, and forums with particular cultures contribute disproportionate volumes of text. Emily Bender, Timnit Gebru and their co-authors made this point forcefully in their 2021 paper on the risks of very large language models: the size of a corpus is no guarantee of its diversity, and the filtering steps meant to remove offensive content can themselves remove the speech of marginalised communities, for instance by filtering out text that discusses sexuality or uses reclaimed slurs.7 An archive can be enormous and still have a narrow point of view. When the archive is written into the formula Bias does not need machine learning to enter a decision rule. Some of the clearest examples of history being encoded as if it were biology come from clinical medicine, where simple equations derived from population data have been used for decades, and where the debate about them has sharpened the thinking about learned models too. For many years, the standard equations for estimating kidney function from a blood test of creatinine included a coefficient for race. A patient identified as Black was assigned an estimated glomerular filtration rate around 16 to 21 percent higher than an otherwise identical patient who was not, depending on the version of the equation. The adjustment rested on observations, from the study populations used to fit the equations, that Black participants had higher average creatinine levels, which researchers attributed to differences in muscle mass. The effect in practice was that Black patients appeared to have healthier kidneys than they did. Because eligibility for specialist referral and for the kidney transplant waiting list depends on estimated kidney function, the coefficient could delay both. In 2021 a joint task force of the US National Kidney Foundation and the American Society of Nephrology recommended a refitted equation without the race term, and in 2022 the organisation that manages the US transplant system required transplant hospitals to use race-neutral calculations, later allowing affected Black patients to have their waiting time adjusted. A similar story played out with a calculator used to estimate the chance of a successful vaginal birth after a previous caesarean section. Its original version included race and ethnicity as inputs and predicted lower success for Black and Hispanic women, which could steer them toward repeat surgery. It was revised in 2021 to remove those variables. These cases matter for machine learning because they show the same structure in its plainest form. A statistical regularity in an archive of past patients, itself shaped by who was studied, how race was recorded and what social conditions affected health, was turned into a rule applied to future individuals. The rule was accurate on average for the population on which it was fitted. Its use of race as a stand-in for unmeasured physiology or circumstance meant that its errors fell systematically on one group. When learned models absorb similar regularities without anyone writing them down, the same thing happens, but no committee ever votes on a coefficient that could later be removed. Why "more data" does not fix it A common first response to these cases is to propose collecting more data, or more representative data. Sometimes this is exactly right. Where a group is underrepresented in the training set, so that the model has too few examples to learn their patterns, adding examples genuinely helps; the facial analysis cases discussed in Chapter 8 are the standard illustration. But representation and history are different problems, and more data only cures the first. If the labels themselves encode past discrimination, more data of the same kind simply gives the model more confidence in the discriminatory pattern. Ten million loan outcomes shaped by redlining are not fairer than ten thousand. If the target variable is a poor proxy for what the decision-maker cares about, as health costs were for health need, more data makes the proxy more precisely estimated without making it a better proxy. If the features carry the imprint of segregation, adding rows does not remove it. The more useful response is to treat the dataset as a historical source, with the same scepticism a historian brings to an archive. Who created these records, and for what purpose? Which events were recorded and which were not? Whose judgement produced the labels, and what pressures were they under? What would the record look like if the past had been different? These questions do not have tidy computational answers, but asking them is the precondition for every technical method described later in this booklet. A team that has not asked them does not know what it is correcting for. The difference between reflecting and amplifying One final distinction will recur throughout. A model that perfectly reproduced the patterns in its training data would already be a problem when those patterns are unjust. But models often do something worse: they amplify them. Jieyu Zhao and colleagues demonstrated this in a 2017 study of image recognition models trained to label activities in photographs.8 In the training data, images of cooking showed women about two-thirds of the time. A model trained on that data, when shown new cooking images, predicted that the person was a woman far more often than two-thirds of the time, around 84 percent in their measurements. The model had learned that gender was a useful cue for the activity and had leaned on it more heavily than the data justified, because doing so improved its average accuracy. Amplification arises because standard training optimises aggregate accuracy, and a model that can gain a little accuracy by exploiting a correlation with a protected attribute will usually do so, often to a greater degree than the correlation in the data warrants. Thresholding then converts small differences in probability into categorical decisions. If a model's scores for one group sit slightly below a cutoff and those for another sit slightly above, a modest statistical difference becomes a stark difference in outcomes. The upshot is that bias is not only inherited but can be intensified by the very process of learning. Understanding where, in the life of a system, these effects enter is the task of the next chapter. CHAPTER 2 Where Harm Enters the Pipeline It is tempting to locate algorithmic bias in a single place, usually "the data", and to treat everything downstream as neutral machinery. The cases in the previous chapter already strain that picture. In the Obermeyer health study the data was accurate; the problem was the choice of target. In the Oakland policing analysis the problem was not only the historical record but the loop by which predictions would generate future records. In the image recognition study the model amplified a correlation beyond what the data contained. A more useful picture treats a machine learning system as a sequence of decisions, each of which can introduce or compound unfairness. Harini Suresh and John Guttag set out one of the clearest versions of this picture in a framework first circulated in 2019 and published in 2021, identifying distinct sources of harm at different stages of the life cycle.1 Their categories, with some adaptation, structure this chapter. The value of distinguishing them is practical: each calls for a different diagnosis and a different remedy, and a team that has only one word for the problem tends to reach for only one fix. Table 1 summarises the stages and the characteristic failure at each, before the chapter takes them in turn. Table 1. Sources of harm across the machine learning life cycle. Stage Source of harm Typical example First question to ask Problem formulation Target choice Predicting cost as a proxy for health need Is the label what we actually care about? Data generation Historical bias Hiring outcomes shaped by past discrimination Whose decisions produced these labels? Data collection Representation bias Face datasets dominated by lighter-skinned men Who is missing or scarce in the sample? Measurement Measurement bias Arrests used as a measure of crime Does the feature mean the same thing for every group? Modelling Aggregation and learning bias One model fitted to groups with different patterns Does one model fit all subpopulations? Evaluation Evaluation bias Benchmarks that underrepresent affected groups Who is the test set testing for? Deployment Deployment and feedback bias Policing predictions that shape future data How will outputs be used, and what do they change? Source: adapted from Suresh and Guttag (2021). Before the data: choosing what to predict The first and most consequential decision in any predictive system is what to predict. Real-world goals are rarely directly measurable. A hospital wants to identify patients who would benefit most from care management, but "benefit" is a counterfactual quantity that is never observed. A lender wants to know who will repay, but repayment is only observed for people who were given loans. An employer wants good employees, but "good" is a judgement, not a measurement. In every case the team must choose a measurable stand-in, and the choice embeds assumptions that may hold for some groups and fail for others. Barocas and Selbst call this the problem of defining the "target variable" and "class labels," and they note that it is where some of the most significant discretion in a project lies, often exercised without anyone treating it as an ethical decision.2 The Obermeyer case is the cleanest illustration: health costs track health need reasonably well on average, but systematically less well for Black patients, because of unequal access. The proxy is not wrong in general. It is wrong differentially, and differential error is the shape that bias usually takes. A second subtlety concerns the selective observation of outcomes. In lending, the repayment behaviour of rejected applicants is never seen. A model trained only on approved loans learns about the population that past lenders were willing to serve. If a group was historically rejected at high rates, the model sees few of its members, and those it does see were selected by a process that may have required them to clear a higher bar. This is the "selective labels" problem, and it means the training data is not a sample from the population on which the model will be used. Statisticians have methods for some versions of it, including reject inference in credit scoring, but none of them can recover information that was never collected. Inside the data: history, representation and measurement The previous chapter discussed historical bias at length: the labels faithfully record an unjust world. Two further failures can occur even when the world being recorded is itself fair. Representation bias arises when the sample underrepresents some part of the population the system will serve. The best-known instances are in computer vision, where benchmark face datasets assembled in the 2000s and early 2010s skewed heavily toward lighter-skinned and male subjects. Medical datasets show the same pattern. Dermatology image collections used to train skin cancer classifiers have contained very few images of darker skin, and clinical trial populations, which feed many risk equations, have historically underrepresented women, older adults and minority groups. A model trained on such data may perform well on average and poorly for exactly the people who were scarce. Representation bias can also run in the opposite direction, with a group overrepresented in ways that distort what the model learns about it. Virginia Eubanks's account of the Allegheny Family Screening Tool, a model introduced in 2016 in the Pennsylvania county that includes Pittsburgh to help screen calls to the child-welfare hotline, illustrates the point. The model drew on the county's integrated records of public services: benefits, public mental health and addiction treatment, county jail, juvenile probation and similar systems. Families who relied on public services were therefore richly documented, while families who paid for private therapy, private addiction treatment or private childcare left no trace in the data. A history of contact with public systems became, in the model, a signal of risk, and that history is far more common among poor families. The county has revised the tool several times and commissioned independent evaluations, and its defenders argue that it improved on the inconsistency of unaided screeners. But the asymmetry in what the data could see was built into its foundations. Measurement bias arises when the features or labels are measured differently, or mean something different, across groups. Arrests are the standard example, because the same underlying behaviour leads to an arrest at different rates depending on where one lives and how one looks. But measurement bias is often subtler. Pulse oximeters, which estimate blood oxygen by shining light through the skin, have been shown to overestimate oxygen saturation in patients with darker skin, so that hidden hypoxaemia occurs more often in Black patients. A model that takes oximeter readings as inputs will inherit that error without any sign of it in the data, because the data contains only the readings, not the truth. Similarly, a "years of continuous employment" feature means something different for people who took time out for caregiving, a burden that falls disproportionately on women. The number is accurate. Its meaning is not uniform. Labels produced by human annotators carry a further version of the same problem. Much of modern machine learning depends on people paid to label images, rate text or flag content, and those people bring their own interpretive habits to the task. Maarten Sap and colleagues showed in 2019 that in widely used datasets for detecting hate speech and offensive language, tweets written in African American English were substantially more likely to be labelled offensive than tweets in other varieties of English, and that classifiers trained on those datasets reproduced the disparity.3 Much of the difference came from annotators reading dialect features, and reclaimed in-group uses of certain words, as markers of offence. A moderation system trained on such labels would remove the speech of Black users more often, not because anyone designed it to, but because the definition of "offensive" in its training data had been set by people unfamiliar with the dialect. The authors found that priming annotators with information about the dialect and the likely race of the author reduced the rate at which they labelled such tweets offensive, which shows that the label was a judgement shaped by context rather than an observation of a fixed property. What makes measurement bias treacherous is that it cannot be seen by inspecting the dataset. Everything looks clean. Detecting it requires external knowledge about how the measurements were produced, which is one reason why fairness work cannot be done by data scientists alone. Inside the model: aggregation and the pull of the majority Even with good data, the modelling process introduces its own pressures. Aggregation bias arises when a single model is fitted to a population whose subgroups follow different patterns. A classic medical example is the relationship between HbA1c, a blood marker used to diagnose and monitor diabetes, and average blood glucose, which differs somewhat across ethnic groups. A single threshold or model calibrated to the pooled population will be systematically off for groups whose relationship differs from the majority's. Learning bias, in Suresh and Guttag's terms, arises from the modelling choices themselves. Standard training minimises the average loss over all examples. When one group makes up ninety percent of the training data, the model is rewarded far more for getting that group right than for getting the remaining ten percent right, and the optimiser will happily sacrifice minority accuracy for small majority gains. Regularisation and model compression can make this worse. Sara Hooker and colleagues showed in 2020 that pruning and quantising neural networks, which barely affects overall accuracy, can disproportionately damage performance on underrepresented subgroups and rare examples. The headline metric hides the cost. The amplification effect described at the end of the previous chapter belongs here too. A model that finds a protected attribute, or a proxy for it, useful for prediction will tend to rely on it, and may rely on it more heavily than the underlying correlation warrants. Proxies: why removing the protected attribute does not work The most common naive defence against algorithmic discrimination is to remove the protected attribute from the inputs. The thinking is that a model which never sees race cannot discriminate by race. This is sometimes called "fairness through unawareness," and it fails for a simple reason: in most real datasets, the protected attribute can be predicted from the remaining features. Postcode predicts race in segregated cities. First names predict sex and often ethnicity. Purchase histories, browsing patterns, the language of a résumé, the type of phone a person uses, the time of day they apply for credit: any of these can correlate with protected characteristics, and the combination of many weakly correlated features can recover the attribute with surprising accuracy. A model optimising for its target will make use of whatever information the features carry, including information about group membership, if that information helps prediction. The protected attribute has not been removed. It has been made invisible to the people who might check for it. There is a stronger point. Removing the protected attribute can make outcomes less fair, not more. If a feature means something different for different groups, as employment gaps do for men and women, a model that knows the group can learn to interpret the feature accordingly. A model that does not know the group must apply a single interpretation, which will be the majority's. Several of the debiasing techniques in later chapters require the protected attribute to be available during training or evaluation, precisely so that its influence can be measured and controlled. The European Union's AI Act recognises this, carving out a limited permission in Article 10 for providers of high-risk systems to process sensitive personal data where strictly necessary for bias detection and correction. Blindness is not neutrality. It is an absence of oversight. After the model: evaluation and deployment Evaluation bias occurs when the data used to test a model does not represent the population on which it will be used, or when the metrics used to judge it ignore the dimensions on which it fails. If a speech recognition system is benchmarked on audiobook recordings by predominantly standard-accent speakers, a strong benchmark score says little about its performance for speakers of regional or ethnic dialects. A 2020 study by Allison Koenecke and colleagues in the Proceedings of the National Academy of Sciences tested five commercial speech recognition systems and found that the average word error rate for Black speakers was nearly double that for white speakers.4 None of this would appear in a single aggregate score. Deployment bias arises when a system is used in a context, or for a purpose, different from the one it was designed and evaluated for. A risk score built to inform decisions about pretrial supervision may be used to inform sentencing. A tool designed to support human judgement may in practice be followed mechanically because staff are overworked or because overriding it requires paperwork. Conversely, humans may selectively override a tool in ways that reintroduce bias: deferring to it when it confirms their expectations and overriding it when it does not. Research on pretrial risk assessment in several US jurisdictions has found that judges' adherence to the recommendations varied by defendant characteristics, which can mean that a tool that is fair on paper produces unfair outcomes in practice. The relevant unit of analysis is not the model but the decision system of which the model is one part. Feedback loops The most dynamic form of deployment bias is the feedback loop, in which a system's decisions shape the data on which it will later be retrained. Danielle Ensign and colleagues formalised this for predictive policing in a 2018 paper with the apt title "Runaway feedback loops in predictive policing."5 They modelled a city with two districts and a policing algorithm that allocated patrols in proportion to recorded crime. Because crime is recorded mainly where police are present, a district that receives slightly more patrols records slightly more crime, which leads to more patrols, which leads to more records. In their model, even when the true crime rates in the two districts were equal, the system could lock in on one of them and send nearly all patrols there indefinitely. The same structure appears elsewhere. A lending model that denies credit to a group prevents its members from building credit histories, which makes the next model even less willing to lend to them. A content recommendation system that shows certain creators less often gives them fewer opportunities to accumulate engagement, which the system then reads as evidence of lower quality. A hiring model that screens out applicants from certain backgrounds ensures that the company's future workforce, and thus its future training data, contains fewer of them. Feedback loops matter for fairness because they convert static bias into dynamic bias. A small initial disparity, which an audit at launch might judge tolerable, can grow over successive retraining cycles into a large one. They also mean that fairness cannot be certified once and left alone. A system has to be monitored over time, and the monitoring has to look at the data-generating process, not just the model. What this anatomy implies Two practical conclusions follow from treating bias as a property of a pipeline rather than of a dataset. The first is that the remedies are stage-specific. Representation bias can often be addressed by collecting more data from underrepresented groups. Historical bias cannot, because more data of the same kind reinforces the history. Measurement bias needs better measurements or explicit correction for known errors. Target-choice bias needs a different target. Learning bias can be addressed by constraints or reweighting during training. Feedback loops need changes to how the system's outputs are used and how its training data is collected. A team that diagnoses every problem as "biased data" and responds with resampling will fix some problems and leave the others untouched, while believing it has done the job. The second is that measuring fairness requires a definition of what fair outcomes would look like, and that definition must be stated in terms precise enough to compute. Saying that a model "disadvantages" a group presupposes a standard against which disadvantage is judged: equal rates of approval, equal error rates, equally reliable scores. These standards are not the same, and as later chapters show, they cannot in general all hold at once. Before any of the remedies above can be evaluated, the team has to decide what it is trying to achieve. The next chapter sets out the main candidates. CHAPTER 3 Counting Fairness: The Group Metrics To say that a system is unfair to a group is to make a comparison: the group receives worse treatment than some other group, or worse than it should. Turning that intuition into something a computer can check requires choosing what to compare. Over the past fifteen years the research community has produced dozens of formal fairness criteria, but most of the ones used in practice belong to a small family built from the same raw material, the confusion matrix of a classifier computed separately for each group. This chapter explains that family carefully, because nearly every argument about algorithmic fairness eventually turns into an argument about which of its members to use. The setting is a binary decision. A model produces a prediction for each person, say "approve" or "deny", "high risk" or "low risk", and there is a true outcome that the prediction is trying to anticipate, such as whether the person repaid the loan or was rearrested. People also belong to groups, defined by a protected attribute. For simplicity, imagine two groups, A and B, while remembering that real applications involve many groups and their intersections, a problem returned to at the end of the chapter. The confusion matrix, group by group For each group, every person falls into one of four cells. True positives are people predicted positive who turned out positive. False positives were predicted positive but turned out negative. True negatives were predicted negative and were negative. False negatives were predicted negative but were in fact positive. From these counts a number of rates follow. The selection rate, or positive prediction rate, is the share of the group predicted positive. The base rate, or prevalence, is the share of the group that is actually positive. The true positive rate, also called sensitivity or recall, is the share of actual positives that the model correctly identifies. The false positive rate is the share of actual negatives that the model wrongly flags. The positive predictive value, or precision, is the share of positive predictions that turn out to be correct. The negative predictive value is the same thing for negative predictions. Each of the main fairness metrics asks that one or more of these rates be equal, or nearly equal, across groups. The differences between them come down to which rate they equalise, and Table 2 sets out the principal choices. Table 2. The main group fairness criteria for binary classifiers. Criterion What must be equal across groups Also known as Ignores the true outcome? Demographic parity Selection rate Statistical parity, independence Yes Disparate impact ratio Ratio of selection rates, judged against a threshold Four-fifths rule (as a test) Yes Equal opportunity True positive rate — No Equalised odds True positive rate and false positive rate Separation No Predictive parity Positive predictive value Part of sufficiency No Calibration within groups Observed outcome rate at each score level Sufficiency No Source: definitions as in Barocas, Hardt and Narayanan, Fairness and Machine Learning (2023). Demographic parity and the four-fifths rule Demographic parity is the simplest criterion. It requires that each group be selected at the same rate. If 40 percent of applicants from group A are approved for a loan, then 40 percent of applicants from group B should be too. In probability notation, the prediction is independent of group membership, which is why Barocas, Hardt and Narayanan call this family of criteria "independence".1 Its appeal is that it does not depend on the labels at all. If the labels are themselves products of historical discrimination, as Chapter 1 argued they often are, a criterion that ignores them is immune to that contamination. Demographic parity says, in effect, that whatever the past record shows, the system should distribute opportunities in proportion to the groups' presence in the applicant pool. In practice the criterion is usually relaxed into a ratio. The disparate impact ratio divides the selection rate of the less-favoured group by that of the more-favoured group. In US employment law, the Uniform Guidelines on Employee Selection Procedures, adopted by federal enforcement agencies in 1978, describe a selection rate for any group that is less than four-fifths of the rate for the group with the highest rate as generally regarded as evidence of adverse impact.2 This "four-fifths rule" was intended as a rule of thumb for when to look more closely, not as a legal standard of liability, but it has become the default threshold in many fairness toolkits and was written into New York City's 2021 law requiring bias audits of automated employment decision tools, which requires those audits to report impact ratios. A worked example makes the arithmetic concrete. Suppose a résumé screening model advances 300 of 1,000 applicants from group A and 180 of 900 applicants from group B. The selection rates are 30 percent and 20 percent. The ratio is 20 divided by 30, or about 0.67, which falls below 0.8 and would trigger scrutiny under the four-fifths rule. The difference in selection rates, sometimes reported instead, is ten percentage points. The weakness of demographic parity is equally clear. If the groups genuinely differ in the outcome the system is predicting, for reasons that are not themselves the product of the system, then enforcing equal selection rates means either selecting less-qualified members of one group or rejecting more-qualified members of another. Cynthia Dwork and her colleagues pointed out in 2012 that demographic parity can be satisfied in perverse ways, for example by selecting the best candidates from one group and a random sample from the other, which meets the statistical requirement while being plainly unfair to the individuals in the second group.3 Equal rates are not the same as equal treatment. Whether group differences in the true outcome are "genuine" in the relevant sense is, of course, often the heart of the dispute. A difference in measured repayment rates may reflect differences in wealth that are themselves the legacy of discrimination. Choosing demographic parity amounts to a judgement that the observed differences should not be allowed to determine the allocation, either because they are artefacts of biased measurement or because the purpose of the decision is partly to correct them. That judgement may be right. It should be made deliberately. Equal opportunity and equalised odds The second family conditions on the true outcome. Instead of asking whether groups are selected at equal rates overall, it asks whether people who share the same true outcome are treated alike regardless of group. Equal opportunity, introduced by Moritz Hardt, Eric Price and Nathan Srebro in 2016, requires equal true positive rates.4 Among people who would in fact repay a loan, the share who are approved should be the same in every group. Among people who would succeed in a job, the share who are advanced should be the same. The name captures the intuition: qualified people should have equal chances of being recognised as qualified. Equalised odds is stricter. It requires equal true positive rates and equal false positive rates. In a pretrial context, among people who will not be rearrested, the share wrongly labelled high risk should be the same in every group, and among people who will be rearrested, the share correctly labelled high risk should be the same too. Barocas, Hardt and Narayanan call this family "separation", because it requires the prediction to be independent of group membership once the true outcome is known. Return to the worked example, now with outcome data. Suppose that in group A, 500 of the 1,000 applicants would actually succeed in the job, and the model advances 250 of them along with 50 who would not. The true positive rate is 250 out of 500, or 50 percent, and the false positive rate is 50 out of 500, or 10 percent. In group B, suppose 300 of 900 would succeed, and the model advances 120 of them along with 60 who would not. The true positive rate is 120 out of 300, or 40 percent, and the false positive rate is 60 out of 600, also 10 percent. The model satisfies one half of equalised odds, with equal false positive rates, but violates equal opportunity: qualified members of group B are recognised less often than qualified members of group A. These criteria have a property that demographic parity lacks: a perfect classifier satisfies them automatically, since a model that is always right has a true positive rate of one and a false positive rate of zero for everyone. They therefore do not force a trade against accuracy in principle, only against the accuracy achievable with imperfect predictors. Their weakness is the converse of demographic parity's strength. They take the labels as ground truth. If the labels are biased, as with arrests standing in for offending, equalising error rates relative to those labels can entrench the bias rather than correct it. Predictive parity and calibration The third family looks at the problem from the side of the decision-maker who receives the prediction. When the model says "high risk", how often is it right, and is that the same for every group? Predictive parity requires equal positive predictive value. If 60 percent of the people labelled high risk in group A go on to reoffend, then 60 percent of those labelled high risk in group B should too. In the worked example, the positive predictive value for group A is 250 out of 300, or about 83 percent, and for group B it is 120 out of 180, or about 67 percent. An employer using this model would find that a recommendation to advance means less for applicants from group B than for those from group A. Calibration within groups extends this from a binary label to a continuous score. A risk model is calibrated within groups if, for every score value, the observed rate of the outcome among people with that score is the same in every group, and equal to the score. If the model assigns a 30 percent risk, then about 30 percent of the people with that score should experience the outcome, whichever group they belong to. Barocas, Hardt and Narayanan call this family "sufficiency", because the score is then sufficient: once you know it, knowing the group adds no further information about the outcome. Calibration is the criterion that practitioners in medicine, insurance and lending tend to find most natural, because it guarantees that a score means the same thing for everyone. A doctor who sees a 20 percent risk of a heart attack wants that to be a 20 percent risk whether the patient is a man or a woman. Most well-built statistical models trained on adequate data are approximately calibrated by default, at least for groups well represented in the training set. Calibration's weakness is that it is compatible with very unequal treatment. A score can be perfectly calibrated within groups and still produce very different error rates across them, if the groups have different base rates. It can also be calibrated while being much less informative for one group, assigning everyone in that group scores close to the group average while discriminating sharply among members of the other group. In that case members of the first group who are in fact low risk are not distinguished from those who are high risk, which is a real harm even though the scores are, in a narrow sense, accurate. Beyond binary groups The metrics above are usually presented for two groups, but real deployments involve many. Race and ethnicity categories vary across jurisdictions and datasets; sex and gender are not always binary; age is continuous. Extending the metrics to multiple groups is mechanically straightforward: compute the rate for every group and compare, typically by reporting the ratio of the lowest to the highest or the largest pairwise difference. The harder extension is to intersections. Kimberlé Crenshaw introduced the term "intersectionality" in 1989 to describe how Black women faced discrimination that could not be understood as the sum of race discrimination and sex discrimination considered separately. The same structure appears in algorithmic systems. A model can satisfy demographic parity with respect to race and with respect to sex while badly failing a subgroup defined by both, a phenomenon Michael Kearns and colleagues named "fairness gerrymandering" in a 2018 paper that also proposed methods for auditing and training against it over rich classes of subgroups.5 The facial analysis audits discussed in Chapter 8 found their starkest disparities at exactly such an intersection. The obstacle to intersectional auditing is statistical. Each intersection divides the data further, and small subgroups yield noisy estimates. A difference in false positive rates between two subgroups of fifty people each may be indistinguishable from chance. Responsible reporting therefore includes confidence intervals, not just point estimates, and treats a large apparent disparity in a small group as a reason to collect more data rather than a reason to dismiss the finding. Rankings and the problem of exposure Many consequential systems do not make yes-or-no decisions at all. They rank. A recruiting platform orders candidates for a recruiter's search; a marketplace orders sellers; a search engine orders results; a lender's queue orders applications for manual review. The binary metrics above do not apply directly, because nobody is formally rejected. But position matters enormously. People examine the top of a list far more than the bottom, so a candidate ranked fifth receives much more attention than one ranked fiftieth even if both are technically "included". Ashudeep Singh and Thorsten Joachims formalised this in 2018 with the idea of fairness of exposure.6 Each position in a ranking carries a certain expected amount of attention, which declines steeply down the list. A ranking can then be evaluated by how much total exposure it gives each group. One natural criterion asks that exposure be distributed in proportion to the groups' presence among relevant candidates; another asks that exposure be proportional to merit as estimated by the system, so that a group with the same average relevance receives the same average attention. The distinction mirrors the one between demographic parity and the outcome-conditioned criteria. The first ignores relevance; the second accepts the system's relevance estimates and asks only that attention follow them consistently. There are practical examples of ranking fairness being applied at scale. Engineers at LinkedIn described in 2019 a fairness-aware re-ranking method deployed in its recruiter search product, designed so that the gender composition of the top results for a query reflected the gender composition of the qualified candidates matching that query.7 The method did not alter the underlying relevance model; it adjusted the order of results after scoring, so that one group was not systematically pushed below the point where recruiters stop looking. Reported tests found the change made the top results substantially more representative without a measurable loss in recruiter engagement. Rankings also show why small score differences matter. If two groups' relevance scores overlap heavily but one is shifted slightly lower, a binary threshold might treat them similarly, since most members of both groups fall on the same side of it. A ranking, by contrast, converts that slight shift into a systematic difference in position, and a steep attention curve converts the position difference into a large difference in exposure. A bias that would barely register in a classifier's confusion matrix can dominate who gets seen. From metric to decision None of these metrics is a verdict on its own. A model that fails the four-fifths rule may be justified if the selection criterion is genuinely related to the job and no less discriminatory alternative exists; that is how US disparate-impact doctrine has worked since Griggs v. Duke Power Co. in 1971. A model that satisfies equalised odds may still be unfair if the labels against which its errors are measured are corrupt. A calibrated model may still be harmful if it is far less informative for one group. What the metrics provide is precision about what is being claimed. "This model is biased against women" is an accusation that cannot be evaluated. "This model's true positive rate for qualified female applicants is twelve points lower than for qualified male applicants, with a 95 percent confidence interval of seven to seventeen points" is a finding that can be checked, contested and acted on. The trouble, as the next chapter shows, is that the criteria cannot all be satisfied together, so that choosing which numbers to report is already a substantive ethical decision. CHAPTER 4 When Fairness Conflicts with Fairness In May 2016 the investigative newsroom ProPublica published an article by Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner under the headline "Machine Bias."1 It examined COMPAS, a commercial risk assessment tool sold by a company then called Northpointe and used by courts in several US states to inform decisions about pretrial release, supervision and, in some jurisdictions, sentencing. The reporters obtained risk scores for roughly seven thousand people arrested in Broward County, Florida, in 2013 and 2014, and checked how many were charged with new crimes over the following two years. Their central finding concerned error rates. Among defendants who did not go on to reoffend, Black defendants were far more likely than white defendants to have been labelled higher risk. Among defendants who did reoffend, white defendants were far more likely than Black defendants to have been labelled lower risk. Table 3 gives the figures ProPublica reported for the tool's general recidivism predictions. Table 3. COMPAS error rates by race in Broward County, as reported by ProPublica. Measure White defendants Black defendants Labelled higher risk, but did not reoffend 23.5% 44.9% Labelled lower risk, yet did reoffend 47.7% 28.0% Source: Angwin, Larson, Mattu and Kirchner, "Machine Bias," ProPublica (2016). In the language of the previous chapter, the false positive rate for Black defendants was nearly double that for white defendants, and the false negative rate for white defendants was substantially higher. The tool failed equalised odds, badly. ProPublica concluded that it was biased against Black defendants. Northpointe replied with a technical report arguing that ProPublica had used the wrong standard.2 The company showed that the tool satisfied predictive parity: among people given a high-risk label, the share who reoffended was about the same for Black and white defendants, and the scores were similarly calibrated across groups. A score of seven meant roughly the same probability of rearrest regardless of race. By that standard, the company argued, the tool was fair. Both analyses were, in their own terms, correct. That was the unsettling part, and it set off a burst of research that produced one of the most important results in the field. The impossibility theorems Within months, two independent papers showed that the conflict between ProPublica and Northpointe was not an artefact of this particular tool but a mathematical necessity. Alexandra Chouldechova, a statistician at Carnegie Mellon University, demonstrated that if two groups have different base rates of the outcome, then a classifier that satisfies predictive parity cannot also have equal false positive rates and equal false negative rates across the groups, unless it is perfect.3 The proof is short. The false positive rate, the false negative rate, the positive predictive value and the base rate are linked by an identity that follows directly from the definitions. Fix the positive predictive value and the false negative rate to be equal across groups, and the false positive rates are forced apart in proportion to the difference in base rates. Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan proved a closely related result for continuous risk scores.4 They considered three conditions: calibration within groups; balance for the positive class, meaning that people who actually experience the outcome receive the same average score in each group; and balance for the negative class, meaning the same for people who do not. They showed that all three can hold together only in two degenerate cases: when the predictor is perfect, or when the groups have exactly equal base rates. In every realistic situation, at least one must fail. In Broward County, the recorded rearrest rate was higher for Black defendants than for white defendants. Given that difference, no tool, however carefully designed, could have satisfied both Northpointe's standard and ProPublica's. Fixing the error-rate disparity would have required breaking calibration, so that a score of seven meant something different for Black and white defendants. Preserving calibration guaranteed the error-rate disparity. It is worth stating the intuition without algebra. Suppose a tool assigns scores that mean the same thing for everyone. If one group has more people who will in fact be rearrested, then more of that group will receive high scores, because the scores track the outcome. Among the people in that group who will not be rearrested, some will inevitably have characteristics that resemble those who will, and since the group has more high scorers overall, more of those non-reoffenders will be swept up into the high-score range. Equal meaning for scores produces unequal error rates. The only escape is to make the scores mean different things for different groups, or to have a perfect tool, or to have equal base rates. The arithmetic is worth seeing once. Chouldechova's identity states that the false positive rate equals the ratio of positives to negatives in the group, multiplied by the ratio of wrong to right positive predictions, multiplied by the true positive rate. Take two groups of 1,000 people each. In group A, 500 people will experience the outcome; in group B, 300 will. Suppose a tool is equally good for both in two senses: it catches 70 percent of the people who will experience the outcome in each group, and 70 percent of the people it flags in each group turn out to experience it. In group A the tool catches 350 of the 500, and since 70 percent of its flags are correct, it must flag 500 people in total, of whom 150 are false positives. There are 500 people in group A who will not experience the outcome, so the false positive rate is 150 out of 500, or 30 percent. In group B the tool catches 210 of the 300, flags 300 people in total, and so makes 90 false positives among the 700 negatives, a false positive rate of about 13 percent. Equal sensitivity and equal precision, combined with unequal base rates, have produced a false positive rate in group A more than twice that in group B. Nothing about the tool's design produced this; it follows from the counting. What the impossibility does and does not say The impossibility results are sometimes read as a counsel of despair: if fairness is mathematically impossible, why try? This reading is mistaken in three ways. First, the results do not say that fairness is impossible. They say that several specific fairness criteria cannot all be met simultaneously when base rates differ. One of them can still be met, and others can be approximately met. The consequence is that a choice must be made, and the choice must be justified by reference to something outside the mathematics. Second, the conflict depends on the base rates, and the base rates are not facts of nature. In the COMPAS case the base rate was the recorded rate of rearrest, and as Chapter 1 discussed, arrest rates reflect policing patterns as well as behaviour. If part of the difference in base rates is an artefact of differential enforcement, then insisting on calibration to those rates means calibrating to a biased label. The impossibility theorem takes the labels as given. A critic of the labels can reasonably reject the premise. Third, the results clarify what is at stake in choosing. Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel and Aziz Huq showed in a 2017 paper that, for a given risk model, the decision rule that maximises public safety subject to a fixed detention rate is to apply a single threshold to all defendants, and that this rule generally violates error-rate parity.5 Satisfying error-rate parity would require using different thresholds for different groups, which means detaining some lower-risk people from one group while releasing some higher-risk people from the other. That trade has costs that can be counted, in crimes not prevented or in people needlessly detained, and those costs fall on identifiable people. Corbett-Davies and Goel later argued that many popular fairness criteria, applied mechanically, can harm the very groups they are meant to protect.6 Others have argued with equal force that a single threshold applied to scores trained on biased labels simply launders historical injustice. The disagreement is real, and the mathematics makes its terms explicit. Individual fairness and counterfactual fairness The criteria discussed so far are all statistical properties of groups. They say nothing directly about how any particular person is treated. Two influential alternatives start from the individual instead. Dwork and her colleagues proposed in 2012 that a fair classifier should treat similar individuals similarly.7 Formally, they asked for a Lipschitz condition: the difference between two people's outcomes should be no greater than the distance between them according to some task-specific similarity metric. The idea captures something group metrics miss, the intuition that two applicants with essentially identical qualifications should receive essentially identical decisions whatever their group. Its weakness is that it pushes the entire ethical question into the similarity metric. Deciding which differences between people are relevant to a loan decision, and which are not, is precisely the contested question. Individual fairness provides a framework for stating an answer, not the answer itself. Counterfactual fairness, introduced by Matt Kusner, Joshua Loftus, Chris Russell and Ricardo Silva in 2017, asks a different question: would this person's prediction have been the same if they had belonged to a different group, with everything causally downstream of group membership adjusted accordingly?8 Answering requires a causal model of how the protected attribute influences the other variables. If race influences neighbourhood, which influences school quality, which influences test scores, then a counterfactually fair predictor cannot rely on test scores in a way that transmits that influence. The approach has the great merit of distinguishing between legitimate and illegitimate pathways, which statistical criteria cannot do. Its weakness is that the causal model must be specified by hand, and different plausible causal models yield different answers. It also raises philosophical difficulties about what it means to imagine a person with a different race or sex, holding "everything else" fixed, when so much of a life is shaped by those characteristics. Choosing a criterion Since no single criterion can be universally right, the choice has to be made case by case, and a few questions do most of the work. The first is what kind of harm the decision can do. Where the decision allocates a benefit, such as a job interview, a loan, a place in a care programme or a scholarship, the most salient harm is a qualified person being passed over. That points toward equal opportunity: qualified people in every group should have an equal chance of being selected. Where the decision imposes a burden, such as pretrial detention, a fraud investigation, a child-welfare inquiry or additional security screening, the most salient harm is an innocent person being wrongly burdened. That points toward equal false positive rates. Where the score will be read by a professional who must combine it with other information, such as a clinician using a risk estimate, calibration matters most, because the professional needs the score to mean the same thing for every patient. The second is how trustworthy the labels are. Where the outcome is measured accurately and equally for all groups, criteria that condition on the outcome, such as equalised odds or calibration, are well founded. Where the outcome is itself a product of biased processes, as with arrests, performance reviews or historical hiring decisions, conditioning on it builds the bias into the definition of fairness. In those cases a criterion that ignores the labels, such as demographic parity, or a deliberate correction of the labels, may be more defensible. The third is what the decision is for. Some allocations are explicitly intended to correct existing disadvantage; others are intended to identify the most qualified candidates by a fixed standard. A scholarship programme aimed at widening participation is supposed to change who gets in. A safety-critical licensing exam is not. The fairness criterion should follow the purpose, and the purpose is a policy question for the institution, not a technical question for the data scientist, though the data scientist should insist that it be answered. The fourth is the law. In the United States, demographic disparities in hiring trigger the four-fifths rule and disparate-impact analysis; explicit use of race to equalise outcomes can itself be unlawful disparate treatment, as the Supreme Court's 2009 decision in Ricci v. DeStefano and its 2023 decision on affirmative action in university admissions illustrate. In the European Union, non-discrimination law takes a somewhat different approach, and the AI Act adds its own requirements. A technically elegant solution that is illegal is not a solution. The question behind the question The COMPAS debate concentrated so heavily on which fairness criterion to apply that a more basic question received less attention than it deserved: how good was the tool at predicting anything? In 2018 Julia Dressel and Hany Farid published a study in Science Advances that used the same Broward County data.9 They recruited several hundred people through an online crowdsourcing platform, none with criminal justice expertise, and showed each a short description of a defendant containing seven pieces of information, including sex, age and criminal history, but not race. Asked to predict whether the defendant would reoffend within two years, the participants were correct about 67 percent of the time when their answers were pooled, compared with about 65 percent for COMPAS. A simple logistic regression using only two inputs, the defendant's age and number of prior convictions, matched the commercial tool's accuracy. The participants' predictions, like the tool's, showed racial disparities in false positive rates, even though race was not shown to them, because the inputs they did see were correlated with race. The finding is important for fairness in two ways. First, it shows that the impossibility results bite hardest when predictors are weak. The theorems say that the conflict disappears only for a perfect predictor; the further a tool is from perfect, the larger the error-rate disparities that calibration forces when base rates differ. A tool that adds little to what age and prior record already reveal carries the full weight of those disparities while contributing little predictive value in return. Second, it shifts the burden of justification. A proprietary system that is no more accurate than a two-variable formula, or than untrained members of the public, is hard to defend on grounds of expertise, and its opacity becomes a cost with no compensating benefit. Before arguing about how a tool's errors should be distributed, it is worth asking whether its predictions are good enough to justify using it at all. Living with the trade-off The impossibility results changed the field because they made it impossible to pretend that fairness was a single property waiting to be optimised. They forced a recognition that building a fair system involves choosing among values, and that the choice cannot be delegated to an algorithm. For practitioners, the practical consequence is a discipline of reporting. A fairness assessment that reports one metric invites the suspicion that the others were bad. A better practice is to report the main criteria together, to state which one the team chose to prioritise and why, and to quantify what that choice costs on the others. In the COMPAS case, such a report would have stated that the tool was calibrated across racial groups, that as a necessary consequence its false positive rate for Black defendants was nearly twice that for white defendants, and that the court system using it had decided, or had failed to decide, which of these mattered more. Neither ProPublica nor Northpointe's framing would then have come as a surprise. These group-level metrics, however, describe only what a system does with its inputs. They say nothing about how the bias got inside it. To see that, the next chapter goes beneath the predictions to the representations from which they are computed. CHAPTER 5 Bias in Geometry: How Latent Spaces Encode Society Modern machine learning systems rarely operate on raw inputs directly. A language model does not process letters; it maps words or word fragments to vectors of several hundred or several thousand numbers. An image classifier transforms pixels into progressively more abstract internal representations. A recommendation system represents each user and each item as a point in a shared space, so that the distance between them predicts whether the user will like the item. These learned spaces are usually called embedding spaces or latent spaces, "latent" because their dimensions are not specified by any designer but discovered by the training process. The usefulness of latent spaces comes from the fact that their geometry carries meaning. Items that behave similarly in the training data end up close together. Relationships between items become directions: the displacement from one point to another can encode a consistent semantic relationship that applies across many pairs. This is what makes the representations powerful, and it is also what makes them a vehicle for bias. If the training data associates certain concepts with certain groups of people, those associations take the form of geometric structure, and every system built on top of the representation inherits that structure. Words as vectors The modern history of this problem begins with word embeddings. In 2013 a team at Google led by Tomas Mikolov released word2vec, a method for training dense vector representations of words from large text corpora. The core idea, older than word2vec but popularised by it, is the distributional hypothesis: words that appear in similar contexts tend to have similar meanings. A model trained to predict a word from its neighbours, or its neighbours from the word, ends up assigning similar vectors to words that occur in similar contexts. "Doctor" and "physician" end up close together. So do "Paris" and "Lyon". The feature that caught public attention was the arithmetic of analogies. Taking the vector for "king", subtracting the vector for "man" and adding the vector for "woman" produced a point closest to "queen". The vector from "France" to "Paris" was approximately parallel to the vector from "Italy" to "Rome". The model had learned, without supervision, that there is a consistent direction in the space corresponding to the relationship between a country and its capital, and another corresponding to the relationship between male and female forms. It did not take long for researchers to ask what else those directions encoded. In 2016 Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama and Adam Kalai published a paper whose title became one of the most quoted phrases in the field: "Man is to Computer Programmer as Woman is to Homemaker?"1 Working with word2vec embeddings trained on Google News articles, they generated analogies along the direction from "he" to "she" and asked crowd workers to judge them. Many were appropriate, such as "brother" to "sister". Many others reflected occupational and social stereotypes: the analogy that gave the paper its title, "surgeon" to "nurse", "architect" to "interior designer". They showed that gender-neutral occupation words were systematically arranged along the gender direction, with words like "homemaker", "nurse" and "receptionist" toward the female end and "maestro", "skipper" and "architect" toward the male end. Measuring association: WEAT and its descendants The Caliskan, Bryson and Narayanan study introduced in Chapter 1 provided a rigorous measurement method to complement this qualitative picture. The Word Embedding Association Test works by computing cosine similarities, the standard measure of how closely two vectors point in the same direction.2 For two sets of target words, such as male and female names, and two sets of attribute words, such as career-related and family-related terms, it computes how much more each target word is associated with one attribute set than the other, sums these differences, and reports an effect size analogous to Cohen's d in psychology, together with a permutation test for statistical significance. The method's strength was that it borrowed its target and attribute word lists directly from published Implicit Association Test studies on human subjects, so the results could be compared with known human biases. The embeddings reproduced every bias the authors tested, including associations of European-American names with pleasant terms and African-American names with unpleasant ones, of men with science and mathematics, of women with arts, and of young people with pleasant terms relative to old people. WEAT became the template for a family of tests. When language representations moved from static word vectors to contextual ones, in which the same word receives a different vector depending on the surrounding sentence, Chandler May and colleagues adapted the method into the Sentence Encoder Association Test, inserting target and attribute words into simple template sentences.3 They found that the results for contextual encoders were less consistent than for static embeddings, which illustrates a recurring difficulty: as representations become more complex, bias measurement becomes more sensitive to the precise way the test is constructed. Why bias becomes a direction It is worth understanding why social bias should take such a clean geometric form. A word embedding is trained to capture co-occurrence statistics. If a word appears frequently near "he", "him", "his" and "man", and another appears frequently near "she", "her" and "woman", the training process will place them in different regions of the space, and the most efficient way to encode a consistent difference across many such words is a single direction. Because gendered pronouns and nouns are extremely common, the gender signal in text is strong and pervasive, and so the gender direction is one of the most prominent structures in the space. The same principle applies to any attribute that consistently shapes which words appear together. Race and ethnicity are encoded through names, places, cultural references and, unfortunately, through the language of crime reporting and social commentary. Religion is encoded through its associations with words related to violence, peace, holidays and practices. Age, class and disability all leave traces. Each produces a subspace, sometimes a single direction, sometimes a more diffuse structure, along which otherwise neutral words are arranged according to their statistical associations with the groups. The crucial point is that these structures are not separate from the useful information in the embedding. The same co-occurrence statistics that put "nurse" near "she" also put "nurse" near "hospital", "patient" and "care". A representation that faithfully captures how words are used will capture the social patterns in that usage, because the patterns are part of the usage. This is why, as the next chapter shows, removing bias from a representation is harder than it first appears. Contextual representations and large language models Since around 2018, static word embeddings have largely been superseded in natural language processing by contextual representations produced by transformer models such as BERT and, later, by the very large generative models now in widespread use. In these systems a word does not have a single vector. Its representation at each layer of the network depends on its context, and the representations at the final layers are used to predict the next token or to perform downstream tasks. The bias does not go away in this transition. It becomes more diffuse and more varied in form. Contextual models show stereotyped associations when probed with fill-in-the-blank tasks, when used for coreference resolution, and when generating text. Several benchmark datasets were developed to measure these effects. WinoBias, created by Jieyu Zhao and colleagues in 2018, tests whether coreference systems resolve pronouns differently depending on whether the occupation involved is stereotypically male or female: whether a system correctly identifies that "she" refers to "the physician" in a sentence where it should, as reliably as it identifies "he".4 The systems they tested performed markedly better on stereotype-consistent sentences. StereoSet and CrowS-Pairs measure whether models assign higher likelihood to stereotyped sentences than to minimally different anti-stereotyped ones. The Bias Benchmark for Question Answering, or BBQ, published by Alicia Parrish and colleagues in 2022, tests whether question-answering models fall back on stereotypes when the context is ambiguous, for example by assuming that the older person in a scenario is the one who is forgetful when the text does not say so.5 Machine translation provides an everyday demonstration of how these associations surface in products. Turkish, like Finnish, Hungarian and many other languages, uses a single gender-neutral third-person pronoun. When translation systems rendered such sentences into English, which forces a choice between "he" and "she", they for years tended to produce "he is a doctor" and "she is a nurse", "he is an engineer" and "she is a cook", following the statistical associations in their training text. The pattern was documented by researchers and widely circulated, and in late 2018 Google began offering both feminine and masculine translations for some gender-neutral queries in a limited set of languages. The fix addressed the output for single short sentences; it did not change the associations in the underlying representation, which continued to shape translations of longer passages where offering alternatives was impractical. These benchmarks have well-documented limitations. Su Lin Blodgett and colleagues reviewed a large body of work on bias in natural language processing in 2020 and found that many papers did not clearly state what harm they were measuring or why, and a follow-up study in 2021 found that several popular benchmarks contained ambiguous, inconsistent or mislabelled test items. A benchmark score is therefore a signal, not a certificate. But the cumulative evidence is consistent: models trained on human text represent and reproduce the associations in that text, and larger models do not automatically reduce them. Images and multimodal spaces The same geometry appears in vision and in models that combine vision and language. Models such as CLIP, released by OpenAI in 2021, learn a joint embedding space for images and text by training on hundreds of millions of image-caption pairs collected from the web. An image and a caption that describe the same thing end up close together. This makes the model highly flexible, since it can classify images into any categories described in words, but it also means that the associations in web captions are written into the space. Studies of these joint spaces have found that images of people are associated with occupation words, crime-related words and other attributes in ways that track race and gender. When such representations are used to guide image generation, the associations become visible. In 2023 Federico Bianchi and colleagues published a study of popular text-to-image systems titled "Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale."6 They found that prompts mentioning ordinary occupations or descriptors produced images that skewed sharply toward stereotypes, for instance generating predominantly lighter-skinned men for high-status jobs and darker-skinned people for prompts associated with poverty, often in ways more extreme than labour statistics would suggest. Prompts that did not mention race or gender at all produced images in which race and gender were strongly determined by the other words in the prompt. Generative systems make the latent-space problem directly visible in a way that classification systems do not. A classifier's bias shows up only in its error rates. A generator's bias is on the screen. People as points: recommendation and advertising Latent spaces are not only built for words and images. Recommendation and advertising systems represent people too. Each user becomes a vector learned from their clicks, purchases, follows and viewing history, and each item, advertisement or job listing becomes a vector in the same space. The system predicts engagement from proximity. Nobody tells it the users' race or sex, but those characteristics shape behaviour, and so they shape the geometry. Latanya Sweeney documented an early version of the consequences in 2013. Searching for names more typical of Black Americans on a major search engine was more likely to produce advertisements suggesting an arrest record, for a background-check company, than searching for names more typical of white Americans.7 The advertiser had supplied ad copy for both kinds of name; the delivery system, optimising for clicks, learned which version to show. Muhammad Ali, Piotr Sapiezynski and colleagues showed in 2019 how this works in a modern advertising platform.8 They ran their own advertisements on Facebook, with identical neutral targeting, and varied only the content. Job advertisements for lumber-industry positions were delivered to audiences that were overwhelmingly male and white; advertisements for supermarket cashier positions went to audiences that were mostly female; advertisements for taxi-driver jobs reached audiences with a high proportion of Black users. Housing advertisements showed similar skews by race depending on the images used. The advertisers had asked for no such targeting. The platform's delivery optimisation, predicting which users would engage with which content from their position in its learned representation, produced the segregation on its own. The legal consequences were concrete. Advertising for housing, employment and credit is covered by US anti-discrimination law, and in 2022 Meta settled a lawsuit brought by the US Department of Justice alleging that its ad delivery system discriminated in housing advertisements. Under the settlement the company agreed to build a system intended to reduce the gap between the eligible audience for a housing advertisement and the audience that actually saw it, measured by estimated sex and race and ethnicity. The case is one of the clearest legal acknowledgements that a latent-space system can discriminate without any input that mentions a protected characteristic, simply because its geometry has absorbed the patterns in its users' behaviour. Representation versus allocation It helps to distinguish two kinds of harm that biased representations can cause, a distinction drawn by Kate Crawford in a 2017 keynote and developed by Barocas, Crawford and others. Allocative harms occur when a system withholds a resource or opportunity from some group: a loan denied, a job advertisement not shown, a medical intervention not offered. Representational harms occur when a system reinforces the subordination of some group by the way it represents them: stereotyping, denigrating, erasing, or failing to recognise them. Biased embeddings can cause both. When a résumé-ranking system uses embeddings in which "engineer" sits closer to male-associated words, the result may be an allocative harm, with female applicants ranked lower. When an image generator depicts every doctor as a man, or a search engine's autocomplete suggests degrading completions for queries about a group, the harm is representational. It does not deny anyone a specific resource, but it shapes how people see themselves and others, and it often goes unmeasured because it does not appear in any decision log. Representational harms are harder to quantify, which makes them easy to discount. But as generative models come to produce a growing share of the images, text and summaries people encounter, the cumulative effect of their representational choices is likely to matter as much as the allocative decisions of any single classifier. Bias as a property of the space The lesson of this chapter is that in modern machine learning, bias is not only a property of outputs that can be checked with the metrics of Chapter 3. It is a property of the internal representations on which those outputs depend. Those representations are learned once and reused many times. A single pretrained language model or image encoder may underlie thousands of downstream applications, from résumé screening to customer service to content moderation, built by teams who never examine the representation itself. This concentration has a double implication. It means that bias in a widely used foundation model propagates widely, which raises the stakes for the organisations that build those models. And it suggests a point of leverage: if the bias can be removed from the representation, every system built on it might benefit at once. Whether that is actually possible, and how far current methods succeed, is the subject of the next chapter. CHAPTER 6 Debiasing the Latent Space If bias in a representation takes the form of geometric structure, the natural remedy is to change the geometry. Since 2016 a substantial body of research has pursued this idea, producing methods that range from simple linear projections to adversarial training to closed-form solutions with formal guarantees. This chapter explains how the main approaches work, what they achieve, and, just as importantly, what they leave untouched. The history of the area is instructive because it repeats a pattern seen throughout fairness work: an elegant method appears to solve the problem, closer inspection shows that it solved a narrower problem than claimed, and the next generation of methods is built on a more honest understanding of what "removing bias" can mean. Table 4 compares the principal families of methods before the chapter discusses each. Table 4. Principal approaches to removing protected information from learned representations. Approach Core mechanism What it guarantees Main limitation Hard debiasing (Bolukbasi et al., 2016) Project out a single bias direction from neutral words No component along that one direction Bias remains recoverable from neighbourhood structure Iterative nullspace projection (Ravfogel et al., 2020) Repeatedly train linear classifiers for the attribute and project out their directions Linear classifiers fail to predict the attribute Removes some useful information; nonlinear leakage remains Adversarial representation learning (Edwards and Storkey, 2016; Madras et al., 2018) Train an encoder to defeat an adversary that predicts the attribute Specific adversary is defeated during training Unstable training; new adversaries often succeed Linear concept erasure (Ravfogel et al., 2022; Belrose et al., 2023) Find the minimal linear change that makes the attribute linearly unpredictable Provable linear guardedness with minimal distortion Linear only Counterfactual data augmentation Train on data with group terms swapped Model sees balanced associations Swap lists are crude; cannot cover implicit cues Hard debiasing: projecting out a direction The first prominent method came from the same paper by Bolukbasi and colleagues that documented gender stereotypes in word embeddings.1 Their approach, which they called hard debiasing, had two steps. The first step was to identify the gender subspace. They took a set of definitional word pairs, such as "she" and "he", "her" and "his", "woman" and "man", "daughter" and "son", and computed the difference vector for each pair. Principal component analysis on these differences revealed that most of the variation lay along a single dominant direction, which they treated as the gender direction. The second step was to modify the embeddings in two ways. For words that should be gender-neutral, such as occupation words, they "neutralised" the embedding by removing its component along the gender direction, projecting it onto the subspace orthogonal to that direction. After this step, "nurse" and "engineer" had no projection on the gender direction at all. For sets of words that differ only in gender, such as "grandmother" and "grandfather", they "equalised" the embeddings so that each word in the set was equidistant from every neutral word. "Babysit" would then be equally close to "grandmother" and "grandfather". The results looked impressive. Generated analogies along the gender direction became far less stereotyped according to crowd-worker ratings, and performance on standard word-similarity and analogy benchmarks was largely preserved. The method was simple, cheap and easy to apply to any pretrained embedding. Lipstick on a pig In 2019 Hila Gonen and Yoav Goldberg published a paper with a deliberately provocative title: "Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them."2 They showed that hard debiasing removed the projection of neutral words onto the gender direction but left the rest of the geometry largely intact. Their tests were straightforward. They took the words that had been most strongly associated with each gender before debiasing and checked whether those words were still distinguishable after it. They were. Words that had been male-biased still clustered together, and female-biased words still clustered together, with clustering recovering the original gender association with high accuracy. A simple classifier trained on a subset of words could predict the original gender bias of held-out words from their debiased embeddings with high accuracy. The nearest neighbours of a previously female-associated occupation were still mostly other previously female-associated words. The explanation is geometric. The gender direction captures one component of how gender is encoded, but gendered associations also affect a word's relationship to many other words. "Nurse" is near "receptionist" and "housekeeper" not only because all three have a component along the gender direction but because they share contexts in the training text, and those shared contexts are themselves shaped by gender. Removing the projection on one axis does not move the words away from each other. The bias was not a single direction; it was distributed across the neighbourhood structure of the whole space. The paper did not show that hard debiasing was useless. It reduced some measurable effects, and for applications that relied only on projections onto the gender direction it may have been sufficient. But it showed that the measure used to validate the method, the projection itself, was too narrow to establish that bias had been removed. This lesson, that a debiasing method should be judged by tests other than the one it was designed to pass, recurs throughout the area. Iterative nullspace projection Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton and Yoav Goldberg responded in 2020 with a more aggressive linear method called Iterative Nullspace Projection, or INLP.3 Rather than defining the bias direction by hand from a list of word pairs, INLP learns it from data. The procedure works as follows. Train a linear classifier to predict the protected attribute from the representations. The classifier's weight vector identifies a direction along which the attribute is linearly encoded. Project all representations onto the nullspace of that classifier, removing the information it used. Then train a new linear classifier on the projected representations; if it can still predict the attribute, it has found another direction. Project that out too, and repeat until no linear classifier can predict the attribute better than chance. INLP removes far more than hard debiasing. After enough iterations, linear probes can no longer recover the attribute, and the clustering tests that defeated hard debiasing are much less successful. The method can be applied to contextual representations from models like BERT as well as to static embeddings, and to attributes other than gender. Its costs are also clearer. Each iteration removes a full direction from the space, and the classifiers may pick up directions that carry useful information correlated with the attribute, so downstream performance can degrade. It is also restricted to linear information. A nonlinear classifier, such as a small neural network, can often still recover the attribute from INLP-processed representations. Whether that matters depends on whether the downstream system is linear. A linear classifier trained on top of the representation cannot exploit nonlinear encodings; a neural network fine-tuned on it can. Adversarial representation learning A different line of work, which developed in parallel and partly predates the embedding-debiasing literature, attacks the problem during training rather than after it. The idea is to learn representations that are useful for the main task but from which the protected attribute cannot be predicted. Richard Zemel and colleagues proposed one of the first versions in 2013, with "Learning Fair Representations", which mapped individuals to a set of prototypes chosen to preserve information about the outcome while obscuring group membership.4 Harrison Edwards and Amos Storkey introduced the adversarial formulation in 2016: an encoder produces a representation, a predictor uses it for the main task, and an adversary tries to predict the protected attribute from it.5 The encoder is trained to help the predictor and hinder the adversary. David Madras, Elliot Creager, Toniann Pitassi and Richard Zemel extended this in 2018 with LAFTR, showing how different adversarial objectives correspond to different group fairness criteria, so that the representation could be tuned toward demographic parity or toward equalised odds.6 Brian Hu Zhang, Blake Lemoine and Margaret Mitchell applied a related adversarial approach in 2018 directly to model predictions, with the adversary trying to infer the protected attribute from the output, and also demonstrated it on word-embedding analogies.7 Adversarial methods are attractive because they are flexible. They can in principle remove nonlinear as well as linear information, and they can target a specific fairness criterion. In practice they have two serious weaknesses. Adversarial training is notoriously unstable, sensitive to hyperparameters and prone to oscillation. More fundamentally, defeating the adversary used during training does not mean the information is gone. Yanai Elazar and Yoav Goldberg showed in 2018 that after adversarial training to remove demographic attributes from text representations, a fresh classifier trained after the fact could still recover the attributes well above chance.8 The encoder had learned to hide the information from one particular adversary, not to remove it. Concept erasure with guarantees The most recent generation of methods returns to the linear setting but aims for formal guarantees. Ravfogel and colleagues introduced Relaxed Linear Adversarial Concept Erasure, or R-LACE, in 2022, which frames the problem as a minimax game between a projection that removes a low-rank subspace and a linear classifier that tries to recover the attribute, solved with a convex relaxation.9 It removes the attribute with fewer dimensions than INLP and less damage to other information. Nora Belrose and colleagues went further in 2023 with LEACE, for LEAst-squares Concept Erasure.10 They proved that a representation is "linearly guarded" against an attribute, meaning that no linear classifier can predict the attribute better than a constant predictor, if and only if the class-conditional means of the representation are equal across attribute values. They then derived, in closed form, the transformation that achieves this condition while moving the representations as little as possible in a precise least-squares sense. The method requires no iterative training, is cheap to compute, and comes with a mathematical guarantee: after LEACE, no linear model can exploit the attribute. The authors also showed that the method could be applied at every layer of a large language model, a procedure they called concept scrubbing, and measured how much this affected the model's performance. That opens the door to debiasing internal representations of deployed models, not only the static embeddings of a decade ago. LEACE is the cleanest answer so far to the question of linear removal. But it remains, by design, a linear guarantee. Nonlinear information can survive, and large neural networks are very good at exploiting nonlinear structure. Data-side approaches All of the methods above operate on the representation after it has been learned or modify the training objective. A complementary approach changes the data. Counterfactual data augmentation creates copies of training examples with protected-group terms swapped, so that for every sentence about "he" being a doctor there is a matching one about "she", and trains on the combined data. Variants have been applied to word embeddings, coreference systems and language models. The approach is simple and often effective at reducing measured stereotype associations, but its swap lists are necessarily crude. Names, dialect features, cultural references and countless subtler cues carry demographic information that no list captures, and swapping words can produce sentences that are nonsensical or that erase real differences in experience. For generative image models, analogous techniques include rebalancing the training data, adjusting the guidance signal during generation, and editing the directions in the model's latent space associated with demographic attributes. Several commercial providers have also intervened at the prompt level, rewriting user prompts to add demographic diversity. That approach produced a well-publicised failure in early 2024, when Google paused its Gemini model's ability to generate images of people after it produced racially diverse depictions of historical figures where such diversity was historically inaccurate. The episode illustrates a general risk: a surface-level fix applied uniformly, without understanding the context in which demographic attributes are and are not relevant, can produce new errors as conspicuous as the ones it was meant to correct. Overt and covert bias in aligned language models Large language models add a further layer to the story. Before release, most such models undergo additional training, often using human feedback, to make them more helpful and to reduce harmful outputs. Among other things, this teaches them to refuse stereotyping requests and to avoid overtly prejudiced language. Measured on benchmarks that ask about groups explicitly, the resulting models often look much less biased than their pretrained predecessors. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky and Sharese King tested whether this improvement goes all the way down. In a study published in Nature in 2024, they compared how language models responded to the same content written in African American English and in Standardised American English, without ever mentioning race.11 The models associated speakers of African American English with negative stereotypes more strongly than any recorded human stereotypes about African Americans in decades of survey research. When asked to match speakers to jobs, the models assigned the African American English speakers to less prestigious occupations; in hypothetical criminal cases, they were more likely to convict them and, in a hypothetical capital case, to recommend death. At the same time, when asked directly about African Americans, the same models produced much more positive associations. The training that suppressed overt prejudice had left the covert, dialect-based prejudice in place, and in some cases the gap between the two grew with the amount of human-feedback training. The parallel with Gonen and Goldberg's result is exact. In both cases a method succeeded on the test it was designed to pass, the projection onto a gender direction in one case and explicit questions about race in the other, while the underlying association survived in a form the test did not look for. Alignment training, like hard debiasing, can teach a system to avoid saying the wrong thing without changing what it has learned. What debiasing can and cannot do Several general lessons emerge from a decade of work on latent-space debiasing. The first is that "removing bias" from a representation is an ambiguous instruction. It can mean removing the ability of a linear probe to recover a protected attribute, removing stereotyped associations as measured by a particular test, removing differences in downstream task performance, or changing the behaviour of a generative system. These are different goals, and a method that achieves one may do little for the others. Seraphina Goldfarb-Tarrant and colleagues showed in 2021 that intrinsic bias measures in embeddings, such as WEAT scores, did not reliably correlate with bias in downstream applications built on them. Reducing the measurable bias in the representation did not reliably reduce the bias in what the system did. The second is that protected information is usually entangled with useful information, so that removing it has costs. A representation from which gender has been perfectly erased cannot help resolve "she" to the right noun, or answer questions about gendered medical conditions. Debiasing must be targeted to the information that should not influence a given decision, which depends on the application. The third is that representation-level debiasing addresses only one point in the pipeline described in Chapter 2. It does nothing about biased labels, biased target choices or biased deployment. A perfectly debiased embedding used to predict a biased outcome will learn the bias again from the labels. The fourth, and most important, is that every debiasing claim should be tested against measures other than the one used to design the method. Hard debiasing passed the projection test and failed the clustering test. Adversarial training defeated its own adversary and failed against a new one. Any team using these techniques should expect the same pattern and design its evaluation accordingly, with a range of probes, downstream task metrics broken down by group, and the group fairness criteria from Chapter 3 computed on the final system's actual decisions. Debiasing the latent space is one useful tool. It is not, on its own, a fair system. CHAPTER 7 Intervening Across the Pipeline Debiasing a representation is one intervention among many. The broader toolkit of fair machine learning is conventionally divided into three families according to where in the pipeline they act: pre-processing methods change the training data, in-processing methods change the learning algorithm, and post-processing methods change the predictions after the model is trained. To these should be added a fourth family, which the technical literature discusses less often but which often matters most: changing the problem itself, by reconsidering the target, the inputs or the decision that the model is meant to support. This chapter takes each family in turn, with attention to what each method requires, which fairness criteria it can achieve, and what it costs. The aim is not a catalogue but a way of thinking about which lever to pull. Pre-processing: changing the data Pre-processing methods transform the training data so that a standard learning algorithm, applied to the transformed data, produces fairer predictions. Their great practical advantage is that they leave the modelling pipeline untouched; a team can apply them without changing its training code. The simplest is reweighing, proposed by Faisal Kamiran and Toon Calders.1 Each training example is assigned a weight so that, in the weighted data, the protected attribute and the label are statistically independent. If positive outcomes are underrepresented among group B in the original data, examples of group B with positive labels receive higher weights, and examples of group B with negative labels lower ones. A model trained on the weighted data sees a world in which group membership does not predict the outcome. Kamiran and Calders also proposed "massaging" the data by changing the labels of selected examples near the decision boundary, and resampling as an alternative to weighting. These methods target demographic parity, since they aim to break the dependence between group and outcome. A second approach transforms the features rather than the labels. Michael Feldman, Sorelle Friedler, John Moeller, Carlos Scheidegger and Suresh Venkatasubramanian proposed in 2015 a method for "repairing" features so that their distributions are the same across groups while preserving the rank order of individuals within each group.2 If credit scores are distributed differently for two groups, the repair maps each person's score to the corresponding quantile of a common distribution, so that the person at the 70th percentile of group A and the person at the 70th percentile of group B end up with the same repaired score. A tunable parameter controls how complete the repair is, allowing a team to trade fairness against fidelity to the original data. The fair representation learning methods discussed in the previous chapter, such as Zemel's prototypes, also belong to this family. Pre-processing methods have important limitations. They generally require the protected attribute at training time, which raises legal and privacy issues in some settings. They act only on the training distribution and offer no guarantee about the fairness of predictions on new data. And because they target demographic parity, they are open to the objections discussed in Chapter 3: where outcomes genuinely differ for reasons that should be respected, breaking the dependence between group and label may reduce accuracy in ways that harm the people the system serves. In-processing: changing the objective In-processing methods build fairness into the training procedure, usually by adding a constraint or penalty to the learning objective. Instead of minimising prediction error alone, the algorithm minimises error subject to a requirement that some fairness metric stay within a tolerance, or minimises a combination of error and a fairness penalty. Early methods of this kind were tied to particular model types. Muhammad Bilal Zafar and colleagues proposed in 2017 constraints for logistic regression and support vector machines that limited the covariance between the protected attribute and the decision boundary, as a tractable approximation to demographic parity or to equal error rates. Toshihiro Kamishima and colleagues proposed a "prejudice remover" regulariser that penalised the mutual information between predictions and the protected attribute. A more general framework came from Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford and Hanna Wallach in 2018, in "A Reductions Approach to Fair Classification."3 They showed that the problem of finding the most accurate classifier subject to a broad class of fairness constraints, including demographic parity and equalised odds, could be reduced to a sequence of ordinary weighted classification problems. Any standard learning algorithm that accepts sample weights can be used as a black box, and the method returns a randomised combination of classifiers that satisfies the constraint up to a specified tolerance. This approach underlies the mitigation algorithms in Fairlearn, the open-source toolkit originally developed at Microsoft, which, together with IBM's AI Fairness 360, made many of the methods in this chapter accessible to practitioners without requiring them to implement the algorithms from scratch. The adversarial methods from the previous chapter are also in-processing methods when applied to the predictions directly, as in the Zhang, Lemoine and Mitchell approach of training a predictor alongside an adversary that tries to infer the protected attribute from its outputs. In-processing methods offer the most direct control over the trade-off between accuracy and fairness, and can target any of the criteria that can be written as constraints on the confusion matrix. Their costs are complexity, the need for the protected attribute during training, and the fact that a constraint satisfied on training data may not hold on deployment data, especially if the population shifts. Post-processing: changing the decisions Post-processing methods leave the trained model untouched and adjust how its outputs are turned into decisions. The canonical example comes from the Hardt, Price and Srebro paper that introduced equal opportunity.4 Given a trained risk score and the protected attribute, they showed how to choose group-specific thresholds, with randomisation where necessary, so that the resulting decisions satisfy equalised odds or equal opportunity while losing as little accuracy as possible. The geometry of the solution is intuitive. For each group, plot the trade-off between true positive rate and false positive rate as the threshold varies, which is the group's receiver operating characteristic curve. Equalised odds requires choosing an operating point that is achievable for every group. If one group's curve dominates another's, meaning the score is more informative for that group, the shared operating point must be one achievable by the weaker curve, and the method may need to randomise decisions for the stronger group to bring it down to that point. Equal opportunity is less demanding, requiring only equal true positive rates, and can usually be met by threshold adjustment alone. Post-processing is the easiest family to implement and the easiest to audit. It requires only the scores, the labels and the group attribute, and it can be applied to a model the team did not build, such as a vendor's black box. Geoff Pleiss and colleagues showed in 2017, however, that post-processing for equalised odds is in general incompatible with calibration, as the impossibility results would predict, and that the attempt to reconcile them can require withholding information from some predictions.5 Post-processing also faces a legal obstacle that is sometimes decisive. It requires using the protected attribute at decision time, explicitly treating people differently according to their group. In many jurisdictions and many domains, especially US employment and credit, that is unlawful disparate treatment. A technique that is mathematically optimal can be legally unavailable, and teams need to understand the law before they choose. A worked adjustment, and the hidden choice among equal models Return to the hiring example from Chapter 3. The model advanced 250 of 500 qualified applicants from group A and 120 of 300 qualified applicants from group B, a true positive rate of 50 percent against 40 percent. Suppose the team decides that equal opportunity is the right criterion, since the decision allocates a benefit and the harm of concern is qualified people being overlooked. Lowering the threshold for group B until its true positive rate reaches 50 percent means advancing 150 qualified applicants from that group instead of 120. Because the threshold drops, some unqualified applicants also cross it: if the score separates qualified from unqualified applicants in group B about as well as before, perhaps 75 or 80 unqualified applicants are now advanced instead of 60. Group B's false positive rate rises from 10 percent to around 13 percent, and its positive predictive value falls slightly. The cost of the adjustment is concrete and countable: a few dozen additional interviews with candidates who would not have succeeded, in exchange for thirty additional qualified candidates who would otherwise have been screened out. Whether that is a good trade is a judgement the organisation must make, but it can make it with the numbers in front of it. There is a less visible choice that precedes any such adjustment. For most real prediction problems there is not one best model but many, trained with different random seeds, slightly different feature sets or different algorithms, all with nearly identical overall accuracy. Statisticians call this the Rashomon effect, after the film in which several witnesses give equally plausible but incompatible accounts of the same event. Emily Black, Manish Raghavan and Solon Barocas drew out its implications for fairness in 2022.6 Models that are indistinguishable in accuracy can differ considerably in their disparities across groups, and can disagree about the predictions for many individuals. Choosing among them by accuracy alone, or by whichever happened to train first, is effectively choosing their fairness properties at random. This matters legally as well as ethically. In US disparate-impact doctrine, a practice with an adverse impact can be defended if it serves a legitimate business need, but the plaintiff can still prevail by showing that a less discriminatory alternative would serve the same need. Model multiplicity suggests that such alternatives will often exist and can be found by searching among comparably accurate models. Black, Raghavan and Barocas argued that this gives developers a duty, or at least a strong reason, to search deliberately for the least discriminatory model among those that perform equally well, rather than accepting the first one that meets the accuracy target. It is one of the cheapest fairness interventions available, because it requires no change to the data, no new constraint and no sacrifice of accuracy, only the decision to look. Changing the problem The three families above all accept the problem as posed: given these features, this label and this decision, make the predictions fairer. Some of the most effective interventions in the record step outside that frame. The first and most powerful is changing the target. The Obermeyer case from the Introduction is the clearest demonstration. No amount of reweighing, constraint or threshold adjustment applied to a model predicting cost would have addressed the core problem, which was that cost was a biased proxy for need. Changing the label to a combination of cost and direct measures of health did more than any of the standard methods could have done, with little loss in the tool's usefulness. The researchers and the manufacturer framed their fix as label choice, and argued that it deserved as much scrutiny as model design. The second is changing the inputs. Sometimes a feature carries so much historical bias, or so little relevant information, that the right response is to stop using it. Some US states have restricted the use of criminal history or credit history in hiring decisions; some lenders have experimented with "alternative data" such as rent and utility payment histories, which can allow people with thin credit files to demonstrate creditworthiness. Neither change is automatically fairer, since new data sources can carry their own biases and privacy costs, but they illustrate that the feature set is a design choice, not a given. The third is changing the decision. A risk score can inform many different interventions. A high pretrial risk score can lead to detention, or to a reminder system that reduces missed court dates, or to transport assistance. A high score in a child-welfare screening tool can lead to an investigation, or to an offer of voluntary support services. The same predictive model, with the same statistical biases, produces very different harms depending on whether a high score triggers a punishment or a benefit. Choosing supportive interventions over punitive ones can reduce the stakes of the model's errors, which is often more achievable than eliminating the errors themselves. The fourth is deciding not to build the system. Some problems should not be automated, either because the available data is too contaminated to support fair predictions, or because the decision is one that people have a right to have made by a human who can be held accountable, or because the cost of errors is too high relative to the benefit. The decision not to deploy is underused partly because it is rarely anyone's job to make it, a problem taken up in Chapter 9. The long run A final consideration complicates all of the above. Most fairness methods evaluate a system at a single point in time. But decisions affect people's futures, which affect future data. Lydia Liu, Sarah Dean, Esther Rolf, Max Simchowitz and Moritz Hardt showed in a 2018 paper, "Delayed Impact of Fair Machine Learning," that imposing a fairness constraint can, under plausible conditions, harm the group it is meant to protect over time.7 Their example was lending. If a constraint forces a lender to extend more loans to a disadvantaged group than an unconstrained lender would, and some of the additional borrowers default, their credit scores fall, potentially leaving the group worse off than before. Whether the constraint helps or hurts depends on the precise relationship between scores, repayment probabilities and the effects of default. The point is not that fairness constraints are harmful. It is that their effects play out in a dynamic world, and that a static fairness metric is an incomplete guide to whether a system is making people's lives better or worse. The result reinforces the case for monitoring outcomes over time and for thinking about the full consequences of a decision, not only its statistical properties at the moment it is made. Choosing a lever How should a team decide which intervention to use? A few principles follow from the material in this and earlier chapters. Start upstream. Before applying any mitigation algorithm, ask whether the target is the right one, whether the features are measured equally well for all groups, and whether the data represents the population the system will serve. Problems at these stages cannot be fully repaired downstream, and fixing them directly is usually cheaper and more robust. Choose the criterion before the method. Each technique targets particular criteria. Reweighing aims at demographic parity; threshold adjustment can achieve equal opportunity or equalised odds; calibration is most naturally preserved by leaving a well-built model alone. The choice of criterion, as Chapter 4 argued, depends on the type of harm, the reliability of the labels and the purpose of the decision. Check the law. Methods that use the protected attribute at decision time may be prohibited. Methods that use it at training time may be permitted, required or restricted, depending on the domain and jurisdiction. Measure on the deployed decision. Whatever method is used, the fairness metrics that matter are those computed on the actual decisions the system makes in use, on representative data, broken down by group and by intersections of groups, and tracked over time. A model that was fair in development and is unfair in production is an unfair model. Expect trade-offs and report them. Most mitigations reduce some measure of accuracy, and most improve some fairness criteria at the expense of others. These costs should be quantified and documented, not hidden, so that the people responsible for the decision understand what they are choosing. That documentation is the beginning of accountability, and the systems that produce it are the subject of the next chapter. CHAPTER 8 Auditing Systems in the World Methods for measuring and mitigating bias are only as useful as the institutions that apply them. A fairness metric that nobody computes, a model card that nobody reads and a mitigation that nobody verifies after deployment are, in practice, the same as none at all. Over the past decade a set of practices and legal requirements has grown up around the task of checking algorithmic systems once they are built: internal and external audits, structured documentation, benchmark testing and, increasingly, statutory obligations. This chapter examines what those practices have achieved, where they fall short, and what a credible audit actually involves. The audit that changed an industry The most influential algorithmic audit to date was published in 2018 by Joy Buolamwini and Timnit Gebru under the title "Gender Shades."1 They evaluated three commercial facial analysis systems, from IBM, Microsoft and the Chinese company Face++, on the task of classifying the gender of a face in a photograph. Existing benchmark datasets were heavily skewed toward lighter-skinned subjects, so the authors built their own, the Pilot Parliaments Benchmark, consisting of images of 1,270 members of parliament from three African and three European countries, selected to give a more balanced distribution of skin types and labelled by gender and by skin type on the dermatological Fitzpatrick scale. The results were stark. All three systems performed better on male faces than on female faces and better on lighter-skinned faces than on darker-skinned faces. The disparity was greatest at the intersection: error rates for darker-skinned women reached 34.7 percent for the worst-performing system, while the maximum error rate for lighter-skinned men was 0.8 percent. The aggregate accuracy figures that the companies reported concealed this almost entirely, because darker-skinned women were a small fraction of the test sets they had used. What made Gender Shades exceptional was what happened next. Buolamwini and Gebru shared their results with the companies before publication. In a follow-up study published in 2019, Inioluwa Deborah Raji and Buolamwini found that all three targeted companies had released new versions of their systems within seven months, with substantially reduced error-rate disparities, while companies that had not been named in the original audit showed larger disparities.2 The authors called this "actionable auditing": a public, specific, intersectional audit that named the companies involved produced measurable change. The broader story of facial recognition did not end there. In 2020, amid protests over policing in the United States, IBM announced that it would stop offering general-purpose facial recognition, Amazon announced a moratorium on police use of its Rekognition product, and Microsoft said it would not sell facial recognition to US police departments until federal legislation was in place. A US National Institute of Standards and Technology evaluation published in December 2019 had found demographic differentials in false positive rates across a large majority of the nearly two hundred algorithms it tested, though the most accurate algorithms showed much smaller differences. Several wrongful arrests in the United States, in which Black men were misidentified by facial recognition and detained, became public in the years that followed. The audit had shown the problem; the deployment context determined how much harm it did. Internal and external audits Audits differ in who conducts them and what access they have. Internal audits, carried out by the organisation that builds or deploys a system, have full access to the model, the training data and the deployment logs. They can measure fairness on real decisions and investigate causes. Their weakness is independence: the people conducting them often report to the people whose product is being evaluated, and unwelcome findings can be softened, delayed or buried. External audits, carried out by journalists, academics, civil society groups or regulators, have the independence that internal audits lack but usually far less access. Gender Shades worked through the public application programming interfaces of the systems it tested. ProPublica's COMPAS investigation relied on records obtained through public records requests. The Markup's mortgage analysis used government disclosure data. Each of these was limited by what could be observed from outside: the auditors could measure outcomes but could not inspect training data or model internals, and the organisations being audited could, and did, contest the findings on the grounds that the auditors lacked crucial variables. Raji and colleagues proposed in 2020 an end-to-end framework for internal algorithmic auditing, modelled on audit practices in finance and aerospace, that tried to combine the access of internal review with structured accountability: scoping documents, risk assessments at each stage, checklists, and a requirement that the audit trail be retained so that decisions could be reconstructed later.3 A related body of work has explored "second-party" audits by contracted firms with privileged access, which is the model that New York City's employment law adopted. The quality of such audits depends heavily on the auditor's independence and on clear standards for what they must examine, and both remain underdeveloped. When an audit finds no violation Not every audit confirms the suspicion that prompted it, and the cases where it does not are as instructive as those where it does. In November 2019, shortly after Apple and Goldman Sachs launched a co-branded credit card, a prominent software developer posted on social media that he had received a credit limit twenty times higher than his wife's, although they filed joint tax returns and she had the better credit score. Others, including Apple's co-founder Steve Wozniak, reported similar experiences. The posts spread quickly, and the New York State Department of Financial Services opened an investigation into whether the underwriting algorithm discriminated by sex. The department published its report in March 2021.4 It had examined the bank's underwriting data and procedures, including the applications of the couples who had complained, and found no violation of fair lending law. The credit decisions were explained by individual credit factors, such as the applicants' own credit histories, utilisation and debt, which could differ substantially between spouses even when they shared finances. A partner who had been an authorised user on the other's cards, rather than the primary account holder, might have a thinner individual record. Sex was not used, and the department found no evidence that the model was relying on a proxy for it. It did criticise the bank's lack of transparency and its customer service, which had left applicants unable to learn why their limits differed. The episode is often remembered as a false alarm, but that reading misses two points. First, the audit was able to reach a clear conclusion only because a regulator with legal authority could compel access to the data and the model. An external researcher working from social media posts could not have resolved the question either way. Second, the report identified a structural issue that sits outside the algorithm: credit systems built on individual histories disadvantage people whose financial lives have been conducted through a partner's accounts, a pattern with a long gendered history. The model was not discriminating in the legal sense. It was faithfully applying a framework in which an archive of individual credit, built up in a world where married women often had none of their own, determines who is trusted with credit now. An audit that asks only whether the model is lawful will miss that; an audit that asks where the disparity comes from will find it. Documentation as infrastructure A precondition for any audit is knowing what a system is, how it was built and what it was tested for. Two proposals from 2018 and 2019 have become standard reference points. "Datasheets for Datasets," by Timnit Gebru and colleagues, drew an analogy with the datasheets that accompany electronic components, which specify operating characteristics, recommended uses and limitations.5 The authors proposed that every dataset be accompanied by a document answering a structured set of questions: why the dataset was created and by whom, what it contains, how the data was collected and labelled, what preprocessing was done, what uses it is and is not suitable for, how it is distributed and maintained. Many of the questions bear directly on fairness, such as whether the dataset identifies subpopulations, whether it contains data that might be considered sensitive, and whether any individuals were compensated or consented. "Model Cards for Model Reporting," by Margaret Mitchell and colleagues, proposed an equivalent for trained models: a short document reporting the model's intended uses, the factors such as demographic groups or environmental conditions across which its performance might vary, the metrics used, the evaluation data, and quantitative results disaggregated across those factors.6 The emphasis on disaggregation is the key fairness contribution: a model card should show how the system performs for each relevant group, not only on average. Both proposals have been widely adopted in form, particularly by large technology companies and in public model repositories, where model cards are now routinely published. Their substance is more uneven. Studies of published model cards have found that many omit disaggregated evaluations or describe limitations in generic terms. Documentation is valuable only when it is honest and specific, and nothing in the format guarantees that. Auditing without demographic data A practical obstacle to fairness auditing is that the organisations best placed to measure disparities often do not have the demographic data needed to do so. In US consumer lending outside mortgages, creditors are generally prohibited from asking applicants about race or ethnicity, precisely to prevent discrimination. Many employers do not collect such data, or collect it separately from the hiring system. In the European Union, data protection law classifies racial or ethnic origin as special-category data that may be processed only under specific conditions. The result is that discrimination is prohibited but often cannot be measured. Several workarounds exist, none perfect. The US Consumer Financial Protection Bureau has used a method called Bayesian Improved Surname Geocoding, which estimates the probability that a person belongs to each racial and ethnic group from their surname and the demographic composition of their census block, to estimate disparities in auto lending and other markets. Such proxies are reasonably accurate in aggregate but can be biased for particular groups, and using them to estimate disparities introduces its own errors. Other approaches include voluntary self-identification collected separately from decisions, secure multiparty computation that allows disparities to be measured without any single party holding the full data, and, under the European Union's AI Act, the narrow permission discussed in Chapter 2 for providers of high-risk systems to process special-category data where strictly necessary for bias detection and correction. The tension here is real and not merely technical. Collecting demographic data creates privacy risks and, in some contexts, opportunities for misuse. Not collecting it makes discrimination invisible. A mature approach treats demographic data used for auditing as a distinct category, governed separately from the data used for decisions, with strict limits on access and purpose. Auditing generative models Large language models and image generators pose new challenges for auditing because their outputs are open-ended and their uses are not fixed in advance. A résumé classifier makes one kind of decision; a general-purpose language model may be used for summarising résumés, writing performance reviews, answering medical questions and generating marketing copy, each with different potential for harm. Several audit methods have emerged. Benchmark suites such as BBQ, discussed in Chapter 5, test for stereotyped responses in controlled settings. Paired-prompt audits present the model with matched inputs that differ only in a demographic cue, such as a name, and compare the outputs; this method, borrowed from the correspondence studies long used to measure hiring discrimination, has been applied to language models asked to rank résumés, set salaries or evaluate loan applications. Studies of this kind have repeatedly found that names associated with different racial or gender groups can shift model outputs, though the magnitude and direction vary across models and tasks. Red-teaming exercises, in which testers deliberately try to elicit harmful outputs, have become a standard part of pre-release evaluation at major developers. The difficulty is that each of these methods tests a sample of a vast space of possible uses. A model that performs well on a fairness benchmark may behave very differently when embedded in a specific application with a specific prompt and specific users. For this reason, auditing generative systems needs to happen at two levels: the model developer evaluating the general-purpose system, and the deployer evaluating the specific application, on its own data and its own users. The law catches up For most of the period covered by this booklet, algorithmic fairness was largely a matter of voluntary practice in the technology industry, with existing anti-discrimination law applying in principle but rarely enforced against automated systems. That has begun to change, though unevenly. Table 5 summarises some of the principal legal instruments that now bear on algorithmic bias. Table 5. Selected legal instruments addressing algorithmic discrimination. Instrument Jurisdiction Scope Key bias-related requirement Title VII and Uniform Guidelines (1964, 1978) United States Employment Disparate-impact liability; four-fifths rule as screening test Equal Credit Opportunity Act and Regulation B (1974) United States Credit Prohibits discrimination; adverse-action notices must give specific reasons Local Law 144 (2021, enforced from 2023) New York City Automated employment decision tools Annual independent bias audit; published impact ratios; candidate notice Colorado Artificial Intelligence Act (2024) Colorado High-risk systems in consequential decisions Reasonable care against algorithmic discrimination; impact assessments AI Act, Regulation (EU) 2024/1689 European Union High-risk systems, including hiring, credit, education and public services Data governance and bias examination; documentation; human oversight In the United States, the oldest and still the most important instruments are the general anti-discrimination statutes. Title VII of the Civil Rights Act of 1964 prohibits employment discrimination, and since Griggs v. Duke Power Co. in 1971 has been read to prohibit practices that have a disparate impact on protected groups and are not justified by business necessity; Congress codified that standard in the Civil Rights Act of 1991. The Equal Credit Opportunity Act prohibits discrimination in credit, and its implementing regulation requires lenders to give applicants specific reasons for adverse decisions. The Consumer Financial Protection Bureau stated in 2022 that this requirement applies in full to lenders using complex algorithms: the complexity of a model is not a defence for failing to explain its decisions.7 These statutes apply to algorithmic decisions as much as human ones, but enforcement priorities shift. In April 2025 an executive order directed federal agencies to deprioritise enforcement based on disparate-impact theories. Because disparate-impact liability under Title VII rests on statute, private litigants can still bring such claims, but the federal posture has changed. At the state and local level, New York City's Local Law 144 became the first law in the United States to require bias audits of automated hiring tools, though early studies of compliance found that only a small fraction of employers had published the required audits, in part because the law allowed employers considerable discretion in deciding whether their tools were covered. Colorado enacted a broader statute in 2024 requiring developers and deployers of high-risk AI systems to use reasonable care to protect consumers from algorithmic discrimination; its effective date was subsequently pushed back. In the European Union, the AI Act entered into force in August 2024 and is being phased in over several years. It classifies AI systems used in employment, credit scoring, access to education, access to essential public services and certain law enforcement and migration contexts as high-risk. Providers of such systems must, among other things, implement data governance practices that include examination of possible biases likely to affect fundamental rights or lead to discrimination, maintain technical documentation, and design their systems to allow effective human oversight. Public bodies and certain private deployers must also carry out a fundamental rights impact assessment before use. The obligations for high-risk systems were originally scheduled to apply from August 2026; in November 2025 the European Commission proposed to delay parts of that timetable, so the precise start dates have been subject to legislative revision. The Act sits alongside existing EU non-discrimination directives and the General Data Protection Regulation, whose Article 22 already restricts decisions based solely on automated processing that produce legal or similarly significant effects. What a credible audit involves Drawing these threads together, a credible fairness audit of a deployed system has several elements, whatever the legal regime. It defines its scope precisely: which system, which decisions, which population, which period. It identifies the groups at risk, including intersections, and explains how group membership was determined. It states the fairness criteria it uses and why they are appropriate to the harm in question, reporting several rather than one. It measures on real decisions in deployment, not only on development data, and reports uncertainty alongside point estimates. It investigates causes, tracing disparities back through the pipeline stages described in Chapter 2 rather than stopping at a number. It is conducted by people with enough independence to report unwelcome findings and enough access to understand the system. And it produces consequences: a documented decision about what will be changed, by whom, and how the change will be verified. The last element is the one most often missing. An audit that finds a problem and changes nothing has, in a sense, done harm, because it lends the system a veneer of scrutiny. Whether audits produce change depends ultimately on the people inside the organisations that build and deploy these systems, and on what they believe they owe to the people affected. That is the subject of the final chapter. CHAPTER 9 The Responsibilities of the Data Scientist On 15 January 2021 the Dutch government of Prime Minister Mark Rutte resigned. The immediate cause was a parliamentary inquiry report, published the previous month under the title Ongekend onrecht, "Unprecedented Injustice", into the administration of childcare benefits by the Dutch Tax and Customs Administration. Over a period of years, tens of thousands of families had been wrongly accused of fraud, ordered to repay benefits that sometimes amounted to tens of thousands of euros, and denied the information they needed to contest the decisions. Many were driven into debt; some lost their homes; children were taken into care. The scandal had many causes, including a political climate that demanded harsh measures against benefit fraud and an administrative culture that treated small paperwork errors as evidence of intent. But one strand was algorithmic. The tax authority used a risk-classification model to select benefit applications for closer scrutiny, and among the indicators it used was whether the applicant held Dutch nationality. In 2021 the Dutch Data Protection Authority fined the tax authority for unlawful and discriminatory processing of data on dual nationality. Amnesty International's report on the case, published in October 2021 under the title "Xenophobic Machines," argued that the system had automated ethnic profiling and that its opacity had prevented those affected from understanding or challenging what had happened to them.1 Many people were involved in building and running that system: policy officials who set the goals, managers who specified the requirements, analysts who selected the risk indicators, engineers who implemented the model, caseworkers who acted on its outputs. It is hard to identify any one of them as the person responsible. That diffusion is the central difficulty this chapter addresses. The problem of many hands Philosophers call it the problem of many hands: when an outcome results from the contributions of many people, each of whom made a small and individually defensible decision, it can become impossible to hold anyone accountable for the whole. Each participant can point to the constraints imposed by others. The analyst followed the specification. The manager relied on the analyst's expertise. The policy official trusted that the technical staff would flag any problem. The caseworker followed the system's recommendation. Responsibility evaporates. Machine learning systems are especially prone to this because their behaviour emerges from the interaction of many choices, few of which look like ethical decisions at the time they are made. Choosing a target variable looks like a technical question about data availability. Dropping rows with missing values looks like data cleaning. Selecting a model by its aggregate accuracy looks like standard practice. Setting a decision threshold looks like a business parameter. Yet as the preceding chapters have shown, each of these choices can determine who is harmed and how badly. The data scientist occupies a distinctive position in this web of responsibility. Not the most senior person, often not the one who decides whether a system is deployed, but very often the only person who understands in detail how the system works and where it might fail. That knowledge creates obligations. A doctor who notices that a treatment protocol is harming certain patients cannot discharge her responsibility by noting that she did not write the protocol. A structural engineer who sees a design flaw cannot stay silent because someone else approved the plans. The same applies here. What professional codes say Data science is not a licensed profession in the way that medicine or civil engineering is. There is no professional body that can strike a data scientist off a register. But the major computing professional associations have codes of ethics, and they are clearer on this subject than many practitioners realise. The Association for Computing Machinery's Code of Ethics and Professional Conduct, revised in 2018, states as one of its general ethical principles that computing professionals should "be fair and take action not to discriminate."2 The accompanying guidance notes that the use of information and technology may cause new, or enhance existing, inequities, and that technologies and practices should be as inclusive and accessible as possible. The code also requires professionals to avoid harm, to give comprehensive and thorough evaluations of computer systems and their impacts, including analysis of possible risks, and to take particular care with systems that become integrated into the infrastructure of society. It says that a computing professional who has reason to believe a system may cause harm has a responsibility to report it, and it explicitly contemplates that this may sometimes require whistleblowing. These are not aspirational sentiments for keynote speeches. They describe concrete duties that bear directly on the problems in this booklet, and they are the standard against which the conduct of a computing professional can reasonably be judged. Five points of responsibility What does this mean on an ordinary working day? The responsibilities of a data scientist building a system that makes or informs decisions about people can be organised around five points in the work. The first is problem formulation. Before any code is written, someone decides what the system will predict and how its predictions will be used. Andrew Selbst and his colleagues identified in 2019 a set of "abstraction traps," ways in which technical work goes wrong by abstracting away the social context in which a system will operate.3 The most fundamental is what they called the framing trap: defining the problem so narrowly, in terms of inputs and outputs, that the system's effects on the wider social context are invisible. The data scientist's responsibility at this stage is to ask the questions that the framing hides. What is the target variable actually measuring, and is it the same thing for every group? Who will be affected by the decisions, and how? What happens to people who are wrongly classified? What is the decision for, and would a different intervention serve that purpose better? These questions are often not in the data scientist's formal remit, but the data scientist is usually best placed to see why they matter. The second is data. The data scientist is responsible for understanding where the training data came from, whose decisions produced its labels, who is represented in it and who is missing, and whether its features mean the same thing across groups. This is the historian's discipline described in Chapter 1, and it requires talking to people who know how the data was generated, not only inspecting the data itself. Writing a datasheet for the dataset, even an informal one, is a practical way of discharging this responsibility. The third is evaluation. The data scientist chooses what to measure, and what is not measured is not seen. At a minimum, this means disaggregating performance by the groups at risk, including their intersections; computing several fairness criteria rather than one; reporting uncertainty; and testing on data that represents the deployed population. It also means resisting the pressure to report only the flattering numbers. A model card with honest disaggregated results is the natural record of this work. The fourth is communication. The people who decide whether and how to deploy a system usually cannot evaluate it themselves. They depend on the data scientist's account of what the system does and does not do. That account must be honest about limitations, explicit about the fairness trade-offs that were made and why, and clear about the conditions under which the system should not be used. It must also be comprehensible to people without technical training. A warning buried in an appendix, or expressed in jargon, has not been communicated. The fifth is deployment and monitoring. Because of the feedback loops and population shifts described in earlier chapters, a system that was fair at launch may become unfair over time. The data scientist has a responsibility to insist on monitoring, to specify what should be monitored and how often, and to define in advance the conditions under which the system should be retrained, modified or withdrawn. A cautionary case from education The 2020 school-examination crisis in England illustrates several of these points at once. When the COVID-19 pandemic forced the cancellation of A-level examinations, which largely determine university admission, the exam regulator Ofqual developed a statistical model to assign grades. Teachers provided predicted grades and a ranking of students within each subject at each school, and the model adjusted these to match each school's historical distribution of results. When results were released in August 2020, nearly 40 percent of teacher-assessed grades had been lowered. The adjustment fell hardest on high-achieving students at schools with weaker historical results, who were capped by their school's past performance, and least hard on students at schools with small class sizes, for whom the model relied more heavily on teacher assessments because there was too little historical data. Since small classes are more common at independent fee-paying schools, the model systematically favoured students who were already advantaged. After widespread protests, with students chanting against the algorithm, the government abandoned the model within days and allowed students to receive their teachers' assessed grades. The Ofqual model was not built carelessly. Its designers had considered fairness and had tested whether the model introduced bias by protected characteristics, and on the measures they chose it did not. But the problem was formulated in a way that made the central injustice invisible: the model's goal was to preserve the national distribution of grades and the historical distribution at each school, and an individual student's grade was therefore partly determined by the past performance of other students at the same school. The framing trap was sprung before any modelling began. Fairness to schools' historical distributions was achieved at the expense of fairness to individual students, and the people who would bear the cost were never asked. The right to say no The hardest responsibility is the one that involves refusing. Sometimes a data scientist will conclude that a system cannot be built fairly with the available data, that the trade-offs it requires are unacceptable, or that its intended use is harmful. What then? The first step is always to raise the concern, clearly and in writing, with the people who have authority over the project. Many concerns are resolved at this stage, because the decision-makers did not know about the problem and are glad to be told. Documenting the concern matters because it creates a record that the issue was identified and forces an explicit decision about it. If the concern is dismissed, the data scientist faces a harder choice. Options include escalating within the organisation, seeking advice from an internal ethics or compliance function where one exists, declining to continue working on the project, or, in serious cases, reporting the problem externally to a regulator. Each option carries personal and professional costs, and the field has seen researchers lose their positions after raising concerns about the harms of the systems their employers were building. The ACM Code recognises these costs while maintaining that they do not eliminate the duty. Organisations can make this easier or harder. Those that take fairness seriously create clear channels for raising concerns, protect people who use them, give ethics reviews real power to stop projects, and reward people who find problems early rather than treating them as obstacles to delivery. Those that do not take it seriously tend to rely on the individual heroism of employees, which is an unreliable and unfair basis for preventing harm. A data scientist choosing where to work can reasonably ask which kind of organisation they are joining. The responsibilities in practice It helps to see how these duties play out on a single, ordinary project. Imagine a data scientist at a property management company asked to build a model that scores rental applicants, to replace a manual review that the company finds slow and inconsistent. The brief is to predict which applicants are likely to fall behind on rent. At the formulation stage, the first question is what "falling behind" means in the data. If the label is eviction filings, it reflects not only tenants' payment behaviour but landlords' decisions about when to file, which vary by neighbourhood and have been shown in US research to fall disproportionately on Black women. If the label is late payment recorded in the company's own systems, it reflects which tenants were pursued and which were quietly given grace periods. Neither is a clean measure of the thing the company cares about. The data scientist's first contribution is to say so, and to propose a label, perhaps sustained arrears above a set amount after a fixed period, that depends less on discretionary decisions. At the data stage, the questions concern features. Criminal records, prior eviction filings and credit scores are standard in tenant screening and each carries the history described in Chapter 1. Several US jurisdictions restrict the use of criminal history or of eviction filings that did not result in a judgement. The data scientist should know those rules, and beyond them should ask whether each feature measures something relevant to paying rent or merely something correlated with group membership. Income-to-rent ratios, for instance, can penalise applicants whose rent is paid partly by housing vouchers, and some jurisdictions prohibit discrimination by source of income. At the evaluation stage, the data scientist needs demographic information to measure disparities, which the company probably does not collect. A defensible approach might combine voluntary self-identification gathered separately from the application with a proxy method for estimating aggregate disparities, and report results with honest uncertainty. The metrics should be chosen with the harm in view. Denial of housing is a severe burden, so false positives, applicants wrongly scored as high risk, deserve particular attention, and their rates should be compared across groups. At the communication stage, the people who will decide whether to use the model need to hear plainly what it can and cannot do: how much better it predicts than the manual process it replaces, how its errors are distributed, which trade-offs were made and why, and which uses it was not designed for, such as setting rents or deciding whom to evict. If the model offers only a marginal gain in accuracy over a simple rule while carrying measurable disparities, that should be said. At the deployment stage, the data scientist should specify what happens to applicants the model scores poorly. A rule that requires a human review, or that offers a co-signer or deposit alternative instead of outright rejection, changes the cost of the model's errors. Monitoring should track approval rates and subsequent outcomes by group, and there should be a defined threshold at which the model is suspended for review. None of these steps is exotic. Each takes some hours of work and some willingness to raise questions that the brief did not ask. Taken together, they are the difference between a system whose fairness was considered and one whose unfairness was simply never examined. Beyond compliance It would be convenient if all of this could be reduced to legal compliance. Laws like the European Union's AI Act and New York City's audit requirement make some fairness practices mandatory, and compliance is necessary. But it is not sufficient, for three reasons. First, the law is incomplete. It covers some domains and some protected characteristics, and leaves others untouched. Many systems that affect people's lives fall outside every regulatory regime. Second, the law lags behind the technology. The legal frameworks described in Chapter 8 were mostly written with traditional classifiers in mind and fit generative systems awkwardly. Practitioners working at the frontier will regularly encounter situations the law has not anticipated. Third, and most fundamentally, the law sets a floor, not a goal. A system can comply with every applicable rule and still treat people unfairly in ways that its builders could have prevented. The question a data scientist should ask is not only "is this legal?" but "would I be able to justify this to the people it affects?" That question cannot be delegated to a compliance department. The phrase "people it affects" deserves emphasis. Many of the failures in this booklet occurred because the people who bore the costs of the system had no voice in its design. The families accused of fraud in the Netherlands, the students in England, the patients in the Obermeyer study and the defendants in Broward County were all subjects of decisions made by systems whose designers never consulted them. Participatory design approaches, which involve affected communities in defining problems, choosing metrics and evaluating outcomes, are slower and messier than purely technical development. They are also the only way to find out what fairness means to the people for whom it matters most. Conclusion: Fairness as a Design Decision The cases in this booklet span health care, criminal justice, lending, hiring, policing, education, welfare administration and the representations that underlie modern language and image models. They differ in their details, but they share a structure. In each, a system learned from a record of the past. In each, that record bore the marks of unequal treatment, sometimes in the labels, sometimes in the features, sometimes in who was counted at all. And in each, the people who built the system made choices that determined whether those marks would be carried forward, amplified or corrected, often without recognising that they were making such choices. Several conclusions follow from putting these cases side by side, and they go beyond the sum of the individual chapters. Neutrality is not available The first is that there is no neutral position. A data scientist who declines to make any fairness intervention has not avoided a choice; they have chosen to reproduce whatever patterns the data contains, with whatever amplification the learning algorithm adds. A team that removes the protected attribute from its inputs has not made its system blind; it has made its system's use of proxies harder to detect. An organisation that reports aggregate accuracy alone has not avoided taking a position on fairness; it has chosen not to look. This matters because the claim of neutrality is the most common defence of biased systems, and it is almost always false. Every system embodies some answer to the question of how errors and benefits should be distributed across groups. The only choice is whether that answer is made deliberately, stated openly and defended on its merits, or arrived at by default and left unexamined. The metrics are tools for argument, not substitutes for it The second conclusion concerns the mathematics. The formal fairness criteria of Chapter 3 and the impossibility results of Chapter 4 are among the genuine intellectual achievements of the field. They replaced vague accusations and vague reassurances with precise, checkable claims. They showed that some intuitively appealing goals cannot be achieved together, which ended a great deal of confused debate. But the criteria do not tell anyone which to choose. That choice depends on what kind of harm the decision can do, how trustworthy the labels are, what the decision is for and what the law allows. These are questions of values and of institutional purpose, and they must be answered by people, preferably including the people who will bear the consequences. The appropriate use of the mathematics is to make the stakes of that choice visible: to show, for a particular system, exactly what is gained and lost under each candidate standard. The inappropriate use is to treat the selection of a metric as a technical detail, settled by convention or by whichever library function happens to be the default. Representations need the same scrutiny as decisions The third conclusion is that as machine learning has moved toward large pretrained models whose internal representations are reused across many applications, bias has become a property of shared infrastructure. A biased embedding space or foundation model does not cause harm only in one product. It causes harm, in varying ways, in every product built on it. The debiasing methods of Chapter 6 have advanced considerably, from the hand-built projections of 2016 to the provable linear erasure of recent work. But their history is a sequence of methods that solved narrower problems than first claimed. The honest current position is that representation-level debiasing can remove particular, well-defined kinds of information with measurable guarantees, and that it cannot, on its own, make a downstream system fair. It must be combined with evaluation of the actual decisions the system makes, on the actual population it serves. The organisations that build widely used foundation models carry a particular responsibility here, because their choices propagate furthest and are hardest for downstream users to inspect. The most powerful interventions are upstream The fourth conclusion, which may be the most practically important, is that the most effective fairness interventions in the record rarely involved fairness algorithms at all. The Obermeyer health algorithm was substantially repaired by changing what it predicted. The facial analysis disparities exposed by Gender Shades were reduced by collecting more representative training data and evaluating on balanced benchmarks. The worst harms of the English examination algorithm and the Dutch benefits system could have been avoided by different problem formulations, different inputs and different decisions about how outputs would be used. The reweighing, constraint and threshold methods of Chapter 7 are valuable, and in some settings they are exactly the right tool. But they operate on a problem that has already been defined. Many of the most serious failures happen before that point, when someone decides what to predict, which data to use and what will happen to the people the model flags. Practitioners who reach first for a mitigation algorithm, before examining these upstream choices, are working on the part of the problem that matters least. Responsibility cannot be diffused away The fifth conclusion concerns people. Algorithmic systems are built by many hands, and that diffusion creates a constant temptation to treat bias as nobody's fault. The data were biased; the requirements came from elsewhere; the decision to deploy was made above one's pay grade; the users chose how to act on the outputs. All of these may be true, and none of them dissolves the obligation of the person who understood how the system worked and could see where it would fail. The professional codes of the computing field already state this obligation clearly. What is often missing is the organisational support that makes it possible to act on it: clear channels for raising concerns, real authority for ethics reviews, and a culture that treats the discovery of a fairness problem as a success rather than an inconvenience. Building those structures is as much a part of building fair systems as any technical method. What remains unsettled Some important questions remain open. The relationship between fairness constraints and long-term outcomes, highlighted by the work on delayed impact discussed in Chapter 7, is poorly understood in most real domains, and there is little longitudinal evidence about how algorithmic decisions affect the people subject to them over years rather than months. Methods for auditing generative models are still immature, and there is no consensus on what a fair language model would look like, given the range of tasks such models perform. The legal frameworks in both the United States and the European Union are in flux, with enforcement priorities shifting and implementation timetables being revised. And the problem of measuring discrimination without collecting sensitive data, or of collecting it without creating new risks, has no fully satisfactory solution. These open questions are not reasons to wait. The core practices are well established: ask what the target really measures; understand where the data came from; disaggregate evaluation by group and intersection; report several fairness criteria and justify the one prioritised; document honestly; monitor in deployment; involve the people affected; and be prepared to recommend against building a system that cannot be made fair. None of these requires a research breakthrough. All of them require someone to decide that they matter. The training data will always be a record of the past, and the past was not fair. Systems trained on it will tend to perpetuate that unfairness unless someone intervenes. The question for anyone who builds them is not whether to make a choice about fairness, since that choice is unavoidable, but whether to make it knowingly, defend it honestly and accept responsibility for its consequences. Notes Introduction 1. Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan, "Dissecting racial bias in an algorithm used to manage the health of populations," Science 366, no. 6464 (2019): 447–453. Chapter 1 1. Solon Barocas and Andrew D. Selbst, "Big Data's Disparate Impact," California Law Review 104 (2016): 671–732. 2. Emmanuel Martinez and Lauren Kirchner, "The Secret Bias Hidden in Mortgage-Approval Algorithms," The Markup, August 2021. 3. Robert Bartlett, Adair Morse, Richard Stanton and Nancy Wallace, "Consumer-lending discrimination in the FinTech era," Journal of Financial Economics 143, no. 1 (2022): 30–56. 4. Jeffrey Dastin, "Amazon scraps secret AI recruiting tool that showed bias against women," Reuters, 10 October 2018. 5. Kristian Lum and William Isaac, "To predict and serve?" Significance 13, no. 5 (2016): 14–19. 6. Aylin Caliskan, Joanna J. Bryson and Arvind Narayanan, "Semantics derived automatically from language corpora contain human-like biases," Science 356, no. 6334 (2017): 183–186. 7. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021). 8. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez and Kai-Wei Chang, "Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints," Proceedings of EMNLP (2017). Chapter 2 1. Harini Suresh and John Guttag, "A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle," Proceedings of Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO '21) (2021). 2. Barocas and Selbst, "Big Data's Disparate Impact." 3. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi and Noah A. Smith, "The Risk of Racial Bias in Hate Speech Detection," Proceedings of ACL (2019). 4. Allison Koenecke et al., "Racial disparities in automated speech recognition," Proceedings of the National Academy of Sciences 117, no. 14 (2020): 7684–7689. 5. Danielle Ensign, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger and Suresh Venkatasubramanian, "Runaway Feedback Loops in Predictive Policing," Proceedings of the Conference on Fairness, Accountability and Transparency, PMLR 81 (2018). Chapter 3 1. Solon Barocas, Moritz Hardt and Arvind Narayanan, Fairness and Machine Learning: Limitations and Opportunities (MIT Press, 2023). Uniform Guidelines on Employee Selection Procedures (1978), 29 C.F.R. § 1607.4(D). 3. Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold and Richard Zemel, "Fairness through awareness," Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (2012). 4. Moritz Hardt, Eric Price and Nathan Srebro, "Equality of Opportunity in Supervised Learning," Advances in Neural Information Processing Systems 29 (2016). 5. Michael Kearns, Seth Neel, Aaron Roth and Zhiwei Steven Wu, "Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness," Proceedings of ICML (2018). 6. Ashudeep Singh and Thorsten Joachims, "Fairness of Exposure in Rankings," Proceedings of KDD (2018). 7. Sahin Cem Geyik, Stuart Ambler and Krishnaram Kenthapadi, "Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search," Proceedings of KDD (2019). Chapter 4 1. Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner, "Machine Bias," ProPublica, 23 May 2016. 2. William Dieterich, Christina Mendoza and Tim Brennan, "COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity," Northpointe Inc. (2016). 3. Alexandra Chouldechova, "Fair prediction with disparate impact: A study of bias in recidivism prediction instruments," Big Data 5, no. 2 (2017): 153–163. 4. Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan, "Inherent Trade-Offs in the Fair Determination of Risk Scores," Proceedings of the 8th Innovations in Theoretical Computer Science Conference (2017). 5. Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel and Aziz Huq, "Algorithmic Decision Making and the Cost of Fairness," Proceedings of KDD (2017). 6. Sam Corbett-Davies and Sharad Goel, "The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning," arXiv:1808.00023 (2018); revised version in Journal of Machine Learning Research 24 (2023). 7. Dwork et al., "Fairness through awareness." 8. Matt J. Kusner, Joshua Loftus, Chris Russell and Ricardo Silva, "Counterfactual Fairness," Advances in Neural Information Processing Systems 30 (2017). 9. Julia Dressel and Hany Farid, "The accuracy, fairness, and limits of predicting recidivism," Science Advances 4, no. 1 (2018): eaao5580. Chapter 5 1. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama and Adam Kalai, "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings," Advances in Neural Information Processing Systems 29 (2016). 2. Caliskan, Bryson and Narayanan, "Semantics derived automatically from language corpora contain human-like biases." 3. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman and Rachel Rudinger, "On Measuring Social Biases in Sentence Encoders," Proceedings of NAACL (2019). 4. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez and Kai-Wei Chang, "Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods," Proceedings of NAACL (2018). 5. Alicia Parrish et al., "BBQ: A Hand-Built Bias Benchmark for Question Answering," Findings of ACL (2022). 6. Federico Bianchi et al., "Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale," Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (2023). 7. Latanya Sweeney, "Discrimination in Online Ad Delivery," Communications of the ACM 56, no. 5 (2013): 44–54. 8. Muhammad Ali, Piotr Sapiezynski, Miranda Bogen, Aleksandra Korolova, Alan Mislove and Aaron Rieke, "Discrimination through Optimization: How Facebook's Ad Delivery Can Lead to Biased Outcomes," Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019). Chapter 6 1. Bolukbasi et al., "Man is to Computer Programmer as Woman is to Homemaker?" 2. Hila Gonen and Yoav Goldberg, "Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them," Proceedings of NAACL (2019). 3. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton and Yoav Goldberg, "Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection," Proceedings of ACL (2020). 4. Richard Zemel, Yu Wu, Kevin Swersky, Toniann Pitassi and Cynthia Dwork, "Learning Fair Representations," Proceedings of ICML (2013). 5. Harrison Edwards and Amos Storkey, "Censoring Representations with an Adversary," Proceedings of ICLR (2016). 6. David Madras, Elliot Creager, Toniann Pitassi and Richard Zemel, "Learning Adversarially Fair and Transferable Representations," Proceedings of ICML (2018). 7. Brian Hu Zhang, Blake Lemoine and Margaret Mitchell, "Mitigating Unwanted Biases with Adversarial Learning," Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (2018). 8. Yanai Elazar and Yoav Goldberg, "Adversarial Removal of Demographic Attributes from Text Data," Proceedings of EMNLP (2018). 9. Shauli Ravfogel, Michael Twiton, Yoav Goldberg and Ryan Cotterell, "Linear Adversarial Concept Erasure," Proceedings of ICML (2022). 10. Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff and Stella Biderman, "LEACE: Perfect linear concept erasure in closed form," Advances in Neural Information Processing Systems 36 (2023). 11. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky and Sharese King, "AI generates covertly racist decisions about people based on their dialect," Nature 633 (2024): 147–154. Chapter 7 1. Faisal Kamiran and Toon Calders, "Data preprocessing techniques for classification without discrimination," Knowledge and Information Systems 33, no. 1 (2012): 1–33. 2. Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger and Suresh Venkatasubramanian, "Certifying and Removing Disparate Impact," Proceedings of KDD (2015). 3. Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford and Hanna Wallach, "A Reductions Approach to Fair Classification," Proceedings of ICML (2018). 4. Hardt, Price and Srebro, "Equality of Opportunity in Supervised Learning." 5. Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg and Kilian Q. Weinberger, "On Fairness and Calibration," Advances in Neural Information Processing Systems 30 (2017). 6. Emily Black, Manish Raghavan and Solon Barocas, "Model Multiplicity: Opportunities, Concerns, and Solutions," Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (2022). 7. Lydia T. Liu, Sarah Dean, Esther Rolf, Max Simchowitz and Moritz Hardt, "Delayed Impact of Fair Machine Learning," Proceedings of ICML (2018). Chapter 8 1. Joy Buolamwini and Timnit Gebru, "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification," Proceedings of the Conference on Fairness, Accountability and Transparency, PMLR 81 (2018): 77–91. 2. Inioluwa Deborah Raji and Joy Buolamwini, "Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products," Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (2019). 3. Inioluwa Deborah Raji, Andrew Smart et al., "Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing," Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (2020). 4. New York State Department of Financial Services, Report on Apple Card Investigation (March 2021). 5. Timnit Gebru et al., "Datasheets for Datasets," Communications of the ACM 64, no. 12 (2021): 86–92. 6. Margaret Mitchell et al., "Model Cards for Model Reporting," Proceedings of the Conference on Fairness, Accountability, and Transparency (2019). 7. Consumer Financial Protection Bureau, Circular 2022-03, "Adverse action notification requirements in connection with credit decisions based on complex algorithms" (May 2022). Chapter 9 1. Amnesty International, Xenophobic Machines: Discrimination through Unregulated Use of Algorithms in the Dutch Childcare Benefits Scandal (October 2021). 2. Association for Computing Machinery, ACM Code of Ethics and Professional Conduct (2018), Principle 1.4. 3. Andrew D. Selbst, danah boyd, Sorelle A. Friedler, Suresh Venkatasubramanian and Janet Vertesi, "Fairness and Abstraction in Sociotechnical Systems," Proceedings of the Conference on Fairness, Accountability, and Transparency (2019). Further Reading Solon Barocas, Moritz Hardt and Arvind Narayanan, Fairness and Machine Learning: Limitations and Opportunities (MIT Press, 2023).** The standard technical text on the subject, written by three of the field's leading researchers. Its treatment of independence, separation and sufficiency, and of the limits of observational criteria, underlies much of Chapters 3 and 4. A free online edition is available from the authors. Michael Kearns and Aaron Roth, The Ethical Algorithm: The Science of Socially Aware Algorithm Design (Oxford University Press, 2019).** An accessible account by two computer scientists of how values such as fairness and privacy can be built into algorithms, with a clear explanation of the trade-offs involved. Cathy O'Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (Crown, 2016).** A former quantitative analyst's influential account of opaque, large-scale scoring systems in education, employment, credit and criminal justice, and of the feedback loops that make them self-reinforcing. Virginia Eubanks, Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor (St. Martin's Press, 2018).** Detailed case studies of automated systems in welfare eligibility, homelessness services and child protection in the United States, written from the perspective of the people subject to them. Safiya Umoja Noble, Algorithms of Oppression: How Search Engines Reinforce Racism (NYU Press, 2018).** An examination of representational harms in commercial search, focusing on how Black women and girls were depicted in search results. Ruha Benjamin, Race After Technology: Abolitionist Tools for the New Jim Code (Polity, 2019).** A sociologist's argument that technologies presented as neutral or even progressive can encode and extend racial hierarchy, with a useful vocabulary for describing how. Joy Buolamwini, Unmasking AI: My Mission to Protect What Is Human in a World of Machines (Random House, 2023).** The author of the Gender Shades audit describes the research, the companies' responses and the policy campaigns that followed. Brian Christian, The Alignment Problem: Machine Learning and Human Values (W. W. Norton, 2020).** A wide-ranging and well-reported history of the effort to make machine learning systems behave as intended, with substantial chapters on representation, fairness and the COMPAS debate. Meredith Broussard, More than a Glitch: Confronting Race, Gender, and Ability Bias in Tech (MIT Press, 2023).** A data journalist's case that algorithmic bias is a structural feature of technology built on unequal data rather than an occasional malfunction, with examples from medicine, policing and education. Hashtags: #AIEthics #AlgorithmicBias #FairAI #EquitableAI #ResponsibleAI #EthicalArtificialIntelligence #AlgorithmicFairness #MachineLearningEthics #BiasInAI #AIFairness #AlgorithmicDiscrimination #FairMachineLearning #DataBias #TrainingDataBias #DemographicParity #EqualizedOdds #EqualOpportunity #PredictiveParity #AlgorithmicAccountability #BiasMitigation #FairnessMetrics #AITransparency #ExplainableAI #RepresentationBias #AdversarialDebiasing #BiasAuditing #EthicalMachineLearning #InclusiveTechnology #AIGovernance #HumanCenteredAI

Download the Book (PDF): Introduction A working molecular biologist in 2026 spends more time in a text editor than at a bench. The sequence of a guide RNA is typed, not pipetted. The design of a therapeutic editor exists first as a string, then as a scoring report, then as a set of candidate constructs ranked by predicted efficiency and predicted collateral damage, and only after all of that as something in a tube. The archive that holds a company's reference genomes is a file system. The library of promoters a team uses to tune a gene circuit is a spreadsheet with measured transfer functions in it. The vocabulary has followed: people speak of parts and registries, of assembly and compilation, of debugging a strain, of a design-build-test-learn loop. None of this is metaphor imposed from outside by enthusiasts. It is how the work is actually organised. This book takes that organisation seriously and asks what follows from it. The claim is not that cells are computers. Cells are not computers in any useful sense; they are self-modifying, self-replicating, thermally noisy chemical systems that were not designed and do not expose clean interfaces. The claim is narrower and more useful: that the problems which dominate modern genetic engineering are, structurally, software engineering problems, and that the techniques which solved them in software — formal specification of intent, addressing schemes, error-correcting codes, layered abstraction, systematic testing, separation of logic from deployment — are the techniques now being imported into biology, where they work partially, in ways that are instructive precisely where they fail. Three activities make the case concretely, and they organise what follows. The first is guide RNA design. To edit a human gene with a CRISPR nuclease, you must specify a twenty-letter address in a three-billion-letter genome and be confident that the machinery you send will bind that address and no other. This is a search and specificity problem with a definable error model. It is attacked with sequence-matching algorithms, thermodynamic models, machine-learned efficiency predictors trained on tens of thousands of measured edits, and empirical assays that enumerate unintended binding sites across the whole genome. The design pipeline looks like a compiler toolchain with a static analyser bolted on, and the reason it looks that way is that the underlying question — does this instruction do what I meant, and only what I meant — is the same question. The second is DNA data storage. Synthetic DNA is the densest known information medium: on the order of a quintillion bytes per gram if you count only the polymer, and durable for centuries to millennia under the right conditions, as the recovery of readable sequence from hundred-thousand-year-old remains demonstrates. Writing arbitrary digital files into DNA requires an encoding scheme, because the physical medium has constraints that binary does not: no long homopolymer runs, balanced composition, no sequences that fold back on themselves, short fragment lengths, and a write process that produces errors at rates a disk manufacturer would find unthinkable. The field's progress has come mostly from information theory — fountain codes, Reed-Solomon outer codes, indexed random access — layered over chemistry that is, in engineering terms, a very bad disk drive with extraordinary density. The result is a real storage system with a real cost curve, and a genuinely open question about whether the curve bends far enough. The third is genetic circuit design. For twenty-five years people have built logic into living cells: switches, oscillators, sensors wired to outputs. The ambition has been a compiler — write the behaviour you want in a high-level language, have a tool select parts, wire them together, and emit a DNA sequence to synthesise. Such tools exist and work, and their limits are the most interesting thing in this book, because they expose what biology does not share with silicon: no clock, no isolation between components, a shared and finite resource pool, mutation under selection, and a population of instances rather than a single deterministic machine. The controlling argument that runs through all three is this: in engineered biology, the hard part has moved from the editing operation to the verification and delivery of it. Making a cut in DNA is now cheap, reliable, and nearly commoditised. Knowing exactly what you changed, proving you changed nothing else, and getting the machinery into the right cells in a living person — these are where the cost, the risk, and the failure modes now live. The same is true of DNA storage, where writing is solved in principle and expensive in practice, and reading back without error requires more cleverness than writing does. And it is true of genetic circuits, where the design is the easy half and the behaviour of the design inside a growing, dividing, evolving cell is the hard half. Every chapter that follows is, in one way or another, about that shift. A note on what this book assumes and what it does not. It assumes you know that DNA is a double-stranded polymer of four bases, that genes are transcribed into RNA and often translated into protein, and that proteins do most of the work in a cell. It does not assume you can read a sequence alignment, know what a PAM is, or have an opinion about Cas12a. Technical material is built up in words, step by step, with real numbers from real systems wherever numbers exist. Where a figure is contested or uncertain, it is labelled as such rather than rounded into false confidence. The clinical material deserves a word too. Genome editing stopped being a laboratory technique in late 2023, when regulators in the United Kingdom and United States approved an edited-cell therapy for sickle cell disease and transfusion-dependent beta thalassaemia. Since then the field has produced a small but growing body of human evidence: in vivo editing delivered to the liver by lipid nanoparticle, base editing that corrects a specific point mutation in treated patients, a prime editing therapy dosed in a patient for the first time, and — in a case that will be discussed in detail — a bespoke editor designed, manufactured, and administered to a single infant with a rare urea cycle disorder within about six months of diagnosis. These are legitimate, published, regulated therapeutic programmes, and the engineering lessons in them are the reason they appear here. The discussion stays at the level of a general technical account of how such designs are conceived, assessed, and governed, which is where the interesting engineering is anyway. What this book is not is a survey. There is no attempt to cover every editor, every chassis organism, every company. Depth on a narrow front is more useful than coverage, and the narrow front chosen here is the engineering discipline itself: how intent becomes sequence, how sequence becomes verified behaviour, and what the gap between those two steps costs. One more framing idea, which will recur. Software engineers distinguish between a program's source, its compiled form, its deployment, and its runtime behaviour. Bugs live in different places depending on which layer failed, and the discipline of debugging is largely the discipline of localising the layer. Engineered biology has the same layers and they are routinely confused. A guide RNA that fails may have failed because its sequence was a poor choice (source), because the construct was assembled wrong (compilation), because the nanoparticle did not reach the target tissue (deployment), or because the cell's repair machinery resolved the break in an unintended way (runtime). Each of those failures has a different diagnostic and a different fix, and a large part of what follows is the business of telling them apart. Chapter 1: The Medium Is Not Silicon Start with the substrate, because every honest comparison between biology and computing has to begin by being precise about what DNA actually is as a medium and what it is not. DNA is a linear polymer. Its backbone is a chain of sugar and phosphate groups, and hanging off that chain is a sequence of four chemical groups: adenine, cytosine, guanine and thymine, written A, C, G and T. The backbone has a direction — one end is chemically distinct from the other — so a sequence is read in a defined order, conventionally from the 5-prime end to the 3-prime end. Two strands pair up with their directions opposed, A opposite T and C opposite G, which means either strand determines the other completely. This redundancy is the physical basis of copying and of repair, and it is also the reason a twenty-letter search query in a genome actually has two places to look, one on each strand. So far this sounds like a four-symbol tape, and at the level of information content it is: two bits per position. A human genome is about 3.1 billion base pairs, so roughly 6.2 gigabits or 775 megabytes of raw sequence per haploid copy, which is why a reference genome fits comfortably on a phone. But the tape analogy fails almost immediately, in three ways that matter for everything that follows. What DNA is besides a sequence First, DNA is a physical object with a shape, and the shape depends on the sequence. The double helix is not uniformly stiff; some sequences bend more easily than others, some locally unwind more easily, and some fold back on themselves into hairpins and more exotic structures when single-stranded. A stretch of alternating G and C is harder to pull apart than a stretch of A and T, because G-C pairs make three hydrogen bonds and A-T pairs make two. This is not a detail. It is why guide RNAs with certain compositions bind their targets more tightly and more promiscuously than others, and it is why DNA data storage schemes forbid certain sequences outright. In a disk drive, the bit pattern does not change the physics of the platter. In DNA, the bit pattern is the physics. Second, the molecule carries a second layer of information that is not in the sequence. Cytosine bases can be chemically decorated with a methyl group, and in mammalian genomes this methylation is heritable through cell division and powerfully affects whether nearby genes are transcribed. Beyond that, the molecule is wound around histone proteins, and those proteins carry their own chemical marks. The practical upshot for an engineer is that two cells with identical sequence can behave entirely differently, and a target site that is accessible in one cell type may be physically wrapped up and unreachable in another. There is no equivalent in silicon of an address that exists but cannot be written to because of how the memory happens to be folded today. In biology that is routine, and the best current guide-design tools take chromatin accessibility data as an input for exactly this reason. Third, DNA in a living cell is not read-only and it is not static. It is copied every time a cell divides, with an error rate low but nonzero; it is attacked constantly by reactive chemistry and radiation; and it is repaired continuously by a set of enzymatic pathways whose job is to restore the molecule when it breaks. That repair machinery is the hidden collaborator in every genome editing experiment. A CRISPR nuclease does not, strictly speaking, edit anything. It breaks the molecule at a chosen place. What happens next is up to the cell's repair pathways, and the outcome distribution — which insertions and deletions appear, at what frequencies, and whether a supplied template gets used — is a property of the cell, not of the enzyme. This is the single most important asymmetry between genetic engineering and programming, and it will come back repeatedly. Where the software stack maps, and where it stops With that in hand, the software analogy can be made properly, with its boundaries marked. There is a genuine instruction set. The genetic code — three-base codons specifying amino acids — is a lookup table, and it is nearly universal across life. It is also redundant: 64 codons for 20 amino acids plus a stop signal, so most amino acids have several codons. That redundancy is exploited constantly in practice. When a gene from one organism is expressed in another, its sequence is typically rewritten codon by codon to match the host's preferred usage, without changing the protein at all. This is a compilation step in the strict sense: same semantics, different encoding, chosen for the target platform. Codon optimisation tools are mature, commercial, and genuinely improve expression, though by less than their marketing suggests and occasionally by a negative amount, because codon choice also affects translation speed, messenger RNA folding, and protein folding in ways the simple models miss. There are genuine modular parts. A promoter is a sequence that recruits the transcription machinery and determines how strongly a downstream gene is expressed. A ribosome binding site determines how efficiently the resulting messenger RNA is translated. A terminator stops transcription. These are composable in practice: you can take a strong promoter from one context and put it in front of a different gene and get strong expression, usually. Registries of characterised parts exist, with measured strengths. Standard assembly methods — a set of techniques using sequence overlaps or type II restriction enzymes to join fragments in defined order — let you snap parts together much the way a build system links object files. The analogy is real enough that the standards bodies, the version control habits and the complaints about undocumented parts all transferred intact. And there is a genuine toolchain. Sequence design software, in silico assembly simulation, oligonucleotide ordering through a web form, sequencing-based verification of what came back. A gene is specified as a text file and arrives as a physical plasmid in the post, typically within a week or two for routine constructs. It costs tens to low hundreds of dollars for a kilobase-scale gene, and for most academic labs, that economics is now so settled that nobody builds routine constructs by hand. But four things do not map, and they are the four things this book keeps returning to. There is no clock. Digital logic works because state changes are synchronised to a global signal, which means a designer can reason about a circuit as a sequence of discrete steps. In a cell, every reaction proceeds at its own rate, concentrations change continuously, and there is nothing to synchronise to. A genetic logic gate's output is a steady-state concentration reached after a delay measured in cell generations, not a value latched on a clock edge. Composing such gates means composing analogue transfer functions and hoping the signal levels still match at the interfaces. There is no isolation. Two programs on a machine share a processor but have separate memory. Two genetic circuits in a cell share everything: the same pool of RNA polymerases, ribosomes, amino acids, and energy. Expressing a circuit strongly enough to be useful can slow the host's growth, which changes every rate in the cell, which changes the circuit's behaviour. This phenomenon — resource competition producing apparent coupling between components that have no designed connection — has no good software analogue except perhaps a shared-memory system under extreme contention, and it is the normal operating condition for engineered cells rather than a pathological one. There is no stable instance. A program runs once; an engineered cell divides and becomes a population, with mutations arising at a low rate and selection acting on them. If a circuit imposes a fitness cost, which almost all do, then cells that break it grow faster and take over. Engineered strains are routinely observed losing function over tens to hundreds of generations, and keeping them working is a matter of designing for evolutionary stability — minimising repeated sequences that invite recombination, reducing burden, sometimes coupling the circuit to something the cell needs. Imagine software that mutates while running and in which the mutants compete with the original for processor time. That is the actual situation. And the write operation is indirect. This is worth dwelling on. When you edit a file, the bytes you specify are the bytes that land. When you edit a genome with a nuclease, you specify a location, the enzyme makes a break there, and the cell repairs the break using whatever pathway happens to engage. The two dominant pathways give very different results. Non-homologous end joining rejoins the broken ends quickly and sloppily, frequently losing or gaining a few bases in the process — which is useful if your intent was to destroy a gene's function, because a one- or two-base insertion shifts the reading frame and garbles everything downstream. Homology-directed repair uses a template with matching flanking sequence to restore the break faithfully, which allows precise substitution, but it operates mainly in dividing cells and at much lower efficiency. So the engineer's intent — disrupt this gene, or change this one base to that one — is expressed as a choice of break location, a choice of pathway to bias towards, and a hope about outcome frequencies. Base editors and prime editors, treated in a later chapter, exist largely to remove this indirection, and they are best understood as making the write operation specify its own result rather than delegating it to the cell. The reference problem One more property of the medium belongs here, because it causes a class of error that is pure software engineering and nothing else: the genome you design against is a file, and the file is not the territory. Every guide design, every off-target search, every variant call is performed against a reference genome — a specific, versioned, downloadable sequence. The human reference in standard use for the last decade is a mosaic assembled from several donors, built up over many releases, and it was for most of its life incomplete. Roughly eight per cent of it consisted of gaps: regions the sequencing technologies of the time could not assemble, concentrated in repetitive areas near the centromeres, in the long arms of certain chromosomes, and in segmental duplications. Those gaps were represented in the file as runs of the letter N, meaning unknown. A guide design tool searching for off-target sites in a reference with eight per cent of the sequence replaced by N will report, with complete confidence, that there are no matches in those regions. That gap was closed in 2022, when a consortium using long-read sequencing published the first genuinely complete assembly of a human genome, end to end with no gaps, derived from a single uniform cell line. It added on the order of two hundred million base pairs of sequence that had never been assembled, including large arrays of repeats and a substantial number of previously unknown gene duplicates. Some of that new sequence contains perfectly good CRISPR target sites, which means that guides designed and published before 2022 were assessed against a reference that could not have found a subset of their potential off-targets. The deeper problem is that a single reference, however complete, misrepresents the object. Any two unrelated people differ at several million positions, plus tens of thousands of larger insertions, deletions and rearrangements. A reference built from a handful of donors represents the variation of those donors, and its ancestry composition skews heavily towards European sequence. For guide design that matters directly: a target site that is a unique twenty-base match in the reference may have a close variant in some populations and not others, and the same guide can therefore have systematically different off-target profiles in different patients. This is not a marginal statistical worry — it is a mechanism by which a therapy validated in one group could carry unmeasured risk in another. The field's response has been to replace the single reference with a graph. A pangenome reference represents many genomes together, with common variation encoded as alternative paths through a shared structure rather than as annotations on one linear string. A draft of such a reference for humans, published in 2023, assembled dozens of individuals from diverse ancestries into a single graph, with the intention of expanding further. Tools that align and search against a graph rather than a string are harder to write, and the ecosystem of software assuming a linear reference with fixed coordinates is enormous, so adoption has been slow — a migration problem with exactly the shape of any infrastructure change in software, where the data model improves and a decade of tooling has to catch up. For anyone designing against a genome, the practical rules fall out directly. State the reference version, because a coordinate without a version is meaningless and the two major human assemblies differ by millions of positions in their numbering. Search variant databases alongside the reference, because the reference's absence of a match is not evidence of a site's absence. For an individualised therapy, sequence the patient and design against their genome, which is now cheap enough to be routine and is exactly what the bespoke programmes do. And treat any off-target assessment performed against an older assembly as incomplete rather than as negative. The consequence for how work is organised Put these differences together and a pattern emerges in how competent groups actually operate. Because the write is indirect and stochastic, every edit is characterised after the fact by sequencing, and the result is reported as a distribution of outcomes rather than a binary success. A good paper reporting an edit does not say "we knocked out the gene"; it says what fraction of alleles carried which specific insertions and deletions, because that distribution determines whether the experiment means anything. Because there is no isolation and no clock, circuit design is empirical in a way that chip design has not been for forty years. You build a library of variants, measure them, and pick. The design tools narrow the library from millions to dozens; they do not hand you an answer. Because there is no stable instance, long-running engineered systems are designed with evolution in mind, and the engineering literature on this is growing rather than shrinking. Industrial fermentation strains are monitored by sequencing precisely because the organism in the tank at the end of a campaign is not genetically identical to the one that went in. And because the medium's physics depends on its content, sequence design is a constrained optimisation rather than an unconstrained encoding. This is most visible in DNA data storage, where the constraints are written down explicitly as rules — composition balance, homopolymer limits, secondary structure avoidance, minimum distance between the addresses of different fragments — and the encoding scheme has to satisfy all of them while still achieving good information density. A later chapter works through exactly how that is done, because it is the clearest example in the whole field of information theory doing real work against a physical medium's quirks. The right way to hold the computing analogy, then, is as a set of borrowed practices rather than a claim about shared nature. Biology is not computation. But the discipline of specifying intent precisely, encoding it against a medium's constraints, verifying that the encoding produced the intended behaviour, and separating the logic from its deployment — that discipline transfers, and the places it strains under biology's peculiarities are exactly the places where the interesting problems are. Chapter 2: Addressing Three Billion Letters The engineering content of CRISPR is almost entirely an addressing story. Strip away the biology and the system is this: a protein that can cut double-stranded DNA, coupled to a short nucleic acid sequence that determines where it cuts. Change the sequence, change the address. Everything that makes the technology general-purpose follows from that separation of mechanism from target, and everything that makes it hard to use safely follows from the fact that the addressing is approximate rather than exact. The mechanism, stated carefully The best-characterised system uses Cas9, a protein originally from Streptococcus pyogenes, loaded with a single guide RNA. In its engineered form the guide is one molecule of about a hundred nucleotides, of which the first twenty are the variable part — the spacer — and the rest is a fixed scaffold that the protein grips. The protein with its guide loaded does not scan the genome base by base in any orderly way. It diffuses, collides with DNA, and at each collision performs a quick test. The test is not for the spacer sequence. It is for a short motif adjacent to the potential target site called the protospacer adjacent motif, or PAM. For this version of Cas9 the PAM is three bases: any base, then G, then G, written NGG. If the PAM is absent, the protein lets go within milliseconds and carries on diffusing. If the PAM is present, the protein locally unwinds the adjacent DNA and lets the guide's spacer try to pair with one of the two strands. This two-stage design is the key to making the search tractable. An NGG motif occurs on average about once every eight base pairs along a random sequence, counting both strands, so the PAM check is a fast filter that still leaves hundreds of millions of candidate sites in a human genome — but each check costs almost nothing. The expensive step, unwinding and attempting base pairing, happens only at sites that pass the filter. A design intuition familiar from any indexed search: a cheap predicate first, the costly comparison second. If pairing succeeds across enough of the twenty bases, the protein undergoes a conformational change that activates its two cutting domains, and it cleaves both strands at a defined position — three base pairs away from the PAM. The result is a clean double-strand break with blunt ends. Then the protein sits on the cut site, in many cases for minutes to hours, which matters because it affects which repair pathway gets access. Two details of this mechanism have outsized practical consequences. The first is that the PAM is a hard requirement, which means you cannot cut wherever you like. You can only cut near an NGG. Given that NGG density, this is rarely a problem for gene disruption, where any break in an early exon will do. It is frequently a problem for precision work, where you need to cut within a few bases of a specific point mutation to get efficient template-directed repair. This is the main reason the field invested heavily in alternative nucleases with different PAM requirements — Cas9 variants from other bacteria, Cas12a with a T-rich PAM, and engineered Cas9 variants with relaxed PAM recognition that accept NG rather than NGG, widening the accessible target space at some cost in specificity. The second is that pairing need not be perfect. This is the whole problem. Why specificity is approximate Consider the information content of the address. Twenty bases of spacer plus a two-base PAM constraint gives roughly forty-two bits of specification. A human genome has about 6.2 billion addressable positions counting both strands, or roughly thirty-three bits of address space. So on paper the address is comfortably unique — there is about five hundred times more specification than the genome needs — and a perfectly discriminating matcher would find exactly one site. The matcher does not perfectly discriminate. Base pairing is a thermodynamic process, and a mismatch costs energy without necessarily costing enough energy to prevent binding. Worse, the cost is not uniform along the spacer. Pairing proceeds directionally, starting from the PAM-proximal end and zipping outwards. The first eight to twelve bases next to the PAM — the seed region — are checked under conditions where a mismatch will usually abort the process. The distal bases, furthest from the PAM, are checked after the complex has already committed substantial binding energy, and a mismatch there is often tolerated. A single distal mismatch frequently reduces cutting only two- or three-fold. Two or three distal mismatches may still permit measurable cutting. Collapse the effective address to the twelve-base seed plus PAM, and the arithmetic changes completely. Twelve bases is twenty-four bits of specification against thirty-three bits of address space. On that accounting, a given seed sequence should occur by chance several hundred times in a human genome. The real situation sits between the two calculations: full matches are cut efficiently, seed-matched sites with distal mismatches are cut inefficiently but detectably, and the genome-wide off-target profile of any given guide is an empirical fact with a long tail. There are further wrinkles. The complex tolerates small bulges — a site where the DNA has one extra base relative to the guide, or one fewer, can still be bound if the helix accommodates the loop. Guide design tools that enumerate only substitution mismatches therefore systematically miss a class of real off-targets. And the energy landscape depends on the guide's own composition: a guide rich in G and C binds more tightly overall, which raises on-target activity and raises mismatch tolerance together, so the strongest guides are often the least discriminating. A useful mental model, borrowed deliberately from software: the guide is a regular expression with a cheap required suffix and a fuzzy twenty-character prefix in which early characters are matched strictly and later ones loosely, and which also permits occasional single-character insertions or deletions. Querying such a pattern against a three-gigabase corpus will return hits. The job of design is to pick the query whose hit set is a single element, or as close to that as the corpus permits. Kinetics, dose, and the time dimension One more factor turns the specificity problem from a static sequence question into a dynamic one. Whether an off-target site gets cut depends not only on binding affinity but on how long the nuclease is present and at what concentration. Think of it as a race. The on-target site is cut quickly because binding is fast and stable. Off-target sites are cut slowly, because binding is rarer and more often aborts. If the nuclease is present for a short time at modest concentration, the on-target site gets cut and most off-targets do not. If it persists for weeks at high concentration — as happens when the editor is delivered as a DNA construct that integrates and keeps expressing — the slow reactions have time to complete, and off-target editing accumulates. This is the single most consequential practical insight in the field, and it reshaped how editors are delivered. Delivering Cas9 as a pre-formed protein-RNA complex, which does its work within hours and is then degraded, gives dramatically better specificity than delivering a plasmid that expresses it for days. Delivering it as messenger RNA, which is translated for a day or so and then disappears, sits in between and has become the dominant approach for therapeutic in vivo editing. The sequence of the guide is identical in all three cases. The off-target profile is not. Dose and duration are design parameters on equal footing with sequence, and a guide validated under transient delivery cannot be assumed safe under sustained expression. The same logic produced a family of engineered high-fidelity nucleases. The insight was that wild-type Cas9 binds DNA more tightly than it needs to for on-target activity — it has excess binding energy, which it spends tolerating mismatches. Variants were engineered with specific contacts to the DNA backbone removed, lowering overall binding energy so that the complex retains enough grip to cut a perfect match but not enough to cut an imperfect one. Several such variants are in routine use and reduce off-target cutting substantially while retaining most on-target activity. The principle is one any engineer recognises: reduce the margin so that marginal cases fail closed rather than open. A worked example Abstract specificity arguments become clearer with a concrete target, so take a simple and common task: disrupt a protein-coding gene in human cells so that it no longer makes functional product. The intent constrains the location before any sequence is examined. You want the break in a coding exon, because an indel there shifts the reading frame and garbles everything downstream. You want it early in the coding sequence, so that no functional truncated protein survives, but not in the first few codons, because translation sometimes restarts at a downstream start codon and produces a shortened but working protein. You want to avoid the last exon, because a frameshift there may leave most of the protein intact. And you want to avoid exon boundaries, because an indel near a splice site can produce a spliced product that skips the damaged exon entirely and restores the frame. That reasoning typically narrows the usable region to one or two early exons, a few hundred base pairs in total. Within that window, enumerate PAM sites. With an NGG motif occurring roughly once every eight base pairs across both strands, a 400-base-pair window yields somewhere around fifty candidate spacers. Now apply the filters in sequence. Discard those with extreme composition — below about 35 or above about 70 per cent GC — which removes perhaps a fifth. Discard those containing a run of four or more identical bases, or an internal sequence that would pair with the guide scaffold, which removes a few more. Score the remainder for predicted activity and keep the upper half. Perhaps fifteen candidates survive to the specificity analysis. Search each against the genome at up to four mismatches with single-base bulges permitted. The distribution of results across candidates is wide and this is where the real discrimination happens. Many guides will return a handful of off-target sites, all in intergenic regions, all with mismatches in the seed where they matter most. Some will return dozens, including several with only two mismatches, both distal. And occasionally one will return a near-perfect match — one or two mismatches, all distal — which is usually diagnostic of something structural rather than coincidental. That structural something is almost always a paralogue: a gene that arose by duplication and retains high sequence identity with the target. The haemoglobin gene cluster is the textbook case. The beta-globin gene and the delta-globin gene sitting beside it are the products of an ancient duplication and remain highly similar across their coding sequence, so a guide designed against one has excellent prospects of cutting the other. For a therapeutic programme in a blood disorder, cutting the neighbouring gene as well is not a theoretical concern. The same trap exists in every expanded gene family, and it is the reason the uniqueness check must be run against the whole genome rather than against the gene of interest — a mistake that looks absurd stated plainly and that the convenience of gene-centric tools makes easy. Among the candidates with clean profiles, pick three to five with different characteristics: one with the best activity score, one with the cleanest specificity, one targeting a different exon in case the first exon turns out to be inaccessible in this cell type. Order them, test them, measure editing at the on-target site and at each candidate's top off-target sites, and choose. The whole analysis takes an afternoon and costs a few hundred dollars in synthesis. That is the measure of how far the addressing problem has been commoditised. What has not been commoditised is the judgement at each step — why the first exon and not the last, why a near-perfect off-target match signals a paralogue, why three candidates rather than one. A tool that returns a ranked list will not supply any of it, and the failures in this field are overwhelmingly failures of that judgement rather than failures of the search. What this means for design The addressing analysis yields a set of requirements that any serious guide design procedure must meet, and they explain the shape of the pipeline described in the next chapter. The target site must have a valid PAM, which constrains where you can act. Among valid sites, you want one whose full twenty-base sequence appears once in the genome, and whose near-matches — sites differing by one, two, or three bases, with and without bulges — are either absent, or present only in places where a cut would be harmless. "Harmless" has to be defined: an off-target cut inside a tumour suppressor gene is a different proposition from one in a gene desert, and a cut in a cell type that will be discarded is different from one in long-lived stem cells. Off-target risk is therefore not a single number but a list of locations with annotations. You also want the guide to work. Sequence features that predict efficiency are only loosely related to the features that predict specificity, and sometimes oppose them. The site must be physically accessible in the target cell type, which depends on chromatin state and cannot be read off the sequence at all. And the guide itself must be manufacturable and stable, which rules out certain compositions and self-complementary sequences that fold into hairpins and sequester the spacer. Finally, if the goal is precise correction rather than disruption, the cut must land close enough to the intended change for a repair template to be used efficiently — generally within a few tens of bases, with efficiency falling off steeply beyond that. This constraint often conflicts with the specificity constraint, because the small number of PAM-proximal sites near a given mutation may all have poor off-target profiles. When it does, the honest answer is to change modality rather than to force the design, which is one reason base and prime editors have displaced nuclease-plus-template approaches for most point-mutation correction. That is the specification. Satisfying it is a computational problem with real structure, a real literature, and real tools, and it is what the next chapter is about. Chapter 3: Designing the Guide Guide design is a pipeline. Input: a genome, a target gene or a specific position, a cell type, and an intent — disrupt, correct, insert, modulate. Output: a ranked list of candidate spacer sequences with predicted on-target activity, an enumerated set of potential off-target sites with annotations, and a decision about which candidates go to the bench. Everything in between is sequence analysis, statistical prediction, and constraint checking, and all of it runs before any molecule is made. The pipeline has five stages. They are worth walking through in order, because each stage has a characteristic failure mode and the failures are not interchangeable. Enumeration and uniqueness The first stage is mechanical. Take the target region, find every position with a valid PAM on either strand, and write down the twenty bases adjacent to each. For a typical exon of 150 base pairs this yields on the order of thirty to forty candidates. For a whole gene, several hundred. The enumeration is trivial; what matters is the filtering that follows. Each candidate is then searched against the whole genome to count exact matches and near-matches. This is where the computational interest starts, because the search is not a simple string match. You need, for each candidate, all genomic sites within some edit distance — conventionally up to three or four mismatches — and ideally all sites reachable with a one-base bulge as well. A naive scan of 6.2 billion positions against several hundred queries with approximate matching is expensive, so the tools use index structures: suffix arrays, Burrows-Wheeler transforms, or hash tables keyed on the PAM-proximal seed. The standard approach exploits the mechanism directly, indexing on the seed region because a site with mismatches concentrated in the seed is unlikely to be cut and can be discarded cheaply. The important design decision at this stage is the mismatch threshold, and it is a genuine trade-off rather than a technical detail. Searching to three mismatches is fast and finds the sites most likely to be cut. Searching to five finds a vastly larger set — tens of thousands of sites for many guides — of which almost none will be cut, and the list is useless in that form. Searching to four with bulges allowed is roughly where most therapeutic programmes operate, and the resulting list is typically tens to hundreds of sites requiring triage. The characteristic failure of this stage is an incomplete reference. If the patient or cell line being edited has a variant that the reference genome does not, a site that looks like a three-mismatch off-target in the reference may be a perfect match in that individual, or vice versa. This is not hypothetical: common human variation is dense enough that a non-trivial fraction of designed guides have altered match profiles in some populations. Serious programmes now search against variant databases as well as the reference, and for individualised therapies, against the patient's own sequenced genome. On-target activity prediction Of the candidates that survive, many will cut poorly or not at all, and the difference between a good and a bad guide at the same locus is routinely twenty-fold. Predicting which is which from sequence alone has been one of the field's genuine machine learning successes, achieved the way such successes usually are: by generating a very large labelled dataset. The datasets came from pooled screens. Thousands of guides are synthesised as a library, delivered to a population of cells, and the resulting editing efficiency at each site is measured by sequencing — either directly, by amplifying each target, or indirectly through a functional readout such as loss of a surface protein. Several such datasets covering tens of thousands of measured guides were published in the mid-2010s and remain the basis of the models in use. What the models learned is instructive. Sequence composition at specific positions matters: the base immediately adjacent to the PAM has a strong effect, with G favoured and C disfavoured; certain positions in the seed show clear preferences; overall GC content has an optimum around 40 to 60 per cent, with both extremes performing worse. Features beyond the twenty-base spacer matter too — the bases flanking the PAM influence activity, which is why models take a longer window as input. Predicted secondary structure of the guide RNA matters, since a spacer that pairs with the scaffold cannot pair with DNA. And thermodynamic features — the predicted melting behaviour of the guide-target duplex, and of the DNA duplex that must be displaced — carry real signal, which is reassuring because it means the models are partly learning mechanism rather than only correlation. Successive generations of model moved from hand-crafted features with gradient-boosted trees to convolutional neural networks taking raw sequence, with modest gains. Published correlations between prediction and measurement, on held-out data from the same experimental platform, run around 0.7 to 0.85 as Spearman coefficients. Across platforms and cell types they fall considerably, often to 0.5 or below. This generalisation gap is the honest headline. The models rank candidates usefully; they do not predict absolute efficiency in a new context, and treating a score as a number rather than a rank is a mistake people still make. What the models cannot see at all is chromatin. A target site wrapped in tightly packed heterochromatin may be inaccessible regardless of its sequence quality, and accessibility is cell-type specific. Where data exists — from assays that map open chromatin genome-wide in particular cell types — incorporating it measurably improves prediction. Where it does not, the prediction is blind to one of the larger sources of variance, and the practical remedy is to carry three to six candidates to the bench rather than one. Off-target scoring and triage Now the enumerated off-target list has to be turned into a risk assessment. Two generations of approach coexist. The older approach assigns each potential off-target site a cutting likelihood based on a position-specific mismatch penalty — a weight for each position in the spacer, derived from experiments that measured how much a mismatch at that position reduced activity — and then aggregates the site scores into a single specificity number for the guide. This is cheap, interpretable, and crude. It assumes mismatch penalties are independent, which they are not; two mismatches close together are far more damaging than the product of their individual penalties suggests. The newer approach trains a model on measured off-target activity, taking the aligned guide-target pair as input and predicting cleavage. These models capture interaction effects and handle bulges, and they outperform the weighted-penalty schemes on benchmark data. They remain limited by the training data available, which is biased towards the guides and cell types people happened to study. Both approaches produce a ranked list, and the ranking is only half the job. The other half is annotation, which is a question about consequence rather than probability. A site with a 1 per cent predicted cutting rate in an intergenic region with no regulatory annotation is a different object from a site with a 0.1 per cent rate in the coding sequence of a gene whose disruption drives leukaemia. Therapeutic programmes classify candidate off-targets by genomic context — coding, splice site, promoter, enhancer, cancer-associated gene, essential gene — and the list that goes forward for empirical testing is selected on expected harm, not on probability alone. Table 1 sets out the main classes of score a design pipeline produces, what each is computed from, and what each is actually good for. Table 1. Classes of score in a guide design pipeline. Score class Computed from Predicts Main limitation Position-weighted mismatch Per-position penalties from early cleavage assays Relative off-target cutting Assumes independent mismatches; ignores bulges Learned off-target model Measured cleavage on guide-target pairs Off-target cutting, incl. bulges Training data biased to studied guides and cell types Learned on-target model Pooled screens of 10,000+ guides Relative editing efficiency Poor transfer across cell types and delivery formats Accessibility-informed Open-chromatin maps for the target cell type Whether the site is reachable Needs matched data, rarely available for primary cells Outcome-profile model Deep sequencing of repair products Which indels or allele outcomes dominate Locus- and cell-specific; moderate accuracy Outcome prediction The fourth stage is newer and less widely appreciated. Knowing that a site will be cut efficiently does not tell you what the cell will do with the break, and the answer is not random. Repair outcomes at a given site are strongly reproducible: the same guide in the same cell type gives the same distribution of insertions and deletions across experiments, dominated by a handful of products determined by microhomologies in the sequence flanking the cut. This reproducibility made outcome prediction learnable, and models now exist that take a target sequence and predict its repair outcome spectrum. The practical value is direct. If the intent is to knock out a gene, you want a cut whose dominant outcomes shift the reading frame; a site where the most frequent product is a clean three-base deletion will leave the protein largely intact and the experiment will quietly fail. If the intent is precise correction, you want to know how much of the population will instead acquire a disruptive indel, because that fraction is a side effect that has to be counted. Outcome prediction also changed how people think about therapeutic design. The approved sickle cell therapy does not repair the mutation that causes the disease. It disrupts a regulatory element controlling a transcription factor that represses fetal haemoglobin, so that edited cells make fetal haemoglobin again and the sickling defect becomes clinically irrelevant. That design choice — attack a disruptable regulatory node rather than attempt precise repair of the causal base — follows directly from the asymmetry in difficulty between disruption and correction, and it is the kind of reframing that good engineering produces. Construct and manufacturability checks The last stage is the least glamorous and a surprisingly common source of failure. The chosen spacer has to be realised physically, and that imposes its own constraints. If the guide is to be transcribed inside cells from a standard promoter, that promoter has sequence preferences at the start of transcription — a requirement often satisfied by adding a base, which changes the guide's length and can alter its behaviour. If the guide is chemically synthesised as RNA, as it is for most therapeutic applications, then its sequence must be synthesisable at length and the molecule is typically chemically modified at its ends and at selected internal positions to resist degradation; those modifications interact with activity and must be validated rather than assumed. Self-complementary spacers fold into hairpins and work poorly. Spacers containing sequences that resemble regulatory motifs can have unintended effects on the construct itself. And if the editor is delivered as DNA, the whole cassette must be checked for the things that break any synthetic construct: repeated sequences that invite recombination, restriction sites that interfere with the assembly method, inadvertent open reading frames, and sequence stretches that cannot be synthesised reliably. Commercial synthesis providers run these checks and reject or redesign problem sequences, which is why "the sequence came back with changes" is a familiar annoyance. What the pipeline is for Taken together, the five stages reduce a target locus to a shortlist. The shortlist is not an answer. The output of a well-run design pipeline is three to six candidate guides with complementary predicted properties and an explicit list of the off-target sites that will be checked empirically. Everything after that is measurement. This is worth stating plainly because the tooling's polish invites over-confidence. A web interface that returns a guide with a specificity score of 87 looks like a determination. It is a prior. The next chapter is about how that prior gets tested, and about the methods that exist precisely because computational off-target prediction, after a decade of effort, is still not accurate enough to be trusted on its own. Hashtags: #SyntheticBiology #Bioinformatics #CRISPR #CRISPRDesign #GenomeEditing #GeneticEngineering #DNADataStorage #ComputationalBiology #MolecularBiology #GeneEditing #GuideRNADesign #CRISPRCas9 #GeneTherapy #PrecisionMedicine #OffTargetAnalysis #GenomicSequencing #DNAEncoding #DNASequencing #InformationTheory #ErrorCorrectingCodes #GeneticCircuits #BiologicalComputing #SyntheticGenomics #GenomeEngineering #BaseEditing #PrimeEditing #MachineLearningInBiology #BiotechnologyInnovation #Bioengineering #FutureOfBiotechnology