Download the Book (PDF): Introduction A nurse in Dubai who wants to send two hundred dollars to her mother in Manila faces a strange fact of modern life. She can stream a film from a server on another continent in a second. She can video-call her mother for free. But the money, which is nothing more than a number in a bank's database, may take a day or more to arrive, and on the global average something like thirteen dollars of it will disappear on the way. If she sends it through a bank rather than a specialist money transfer company, the World Bank's own price survey suggests she could lose more than a seventh of it. The usual explanation is that the banking system is old. That is true, and it is also not very useful. Old systems can be fast; the card networks that approve a purchase in under two seconds were designed in the 1970s. The deeper explanation, and the one this book is built on, is that a payment is not one thing. It is three things that have to happen together: a piece of information has to travel from the payer to the payee's bank; a quantity of value has to move from one balance sheet to another; and enough trust has to exist between all the institutions involved that each is willing to act on the information before, or without, seeing the value. Cross-border payments are slow and expensive because, across borders, all three of those things are fragmented at once. Messages pass through formats that drop data. Value has to be pre-positioned in dozens of accounts in dozens of currencies. And trust has to be rebuilt at every hop, through compliance checks that each bank performs on its own, often on data it cannot fully read. The modernization of payment infrastructure now under way is best understood as three separate attacks on those three problems. The migration to ISO 20022, which the global banking network completed for cross-border payment instructions in November 2025, is an attack on the information problem: it replaces cramped, decades-old message formats with structured, data-rich ones that can carry a full name, a parsed address, a purpose code and an invoice reference without truncation. The spread of real-time payment systems, from Britain's Faster Payments in 2008 to India's UPI, Brazil's Pix, the United States' RTP network and FedNow, and the euro area's instant credit transfer mandate, is an attack on the value problem within a single currency: settlement happens in seconds, around the clock, so nobody has to extend credit while waiting for the books to close. And stablecoins, dollar-denominated tokens that move on public blockchains, are an attack on the value problem across borders: they promise to let a dollar travel from Dubai to Manila as a bearer instrument, without a chain of correspondent banks each holding prefunded accounts for the others. The argument of this book is that each of these attacks succeeds against the problem it targets and leaves the other two standing. Rich data makes a payment easier to screen and reconcile, but it does not make the money move any faster. Instant domestic rails make settlement nearly free within a country, but they do not on their own solve the problem of exchanging one currency for another, and their speed creates a new class of fraud. Stablecoins can move value across a border in minutes at trivial network cost, but the expensive parts of a remittance, converting local currency into dollars at one end and dollars back into pesos at the other, while satisfying two sets of financial crime rules, are still there, now performed by exchanges and payout partners instead of correspondent banks. The cost of a cross-border payment is not in the wire. It sits in foreign exchange, in liquidity tied up in advance, and in establishing who is paying whom and why. Any technology that claims to transform cross-border payments has to be judged by which of those it actually reduces. That framing has a practical consequence. It allows a reader to look at any new announcement in this field, a new instant payment link between two countries, a bank's tokenized deposit, a remittance startup built on a blockchain, a central bank's digital currency trial, and ask three plain questions. What happens to the information? Where does the value sit, and who has to prefund it? And who carries the risk that the payment is fraudulent, sanctioned or mistaken? The answers almost always reveal more than the press release. What this book covers The book moves from foundations to frontier. The first chapter explains what a payment is at the level of ledgers: the difference between clearing and settlement, why central bank money sits at the top of the system, and how gross and net settlement systems trade liquidity against risk. Without this, nothing that follows makes sense, because every innovation discussed later is a rearrangement of the same parts. The second chapter follows a cross-border payment through the correspondent banking chain and counts where the cost and delay come from. It draws on the World Bank's long-running survey of remittance prices and on the Committee on Payments and Market Infrastructures' data showing that correspondent relationships have been withering for more than a decade, cutting smaller economies off from the system. The third and fourth chapters deal with ISO 20022. The third explains what the standard actually is, a modelling methodology and a catalogue of messages rather than a single format, and what changed when banks moved from the old SWIFT MT103 to the ISO 20022 pacs.008. The fourth asks what that rich data buys in practice, in sanctions screening, reconciliation and fraud prevention, and why the benefits arrive slowly, if at all, when banks translate old data into new formats rather than capturing it properly at the source. The fifth and sixth chapters deal with real-time payments. The fifth compares how the major instant systems are designed, including the choices they make about settlement, operating hours and access, and why some, like Pix and UPI, achieved mass adoption while others grew slowly. The sixth deals with what instant, irrevocable payment costs: authorised push payment fraud, the liability rules that have grown up to address it, and the liquidity strain on banks that must settle at three o'clock on a Sunday morning. The seventh chapter looks at the effort to link domestic instant systems across borders, through bilateral links like Singapore's PayNow and Thailand's PromptPay and multilateral schemes like Project Nexus, and asks what remains unsolved once the rails are connected. The eighth and ninth chapters turn to stablecoins: first the mechanics of using them as a cross-border rail and an honest accounting of where the cost goes, then the regulatory settlement that has emerged in the United States and Europe and the way banks and the SWIFT network itself are responding with tokenized deposits and shared ledgers. A note on what is left out This is not a book about card payments, buy-now-pay-later, or the economics of merchant acquiring, though all of them ride on the rails described here. It is not a book about cryptocurrency investment; stablecoins appear only as a means of moving money. Central bank digital currencies appear where they bear on cross-border settlement, not as a topic in their own right. And it is not a technical manual: readers who need the exact schema of a pacs.008 message will find the published standards and the Further Reading more useful than any book. The aim is narrower and, I hope, more durable. Payment technology changes quickly and its marketing changes faster. The underlying problem does not change: getting the right information, the right value and enough trust to the same place at the same time, across a boundary where the currency, the law and the banks are all different. A reader who understands that problem will be able to judge the next decade of payment innovation on its merits. Chapter 1: Ledgers, Messages and Settlement Almost all the money in a modern economy exists as entries on ledgers. The banknotes in a wallet are a small fraction of the total; the rest is a set of claims recorded in databases. When a person says they have a thousand dollars in the bank, what they have is a promise from the bank, recorded as a liability on its books, to pay them a thousand dollars on demand. When they pay someone, that promise does not travel. What happens instead is that one ledger entry is reduced and another is increased, and if the two accounts sit in different banks, the banks themselves have to adjust their claims on each other. Every payment system ever built, from a medieval merchant's bill of exchange to a stablecoin on a public blockchain, is a method for making those adjustments agree. Understanding this is the key to reading everything else in this book. A payment that looks like a single act, a tap of a phone or a click on a banking app, is really a chain of ledger changes in different institutions, coordinated by messages. The question of how fast and how cheaply money moves comes down to how many ledgers have to change, how they learn that they should, and what each one risks by changing before the others. Two banks and a message Take the simplest case that involves more than one bank. Alice holds an account at Bank A and wants to pay Bob, who banks at Bank B. Bank A cannot reach into Bank B's ledger and credit Bob. It can only reduce Alice's balance on its own books and tell Bank B to increase Bob's. For Bank B to do so, it needs two things: information, meaning who to credit, how much, and on whose behalf; and value, meaning some assurance that Bank A will actually compensate it for the new liability it is taking on toward Bob. Payments professionals separate these into two steps. Clearing is the exchange and reconciliation of payment information: transmitting the instruction, checking it, and calculating what each bank owes the others. Settlement is the discharge of those obligations: the moment when Bank A's debt to Bank B is extinguished by a transfer of some asset that Bank B accepts as final. In everyday banking the two are often separated by hours or days. A traditional cheque is cleared when the banks exchange and verify it, and settled when the net amounts are transferred between them at the end of a cycle. What asset do banks use to settle with each other? They could, in principle, use claims on each other: Bank A could simply owe Bank B. But then Bank B would be exposed to Bank A's failure, and in a system of thousands of banks, chains of such exposures would allow one failure to spread. So in nearly every country the answer is the same. Banks hold accounts at the central bank, and settle by moving balances between those accounts. Central bank reserves are the one asset in the system that carries no credit risk in its own currency, because the central bank cannot run out of the money it issues. This is why payments specialists speak of a hierarchy of money: banknotes and central bank reserves at the top, commercial bank deposits beneath them, and claims on payment companies and e-money issuers beneath those. A payment between two customers at different banks is, at bottom, an exchange of one bank's liability for another's, made safe by a movement of the top layer between the banks. The message and the money therefore travel on different paths. The message goes from Bank A to Bank B, often through a clearing house or a network such as SWIFT, which carries only information and holds no money at all. The value moves across the central bank's books. A good deal of confusion about payments comes from mistaking the first for the second. When people say that "SWIFT moves trillions of dollars a day," they mean that SWIFT carries messages instructing banks to move trillions of dollars; the money itself moves on ledgers that SWIFT never touches. Gross or net: the oldest trade-off in payments Once there is a central bank ledger to settle on, the system designer faces a basic choice. Should each payment be settled individually, as it happens, or should payments be accumulated and only the net differences settled periodically? The first approach is called real-time gross settlement, or RTGS. Every payment is settled one by one, in full, in central bank money, the moment it is processed. Once settled, it is final and cannot be unwound. The Federal Reserve's Fedwire Funds Service, the euro area's T2, the Bank of England's RTGS system that underpins CHAPS, and their equivalents in almost every country work this way. Their virtue is safety: no bank ever owes another for longer than an instant, so the failure of one bank cannot leave others holding unpaid claims. Their cost is liquidity. To make a payment, a bank must have the full amount in its central bank account at that moment. A bank that has to pay out a billion dollars at ten in the morning, and expects to receive a billion dollars at eleven, still needs a billion dollars on hand at ten. The second approach is deferred net settlement. Payments are exchanged throughout a period, a day, an hour, a few minutes, and at the end only each bank's net position is settled. If Bank A owes Bank B a hundred million and Bank B owes Bank A ninety-five million, only five million moves. Netting can reduce the liquidity needed to settle by very large fractions. The automated clearing house systems that process payroll and direct debits in most countries work this way, as did the cheque systems before them. The cost is risk. Between the moment a payment is accepted and the moment the net position is settled, banks are effectively extending credit to each other. If one fails before settlement, the others may find that payments they have already credited to their customers are not covered. The history of payment systems design is largely the history of managing this trade-off. The most famous lesson came in June 1974, when German regulators closed Bankhaus Herstatt in the middle of the business day. Herstatt's counterparties had already paid it Deutsche marks in Frankfurt and were waiting to receive dollars in New York, several hours behind. The dollars never came. "Herstatt risk," the risk that one leg of a currency trade settles and the other does not, became the name for a whole class of problems, and it took until 2002 for the industry to build CLS, a specialised system that settles both legs of foreign exchange trades simultaneously. That gap of nearly three decades is a reminder of how slowly the plumbing of finance changes, even after a crisis has shown exactly what is wrong. Modern systems blend the two approaches. Many RTGS systems include liquidity-saving mechanisms that hold queued payments and release them in offsetting batches. Many net systems require participants to post collateral covering their maximum possible net debit, so that a failure can be absorbed. And, as later chapters show, most instant payment systems deal with the trade-off in a distinctive way: they settle in real time but require banks to prefund a special account in advance, so that every payment is covered by money set aside before it is made. Finality and the problem of the irrevocable payment Settlement matters for one reason above all: it determines when a payment becomes final. Finality means that the payment cannot be revoked or unwound, by the payer, by the payer's bank, or by a liquidator if one of the banks fails. Legal systems give special protection to final settlement in designated payment systems precisely so that a bank receiving funds can rely on them absolutely. Finality for banks and finality for customers are different things, and the gap between them is where much of the risk in payments lives. In a card payment, the merchant is paid, but the cardholder retains the right to dispute the charge for months, and the card network's rules allocate the loss if the dispute succeeds. In a direct debit, the payer can often claim a refund for weeks. In a credit transfer, a push payment initiated by the payer, there is usually no such right: once the money has been sent to the account the payer named, the payer's bank has done what it was asked. This distinction between pull payments, where the payee takes money with the payer's authority, and push payments, where the payer sends it, shapes both the convenience and the fraud profile of every system. Instant payment systems are push systems with immediate finality, and that combination, as the sixth chapter shows, is precisely what makes them attractive to fraudsters. Access: who gets an account at the top If settlement in central bank money is the foundation, then the question of who may hold an account at the central bank is the question of who can build a payment business without depending on a bank. For most of the twentieth century the answer was simple: banks, and only banks. A payment company, a money transfer operator, or a mobile wallet that wanted to send money had to do so through a sponsoring bank's account. This rule has slowly loosened. The Bank of England opened settlement accounts to non-bank payment service providers in 2017. The European Union's revision of its Settlement Finality Directive, agreed as part of the instant payments package in 2024, allows payment and e-money institutions to participate directly in designated payment systems. Brazil's central bank designed Pix to admit regulated payment institutions alongside banks from the start. In the United States, access to Federal Reserve accounts remains narrowly held and contested, and that matters, because a firm that cannot settle in central bank money must always have a bank between it and the top of the hierarchy, with all the costs and compliance gatekeeping that implies. Access is also the hinge on which the stablecoin argument turns. A stablecoin issuer holds reserves, cash and short-term government securities, at banks and custodians, and issues tokens that circulate among anyone with a compatible wallet. In effect it creates a new ledger outside the banking system's hierarchy, on which value can move without touching the central bank at all until someone redeems. Whether that is an innovation or a regression, and whether it is safe at scale, depends on exactly the questions this chapter has introduced: what backs the claim, when settlement is final, and who bears the loss when something fails. One payment, four ledgers It helps to follow a single payment through the books, because the abstractions become concrete very quickly. Suppose Alice pays Bob one thousand dollars, from her account at Bank A to his at Bank B, and suppose both banks settle through the central bank. Before the payment, four ledgers hold the relevant entries. Bank A's ledger shows a liability of, say, five thousand dollars to Alice. Bank B's ledger shows a liability of two thousand dollars to Bob. The central bank's ledger shows reserve balances for Bank A and Bank B, which are liabilities of the central bank and assets of the banks. And each bank's own books record its reserve balance as an asset. When the payment is made, Bank A reduces its liability to Alice by one thousand dollars. To keep its books balanced, it must also reduce an asset by one thousand dollars, and the asset it gives up is reserves: it instructs the central bank to move one thousand dollars from its reserve account to Bank B's. The central bank debits Bank A and credits Bank B; its total liabilities are unchanged, merely reassigned. Bank B now has one thousand dollars more in reserves, a new asset, and it records a matching new liability by crediting Bob's account. Every ledger balances. Alice has a smaller claim on Bank A, Bob a larger claim on Bank B, and the banks have settled the difference in the one asset both trust. Three things about this sequence are worth noticing. First, nothing physical moved; four ledgers were updated in a coordinated way. Second, the order matters. If Bank B credits Bob before it receives the reserves, it is extending credit to Bank A, and if Bank A fails in the meantime, Bank B bears the loss. If Bank A debits Alice but the message to Bank B is lost, Alice's money is in limbo. The whole art of payment system design lies in making the updates happen in an order, and with safeguards, that leave no one exposed to a loss they did not agree to bear. Third, the information in the message is what tells Bank B whom to credit. If the message says "pay account 12345" but omits Bob's name, Bank B must trust the account number. If the account number is wrong, the reserves arrive but the wrong customer is credited, or the payment cannot be applied at all and must be returned, which means the whole sequence runs again in reverse. Now change one assumption. Suppose Bank B is in another country, with another currency, and has no account at Bank A's central bank. Bank A cannot instruct the central bank to credit Bank B, because Bank B is not on that ledger. It must find some institution that is on both ledgers, or a chain of institutions that together span them, and route the payment through each of them in turn. Every link in that chain is another pair of ledger updates, another message, another moment when one party is exposed to another. That, in miniature, is the cross-border payment problem. Messages are the slow part One more distinction completes the toolkit. Payment systems have historically been slow not mainly because moving central bank money is slow (an RTGS system settles a payment in seconds) but because of everything around it: batch processing schedules, operating hours, the need for human review of unclear instructions, and the translation of information between formats as a payment passes from one system to another. Consider the information that a business needs when it receives a payment. It needs to know who paid, and for what: which invoice, which order, which account. A payment that arrives as "ACME CORP 4500.00" with no reference forces a clerk to search through open invoices and guess. Multiply that by thousands of payments a day and the cost of poor information becomes one of the largest hidden costs in commerce. The same is true of compliance: a bank screening a payment against sanctions lists needs to know the full name and address of the parties, and a name that has been truncated to fit a short field may match, or fail to match, the wrong person. For most of the history of electronic payments, the formats used to carry this information were designed when bandwidth and storage were expensive. Fields were short, fixed in length and loosely structured. A name and address had to fit into four lines of thirty-five characters. A remittance reference had to fit into a similar space. Information that did not fit was cut off or left out. Every time a payment passed between systems with different formats, more could be lost. The rich data that a business had in its invoicing system rarely survived the journey to the payee. This is the background against which ISO 20022 should be understood. It does not make money move faster in itself; settlement speed is determined by the settlement system. What it changes is the quality of the information that travels alongside the money, and therefore how much human work, delay and error accumulate at each step where someone has to decide whether to act on an instruction. In domestic payments, the information problem can be solved inside one system with one format. In cross-border payments, where a payment may pass through four or five institutions in three or four countries, it is the information problem as much as the value problem that determines whether a payment takes ten minutes or three days. The next chapter follows such a payment through the correspondent banking chain to see exactly where those days and dollars go. Chapter 2: The Correspondent Chain There is no global central bank. That single fact explains most of what is slow and expensive about sending money across a border. Within a country, every bank can settle with every other bank on the central bank's books. Between countries, there is no common ledger at the top of the hierarchy, and there are many currencies rather than one. A bank in Manila cannot hold an account at the Federal Reserve as a matter of course, and a bank in Dubai cannot settle directly in Philippine pesos on the books of the Bangko Sentral ng Pilipinas. So international payments have always relied on a workaround: banks hold accounts with each other. Nostro, vostro and the prefunding problem The arrangement is called correspondent banking. A bank that wants to make and receive payments in a foreign currency opens an account with a bank in that currency's home country, or with a large international bank that has access to that currency's settlement system. From its own point of view, the account it holds abroad is its nostro account, Latin for "ours". From the foreign bank's point of view, the same account is a vostro, "yours". When a Philippine bank holds a dollar account at a large New York bank, it can receive dollars into that account and pay dollars out of it, and the New York bank settles those dollar movements on its own books or through Fedwire and CHIPS. The trouble is scale. No bank can hold accounts in every currency with every other bank. A regional bank in the Gulf might hold nostro accounts in a dozen currencies with a handful of large correspondents. A small bank in West Africa might hold one dollar account and one euro account, both with banks in Europe. When a payment has to go from a customer of one small bank to a customer of another small bank in a different region, it often has to pass through two, three or more correspondents, each of which holds an account for the next. This is the correspondent chain. Each link in the chain requires money to be sitting in advance in the right place. If a bank wants to be able to pay out dollars on behalf of its customers, it must keep enough dollars in its nostro account to cover the payments it expects, and that money earns little and cannot be used for anything else. Industry estimates of the total trapped in nostro accounts globally run into trillions of dollars, though they are rough and should be treated as illustrations of scale rather than measurements. The point does not depend on the precise number. Prefunded liquidity is a real cost, and it is ultimately paid for by the people who send money, in fees and in exchange-rate margins. Liquidity also explains why smaller corridors are so much more expensive than large ones. A bank sending dollars to Mexico can rely on enormous, constant two-way flows; its correspondent accounts are always being replenished. A bank sending money from South Africa to Malawi, or from Japan to Nepal, sees much thinner and more one-directional flows. It must hold proportionally larger buffers, its correspondents earn less volume per relationship, and the fixed costs of maintaining the relationship are spread across fewer payments. Following a payment through the chain Imagine a small manufacturer in Ohio paying a supplier in Vietnam twenty thousand dollars, to be received in Vietnamese dong. The manufacturer's community bank does not have a relationship with any Vietnamese bank. It sends the instruction to its own correspondent, a large US bank. That bank holds a vostro account for a large Asian bank with operations in Vietnam, or, more likely, has a relationship with an intermediary that does. The dollars move from the community bank's account to the large bank, from the large bank to the intermediary, and from the intermediary to the Vietnamese beneficiary's bank, where they are converted into dong and credited. There are two ways the information and the value can travel along this chain. In the serial method, a single customer payment message passes from bank to bank, each one debiting the sender's account and crediting the next. Every bank sees the full payment details. In the cover method, the ordering bank sends the customer payment message directly to the beneficiary's bank, and separately sends a bank-to-bank "cover" payment through the correspondents to move the money. The cover method is faster for information but historically allowed intermediaries to move funds without seeing who the ultimate customers were. After investigations in the late 2000s found that this had been used to hide sanctioned parties, SWIFT introduced a special cover message, the MT202 COV, in 2009 that carries the underlying customer information through the chain. The episode is a small illustration of a recurring pattern: payment data became richer because regulators demanded transparency, not because banks wanted it. At each step, things can go wrong. A field may be filled in inconsistently, and the next bank's systems may reject it or route it to a queue for manual repair. A name may produce a possible match against a sanctions list and the payment may be held while an analyst investigates. A cut-off time may be missed, so the payment waits for the next business day. If the payment crosses several time zones, the window in which all the banks along the chain are open at once may be short. And at the end, the beneficiary's bank may hold the funds while it seeks information it needs under its own country's rules, such as the purpose of the payment, which in some countries must be reported for foreign exchange controls. Each step can also cost money. There are three ways of allocating charges on an international wire, conventionally abbreviated as OUR (the sender pays all charges), SHA (charges are shared, each side paying its own bank, with intermediaries deducting their fees from the principal) and BEN (the beneficiary pays everything). Under SHA, the most common option, each intermediary in the chain may take a "lifting fee" out of the amount in transit. The sender does not know in advance how many intermediaries there will be or what each will charge. The supplier in Vietnam expects twenty thousand dollars and receives something less, with no clear explanation of where the difference went. Then there is the exchange rate. The rate applied to convert dollars to dong is set by the converting bank, and the margin above the wholesale rate is often the largest single component of the total cost, but it is rarely shown as a fee. The hidden price: exchange-rate margins Of all the costs in the chain, the exchange-rate margin is the one customers understand least and the one that most rewards opacity. A bank or money transfer company that converts currency on a customer's behalf rarely charges a visible conversion fee. Instead it applies its own rate, which is less favourable than the rate at which banks trade with each other in the wholesale market. The difference is a charge in all but name. A simple example shows why this matters. Suppose a customer sends one thousand dollars and is told the fee is fifteen dollars. The provider converts the remaining 985 dollars at a rate two percent worse than the wholesale mid-market rate. The customer sees a fifteen-dollar fee; the true cost is fifteen dollars plus roughly twenty dollars of exchange-rate margin, more than double what was advertised. If the payment travels under shared charges and an intermediary deducts its own lifting fee from the principal, the beneficiary receives less again, and neither party can easily reconstruct where the money went. The arithmetic here is illustrative, but the structure is exactly how international transfers have long been priced. Regulators have tried to force this cost into the open. In the United States, the Dodd-Frank Act of 2010 led the Consumer Financial Protection Bureau to adopt a remittance transfer rule that took effect in 2013, requiring providers to disclose to consumers, before payment, the exchange rate, the fees and the amount the recipient will receive, and giving senders a short window to cancel. In the European Union, a 2019 amendment to the cross-border payments regulation required providers to disclose currency conversion charges as a percentage markup over the latest euro foreign exchange reference rates published by the European Central Bank, so that customers can compare offers. These rules have helped, and the success of newer remittance firms that advertise the mid-market rate and a separate transparent fee owes something to them. But disclosure rules cover consumer remittances in particular jurisdictions; they do not reach most business payments, and they cannot, by themselves, reduce the underlying cost of converting a thinly traded currency. The margin is also, in part, a payment for genuine risk and genuine work. A bank that quotes a rate for a payment that will be converted later bears the risk that the market moves. A bank that must hold balances in a volatile currency to pay out on demand bears the cost of that inventory. In the most liquid currency pairs the justified margin is tiny; in the least liquid, it can be substantial. The problem is less that margins exist than that, for decades, nobody outside the bank could tell how much of a given margin was payment for risk and how much was simply what the market would bear. What the numbers say The best long-running evidence on what this system costs ordinary people comes from the World Bank's Remittance Prices Worldwide database, which since 2008 has surveyed the price of sending the equivalent of two hundred dollars, and of five hundred dollars, across several hundred corridors from sending to receiving countries. It records both fees and the exchange-rate margin, so it captures the full cost to the sender. Its findings have been remarkably stable. In the first quarter of 2025, the global average cost of sending two hundred dollars was 6.49 percent, up from 6.26 percent in the previous quarter. The United Nations' Sustainable Development Goals include a target of reducing remittance costs below three percent by 2030 and eliminating corridors costing more than five percent. The world is not close. The breakdown by type of provider shows where the correspondent chain bites hardest, as Table 1 sets out. Banks, which mostly rely on traditional correspondent arrangements, were by far the most expensive channel. Specialist money transfer operators were around five percent. Mobile operators, which typically collect and pay out through their own networks and settle in bulk, were slightly cheaper still. The World Bank also reports that sending through digital channels averaged 4.85 percent against 7.16 percent for non-digital ones. Table 1. Average cost of sending $200, by type of provider, first quarter 2025. Provider type Average total cost Source Banks 14.55% World Bank RPW, Q1 2025 Post offices 7.71% World Bank RPW, Q1 2025 Money transfer operators 5.04% World Bank RPW, Q1 2025 Mobile operators 4.97% World Bank RPW, Q1 2025 Global average, all providers 6.49% World Bank RPW, Q1 2025 Region matters too. In the same survey, sending money to South Asia was the cheapest, at 4.80 percent on average, while sending to sub-Saharan Africa was the most expensive, at 8.78 percent. The pattern fits the liquidity story: India, Pakistan and Bangladesh receive huge, steady remittance flows, and competition among providers is intense. Many African corridors are smaller, involve currencies that trade thinly, and pass through more intermediaries. These are not small sums. The World Bank estimated that officially recorded remittances to low- and middle-income countries reached 685 billion dollars in 2024, with India alone receiving 129 billion and Mexico 68 billion. Over the preceding decade, remittances had grown by more than half while foreign direct investment to those countries fell. A cost of six or seven percent on that flow is a transfer of tens of billions of dollars a year from migrant workers and their families to the payment industry. That is why remittance costs have become a matter of development policy and why the Group of Twenty, in 2020, adopted a roadmap for enhancing cross-border payments, followed in 2021 by quantitative targets. For retail cross-border payments generally, the targets call for an average cost of no more than one percent by the end of 2027, with three-quarters of payments arriving within an hour; for remittances, they adopt the three percent goal, with no corridor above five percent, by 2030. The Financial Stability Board, which monitors progress, has acknowledged that the 2027 targets are unlikely to be met. Speed: faster than its reputation, slower than it should be The correspondent system's reputation for slowness is partly out of date. In 2017 SWIFT launched its global payments innovation service, known as gpi, which introduced a unique end-to-end transaction reference, the UETR, that travels with each payment and lets every bank in the chain report its status to a central tracker. Banks that join commit to processing payments on the day they receive them and to passing on the full amount and charges information. The result, according to SWIFT's own figures published in late 2024, is that about 90 percent of cross-border payments on its network reached the beneficiary's bank within an hour. The bottleneck is the last step: only 43 percent were credited to the end customer's account within an hour. That gap between reaching the bank and reaching the customer is telling. SWIFT attributes it to domestic factors at the receiving end: the operating hours of local payment systems, compliance checks the receiving bank performs, domestic verification practices, and currency controls. In other words, the slowest parts of a cross-border payment are no longer in the network that carries the message between countries; they are in the information and trust problems that arise when a foreign payment lands in a local system with its own rules. That is precisely the problem that data-rich messaging and linked instant payment systems are intended to address. The withering of the network The most troubling trend in correspondent banking is not cost or speed but contraction. Since the 2010s, the Committee on Payments and Market Infrastructures, a body of central banks hosted by the Bank for International Settlements, has published data on the number of active correspondent relationships worldwide, drawn from SWIFT traffic. Its data show a steady decline. Between 2011 and 2022, the number of active correspondents fell in every region, with steep drops in Oceania, Africa, and the Americas outside North America, where it fell by close to half. The number of correspondents active in US dollars and in euros each fell by around two-fifths. Some individual countries lost the large majority of their correspondent relationships. The reasons are well understood. After a series of large enforcement actions against international banks for sanctions and money-laundering failures in the late 2000s and early 2010s, the largest correspondent banks re-examined their relationships and concluded that many were not worth the risk. A correspondent bank is, in effect, vouching for its respondent's customers, whom it cannot see directly. The cost of the due diligence needed to do that properly, "know your customer's customer," is high and largely fixed, while the revenue from a small respondent bank in a small country is low. Rather than manage the risk, banks exited it. The practice became known as de-risking. De-risking has consequences that run against the stated aims of financial crime policy. When a small country loses its correspondent relationships, payments do not stop; they move to fewer, more concentrated channels, or to informal ones beyond the view of any regulator. Remittance companies serving high-risk corridors have repeatedly found their own bank accounts closed. Pacific island states, which depend heavily on remittances from Australia and New Zealand, have been particularly exposed. The irony is that the network was hollowed out by the difficulty of establishing trust over thin information: banks could not see enough about the payments passing through them to be confident they were legitimate. What the chain teaches The correspondent chain reveals the three problems identified in the introduction in their sharpest form. The information problem shows up as payments that are truncated, misread or held for investigation because a field is unclear. The value problem shows up as trillions in prefunded nostro balances, thin corridors that cost more, and exchange-rate margins that dwarf visible fees. The trust problem shows up as compliance holds, lifting fees for every hop, and, at the extreme, the withdrawal of whole countries from the network. None of these is a technological problem in the narrow sense. The messages move in seconds. Settlement in each currency, at each end, is fast. What is slow and costly is the repeated work of establishing, at each bank in the chain, that it is safe to act. The question for every modernization effort discussed in the rest of this book is which of these three costs it reduces. The first and most comprehensive attempt, a twenty-year project to replace the language in which banks speak to each other, is the subject of the next two chapters. Chapter 3: A New Language for Money On the weekend of 22 and 23 November 2025, the international banking system quietly changed the language it speaks. After that weekend, SWIFT, the cooperative that carries payment instructions among more than eleven thousand five hundred financial institutions in over two hundred countries and territories, ended the period during which banks could send cross-border payment instructions in either the old MT format or the new ISO 20022 format. ISO 20022 became mandatory for payment instructions. On the first business day after the cutover, SWIFT reported, 97 percent of payment instructions on its network were already being sent in ISO 20022, with a conversion service handling the stragglers for a transition period. It was the end of a migration that began in earnest in March 2023 and that had been under discussion for nearly twenty years before that. Few events of such significance to global commerce have passed so unnoticed by the public. To understand why the change mattered and why it took so long, it is necessary to understand what was being replaced, and what ISO 20022 actually is. The common description, "a new messaging format", is both accurate and misleading. The world of MT SWIFT went live in 1977, replacing the telex as the means by which banks sent each other payment and other instructions. Its message types, known as MT messages, were numbered by category: the MT1xx series for customer payments, MT2xx for bank-to-bank transfers, MT9xx for statements and confirmations. The MT103, the single customer credit transfer, became the workhorse of international payments, the message behind most international wires sent by companies and individuals for nearly half a century. The MT format reflected its age. It was a text format made of numbered fields, each with tight rules on length and character set. The ordering customer, field 50, and the beneficiary customer, field 59, each allowed a name and address in four lines of thirty-five characters. The remittance information, field 70, which tells the beneficiary what the payment is for, allowed another four lines of thirty-five characters. There was no separate field for a street, a postcode or a city; these were typed into the free lines in whatever order the sender's system chose. There was no standard way to identify a company by a global identifier other than the bank's own code, the BIC. A payment might say "ACME TRADING CO LTD 14 HARBOUR RD KOWLOON HONG KONG" across two lines, and every system downstream had to guess which part was the name, which the street and which the city. The MT standard was extraordinarily durable, and for good reasons. It was compact, every bank in the world could process it, and decades of software had been written around it. But it was a format that could not grow. Each new regulatory demand, more information about the payer for anti-money-laundering rules, a code for the purpose of the payment, an identifier for the ultimate beneficiary behind an intermediary, had to be squeezed into existing fields or added through awkward conventions. Domestic payment systems, meanwhile, had developed their own formats, so a payment crossing from one system to another often had to be translated twice, losing information each time. What ISO 20022 actually is ISO 20022 is a standard published by the International Organization for Standardization, first in 2004, under the title "Universal financial industry message scheme". The important word is scheme. ISO 20022 is not one message format but a method for designing message formats, combined with a central repository of the business concepts and messages that have been designed with it. The method works in layers. At the top is a business model: a description, independent of any technology, of the actors and activities in a financial process, such as a debtor, a creditor, a debtor's agent, an instruction to pay, a status report. Beneath that is a logical model of the messages those activities require, built from a shared dictionary of business components: a party has a name, a postal address, and identifiers; a postal address has a street name, a building number, a post code, a town name and a country; an amount has a value and a currency. Only at the bottom is the syntax, the actual encoding used to send a message. In practice that syntax is almost always XML, which is why ISO 20022 messages are often called "MX" messages in SWIFT's terminology, but the standard allows for others. Because every message is assembled from the same dictionary, the same concept means the same thing everywhere. A creditor's postal address in a payment initiation from a company to its bank has exactly the same structure as a creditor's postal address in the interbank payment and in the statement that reports it. Data can flow from one stage to the next without translation. That is the fundamental advance over the MT world, where each stage used a different format and every handoff was an opportunity for loss. The messages are grouped into business areas, identified by a four-letter prefix. Three matter most for payments. The pain messages, for payments initiation, carry instructions from a customer to its bank: the pain.001 is a company's instruction to its bank to make one or more credit transfers. The pacs messages, for payments clearing and settlement, carry instructions between banks: the pacs.008 is the interbank customer credit transfer, the direct successor to the MT103; the pacs.009 is a bank-to-bank transfer, succeeding the MT202; the pacs.002 reports the status of a payment; the pacs.004 returns one. The camt messages, for cash management, carry reports and exceptions: the camt.053 is an end-of-day bank statement; the camt.054 a debit or credit notification; the camt.056 a request to cancel a payment. A full payment lifecycle, from a company's accounts-payable system to its supplier's bank statement, can now in principle be expressed in one coherent family of messages. What changed in the payment itself The difference between the old and new formats is clearest when they are set side by side for the elements that matter most to banks and their customers. Table 2 summarises the main contrasts between the MT103 and the pacs.008 as used for cross-border payments. Table 2. How key payment data is carried in the MT103 and the ISO 20022 pacs.008. Data element MT103 pacs.008 (ISO 20022) Party name Shares four lines of 35 characters with the address Dedicated name element, up to 140 characters Postal address Free-text lines in no fixed order Separate elements for street, building, post code, town, country Remittance information Four lines of 35 characters Unstructured text or structured references to invoices Ultimate parties No dedicated field Ultimate debtor and ultimate creditor elements Organisation identifiers No dedicated field outside bank codes Dedicated elements, including the Legal Entity Identifier Purpose of payment No standard field Purpose element using a shared external code list Several of these differences deserve explanation, because each corresponds to a real-world problem. The name and address changes matter for sanctions screening and for fraud prevention. When a name is cut short to fit a line, or merged with an address, a screening system may flag an innocent payment because a fragment matches a listed name, or may miss a real match because the listed name is split across lines. When an address is structured, a bank can tell immediately which country the beneficiary is in, rather than parsing free text and hoping it found the country rather than a street named after one. The ultimate party elements matter because so many modern payments are made on behalf of someone else. When a payment company pays a freelancer on behalf of a client, or a corporate treasury centre pays a supplier on behalf of a subsidiary, the account holder is not the party with the real economic interest. In the MT world, the real payer either disappeared or was squeezed into free text. In ISO 20022, the ultimate debtor and ultimate creditor have their own place, so the information can be screened and reported. The organisation identifier elements matter because names are ambiguous and identifiers are not. The Legal Entity Identifier, a twenty-character code administered by the Global Legal Entity Identifier Foundation since the aftermath of the 2008 financial crisis, uniquely identifies a legal entity worldwide. A payment that carries the LEI of both payer and payee can be matched against company records with certainty rather than by fuzzy name matching. The structured remittance elements matter most to businesses. A payment can carry, in machine-readable form, the identifiers of the invoices it settles, their amounts and any adjustments. A company receiving it can match it to its accounts receivable automatically rather than by a clerk reading free text. Reading a payment message It is worth walking through the logical content of a pacs.008, without the XML, to see how the standard thinks about a payment. Take the Ohio manufacturer paying its Vietnamese supplier from the previous chapter. The message begins with a group header: a unique identifier for the message, the date and time it was created, the number of transactions it contains, and the method by which it will be settled, for example through an account the sending bank holds with the receiving bank, or through a clearing system. The header describes the envelope, not the payment. Then comes the credit transfer transaction information, the body of the message, which can repeat if the message carries several payments. It opens with a set of identifiers. There is the instruction identifier, assigned by the bank that sends this particular message and meaningful only between it and the next bank. There is the end-to-end identifier, assigned by the ordering customer, which must pass unchanged all the way to the beneficiary; this is how the supplier can tell the manufacturer which payment arrived. And there is the UETR, the universally unique reference introduced for SWIFT's tracking service, which lets any bank in the chain report the payment's status against a single identifier. Next comes the money. The interbank settlement amount is what moves between this pair of banks, in a stated currency, on a stated settlement date. Separately, the instructed amount records what the customer originally asked to send, so that if fees are deducted along the way, the gap is visible. A charge bearer code states whether the debtor, the creditor or both bear the charges, the ISO 20022 equivalent of the OUR, BEN and SHA options. If intermediaries take charges, each can record what it took in a charges element, so that the beneficiary can see where the shortfall arose rather than simply receiving less. Then come the parties, each in its own place. The debtor is the manufacturer, with its name, structured address and, ideally, its Legal Entity Identifier. The debtor account identifies the account debited. The debtor agent is the manufacturer's bank, identified by its BIC. Then the instructing agent and instructed agent identify the two banks exchanging this particular message, which change at every hop, and up to three intermediary agents can be listed if the payment must pass through others. On the far side, the creditor agent is the supplier's bank, the creditor is the supplier, again with name, structured address and identifiers, and the creditor account is the account to credit. If the manufacturer was paying on behalf of a parent company, or the supplier receiving on behalf of a factoring firm, the ultimate debtor and ultimate creditor would name them. Finally come the reasons. A purpose code from the external code list can say that this is a payment for goods, or salary, or a tax payment, which some countries require for their foreign exchange reporting. A regulatory reporting element can carry codes that the destination country's central bank requires. And the remittance information tells the supplier what the payment is for, either as free text or, in structured form, as a list of the invoices being settled with their numbers, dates and amounts. What this walk-through shows is not complexity for its own sake. Each element answers a question that, in the MT world, a bank or a customer had to answer by guessing, by telephoning, or by reading free text. Who exactly is paying? On whose behalf? Where is the beneficiary located? Why is the money being sent? What did the original sender intend the recipient to receive? A well-formed pacs.008 answers all of them in a form a machine can act on. Standards inside the standard ISO 20022's flexibility is also its weakness. Because the standard defines a vast catalogue of optional elements, two systems can both be "ISO 20022 compliant" and still be unable to exchange messages cleanly, if one requires an element that the other omits or uses an element differently. A standard designed to remove fragmentation can reproduce it at a finer grain. The industry's answer has been usage guidelines: agreed rules that narrow the standard for particular uses. For cross-border payments over SWIFT, the governing rules are known as CBPR+, for Cross-Border Payments and Reporting Plus, developed by a SWIFT-led industry group. For high-value domestic payment systems, a parallel set called HVPS+ was developed so that the large RTGS systems of different countries would implement ISO 20022 consistently. Each market infrastructure then publishes its own detailed specification. The Federal Reserve's Fedwire Funds Service, the euro area's T2, the Bank of England's CHAPS, and the Clearing House's CHIPS each implemented ISO 20022 to their own published rules, aligned with but not identical to the global guidelines. Even in instant payment systems, such as the euro area's SCT Inst scheme, the US RTP network and FedNow, all of which were built on ISO 20022 from the start, the details differ. This matters because the benefits of rich data depend on its surviving the whole journey. A payment that begins in a company's system as a fully structured pain.001, travels as a pacs.008 through two correspondents, and arrives in a domestic system that drops half the elements because its local implementation does not carry them, delivers to the beneficiary only what the narrowest link allowed through. The harmonisation work of the last few years, in which the Committee on Payments and Market Infrastructures published harmonised data requirements for cross-border payments in 2023 and the major infrastructures aligned their implementations, is an attempt to make the narrowest link wide enough. Why it took twenty years If ISO 20022 was published in 2004, why did the cross-border migration only finish in 2025? The answer lies in the economics of networks. No single bank benefits from sending rich data unless the banks downstream can receive and use it. And the costs of migrating, rewriting payment engines, screening systems, reconciliation tools, data warehouses and every interface between them, fall on each bank individually. For a long time, each bank's rational choice was to wait for others to move first. What broke the deadlock was coordination from the top. Large domestic infrastructures began migrating, with fixed dates that their participants could not ignore. SWIFT set a date for the end of coexistence on its cross-border network, and although the start of the migration slipped, first from November 2021 to November 2022 and then to March 2023, the November 2025 end date held. Regulators made clear they expected the richer data to be used. Once enough of the network was committed, the remaining banks had no choice. The migration was also made more tolerable by a compromise that would later prove costly. During the coexistence period from March 2023 to November 2025, a payment could start in ISO 20022 and be converted to MT and back again as it passed through banks that had not yet migrated. Every conversion from a rich message to a poor one could lose data, and a payment converted back to ISO 20022 could not recover what had been lost; it would arrive in the new format with the old format's limitations. SWIFT built services to preserve the original data alongside converted messages, but the coexistence period meant that for over two years the system carried many payments that were ISO 20022 in form and MT in substance. The next chapter asks what the industry has actually gained, now that the language has changed, and why the answer depends less on the format than on what banks put into it. Chapter 4: What Rich Data Buys A new language is only valuable if people say new things in it. The migration to ISO 20022 replaced the format of the world's payment messages, but a format is a container. A pacs.008 that carries a truncated name in its name element, an unparsed address in its address lines and nothing at all in its remittance and purpose elements is no more useful than the MT103 it replaced. The real question about ISO 20022 is not whether banks have migrated, which they largely have, but whether the richer data the standard permits is being captured at the source, carried through the chain intact, and used at the other end. The answer, in late 2026, is: partly, unevenly, and more slowly than the case for migration suggested. That is not a failure of the standard. It is the information problem reasserting itself, because the hard part of good payment data was never the container. It was getting the data into it. The migration timetable ISO 20022 arrived in the world's large payment systems over roughly three years, in a sequence that Table 3 sets out. The order mattered: by the time SWIFT ended coexistence on the cross-border network, the high-value systems in the major currencies had already moved, so a cross-border payment arriving in euros, sterling or dollars could be handed to the domestic settlement system without dropping into an older format. Table 3. Major ISO 20022 migrations in high-value and cross-border payments. System Currency or scope ISO 20022 live Operator T2 (formerly TARGET2) Euro March 2023 Eurosystem SWIFT cross-border, coexistence begins Cross-border March 2023 SWIFT CHAPS Sterling June 2023 Bank of England CHIPS US dollar April 2024 The Clearing House Fedwire Funds Service US dollar July 2025 Federal Reserve Banks SWIFT cross-border, coexistence ends Cross-border November 2025 SWIFT Several of these migrations were deliberately "like for like" at first. CHAPS, for instance, initially allowed participants to send messages that were ISO 20022 in syntax but carried only the data the old format could hold, with enhanced data requirements following in later stages. The Federal Reserve's migration of Fedwire on 14 July 2025 was a single-day cutover, with no extended coexistence, but it too accepted that the full benefit would come as participants enriched their data over time. This staged approach reduced the risk of a failed cutover in systems that move trillions of dollars a day, at the cost of postponing the gains. Sanctions screening and the false positive problem The most immediate operational benefit of structured data is in sanctions and financial crime screening. Every bank that handles a cross-border payment must check the parties against sanctions lists issued by the United States, the European Union, the United Kingdom, the United Nations and others. Screening is done by software that compares names, addresses and other details against the lists, using fuzzy matching to catch spelling variants, transliterations and deliberate evasion. Fuzzy matching on unstructured text produces enormous numbers of false alarms. If a payment's name and address are run together in free lines, the software must check every word against every list entry. A street called "Lenin Street" or a town that shares a name with a sanctioned region, or a common surname that happens to belong to a listed person, can trigger a hit. Each hit must be reviewed by a human analyst, and while it is reviewed, the payment waits. Banks' compliance departments have historically reported that the overwhelming majority of screening alerts are false positives, and the delays they cause are among the main reasons a cross-border payment that reached the beneficiary's bank in minutes is not credited for hours or days. Structured data narrows the problem. When the name, the street, the town and the country are in separate elements, the screening engine can compare names only with names and countries only with countries. A country code removes ambiguity about jurisdiction at once. An LEI or other identifier can confirm or rule out a match with certainty. The expected effect is fewer false alerts, faster clearance of the true ones and better detection of real evasion, since a bad actor can no longer rely on ambiguous free text to slip past a filter. This benefit depends entirely on the data being structured at the source. A bank that receives an unstructured address from a corporate client and simply copies it into the address-line elements of a pacs.008 has gained nothing. That is why SWIFT announced that fully unstructured postal addresses would no longer be accepted in cross-border payment messages after November 2026, while allowing a "hybrid" form in which at least the town and country are structured. The deadline proved hard to meet. In August 2026, SWIFT accepted an industry request to extend the structured-address migration, a decision that illustrates how difficult the source-data problem is: banks had to go back to millions of customer records, captured over decades in free-text fields, and parse them into components, often by hand or with specialised software, and often by contacting the customer. Reconciliation: the corporate prize For businesses, the most valuable promise of ISO 20022 is the end of manual cash application: the work of matching incoming payments to the invoices they pay. A large company may receive tens of thousands of payments a month, many of them paying several invoices at once, net of discounts, deductions or disputes. When a payment arrives with a truncated reference, a person must work out what it pays. ISO 20022 allows the payer to include structured remittance information: the reference of each invoice paid, its amount, and any adjustments, in machine-readable form. A receiving company whose systems read camt.054 notifications or camt.053 statements can match the payment automatically. In principle this closes the loop: an invoice issued electronically is paid with its reference attached, and the payment is applied to the invoice without anyone touching it. The reality is that this loop closes only where both sides, and every bank between them, have done the work. Many corporate systems still send payment files in older formats that their banks translate. Many banks still provide statements to their corporate clients in older formats, so even rich incoming data is flattened before the client sees it. And the cross-border usage guidelines, in order to keep message sizes manageable and to limit the risk of misuse, restrict how much remittance detail can travel inside the payment message itself; very long remittance information is meant to travel separately, linked by reference. The capability exists; the habit of using it does not yet, outside the most sophisticated treasuries. Verification of payee and the fraud dividend A third benefit of better data is in fraud prevention, and here it has been joined by regulation. Many frauds against payers rely on the fact that, in most payment systems, the name of the beneficiary is not checked against the account number. A fraudster who intercepts an invoice and changes the bank details can direct a payment to their own account while the payment still names the legitimate supplier. Payment systems traditionally routed money using only the account identifier. Verification of payee services close this gap by checking, before the payment is sent, whether the name the payer entered matches the name on the receiving account. The United Kingdom introduced Confirmation of Payee in 2020. The European Union's Instant Payments Regulation, adopted in March 2024, made verification of payee mandatory for euro credit transfers from October 2025, with results returned to the payer as a match, a close match or no match before the payment is authorised. Similar services exist or are planned in Brazil, India, Australia and elsewhere. These services work only if the name held by the receiving bank and the name entered by the payer can be compared meaningfully, which is again a data quality question. The international standard-setters have moved in the same direction. In June 2025, the Financial Action Task Force, which sets global anti-money-laundering standards, revised its Recommendation 16 on payment transparency. The revised standard specifies the information that must accompany cross-border payments, including, for payments above one thousand US dollars or euros, the name and address of the payer and the payee, and introduces expectations for checking that the beneficiary's name aligns with the account. Countries are expected to implement the changes by the end of 2030. In effect, the global anti-money-laundering standard now presumes a world of structured, rich payment data of the kind ISO 20022 can carry. Where the data goes wrong There are three ways in which the promise of rich data can fail, and each is visible today. The first is translation. Wherever a system that holds poor data feeds a system that expects rich data, someone writes a mapping. Mappings fill required fields with defaults, put unparsed text into whatever element will accept it, and drop anything that has no obvious home. The resulting message is valid but no richer than its source. The coexistence period, during which payments could pass between MT and ISO 20022 formats, created exactly this risk at scale, and its effects outlast it: many banks' internal systems still store data in the old structures and convert at the boundary. The second is inconsistency. ISO 20022 offers many places to put the same information, and usage guidelines narrow but do not remove the choices. One bank puts a tax identifier in one element, another in a different one, a third in free text. A purpose code list that one country makes mandatory for foreign exchange reporting is unknown to the sender in another. The richer the standard, the more ways there are to use it differently, and the more any one bank's benefit depends on everyone else's discipline. The third is incentive. The costs of capturing good data fall mostly on the sending side, on the payer's bank and its corporate clients who must collect and structure the information. The benefits fall mostly on the receiving side and the intermediaries, who screen faster and reconcile automatically. Banks that originate many payments have limited reason to invest beyond the minimum their market infrastructure requires. That is why deadlines, rejection rules and regulation have done more to improve data quality than the business case on its own. A treasury's view To see how these problems play out in a single organisation, consider a composite that reflects the experience of many mid-sized multinationals: a manufacturer with subsidiaries in a dozen countries, a central treasury, and a single enterprise resource planning system that holds its invoices and supplier records. Before the migration, the treasury sent payment files to its banks in a mixture of formats: a national format for domestic payments in each country, the banks' proprietary formats for international ones, and a few legacy file layouts nobody fully understood. Supplier addresses were held in the system as three free lines, entered by different clerks in different countries over twenty years. When the company's main bank announced that it would require structured addresses for international payments, the treasury discovered that it could not produce them. Its system had a single field for the whole address. Splitting it into street, building, post code and town required a data cleansing project across tens of thousands of supplier records, many in languages and address conventions the central team did not know. On the receiving side, the company's customers paid it from all over the world. Its cash application team spent its days matching incoming payments to open invoices, using whatever fragment of a reference the payment carried. When its banks began to supply ISO 20022 statements, the team initially saw no difference: most of its customers were still sending payments with the same thin references as before, now carried in a richer container. The gains arrived only where the company worked with its largest customers to agree on how invoice references would be passed, and where those customers' banks carried the structured data through. This pattern, an expensive cleansing project on the sending side and slowly accumulating benefits on the receiving side, is typical. It explains why corporate treasurers often describe ISO 20022 as a compliance burden first and an opportunity second. The opportunity is real, but it is collective: each company's benefit depends on the data discipline of its counterparties and their banks. The privacy objection There is a serious objection to the rich-data project that deserves a hearing. More data in every payment means more information about people's financial lives flowing through more institutions in more countries. A structured address, a date of birth, a purpose code and an invoice reference, carried through four banks in three jurisdictions and retained by each for years under record-keeping rules, are a considerable intrusion into privacy, especially for small personal payments that pose little risk. Data protection law and payment transparency law pull in opposite directions here. European data protection rules require that personal data collected be limited to what is necessary for a purpose. Anti-money-laundering rules require that payments carry enough information to identify the parties. The revised FATF standard tries to manage this tension with thresholds, requiring the fullest information only above a certain value, and with the principle that information should be shared only with those who need it. But every additional data element in a cross-border payment is copied into the systems of every institution that handles it, and some of those institutions sit in countries with weak data protection. The objection does not defeat the case for structured data; payments that carry no reliable information about their parties are a gift to criminals and a source of delay for everyone else. It does mean that the value of rich data has to be weighed against the cost of disseminating it, and that a payment architecture in which information is checked once, at the edges, by the institutions that hold the customer relationship, may be preferable to one in which every intermediary receives and stores everything. That question will become more pressing as stablecoins, whose transfers carry almost no personal data on the ledger itself, compete with bank rails whose messages carry more than ever. The honest scorecard What, then, did the industry buy with its twenty-year project? It bought a common grammar, which is no small thing: for the first time, a payment can in principle be described in the same terms from a company's ledger to the central bank's settlement account to the beneficiary's statement, across borders. It bought the possibility of screening payments on structured data, and the regulatory pressure, from SWIFT's address rules to FATF's revised standard, that is slowly converting the possibility into practice. It bought the foundation on which instant payment systems and their cross-border links are being built, since nearly every modern instant system speaks ISO 20022 natively. What it did not buy is speed or lower cost on its own. ISO 20022 does not move money. It does not reduce the liquidity a bank must prefund in its nostro accounts, nor the margin it charges on foreign exchange, nor the number of banks in a correspondent chain. Its effect on cost is indirect: fewer payments held for investigation, fewer returns and repairs, fewer clerks matching receipts. Those effects are real but accumulate slowly, at the rate that data quality improves. That is the first lesson of the modernization story. The information problem is a necessary condition for faster, cheaper cross-border payment, not a sufficient one. Clean data at the source makes everything downstream easier. But once the information can travel well, the binding constraint becomes the movement of value itself, and for that the world has been building a different kind of infrastructure: payment systems that settle in seconds, at any hour, every day of the year. Chapter 5: Settlement in Seconds For most of the history of electronic banking, a payment between accounts at different banks took at least a day. Payroll, bill payments and supplier payments were collected into files, exchanged between banks in batches at fixed times, netted, and settled overnight or the next morning. The money was gone from the payer's account before it arrived in the payee's, and for the hours in between it sat in the machinery of the clearing system. Weekends and public holidays stopped everything. A person paid on Friday evening might wait until Tuesday. The real-time payment systems built over the last two decades abolish that delay within a single currency. A payment is initiated, cleared, settled and credited to the payee's account in seconds, at any hour of any day. The payer and payee both know, almost immediately, that the money has moved. This changes what payments are used for. It allows a gig worker to be paid at the end of a shift, an insurer to settle a claim while the customer is still on the phone, a landlord to see rent arrive before handing over keys, and a market trader with no card terminal to accept payment from a customer's phone. Where instant payment systems have been designed well, they have displaced cash and cards on a scale no one predicted. But "real-time payments" covers systems that differ a great deal under the surface. They differ in how they settle, who runs them, who can join, what they cost and what services they carry beyond the basic transfer. Those differences explain why some reached most of their country's adults within a few years while others, technically excellent, grew slowly. How instant settlement works An instant payment system must solve a problem that the first chapter described: the trade-off between the liquidity needed for gross settlement and the credit risk of net settlement. If a payment is to be final and credited to the payee within seconds, the payee's bank must be confident it will be paid. If the systems settled each payment individually in central bank money, every bank would need its central bank account to be open and funded every second of every day, and central bank settlement systems historically closed in the evening and at weekends. Designers have used three main approaches. The first is settlement in central bank money in real time, with the central bank's settlement system running around the clock. The euro area's TARGET Instant Payment Settlement service, TIPS, launched by the Eurosystem in 2018, works this way: banks move liquidity from their main central bank accounts into dedicated TIPS accounts, and each instant payment is settled individually, in central bank money, at any hour. The Federal Reserve's FedNow Service, launched in July 2023, settles each payment in the participants' accounts at the Federal Reserve. Brazil's Pix settles in real time on accounts at the Banco Central do Brasil. The second is real-time clearing against a prefunded pool, with the pool held in central bank money. The Clearing House's RTP network in the United States, launched in November 2017, holds a joint account at the Federal Reserve Bank of New York. Each participating bank's share of that account is recorded on the RTP system's own books. When a payment is made, the RTP system moves the shares instantly; the total in the joint account does not change. Banks top up their positions during Fedwire operating hours. Because every payment is backed by money already in the pool, there is no credit risk, and the system can run at all hours even when Fedwire is closed. The third is real-time clearing with deferred net settlement, backed by collateral or prefunding. The United Kingdom's Faster Payments, launched in 2008, originally settled net positions several times a day across the Bank of England's books; since 2015 participants have been required to prefund their maximum net exposure, so that the system can absorb the failure of a participant. India's Unified Payments Interface settles banks' net positions in multiple cycles each day through the Reserve Bank of India, while customers see their money instantly. Table 4 compares the main systems discussed in this chapter. The differences in settlement model are invisible to the customer, who sees the same thing in each case: money arriving in seconds. They matter to the banks, because each model asks them to hold different amounts of liquidity in different places, and to the central bank, because each model places risk differently. Table 4. Selected instant payment systems compared. System Jurisdiction Launched Operator Settlement model Faster Payments United Kingdom 2008 Pay.UK Deferred net, prefunded at central bank UPI India 2016 NPCI Deferred net, multiple daily cycles RTP network United States 2017 The Clearing House Prefunded joint account at the Fed TIPS Euro area 2018 Eurosystem Real-time gross, central bank money Pix Brazil 2020 Banco Central do Brasil Real-time gross, central bank money FedNow United States 2023 Federal Reserve Banks Real-time gross, central bank money The overlay: what the customer actually touches A settlement system on its own is not something consumers use. What determines adoption is the layer above it: how a payment is addressed, initiated and requested. The most important innovation is the alias or proxy. In older systems, to pay someone you needed their bank's routing code and their account number, which few people know by heart and which create opportunities for error. Modern instant systems let a payment be addressed to a phone number, an email address, a tax identification number or a chosen handle. Brazil's Pix calls these "keys"; India's UPI uses a virtual payment address, often in the form name@bank. Behind the scenes, a central directory translates the alias into an account. The alias makes paying someone as easy as messaging them, and it is the single feature most responsible for mass adoption. The second innovation is the QR code. A merchant can display a static code at the counter, or generate a dynamic one for each sale encoding the amount and a reference. The customer scans it with a banking or payments app and confirms. This lets the smallest merchant accept electronic payments with no terminal, no card acquiring contract and, in many countries, no fee. India and Brazil built their merchant adoption largely on QR codes. The third is request for payment. Instead of the payer typing in details, the payee sends a request, carrying the amount and an invoice reference, which the payer approves in their app. Requests turn a push payment system into something that can compete with direct debits and cards for bills and online purchases. They are also where ISO 20022's richer data pays off domestically: a request can carry structured remittance information that the resulting payment then returns to the payee, so the payment reconciles itself. Why Pix and UPI took off The two most successful instant payment systems in the world were both built in emerging economies, by or under the direction of public authorities, and both reached mass use within a few years. Pix was designed by the Banco Central do Brasil and launched in November 2020. The central bank did not merely operate the settlement system; it wrote the rules for the whole consumer experience, including the key directory, the QR code standard and the user interface requirements. Participation was mandatory for financial institutions above a size threshold, so customers of every large bank could use it from day one. Payments between individuals were free. Within its first year Pix had overtaken Brazil's older interbank transfer methods, the TED and DOC, in the number of transfers. In 2025, according to central bank data reported in Brazil, it processed close to 80 billion transactions worth over 35 trillion reais, up from 63.5 billion transactions the year before. UPI was launched in 2016 by the National Payments Corporation of India, an organisation owned by banks but created at the urging of the Reserve Bank of India. Its distinctive design choice was to separate the payment app from the bank: any authorised app, whether from a bank or a technology company, could initiate payments from any bank account linked to it. This opened the field to large consumer technology companies, whose apps rapidly became the dominant way Indians paid. The government's policy, from 2020, of prohibiting merchant fees on most UPI payments removed the main obstacle to merchant acceptance. The results are without precedent: UPI recorded about 218.6 billion transactions in the fiscal year ending March 2026, and reached a monthly record of 24.51 billion in August 2026. Neither success can be explained by technology alone. Both systems succeeded because a public authority with the power to coordinate made the decisions that no individual bank would make on its own: universal participation, a single directory, a single QR standard, and pricing that made the system free or nearly free at the point of use. The zero-price policy has costs, and in India there has been a long debate about who pays for the infrastructure if merchants cannot be charged. But in both countries, a coordinating authority solved the network problem that the ISO 20022 migration also had to overcome: no participant benefits until everyone joins. The American exception The United States has two instant payment systems and, by comparison with Brazil or India, modest adoption of both. The RTP network has grown steadily; it set a single-day record of 2.27 million transactions worth 8.62 billion dollars on 1 May 2026, according to figures reported in the trade press. FedNow reported more than fifteen hundred participating institutions by late 2025. Both raised their transaction limits to ten million dollars in 2025, RTP in February and FedNow in November, explicitly to attract business payments that had previously gone by wire. But both remain small next to the card networks and the automated clearing house, and many consumers have never knowingly used either. The reasons are structural. The United States has thousands of banks and credit unions, and there was no mandate requiring them to join either system; each institution decided for itself, often waiting until its core banking provider offered a ready-made connection. Consumers already had person-to-person apps that felt instant, though most settled between banks through slower systems behind the scenes. Card payments, with their rewards and chargeback protections, were entrenched at the point of sale. And the existence of two systems, one private and one public, meant that a bank joining one could not necessarily reach every account on the other. The American experience is a useful corrective to the idea that instant payment technology drives adoption. The technology was available in the United States from 2017. What was missing was the coordinating decision, mandated participation, a common directory, a consumer-facing standard, that made Pix and UPI unavoidable. Europe offers a third model: regulation rather than a public operator. The SEPA Instant Credit Transfer scheme existed from 2017 but grew slowly, partly because many banks charged extra for instant transfers. The Instant Payments Regulation of 2024 changed that, requiring payment providers in the euro area that offer ordinary credit transfers to be able to receive instant transfers by January 2025 and send them by October 2025, at a price no higher than an ordinary transfer, combined with the verification-of-payee requirement described in the previous chapter. Non-euro member states have until 2027. The regulation turns instant payment from a premium product into the default. Who pays for the rails An instant payment system is expensive to build and run. It must operate continuously, with near-perfect availability, screening and routing millions of payments a day. Someone has to pay for it, and the answer shapes the system's adoption as much as any technical choice. There are three basic models. In the first, the operator charges banks a small fee per transaction, and banks decide whether to pass it on. The US systems follow this model: both the RTP network and FedNow charge participating institutions a few cents per payment, with FedNow's published fee for a credit transfer set at four and a half cents. At that level, the operator's fee is negligible; what matters is what the banks charge their customers, and in the United States many banks have charged consumers for instant transfers, or offered them only as a premium feature, which depresses use. In the second model, the public authority runs the system at or near cost and mandates low or zero prices to end users. Brazil's central bank charges institutions a very small fee per transaction and prohibits charging individuals for most Pix transfers. India went further, prohibiting merchant discount rates on most UPI payments, which means that the banks and apps that carry the payments earn nothing directly from them. Indian payment companies and banks have argued for years that this is unsustainable, and the government has provided subsidies to the industry to offset part of the cost. The zero-price model produced the fastest adoption in the world, but it also means that the infrastructure is financed by cross-subsidy and public money rather than by its users, and the question of who pays for continued investment, especially in fraud prevention, remains open. In the third model, regulation sets a price ceiling relative to an existing product. The European Union's Instant Payments Regulation does not set prices but requires that instant transfers cost no more than ordinary ones. Since most European banks charge little or nothing for ordinary domestic transfers within online banking, the effect is to make instant transfers similarly cheap, while leaving banks to recover their costs through account fees. The choice of model matters for cross-border payments because pricing determines who has an incentive to connect domestic systems to the rest of the world. Where banks earn nothing from domestic instant payments, they may see a cross-border link mainly as a cost. Where the system operator is a public authority with a mandate to improve cross-border payments, as several Asian central banks are, the link can be justified as public infrastructure. Business payments and the high-value frontier Instant systems were first used mainly for small payments between individuals, but their most important economic effect may come from business payments. Companies pay and are paid in large amounts, often with long settlement delays, and the working capital tied up in money in transit is substantial. The decisions by both US operators to raise their transaction limits to ten million dollars in 2025 were aimed directly at this market. At that level, an instant payment can replace a wire for most commercial transactions, including property closings, supplier payments and treasury transfers between a company's own accounts, with the advantage that it works outside wire operating hours and carries a request-for-payment option that wires lack. Instant payments also change the economics of the supply chain. A small supplier that is paid instantly on delivery, rather than by a batch payment days later, needs less short-term borrowing. Several countries have seen businesses use instant payments for payroll, so that workers are paid at the end of a shift. High-value instant payments do raise the stakes for fraud and liquidity. A ten-million-dollar payment sent to the wrong account in seconds, at a weekend, is a much more serious event than a fifty-dollar transfer, and banks offering high limits have applied their own lower limits and additional controls for most customers. But the direction is clear. In the long run, the distinction between "instant retail payments" and "high-value wholesale payments" may dissolve into a single expectation that money moves when it is sent. What instant settlement does not solve Instant domestic rails solve the value problem within a currency. They make settlement fast, continuous and cheap, and they remove the credit risk of waiting for a batch. In the best designs, they also carry rich ISO 20022 data, so the information problem is solved along with it. They leave two things untouched. The first is the boundary of the currency. An instant system settles in one currency, among institutions that share a central bank. A payment to another country still needs foreign exchange and still needs value positioned on the other side. Linking instant systems across borders, the subject of the seventh chapter, is possible but runs straight into the liquidity and compliance problems of the correspondent chain. The second is trust. A system in which payments are final in seconds, at any hour, and cannot be recalled, is a system in which a mistake or a deception becomes permanent before anyone can intervene. That is the price of instant, and the next chapter examines who pays it. Chapter 6: The Price of Instant Every property that makes an instant payment useful also makes it useful to a thief. It is fast, so the money is gone before anyone can question it. It is final, so there is no chargeback to reverse it. It runs at all hours, so it works when fraud teams are thin. It is addressed by phone number or handle, so a fraudster can receive money without a payer ever seeing a bank's name. And it is a push payment, initiated by the payer, so when the payer has been deceived into sending it, the payer's bank has, in a narrow sense, done exactly what its customer asked. This is not an argument against instant payments. It is a description of the trust problem that instant settlement creates. Older, slower systems had a hidden safety feature: delay. A batch payment that would not settle until the next morning could be stopped by a call to the bank. A wire held for compliance review could be cancelled. When settlement became instant and irrevocable, that margin disappeared, and with it a whole class of informal protections. The payment industry has spent the last decade trying to rebuild those protections in a different form. The anatomy of an authorised push payment scam The characteristic crime of the instant payment era is the authorised push payment scam, usually called APP fraud. Its defining feature is that the victim authorises the payment. No password is stolen and no account is taken over. Instead, the victim is persuaded to send money, through deception, to an account the fraudster controls. The forms are many. In invoice redirection, a fraudster who has compromised or impersonated a supplier's email sends a customer a message saying the supplier's bank details have changed; the customer pays the next invoice to the new account. In impersonation scams, the fraudster poses as the victim's bank, the police or a tax authority and persuades the victim that their money is at risk and must be moved to a "safe account". In purchase scams, the victim pays for goods advertised online that never arrive. In romance and investment scams, the fraudster builds a relationship over weeks or months before asking for money. What unites them is that, from the payment system's point of view, everything is in order: a genuine customer, properly authenticated, instructs a payment to a valid account. The receiving account is the weak point. To receive the money, the fraudster needs an account at a regulated institution. Usually it is a money mule account: an account opened with stolen or synthetic identity documents, or the genuine account of a person recruited, often through social media, to let criminals use it in return for a cut. Within minutes of receiving the funds, the mule account pays them onward to other accounts, to cryptocurrency exchanges, or to cash withdrawals, so that by the time the victim realises what has happened, the money has scattered through several institutions. The British experiment in liability The United Kingdom was the first large economy to confront APP fraud at scale, partly because Faster Payments made the UK one of the first countries with ubiquitous instant payments, and partly because its regulators chose to measure the problem publicly. Industry figures published by UK Finance showed APP losses running at several hundred million pounds a year through the early 2020s. The question was who should bear the loss. Under traditional banking law, a bank that executes a payment its customer authorised has no liability for it, even if the customer was deceived. For several years, UK banks operated a voluntary code under which they reimbursed some victims, but outcomes varied widely between banks and many victims received nothing. In 2023, the Payment Systems Regulator decided to make reimbursement mandatory. The resulting rules took effect on 7 October 2024. Payment providers must reimburse consumers, micro-enterprises and charities who fall victim to APP scams over Faster Payments, up to a maximum of 85,000 pounds per claim, unless the victim acted fraudulently or with gross negligence. Crucially, the cost is split equally between the sending provider and the receiving provider. That split was the most important design choice. By making the bank that holds the mule account pay half of every loss, the rules gave receiving banks a direct financial reason to detect and close mule accounts, which they had previously had little incentive to do. The early evidence suggests the rules changed behaviour. According to the regulator's data and its one-year evaluation, the share of eligible claims reimbursed rose sharply, to 88 percent in the first year, and in-scope scam losses over Faster Payments fell by roughly a fifth over the first twelve months. The evaluation also found large differences in reimbursement rates between providers, and evidence that fraud was shifting to channels outside the rules: international payments, which the UK rules do not cover, and transfers to cryptocurrency exchanges. Fraud, like water, finds the gap. Other designs: Brazil, Europe, the United States Brazil took a different approach, centred on recovering money rather than reimbursing it. When Pix grew quickly, so did fraud and a more violent crime: so-called lightning kidnappings, in which criminals forced victims to make Pix transfers. The central bank responded with transaction limits, including lower default limits at night, and in 2021 launched the Mecanismo Especial de Devolução, or MED, a special return mechanism under which a victim's bank can ask the receiving bank to block suspected fraudulent funds and return them. The original mechanism could only block funds in the first receiving account, which criminals defeated by moving the money on immediately. Its successor, MED 2.0, which the central bank made mandatory for all Pix providers from February 2026, is designed to trace funds along the chain of onward transfers and block them wherever they land, and lets users contest a payment directly in their banking app. Europe has relied mainly on prevention at the point of payment. The verification-of-payee requirement in the Instant Payments Regulation, in force for euro payments from October 2025, attacks invoice redirection directly: if the name a payer enters does not match the account, the payer is warned before sending. The EU's broader revision of its payment services rules has also been negotiated with provisions extending liability in some impersonation cases. The United States has no general reimbursement regime for push payment fraud; consumer protections under its electronic fund transfer law cover unauthorised transactions, not authorised ones obtained by deception, though there has been continuing political pressure for change. The operators have added tools instead: the Federal Reserve, for instance, announced in late 2025 a pilot service letting FedNow participants verify accounts before a payment and a standard classification model for reporting scams. Behind these differences lies a common realisation. The defence against APP fraud cannot rest with the payer alone, who is the target of the deception, or with the sending bank alone, which sees only its own customer. It needs information from the receiving side: whether the name matches the account, whether the account is new, whether it is receiving many payments from unrelated people and sending them on at once. Fraud prevention in instant payments has become, like compliance in cross-border payments, a problem of sharing information between institutions fast enough to act on it. One fraud, four regimes It is useful to follow a single, typical fraud through the different regimes, because the outcome for the victim depends almost entirely on where it happens. A small accounting firm pays its office landlord eight thousand in local currency each month by bank transfer. One morning an email arrives, apparently from the landlord's managing agent, saying the agent has changed banks and giving new account details for future rent. The email is in fact from a fraudster who has gained access to the agent's email account and has been reading its correspondence for weeks; the message refers correctly to the building, the lease and the last payment. The firm's bookkeeper updates the payee details and sends the next month's rent by instant transfer. The money arrives in a mule account, opened two weeks earlier with false identity documents, and within twenty minutes it has been sent on to three other accounts and from there to a cryptocurrency exchange. In the euro area since October 2025, the bookkeeper would have entered the new account number and the landlord's name, and the verification-of-payee check would have returned "no match", since the account is in a different name. The payment might still have been sent, since the payer can proceed after a warning, but in most cases the warning stops the fraud before it happens. In the United Kingdom, the same Confirmation of Payee check would have produced the same warning. If the firm proceeded and lost the money, it would, as a micro-enterprise, be eligible under the reimbursement rules introduced in October 2024, with the cost split between its own bank and the bank that hosted the mule account, unless the bank could show gross negligence. In Brazil, the firm's bank would use the return mechanism to ask the receiving institution to block the funds, and under MED 2.0 the system would follow the money through the onward transfers in an attempt to recover what remained. In the United States, the firm would typically have no legal right to reimbursement, since it authorised the payment; whether it recovered anything would depend on its bank's goodwill and on whether the receiving bank froze the funds in time. The same fraud, the same instant payment, four very different outcomes. The difference lies entirely in the trust machinery around the rail, not in the rail itself. Confirmation as a design principle Several principles have emerged from this decade of experience, and they bear directly on any system, including stablecoin networks, that offers fast, final payment. The first is that names must be checked against accounts. Routing money on an account number alone was acceptable when payments were slow and mostly between known parties. It is not acceptable when a payment can be sent in seconds to anyone. Verification of payee services, supported by the structured name data that ISO 20022 carries, are becoming a standard feature of instant systems. The second is that friction can be targeted. A system can be instant for the great majority of payments and slower for the few that show signs of risk: a first payment to a new payee, an unusually large amount, a transfer at three in the morning to an account opened yesterday. Several jurisdictions now allow, or require, banks to delay a suspicious instant payment for a short period to investigate. The UK, for example, changed its rules in 2024 to allow banks to hold an outbound payment for up to four business days where they have reasonable grounds to suspect fraud. This reintroduces the old safety margin selectively instead of for everyone. The third is that liability drives investment. The UK reforms changed bank behaviour not by mandating any particular technology but by assigning the loss to the institutions best placed to prevent it. Banks that were paying half of every scam into a mule account they hosted began to look much harder at who was opening accounts with them. The fourth, learned the hard way, is that the controls must follow the money across rails. The UK's evaluation found fraud moving to international payments and to transfers into cryptocurrency exchanges, which fall outside the domestic reimbursement rules. Brazil's original return mechanism was defeated by onward transfers until MED 2.0 extended tracing along the chain. A fraudster's goal is to move money from a protected rail to an unprotected one as quickly as possible, and the boundary between a domestic instant system and a cross-border or blockchain-based one is exactly such a gap. Every protection that ends at a border invites fraud to cross it. The liquidity cost of never closing There is a second, less visible price of instant payments, borne by banks rather than customers: the cost of being open all the time. A bank in a deferred net settlement system needs liquidity only at settlement times, and it can borrow from the central bank or the money market during business hours to cover its position. A bank in an instant system needs liquidity at every moment, including weekends and holidays when the central bank's main settlement system and the money markets are closed. If a large corporate customer receives a payroll refund on Saturday night, or if a wave of customers move money out of the bank on a Sunday, the bank must already have enough set aside in its instant-payment account to meet every outgoing payment until the main system reopens. This is why the prefunding models described in the previous chapter matter. On the RTP network, a bank's position in the joint account at the Federal Reserve Bank of New York can only be topped up while Fedwire is open; over a long weekend, it must hold enough to last. Central banks have responded by extending the operating hours of their main systems, so that liquidity can be moved into instant-payment accounts for more of the week; the Federal Reserve has consulted on extending Fedwire's hours, and extended RTGS hours were one of the G20 cross-border payment priorities. Banks, for their part, have had to build new forecasting and treasury routines: models of weekend outflows by customer segment, automated top-ups timed to the last minutes before the main system closes on Friday, and internal limits that throttle large outgoing instant payments if the bank's position falls below a threshold. Smaller institutions often rely on a correspondent or a service provider to manage this for them, which reintroduces a dependency on a larger bank at the very moment the system was meant to give everyone equal access. But the basic tension remains: instant settlement converts what used to be a daily liquidity problem into a continuous one, and the cost of holding idle liquidity at all times falls on the banks. There is a stability dimension too. Instant payments make it possible for depositors to move money out of a bank faster than at any time in history. When Silicon Valley Bank failed in March 2023, depositors tried to withdraw about 42 billion dollars in a single day, mostly through online banking, according to regulators' accounts. Instant, always-on payment rails do not cause bank runs, but they remove the friction that once slowed them. Supervisors now treat that speed as a factor in how much liquidity banks should hold. The lesson for cross-border rails The experience of domestic instant systems carries a clear message for anyone designing a cross-border rail, whether by linking instant systems or by using stablecoins. Speed and finality are valuable, but they transfer risk from the system to its users unless something else carries the trust function that delay used to provide. In domestic systems, that something has become a combination of name checking, targeted friction, shared fraud intelligence and liability rules that make the receiving institution answerable for the accounts it hosts. Across borders, all of these are harder. A verification-of-payee check must query a bank in another country, under a different data protection regime. A fraud investigation must reach an institution in another jurisdiction. A liability rule must be agreed between regulators who may not share priorities. And the money, once it has crossed, is subject to another country's law. The value problem and the trust problem are bound together: the faster value moves, the more the system needs a way of establishing trust that is equally fast. The next chapter examines what happens when instant payment systems are connected to one another across borders, and why the hardest work is not in the connection itself. Chapter 7: Linking the Rails Once instant payment systems existed in dozens of countries, an obvious idea followed. If a person in Singapore can pay anyone in Singapore in seconds using a phone number, and a person in Thailand can do the same in Thailand, why not connect the two systems, so that a Singaporean can pay a Thai phone number directly? The domestic leg at each end is already instant. All that seems to be needed is a bridge in the middle. In April 2021, that bridge opened. Singapore's PayNow and Thailand's PromptPay were linked, allowing customers of participating banks to send money between the two countries using a mobile number, with funds arriving within minutes. It was the first link of its kind between two national instant payment systems. Others followed: India's UPI was linked to Singapore's PayNow in February 2023, and a web of bilateral connections spread across Southeast Asia, with QR-code payment links allowing tourists to pay merchants in a neighbouring country with their home banking app. These links are genuine achievements, and for the people who use them they are dramatically better than a wire transfer. But they also make plain what an instant payment link does and does not solve. Connecting the rails turns out to be the easy part. The hard part is everything the correspondent chain was doing, in its slow and expensive way, that someone still has to do. Three ways to connect There are three basic models for linking instant payment systems across borders, and each distributes the hard problems differently. The first is the bilateral link. Two national systems agree on a connection, technical standards, a message format, scheme rules and a settlement arrangement. Payments flow from one system through a gateway into the other. PayNow–PromptPay and UPI–PayNow are of this kind. Bilateral links are relatively quick to build when two countries have strong motivation, but they scale badly. Connecting every system to every other requires a number of links that grows with the square of the number of systems; sixty systems would need nearly eighteen hundred links, each with its own negotiation, rules and technology. The second is the hub-and-spoke model. Each national system connects once to a central hub, which handles translation, routing and the coordination of currency conversion; through that single connection it can reach every other system on the hub. This is the logic of Project Nexus, an initiative begun by the Bank for International Settlements' Innovation Hub in 2021. Nexus defines a common set of rules, a standard ISO 20022-based message flow and a common interface, so that a national system that connects to Nexus can in principle reach all the others. In its operational phase, the participating central banks, those of India, Malaysia, the Philippines, Singapore and Thailand, with Indonesia as a special observer, agreed to establish a scheme organisation owned by the participating central banks and instant payment operators to run it, with live operation targeted for 2026. The BIS moved to an advisory role. The third is the common platform. Instead of linking separate national systems, several countries use a single shared system that can settle in multiple currencies. Buna, launched in 2020 by an organisation established by the Arab Monetary Fund, is a multi-currency payment platform used by banks across the Arab region. The euro area's TIPS was designed to be able to settle currencies other than the euro, and Sweden's central bank began settling instant krona payments on it in 2024. A common platform can solve some problems elegantly, because all participants share one rulebook and one ledger, but it requires a degree of institutional trust between countries that is rare, and it puts a large piece of each country's payment infrastructure under shared governance. The currency problem does not go away Whichever model is used, a cross-border instant payment between two currencies requires someone to convert one currency into the other, and that someone must hold balances in both. When a Singaporean sends Singapore dollars through PayNow to a Thai mobile number, the Thai recipient must receive baht through PromptPay. Somewhere between the two, a bank or a specialist provider must take in Singapore dollars on one side and pay out baht on the other. In the correspondent chain, that work is done by the converting bank using its nostro balances. In an instant payment link, it is done by participating banks or designated foreign exchange providers, who must hold liquidity in both systems. Nexus formalises this by defining specific roles: foreign exchange providers, which quote rates and convert currency, and settlement access providers, which give those foreign exchange providers access to the domestic instant payment systems in which they need to settle. A foreign exchange provider that wants to offer conversion from rupees to pesos must hold a rupee position in India's system and a peso position in the Philippines, both funded in advance, so that each half of the payment can be settled instantly on its own side. This is the same prefunding problem that the correspondent chain faces, relocated. The link makes the movement of information instant and the settlement on each side instant, but the liquidity that makes the conversion possible must still be positioned in advance, in both currencies. And because instant systems operate around the clock, that liquidity must cover weekends and holidays when the foreign exchange market is thin and the foreign exchange provider cannot easily rebalance. The costs of holding it, and the risk of rates moving over a weekend, are priced into the exchange rate that the sender sees. The economic effect of a well-designed link is therefore subtler than "removing intermediaries". What it removes are the intermediaries between the sending bank and the foreign exchange provider and between the foreign exchange provider and the receiving bank, each of which in the correspondent world might take a fee and add delay. What it keeps, and must keep, is the foreign exchange provider itself. Competition among such providers, which Nexus encourages by having several quote rates for each corridor, can drive margins down. But the cost of conversion, and of the liquidity it requires, is a floor below which cross-border payment cannot go. A linked payment, step by step Following a single payment through a hub such as Nexus makes the division of labour concrete. Suppose a shop owner in Kuala Lumpur wants to pay a supplier in Manila the equivalent of five hundred dollars, using her banking app. She opens the app, chooses to pay abroad, selects the Philippines and enters the supplier's mobile number, which is registered as an alias in the Philippine instant payment system. Her bank sends a query through the hub to the Philippine side, which resolves the alias to an account and returns the account holder's name. The app shows her the name so she can confirm she is paying the right person. Next, the app asks for an exchange rate. Her bank, through the hub, receives quotes from foreign exchange providers that are active in the ringgit-to-peso corridor. Each quote is a firm rate, valid for a short period. The bank selects one, or lets her choose, and shows her the exact amount in ringgit that will be debited, any fee, and the exact amount in pesos the supplier will receive. Only then does she confirm. When she confirms, her bank runs its sanctions and fraud checks and sends an ISO 20022 payment instruction. On the Malaysian side, the ringgit move instantly through the domestic system from her bank to the foreign exchange provider's account, or the account of its settlement access provider. On the Philippine side, the foreign exchange provider pays the pesos out of its prefunded position through the Philippine instant system to the supplier's bank, which credits the supplier. Both legs are linked so that if the second fails, the first is reversed or not executed. The hub coordinates the timing and passes confirmations back. The target is for the whole process to complete within sixty seconds. At no point does any bank need a correspondent relationship with any other bank across the border. The shop owner's bank deals with the hub and with its own domestic system. The supplier's bank deals with its own domestic system. The only institution that spans the border is the foreign exchange provider, which holds balances in both currencies, and that is where the cost and the risk concentrate. The money that never crosses The specialist remittance companies that grew up in the last fifteen years discovered a version of the same insight before the instant links were built. A firm that receives many payments from customers in Britain who want to send money to Europe, and many from customers in Europe who want to send money to Britain, need not move each payment across the border at all. It can hold accounts in both places, pay each British customer's European recipient out of its euro account and each European customer's British recipient out of its sterling account, and only occasionally rebalance the two positions with a single large foreign exchange trade. Firms such as Wise built their businesses on this model of matching flows through networks of local accounts, and some went further by obtaining direct access to domestic payment systems, so that they no longer needed a bank to send domestic payments on their behalf. The model shows that much of the cost of cross-border payment is not intrinsic to crossing the border. Where flows are large and roughly balanced, the money mostly does not cross; it is netted. Where they are small or one-directional, as in many remittance corridors where money flows overwhelmingly from rich to poor countries, it cannot be netted, and someone must buy the receiving currency in the market and hold it in advance. Instant links, specialist remittance firms and stablecoins are all, in different ways, competing to do that job more efficiently. None of them can make it unnecessary. Compliance at the speed of a domestic payment The second problem that linking cannot avoid is compliance. A domestic instant payment is screened by the sending bank and, in principle, the receiving bank, under one country's rules. A cross-border payment is subject to two sets of rules, and often to the requirements of a third or fourth country whose sanctions lists the banks must also apply. Every institution in the payment, including the foreign exchange provider, has to be satisfied that it may lawfully process it. In the correspondent world, compliance checks are a major cause of delay: a payment that triggers a possible sanctions match waits in a queue until an analyst clears it. An instant payment link cannot afford that. Nexus addresses this partly by moving checks earlier. Before the payer confirms a payment, the system retrieves and displays the recipient's name, confirming the account through the destination system's own directory, and the sending side can screen the recipient before the payment is sent. Payments that cannot be cleared in seconds are rejected rather than held. The aim is to ensure that a payment that is sent will clear, with the cost that a few legitimate payments that would have been cleared after review in the correspondent system are refused outright. Compliance in seconds depends on data quality, and here the threads of this book come together. Instant cross-border screening is only workable if the payment carries clean, structured information: a full name, a structured address, a country code, ideally an identifier. ISO 20022, the harmonised data requirements published by the Committee on Payments and Market Infrastructures, and the revised FATF standard on payment transparency are all prerequisites for instant cross-border payments that do not simply shift risk onto the banks. The information problem, solved, is what makes the value problem solvable at speed. Rules, liability and the cost of agreement The third problem is governance. When a domestic instant payment goes wrong, a fraud, a mistake, a dispute, the domestic scheme's rules say who must do what. When a cross-border instant payment goes wrong, whose rules apply? If a Singaporean is tricked into sending money to a fraudster's account in Thailand, does the British-style principle of reimbursement apply, or the Thai rules, or none? Can the sending bank request the return of funds, and is the receiving bank obliged to cooperate within a fixed time? Each link must answer these questions, and the answers require agreement between central banks, regulators and payment system operators in each country. This is the slowest part of building any link. It explains why links are more common among countries with existing close economic and political ties, and why the multilateral approach of Nexus took years of design work before it reached operational stage. A single rulebook that every connected country accepts can, once agreed, be extended to new members at low cost. But reaching that agreement involves negotiating questions of sovereignty, data protection and consumer protection that technology cannot settle. There is also the matter of sanctions politics. Some cross-border payment initiatives have been shaped as much by geopolitics as by efficiency. Project mBridge, a multi-central-bank platform for wholesale cross-border settlement using central bank digital currencies developed with the participation of central banks in China, Hong Kong, Thailand, the United Arab Emirates and, later, Saudi Arabia, attracted attention as a possible route for payments outside the dollar system. The BIS withdrew from the project in October 2024, saying it had reached a mature stage and could be run by the participating central banks. The episode is a reminder that a payment rail is never merely technical; whoever governs it decides who may use it. What links have achieved Measured against the problems this book has set out, instant payment links make a clear contribution. They solve the information problem for the corridors they serve, since they are built on ISO 20022 from the start, with structured names and confirmation of the recipient before the payment is sent. They solve the speed problem, delivering funds in seconds or minutes rather than days. They reduce the number of intermediaries, and with it the stacking of fees. And because they offer a public, transparent rate and fee before the payer confirms, they meet the transparency goal that the correspondent chain struggles with. What they do not solve is the irreducible cost of currency conversion and the liquidity it requires, which still has to be provided by someone with balances in both currencies. And they bring a new constraint: reach. A link connects only the systems that have joined it, and only for the institutions in each country that participate. A migrant worker in a Gulf state sending money to a village in East Africa will not be served by a link between Singapore and Thailand. For the many corridors where no link exists, or where one end has no functioning instant payment system at all, the correspondent chain remains the only regulated route. This is also why the official targets are slipping. The G20's goals for 2027 assumed that faster, cheaper infrastructure, ISO 20022 harmonisation, longer operating hours for settlement systems and links between instant payment systems, would steadily pull down average costs. The infrastructure has improved, and the speed targets are within reach on the best-served routes; SWIFT reports that the great majority of payments now reach the beneficiary's bank within an hour. But averages are dominated by the expensive corridors, and the expensive corridors are expensive for reasons that links do not touch: thin currency markets, weak domestic payment systems at the receiving end, and the cost of compliance where risk is perceived to be high. The Financial Stability Board's own assessments have acknowledged that the cost targets are unlikely to be met by 2027, and its chair has framed the remaining agenda around reducing compliance costs and improving the domestic leg, rather than building more connections. That gap, between where instant links reach and where people actually need to send money, is precisely the gap into which stablecoins have moved. They promise to move value across any border, to anyone with a phone, at any hour, without either side's payment systems having to agree anything at all. The next chapter examines how that works, and where the costs that links could not remove end up. Chapter 8: Stablecoins as a Cross-Border Rail A stablecoin is a token on a blockchain that is designed to hold a fixed value against a conventional currency, almost always the US dollar. The dominant kind is simple in principle. A company, the issuer, accepts dollars from customers and issues an equal number of tokens. It holds the dollars as reserves, in bank deposits, short-term US Treasury bills and similar assets, and promises to redeem each token for a dollar on request. The tokens themselves live on one or more public blockchains, such as Ethereum, Tron or Solana, where anyone with a compatible wallet can hold them and send them to anyone else, at any hour, usually for a small network fee, with the transfer recorded as final within seconds or minutes. That combination, a dollar claim that moves like an email, is why stablecoins have become the most serious challenger to the correspondent chain since the telex. Their total supply passed 300 billion dollars in 2026, according to industry trackers, concentrated in two issuers: Tether's USDT, the largest by a wide margin, and Circle's USDC. Most of their use has historically been inside the cryptocurrency markets, as a dollar-equivalent unit for trading. But a growing share is payment: remittances, business-to-business settlement, payroll for remote workers, and, in countries with unstable currencies, a way for households and firms to hold and move dollars without a US bank account. In February 2025, the governor of Brazil's central bank, Gabriel Galípolo, said that around 90 percent of the flow in Brazil's crypto market was linked to stablecoins, and that much of it appeared to relate to paying for goods abroad. This chapter asks the question that frames this whole book. Which of the three problems, information, value and trust, does a stablecoin actually solve, and where do the costs it does not solve go? The sandwich Very few people want to be paid in a stablecoin and keep it. The nurse in Dubai earns dirhams; her mother in Manila buys rice in pesos. A stablecoin remittance is therefore almost always a three-layer transaction that practitioners call a stablecoin sandwich: local currency in, stablecoin across, local currency out. In the first layer, the on-ramp, the sender's dirhams are converted into a dollar stablecoin. This might happen at a cryptocurrency exchange, through a remittance company that uses stablecoins behind the scenes, or through a payment provider that holds stablecoin inventory. The on-ramp requires converting dirhams into dollars, which is a foreign exchange transaction priced at some margin, and then acquiring the tokens, either by buying them on a market or by having the issuer mint them against dollars deposited in a bank. In the middle layer, the stablecoin moves from the sender's wallet, or the remittance company's wallet, to a wallet controlled by someone in the Philippines. This is the part that is genuinely transformed. It happens in seconds or minutes, at any hour, without correspondent banks, nostro accounts or cut-off times, and on low-cost networks the fee for the transfer itself can be a small fraction of a dollar regardless of the amount sent. In the third layer, the off-ramp, the stablecoins are converted into pesos and paid into the recipient's bank account or mobile wallet, or handed over as cash. The off-ramp requires a local partner with peso liquidity, a licence to operate, and a connection to the Philippine payment system. Its pricing includes a second foreign exchange margin, from dollars to pesos, and the cost of the local payout. Seen this way, a stablecoin rail is a correspondent chain with the middle replaced. The sending bank's nostro account and the chain of intermediaries are replaced by a blockchain transfer. But the ends, which is where the foreign exchange, the local liquidity and most of the compliance cost live, are still there. They are now operated by exchanges, payment companies and licensed payout partners instead of banks. Where the savings come from Table 5 sets out, component by component, how the cost of a cross-border payment is borne in the three models this book has examined: the correspondent chain, a link between instant payment systems, and the stablecoin sandwich. The table is a qualitative summary of the mechanisms described in this and earlier chapters rather than a measurement; the actual cost in any corridor depends on its volume, competition and regulation. Table 5. Where the cost of a cross-border payment sits under three models. Cost component Correspondent chain Instant payment link Stablecoin sandwich Message transmission Low, via SWIFT Low, via the link Low, on-chain Intermediary fees One per hop, often unknown Few, disclosed upfront Network fee, usually small Foreign exchange margin One conversion, set by bank One conversion, competing quotes Two conversions, on and off ramp Prefunded liquidity Nostro balances per corridor Balances in both systems Stablecoin and local-currency inventory Compliance work Repeated at each bank Sender and receiver, in seconds On-ramp and off-ramp providers Recourse if wrong Recall request possible Scheme rules, if agreed Usually none on-chain The middle of the sandwich produces genuine savings in three ways. First, it removes the stack of intermediary fees and the uncertainty about how many there will be. Second, it compresses settlement time from days to minutes, which matters most to businesses: a supplier paid in minutes instead of three days needs less working capital, and a payment company that settles with its partners in minutes needs to prefund less. Third, it is open all the time, so there are no cut-off times, no weekend gaps and no mismatch of time zones. There is also a less obvious saving in liquidity. A payment company operating in fifty corridors through correspondent banks must hold prefunded balances in fifty currencies in fifty places. A company using stablecoins as its settlement layer can hold one large pool of dollar stablecoins and move it to wherever it is needed in minutes, converting into local currency only at the point of payout. The total amount of capital tied up can be much smaller. This is the real economic argument for stablecoins in cross-border payments, and it is the one that has drawn established payment companies to them. Stripe's acquisition of the stablecoin infrastructure company Bridge, completed in early 2025, and the card networks' experiments with settling with their partners in stablecoins, are bets on this liquidity advantage more than on consumers holding tokens. Where the costs go The costs that stablecoins do not remove are worth naming precisely, because they are where marketing claims often go wrong. The foreign exchange cost is still there, and in a sandwich it is incurred twice. Converting dirhams to dollars and dollars to pesos involves two margins, unless the corridor is one where the sender already holds dollars. In large corridors with deep, competitive on- and off-ramp markets, these margins can be small. In thin corridors, where few local exchanges or payout partners exist and local currency liquidity against stablecoins is shallow, they can be large, and can exceed the margins in the correspondent system. The dollar stablecoin is a common middle currency, but it does not create liquidity in the local currencies at either end. The liquidity cost is reduced but not abolished. Someone must hold local currency to pay out pesos when stablecoins arrive, and someone must hold stablecoins, or the dollars to mint them, to accept dirhams. These inventories must be funded and rebalanced, and their holders bear the risk that local currency rates move. The compliance cost is moved, not removed. Regulated on-ramps and off-ramps must identify their customers, screen transactions against sanctions lists and monitor for money laundering, just as banks do. Since 2019 the FATF has applied its "travel rule", the requirement that information about the originator and beneficiary travel with a payment, to virtual asset service providers, meaning that exchanges sending stablecoins to each other must exchange customer information through separate channels, since the blockchain transfer itself carries only wallet addresses. Implementation of the travel rule across jurisdictions remains uneven. Where on- and off-ramps comply fully, stablecoin payments carry compliance costs comparable to other licensed payment providers. Where they do not, the savings come partly from skipping compliance, which is not a durable advantage. The trust problem on a public ledger Stablecoins solve the information problem in an unusual way. The blockchain records every transfer publicly and permanently, so the movement of value is more transparent than in any banking system. But the record shows only wallet addresses and amounts. Who controls the addresses, and why they are paying each other, is not on the chain. Rich data in the ISO 20022 sense, names, addresses, purposes, invoice references, travels, if at all, outside the transfer, between the on-ramp and off-ramp providers. The transparency of the ledger is real but of a different kind from the transparency regulators require. The trust problem appears in three forms. The first is issuer risk. A stablecoin is only as good as the reserves behind it and the issuer's ability and willingness to redeem. In March 2023, when Silicon Valley Bank failed, Circle disclosed that about 3.3 billion dollars of USDC's reserves were held there, and USDC briefly traded well below one dollar until the US authorities guaranteed the bank's deposits. The collapse of TerraUSD in May 2022 showed what happens to a token that claims a dollar peg without holding dollar reserves at all. Regulation, discussed in the next chapter, is largely an answer to this form of trust problem. The second is finality without recourse. A blockchain transfer cannot be reversed by anyone other than the recipient. A payment sent to the wrong address, or to a fraudster, is gone unless the issuer intervenes. Large stablecoin issuers do have the technical ability to freeze tokens held at particular addresses, and they use it in response to law enforcement requests and sanctions, which means the system is not as unstoppable as its early advocates claimed. But there is no equivalent of a scheme rulebook allocating losses or a reimbursement regime. The lessons of the previous chapter about authorised push payment fraud apply with full force: a fast, final, push-only payment system without name checking or liability rules is a gift to fraudsters, and investment scams that end in stablecoin transfers have been one of the largest categories of financial fraud of recent years. The third is integrity. Because stablecoins can be held in self-custodied wallets, outside any regulated institution, they can be used by people and entities that the banking system has excluded, legitimately or not. For a small business in a country with currency controls, that is the appeal. For a sanctions authority, it is the problem. Stablecoin issuers and blockchain analytics firms have built substantial capabilities to trace flows and freeze assets, and enforcement agencies have used them. But a system whose openness is its advantage will always sit in tension with regimes built on knowing who is paying whom. Two corridors, two answers The arithmetic of the sandwich is easiest to see by comparing two corridors with different characteristics. The numbers that follow are illustrative assumptions, chosen to show the mechanism rather than to report any provider's prices. In the first corridor, a worker in the United States sends two hundred dollars to a family in Mexico. This is one of the largest and most competitive remittance corridors in the world. Traditional money transfer operators compete hard, and the price of a transfer through the cheapest of them, fee plus margin, is already low by global standards. A stablecoin route must on-ramp in the United States, which is cheap because the sender already holds dollars, and off-ramp in Mexico, which requires converting dollars to pesos through a local exchange or payout partner. Even if the blockchain transfer costs a few cents, the off-ramp margin and the payout fee leave little room to undercut an established operator. The stablecoin route may be faster, but it is not dramatically cheaper. In the second corridor, a small importer in an African country with foreign exchange shortages needs to pay a supplier in China twenty thousand dollars. Its local bank can supply dollars only after a wait of weeks, through a correspondent chain with several intermediaries, at an official rate the importer cannot actually obtain in practice. A stablecoin route lets the importer buy dollar tokens locally, from an exchange or over-the-counter desk, at a market rate, and pay the supplier's agent, who converts them to yuan. The local on-ramp may carry a sizeable margin, and the legal position may be uncertain, but the alternative is not a cheaper bank wire; it is a long delay, an unattainable official rate, or an informal channel. Here the stablecoin route is decisively better for the user. The difference between the corridors has nothing to do with technology. It comes from how well the existing system serves each one. Stablecoins are most valuable precisely where correspondent banking has withdrawn, where currencies are controlled, and where the formal system is slowest. That is also where their use is most worrying to the authorities, because it is where they are most likely to be used to evade controls that governments regard as legitimate. The dollarization objection That observation leads to the most serious policy objection to stablecoins in cross-border payments, raised repeatedly by emerging-market central banks and by the International Monetary Fund. A dollar stablecoin held on a phone is, in effect, a dollar bank account available to anyone, outside the reach of the local banking system. In a country with high inflation or capital controls, households and firms may shift savings and payments into it on a large scale. For an individual, that may be rational and even liberating. For the country, it can weaken monetary policy, since the central bank's interest rate affects a shrinking share of the money in use; drain deposits from local banks, reducing their ability to lend; and undermine capital controls that the government considers necessary for stability. It also transfers the income that a country earns from issuing its own currency to a private issuer whose reserves sit in US Treasury bills. Several emerging-market central banks have expressed concern that the spread of dollar stablecoins is a new form of the old problem of dollarization, made far easier by technology. There is no simple answer. Countries cannot easily block access to public blockchains, and attempts to ban stablecoin use tend to push it into informal channels. Some are responding by improving their own payment systems and widening access to foreign currency through regulated channels, on the theory that people turn to stablecoins when the formal system fails them. Others are regulating on- and off-ramps tightly, so that conversion between local currency and stablecoins happens only through licensed providers that report to the authorities. The European Union's limits on non-euro stablecoins used as a means of exchange, described in the next chapter, show that even rich economies with strong currencies take the risk seriously. Where stablecoin rails win Put together, the analysis suggests where stablecoin rails make sense, and where they do not. They are strongest where the correspondent chain is weakest: corridors where banks have withdrawn, where currencies are unstable or controlled, where the recipient wants dollars anyway, and where the alternative is a slow, expensive wire through several intermediaries or an informal channel. They are strong for businesses that need to settle frequently with partners in many countries and can benefit from holding a single pool of dollar liquidity. They are strong at weekends and holidays, when other channels are closed. They are weakest where a modern domestic instant payment system and a competitive foreign exchange market already exist at both ends, especially if the two systems are linked. There, the stablecoin sandwich adds a second conversion and new operational risks for little gain in speed. They are also weak wherever the off-ramp is thin, because the cost of converting stablecoins into local currency in a shallow market can wipe out the savings of the middle layer. The pattern is the one this book has traced from the start. The stablecoin attacks the value problem across borders more directly than any previous technology, by creating a bearer dollar that moves without a chain of banks. It leaves the information and trust problems to be handled at the edges, by the same kinds of regulated intermediaries that handle them now. Whether stablecoins become a mainstream payment rail therefore depends less on blockchain technology than on how those edges are governed. Since 2024 the major jurisdictions have begun to answer that question, and the answer is reshaping the banks as much as the stablecoin issuers. That is the subject of the final chapter. Chapter 9: Rules, Tokens and the Banks' Reply For most of their first decade, stablecoins existed in a regulatory twilight. Their issuers were licensed, if at all, as money transmitters or trust companies, under rules written for other businesses. Their reserves were described in attestations of varying quality. Their legal status, money, security, commodity, or something new, was contested. That ambiguity was itself a barrier to their use in mainstream payments: no bank treasurer or regulated payment company could build a business on an instrument whose legal nature might be redefined by a court or an enforcement action. Between 2023 and 2026, the ambiguity was largely resolved in the jurisdictions that matter most. The European Union, the United States, Hong Kong and others adopted specific regimes for stablecoin issuance. These regimes answer the issuer side of the trust problem described in the previous chapter, what backs the token and whether it can be redeemed, and in doing so they have changed the competitive landscape. Banks, which for years treated stablecoins as a curiosity or a threat, have responded by building their own tokenized versions of money. Even SWIFT has announced that it will add a blockchain-based ledger to its infrastructure. The question now is not whether tokenized money will be used in cross-border payments, but which form of it, issued by whom, and connected to the existing system in what way. Europe: MiCA The European Union moved first. Its Markets in Crypto-Assets Regulation, known as MiCA, adopted in 2023, created two categories of stablecoin. An e-money token references a single official currency; an asset-referenced token references a basket or other assets. The rules for both applied from 30 June 2024, ahead of the rest of the regulation at the end of that year. The rules for e-money tokens tie them firmly to the existing payments framework. They may be issued only by banks or licensed electronic money institutions. Holders have a right to redeem at par at any time, free of charge. Issuers may not pay interest. Reserves must be held in secure, liquid assets, with a substantial portion in deposits at credit institutions. Tokens that become large are classified as "significant" and supervised by the European Banking Authority. And MiCA includes limits on stablecoins referencing a non-EU currency when they are widely used as a means of exchange, a provision aimed squarely at the risk of dollar stablecoins displacing the euro in everyday payments. The practical effect was immediate. Circle obtained an electronic money institution licence in France in 2024 and made USDC available under MiCA. Tether did not seek authorisation for USDT, and major exchanges serving European customers restricted or removed USDT trading for them. The episode demonstrated that regulation could reshape the stablecoin market quickly: the largest stablecoin in the world became, for regulated European platforms, largely unavailable. The United States: the GENIUS Act The United States followed with the Guiding and Establishing National Innovation for U.S. Stablecoins Act, the GENIUS Act, signed into law on 18 July 2025. It created the first federal framework for what it calls payment stablecoins. Its core provisions resemble MiCA's in substance, though not in structure. A payment stablecoin must be backed at least one-for-one by a narrow set of reserve assets: US currency, deposits at insured banks, Treasury bills with a maturity of ninety-three days or less, certain repurchase agreements and money market funds investing in such assets. Issuers must publish the composition of their reserves monthly. Holders must be able to redeem promptly at par, and in an issuer's insolvency their claims have priority over other creditors. Issuers may not pay interest or yield to holders. Issuers must have the technical ability to comply with lawful orders to freeze or seize tokens, and they are treated as financial institutions under the Bank Secrecy Act, with the same anti-money-laundering obligations as banks. The act creates a dual system. Issuers with more than ten billion dollars in outstanding stablecoins come under federal supervision; smaller issuers may opt for state regimes that federal regulators judge substantially similar. Banks may issue stablecoins through subsidiaries. Foreign issuers may offer stablecoins in the United States only if their home regime is deemed comparable and they meet certain conditions. The law gave regulators a year to write implementing rules, and it takes effect on the earlier of 18 January 2027 or 120 days after final regulations are issued. At the time of writing, in October 2026, the regulators have published proposed rules, and the industry is preparing for the effective date. Two features of the GENIUS Act matter particularly for cross-border payments. The first is the prohibition on yield. A stablecoin that pays no interest is a poor store of value compared with a bank deposit or a money market fund, but it is a perfectly good means of payment. By banning yield, the law pushes stablecoins toward being used for moving money rather than holding it, and industry data from 2026 suggest some treasurers have indeed moved idle balances out of payment stablecoins into tokenized money market funds that can pay interest. The second is the treatment of foreign issuers, which matters because the largest dollar stablecoin is issued outside the United States. The law effectively offers a choice between becoming subject to US standards and accepting limits on access to the US market. Hong Kong brought its own stablecoin licensing regime into force in August 2025, and other financial centres, including the United Kingdom and Singapore, have built or proposed frameworks. The common direction is clear: in the major jurisdictions, a payment stablecoin is becoming a regulated, fully reserved, non-interest-bearing claim on a supervised issuer, much like electronic money. The regulation solves the issuer-risk part of the trust problem. It does not, on its own, solve the fraud, finality or edge-compliance problems the previous chapter described. The central banks' objection Not everyone accepts that regulated stablecoins are a sound basis for the payment system. The Bank for International Settlements made the case against them most directly in its Annual Economic Report of June 2025, which argued that stablecoins fail three tests that a sound monetary system must pass. The first test is singleness: every unit of money must be accepted at par with every other, so that a dollar in one bank is always worth a dollar in another. Bank deposits achieve this because banks settle with each other in central bank money. A stablecoin, the BIS argued, trades on secondary markets at prices that can deviate from par, and different stablecoins are not settled against each other through a common central asset, so they behave more like the notes of private banks in the nineteenth century, each with its own risk. The second test is elasticity: the system must be able to expand to meet payment needs, which bank deposits do through lending and central bank liquidity; a fully reserved stablecoin cannot. The third is integrity: the system must resist financial crime, which stablecoins, circulating on public blockchains among unidentified wallets, find hard. The BIS's alternative is what it calls a unified ledger: a programmable platform on which tokenized central bank reserves, tokenized commercial bank deposits and tokenized securities coexist, so that settlement can be instant and atomic, with both legs of a transaction occurring together, while money retains its two-tier structure of central bank and commercial bank money. Its Project Agorá, launched in 2024 with seven central banks, including the Federal Reserve Bank of New York, the Bank of England, the Bank of Japan and the Banque de France, and dozens of private financial institutions, explores this design for cross-border payments. One need not fully accept the BIS's conclusions to see the force of its framework. The argument is about the value problem at the deepest level: what asset sits at the top of the hierarchy, and whether a new form of money can be fully integrated with the old. A stablecoin is a claim on an issuer whose reserves sit in banks. It moves outside the banking system but is backed by it. Whether that is a stable arrangement at a scale of trillions rather than hundreds of billions is a question no one can yet answer with confidence. The banks build their own The banks' reply to stablecoins has been to tokenize deposits. A tokenized deposit is an ordinary bank deposit, a liability of a regulated bank, covered by its capital, liquidity rules and deposit insurance where it applies, represented as a token on a blockchain so that it can move with the speed and programmability of a stablecoin. JPMorgan has operated a blockchain-based deposit platform for institutional clients for several years, rebranded as Kinexys in 2024, through which it reports processing large daily volumes of payments between its own corporate clients' accounts in different locations, at any hour. Other large banks have built similar systems. The limitation of tokenized deposits is that each bank's tokens, like each bank's deposits, are claims on that bank. Moving a tokenized deposit from one bank to another still requires interbank settlement. A world of tokenized deposits at many banks therefore needs a common settlement layer, which brings the argument back to the central bank and to the unified ledger concept. SWIFT's response has been the most significant institutional signal. On 29 September 2025, at its annual Sibos conference, SWIFT announced that it would add a blockchain-based shared ledger to its infrastructure, beginning with a prototype built with the technology company Consensys and involving more than thirty banks from sixteen countries, among them Bank of America, HSBC, JPMorgan, Deutsche Bank, DBS and Standard Chartered. The stated first use is real-time, around-the-clock cross-border payments between financial institutions, with the ledger recording and validating transactions through smart contracts and allowing the movement of tokenized regulated value. SWIFT has also announced a new scheme for retail cross-border payments, built on the completed ISO 20022 migration, with commitments on speed and on disclosing fees upfront. The organisation whose messages define the correspondent chain is rebuilding itself so that it can carry value as well as information. What changes for remittance providers For the companies that move money for migrants, the new regime is both an opportunity and a squeeze. The opportunity is obvious: a regulated dollar stablecoin gives a remittance company a settlement asset it can hold in one pool and move to any payout partner in minutes, reducing the prefunded balances it must keep in each corridor. Several established money transfer operators have begun to use stablecoins for treasury settlement with their partners, while still presenting customers with an ordinary transfer in local currency at each end. For the sender, nothing visible changes except, ideally, speed and price. The squeeze comes from two directions. From one side, new entrants that build on stablecoins from the outset can operate with far less capital tied up in corridors, and can reach recipients who hold stablecoin wallets directly, without a payout network at all. From the other side, the domestic instant payment systems at the receiving end, Pix in Brazil, UPI in India, mobile money in East Africa, have made the last mile far cheaper, so that any provider with access to them can deliver funds instantly. The traditional remittance company's advantages, its network of agents and its correspondent relationships, are being eroded at both ends of the chain. What remains valuable is what this book has identified as the irreducible core: access to competitive foreign exchange in the corridors that matter, and the licences and compliance capability to operate legally in many jurisdictions at once. That suggests a likely outcome. The remittance industry will not be replaced by stablecoins; it will absorb them, as it absorbed earlier technologies, using them where they reduce liquidity costs and ignoring them where domestic links or netting serve better. The firms that thrive will be those that treat the choice of rail as an internal routing decision, made payment by payment, rather than a strategic identity. Where central bank digital currencies fit A further contender for the role of cross-border settlement asset is money issued directly by central banks in tokenized or digital form. Central bank digital currencies come in two kinds. A retail CBDC would be available to the public, a digital equivalent of banknotes. A wholesale CBDC would be available only to banks and other financial institutions, a tokenized form of the reserves they already hold at the central bank. Retail CBDCs have proved politically difficult in the large advanced economies. The European Central Bank has been preparing a digital euro, with the legislation still under debate in the European Parliament and Council, and with its designers emphasising holding limits so that it does not drain deposits from banks. In the United States, an executive order in January 2025 prohibited federal agencies from establishing or promoting a CBDC, and policy attention has shifted decisively toward regulated private stablecoins. Several emerging economies have launched retail CBDCs with modest uptake. For cross-border payments, the more important form is wholesale. If central banks issue tokenized reserves on platforms that can interoperate, banks in different countries could settle with each other directly in central bank money, and foreign exchange could settle both legs simultaneously, eliminating the settlement risk that has haunted the system since Herstatt. This is the idea behind Project Agorá and behind the earlier multi-central-bank experiments. It goes to the heart of the value problem: it offers a top-of-the-hierarchy settlement asset across borders, something the world has never had. The obstacle is governance, not technology. A platform on which several central banks issue money must have rules about who can access it, whose law applies, how disputes are settled and what happens when one participant's policy, sanctions for example, conflicts with another's. The experience of mBridge showed how quickly such a platform becomes entangled with geopolitics. The most likely path is not a single global platform but a set of interoperable ones, each governed by a group of central banks with shared interests, connected through common standards. ISO 20022 will almost certainly be one of those standards. Convergence Step back, and the shape of the next decade becomes visible. The three paths that this book has followed, rich data, instant domestic rails and tokenized money, are converging. ISO 20022 is becoming the common data language not only for the correspondent chain but for instant payment systems, their cross-border links, and, through bridges built by SWIFT and others, for blockchain-based payments. A stablecoin or tokenized deposit transfer that is part of a regulated cross-border payment will increasingly be accompanied by an ISO 20022 message carrying the same structured information that a wire would. The information problem is being solved in one language across rails. Instant settlement is becoming the norm for value within currencies, through domestic systems running around the clock, and is being extended across currencies by links, by shared ledgers and by stablecoins. The value problem is being attacked from several directions at once, and the competition among them will determine which approach prevails in which corridors. The trust problem, meanwhile, is being rebuilt through regulation: reserve and redemption rules for stablecoin issuers, verification of payee and reimbursement rules for instant payments, the revised FATF standard for payment transparency. None of these is a technology. All of them are agreements about who must know what, and who bears the loss when things go wrong. The cross-border payment of 2030, if the current trajectory holds, may look something like this: an instruction carrying full structured data, screened in seconds by both sides, settled across a shared ledger in tokenized money or through linked instant systems, with the currency conversion priced competitively and disclosed upfront. It will be much faster than today's correspondent payment and somewhat cheaper. Whether it will be dramatically cheaper depends on what happens to the costs that no rail removes: the foreign exchange margin, the liquidity needed in thin currencies, and the compliance work of establishing who is paying whom. Those costs are the subject of the conclusion. Conclusion The nurse in Dubai from the opening of this book will, within a few years, almost certainly be able to send money to Manila faster than she can today. The message carrying her payment will be structured from the moment she enters it, with her mother's name checked against the receiving account before she confirms. The money may travel through a correspondent chain whose banks all speak ISO 20022 and pass the payment along in minutes, through a link between the Emirates' and the Philippines' instant payment systems, or as a regulated dollar stablecoin moving between a licensed exchange in Dubai and a payout partner in Manila. In each case, the delivery will be close to instant. Whether it will be much cheaper is a different question, and the answer follows from everything this book has argued. A cross-border payment has three costs that no rail can make disappear. Someone must convert one currency into another, at a price that reflects the depth of the market between them. Someone must hold money in advance on the receiving side, so that it can be paid out the moment the payment arrives. And someone must establish, to the satisfaction of two or more legal systems, who is paying whom and why. Modernization has attacked the friction around these costs, the redundant intermediaries, the batch schedules, the truncated data, the manual repairs, and has removed much of it. The costs themselves remain, and in the expensive corridors they are most of the bill. Several conclusions follow, and they are more practical than the debates about technology that dominate the field. The first is that the data project is not finished, and it matters more than it looks. The completion of the ISO 20022 migration in November 2025 was an achievement in the container, not in the contents. The gains from rich data, faster screening, fewer false positives, automatic reconciliation, effective verification of payees, depend on structured information being captured at the source and carried intact. The extension of SWIFT's structured address deadline in 2026 shows how far the industry still has to go. Yet every other improvement depends on it. Instant cross-border payment is only safe if compliance can be done in seconds, and compliance can be done in seconds only on clean data. The same is true of stablecoin payments at regulated edges, which must carry the same information outside the blockchain. A policymaker who wants cheaper cross-border payments should care less about which rail wins and more about whether the information travelling on every rail is good enough to act on without a human looking at it. The second is that speed without trust machinery simply moves losses onto users. The domestic experience of instant payments, from Britain's reimbursement regime to Brazil's return mechanism to the euro area's verification of payee, shows that fast, final, push payments need deliberate protections: name checking, targeted friction for risky payments, shared fraud intelligence, and liability rules that give receiving institutions a reason to close mule accounts. These protections are being built domestically and barely exist across borders. Stablecoin networks, with their irreversible transfers and pseudonymous wallets, have almost none of them at the protocol level. As cross-border payments become instant, the absence of cross-border fraud rules will become the most important gap in the system. Regulators who are now writing stablecoin regimes focused on reserves and redemption have done the easier half of the work. The third is that the remaining cost is concentrated where the existing system serves people worst. The averages that official targets measure are pulled up by thin corridors: small currencies, weak domestic payment systems, de-risked banking relationships, high perceived compliance risk. These are also the corridors where remittances matter most to recipients' livelihoods and where stablecoins have found their strongest use. That convergence is not an accident. Stablecoins succeed where the formal system has withdrawn, which is why they are both genuinely valuable and genuinely worrying to the authorities in those countries. The most durable way to reduce costs in those corridors is not a new rail but the unglamorous work of building domestic instant payment systems at the receiving end, deepening local foreign exchange markets, and making compliance cheaper through shared utilities and better data, so that banks are willing to serve these countries again. The fourth is that competition among rails is the most useful thing modernization has produced. For decades the correspondent chain had no real rival for most cross-border payments, and its pricing showed it. It now faces specialist remittance firms that net flows through local accounts, instant payment links with transparent competing exchange quotes, and stablecoin routes that can settle in minutes. The banks' responses, SWIFT's tracking service, its retail payment scheme, its shared ledger, tokenized deposits, would not have happened without that pressure. The policy aim should be to keep the rails contestable: to let regulated non-banks reach central bank settlement where it is safe, to keep the edges of stablecoin networks open to licensed competitors, and to require upfront disclosure of the full price, including the exchange-rate margin, on every rail, so that users can choose. The fifth is that the form of money at the top of the hierarchy is now genuinely in question, for the first time in generations. Regulated stablecoins, tokenized bank deposits and wholesale central bank digital currencies are three different answers to the question of what asset should settle a cross-border payment. Each has a different relationship to the central bank, a different risk profile and a different politics. The outcome will not be decided by which technology is fastest, since all of them can be fast. It will be decided by which can satisfy the trust requirements of two countries at once: redemption at par that both sides believe in, compliance that both regulators accept, and a way of assigning losses when something goes wrong. None of this is an argument for complacency about the old system or for scepticism about the new. The modernization of payment rails over the last decade has been real and, in places, remarkable. Brazil and India have shown that a well-designed instant payment system can change a country's economic life within a few years. The ISO 20022 migration, slow and frustrating as it was, has given the world a common language for money that will outlast every rail described in this book. Stablecoins have forced a long-overdue reckoning with how much cross-border payment costs and why. But the problem was never only the speed of the wire. A payment is information, value and trust arriving together. Any system that delivers one of the three and leaves the others to someone else has not solved the problem; it has moved it. The test of every innovation in this field, now and in the future, is simple to state and hard to pass: after the payment has arrived, who did the work of conversion, who held the money in advance, who established that the payment was legitimate, and what did each of them charge for it? Hashtags: #FinTech #PaymentRails #FinancialTechnology #ISO20022 #RealTimePayments #RTP #InstantPayments #CrossBorderPayments #CrossBorderRemittances #InternationalMoneyTransfers #PaymentInfrastructure #DigitalPayments #PaymentModernization #SWIFT #SWIFTGPI #CorrespondentBanking #PaymentMessaging #ClearingAndSettlement #RealTimeGrossSettlement #RTGS #FedNow #UPI #PixPayments #PaymentInteroperability #ProjectNexus #StablecoinPayments #ForeignExchange #LiquidityManagement #PaymentSecurity #FinancialInclusion

Download the Book (PDF): Introduction A fourteen-year-old is stuck on a problem about the area of a trapezoid. She types the problem into a chatbot on her phone, and within two seconds she has the answer, a tidy derivation, and an offer to explain anything further. She copies the answer into her homework and moves on. The next morning, on a quiz with no phone allowed, she meets another trapezoid and cannot begin. In a different classroom, a boy working on the same topic in a piece of software gets the problem wrong. The system does not show him the answer. It asks him which two sides are parallel. He guesses. It asks him what the formula needs to know about those sides. Three exchanges later he has the answer, and the software, having watched him struggle on the step that combines the two bases, quietly schedules two more problems that turn on exactly that step, then a review of the same idea four days later. The next morning he gets the quiz question right. Both students used artificial intelligence. Both received instant, personalized help. One learned and one did not, and the difference had almost nothing to do with how clever the machine was. It had to do with two things the second system did and the first did not: it kept a running estimate of what the student actually knew, and it declined to do his thinking for him. That is the argument of this book. The excitement around AI in education, which has been intense since large language models arrived in classrooms in late 2022, tends to focus on the most visible part of the technology: the fluent, patient, tireless conversational partner that can explain photosynthesis at any hour in any language. That part is real and it matters. But on its own it is not a tutor. A tutor is a system that knows what you know, decides what you should do next, and makes you do it. Personalization is a measurement problem before it is a content problem, and an AI tutor helps only as much as it knows what the learner knows and declines to think on the learner's behalf. Three parts of a tutor It helps to divide any tutoring system, human or machine, into three parts, because the history of the field and the argument of this book both follow that division. The first part is a model of the learner. A good human tutor builds one without thinking about it: within a few minutes she knows that this student can factor quadratics but freezes on negative exponents, reads quickly but skims, and gives up after the second wrong attempt. Machines have to build this model deliberately, from the only evidence they have, which is a stream of right and wrong answers, hesitations, and requests for help. The most influential method for doing so, Bayesian knowledge tracing, was published in the mid-1990s and is still running inside commercial software used by millions of students. It treats a student's knowledge of each skill as a hidden quantity that cannot be observed directly, only inferred, and it updates that inference with each answer, allowing for the fact that students sometimes guess right and sometimes slip on things they know. Its descendants include statistical models borrowed from psychometrics and deep neural networks trained on hundreds of millions of student responses. The second part is a policy for what comes next. Once a system has a belief about what a student knows, it has to choose: another problem on the same skill, a harder one, a review of something learned last month, a short explanation, a move to the next topic. These choices add up to a curriculum path, and a personalized curriculum is simply a sequence of such choices made for one student rather than for a class. The science behind good choices is older than computing and surprisingly robust: spaced practice beats massed practice, mixed practice beats blocked practice, and practice that requires recall beats rereading. The engineering challenge is to apply those findings to one learner at a time without wasting their hours on material they have mastered or pushing them into material they are not ready for. The third part is the interaction itself: the hints, explanations, questions and feedback that a student actually sees. For most of the history of intelligent tutoring this was the weakest part. Software could detect that a student had made a particular error in an algebra step and display a pre-written hint about it, but it could not hold a conversation. Large language models have changed that almost overnight. For the first time, a machine can respond to a student's own words, follow a wandering line of reasoning, ask a question that fits the specific confusion in front of it, and do so in something close to the Socratic manner that good tutors have used for millennia. The central claim of this book is that these three parts are not equally novel and are not equally sufficient. The conversational layer is the newest and the most impressive, and it is the one most likely to be mistaken for the whole. A language model with no learner model and no sequencing policy is a brilliant answer engine attached to nothing. Students can and do use it to learn, but they can just as easily use it to avoid learning, and the evidence now emerging from controlled trials shows both outcomes clearly. Why disruption has been slow, and why it may not be now Education technology has a long and humbling record of predicted revolutions. Thomas Edison thought motion pictures would largely replace textbooks. Radio, television, programmed instruction, the personal computer, the internet, and massive open online courses each arrived with forecasts that the classroom as we knew it was finished. The classroom outlasted them all, absorbing each technology into a recognizably similar daily routine. There are good reasons for this resilience, and they are not merely institutional inertia. Schools do many things besides transmit knowledge: they supervise children, socialize them, certify them, and give them a reason to show up. Learning itself depends heavily on motivation, attention and relationship, which a teacher in a room supplies in ways software has struggled to replicate. And the students who benefit most from self-paced technology tend to be the ones who were already motivated and well-supported, which means that technology offered on an opt-in basis often widens the gaps it was meant to close. Does generative AI break this pattern? This book's answer is that it may, but not in the form most often imagined. The disruption is unlikely to be the replacement of teachers by chatbots. It is more likely to be a slow redistribution of the labor of teaching: machines taking over more of the diagnosis, practice and individual feedback that one adult cannot provide to thirty students at once, and teachers spending more of their time on the things only they can do. Whether that redistribution helps students depends on design choices being made right now, by companies, school systems and regulators, often without much attention to the evidence. What this book covers The chapters follow the three-part structure. Chapter 1 sets out the goal that the whole field has chased for forty years, the remarkable effectiveness of one-to-one tutoring, and explains what about tutoring actually produces learning. Chapters 2 and 3 cover the learner model: first Bayesian knowledge tracing in enough detail to see exactly how it reasons, then its limitations and the family of alternatives that has grown up around it. Chapter 4 turns to sequencing and curriculum paths, from prerequisite maps to spaced repetition to attempts at using reinforcement learning to choose what a student should do next. Chapter 5 takes stock of the evidence on adaptive tutoring systems before the arrival of large language models, which is more substantial and more mixed than either enthusiasts or skeptics usually acknowledge. Chapters 6 and 7 cover the new conversational layer: how large language models can be shaped into Socratic tutors, what goes wrong when they are not, and what the first wave of randomized trials has found. Chapter 8 addresses teachers, data and power: what happens to the teacher's role, who owns the detailed records that learner models depend on, how bias enters a model of knowledge, and how regulation is beginning to respond. The conclusion draws out what follows from all of this for those choosing, building or governing these systems. Throughout, the book stays close to evidence. Where a claim rests on a specific study, the study is named and described honestly, including its limits. Where the evidence is thin or contested, the book says so. The field has been badly served by both breathless promotion and reflexive dismissal; the reality is more interesting than either. A note on scope. This book concentrates on academic learning in school and early university, where most of the research has been done, and on subjects such as mathematics, science and language, where intelligent tutors have the longest track record. It touches only lightly on AI for grading, for administrative tasks, or for detecting cheating, important as those are, because they are not where the question of personalized learning is decided. And it treats the student, not the institution, as the unit that matters: a system is judged here by whether the person using it ends up knowing more than they did. Chapter 1: The Two Sigma Problem In 1984 the educational psychologist Benjamin Bloom published a short paper in Educational Researcher that has shaped the ambitions of educational technology ever since. Its title was "The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring." Bloom reported on studies, conducted by his doctoral students at the University of Chicago, that compared three conditions for teaching the same material: a conventional class of about thirty students, a class using mastery learning, in which students were tested regularly and given corrective work until they reached a set standard, and one-to-one or small-group tutoring that also used mastery methods. The results were striking. Students taught by tutors performed, on average, about two standard deviations above students in the conventional class. Put in more intuitive terms, the average tutored student outperformed roughly 98 percent of the students taught conventionally. Mastery learning in groups achieved about one standard deviation. Bloom's "problem" was that one-to-one tutoring was far too expensive to provide to every child. The challenge he set his field was to find methods of group instruction that could come close to the tutoring result. Four decades later, almost every pitch for AI in education begins with Bloom's paper. If a machine could tutor every student individually, the argument runs, it could deliver the two sigma gain at the cost of software. The argument deserves to be taken seriously, but it rests on two assumptions that need examining first: that the two sigma figure is real, and that we understand what about tutoring produces it. How big is the tutoring effect, really? The two sigma figure has not held up well as a general estimate. Bloom's comparison drew on a small number of studies, with relatively short units of instruction, in which the tutored students received not only individual attention but also a carefully aligned mastery curriculum and frequent formative tests. Those are favorable conditions, and later research has found smaller effects. The most careful modern estimate of the tutoring effect comes from a 2020 meta-analysis by Andre Nickow, Philip Oreopoulos and Vincent Quan, published as a National Bureau of Economic Research working paper, which pooled nearly a hundred randomized trials of tutoring programs for students from pre-kindergarten through high school. They found an average effect of about 0.37 standard deviations. That is large by the standards of educational interventions, where an effect of 0.2 is often considered meaningful and many popular reforms produce nothing measurable at all. But it is less than a fifth of two sigma. The largest effects came from programs that were frequent, embedded in the school day, and delivered by trained teachers or paraprofessionals rather than volunteers or parents. A second estimate, more directly relevant to machines, comes from Kurt VanLehn's 2011 review in Educational Psychologist, "The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems." VanLehn gathered studies that compared human tutors and various kinds of computer tutors against students who received no tutoring but studied the same material. Human tutoring produced an average effect of about 0.79 standard deviations. That is impressive, but it is not two sigma, and VanLehn argued that Bloom's figure reflected the special conditions of the original studies rather than the typical power of tutoring. Strong evidence that tutoring works at scale, and an equally strong illustration of its cost, comes from Chicago. Beginning in the early 2010s, the University of Chicago Education Lab evaluated a program run by the nonprofit Saga Education in which ninth and tenth graders in low-income public high schools received a daily class period of mathematics tutoring, in groups of two students per tutor, from trained full-time tutors. Jonathan Guryan and colleagues, reporting on two large randomized trials in the Journal of Policy Analysis and Management in 2023, found gains in mathematics of roughly 0.18 to 0.40 standard deviations over a school year, along with reduced course failures. These are among the largest effects ever recorded for an intervention with adolescents, an age group long thought to be too old for remediation to make much difference. They came at a price of several thousand dollars per student per year, mostly for the tutors' salaries and training. That is affordable for a program; it is far beyond what most school systems could pay for every student in every subject. The cost, not the method, is what keeps tutoring rare, and it is the cost that technology promises to change. So the honest version of the promise is this. Good tutoring, delivered well and often, produces gains somewhere between a third of a standard deviation and three quarters of one, depending on the subject, the student and how the comparison is made. That is still one of the most powerful interventions education knows. A technology that reliably delivered even half of it to every student would be a historic achievement. But anyone who claims an AI tutor will produce two sigma is quoting a number that human tutors do not reliably produce either. What tutors actually do If tutoring works, why does it work? The easy answer is "individual attention," but that answer is too vague to build anything from. The research on tutoring dialogues, much of it conducted in the 1990s by Arthur Graesser, Michelene Chi and their colleagues, who recorded and transcribed hundreds of hours of real tutoring sessions, offers a more precise picture, and parts of it are surprising. The first surprise is that most tutors are not very skilled. Graesser and his colleagues found that typical tutors, including many who were effective, rarely used sophisticated pedagogical strategies. They did not systematically diagnose misconceptions, did not often use Socratic questioning in any rigorous sense, and frequently gave incorrect or vague feedback. Yet their students learned. Whatever makes tutoring work, it does not seem to require expertise in the art of tutoring. The second surprise, from Chi's work, is that the student's own activity matters more than the tutor's. In one series of studies, Chi and colleagues compared sessions in which tutors explained a great deal with sessions in which tutors were instructed to prompt the student to explain instead. Students learned as much or more when the tutor talked less. Learning tracked what the student was doing, not what the tutor was saying. Chi later developed this into the ICAP framework, published with Ruth Wylie in 2014, which ranks learning activities by the kind of engagement they demand: passive (receiving information), active (doing something with it, such as highlighting or copying), constructive (generating something new, such as an explanation or a prediction), and interactive (building on another person's contributions in dialogue). Across many studies, each level tends to produce more learning than the one before. Tutoring is powerful partly because it pushes students up this ladder: it is hard to remain passive when someone is sitting across from you asking what you think happens next. The third finding, which VanLehn formalized, concerns the granularity of interaction. VanLehn distinguished tutoring systems by the size of the step at which they interact with a student. An answer-based system checks only the final answer to a problem. A step-based system checks each step of the solution, such as each line of an algebraic derivation. A substep-based system goes further, engaging the student about the reasoning inside a single step. Human tutors work at all these levels, intervening the moment a student's reasoning goes wrong rather than waiting for a wrong final answer. VanLehn's review found that answer-based computer systems produced effects of about 0.31 standard deviations, while step-based intelligent tutoring systems produced about 0.76, essentially the same as human tutors. Substep-based systems, the ones that tried to emulate tutorial dialogue most closely, did not outperform step-based ones. His interpretation, which he called the interaction plateau, was that once feedback is fine-grained enough to catch errors at the step where they occur, making it finer brings little additional benefit. What mattered was not conversation as such but the timeliness and specificity of feedback, and the fact that the student was the one producing each step. These findings point to a set of mechanisms that any tutoring system, human or machine, has to supply. The student must do the cognitive work. Errors must be caught near where they happen, before they compound. Feedback must be specific to the error. The material must be pitched at the edge of what the student can currently do. And the whole process must keep the student engaged long enough for practice to accumulate. None of these mechanisms is "the tutor explains things clearly," which is the thing language models do best. That observation will matter a great deal later in this book. A century of teaching machines The ambition to automate tutoring is much older than artificial intelligence. In the 1920s Sidney Pressey, a psychologist at Ohio State University, built a mechanical device that presented multiple-choice questions and would not advance until the student pressed the key for the right answer. Pressey predicted an "industrial revolution" in education. It did not arrive, partly because of the Depression and partly because teachers and schools showed little interest. In the 1950s B. F. Skinner revived the idea with his own teaching machines, grounded in his behaviorist theory of learning. Skinner's machines presented material in small steps, required the learner to construct a response rather than choose one, and gave immediate confirmation. Programmed instruction, as the approach came to be called, enjoyed a boom in the early 1960s, with publishers producing programmed textbooks and companies marketing machines to schools. The boom faded within a decade. The materials were tedious, the evidence for their superiority was weak, and the linear sequence of small steps proved poorly suited to the variety of learners it was meant to serve. The historian Audrey Watters tells this story in detail in Teaching Machines (2021), and her account is a useful corrective to the belief that personalized learning is a new idea. Computers entered the picture in the 1960s with systems such as PLATO at the University of Illinois, which by the 1970s supported thousands of terminals, online discussion, and courseware in dozens of subjects. Patrick Suppes at Stanford built computer-assisted instruction programs for arithmetic and reading that adjusted the difficulty of drills to each student's performance. These systems were adaptive in a limited sense, branching on right and wrong answers, but they had no model of why a student answered as they did. The phrase "intelligent tutoring system" emerged in the 1970s and early 1980s, when researchers began to apply ideas from artificial intelligence and cognitive science. The ambition was to give the tutor three kinds of knowledge: knowledge of the domain (how an expert solves the problems), knowledge of the student (what this learner currently understands and misunderstands), and knowledge of teaching (what to do about it). This three-part architecture, often drawn as domain model, student model and pedagogical model, still organizes the field, and it maps closely onto the division used in this book. The most influential line of work came from John Anderson and his colleagues at Carnegie Mellon University. Anderson had developed a theory of human cognition, called ACT and later ACT-R, which held that complex skills are composed of many small units of procedural knowledge, each a rule of the form "if the goal is X and the situation is Y, then do Z." Learning a skill such as algebra, on this view, is a matter of acquiring and strengthening hundreds of such rules. Anderson's group built tutors for geometry, computer programming and algebra in which each rule was explicitly encoded. As a student worked, the tutor tracked which rules the student's steps were consistent with, a technique called model tracing, and intervened when a step matched a known incorrect rule or no rule at all. These became the Cognitive Tutors, later commercialized by Carnegie Learning, and their algebra curriculum was eventually used in thousands of American schools. Why the dream kept slipping If step-based tutoring systems were matching human tutors in VanLehn's review, why had they not transformed education by 2011, or by 2022? The reasons foreshadow much of this book. The first reason is cost of construction. Building a model-tracing tutor for a single course required encoding every correct and incorrect rule a student might use, writing hints for each, and testing them all. Estimates from the Carnegie Mellon group in the early years ran to hundreds of hours of development per hour of instruction. The systems worked best in highly structured domains such as algebra, geometry and programming, where solution steps could be enumerated. They worked poorly or not at all for essay writing, history, or anything where good answers take many forms. The second reason is that results in the laboratory and in short studies often shrank in real classrooms over a full school year. Students used the software less than intended, teachers integrated it unevenly, and the comparison condition in a real school was not a student studying alone but a teacher doing their best. Chapter 5 examines that evidence closely. The third reason is motivation. The tutoring literature's finding that students learn when they do the work has an uncomfortable corollary: students who do not want to do the work will not learn from a system that requires it. Students quickly discovered that many tutoring systems would eventually reveal the answer if they clicked the hint button enough times, a behavior researchers named "gaming the system." Ryan Baker and colleagues documented it extensively in the 2000s and showed that students who gamed learned substantially less. The fourth reason is that the interaction layer, the part students actually experienced, was rigid. Pre-written hints could not respond to a student's own explanation of what they were thinking, could not answer a question the designers had not anticipated, and felt mechanical. Systems built around natural-language dialogue, notably Graesser's AutoTutor, made real progress, but they were expensive to build and limited to narrow topics. Large language models remove the fourth obstacle almost entirely and the first one partially: they can generate fluent, responsive explanations and questions on almost any topic without hand-built content. That is why the current moment feels different. But they do nothing, by themselves, for the second and third obstacles. Indeed, a language model that will produce any answer on request makes gaming the system effortless. And they do not, by themselves, supply the student model or the sequencing policy that made the best older systems effective. The next three chapters explain what those components are and how they work, because without understanding them it is impossible to judge whether a new AI tutor is a tutor at all. The target, restated The lesson of the tutoring research can be compressed into a working definition that the rest of this book will use. A tutor, human or machine, is effective to the extent that it keeps the learner doing productive cognitive work at the edge of their current ability, catches and addresses errors close to where they arise, and sustains this over enough time for practice to accumulate into durable knowledge. Each clause implies a capability. To keep work at the edge of ability, a tutor must know where that edge is, which requires a model of what the student knows. To catch errors where they arise, it must engage with the student's steps, not just their final answers. To sustain practice over time, it must choose what comes next in a way that builds and consolidates knowledge, which requires a sequencing policy. And to keep the learner doing the work, it must resist the temptation, and the student's frequent request, to do the work itself. Bloom's two sigma was always more a direction than a destination. The question for AI in education is not whether it will reach two sigma, which human tutors themselves rarely do, but whether it can provide these capabilities to students who would otherwise have no tutor at all. That is a lower bar, and a far more important one. Chapter 2: Inferring the Invisible: Bayesian Knowledge Tracing Knowledge cannot be observed. A teacher who wants to know whether a student understands how to add fractions with unlike denominators has no direct access to the student's mind. What the teacher can observe is behavior: the student's answers to questions, the steps they write down, the time they take, the expression on their face. From that behavior the teacher infers a state of knowledge, and the inference is always uncertain. A correct answer might be a lucky guess. A wrong answer might be a careless slip by someone who knows the material perfectly well. Every adaptive learning system faces the same problem, with less evidence than a teacher has. It sees only the stream of responses a student produces, and from those it must decide what the student knows and therefore what the student should do next. The method that has done more than any other to shape how machines make this inference is called Bayesian knowledge tracing. It is worth understanding in some detail, because its assumptions are built into a great deal of educational software, and because its logic makes visible what any learner model has to get right. The original problem Bayesian knowledge tracing was introduced by Albert Corbett and John Anderson of Carnegie Mellon University in a paper titled "Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge," published in the journal User Modeling and User-Adapted Interaction in 1995 (the volume is dated 1994). They were working on the ACT Programming Tutor, a system that taught students to write code in the Lisp programming language. The tutor already used model tracing, described in Chapter 1, to follow each student's solution step by step and identify which underlying rules each step reflected. What it needed was a way to decide when a student had learned a given rule well enough to stop practicing it. This is a more consequential question than it sounds. Stop too early and the student moves on with fragile knowledge that will collapse under later demands. Stop too late and the student wastes time on problems they have already mastered, which is not merely inefficient but demotivating. A conventional class solves the problem crudely: everyone does the same number of problems, set by the textbook or the teacher, regardless of who has learned what. Corbett and Anderson wanted each student to practice each skill exactly as much as that student needed. The model in plain terms Bayesian knowledge tracing begins by breaking the subject into small units called skills, or in the language that later became standard, knowledge components. A knowledge component is a piece of knowledge that a student either has or lacks and that can be exercised in a problem step: knowing that the derivative of a constant is zero, knowing how to find a common denominator, knowing that a Lisp function call takes its arguments after the function name. Each problem step is tagged with the knowledge component or components it requires. The mapping from steps to knowledge components is usually built by subject experts and refined with data, and its quality matters enormously, a point taken up in Chapter 3. For each student and each knowledge component, the model maintains a single number: the probability that the student currently knows that component. It treats knowledge as binary. At any moment the student is either in the learned state or the unlearned state, and the model's job is to estimate which, given the evidence so far. Because the true state is hidden and only answers are visible, the model is a hidden Markov model, a standard structure in statistics in which an unobserved state changes over time and produces observable outputs. Four parameters govern the model for each knowledge component. 1. The prior probability of knowing, usually written P(L0). This is the probability that a student already knows the component before any practice in the system, perhaps from earlier instruction. 2. The learning probability, P(T), for transition. This is the probability that a student who does not yet know the component will learn it as a result of one practice opportunity. 3. The guess probability, P(G). This is the probability that a student who does not know the component will nonetheless answer correctly, by guessing, by partial knowledge, or by some shortcut. 4. The slip probability, P(S). This is the probability that a student who does know the component will answer incorrectly anyway, through carelessness, a typing error, or distraction. The standard model makes one more assumption: once learned, a component is never forgotten. A student can move from unlearned to learned, but not back. This assumption is unrealistic, and much later work relaxes it, but it made the model tractable and is less damaging than it sounds over the short time scales of a single unit of instruction. Watching the model reason The model's reasoning is easiest to see in a worked example. Suppose a component has the following parameters, which are typical in magnitude of those found in real tutoring data: a prior of 0.3, a learning probability of 0.1, a guess probability of 0.2, and a slip probability of 0.1. Before the student has answered anything, the system believes there is a 30 percent chance they already know the skill. The student attempts the first problem and answers correctly. The model now asks: given a correct answer, how likely is it that the student knew the skill? It uses Bayes' theorem. If the student knew the skill, a correct answer would occur with probability 0.9, one minus the slip probability. If they did not, it would occur with probability 0.2, the guess probability. Weighting each by the prior, the probability of "knew it and answered correctly" is 0.3 times 0.9, or 0.27. The probability of "did not know it but guessed" is 0.7 times 0.2, or 0.14. Of all the ways a correct answer could have happened, the share that involves knowing is 0.27 divided by 0.41, which is about 0.66. A single correct answer has raised the estimate from 30 percent to 66 percent. The model then accounts for learning. The practice opportunity itself might have taught the student something. If they did not know the skill before, there is a 10 percent chance they learned it now. So the probability of knowing after this opportunity is 0.66 plus 0.1 times the remaining 0.34, which comes to about 0.69. The student answers the second problem correctly too. Repeating the same reasoning, the estimate rises to about 0.91 after the evidence and about 0.92 after allowing for learning. A third correct answer pushes it to about 0.98. Corbett and Anderson used a mastery threshold of 0.95: once the probability of knowing a component exceeded 95 percent, the tutor stopped presenting problems that required it. In this example, three correct answers in a row would be enough. Now consider a different student who answers the first problem correctly but the second incorrectly. After the first answer the estimate stands at about 0.69, as before. A wrong answer could come from a student who knows the skill and slipped, with probability 0.69 times 0.1, about 0.07, or from a student who does not know it, with probability 0.31 times 0.8, about 0.25. The share involving knowing is about 0.22. After allowing for learning, the estimate is about 0.30, back almost exactly where it started. The model has concluded, reasonably, that the first correct answer was probably a guess. This simple machinery has several attractive properties. It is interpretable: each parameter has a plain meaning that a teacher could understand. It is cheap to compute: updating the estimate requires a few multiplications, which mattered a great deal on 1990s hardware and still matters for systems serving millions of students. It handles noise gracefully, neither leaping to conclusions after one answer nor ignoring evidence. And it supports a natural decision rule for mastery learning, the instructional approach Bloom had championed, in which students move on only when they have demonstrated command of the current material. Choosing the threshold The 95 percent threshold looks like a technical detail, but it is a pedagogical decision with real consequences, and it is worth seeing why. Raise the threshold to 99 percent and, with the parameters in the example, a student needs at least four correct answers in a row rather than three, and more if any errors intervene. Lower it to 90 percent and two correct answers may suffice. Across a course with a hundred knowledge components, the difference between these settings amounts to hundreds of problems per student. A higher threshold guards against a specific kind of error: declaring mastery for a student who was actually guessing. A lower one guards against the opposite error: holding back a student who knew the skill all along. Which error is worse depends on the material. For a foundational skill on which much later learning depends, such as place value in arithmetic or the distributive property in algebra, premature advancement is costly, because the gap will resurface repeatedly and in confusing forms. For a skill that is relatively self-contained and will be revisited anyway, over-practice is the bigger waste. A thoughtful designer might set different thresholds for different components, and some systems do. The threshold also interacts with the guess parameter. When a problem format makes guessing easy, such as multiple choice with three options, each correct answer carries less information, and the model will require more of them before declaring mastery. This is the model's way of saying, correctly, that a multiple-choice question is weaker evidence than a problem that requires the student to produce an answer. Designers who want students to reach mastery efficiently therefore have a reason, internal to the model, to prefer questions that are hard to guess. Finally, the threshold depends on the model's estimates being calibrated: when the model says 95 percent, students in that state should in fact know the skill about 95 percent of the time. If the model is systematically overconfident, the real mastery rate at the threshold will be lower than intended. Calibration can be checked against later performance, for instance by testing students on a skill some weeks after they were declared to have mastered it, but few commercial systems publish such checks. Where the parameters come from The four parameters are not set by hand in serious use. They are estimated from data: the response histories of many students who have worked through the same material. The standard method is an algorithm called expectation maximization, which iteratively searches for the parameter values that make the observed sequences of right and wrong answers most probable. Other methods, including a brute-force search over a grid of candidate values, are also common. Fitted parameters often tell designers something useful about their material. A component with a very high guess probability may be one where the problem format gives the answer away, for instance a multiple-choice item with implausible distractors. A component with a very low learning probability may be one where the instruction is not working, or where the component as defined actually bundles several different skills that are learned at different times. A component with a high slip probability may involve a procedure that is error-prone even for experts, such as long arithmetic. Fitting the model to data thus doubles as a diagnostic of the curriculum. Knowledge components and the skill map The worked example assumed that each problem exercises exactly one, well-defined knowledge component. In practice, deciding what the components are is one of the hardest parts of building a learner model, and errors here undermine everything downstream. Consider a problem that asks a student to solve 3x + 7 = 22. To a novice designer this might look like one skill, "solving linear equations." But a student might get it wrong because they do not know to subtract 7 from both sides, because they subtract correctly but then multiply by 3 instead of dividing, or because they make an arithmetic error computing 22 minus 7. These are different pieces of knowledge, learned at different rates, and a model that lumps them together will produce muddled estimates. If a student has mastered the inverse-operation idea but keeps making subtraction errors, a single "linear equations" component will hover at an uncertain middle value and the system will assign more of the same problems, which address neither the mastered skill nor the weak one in particular. Researchers formalize the mapping from problems to components as a matrix, often called a Q-matrix, with a row for each problem step and a column for each knowledge component, and a mark wherever a step requires a component. Kenneth Koedinger and colleagues at Carnegie Mellon developed methods to evaluate and improve such maps with data. One telling signal is the learning curve: if a component is well defined, students' error rates on it should fall smoothly with practice. If the curve is jagged or flat, the component probably bundles distinct pieces of knowledge, and splitting it often produces both a better-fitting model and better instruction. Their technique for searching systematically over possible refinements, Learning Factors Analysis, introduced by Hao Cen, Koedinger and Brian Junker in 2006, became a standard tool. The public repository DataShop, hosted by Carnegie Mellon's LearnLab, made thousands of such datasets available, which allowed researchers to compare models on shared data. Koedinger, Corbett and Charles Perfetti later set out the theory behind this approach in their Knowledge-Learning-Instruction framework, published in Cognitive Science in 2012. Its central claim is that instructional decisions should depend on the kind of knowledge component being learned. Facts, such as the capital of a country, are best acquired through retrieval practice and spacing. Procedures, such as solving an equation, benefit from worked examples followed by practice with feedback. Principles and concepts, such as why an equation can be manipulated as it can, may require explanation, comparison of cases and self-explanation. A learner model that knows only that a component is "mastered" or "not mastered," without knowing what kind of thing it is, will make poor choices about how to teach it. What knowledge tracing gets right It would be easy to dismiss Bayesian knowledge tracing as a crude relic, and Chapter 3 will lay out its weaknesses frankly. But its core insight is correct and is easy to lose sight of in the current enthusiasm for conversational AI. The insight is that knowledge is a latent quantity that must be inferred, with explicit uncertainty, from a stream of noisy evidence, and that the inference must be kept per skill rather than as a global impression. A system that does not do this cannot know when to stop practice, when to advance, or when to return to old material. It can respond to what a student says in the moment, but it cannot plan. This matters directly for language-model tutors. A large language model in conversation with a student has, within the conversation, a rich and immediate sense of what the student just said. But it has no persistent, calibrated estimate of the student's knowledge across the dozens of components in a course, and without special engineering it forgets everything when the session ends. It cannot tell whether a correct answer today reflects mastery or a lucky guess, because it is not counting. It does not know that this student has answered three questions on common denominators correctly over the past week and failed two on equivalent fractions, because it has not been told. Bayesian knowledge tracing, for all its simplifications, does exactly that bookkeeping. The bookkeeping is also what allows a system to be accountable. When a knowledge-tracing tutor declares a student to have mastered a component, a teacher can ask why and receive an answer: these four responses, these parameter values, this threshold. The reasoning can be inspected, criticized and changed. That property, which deep learning approaches often sacrifice, turns out to matter for teachers' trust and for fairness, and later chapters return to it. The assumptions to keep in mind Before turning to the model's limits, it is worth listing its assumptions plainly, since each one is the starting point for a line of later research. Knowledge of each component is binary: learned or not, with nothing in between. The parameters are the same for every student, so a quick learner and a slow one are modeled identically. Components are independent: learning one says nothing about the likelihood of knowing another, even when they are closely related. Nothing is forgotten. Each response depends only on whether the component is known, not on how long the student took, whether they asked for a hint, or what time of day it is. And the map of components is fixed and correct. None of these assumptions is true. The question that occupied the field for the next twenty-five years was which of them matter most, and what it costs to relax them. Chapter 3: Beyond Four Parameters: How Learner Models Grew Up Bayesian knowledge tracing works well enough to have run inside commercial tutors for three decades, but anyone who has fitted it to real data quickly meets its peculiarities. Some of these are technical curiosities. Others go to the heart of what it means for a machine to claim it knows what a student knows. This chapter follows the main lines along which learner models have developed since 1995: the repair of knowledge tracing's own weaknesses, the import of models from psychometrics, and the arrival of deep learning. The thread that runs through all of them is a tension between predicting student answers accurately and representing student knowledge meaningfully. The two are not the same, and for teaching purposes the second matters more. When the parameters lie The first problem researchers found was that knowledge tracing's parameters are not always identifiable. In a 2007 paper, Joseph Beck and Kai-min Chang showed that very different combinations of parameter values can fit the same student data almost equally well. A set of parameters in which students start with low knowledge and learn quickly can produce nearly the same predicted sequence of correct and incorrect answers as one in which students start with moderate knowledge and learn slowly. If the predictions are the same, why does it matter? It matters because the parameters drive decisions. The two models will disagree about when a particular student has reached mastery, and therefore about how many problems that student should do. A model chosen purely because it predicts well might assign practice in a way that makes no pedagogical sense. Worse, fitting algorithms sometimes produce degenerate parameters that have no sensible interpretation at all. A model might conclude that the guess probability exceeds 0.5, implying that students who do not know a skill are more likely to answer correctly than not, or that the slip probability is so high that knowing a skill barely improves performance. In such a model, a correct answer can paradoxically lower the estimated probability that the student knows the skill. Ryan Baker, Albert Corbett and Vincent Aleven documented these degenerate fits in 2008 and showed that they occurred in real tutor datasets. The remedies are partly mathematical, such as constraining parameters to plausible ranges or placing prior distributions on them, and partly a reminder that a model's fit to data is not a guarantee of its meaning. Making the model fit the student The next problem is that standard knowledge tracing treats every student the same. The guess, slip, learning and prior parameters are estimated for each knowledge component, but shared across all learners. Teachers know that this is false: some students learn new procedures after one or two examples and others need ten. Zachary Pardos and Neil Heffernan, working with data from the ASSISTments platform at Worcester Polytechnic Institute, introduced ways of individualizing the prior in 2010, letting each student start with their own initial probability of knowing a skill. Michael Yudelson, Kenneth Koedinger and Geoffrey Gordon went further in 2013, separating each parameter into a component that depends on the skill and one that depends on the student. Their analysis found that individualizing the learning rate improved predictions considerably more than individualizing the prior. In other words, students differ more in how quickly they learn than in what they already know when they arrive, a finding with real implications for how adaptive systems should pace instruction. Baker, Corbett and Aleven's contextual guess-and-slip model, from the same 2008 work, took a different route. Rather than treating guess and slip as constants, it estimated them for each individual response from features of the context: how long the student took, whether they had just asked for help, how many attempts they had made. A correct answer delivered in two seconds immediately after a hint request is more likely to be a guess than one delivered after thirty seconds of unassisted work. This made the model more sensitive to the behavioral evidence that teachers use instinctively. Other extensions tackled the remaining assumptions. Adding a forgetting parameter, which allows a student to move from the learned back to the unlearned state, makes the model more realistic over long time scales. Allowing hints and instruction to have their own learning effects lets the model distinguish practice from teaching. And dynamic Bayesian networks, which link knowledge components to one another, let evidence about one skill inform estimates of related ones, so that mastering multi-digit subtraction raises the system's confidence about single-digit subtraction. Each extension improves the model's fit somewhat. None of them dramatically changes the picture, and each adds parameters that must be estimated from data, which makes the identifiability problem worse. By the mid-2010s many researchers had concluded that knowledge tracing was close to the limit of what its basic structure could do. Borrowing from psychometrics A different tradition of student modeling had been running in parallel for even longer. Psychometrics, the science of educational and psychological measurement, developed item response theory in the mid-twentieth century to make standardized tests fairer and more precise. In its simplest form, the Rasch model, each student has an ability and each test item has a difficulty, both placed on the same scale, and the probability of a correct answer depends on the difference between them through a logistic function. A student whose ability equals an item's difficulty has a 50 percent chance of answering correctly; one well above it is very likely to succeed. Item response theory assumes ability is fixed during a test, which is reasonable for a one-hour exam but not for a learning system, where the whole point is that ability changes. Researchers therefore developed logistic models that add terms for practice. Additive factors models, used in Learning Factors Analysis, predict the probability of a correct answer from a student's general ability, the difficulty of each knowledge component involved, and the number of previous practice opportunities on each component, with each component having its own learning rate. Performance factors analysis, introduced by Philip Pavlik, Hao Cen and Kenneth Koedinger in 2009, refined this by counting prior successes and prior failures separately, on the reasonable grounds that a correct practice attempt and an incorrect one are not equally informative about future performance. A strikingly simple member of this family is the Elo rating system, invented by the physicist Arpad Elo to rank chess players. After each game, the winner's rating rises and the loser's falls by an amount proportional to how surprising the result was. Applied to education, the student and the item play a game: a correct answer is a win for the student. Radek Pelánek, whose 2016 paper in Computers & Education surveyed applications of Elo in adaptive educational systems, showed that this approach, which requires no batch fitting and updates in real time, often predicts student performance about as well as more elaborate models. Several large practice platforms use variants of it, particularly for vocabulary, geography and other domains where items are numerous and each tests a small piece of knowledge. The logistic models give up knowledge tracing's crisp hidden state, the idea that a student either knows a skill or does not. In exchange they handle multiple knowledge components per item naturally, they are less prone to degenerate fits, and their estimates of difficulty and learning rate are often easy to interpret. Pelánek's 2017 review of the field in User Modeling and User-Adapted Interaction, titled "Bayesian Knowledge Tracing, Logistic Models, and Beyond," concluded that the differences in predictive accuracy among these families are generally small and that practical considerations, such as the structure of the domain and the decisions the model must support, should drive the choice. Deep knowledge tracing and the accuracy race In 2015 Chris Piech and colleagues at Stanford University published "Deep Knowledge Tracing" at the NeurIPS conference. They trained a recurrent neural network, a type of model designed for sequences, on the raw histories of student answers. The network took each response as input, along with an indication of which skill it exercised, and learned to predict whether the student would answer the next question correctly. It had no explicit parameters for guessing, slipping or learning. It simply learned whatever patterns in the data were predictive. The initial results appeared to show a dramatic improvement in predictive accuracy over Bayesian knowledge tracing, and the paper attracted wide attention. It seemed to herald the replacement of handcrafted cognitive models with general-purpose learning machines, much as deep learning was then replacing handcrafted features in image and speech recognition. The response from the educational data mining community was swift and instructive. Researchers found that part of the reported gain came from issues in the benchmark data, including duplicated records. Mohammad Khajah, Robert Lindsey and Michael Mozer, in a 2016 paper pointedly titled "How Deep Is Knowledge Tracing?", showed that extending Bayesian knowledge tracing with a few well-motivated features, such as forgetting, individual student abilities and interactions between skills, closed most of the gap. Their interpretation was that the neural network was succeeding largely by capturing these known regularities implicitly, rather than discovering something fundamentally new. The deep learning line nonetheless continued, and the models grew more sophisticated. Attention-based models such as SAKT, introduced by Shalini Pandey and George Karypis in 2019, and AKT, introduced by Aritra Ghosh, Neil Heffernan and Andrew Lan in 2020, borrowed the transformer architecture that underlies modern language models and allowed predictions to draw on any earlier response, weighted by relevance. On the largest datasets, which run to hundreds of millions of responses from commercial platforms, such models do predict better than the classical alternatives. A careful comparison by Théophile Gervet, Ken Koedinger, Jeff Schneider and Tom Mitchell, published in the Journal of Educational Data Mining in 2020, clarified when deep learning pays. On datasets with many interactions per student and per item, deep models had an edge. On smaller datasets, typical of a single course or a new product, well-specified logistic regression models were as accurate or better. The authors also noted that deep models' advantage often came from exploiting details of how data were logged, such as the order in which items appeared, rather than from modeling knowledge. Table 1 summarizes how the main families compare on the properties that matter for teaching rather than for prediction contests. Table 1. Main families of learner models compared on teaching-relevant properties. Model family What it represents Interpretability Data needed Typical use Bayesian knowledge tracing Probability each skill is learned High; four named parameters per skill Modest Mastery decisions in step-based tutors Logistic models (AFM, PFA) Ability, item difficulty, learning per skill High; coefficients readable Modest Curriculum diagnosis, mastery decisions Elo-style ratings Running ability and difficulty scores Moderate Low; updates live Large item banks, practice apps Deep knowledge tracing and attention models Hidden vector summarizing history Low Large Next-answer prediction on big platforms Prediction is not understanding The race for predictive accuracy taught the field an important lesson, one that bears directly on the use of large language models in tutoring. A model that predicts whether a student will answer the next question correctly is not necessarily a model of what the student knows. Consider what a learner model is for. Its job is to support decisions: whether to give this student another problem on this skill, whether to move on, whether to review something old, whether to alert the teacher. A deep network might predict, with high accuracy, that a student will get the next item right, because it has learned that students who just got two items right in a row usually get the third right. That prediction is useful for a leaderboard but nearly useless for teaching, because it says nothing about which skill the student has or lacks, and it cannot answer the question "what should this student practice?" Several research groups have tried to recover interpretable skill estimates from deep models, with partial success. But the more durable conclusion is that the learner model should be designed around the decisions it supports. If the decision is when to stop practicing a skill, the model must produce a calibrated estimate of mastery for that skill. If the decision is how to explain something to a teacher, the model's reasoning must be expressible in terms a teacher understands. Accuracy at predicting the next click is a means to these ends, not an end in itself. There is also a subtler problem, which Shayan Doroudi and Emma Brunskill analyzed in several papers. Learner models are fitted to data generated by students working through a particular system with a particular sequencing policy. Change the policy, for instance by stopping practice earlier, and the data the model sees change too. A model that predicts well under one policy may predict poorly under another. A model that is wrong in its structure, rather than merely imprecise in its parameters, can therefore produce mastery decisions that are systematically off without anyone noticing from the fit statistics. Doroudi and Brunskill went on to show, in a 2019 paper, that such misspecification does not affect all students equally, a point Chapter 8 returns to. Where language models fit Large language models enter this story in two ways. The first is as a new and powerful source of evidence. Classical learner models consume right and wrong answers, sometimes with timing and hint use. A language model can read a student's free-text explanation, a worked solution with a mistake in the middle, or a confused question, and judge what it reveals about the student's understanding. This makes it possible, in principle, to trace knowledge in domains such as writing, history or scientific reasoning, where answers are not simply right or wrong and where classical tutors never worked well. Research systems that use language models to grade short answers, tag them with knowledge components, or identify specific misconceptions are now common, though their reliability varies and is often not reported in ways that allow comparison. The second way is as a candidate learner model in itself. A language model given a student's history in its context can be asked whether the student understands a concept, and it will produce a confident answer. But that answer comes without calibrated uncertainty, cannot easily be inspected, and may be swayed by surface features of the transcript, such as a student's fluency in English. The lessons of the deep knowledge tracing episode apply with greater force: an impressive-sounding judgment is not the same as a reliable estimate, and a system that cannot say why it believes a student has mastered something cannot be held to account when it is wrong. The practical conclusion, which a number of serious developers have reached, is to combine the two. Language models are used to interpret rich student input and convert it into structured evidence about specific knowledge components. A classical learner model, whether knowledge tracing or a logistic model, accumulates that evidence over time and makes mastery estimates that can be inspected and audited. The language model reads the student; the learner model remembers what it has read and keeps score. That division of labor plays to the strengths of each. Chapter 4: Paths Through a Curriculum A learner model answers the question "what does this student know?" A curriculum answers a different question: "what should this student do next?" In a conventional classroom the second question is answered once, for everyone, by the order of chapters in a textbook and the pace of a teacher's plan. Personalized learning promises to answer it separately for each student, every few minutes, in light of what the learner model currently believes. That promise sounds simple and turns out to be hard, for reasons that have little to do with computing power. Choosing well requires knowing how the pieces of a subject depend on one another, how long practice should continue on a skill before moving on, when previously learned material should return, and how to balance a student's immediate success against their long-term retention. Each of these is a question cognitive science has studied for decades, and the answers are not always intuitive. A personalized curriculum built on bad answers is worse than a well-designed fixed one, because it delivers bad sequencing with the authority of an algorithm. Maps of prerequisite knowledge The most basic structure in any curriculum is prerequisite order. A student cannot sensibly practice adding fractions with unlike denominators before understanding what a fraction is, or solve two-step equations before solving one-step ones. Every adaptive system therefore needs some representation of which knowledge components depend on which. The most mathematically developed approach to this problem is knowledge space theory, introduced by Jean-Paul Doignon and Jean-Claude Falmagne in 1985. They observed that if a subject has, say, a hundred distinct items, there are in principle an astronomical number of possible combinations a student might know. But prerequisite relations rule most of these out. A student who can solve quadratic equations by factoring will almost certainly be able to multiply binomials. The set of combinations that are actually feasible, given all the prerequisite relations, is called a knowledge structure, and it is vastly smaller than the set of all combinations. This structure has a practical payoff. Because each question rules out many possible states at once, an adaptive assessment can locate a student within the structure with a surprisingly small number of questions. Once located, the student's outer fringe, the items they are ready to learn because all prerequisites are in place, defines what they should study next. Knowledge space theory became the basis for ALEKS, an adaptive system developed at the University of California, Irvine, with funding from the National Science Foundation, and later acquired by McGraw-Hill. ALEKS is widely used in American schools and universities, particularly for mathematics placement and remediation, and its "ready to learn" display is a direct expression of the outer-fringe idea. Most other adaptive systems use simpler prerequisite graphs, built by subject experts and sometimes refined with data. Khan Academy's mastery system, for example, organizes skills into units and courses with recommended prerequisites. The quality of these graphs varies widely. A graph that is too coarse lets students advance with gaps; one that is too strict traps them in remediation, repeatedly sending them back to foundations they do not need. Data can reveal errors in a hand-built graph, for instance when students who skip a supposed prerequisite perform just as well, but correcting the graph requires a willingness to let some students take unusual paths, and many systems never collect that evidence. When to stop practicing Once a student is working on a skill, the next decision is when to stop. Bayesian knowledge tracing's mastery threshold, described in Chapter 2, offers one answer: stop when the probability of knowing exceeds 95 percent. Simpler systems use heuristic rules, such as "three correct in a row," which is the rule ASSISTments used in its skill-builder assignments and which approximates the knowledge-tracing decision surprisingly well for typical parameter values. The cost of getting this wrong runs in both directions. Studies with the Cognitive Tutor found that knowledge-tracing-based mastery decisions saved students substantial time compared with fixed problem sets, because students who learned quickly were released early. Hao Cen, Kenneth Koedinger and Brian Junker reported in 2007 that refining the learner model in a geometry tutor could have reduced practice time considerably without reducing learning, simply by identifying skills students had already mastered. Over a year, the time saved compounds into weeks. The opposite failure has its own name. Joseph Beck and Yue Gong, in a 2013 paper, called it wheel-spinning: a student practices a skill repeatedly without making progress toward mastery. They found, in data from Cognitive Tutor and ASSISTments, that a substantial share of students who had not mastered a skill within roughly ten attempts were unlikely to master it at all through further practice of the same kind. These students were not lazy; many were trying hard. They needed something other than more of the same problem: a different explanation, a return to a prerequisite, or a human. A system that only knows how to assign another problem will keep them spinning, and the experience of repeated failure carries its own motivational damage. Detecting wheel-spinning early and changing course is one of the clearest places where a learner model can make a practical difference, and one where a language model's ability to change its explanation might complement a classical system's ability to notice the problem. Spacing, retrieval and interleaving Mastery learning addresses the immediate question of whether a student can perform a skill. Cognitive psychology adds a harder question: will they still be able to perform it next month? Here three findings, each replicated many times over, should shape any personalized curriculum. The first is the spacing effect. Practice distributed over time produces more durable memory than the same amount of practice massed together. Hermann Ebbinghaus observed this in his experiments on memory in the 1880s, and a large meta-analysis by Nicholas Cepeda and colleagues, published in Psychological Bulletin in 2006, confirmed it across hundreds of studies. The optimal spacing depends on how long the material needs to be retained: the longer the retention interval, the longer the ideal gap between study sessions. The second is the testing effect, also called retrieval practice. Actively recalling information from memory strengthens it far more than rereading or restudying it. Henry Roediger and Jeffrey Karpicke's experiments in the mid-2000s made this finding widely known, and it has been confirmed in classrooms as well as laboratories. A quiz is not just a measurement; it is a learning event. The third is interleaving. When students practice several types of problem mixed together, rather than in blocks of one type at a time, they perform worse during practice but better on later tests. Doug Rohrer and Kelli Taylor demonstrated this with mathematics problems in 2007, and subsequent classroom studies have reproduced it. The explanation is that blocked practice lets students apply a procedure without having to decide which procedure is needed, whereas interleaved practice forces them to identify the type of problem first, which is precisely the skill a test or real-world use demands. Robert Bjork coined the term "desirable difficulties" for conditions such as these: they make learning feel harder and slower in the moment while making it stronger in the long run. The term captures a central difficulty for adaptive systems. A system optimized to maximize correct answers during practice, or to keep students feeling successful, will tend to avoid desirable difficulties. It will block practice, mass repetition, and present material just before it is needed, because those choices produce high immediate performance. A system optimized for long-term learning has to accept lower immediate performance, which looks worse on the dashboard and can feel worse to the student. Spaced repetition software is the most direct application of these findings. Systems descended from the Leitner box, a paper flashcard method popularized by Sebastian Leitner in the 1970s, schedule each item for review at increasing intervals as long as the learner keeps recalling it, and reset it to short intervals after a failure. Piotr Woźniak's SuperMemo algorithms, starting in the late 1980s, made the intervals adapt to each item's estimated difficulty. Duolingo published one of the best-documented modern versions in 2016: Burr Settles and Brendan Meeder's half-life regression model, presented at the Association for Computational Linguistics conference, estimates for each learner and each word how quickly the memory decays, and schedules practice when the predicted probability of recall drops toward a threshold. They reported that the model predicted recall better than earlier heuristics and that its deployment increased learner engagement in a controlled experiment. Choosing among strategies These findings combine into a handful of distinct strategies for deciding what a student does next, summarized in Table 2. Real systems usually blend several. Table 2. Strategies for sequencing practice in adaptive systems. Strategy Decision it makes Evidence base Main risk Prerequisite mapping Which skills a student is ready to learn Knowledge space theory; expert graphs Rigid maps trap or skip students Mastery thresholds When to stop practicing a skill Corbett and Anderson; Cognitive Tutor studies Wheel-spinning for students who are stuck Spaced retrieval When to revisit learned material Spacing and testing effects, widely replicated Reviews crowd out new learning if overused Interleaving How to mix problem types Rohrer and Taylor; classroom replications Lower practice scores discourage students Reinforcement learning Whole sequence, optimized for an outcome Mixed; strongest where models are well grounded Optimizes the wrong objective or the wrong model One student's week It helps to see how these strategies combine for a single student. Imagine a twelve-year-old working through a unit on ratios in a well-designed adaptive system. On Monday the system's prerequisite map shows that she has mastered multiplication and division facts and simple fractions, so ratios are on her outer fringe. She begins with two worked examples of equivalent ratios and then attempts problems herself. Her first two answers are correct, her third is wrong in a way that suggests she is adding rather than multiplying to scale a ratio, and the learner model's estimate for that component drops. The system gives her a targeted hint and two more problems that turn on the same distinction. By the end of the session she has crossed the mastery threshold for equivalent ratios but not yet for unit rates. On Tuesday, before continuing with unit rates, the system gives her three short retrieval questions: one on equivalent ratios from yesterday and two on fraction operations she mastered three weeks ago, which the scheduler judges to be at risk of fading. She gets the fraction questions right and one ratio question wrong. That error does not send her back to the beginning; it lowers the ratio estimate enough that a ratio problem is scheduled again for Thursday. On Wednesday she struggles with unit rates. After eight attempts with little progress, the system's wheel-spinning detector flags her. Rather than serving a ninth problem, it offers a short explanation using a different representation, a double number line, and alerts her teacher, who stops by during class and spends five minutes working through one problem with her. She then solves three in a row. On Thursday her practice set mixes ratio, unit-rate and fraction problems in an interleaved order. Her accuracy is lower than it was on Monday's blocked set, which the system expects. On Friday the class takes a short quiz on paper, and she gets the ratio and unit-rate questions right. Nothing in this week is exotic. Each step applies a well-supported finding. Yet a system that does all of it consistently, for every student, is rarer than the marketing of adaptive learning suggests, because each step requires a component that is expensive to build and easy to omit: a sound prerequisite map, a calibrated learner model, a review scheduler, a stuck-student detector, and a channel to the teacher. Can a machine learn to teach? The most ambitious approach to sequencing treats it as an optimization problem. Each decision about what to present next is an action; the student's knowledge is the state; and learning, measured somehow, is the reward. Reinforcement learning, the branch of machine learning that learns policies for sequential decisions, can in principle find the sequence of actions that maximizes long-term reward for each student. The same family of methods learned to play Go and to control robots. Could it learn to teach? Shayan Doroudi, Vincent Aleven and Emma Brunskill reviewed four decades of attempts in a 2019 paper in the International Journal of Artificial Intelligence in Education, titled "Where's the Reward? A Review of Reinforcement Learning for Instructional Sequencing." They found that reinforcement-learning policies outperformed baseline policies in only around half of the studies that compared them, and that the successes clustered in particular conditions. Policies did best when they were built on learner models grounded in cognitive psychology, such as models of memory decay for vocabulary learning, and when they were compared against weak baselines. They did worst when they relied on generic statistical models of the student, or when compared against strong, theory-driven baselines designed by experts. Their interpretation is worth dwelling on. A reinforcement-learning policy is only as good as the model of the student it optimizes against. If that model is wrong, the policy will exploit its errors, confidently selecting sequences that look optimal in simulation and fail with real students. And the reward must be chosen carefully. Optimizing for correct answers during practice will rediscover the undesirable shortcuts that desirable difficulties warn against. Optimizing for time spent will produce engagement, not learning. The only defensible reward is some measure of durable learning, which is slow and expensive to observe. Simpler forms of experimentation fare better. Multi-armed bandit methods, which allocate more students to whichever of several options appears to be working best while continuing to explore the others, have been used to test alternative explanations, hints and problem types in live systems. They do not attempt to plan a whole curriculum; they make one decision at a time with clear evidence. Platforms with large user bases, including ASSISTments through its E-TRIALS infrastructure, have made this kind of embedded experimentation a routine part of improving content. Personalization's quiet trade-offs There is a final point about curriculum paths that the vocabulary of personalization tends to obscure. A personalized path is also a separated one. When thirty students in a class work through software at their own pace, they quickly end up in thirty different places. That is the intended result, and for practice it is often appropriate. But it means students no longer share a common experience of the material. Class discussion of a topic becomes difficult when half the class reached it last week and half will not reach it until next month. Teachers who use adaptive software extensively often report spending much of their time managing this dispersion. There is also a risk that personalization becomes tracking by another name. A system that measures a student as behind and assigns easier material, slower pacing and more remediation may keep that student behind indefinitely, since they never encounter the grade-level content on which they will eventually be assessed. TNTP's 2018 report The Opportunity Myth, which observed classrooms in five school systems, found that students spent most of their time on assignments that were not at the level of their grade, and that students who were given grade-appropriate work with support made more progress. The report did not study adaptive software specifically, but its warning applies directly: a curriculum path that keeps a student permanently comfortable can quietly lower the ceiling on what they will learn. Good systems mitigate this by setting floors as well as ceilings, by exposing all students to core grade-level material with scaffolding rather than substituting easier material, and by giving teachers visibility into where each student is and the authority to override the path. These are design choices, not features of the algorithm, and they are where the values of a system become concrete. The upshot for anyone evaluating an adaptive or AI-driven product is a set of plain questions. What map of knowledge does the system use, and who built it? What rule decides when a student has mastered something, and what happens if they never do? Does the system bring back old material, and on what schedule? Does it ever deliberately make practice harder in the short run? What exactly is it optimizing? A vendor who cannot answer these questions is selling a content library with a recommendation engine, not a curriculum. Chapter 5: What the Evidence Showed Before the Chatbots When ChatGPT was released in November 2022, a common reaction among commentators was that education was about to experience, for the first time, the effects of artificial intelligence. That was not quite right. Intelligent tutoring systems and adaptive practice platforms had by then been used by millions of students for two decades, and they had been studied with unusual rigor for an educational technology, including in large randomized trials. That body of evidence is the best guide we have to what happens when machine tutoring leaves the laboratory and meets real schools, and its lessons apply with little modification to tutors built on language models. The short version is that adaptive tutoring works, sometimes impressively, but the effects shrink as studies become larger, longer and more realistic, and they depend heavily on how much students actually use the software and how teachers integrate it. The technology was never the only variable, and often not the most important one. The meta-analyses Several meta-analyses, each combining dozens of studies, have tried to estimate the overall effect of intelligent tutoring systems. They do not agree precisely, and the reasons for their disagreement are instructive. VanLehn's 2011 review, discussed in Chapter 1, found that step-based tutoring systems produced effects of about 0.76 standard deviations compared with no tutoring, nearly matching human tutors. James Kulik and J. D. Fletcher, in a 2016 review in the Review of Educational Research, examined fifty controlled evaluations and found a median effect of 0.66 standard deviations, a large effect by any standard. Wenting Ma, Olusola Adesope, John Nesbit and Qing Liu, in a 2014 meta-analysis in the Journal of Educational Psychology, found that intelligent tutoring systems outperformed teacher-led large-group instruction with an average effect of about 0.41 standard deviations. Saiying Steenbergen-Hu and Harris Cooper, however, in a 2013 meta-analysis also published in the Journal of Educational Psychology, focused on intelligent tutoring for mathematics in kindergarten through twelfth grade and found an overall effect close to zero. Their companion review of college-level studies, published the following year, found moderate positive effects. Why such different numbers? Kulik and Fletcher's analysis supplies much of the answer. They found that effects were far larger in studies that used tests developed by the researchers, closely aligned with what the tutor taught, than in studies that used standardized tests. They were larger in short studies than in long ones, and larger in small studies than in large ones. These are familiar patterns across education research, and they reflect something real: a tutor that teaches a specific procedure will show large gains on a test of that procedure, and smaller gains on a broad standardized test where that procedure is one item among many. They also reflect the fact that small, short studies run by enthusiastic developers tend to have more careful implementation than large, long studies run in ordinary schools. The practical reading is that intelligent tutors can produce large learning gains on the specific material they teach, but that translating those gains into broad improvements in measured achievement over a school year is much harder and yields much smaller effects. The big field trials The most convincing evidence comes from a handful of large randomized trials run in ordinary schools over a full year or more. The largest was a RAND Corporation study of Carnegie Learning's Cognitive Tutor Algebra I curriculum, published by John Pane, Beth Ann Griffin, Daniel McCaffrey and Rita Karam in Educational Evaluation and Policy Analysis in 2014. It involved more than 140 middle and high schools in seven states, randomly assigned to use the curriculum, which combined the tutoring software with textbook materials and classroom teaching, or to continue with their existing algebra curriculum. In the first year, there was no significant effect on student achievement. In the second year, high school students in schools using the Cognitive Tutor scored about 0.2 standard deviations higher than those in comparison schools. The researchers' interpretation was that teachers and schools needed a year to learn to use the program effectively. An effect of 0.2 standard deviations in algebra, a gateway course with high failure rates, is a meaningful result. It is also a long way from the effects in the meta-analyses. A second major trial, by Jeremy Roschelle, Mingyu Feng, Robert Murphy and Craig Mason, published in AERA Open in 2016, tested ASSISTments, a free web-based platform developed at Worcester Polytechnic Institute, used for seventh-grade mathematics homework in Maine. ASSISTments is a modest system by the standards of intelligent tutoring. It gives immediate feedback on homework problems, offers hints, and provides teachers with reports showing which problems students struggled with so that the next class can address them. In schools randomly assigned to use it, students scored about 0.18 standard deviations higher on a standardized mathematics test at the end of the year, with larger gains for students who had entered with lower achievement. Part of the effect seemed to come not from the software's direct interaction with students but from the way teachers used its reports to change what they did in class. A third trial, from outside the United States, is especially relevant for the global debate. Karthik Muralidharan, Abhijeet Singh and Alejandro Ganimian evaluated Mindspark, an adaptive learning program developed in India, in an after-school setting for middle school students in Delhi. Their results, published in the American Economic Review in 2019, showed that students given access to Mindspark plus some group instruction for about four and a half months gained 0.37 standard deviations in mathematics and 0.23 in Hindi compared with a control group. The study's most important insight concerned the reason. Many students in the sample were working several grade levels below their nominal grade, and their classroom instruction, pitched at grade level, left them behind. Mindspark diagnosed where each student actually was and gave them material at that level. Personalization worked largely because the baseline was so badly mismatched to the students. Table 3 summarizes these studies alongside the main meta-analytic estimates, with an emphasis on the conditions that differentiate them. Table 3. Selected evidence on adaptive tutoring before large language models. Study Setting Outcome measure Effect (SD) VanLehn (2011), review Step-based tutors vs. no tutoring Mostly researcher tests About 0.76 Kulik and Fletcher (2016), review Fifty controlled evaluations Mixed Median 0.66 Pane et al. (2014), RAND trial Cognitive Tutor Algebra I, 140+ US schools Standardized algebra test About 0.2 in year two; none in year one Roschelle et al. (2016) ASSISTments homework, Maine Standardized math test About 0.18 Muralidharan et al. (2019) Mindspark after school, Delhi Independent math and Hindi tests 0.37 math; 0.23 Hindi Source: the cited publications; effect sizes as reported by their authors. The dosage problem Every one of these studies reports something that rarely appears in marketing materials: students used the software much less than intended, and the amount they used it strongly predicted how much they gained. This pattern is so consistent that it has become a central problem for the field. Laurence Holt, writing in Education Next in 2024, called it "the 5 percent problem." Reviewing published data from several widely used mathematics platforms, he observed that only a small fraction of students, on the order of 5 percent, used them at the level the developers recommended, and that the impressive gains reported for "students who used the program as intended" applied only to that fraction. The other 95 percent used it sporadically or not at all and showed little benefit. Worse, the students who used it as intended were disproportionately those who were already doing well, so the program's headline results reflected selection as much as learning. The dosage problem points to something the learner model and the curriculum path cannot fix on their own. Personalized learning systems are built on the premise that each student will do the work the system assigns. In a classroom, the teacher's presence, the structure of the day and the social setting of peers all push students to engage. Alone with a screen, many students, particularly those who are already disengaged, simply do not. A tutor that would deliver large gains if used for thirty minutes a day delivers nothing if used for five minutes a week. Justin Reich, in Failure to Disrupt: Why Technology Alone Can't Transform Education (2020), traces how this dynamic plays out across educational technology more broadly. He calls it the edtech Matthew effect, after the biblical line about the rich getting richer. New learning technologies, offered on an opt-in basis, are taken up disproportionately by students who are already advantaged, who have the motivation, the home support and the digital skills to use them well. A 2015 study by John Hansen and Reich in Science found exactly this pattern among the users of massive open online courses from Harvard and MIT, who were drawn disproportionately from more affluent neighborhoods. Unless deliberately designed and implemented to counteract it, technology tends to widen gaps rather than close them. Personalized learning at scale: the cautionary cases The 2010s also produced a series of ambitious attempts to rebuild entire schools around personalized learning, and their trajectories hold lessons that pure efficacy data cannot. AltSchool, founded in San Francisco in 2013 by a former Google executive, raised well over a hundred million dollars from prominent technology investors to build a network of small private schools in which each child would follow a personalized path supported by software and continuous data collection. Within a few years it was closing schools, and in 2019 it shifted away from running schools to selling its software, under the name Altitude Learning. The schools had proved expensive to run, the technology had absorbed more of teachers' time than it saved, and some parents felt their children had been experimental subjects. Summit Learning, a personalized learning platform developed by a California charter network with engineering support from Facebook, was adopted by hundreds of public schools across the United States in the second half of the 2010s. In 2018, students at schools in Brooklyn, New York, and McPherson, Kansas, staged walkouts and protests, complaining that they spent long hours alone in front of screens, that the curriculum was thin, and that they were isolated from teachers and peers. Some districts dropped the program. Supporters argued that implementation in those schools had been poor, which may be true, but the episode showed how quickly students and families can reject a model that replaces the human relationships of school rather than supporting them. These were not tutoring systems in the narrow sense, but they tested the broader idea that self-paced, data-driven software could become the core of schooling. RAND's studies of personalized learning schools funded by the Bill and Melinda Gates Foundation, published in 2015 and 2017, found modest positive effects on mathematics achievement on average, but with wide variation between schools, and noted that "personalized learning" meant very different things in different places. The schools that did best tended to combine technology with strong teaching rather than substitute one for the other. The pandemic test The school closures of 2020 and 2021 imposed an unplanned test of what happens when software and screens take over most of instruction. In much of the world, students spent months learning remotely, and adaptive practice platforms saw enormous growth in usage as schools turned to them to fill the gap. If self-paced personalized learning could substitute for classroom teaching, this was its moment. The results were sobering. In the United States, the National Assessment of Educational Progress, the country's most authoritative test of student achievement, recorded in 2022 the largest declines in mathematics scores in the history of the assessment, with steep drops in reading as well. Studies by researchers at Harvard, Stanford and elsewhere found that learning losses were largest in districts that stayed remote longest and in high-poverty schools, where students were less likely to have reliable devices, connections and adult support at home. Remote learning was not identical to adaptive tutoring, and the losses had many causes, including illness, disruption and stress. But the episode showed, at a scale no study could match, that access to digital learning tools does not substitute for the structure, supervision and relationships of school. The policy response is also telling. When the United States allocated roughly 190 billion dollars in emergency relief to schools, much of the effort to recover lost learning went not into software but into human tutoring, especially high-dosage tutoring of the kind evaluated in Chicago and described in Chapter 1. Many districts did buy adaptive platforms as well, but the consensus among researchers advising on recovery was that the strongest evidence lay with frequent, structured tutoring by trained people. That judgment reflected the evidence available at the time, before language-model tutors had been tested, and it is part of the background against which the newer tools must prove themselves. What carried over Looking across this evidence, several conclusions stand out, and each one bears directly on the arrival of language-model tutors. First, the technology works when students use it. The best step-based tutors, used as intended, produce gains comparable to human tutoring. Nothing about language models changes this basic finding; if anything, their fluency should make them more engaging. Second, getting students to use it is the hard problem. Motivation, routine, accountability and human presence are the binding constraints, not the sophistication of the software. Language models may help with engagement, since conversation is more inviting than a problem set, but they will not by themselves solve the problem of the student who does not log on. Third, effects shrink as studies become more realistic. Gains measured on closely aligned tests in short studies do not translate one-for-one into gains on broad achievement measures over a year. Early studies of language-model tutors, nearly all of which are short and use closely aligned tests, should be read with this in mind. Fourth, the teacher matters. The trials that worked, notably ASSISTments, worked in part because the software made teachers more effective, not because it replaced them. The ones that failed often tried to sideline the teacher. Fifth, personalization does most good where the mismatch between instruction and student is greatest. Mindspark's large effects came from students who were far behind and receiving instruction they could not follow. In classrooms where instruction is already reasonably well matched to students, the room for personalization to help is smaller. None of these conclusions is discouraging. Together they amount to a realistic picture of what educational technology can do: substantial good, under the right conditions, for students who engage with it. The question for the new generation of AI tutors is whether their capabilities change those conditions, or whether they meet the same constraints in new form. The next two chapters take up that question. Chapter 6: The Socratic Machine In Plato's dialogue Meno, Socrates sets out to show that learning is a kind of recollection. He calls over an enslaved boy who has never studied geometry and, using only questions, leads him to discover how to construct a square with twice the area of a given square. The boy makes mistakes along the way, first proposing to double the side, then trying a side one and a half times as long, and each time Socrates asks questions that let him see the error for himself. At the end the boy has found the answer, the diagonal of the original square, without, Socrates insists, having been told anything. The scene has been the emblem of a certain ideal of teaching for nearly two and a half thousand years: the teacher who does not lecture but asks, and the student who arrives at understanding through their own reasoning. It is also the ideal that the designers of today's AI tutors most often invoke. A large language model can ask questions as easily as it can answer them. If it can be made to ask the right ones, it seems to offer every student a Socrates of their own. This chapter examines what it takes to turn a language model into a Socratic tutor, why the default behavior of these systems works against it, and what the design choices look like in practice. The next chapter turns to the evidence on whether it works. What language models change The arrival of capable large language models changed the tutoring problem in several specific ways. The first is coverage. Earlier tutoring systems required hand-built content for every problem and every hint. A language model can discuss nearly any topic in school or university curricula at a reasonable level, generate new practice problems, produce explanations at different levels of sophistication, and respond to questions nobody anticipated. The construction cost that confined intelligent tutors to algebra, geometry and programming largely disappears. The second is open-ended input. Classical tutors could check a student's answer against a list of correct and known incorrect answers. A language model can read a student's explanation in their own words, a paragraph of an essay, or a confused question, and respond to what is actually there. This opens tutoring to domains such as writing, history and conceptual science, where the classical approach never worked well. The third is register and language. A language model can explain the same idea in the vocabulary of a ten-year-old or a graduate student, in English or in Yoruba or Turkish, with examples drawn from football or cooking. For students learning in a second language, or whose teachers do not share their first language, this flexibility is substantial. The fourth is cost. A tutoring conversation with a capable model costs a small fraction of what an hour with a human tutor does, and the cost has fallen steeply since 2023. For the first time, something resembling one-to-one tutoring is economically available to students who could never have paid for a human tutor. These changes address the obstacles that kept classical tutors in a narrow niche: construction cost, domain coverage and the rigidity of the interaction. They do not, on their own, address the other two components of a tutor described in this book's introduction: a model of what the student knows over time and a policy for what they should do next. A language model in a single conversation has neither unless they are added deliberately. The helpful assistant problem The most important fact about general-purpose language models, from a tutoring perspective, is that they have been trained to be helpful in the sense that an assistant is helpful. When a user asks a question, the model is rewarded for answering it, clearly and completely. When a user asks it to solve a problem, it solves the problem. This is what makes these systems useful for most purposes, and it is exactly the wrong behavior for a tutor. A good tutor, confronted with a student who asks for the answer, usually does not give it. Withholding is the point. Chapter 1 described the evidence that students learn from the cognitive work they do themselves, that constructive and interactive engagement produce more learning than passive reception. An assistant that does the work for the student converts every potential learning episode into a passive one. The student reads a correct solution, perhaps understands it in the moment, and leaves without having built the ability to produce it. Researchers documented this mismatch before and after the release of ChatGPT. Anaïs Tack and Chris Piech, in a 2022 paper called "The AI Teacher Test," compared responses from earlier conversational models with those of human teachers in real educational dialogues and found that the models were rated considerably worse on helpfulness to the student's learning, even when their responses were fluent. Jakub Macina and colleagues at ETH Zurich, in their 2023 work on a dataset called MathDial, collected tutoring conversations in which teachers worked with a language model playing the role of a student, and then studied how models behaved when they themselves acted as tutors. They found that models tended to reveal the solution too early, failed to identify the specific error in a student's reasoning, and often affirmed incorrect student work. Their conclusion was that general-purpose language models, though capable of solving the problems, lacked the pedagogical restraint and diagnostic focus that tutoring requires. A related failure is sycophancy: the tendency of models trained on human approval to agree with the user. A student who confidently states a wrong answer and asks for confirmation may receive it, or receive a gentle qualification buried in praise. For a tutor, accurately telling a student they are wrong, kindly but clearly, is essential, and a model biased toward agreeableness may fail at precisely this moment. Finally, language models make errors. They are much better at mathematics and science than they were in 2023, and the strongest current models solve most school and early university problems correctly. But they still sometimes compute incorrectly, misread a diagram described in text, or produce a plausible-sounding explanation that is subtly wrong. A tutor that is wrong a small fraction of the time is a serious problem, because students, by definition, are not in a position to catch its errors. A student who is taught a wrong method confidently will learn it. Building restraint Developers have responded with a range of techniques that, taken together, form the practical craft of building a language-model tutor. The simplest is the system prompt, the instructions given to the model before the conversation that shape its behavior. A tutoring prompt typically tells the model not to give final answers, to ask the student what they have tried, to offer the smallest useful hint, to check the student's understanding before moving on, and to respond to errors by asking a question that would let the student find the mistake. Prompts of this kind can change behavior substantially, and the Harvard physics tutor discussed in the next chapter relied heavily on carefully engineered prompts. But prompts are fragile. A determined student can often talk a model out of its instructions, and the model may drift back toward assistant behavior over a long conversation. A second technique addresses accuracy by grounding the model in vetted solutions. Rather than letting the model solve each problem itself and then tutor from its own solution, the system supplies a correct, expert-written solution in the model's hidden context. The model's job is then not to solve the problem but to guide the student toward the known solution, which reduces the risk of tutoring toward a wrong answer. A related approach has the model work out the problem privately, in reasoning the student does not see, before deciding how to respond, so that it knows where the student's work has gone wrong before choosing a question to ask. A third technique is to train models specifically for tutoring. Google's LearnLM project, described in a 2024 technical report by Irina Jurenka and a large team of collaborators titled "Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach," fine-tuned Gemini models on data designed to instill pedagogical behaviors: encouraging active learning, managing cognitive load, adapting to the learner, stimulating curiosity and deepening metacognition. The report also set out a set of evaluations, including ratings by teachers and education experts of conversations with simulated and real learners, to measure whether the fine-tuned model behaved more like a good tutor than the base model. It did, by those measures, though the report was candid that measuring good tutoring remains difficult. A fourth technique, increasingly common, is to give students an explicit choice of mode. Anthropic introduced a learning mode for Claude in its education offering in April 2025, designed to guide students through reasoning rather than supply answers. OpenAI added a study mode to ChatGPT in July 2025, and Google launched a guided learning mode for Gemini the following month. Each is built on the same insight, that the default assistant behavior is wrong for learning, and each offers a more Socratic alternative. Each also leaves the choice to the student, who can switch back to the ordinary assistant whenever the Socratic one becomes inconvenient. That design choice reflects the commercial position of general-purpose assistants, which serve adults doing work as well as students learning, but it means the restraint is voluntary, and the next chapter shows why that matters. Purpose-built tutoring products make the restraint mandatory. Khan Academy's Khanmigo, launched in pilot form in March 2023 on OpenAI's GPT-4, was designed from the start not to give students answers, and it operates inside Khan Academy's existing exercises and mastery system, so it can be given the problem the student is working on and a correct solution. Because it is deployed through schools, teachers can see students' conversations. Sal Khan, Khan Academy's founder, set out the case for this kind of tutor in his 2024 book Brave New Words. What a Socratic exchange looks like The difference between assistant and tutor behavior is easiest to see in a single exchange. Suppose a student writes: "Why does ice float? Just tell me the answer, I have a test tomorrow." An assistant responds with a clear, correct explanation: water expands when it freezes because hydrogen bonds lock the molecules into an open lattice, so ice is less dense than liquid water, and less dense things float. The student reads it, nods, and will probably remember little of it tomorrow. A tutor responds differently. It might acknowledge the deadline and then ask: "What do you already know about why anything floats? What decides whether a block of wood sinks or floats?" If the student says it is about weight, the tutor might ask whether a large log, which is very heavy, sinks. When the student arrives at density, the tutor asks how ice and water might compare, and then what it would mean for frozen water to take up more space than liquid water. Only when the student has worked out that ice must be less dense does the tutor introduce the molecular explanation, and then it asks the student to put the whole chain into their own words. This takes longer, perhaps five minutes instead of thirty seconds. It is also likely to produce knowledge the student can use on the test and beyond, because the student generated the reasoning. The student has also been told nothing they could not have accepted passively, but has been made to construct it. Done badly, however, Socratic questioning becomes a guessing game in which the student tries to work out what the tutor wants them to say. Graesser's studies of human tutors found that skilled questioning was rare precisely because it is hard: it requires diagnosing what the student currently thinks and choosing a question that moves them from there. A tutor that asks generic questions ("What do you think?") regardless of the student's state is not Socratic but evasive, and students quickly lose patience. The ideal is a tutor that asks questions when the student is close to an insight they can reach, and explains directly when they are not, because a student with no foothold gains nothing from being asked to guess. This is where the learner model returns. Whether to question or explain depends on what the student already knows. A student who has mastered the concept of density can be asked to apply it to ice; a student who has never encountered density needs an explanation first. The research on worked examples supports this: novices learn more from studying worked examples than from solving problems unaided, while more advanced learners benefit more from solving problems, a pattern Slava Kalyuga and colleagues called the expertise reversal effect. A Socratic tutor without a learner model cannot apply this rule consistently, because it does not know which kind of student it is talking to. Voice, images and the illusion of a companion The tutoring interface is no longer confined to typed text. Current models accept photographs, so a student can photograph a page of handwritten working and ask where it went wrong, which brings the system much closer to the step-level feedback that VanLehn identified as the source of tutoring's power. A model that can see each line of a derivation can, in principle, identify the line where the error occurred and ask about that line, rather than responding only to a wrong final answer. Real-time voice conversation, available in the major assistants since 2024, makes it possible for young children who cannot yet type fluently, and for students with some disabilities, to use a tutor at all. Both developments widen access considerably. They also raise new concerns. A voice tutor that is warm, encouraging and always available can feel to a child like a companion rather than a tool. That feeling has pedagogical value, since students work harder for tutors they like, but it brings risks that the field has barely begun to study: children confiding in a system that is not equipped to respond to distress, becoming emotionally dependent on it, or preferring it to the more demanding company of teachers and peers. Responsible deployments in schools generally route certain topics, such as self-harm, to human adults and restrict the tutor to its educational role. These safeguards are part of the design of a tutor just as much as its questioning strategy. A related point concerns the tutor's persona. Some products give their tutors names, faces and personalities, sometimes historical figures or fictional characters. Done carefully, this can make practice more engaging. Done carelessly, it blurs the line between what the system is, a statistical model with no understanding of the child in front of it, and what it appears to be. Students, particularly younger ones, should know that they are talking to software, and teachers should be able to see what it says to them. The architecture that results Putting these pieces together yields a picture of what a well-designed AI tutor looks like, which differs considerably from a chatbot with a tutoring prompt. At its core is a learner model of the kind described in Chapters 2 and 3, holding estimates of the student's mastery of each knowledge component in the course, persisted across sessions. Around it is a curriculum policy of the kind described in Chapter 4, choosing what the student works on next, scheduling review, and recognizing when a student is stuck. The language model sits at the interface. It reads what the student writes, converts it into evidence for the learner model, and generates questions, hints and explanations calibrated to what the learner model says the student knows. It is grounded in vetted content so that it tutors toward correct answers. And it operates under constraints, enforced by the system rather than only by a prompt, that prevent it from doing the student's work. Few products in 2026 implement all of this fully. Many products marketed as AI tutors are closer to the bare chatbot end of the spectrum, with a tutoring prompt and some subject-specific content but no persistent learner model and no curriculum policy. Others are classical adaptive platforms with a language model added as a hint generator. The difference matters, and the next chapter's evidence shows why. Chapter 7: The Crutch and the Evidence Within a year of ChatGPT's release, students had made their own decision about what large language models were for. A Pew Research Center survey published in January 2025 found that about a quarter of American teenagers had used ChatGPT for schoolwork, double the share a year earlier, and by most accounts usage has continued to climb. University surveys in many countries have found majorities of students using generative AI for coursework. Much of this use is for exactly what the tools are best at: producing finished text, solving problems, summarizing readings. Very little of it, by students' own accounts, takes the form of Socratic dialogue. This is the context in which any evidence on AI tutoring has to be read. The question is not only whether a well-designed tutor can produce learning. It is also what happens when students have an answer engine in their pocket, and whether the tutoring use or the answering use prevails. The first wave of rigorous studies, most published in 2024 and 2025, speaks to both questions, and the picture they paint together is sharper than either enthusiasts or critics have generally acknowledged. When help harms The most important single study so far was conducted in Turkey and published in the Proceedings of the National Academy of Sciences in June 2025 under the title "Generative AI Without Guardrails Can Harm Learning: Evidence from High School Mathematics." Its authors, Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı and Rei Mariman, worked with a high school to run a field experiment involving nearly a thousand students over several practice sessions in mathematics. The design was simple and well suited to the question. During practice sessions covering material from the regular curriculum, students were randomly assigned to one of three conditions. Some practiced with their usual materials only. Some had access to a GPT-4-based assistant with a standard interface, which the authors called GPT Base, essentially resembling ChatGPT. And some had access to GPT Tutor, the same underlying model with a prompt designed by teachers to guide students toward solutions with hints rather than give answers, and supplied with correct solutions to the practice problems so that it would not lead students astray with its own errors. After each practice session, all students took an exam on the same material, without access to any AI. During practice, both AI groups performed much better than the control group. Students with GPT Base solved 48 percent more practice problems correctly; students with GPT Tutor solved 127 percent more. On the exams that followed, however, the picture reversed. Students who had practiced with GPT Base scored 17 percent lower than the control group. Students who had practiced with GPT Tutor scored about the same as the control group, with no significant difference in either direction. The authors' analysis of the conversations explains the result. Many students in the GPT Base group simply asked for the answer and copied it. They completed their practice successfully in the sense of producing correct answers, but they did not do the practice in any meaningful sense, and so they did not learn the material as well as students who had worked through it with only a textbook. In the GPT Tutor group, the guardrails prevented the worst of this substitution, which was enough to eliminate the harm, but not enough to produce a measurable benefit over conventional practice. Two features of the study deserve emphasis. First, the harm came from a tool that made practice look dramatically more successful. Any teacher or school judging AI by students' homework performance would have concluded GPT Base was a great success. Second, the protective condition, the one with guardrails and vetted solutions, was the one a careful developer would build. Even it produced no gain on the exam. The study shows that answer-giving can harm learning; it does not show that tutoring with a language model reliably helps. The PNAS study is not alone. Smaller studies point in the same direction. Yizhou Fan and colleagues, in a study of university students published in the British Journal of Educational Technology in 2025 under the title "Beware of Metacognitive Laziness," found that students who used ChatGPT while revising essays produced better essays than comparison groups but did not show greater gains in knowledge or its transfer, and engaged less in the self-monitoring that supports learning. A 2025 preprint from the MIT Media Lab by Nataliya Kosmyna and colleagues, which drew wide media attention, recorded brain activity while a small number of participants wrote essays with and without a language model, and reported weaker neural engagement and poorer recall of their own essays among those who used the model. That study is small and has not yet been through full peer review, and its neural measures are difficult to interpret, but its behavioral finding, that people remember less of work they did not do themselves, is consistent with decades of research on the generation effect. The underlying mechanism is not mysterious. Cognitive psychologists have long described an illusion of competence: when material is presented fluently and clearly, learners feel they understand it and overestimate how well they will remember it. Reading a well-written solution produces a powerful sense of understanding that does not survive the attempt to reproduce it. A language model that produces polished explanations on demand is a near-perfect generator of this illusion. When help helps Against this, several well-designed studies have found substantial gains from AI tutoring. Their designs differ from the Turkish study in ways that matter. At Harvard University, Gregory Kestin, Kelly Miller and colleagues ran a randomized crossover trial in an introductory physics course, published in Scientific Reports in 2025 under the title "AI Tutoring Outperforms In-Class Active Learning." About 194 students each experienced two lessons, on surface tension and on fluid flow, one taught through the course's usual in-class active learning and the other through a custom AI tutor built on GPT-4. The order was randomized so that each student served as their own control. The AI tutor was carefully designed: its prompts implemented principles from the learning sciences, it guided students through problems step by step, and it was supplied with expert-written solutions to keep it accurate. Students working with the AI tutor learned more, with median gains on post-tests more than double those from the in-class lessons and effect sizes the authors estimated at between 0.73 and 1.3 standard deviations. They took a median of 49 minutes, somewhat less time than the class session, and reported higher engagement and motivation. The result is striking, but it has limits the authors acknowledge. Each comparison covered a single lesson, learning was measured immediately afterward rather than weeks later, and the students were Harvard undergraduates, a highly motivated and capable population. The AI tutor was used in place of a single class session, not as a semester-long replacement for teaching. What the study shows is that a carefully engineered AI tutor, grounded in vetted content and designed around known principles, can deliver a lesson more effectively than a good active-learning class for motivated students. That is important. It is not the same as showing that AI tutoring will transform a course. In Nigeria, a team from the World Bank led by Martín De Simone evaluated a six-week after-school program at secondary schools in Benin City, Edo State, in mid-2024. About 800 students in their first year of senior secondary school met twice a week in school computer labs to work on English with Microsoft Copilot, which was then based on GPT-4. Teachers were present as facilitators, guiding students on how to use the tool and following a structured set of activities aligned with the curriculum. Students in the program outperformed a randomly selected comparison group by about 0.3 standard deviations on an English assessment, and the authors reported gains on end-of-year exams as well. They estimated that the gain was equivalent to nearly two years of typical learning in the Nigerian context and placed the program among the most cost-effective education interventions in a large comparative database. Girls benefited more than boys. In Ghana, Owen Henkel and colleagues evaluated Rori, a mathematics tutor delivered through the messaging service WhatsApp on basic phones, used by students roughly aged eight to fourteen. In a study involving about a thousand students in grades three to nine across eleven schools over eight months in 2023, students with access to Rori for two thirty-minute sessions a week, scheduled during the school day, gained about 0.37 standard deviations more in mathematics than a control group. Rori was a hybrid system that combined structured practice content with language-model conversation, and it was designed to work over low-bandwidth connections at very low cost. A different model was tested in the United States. Rose Wang, Ana Ribeiro, Carly Robinson, Susanna Loeb and Dora Demszky at Stanford built Tutor CoPilot, which did not tutor students directly. Instead, it assisted human tutors, many of them relatively inexperienced, who were conducting online mathematics tutoring with elementary students in a low-income school district. When a student made an error, the tutor could press a button and receive suggestions, generated by a language model guided by the strategies of expert teachers, for how to respond: asking the student to explain their reasoning, giving a hint, or prompting a related simpler problem. In a randomized trial involving roughly 900 tutors and 1,800 students in the spring of 2024, students whose tutors had access to Tutor CoPilot were 4 percentage points more likely to master the topics they were working on, and 9 percentage points more likely when paired with tutors who had been rated as less effective. The cost was about twenty dollars per tutor per year. The tool worked mainly by making tutors more likely to use questioning strategies and less likely to simply give answers. Table 4 sets these recent studies side by side with the Turkish one. Table 4. Selected controlled studies of language-model tutoring, 2023 to 2025. Study Setting AI design Teacher role Result Bastani et al. (2025) High school math, Turkey Plain GPT-4 vs. guided tutor Present; not mediating AI use Plain: 17% lower exam scores; guided: no difference Kestin et al. (2025) University physics, Harvard Engineered prompts, vetted solutions Course designers built the tutor 0.73 to 1.3 SD on immediate post-test De Simone et al. (2025) Secondary English, Nigeria Copilot with structured activities Facilitators in the lab About 0.3 SD Henkel et al. (2024) Primary and middle math, Ghana WhatsApp tutor with structured practice Scheduled sessions at school About 0.37 SD Wang et al. (2024) Elementary math tutoring, US Suggestions to human tutors AI supports the human 4 points more mastery; 9 for weaker tutors Source: the cited publications; figures as reported by their authors. Reading the pattern These results are sometimes presented as contradictory, as if the field had not yet decided whether AI tutoring helps or harms. Read alongside the earlier chapters, they are consistent, and the pattern they form is the main empirical support for this book's argument. In every study that found harm or no benefit, students had substantial freedom to use the AI to obtain answers rather than do the work, or the guidance was only strong enough to prevent harm. In every study that found substantial gains, the AI was embedded in a structure that kept the student doing the cognitive work. In the Harvard study, the tutor was engineered to guide rather than tell, with vetted solutions and a structured progression. In Nigeria, the AI was used inside a curriculum-aligned program with teachers present. In Ghana, the language model was combined with structured practice content. In the Tutor CoPilot study, the AI did not interact with students at all; it helped human tutors behave more like good tutors. The common element is not the sophistication of the language model. All of these studies used models of roughly similar capability. The difference is whether the system around the model supplied the other components of tutoring: a structured path through the material, a check on whether the student was actually doing the work, and some form of human or programmatic restraint on answer-giving. Where these were present, the model's conversational ability became a powerful amplifier of learning. Where they were absent, it became a powerful amplifier of avoidance. There are caveats. All of these studies are short, ranging from a single lesson to eight months. Most measure learning soon after the intervention, so we do not yet know how durable the gains are. Several were conducted by teams who built or championed the system being tested, a pattern that in earlier educational technology research was associated with larger effects. None yet tests a full-year, at-scale deployment of the kind the RAND study of Cognitive Tutor provided. The history recounted in Chapter 5 suggests that effects will shrink as studies become larger, longer and more independent. Anyone who cites the Harvard effect sizes as a forecast for what AI tutoring will do across a school system is making the same mistake as those who cited Bloom's two sigma. Learning versus performance The deepest lesson of the Turkish study is one that the learning sciences have long understood but that technology design routinely ignores. Learning and performance are different things, and they can move in opposite directions. Performance is what a student can do right now, with whatever support is available. Learning is the durable change in what they can do later, without support. Students working with GPT Base performed much better during practice and learned less. Students practicing with desirable difficulties, as described in Chapter 4, perform worse during practice and learn more. Any system that measures its success by performance during use, which includes most analytics dashboards and most student satisfaction surveys, will be systematically misled about whether it is producing learning. This has an uncomfortable implication for the market in AI education products. Products that give students answers feel helpful, produce higher immediate performance, and generate satisfied users. Products that withhold answers and require effort feel harder, produce lower immediate performance and generate frustrated users. A market driven by usage and satisfaction will tend to select for the former. Only a buyer who measures delayed learning, on assessments taken without AI, will consistently choose the latter. The implication for designers is that the tutor must be judged on what students can do when the tutor is gone. That is a slower, more expensive and less flattering metric than engagement or practice accuracy. It is also the only one that measures whether a tutor is doing its job. The student's choice One last observation connects this evidence to the products students actually use. The learning modes offered by general-purpose assistants, described in the previous chapter, are well designed in many respects. But they are optional. A student facing a deadline at eleven at night can switch to the ordinary assistant and get the answer. The Turkish study suggests that many will. This is not a criticism of students. Students who copy answers under time pressure are behaving rationally given the incentives they face, which usually reward completed work over learning. It is a reminder that the decisive design choices are often not in the software. They are in the structure of assignments, the design of assessments, and the presence of adults who can see what students are doing. A school that assigns problem sets graded on correctness and allows unrestricted AI access has, in effect, chosen the GPT Base condition for all its students. A school that assigns practice with a constrained tutor and assesses learning in person has chosen something closer to the GPT Tutor condition or better. The evidence does not say that AI tutoring works or that it fails. It says that a language model is an amplifier, and that what it amplifies depends on the system, human and technical, that surrounds it. Chapter 8: Teachers, Data and Power The previous chapters treated AI tutoring as a technical problem: how to model what a student knows, how to choose what comes next, how to make a language model behave like a tutor. The evidence in Chapters 5 and 7 has already shown that technical quality is not enough. What happens around the software, especially what teachers do, determines much of whether it works. This chapter turns to that surrounding system, and to three questions that the technology raises but cannot answer: what becomes of the teacher, who controls the data that learner models depend on, and whose interests are served when a machine decides what a child should learn. The teacher in the loop Every successful deployment described in this book had a teacher somewhere in the loop. ASSISTments worked partly through the reports it gave teachers. The Nigerian program had teachers in the computer lab. Tutor CoPilot made human tutors better rather than replacing them. And the cautionary cases, from Summit Learning's protests to the low usage that plagues self-directed platforms, often involved systems that pushed the teacher to the margins. This is not sentimentality about teaching. It reflects a division of labor in which humans and machines are good at different things. Machines are good at tirelessly tracking dozens of skills for each of thirty students, giving immediate feedback on routine practice, and scheduling review. They are poor at noticing that a student is upset about something at home, at persuading a reluctant fifteen-year-old that algebra is worth the effort, at leading a discussion in which students build on each other's ideas, and at judging when the plan should be abandoned because something more important has come up. Teachers are good at all of these, and they cannot track thirty students' mastery of forty skills in their heads. Researchers have explored how to combine the two. Kenneth Holstein, Bruce McLaren and Vincent Aleven at Carnegie Mellon built Lumilo, a system that gave teachers mixed-reality smart glasses through which they could see, floating above each student's head, indicators drawn from the tutoring software's learner model: whether a student was struggling, idle, gaming the system, or making rapid progress. In a classroom study reported in 2018, students in classes where teachers used Lumilo learned more from the same tutoring software than students in classes without it, with the largest benefits for students who started with less knowledge. The mechanism was simple. Teachers spent their time with the students who most needed them, because the software told them who those students were. The glasses were a research prototype, but the principle has spread into ordinary teacher dashboards: the learner model's job is not only to make decisions for the software but also to inform decisions by the teacher. A dashboard that shows which students have been stuck on the same skill for twenty minutes, which ones are rushing through hints, and which skills the whole class is struggling with turns the learner model into a tool for teaching rather than a substitute for it. A well-designed AI tutor generates the same kind of signal from conversations: which students asked for answers repeatedly, which revealed a specific misconception, which seem disengaged. Teachers also use generative AI directly, for preparing lessons, writing differentiated materials, giving feedback on written work and handling administrative tasks. Surveys of teachers in the United States and the United Kingdom since 2024 have found rapidly growing use, and teachers who use AI regularly report substantial time savings. These reports are self-assessments and should be treated cautiously, but the direction is clear. If teachers can spend less time on clerical work and first-draft feedback, they can spend more on the relational and diagnostic work that machines do poorly. The risk runs the other way as well. A system that saves teacher time can be used to justify fewer teachers rather than better teaching, and in school systems under financial pressure that temptation is real. The evidence surveyed in this book offers no support for the idea that AI tutoring can substitute for teachers in general schooling. It supports the narrower claim that AI tutoring can extend what a teacher can do, and can provide something to students who have no teacher or tutor available at the moment they need one. The data that learner models eat A learner model is a detailed record of a child's learning. To estimate mastery of each skill, it records every response, often with timing, hint use and, increasingly, the full text of conversations with a language model. Over a school year, this adds up to a remarkably intimate portrait: what the student knows and does not know, how they respond to difficulty, when they give up, what they write when they are confused, and sometimes what they confide to a chatbot that seems to be listening. This data has obvious value for teaching. It also has value for other purposes: training commercial models, marketing products, profiling students for admissions or employment, and, in less benign hands, surveillance. The question of who controls it has been politically explosive before. In 2013, a nonprofit called inBloom, funded with about a hundred million dollars from the Bill and Melinda Gates Foundation and the Carnegie Corporation of New York, set out to build shared infrastructure for storing student data from multiple states and districts, so that educational software could draw on it to personalize learning. Parents and privacy advocates objected strongly to the aggregation of detailed student records in a single private system, particularly given unclear limits on how vendors could use them. States withdrew one after another, and inBloom shut down in 2014. Its failure made clear that the public will not accept data infrastructure for personalized learning that it does not understand and trust. In the United States, the main legal protections are the Family Educational Rights and Privacy Act of 1974, which governs access to educational records, and the Children's Online Privacy Protection Act of 1998, which requires parental consent for collecting personal information from children under thirteen, and whose implementing rule the Federal Trade Commission updated in 2025. Many states have added their own student privacy laws. In Europe, the General Data Protection Regulation applies to student data with particular care for children. None of these frameworks was designed with continuous conversational logs from language-model tutors in mind, and questions such as whether a student's conversations may be used to train the vendor's next model are often answered in contract terms that schools do not read closely. Practical safeguards are not complicated. Schools can require that student data be used only for the educational purpose for which it was collected, not for training general models or for advertising; that data be deleted after a defined period; that conversation logs be accessible to teachers and parents; and that vendors disclose what their learner models record and infer. Several state education agencies and large districts have adopted contract templates along these lines. What is lacking in many places is not the template but the capacity to negotiate and enforce it. Bias in a model of knowledge A learner model makes consequential decisions about students: who has mastered a skill, who needs more practice, who is ready for the next topic. If those decisions are systematically less accurate for some groups of students than others, the system is unfair in a concrete sense, even if no one intended it. The way bias enters a learner model is often subtle. Shayan Doroudi and Emma Brunskill, in a 2019 paper titled "Fairer but Not Fair Enough: On the Equitability of Knowledge Tracing," examined what happens when students learn at different rates but are modeled with a single set of parameters, as in standard Bayesian knowledge tracing. A population-level model is fitted to the average learner. When it is applied to students who learn more slowly than average, it tends to conclude they have reached mastery too early and moves them on before they are ready. Students who learn faster than average are kept practicing longer than they need. The result is that a system designed to give each student the right amount of practice systematically under-serves the students who need the most help. Individualized models reduce the problem, but the authors showed they do not eliminate it. Ryan Baker and Aaron Hawn, in a 2022 review of algorithmic bias in education published in the International Journal of Artificial Intelligence in Education, documented many other routes. Models trained mostly on data from one population, such as suburban American students, may perform worse for students from other backgrounds. Models that use features like response time may misinterpret students who read more slowly because they are learning in a second language. And models built on data from one country may encode its curricular assumptions in ways that fail elsewhere. Language models add new sources of bias. A model's judgment of whether a student understands a concept may be swayed by the fluency of the student's English, by dialect, or by stylistic features unrelated to understanding. Weixin Liang and colleagues showed in a 2023 paper in the journal Patterns that software for detecting AI-generated text misclassified essays written by non-native English speakers as machine-generated at high rates, a stark illustration of how surface features of language can drive consequential judgments. A tutor that interprets student writing will need to be tested for exactly this kind of disparity. The remedy is not to abandon learner models but to audit them. A responsible developer measures whether mastery decisions are equally accurate across groups of students, whether students from different groups spend comparable time stuck, and whether the system's recommendations lead to comparable outcomes. These analyses require collecting demographic data, which raises its own privacy tensions, and they are rarely published. A school system choosing a product is entitled to ask for them. Surveillance and the limits of measurement One line of research in educational AI has sought to detect not only what students know but how they feel: whether they are bored, confused, frustrated or engaged. Ryan Baker and Sidney D'Mello, among others, built detectors that infer these states from interaction logs, and some systems have used cameras and facial expression analysis for the same purpose. The pedagogical case is real, since confusion and boredom affect learning and a good tutor notices them. But the line between noticing and monitoring is thin, and emotion recognition from faces rests on contested science. The European Union's AI Act, adopted in 2024, drew a firm line. It prohibits the use of AI systems to infer the emotions of people in educational institutions, except for medical or safety reasons, a prohibition that took effect in February 2025. The same act classifies several educational uses of AI as high-risk: systems that determine admission, that evaluate learning outcomes or steer the learning process, that assess the level of education an individual should receive, and that monitor students for prohibited behavior during tests. High-risk systems must meet requirements for risk management, data quality, transparency, human oversight and accuracy. The original deadline for these obligations was August 2026; an amending package agreed in 2026 moved the date for such stand-alone high-risk systems to December 2027. When they apply, AI tutors that steer students' learning paths in the EU will need documented risk assessments and meaningful human oversight, requirements that align closely with the design principles this book has argued for on pedagogical grounds. Other jurisdictions have moved more slowly or in different directions. In the United States, an executive order issued in April 2025 promoted AI education and AI literacy for students and teachers rather than imposing new restrictions, and the Department of Education subsequently issued guidance on using federal funds for AI tools. UNESCO's 2023 guidance on generative AI in education and research urged governments to regulate data protection and suggested a minimum age of thirteen for the use of generative AI tools in the classroom. China, Singapore, South Korea and others have pursued national strategies that include both AI literacy curricula and state-sponsored educational platforms. Who decides what a child learns? Beneath these specific issues lies a larger question. Curriculum has always been a matter of public decision, contested and negotiated through school boards, ministries, examination bodies and teachers. When a personalized learning system decides, for each student, which topics to study, in what order, at what pace and to what standard, it is making curricular decisions at a granularity no public body has ever controlled. Those decisions are embedded in prerequisite maps, mastery thresholds and optimization targets chosen by a vendor's engineers and content designers, usually without public scrutiny. Most of the time this is harmless; the vendor's choices are sensible and broadly match the official curriculum. But the cases discussed in Chapter 4, where personalization slides into lower expectations for students judged to be behind, show how the details of an algorithm can encode consequential value judgments. A system that keeps a struggling student on remedial material indefinitely has decided, in effect, that this student will not encounter grade-level content this year. That is a decision a teacher or parent might well want to contest, if they knew it had been made. The principle that follows is that the human beings responsible for a child's education must be able to see and override what the system decides. Teachers should be able to see each student's path and change it. Parents should be able to understand, in plain terms, what the system believes about their child and why. Students themselves, especially older ones, should be able to see their own learner model, which has the added benefit, documented in research on open learner models by Susan Bull and others, of supporting reflection and self-regulation. Opaque personalization, however accurate, transfers authority over learning from accountable people to unaccountable software. Transparent personalization keeps it where it belongs. Access and the global picture A last question concerns who gets these tools. In wealthy countries, the concern is often that AI tutoring will widen gaps, because advantaged students will use it well and disadvantaged students will use it poorly or not at all, the Matthew effect described in Chapter 5. In much of the world, the picture is different. Hundreds of millions of children are in school without learning basic literacy and numeracy, in classrooms of fifty or more, with teachers who are often undertrained. For them, the alternative to an AI tutor is not a human tutor; it is nothing. The studies from Nigeria and Ghana, and the earlier Mindspark study in India, suggest that well-structured AI tutoring can produce substantial gains in such settings, at very low cost, sometimes over basic phones and low-bandwidth connections. These are among the most hopeful results in the field. They come with the same conditions as elsewhere: structure, curriculum alignment, supervised time, and design for the specific context. They also raise questions about language, since most capable models perform best in English and a handful of other widely used languages, and about dependence on technology companies based far from the communities they serve. But the potential to extend some form of tutoring to children who have never had any is real, and it is arguably where AI in education can do the most good. Conclusion The two students in the introduction of this book used AI in ways that looked equally modern and produced opposite results. The chapters since have explained why. One received answers from a system that knew nothing about her and asked nothing of her. The other worked inside a system that kept track of what he knew, chose what he should do next, and made him do it. The conversational fluency of the machine was the least important difference between them. Taken together, the evidence and history in this book support a conclusion that is more specific than "AI will transform education" and more hopeful than "AI will ruin it." The value of an AI tutor depends on what it knows about the learner and what it refuses to do for the learner. Everything else, including the impressive fluency of modern language models, is an amplifier of whichever way those two answers point. Several consequences follow, and they are not the ones most often drawn. The scarce ingredient is not explanation The first consequence is that the bottleneck in education was never the supply of clear explanations. Textbooks, lectures and, for two decades, free online videos have offered good explanations of nearly everything in the school curriculum. Students who did not learn did not, in most cases, lack access to an explanation. They lacked practice at the right level, timely feedback on their errors, someone who noticed when they were stuck, and a reason to keep going. Large language models have made explanations more abundant, more personal and more responsive, which is a real gain. But they have done least for the ingredients that were actually scarce. Those ingredients are supplied by the older and less glamorous parts of the tutoring architecture: the learner model that estimates what a student knows, the sequencing policy that turns those estimates into a path, and the human or programmatic structure that keeps the student doing the work. The most promising AI tutors are therefore not the newest chatbots but hybrids that put a language model on top of these foundations. Building such systems is harder than writing a tutoring prompt, and the market rewards the prompt, because it is cheaper and its users are satisfied. The gap between what sells and what teaches is likely to be the defining problem of the field for the next several years. Disruption is a redistribution, not a replacement The second consequence concerns what "disruption of the classroom" can realistically mean. The evidence gives no support to the idea that AI tutors will replace teachers in general schooling. Every successful deployment had a human in the loop, and the most cost-effective tool tested so far, Tutor CoPilot, worked by making human tutors better. What AI can do is redistribute the work of teaching. A single teacher cannot track every student's mastery of every skill, give immediate feedback on every practice problem, or sit beside every student at the moment of confusion. Software can do much of this, and do it well, if it is built properly. That frees the teacher, in principle, for the work only a person can do: motivating, discussing, noticing, and deciding when the plan should change. The phrase "in principle" matters, because the same redistribution can be used to cut teaching staff rather than improve teaching, and school systems under budget pressure will be tempted. The pedagogical case is for using AI to give each teacher better information and more time, not to give each teacher more students. Assessment has to move The third consequence is less obvious and possibly the most urgent. The Turkish experiment showed that a widely available tool can raise students' performance on practice while lowering what they actually learn. If schools judge students, and teachers judge themselves, by the quality of work produced outside supervised conditions, AI will make that work look better and the underlying learning worse, and nobody will notice until the gap appears in some later, unassisted test. The response cannot be to ban the tools, which students will use anyway, nor to rely on AI detection software, which is unreliable and, as Chapter 8 noted, biased against some groups of students. It has to be a shift in what counts as evidence of learning: more assessment in supervised conditions, more oral explanation, more work in which students show their reasoning live, and homework treated as practice rather than as proof. That is a return to older forms of assessment as much as an innovation, and it will cost teacher time, which is one more reason to use AI to save that time elsewhere. What different readers should do For school leaders and those who buy educational technology, the practical upshot is a short set of questions to ask of any product sold as an AI tutor. Does it maintain a persistent model of what each student knows, by skill, and can a teacher see it? Does it have a curriculum policy, including when to stop practice and when to return to old material? Does it prevent the student from obtaining answers without doing the work, by design rather than by a prompt the student can talk around? Is it grounded in vetted content so that it does not tutor toward wrong answers? Has it been evaluated on learning measured later and without the tool, rather than on usage or satisfaction? And does the vendor commit to using student data only for the student's education? A product that fails most of these questions may still be a useful resource, but it is not a tutor. For developers, the lesson is that the competitive advantage lies not in the language model, which is increasingly a commodity, but in the learner model, the curriculum, the content and the evaluation. The firms and nonprofits that will matter are those willing to measure delayed learning honestly and to build the less visible components that make a tutor more than a chatbot. The work of the educational data mining community over thirty years, on knowledge tracing, skill maps, mastery decisions and wheel-spinning, is directly relevant and largely unused by newcomers to the field. For teachers, the practical guidance is to treat AI tools as practice partners whose use must be structured. Students should use them inside tasks that require their own reasoning, with the tutor's restraint enforced where possible and the teacher able to see the conversations. The question to ask of any AI-assisted assignment is the one the Turkish study answers: will the student be able to do this tomorrow, without the tool? For parents and students, the advice is the simplest and the hardest to follow. When the goal is learning, the effort is the point. A tool that removes the effort removes the learning with it. Asking an AI to explain, question and check is likely to help; asking it to do the work is likely to harm, however good the result looks. What remains unsettled Several important questions remain open, and honest readers should hold their conclusions with corresponding modesty. We do not yet know whether the gains from well-designed AI tutoring persist over months and years, or whether, like many educational interventions, they fade. We do not know how large the effects will be when these systems are deployed across whole school systems by ordinary staff rather than in studies run by their designers. We do not know whether language models can themselves become reliable, calibrated learner models, or whether they will always need a classical model to keep score. We do not know what sustained use does to students' motivation, independence and willingness to struggle, which may matter more in the long run than any test score. And we do not know how teaching as a profession will change if much of its diagnostic work is handed to software, whether teachers' own expertise will grow by being freed for higher-level work or wither by disuse. These questions will be answered by the next decade of deployment and research. What can be said now is that the answers will depend far more on design and implementation choices than on further improvements in the underlying models. Bloom's two sigma problem was never a problem of explanation. It was a problem of knowing each student, keeping each one working at the edge of their ability, and doing so at a cost society could afford. AI has, for the first time, brought the cost within reach. Whether it solves the rest of the problem is a choice, not a forecast. Hashtags: #AIInEducation #ArtificialIntelligence #EducationalTechnology #EdTech #PersonalizedLearning #AdaptiveLearning #IntelligentTutoringSystems #AITutors #IntelligentTutors #GenerativeAI #LargeLanguageModels #StudentCenteredLearning #IndividualizedInstruction #LearningAnalytics #StudentModeling #BayesianKnowledgeTracing #DeepKnowledgeTracing #KnowledgeTracing #MasteryLearning #AdaptiveCurriculum #CurriculumPersonalization #SocraticTutoring #SocraticLearning #FormativeAssessment #SpacedRepetition #RetrievalPractice #LearningOptimization #EducationalDataMining #AIEnhancedTeaching #ResponsibleAIInEducation

Download the Book (PDF) Introduction A language model that answers in under a second on a phone, without a network connection, without sending a word of the user's data to anyone, is no longer a research demonstration. In 2026 it is a product requirement. Apple ships a roughly three-billion-parameter model inside its operating systems and exposes it to developers. Google ships Gemma models built to run on phones and single-board computers, and an on-device Gemini Nano through Android. Qualcomm, MediaTek, Intel, AMD and Apple all sell chips with dedicated neural accelerators, and every one of them markets its ability to run language models locally. Open-weight families from Alibaba's Qwen team, Google, Microsoft, Hugging Face, Liquid AI and others now include models at one, two, four and nine billion parameters that are useful enough to build on. Yet the gap between a model that exists and a model that runs well on a particular device is large, and most teams discover it the hard way. A four-billion-parameter model stored in the 16-bit format in which it was trained occupies about eight gigabytes. A mid-range phone may have eight gigabytes of memory in total, shared with the operating system, the camera pipeline, the browser and every other app the user has open. Even when the model fits, the speed at which it can produce text is governed less by how many arithmetic operations the chip can perform than by how quickly it can drag the model's weights out of memory, once for every token it generates. And the battery, the thermal envelope and the user's patience all set limits that a data-centre deployment never meets. Compression is how that gap is closed. Quantization stores each weight in fewer bits. Pruning removes weights, neurons, attention heads or whole layers. Distillation trains a small model to imitate a large one. Low-rank adaptation lets a compressed model be specialised, or repaired, by training a tiny fraction of its parameters. Each of these has a literature of its own, a toolchain of its own and a set of folk beliefs that are only partly true. The argument of this book This book argues one thing: compressing a language model for the edge is a budgeting problem, and it goes well only when it is treated as one. There are two budgets. The first is the device budget: bytes of memory, bytes per second of memory bandwidth, operations per second, joules per token, and milliseconds the user will tolerate. The second is the quality budget: how much of the original model's behaviour on your tasks you are prepared to lose. Every compression technique is a way of spending one budget to save the other, and each spends it at a different price, paid in a different currency: calibration data, GPU hours, training tokens, engineering effort, or hardware support. Treated this way, the field becomes much easier to navigate. The order of operations stops being mysterious. You choose a base model whose architecture already respects the device budget. You remove bytes in the cheapest places first, which almost always means weight quantization to about four bits with a calibration-aware method. You spend training compute — distillation, quantization-aware training, low-rank recovery — only when cheap methods have hit their limit and only on the capability you need. You choose pruning when the hardware can actually exploit what you remove, which is less often than the papers suggest. And you measure quality on the tasks the product depends on, because the most common way to fail is to ship a model whose perplexity looks fine and whose behaviour on a long, structured, multilingual or tool-calling task has quietly collapsed. A second, less obvious claim runs alongside the first. The bottleneck at the edge is almost never arithmetic. It is memory: its capacity, which decides whether the model loads at all, and its bandwidth, which decides how fast the model speaks. That single fact explains why four-bit weights roughly double generation speed over eight-bit weights on the same chip, why unstructured sparsity so rarely makes a model faster on a phone, why the key–value cache becomes the dominant cost in long conversations, and why the neural processing units that dominate marketing material often matter less for chat-style generation than the humble width of the memory bus. Readers who keep that fact in view will make fewer expensive mistakes than readers who memorise the names of a dozen algorithms. Who this is for The reader this book has in mind is technically literate but not necessarily a specialist in model compression: an engineer asked to make a model run on a product, a technical lead deciding whether to build on an on-device model or a cloud API, a researcher in an adjacent field who needs a working map, or a product manager who wants to understand why the team keeps saying "it fits but it's too slow." Familiarity with what a neural network is, and roughly how a transformer works, will help. Nothing beyond that is assumed. Every technical term is defined when it first appears, and a glossary at the end collects the ones that recur. The book is practical without being a manual. Tools change month by month, and any book that tried to be a command reference would be stale before it was printed. What changes much more slowly is the reasoning: why a particular method works, what it costs, where it breaks, and how to tell whether it has broken. Where specific tools matter — llama.cpp and its GGUF format, Apple's MLX and Core ML, PyTorch's ExecuTorch, Google's LiteRT, Microsoft's ONNX Runtime, the quantization libraries built around GPTQ and AWQ — they are described as they stood in late 2026, with an eye to the design decisions that will outlast any particular version. How the book is arranged The chapters build in the order a practitioner actually meets the problems. The first establishes the arithmetic of the edge: where the bytes and the milliseconds go when a transformer generates text on a phone or laptop, and why memory bandwidth dominates. The second covers the choice of starting point, because the cheapest compression is the one you avoid by picking a model whose architecture was designed for small devices. Four chapters then take quantization apart. The first lays out the fundamentals: number formats, scales, granularity, and the outlier problem that makes language models harder to quantize than the vision networks that preceded them. The next covers post-training quantization of weights — GPTQ, AWQ, the k-quant and importance-matrix approach of llama.cpp, and the vector and lattice methods that push below four bits. Another turns to activations, the key–value cache, rotation methods and the new four-bit floating-point formats. A later chapter covers quantization-aware training and the sub-four-bit frontier, including ternary models. Pruning, distillation and low-rank adaptation each receive a chapter, and each is treated as a tool with a specific price rather than a magic word. The runtime and hardware chapter matches formats to chips, which is where many good compression results go to die. The evaluation chapter explains how to measure what compression has cost, using signals that are more sensitive than the benchmarks on a model card. The final chapter assembles everything into a blueprint: a sequence of decisions, with the questions to ask at each, for taking a model from a checkpoint to a device. A note on numbers. Compression research reports a great many results, and the benchmarks on which they are reported shift constantly. Where this book gives a figure, it is because the figure comes from a specific, named source or follows directly from arithmetic the reader can check. It does not reproduce leaderboard scores for particular models, which are often measured under incomparable conditions and are superseded within months. The reader who needs a current number for a decision should measure it on their own task, on their own device; the evaluation chapter explains how. The goal is that, by the end, the reader can look at a device and a task and reason, from first principles, about which model to start from, which bytes to remove first, what that will cost in quality, how to recover what was lost, and how to know whether the result is good enough to ship. Chapter 1. The Arithmetic of the Edge Before choosing a compression method it helps to know what, precisely, is being compressed and why. A language model running on a device consumes four resources: memory capacity, memory bandwidth, arithmetic throughput and energy. They are not equally scarce, and they are not consumed evenly across the life of a request. Most of the confusion in edge deployment comes from treating "the model is too big" as a single problem when it is really several, each with a different cure. Where the bytes live A transformer language model at inference time holds three kinds of data in memory. The first is its weights: the matrices that define the model. For a dense model the weight memory is simply the number of parameters multiplied by the bytes used to store each one. A model with four billion parameters stored in 16-bit floating point, the format in which most models are released, needs eight billion bytes, or about eight gigabytes. Stored at eight bits per weight it needs about four gigabytes; at four bits, about two; at two bits, about one. Real formats add a few percent of overhead for the scale factors that accompany quantized weights, which is why a "four-bit" file is usually closer to 4.5 bits per weight when measured honestly. The second is the key–value cache, usually shortened to KV cache. A transformer's attention mechanism lets each new token look back at every earlier token in the context. To avoid recomputing the representations of earlier tokens at every step, the runtime stores, for each layer, a key vector and a value vector for every token processed so far. The size of this cache grows linearly with the length of the context. It can be calculated exactly: two (for keys and values) multiplied by the number of layers, the number of key–value heads, the dimension of each head, the number of tokens, and the bytes per element. Take a concrete case shaped like many current small models: 28 layers, 8 key–value heads, a head dimension of 128, stored in 16-bit precision. Each token then costs 2 × 28 × 8 × 128 × 2 bytes, which is 114,688 bytes, or a little over 112 kilobytes. A conversation of 4,000 tokens therefore needs roughly 460 megabytes of cache; 32,000 tokens needs about 3.7 gigabytes. For a model whose quantized weights occupy two gigabytes, the cache for a long document can easily exceed the model itself. This is the first counterintuitive fact of edge deployment: for long contexts, compressing the weights further buys less than compressing, shrinking or bounding the cache. The third is activations and working buffers: the intermediate results of each layer, scratch space for matrix multiplications, and the logits over the vocabulary at the output. During token-by-token generation these are small, because only one token is processed at a time. During the initial processing of a long prompt they can be substantial, because the runtime processes many tokens together. Runtimes typically allocate these buffers once, sized to the maximum batch of prompt tokens they will handle. Add an operating system, a user interface and whatever else is running, and the practical memory budget for a model on a phone with eight gigabytes of RAM is perhaps two to three gigabytes before the system starts killing background apps or the model itself. On a twelve- or sixteen-gigabyte flagship it is more comfortable, but the model still competes with everything else. On a laptop with sixteen gigabytes of unified memory, a model of eight billion parameters at four bits is routine; at thirty-two gigabytes, much larger models become possible. Embedded boards and microcontrollers live in a different universe altogether, where hundreds of megabytes is a luxury. Table 1 sets out the weight memory for some common model sizes at common precisions. The figures are straightforward arithmetic — parameters multiplied by bits per weight, divided by eight — and they ignore the small overhead of scale factors, the KV cache and runtime buffers, all of which must be added on top. Table 1. Approximate weight memory by model size and storage precision (parameters × bits ÷ 8; excludes scales, KV cache and buffers). Parameters 16-bit 8-bit 4.5-bit (typical "4-bit" file) 2-bit 0.5 billion 1.0 GB 0.5 GB 0.28 GB 0.13 GB 1 billion 2.0 GB 1.0 GB 0.56 GB 0.25 GB 3 billion 6.0 GB 3.0 GB 1.7 GB 0.75 GB 4 billion 8.0 GB 4.0 GB 2.3 GB 1.0 GB 8 billion 16.0 GB 8.0 GB 4.5 GB 2.0 GB Two lessons follow. First, for phones, the useful range of models at four bits runs from well under one billion parameters up to about four billion, with eight billion feasible only on devices with generous memory and little else running. Second, the difference between eight and four bits is the difference between a three-billion-parameter model fitting comfortably on a mid-range phone and not fitting at all. Precision is the largest single lever on capacity. Why speed is a bandwidth problem Capacity decides whether a model loads. Bandwidth decides how fast it talks. Generation in a transformer proceeds in two phases. In the prefill phase the model reads the entire prompt — the system instructions, the user's message, any retrieved documents, the conversation so far — and computes the KV cache for all of it. Because all prompt tokens are available at once, the runtime can process them together as large matrix multiplications. Each weight fetched from memory is used for hundreds or thousands of tokens, so the work is dominated by arithmetic. Prefill is compute-bound. In the decode phase the model produces one token at a time. To produce each token it must run the entire network for that single token, and that means reading essentially every weight from memory — once per token. Each weight is fetched, multiplied by a single vector, and discarded until the next token. The arithmetic per byte fetched is tiny. Decode is memory-bandwidth-bound. This gives a simple, surprisingly accurate upper bound on generation speed. If a device can move B bytes per second from memory to its compute units, and the model's weights occupy W bytes, the model cannot produce more than about B ÷ W tokens per second, however powerful the processor. A phone memory system using LPDDR5X on a 64-bit bus at 8,533 million transfers per second has a theoretical peak of about 68 gigabytes per second; real sustained bandwidth for a single workload is lower. A two-gigabyte model on such a device therefore has a hard ceiling somewhere around thirty tokens per second, and achieved speeds below that ceiling are normal. A Raspberry Pi 5, with a 32-bit LPDDR4X interface at 4,267 million transfers per second, has a theoretical peak around 17 gigabytes per second; the same two-gigabyte model could not exceed about eight tokens per second there. Apple's own engineering notes on its M5 chip make the split explicit. When Apple's machine learning research team compared the M5 with the M4 using its MLX framework, it reported that time to first token — a prefill measurement — improved by between roughly three and four times, thanks to new matrix units in the GPU, while generation speed improved by about 19 to 27 percent, which it attributed to the M5's memory bandwidth rising from 120 to 153 gigabytes per second. The arithmetic units became several times faster; generation followed the memory bus. The consequences for compression are direct: • Halving the bytes per weight roughly doubles the decode ceiling. Moving from 8-bit to 4-bit weights does not just save memory; on bandwidth-bound hardware it is the most reliable speed-up available. • Arithmetic savings help prefill, not decode. Techniques that reduce the number of operations without reducing bytes fetched — including most forms of unstructured sparsity, discussed in the pruning chapter — rarely speed up generation on edge devices. • Dequantization overhead can matter. Because weights must be unpacked from their compressed form into numbers the hardware can multiply, an exotic format that saves bytes but requires expensive decoding can lose its advantage. A large part of the art of practical quantization is designing formats whose decoding is cheap enough that the bandwidth saving survives. • The KV cache is also read on every step. During decode, attention reads the cache for every previous token. For long contexts this read can rival the weight read in volume, which is why cache compression speeds up long-context generation as well as saving memory. A useful habit is to compute the bandwidth ceiling before measuring anything. If a runtime achieves 70 percent of the ceiling, further work on kernels has limited upside, and the only way to go faster is to move fewer bytes. If it achieves 20 percent, the problem is software — poor kernels, unfortunate memory layout, contention with other processes — and compressing the model further may not help. Latency the user feels Users do not experience tokens per second. They experience how long they wait before anything appears and how fast text arrives once it does. Time to first token is dominated by prefill. A conversational assistant with a long system prompt, a retrieval-augmented application that stuffs several documents into the context, and an agent that receives a long list of tool definitions all pay this cost before the first word appears. On a phone, prefilling a few thousand tokens on the CPU can take several seconds; moved to a GPU or NPU that can run dense matrix multiplications efficiently, it can fall well below a second. Because prefill is compute-bound, this is where neural accelerators earn their keep. It is also where caching helps: a system prompt that never changes can have its KV cache computed once and reused, which many runtimes support. Decode speed is perceived relative to reading speed. A rate above roughly ten tokens per second keeps up with most readers when text streams onto the screen. For tasks where the output is consumed by software rather than read — structured extraction, classification, tool calls — what matters is the total time to the final token, and both phases count. There is also sustained performance. A phone can run its processors at full power for a short burst before heat forces it to slow down. A benchmark that measures the first thirty seconds of generation may report numbers that a real user session, running for minutes with the screen on and the radio active, never sees. Edge deployment must be measured over realistic durations, at realistic ambient temperatures, on battery as well as on charger. Energy and the cost of moving data Energy follows the same logic as speed. Moving a byte from off-chip memory costs far more energy than performing an arithmetic operation on it once it has arrived; the precise ratio depends on the process technology and memory type, but the gap is large and well established in computer architecture research. A model that fetches half as many bytes per token therefore uses substantially less energy per token, not merely less time. For a battery-powered device this changes priorities. A cloud deployment can sometimes trade energy for latency by running larger batches; a phone serves one user and pays for every byte fetched. Formats that reduce memory traffic, runtimes that keep data in on-chip memory where possible, and accelerators that achieve high efficiency at low precision all compound into battery life. Hardware vendors now advertise performance per watt for their neural units precisely because sustained on-device inference has become a design target. The two budgets With this arithmetic in hand the problem can be stated cleanly. For a given device and application there is a device budget: a ceiling on weight memory (set by capacity minus everything else running), a ceiling on KV cache (set by the longest context the product must support), a floor on decode speed (set by user experience), a ceiling on time to first token, and an energy envelope. And there is a quality budget: a floor on how well the model must perform on the tasks the product depends on. Every compression technique in this book is a way of moving the model inside the device budget while staying above the quality floor. They differ in which part of the device budget they relieve and in what they cost. • Weight quantization relieves capacity and decode speed directly. It costs calibration data and some quality, increasing sharply below about four bits. • KV-cache quantization and architectural choices such as grouped-query attention relieve long-context memory and long-context decode speed. • Activation quantization relieves prefill compute on hardware with fast low-precision arithmetic, and is often required to use integer-only accelerators at all. • Structured pruning relieves everything at once — fewer parameters means fewer bytes, fewer operations and a smaller cache if attention heads or layers are removed — but costs quality that usually has to be recovered with training. • Unstructured pruning relieves capacity only if the runtime stores sparse weights compactly, and speed only on hardware that skips zeros, which most edge hardware does not. • Distillation produces a smaller model of a chosen architecture that is better than the same architecture trained from scratch; its cost is training compute and data. • Low-rank adaptation does not compress in itself, but it makes specialisation and quality recovery cheap enough to be practical on compressed models. The order in which to apply these is the subject of the last chapter. But the shape of the answer is already visible: methods that move fewest bytes for least quality, at the lowest engineering cost, come first. That almost always means choosing a well-designed small model and quantizing its weights to around four bits with a calibration-aware method. Everything after that is spending more to gain less, and must be justified by a specific gap between what the device budget demands and what the cheap methods deliver. What the arithmetic leaves out Three caveats keep this framework honest. First, mixture-of-experts models change the arithmetic. In such a model only a few "expert" sub-networks are activated for each token, so the bytes fetched per token reflect the active parameters while the memory required reflects all parameters. A model with tens of billions of total parameters but only three or four billion active per token can decode at the speed of a small dense model — if it fits. On a laptop with ample unified memory this is an attractive trade; on a phone, where capacity is the binding constraint, it usually is not. Some designs, such as Google's Gemma 4 "effective" models, place part of their parameters in large per-layer embedding tables that need only be looked up rather than multiplied, blurring the line between total and active size in a different way. When reading a model card, it pays to ask which number determines memory and which determines speed. Second, batching changes the arithmetic. If a device serves several requests at once — a laptop running background agents alongside a chat, or a gateway box serving several users — each weight fetched can be reused across the batch, and decode moves toward the compute-bound regime. Most phone deployments serve one request at a time, but speculative decoding, discussed in the runtime chapter, creates an internal batch of draft tokens to exploit exactly this effect. Third, the quality budget is not a single number. Compression degrades capabilities unevenly. Arithmetic, long-context retrieval, rare languages, strict output formats and multi-step tool use typically suffer before general conversational fluency does. A model can lose almost nothing on a broad knowledge benchmark while losing a great deal on the narrow behaviour a product depends on. The evaluation chapter returns to this at length, because it is where most compressed deployments that fail in the field actually fail. Chapter 2. Choosing the Starting Point The cheapest compression is the one you never have to perform. A model whose architecture was designed with small devices in mind, trained on enough data to make every parameter count, will usually beat a larger model hammered down to the same size. This chapter is about choosing that starting point well: what makes a small model good, what architectural features matter for edge deployment, and how to read the claims made for the many small open-weight models now available. Why small models got good For several years the default assumption was that capability came from scale: more parameters, more data, more compute. The 2022 "Chinchilla" study from DeepMind refined that assumption by showing that, for a fixed training budget, many large models of the time were undertrained — they would have been better as smaller models trained on more tokens. Chinchilla's recommendation, roughly twenty training tokens per parameter, was a recipe for minimising the cost of training a model of given quality. Edge deployment cares about something different: the cost of running the model, every time, on every device. If a model will be used billions of times, it is worth spending far more on training to make it smaller at inference. This logic, sometimes described as training "past Chinchilla-optimal," has driven the design of nearly every capable small model since. Meta's Llama 3 8B, released in 2024, was trained on more than fifteen trillion tokens — nearly two thousand tokens per parameter, a hundred times the Chinchilla ratio. Small models in the families that followed have continued the pattern. The result is that a model of three or four billion parameters in 2026 is typically far more capable than a model of the same size from three years earlier, and often more capable than much larger models from that period. Three other forces have contributed. Data quality has become a central research topic: Microsoft's Phi series, beginning with the 2023 paper "Textbooks Are All You Need," argued that carefully filtered and synthetic "textbook-quality" data lets small models punch above their weight. Distillation from larger teacher models, discussed in its own chapter, has become standard: Google described its Gemma 2 small models as trained with knowledge distillation, and Meta built its Llama 3.2 1B and 3B models by pruning the 8B model and then training with logits from the 8B and 70B models as targets. And post-training — instruction tuning, preference optimisation, reinforcement learning on verifiable tasks — has become far more effective, with reasoning abilities increasingly transferred into small models through distillation of long chains of thought from large reasoning models, as DeepSeek did when it released distilled versions of its R1 model in early 2025. The practical consequence is that a team building for the edge in late 2026 has an unusually rich menu. Alibaba's Qwen team released a Qwen3.5 small series in early 2026 at 0.8, 2, 4 and 9 billion parameters under the Apache 2.0 licence, with native vision input and a hybrid architecture discussed below. Google released Gemma 4 in April 2026, including E2B and E4B models aimed at phones and small boards, also under Apache 2.0. Microsoft's Phi-4-mini sits at 3.8 billion parameters. Hugging Face's SmolLM line publishes not just weights but training recipes and data. Liquid AI's LFM2 and LFM2.5 families target on-device use with non-transformer components and sizes around one to three billion parameters, and Liquid AI reported its LFM2.5 1.2B instruct model generating dozens of tokens per second on a current Samsung phone. Meta's Llama 3.2 1B and 3B remain widely deployed, and Apple's own on-device model — not openly downloadable, but available to developers through Apple's Foundation Models framework — is a roughly three-billion-parameter model built for its silicon. This menu will have changed by the time this book is read. The principles for choosing from it will not. Architecture features that matter on a device Two models with the same parameter count can differ substantially in how well they run at the edge. The differences lie in a handful of design decisions. Grouped-query attention. In the original transformer, each attention head has its own key and value projections, so the KV cache scales with the number of heads. Multi-query attention, proposed by Noam Shazeer in 2019, shares a single key and value head across all query heads; grouped-query attention, introduced by Ainslie and colleagues in 2023, shares each key–value head across a group of query heads. A model with 32 query heads and 8 key–value heads has a KV cache a quarter the size of the same model with full multi-head attention. Nearly every recent small model uses grouped-query attention, but the ratio varies, and it determines how much context fits in a given memory budget. When comparing models for long-context use, compute the per-token cache cost from the configuration file rather than trusting a headline context length. Hybrid and linear attention. Standard attention's cost grows with context length in both compute and cache. A number of recent designs replace most attention layers with recurrent or linear-attention layers whose state is a fixed size regardless of how many tokens have been processed, keeping only a few full-attention layers for precise retrieval. Qwen3.5's small models use this approach: their published configuration alternates three Gated DeltaNet linear-attention blocks with one full gated-attention block, so only a quarter of the layers maintain a growing KV cache. Liquid AI's models combine short convolutions with a minority of attention layers. For edge deployment this is valuable — a long conversation costs far less cache — but it carries a tooling cost: runtimes and NPU compilers must support the new layer types, and quantization methods tuned on standard transformer layers may not transfer cleanly to recurrent state updates. Always confirm that your target runtime supports the architecture efficiently before committing to it. Sliding-window attention. Another way to bound the cache is to let most layers attend only to a recent window of tokens, with a few global layers attending to everything. Google's Gemma models have used interleaved local and global attention for this reason. The effect on memory is similar to hybrid designs, and support in runtimes is now common. Depth versus width. At a fixed parameter budget, a model can be wide and shallow or narrow and deep. Meta's MobileLLM work, presented at ICML 2024, argued that for sub-billion-parameter models, depth matters more than conventional scaling laws suggested — that "deep and thin" architectures with weight sharing and grouped-query attention outperform wider ones of the same size. Depth has a cost at inference, because layers execute sequentially and each layer carries fixed overheads, but for very small models the quality gain usually dominates. Embedding and vocabulary size. Modern multilingual tokenisers have vocabularies of 150,000 to more than 250,000 tokens. The embedding matrix that maps tokens to vectors, and the output matrix that maps the final hidden state back to token probabilities, each have one row per vocabulary entry. For a model with a hidden size of 2,048 and a vocabulary of 256,000, each such matrix holds over half a billion parameters. In a one-billion-parameter model the embeddings can therefore be a large fraction of the total. Several design responses exist: tying the input and output embeddings so the same matrix serves both (common in small models), quantizing embeddings separately (Apple reports quantizing its on-device model's embedding table to 4 bits while its other weights are at 2), and keeping embeddings in slower storage since only one row per token needs to be read on input. Gemma 4's E-series models use per-layer embedding tables that hold a substantial share of their parameters; Google describes the E2B model as having about 2.3 billion "effective" parameters doing the expensive multiplication work. The vocabulary also affects compression in a subtle way: the output projection is often the most quantization-sensitive matrix in the model, and many quantization recipes keep it at higher precision. Native multimodality. Many small models now accept images, and some accept audio. Vision encoders add parameters and, more importantly, add tokens: a single image can become hundreds of tokens in the context, raising both prefill cost and cache size. Encoder weights often quantize differently from language-model weights. If the product needs only text, a text-only variant or a runtime that skips loading the encoder saves memory. Licences and provenance The legal terms attached to open-weight models vary more than their marketing suggests. Apache 2.0 and MIT licences impose few restrictions. Other licences add acceptable-use policies, restrictions on very large deployments, attribution requirements, or terms on using the model's outputs to train other models. Google moved Gemma 4 to Apache 2.0 after earlier Gemma versions shipped under a custom licence; Meta's Llama models carry Meta's own community licence. Before building a product, read the licence of the specific checkpoint, and note that derived works — quantized, pruned, fine-tuned — usually inherit it. Distillation from a teacher model raises its own question: some licences restrict using outputs to train competing models, which may matter if the plan is to distil from a proprietary model. Provenance matters for another reason. Quantized checkpoints on public model hubs are often produced by third parties. Most are made in good faith with standard tools, but they vary in method, calibration data and care. A quantized file whose creator used a poor calibration set or an outdated tool can be measurably worse than one produced carefully from the same base. For anything that ships, produce your own quantized artefacts from official weights, or at least verify third-party ones against your own evaluation. Reading a model card honestly Model cards report scores on public benchmarks. These numbers are useful for a rough ranking and dangerous for anything finer. Several cautions apply. Benchmarks are measured under different conditions — number of examples in the prompt, whether "thinking" mode is enabled, which answer-extraction script is used — and small differences in these settings can move scores by several points. Comparisons across model cards from different vendors are rarely apples to apples. Public benchmarks also leak: their questions appear on the web and can end up in training data, inflating scores in ways that do not transfer to new tasks. And benchmark scores are almost always reported for the unquantized model. The compressed model you will ship may behave differently, and different architectures tolerate compression differently. The model card is therefore a shortlist generator, not a decision. The decision comes from running two or three candidate models, compressed the way you intend to ship them, on an evaluation set drawn from your own product, on your target device. Building that evaluation is the subject of a later chapter, and it is worth doing before choosing a base model, not after. A selection procedure Combining the device arithmetic of the previous chapter with these considerations gives a practical procedure. Start from the device budget. Subtract the operating system and the rest of your application from available memory to obtain a ceiling for model weights plus cache. Decide the longest context the product must support and compute the cache cost per token for each candidate from its configuration. Decide the minimum acceptable decode speed and divide the device's sustained bandwidth by it to obtain a ceiling on bytes read per token. From these, derive a ceiling on parameters at roughly 4.5 bits per weight. That is the size class to shop in. Resist the temptation to pick the largest model that might fit at three or two bits; the quality loss below four bits rises steeply for post-training methods, and a smaller model at four bits is usually the better bet unless you are prepared to invest in quantization-aware training. Within the size class, prefer models that use grouped-query attention with a generous query-to-key ratio, or a hybrid design if your runtime supports it; that have a licence compatible with your product; that offer an instruction-tuned version close to your use case; and that have strong support in your target runtime. Check whether the vendor publishes quantization-aware-trained checkpoints — Google released quantization-aware-trained Gemma 3 checkpoints in 2025, and Meta released quantized Llama 3.2 1B and 3B variants trained with quantization-aware training plus low-rank adapters, and with the SpinQuant rotation method, in late 2024. Such checkpoints often beat anything you can produce with post-training methods. Then evaluate. Quantize each candidate the way you intend to ship it and measure on your own tasks. Very often the ranking changes: a model that led on its card loses more under quantization than its rival, or handles your output format worse, or proves weaker in a language your users need. When to compress a larger model instead There are circumstances in which starting from a larger model and compressing aggressively is the right choice. One is when the target capability is simply absent in available small models — a specialist domain, a rare language, a reasoning skill — and present in a larger model. Then structured pruning and distillation from the larger model to a custom small architecture may be the only route. NVIDIA's Minitron work, published in 2024, showed that pruning a fifteen-billion-parameter model to eight and four billion parameters and retraining with distillation could produce models competitive with similarly sized models trained from scratch, using a small fraction of the training tokens. That is an option for organisations with training infrastructure; it is not a weekend project. Another is when the device is not really an edge device in the phone sense: a workstation-class laptop with 64 or 128 gigabytes of unified memory, or an industrial box with a large GPU. There, a mixture-of-experts model with a large total size but a small active size may be the best trade, and aggressive quantization of the expert weights becomes the main lever. A third is when the vendor already did the compression for you, as with quantization-aware-trained checkpoints of a larger model that fit your budget. That is often the best of both worlds. For everyone else, the advice is simple. Choose the best model you can find in the size class your device budget allows, and spend your effort on quantizing it well, measuring it honestly and specialising it to your task. The following chapters explain how. Chapter 3. Quantization from First Principles Quantization is the workhorse of edge deployment. Almost every model that runs on a phone or laptop today is quantized, and for most deployments quantization is the only compression applied. It is also the technique most often applied carelessly, because the tools make it look like a single command. This chapter builds the ideas from the ground up — number formats, scales, granularity, and the peculiar statistics of language-model activations — so that the methods in the following chapters can be understood as answers to specific problems rather than as recipes. Numbers with fewer bits A trained model's weights are stored, by default, as 16-bit floating-point numbers, usually in the bfloat16 format, which keeps the eight exponent bits of a standard 32-bit float but only seven bits of mantissa. A floating-point number spends its bits on a sign, an exponent that sets its order of magnitude, and a mantissa that sets its precise value within that order of magnitude. This gives a huge dynamic range: bfloat16 can represent numbers from about 10⁻³⁸ to 10³⁸. Neural network weights do not need that range. Within a single weight matrix, values typically cluster around zero in a roughly bell-shaped distribution, with a spread spanning a few orders of magnitude at most. Quantization exploits this by mapping the weights onto a much smaller set of representable values. The simplest scheme is uniform integer quantization. Choose a number of bits, say four, giving sixteen possible integer codes. Choose a scale s, a real number. A weight w is stored as the integer q = round(w ÷ s), clipped to the representable range, and recovered approximately as w' = q × s. With symmetric quantization the integer range is centred on zero (for example −8 to 7 for four bits). With asymmetric quantization a zero-point z is added, so w' = (q − z) × s, allowing the grid to cover a range that is not centred on zero — useful when values are skewed, as activations after some nonlinearities are. The error introduced is the difference between w and w'. For uniform quantization it is at most half a step, s ÷ 2, for values inside the range, and potentially large for values clipped at the edges. Everything else in this chapter is about choosing s, z and the set of representable values so that the errors that matter are small. Floating-point formats with few bits are an alternative. An 8-bit float, in the E4M3 variant with four exponent bits and three mantissa bits, or the E5M2 variant with five and two, spaces its representable values logarithmically, densely near zero and sparsely far from it. That matches bell-shaped distributions better than an even grid does, and FP8 has become standard for training and inference on data-centre GPUs since NVIDIA's Hopper generation. At four bits, the E2M1 layout — one sign bit, two exponent bits, one mantissa bit — has only eight magnitudes, ranging from 0 to 6: 0, 0.5, 1, 1.5, 2, 3, 4 and 6. On its own that is far too coarse to represent a weight matrix, which is why four-bit floating-point formats always come paired with per-block scales, discussed below. Non-uniform codebooks go further. Instead of a fixed grid, store a small table of, say, sixteen representative values chosen to fit the actual distribution, and store each weight as an index into the table. The NF4 ("4-bit NormalFloat") format introduced with QLoRA in 2023 uses a fixed codebook whose sixteen values are spaced according to the quantiles of a normal distribution, on the theory that normalised weights are roughly normally distributed. Vector quantization, discussed in the next chapter, stores indices into a codebook of short vectors rather than single values. Scales and granularity The single most consequential choice in quantization is how many weights share one scale. With per-tensor quantization, one scale covers an entire weight matrix. It is cheap — one number per matrix — but disastrous at low bit-widths, because a single large value forces a large scale, and then the many small values all round to the same few codes. With per-channel quantization, each output row of a weight matrix gets its own scale. This was the standard for 8-bit integer inference on mobile hardware in the convolutional-network era and works well at eight bits. With group-wise or block-wise quantization, each contiguous block of weights — typically 32, 64 or 128 values along the input dimension — gets its own scale, and sometimes its own zero-point or offset. Group-wise quantization is what makes four-bit weight storage work for language models. The cost is overhead: a 16-bit scale for every 32 weights adds half a bit per weight, which is why "four-bit" formats usually land between 4.25 and 5 bits per weight in practice. The llama.cpp project's "k-quant" formats reduce this overhead with a two-level scheme in which blocks of weights are grouped into larger super-blocks, and the block scales are themselves quantized relative to a super-block scale. The four-bit floating-point formats standardised in 2023–2025 make granularity explicit in their names. The MXFP4 format, part of the Open Compute Project's Microscaling specification, stores blocks of 32 E2M1 values sharing one 8-bit power-of-two scale (an E8M0 value, which is all exponent and no mantissa). NVIDIA's NVFP4, introduced with its Blackwell GPUs, uses smaller blocks of 16 values with an FP8 E4M3 scale per block, plus a 32-bit floating-point scale per tensor. The smaller blocks and finer-grained scale make NVFP4 more accurate than MXFP4 at a slightly higher overhead. OpenAI released its gpt-oss models in 2025 with mixture-of-experts weights stored in MXFP4, which helped make the format a practical target for open runtimes. The general principle is that finer granularity reduces error at the cost of overhead bits and, sometimes, slower decoding. Most production formats settle on blocks of 16 to 128 weights. What makes language models hard Quantizing convolutional networks for mobile was largely a solved problem by 2020: eight-bit integers for weights and activations, per-channel weight scales, a little calibration, and accuracy losses of a fraction of a percent. Large language models broke this recipe, and understanding why is the key to everything that follows. The problem is activation outliers. In 2022, Tim Dettmers and colleagues, in the paper introducing the LLM.int8() method, reported that once transformer language models grow past roughly 6.7 billion parameters, a small number of hidden-state dimensions begin to carry values far larger than the rest — tens of times larger — and that these "emergent outlier features" appear systematically across most layers and most tokens. The outliers are few, a handful of dimensions out of thousands, but they are essential: zeroing them out destroys the model's performance. Outliers are fatal to naive activation quantization. If one dimension of the input to a matrix multiplication can reach a hundred while the rest stay below one, a per-tensor scale for that input must accommodate the hundred, and the small values — which carry most of the information — collapse to zero. LLM.int8() handled this by splitting the computation: the few outlier dimensions were multiplied in 16-bit precision and everything else in 8-bit. It worked, but the mixed-precision decomposition was slow. Later research added detail. Some outliers are tied to specific tokens, such as the beginning-of-sequence token or punctuation, and appear to function as "attention sinks" — places where attention heads park probability mass when they have nothing useful to attend to. A 2024 paper by Sun and colleagues described "massive activations" of this kind, values thousands of times larger than typical ones, concentrated in a few dimensions and positions. Weights have outliers too, though milder: a small fraction of weights have an outsized effect on output, and destroying them hurts disproportionately. Small models are not immune. Although the Dettmers paper located the emergence of systematic outliers at the multi-billion scale, models trained far past Chinchilla-optimal ratios, as small models now routinely are, often show quantization sensitivity that their size alone would not predict. A 2024 study by Kumar and colleagues on scaling laws for precision found that models trained on more data become harder to quantize after training — the more information is packed into each parameter, the more each rounding error costs. This is one reason why the best small models of 2026 are frequently shipped with vendor-produced quantization-aware checkpoints rather than left to post-training methods. Weights only, or weights and activations The outlier problem divides quantization into two families with different purposes. Weight-only quantization stores weights in a low-bit format and converts them back to 16-bit (or 8-bit) floating point just before multiplication; activations stay in higher precision. This saves memory and memory bandwidth — exactly what decode needs — without touching activation outliers. It is the dominant approach for edge generation, and it is what llama.cpp's GGUF formats, Apple's MLX quantization, and the GPTQ and AWQ methods primarily do. Weight-only quantization is described with notation such as W4A16: four-bit weights, sixteen-bit activations. Weight-and-activation quantization quantizes both inputs to each matrix multiplication, so the multiplication itself can run on low-precision integer or floating-point units: W8A8, W4A8 or W4A4. This speeds up compute-bound work — prefill and batched serving — and is often necessary to use neural processing units at all, since many NPUs have very fast integer pipelines and much slower floating-point ones. It requires dealing with activation outliers, which is the job of methods such as SmoothQuant and the rotation methods covered in a later chapter. A third target, the KV cache, has its own outlier structure — keys in particular tend to have outlier channels — and its own methods, also covered later. The edge practitioner therefore has to ask which resource is binding. For chat on a phone with a CPU or GPU, weight-only quantization addresses the bottleneck directly. For long-prompt applications where time to first token matters, or for deployment on an NPU, activation quantization becomes necessary too. Calibration and what to minimise The simplest way to quantize a weight matrix is round-to-nearest: compute a scale for each group from the group's minimum and maximum values, and round each weight to the nearest code. Round-to-nearest at eight bits is nearly lossless for most models. At four bits with small groups it is acceptable for many models but measurably worse than better methods. At three bits and below it usually fails. Better methods share an insight: the goal is not to make the quantized weights close to the original weights, but to make the quantized layer's outputs close to the original layer's outputs on realistic inputs. Two weights with the same rounding error can have very different effects, depending on how large the inputs that multiply them typically are. A weight attached to an input channel that is usually near zero can be rounded crudely; a weight attached to an outlier channel cannot. That requires knowing what realistic inputs look like, which is the role of calibration data: a small set of example texts — typically a few hundred sequences of a few thousand tokens — that are passed through the model to record activation statistics. Calibration is not training; no gradients flow through the model's weights. It is measurement. The choice of calibration data matters more than many practitioners expect. Older papers routinely calibrated on random excerpts from a web-text corpus or Wikipedia. For a chat model that will be used for code, or for a language other than English, or for a specific structured task, calibration on generic English prose under-represents the activation patterns that matter. The llama.cpp community's practice of building "importance matrices" from diverse calibration text, the use of instruction-formatted calibration data for chat models, and the inclusion of the target languages and domains all reflect this. A useful rule: calibration data should look like what the model will see in production, including its chat template, and should be diverse enough to activate the capabilities you care about. Two warnings about calibration. The first is overfitting: methods that optimise heavily against a small calibration set can produce a model that performs well on text like the calibration set and worse elsewhere. Evaluate on held-out data that differs from it. The second is that calibration statistics are only as good as their coverage of rare but important behaviours: if no calibration example exercises a tool-calling format, nothing in the quantization procedure protects the weights that implement it. Mixed precision Not all layers are equally sensitive. Across many models, the embedding and output projection matrices, the first and last few transformer blocks, and some attention projections are more sensitive than the large feed-forward matrices in the middle of the network. Practical recipes therefore mix precisions: llama.cpp's "Q4_K_M" format, for example, keeps certain tensors — parts of the attention value projections and feed-forward down projections in some layers — at higher precision than the nominal four bits, which is why it lands around 4.8 bits per weight on average for many models. Apple's on-device model keeps its embedding table at a different bit-width from its transformer weights. Sensitivity can be measured directly: quantize one layer at a time and observe the change in a quality signal such as perplexity on held-out text or divergence from the original model's output distribution. Some tools automate this, assigning more bits to sensitive layers within a fixed total budget. The gains are real but modest at four bits; at three bits and below, mixed precision becomes essential. A mental model of error It helps to think of quantization error as noise injected into each layer. Each matrix multiplication produces outputs equal to the true outputs plus an error term. Normalisation layers and residual connections dampen some of this noise, but errors accumulate across dozens of layers, and generation compounds them, because each generated token becomes input to the next step. A small per-layer error that barely moves a single-token benchmark can, over hundreds of generated tokens, steer the model into a different trajectory: a lost thread in a long answer, a broken JSON bracket, a slip into the wrong language. This explains a pattern familiar to anyone who has evaluated quantized models. Single-step metrics — perplexity, multiple-choice accuracy — degrade smoothly and modestly down to about four bits and then fall off a cliff. Multi-step behaviours — long-form generation, reasoning chains, agentic tool use — begin degrading earlier and less predictably. The rest of the quantization chapters are about reducing per-layer error in increasingly sophisticated ways; the evaluation chapter is about detecting the cumulative effects that per-layer metrics miss. Chapter 4. Post-Training Weight Quantization Post-training quantization (PTQ) compresses a trained model without further training. It needs only the model, a small calibration set and, for the better methods, a GPU for minutes to a few hours. Because it is cheap, it is the first thing to try and, for most edge deployments, the last thing needed. This chapter covers the main methods for quantizing weights after training: what each does, why it works, where it breaks, and how the practical file formats of the edge ecosystem relate to the research methods behind them. The baseline: round-to-nearest with groups Every method should be compared against the simplest one. Group-wise round-to-nearest (RTN) divides each weight row into groups of, say, 32 or 128 values, computes a scale (and possibly an offset) from each group's range, and rounds. It needs no calibration data and runs in seconds. At 8 bits, RTN is effectively lossless for nearly all language models, and 8-bit weight files, such as llama.cpp's Q8_0, are a sensible default when memory permits. At 6 and 5 bits with small groups it remains close to lossless. At 4 bits it works acceptably for many models but leaves quality on the table compared with calibrated methods, and the gap widens for small, heavily trained models. At 3 bits, RTN usually produces visibly degraded output; at 2 bits, gibberish. The methods below exist to push the "acceptable" boundary down by one or two bits. They do this by using information RTN ignores: how the inputs to each layer are distributed, and how errors in one weight can be compensated by adjustments to others. GPTQ: compensating errors as you go GPTQ, published by Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh in 2022 and presented at ICLR 2023, was the method that first made 4-bit and 3-bit weight quantization of very large language models practical. The authors reported quantizing models with 175 billion parameters in roughly four GPU hours. Its core idea comes from an older line of work on network pruning, the "Optimal Brain Surgeon" framework of the early 1990s, which asked: if I remove or perturb one weight, how should I adjust the remaining weights to minimise the change in the layer's output? For a linear layer and a squared-error objective, the answer depends on the second-order statistics of the layer's inputs, summarised in a matrix often called the Hessian, computed from calibration data as the sum of outer products of input vectors. GPTQ applies this one column at a time. It quantizes the first column of a weight matrix, measures the error introduced, and then updates all not-yet-quantized columns to compensate for that error, using the inverse Hessian to decide how much each should move. Then it quantizes the next column, and so on. Errors from early columns are absorbed by later ones. The algorithmic contributions of the paper were the engineering that made this tractable for billions of weights: processing columns in a fixed order shared across rows, batching updates in blocks for GPU efficiency, and a Cholesky-based reformulation for numerical stability. In practice GPTQ is used with group-wise scales (often groups of 128) and sometimes with "act-order," which quantizes columns in order of decreasing activation magnitude so that the most important columns are handled first, while later columns absorb their errors. GPTQ quantizes a small model in minutes on a single GPU. GPTQ's weakness is a tendency to overfit its calibration set. Because it actively adjusts weights to minimise error on the calibration inputs, a calibration set unrepresentative of real use can produce a model that is well tuned to the wrong distribution. Diverse, task-representative calibration data matters more for GPTQ than for methods that only choose scales. AWQ: protect the salient channels AWQ (Activation-aware Weight Quantization), from Ji Lin, Song Han and colleagues at MIT, was published in 2023 and received a best paper award at MLSys 2024. It starts from an observation: not all weights are equally important, and the important ones can be identified not by looking at the weights themselves but at the activations they multiply. Weights in input channels that typically see large activations matter far more to the output than weights in channels that see small ones. The authors reported that keeping roughly one percent of these salient weight channels in higher precision recovered much of the loss from 4-bit quantization. Mixed-precision storage is awkward for hardware, so AWQ instead uses a mathematical trick. Multiplying a weight channel by a factor greater than one and dividing the corresponding input activation by the same factor leaves the layer's output unchanged in exact arithmetic. But after quantization, the scaled-up weights occupy more of the quantization grid, so their relative rounding error falls. AWQ searches, per layer, for scaling factors that minimise output error on calibration data, folds the inverse scaling into the preceding operation (often a normalisation layer), and then quantizes with ordinary group-wise RTN. Because AWQ only chooses per-channel scales rather than rewriting individual weights, it is less prone to overfitting the calibration set than GPTQ and tends to generalise well across domains, including to instruction-tuned and multimodal models. It is also fast. AWQ and GPTQ are often combined or compared directly; the honest summary of the literature is that at 4 bits with group size 128 both are good, the winner varies by model, and both are better than RTN at the same bit-width. HQQ and calibration-free methods Sometimes calibration data is unavailable or calibration is undesirable — for example, when quantizing on the fly at load time. Half-Quadratic Quantization (HQQ), published by Mobius Labs in 2023, optimises scales and zero-points per group to minimise weight-reconstruction error under a robust loss, without any calibration data, and runs in seconds to minutes. It is better than plain RTN and competitive with calibrated methods at 4 bits for some models, though it lacks the activation awareness that makes AWQ and GPTQ stronger at lower bit-widths. The general lesson is that calibration-free methods are fine at 4 bits and above for robust models; below that, information about the input distribution becomes indispensable. The llama.cpp family: k-quants and importance matrices The most widely used quantized format at the edge is not a research paper's output but the GGUF file format of the llama.cpp project, created by Georgi Gerganov and maintained by a large open community. GGUF is a single-file container that stores the model's weights in a variety of quantized types along with its tokenizer, configuration and chat template. It is the native format of llama.cpp and of the tools built on it, such as Ollama and LM Studio. GGUF's quantization types fall into three generations. The legacy types, such as Q4_0 and Q8_0, are block-wise RTN with blocks of 32 weights and a 16-bit scale per block (Q4_1 adds an offset). Q8_0 remains a good near-lossless baseline. Q4_0 is coarse but has the advantage of very simple decoding, which has made it a favoured target for hardware-specific kernels on ARM CPUs and for some accelerator backends. The k-quants, contributed by Iwan Kawrakow in 2023, use super-blocks of 256 weights subdivided into smaller blocks, with block scales and minimums themselves quantized to a few bits. The types run from Q2_K through Q6_K, with suffixes S, M and L indicating mixes in which more sensitive tensors receive higher-precision types. Q4_K_M has become the community's default "good 4-bit" choice, averaging somewhat under 5 bits per weight once its mixed tensors are counted; Q5_K_M and Q6_K are near-lossless for most models. The i-quants (IQ1 through IQ4 types, also largely Kawrakow's work) bring ideas from the research frontier — lattice-based codebooks, discussed below — into the format, making 2- and 3-bit files usable for larger models. IQ4_XS, at about 4.25 bits per weight, often matches or beats Q4_K_M in quality at a smaller size, at the cost of somewhat more expensive decoding on some hardware. What lifts these formats above RTN is the importance matrix (imatrix). Running the model over a calibration text, llama.cpp records, for every weight matrix, the average squared magnitude of the activations reaching each input channel. During quantization, the rounding and scale selection for each block minimise an error weighted by these importances, so that weights in channels that see large activations are reproduced more faithfully. This is the same insight as AWQ, implemented as a weighted rounding objective rather than a scaling transform, and it makes a large difference at low bit-widths; the community's guidance is that the i-quants in particular should not be used without one. Two practical points follow. First, imatrix files vary in quality with their calibration text, and the community has iterated on diverse calibration sets that mix prose, code, multiple languages and chat-formatted data. Second, because GGUF quantizations are cheap to produce, model hubs host many variants of every popular model, produced by different people with different calibration data. They are not interchangeable, and a quality difference between two files with the same type name is entirely possible. Other ecosystems Different runtimes use different formats, and the methods above appear in several guises. MLX, Apple's array framework for its silicon, quantizes weights with group-wise affine quantization (scales and biases per group, commonly groups of 64) at 2 to 8 bits, and has added further types since; the MLX community publishes converted models on Hugging Face. MLX quantization has historically been closest to RTN, though calibrated methods such as AWQ-style scaling and learned rounding have been added to its tooling. Core ML, Apple's deployment framework, supports palettization (non-uniform codebooks of 1 to 8 bits, optionally per-group), linear quantization, and pruning, through its coremltools optimisation API. Palettization suits Apple's Neural Engine, which handles lookup-table weights efficiently. GPU-serving formats built around GPTQ and AWQ — produced with tools such as vLLM's llm-compressor and GPTQModel — target Linux machines with NVIDIA or AMD GPUs and are relevant at the edge mainly on workstation-class or embedded GPU boards such as NVIDIA's Jetson line. ExLlama's EXL2 and EXL3 formats allow fractional average bit-widths with per-layer mixing, popular among enthusiasts with consumer GPUs. Mobile and NPU toolchains — ExecuTorch with its backends, Google's LiteRT and its LLM runtime, Qualcomm's AI Engine tools, ONNX Runtime and its GenAI extension — generally implement group-wise integer weight quantization with calibrated methods, plus 8-bit or 16-bit activations where an accelerator requires them. The chapter on runtimes discusses how to match these. Below four bits: vectors and lattices At 4 bits, scalar methods with good calibration are nearly lossless for many models. Below about 3.5 bits, scalar quantization runs into a geometric limit: with only four or eight levels per weight, the grid is too coarse no matter how cleverly scales are chosen. Two ideas break through. The first is incoherence processing. QuIP, published by Chee, Cai, Kuleshov and De Sa in 2023, observed that quantization is easier when weights have no outliers and are spread evenly — "incoherent" in their terminology. It is possible to make them so by multiplying the weight matrix on both sides by random orthogonal matrices, which preserves the layer's function (the inverse rotation is applied to inputs and outputs) while spreading any large values across all entries. QuIP# (Tseng, Chee, Sun, Kuleshov and De Sa, ICML 2024) made this efficient using randomised Hadamard transforms, which can be applied quickly, and combined it with the second idea. The second is vector quantization. Instead of rounding each weight independently, group weights into short vectors — say, eight at a time — and replace each vector with the nearest entry in a codebook of representative vectors, storing only the index. At 2 bits per weight, an 8-dimensional vector needs a 16-bit index, so the codebook has 65,536 entries. Because the codebook can be shaped to the joint distribution of weights, vector quantization achieves lower error than scalar quantization at the same bit rate. QuIP# uses a codebook derived from the E8 lattice, a highly symmetric arrangement of points in eight dimensions known to give exceptionally dense sphere packings, which can be decoded quickly without storing the whole codebook. AQLM (Egiazarian and colleagues, ICML 2024) uses learned additive codebooks, representing each vector as a sum of entries from several codebooks, and fine-tunes them. QTIP, from the same Cornell group, uses trellis-coded quantization to go further still. These methods made 2-bit quantization of large models produce coherent, useful text for the first time, and they inspired the llama.cpp i-quants. For small models on edge devices their practical value is more limited, for two reasons. Decoding codebook-quantized weights is more expensive than unpacking scalar ones, which matters when the device is bandwidth-bound but has modest compute. And small models have less redundancy to begin with: a 2-bit eight-billion-parameter model and a 4-bit four-billion-parameter model occupy similar memory, and the evidence from the community's experience is that the larger model at 2 bits is not reliably better, while it is reliably more fragile on multi-step tasks. Where 2- and 3-bit weights are needed on phones, quantization-aware training, covered in a later chapter, is usually the stronger route, as Apple's 2-bit on-device model illustrates. Choosing among methods Table 2 summarises the main weight-only PTQ approaches along the dimensions that matter for an edge decision. The entries are qualitative and reflect the published descriptions of each method and common practice rather than a controlled benchmark. Table 2. Weight-only post-training quantization methods compared (qualitative, from the methods' original papers and common practice). Method Needs calibration Core mechanism Typical sweet spot Main caution Group-wise RTN No Round each group to its own scale 8–5 bits; acceptable at 4 Degrades fast below 4 bits HQQ No Optimise scales and zero-points per group 4 bits, quick at load time Lacks activation awareness GPTQ Yes Second-order error compensation across columns 4 and 3 bits Can overfit calibration data AWQ Yes Activation-aware per-channel scaling, then RTN 4 bits Gains shrink at very low bits llama.cpp k-quants with imatrix Yes (imatrix) Super-block scales, importance-weighted rounding Q4_K_M to Q6_K Variant quality depends on producer QuIP#, AQLM, QTIP, i-quants Yes Rotations plus vector or lattice codebooks 2 to 3 bits Decode cost; fragile on small models For most edge projects the decision collapses to a short rule. Start at 4-bit group-wise weights with a calibrated method native to your runtime — an imatrix-calibrated Q4_K_M or IQ4_XS for llama.cpp, AWQ or GPTQ for GPU runtimes, the vendor toolchain's calibrated mode for NPU targets. Evaluate. If quality is adequate and the model fits, stop. If memory is still too tight, first consider a smaller base model at 4 bits; only then consider 3 bits with mixed precision and careful calibration, and treat 2 bits as territory for quantization-aware training. If quality at 4 bits is inadequate, try 5 or 6 bits for the most sensitive layers, try a better calibration set, and consider low-rank recovery, covered in its own chapter. Above all, judge every quantized artefact by evaluation on your own tasks rather than by its type name. Two "Q4" files of the same model can differ more than a Q4 and a Q5 file, and the difference shows up first where your product is most demanding. Chapter 5. Activations, the KV Cache and Rotations Weight-only quantization addresses the decode bottleneck, but it leaves two costs untouched. The first is compute during prefill, which matters whenever prompts are long and time to first token is visible to users; it also matters whenever the target hardware is a neural processing unit whose fast pipelines operate on low-precision integers. The second is the key–value cache, which for long contexts can dwarf the weights in both memory and per-token traffic. Both problems run into the activation outliers described in the fundamentals chapter. This chapter covers the methods that tame them: smoothing, rotation, specialised cache quantization, and the new four-bit floating-point formats that hardware now supports directly. Why activations resist quantization Recall the structure of the problem. A linear layer computes Y = XW, where X is a matrix of activations (one row per token) and W is the weight matrix. To run this on integer hardware, both X and W must be quantized. Weights are fixed and can be quantized offline with care. Activations change with every input and must be quantized on the fly, usually with a scale computed from a calibration set (static quantization) or from the actual values at runtime (dynamic quantization). The trouble is that activations in language models have a few input channels with enormous values. With per-tensor or per-token activation scales — the granularities that keep integer matrix multiplication efficient — those channels set the scale, and everything else is crushed. Per-channel activation scales would solve the problem, but they cannot be factored out of the matrix multiplication, because the scale varies along the dimension that the multiplication sums over. The integer hardware would have to rescale each product individually, defeating the purpose. So the outliers have to be dealt with before quantization. There are three broad strategies: keep them in higher precision (as LLM.int8() did, at the cost of speed); shift them into the weights, where offline quantization can cope; or spread them out so that no channel stands out. SmoothQuant: moving the difficulty SmoothQuant, by Guangxuan Xiao, Ji Lin, Song Han and colleagues, presented at ICML 2023, takes the second route. It uses the same mathematical identity as AWQ: dividing an activation channel by a factor and multiplying the corresponding weight row by the same factor leaves the product unchanged. If a channel's activations are a hundred times larger than typical, dividing them by, say, ten and multiplying the weights by ten makes both tensors moderately hard to quantize instead of one being impossibly hard. The factor per channel is chosen by a "migration strength" parameter, usually written α, which balances how much difficulty is transferred: the scale is the activation's maximum magnitude raised to α, divided by the weight's maximum magnitude raised to 1 − α. The paper found α around 0.5 worked for many models, with some needing more. The scaling is folded offline into the preceding normalisation layer's parameters, so it costs nothing at inference time. SmoothQuant made W8A8 inference — 8-bit weights and 8-bit activations with integer matrix multiplication — accurate for large language models, roughly halving memory and speeding up compute on hardware with fast INT8 units. It became a standard component in data-centre inference stacks and in vendor toolchains for NPUs. Its limitation is the bit-width: at 4-bit activations, smoothing alone is not enough, because the remaining spread in both tensors exceeds what sixteen levels can represent. Rotations: spreading the outliers The third strategy — spread the outliers out — is where most of the progress since 2024 has come from. The idea, introduced for weights by QuIP and extended to activations by later work, is to multiply activations by an orthogonal matrix before quantization. An orthogonal transformation preserves lengths and angles; it can be undone exactly by multiplying by its transpose. If a vector has one huge entry and many small ones, a well-chosen rotation turns it into a vector whose entries are all of moderate size — the energy of the outlier is spread across every dimension. A vector with no outliers quantizes well. Crucially, in a transformer many such rotations can be absorbed into the weights. Consider a sequence of linear layers with a residual stream between them. If the residual stream is rotated by an orthogonal matrix Q, every weight matrix that reads from the stream can be pre-multiplied by Q's transpose and every matrix that writes to it post-multiplied by Q, and the network computes exactly the same function — provided the normalisation layers commute with the rotation, which RMSNorm (used in most modern models) does after a small adjustment that folds its scaling parameters into adjacent weights. The rotation then costs nothing at runtime. A few rotations cannot be absorbed — for example, inside the attention computation or before the feed-forward down projection — and must be applied online, but Hadamard matrices, which contain only +1 and −1 entries (scaled), can be applied with a fast transform costing a small fraction of a matrix multiplication. QuaRot, by Ashkboos and colleagues (2024), used randomised Hadamard rotations in this way to remove outliers from the residual stream, the attention inputs and the KV cache, and reported end-to-end 4-bit quantization of weights, activations and cache for Llama 2 models with modest quality loss. SpinQuant, from Meta researchers Zechun Liu and colleagues (2024, presented at ICLR 2025), showed that random rotations vary considerably in quality and that rotations can instead be learned, optimising them on calibration data on a mathematical surface that keeps them orthogonal. Meta subsequently used SpinQuant to produce one of its official quantized Llama 3.2 1B and 3B variants for mobile deployment, alongside a quantization-aware-trained variant. Rotation methods have three practical implications for edge work. First, they are compatible with weight-only methods: one can rotate and then apply GPTQ to the rotated weights, and this combination is among the strongest post-training recipes at 4 bits and below. Second, they make 4-bit activations — and therefore 4-bit integer or floating-point matrix multiplication — feasible, which matters on hardware with fast 4-bit units. Third, they need runtime support for the online rotations, which not every edge runtime provides; check before choosing a rotation-based checkpoint. Quantizing the KV cache The KV cache deserves its own treatment, because its statistics differ from those of weights and of other activations, and because for long contexts it is often the dominant cost. The key observation, documented in work such as KIVI (Zirui Liu and colleagues, ICML 2024) and KVQuant (Coleman Hooper and colleagues, NeurIPS 2024), is that keys and values have different outlier structures. Keys tend to have a few channels with consistently large magnitudes across tokens; values do not, but vary more from token to token. KIVI therefore quantizes keys per-channel and values per-token, and reported usable 2-bit caches with careful handling of the most recent tokens, which are kept in full precision until a group fills up. KVQuant added further refinements: quantizing keys before the rotary position embedding is applied (because rotary embeddings mix channels and smear the outliers in a way that hurts quantization), using non-uniform per-layer codebooks, and storing a small number of outlier values separately in sparse form. In mainstream edge runtimes, the practical options are more conservative. llama.cpp supports storing the cache in 8-bit (q8_0) and 4-bit types, configurable separately for keys and values. Apple reports quantizing its on-device model's KV cache to 8 bits. For most deployments, an 8-bit cache roughly halves cache memory with negligible quality loss; a 4-bit cache halves it again and is usually acceptable for values but riskier for keys, particularly at long context lengths, where small errors in attention scores across thousands of positions accumulate into misdirected attention. As always, test on long-context tasks specifically; a short-prompt benchmark will not reveal a cache-quantization problem. Quantization is not the only way to shrink the cache. Architectural measures — grouped-query attention, sliding-window layers, hybrid linear-attention designs, and the cross-layer KV sharing used in Apple's on-device model, in which a later block of layers reuses the cache produced by an earlier block — reduce the cache by design and were discussed in the chapter on choosing a model. Apple's 2025 technical report describes splitting its roughly three-billion-parameter on-device model into two blocks, with the second block, containing 37.5 percent of the layers, having its key and value projections removed and reading the first block's cache, cutting cache memory by 37.5 percent. Eviction methods keep only a subset of past tokens. The StreamingLLM work of Xiao and colleagues (2023) found that keeping the first few tokens — the attention sinks — plus a sliding window of recent tokens allowed models to run stably over very long streams, though it does not let the model recall information from evicted tokens. Methods such as H2O and SnapKV choose which tokens to keep based on accumulated attention scores. Eviction trades recall for memory and is risky for tasks that require retrieving a specific fact from a long document, which is precisely when long contexts are most needed. The right combination depends on the product. A chat assistant with conversations of a few thousand tokens can use an 8-bit cache and stop worrying. A document-question-answering app processing tens of thousands of tokens on a phone needs a model with an efficient attention architecture, a quantized cache, and careful testing of retrieval accuracy at the far end of the context. Four-bit floating point in hardware Until recently, low-precision inference on most hardware meant integers: INT8 everywhere, INT4 on some accelerators. Since 2024 the hardware landscape has shifted toward small floating-point formats with block scales, and this has changed what is practical. The microscaling formats introduced in the fundamentals chapter — MXFP8, MXFP6 and MXFP4 under the Open Compute Project specification, and NVIDIA's NVFP4 — store blocks of low-precision floating-point values with a shared scale per block. NVIDIA's Blackwell GPUs execute FP4 matrix multiplications natively, as do AMD's MI355X data-centre accelerators for MXFP4. The edge relevance comes through two routes. One is workstation and embedded GPUs: NVIDIA's consumer and embedded Blackwell parts support FP4 tensor operations, and the llama.cpp project merged NVFP4 support across several backends in the spring of 2026. The other is models released natively in these formats: OpenAI's gpt-oss models, released in August 2025, shipped their expert weights in MXFP4, making the smaller of them, with about 21 billion total parameters, runnable on machines with around 16 gigabytes of memory. Block-scaled FP4 has two advantages over INT4 for activations. The logarithmic spacing of floating-point values fits bell-shaped distributions better than a uniform grid. And the small blocks — 16 or 32 values — mean that an outlier only affects the scale of its own block rather than a whole row. Research from NVIDIA and others has shown NVFP4 retaining accuracy on large models at 4-bit weights and activations with appropriate calibration, and some groups have trained models directly in FP4 precision. On phones and NPUs the picture is more varied. Mobile NPUs have generally offered fast INT8 and INT4 (and in some cases INT16 and FP16) paths; support for micro-scaled floating-point formats is arriving but uneven, and each vendor's toolchain dictates which formats compile efficiently. Here, more than anywhere, the practical rule is to start from what the hardware accelerates and choose the quantization method to fit, rather than the other way around. Putting activation quantization to work For the edge practitioner, the decision whether to quantize activations reduces to three questions. Is prefill time a problem? If the application has short prompts and the main complaint is decode speed, weight-only quantization addresses the bottleneck and activation quantization adds risk for little gain. If prompts are long — retrieval-augmented generation, document processing, agents with large tool schemas — and time to first token is unacceptable, activation quantization on a compute-capable accelerator is the lever. Does the target accelerator require it? Many NPUs execute integer arithmetic far faster than floating point, and some execute only quantized integer graphs efficiently. Deploying to such an NPU means producing a W8A8, W8A16 or W4A8-style model in the vendor's format, with calibrated activation scales. The vendor toolchains — Qualcomm's AI Engine Direct and AI Hub, MediaTek's NeuroPilot, Apple's Core ML tools, Intel's OpenVINO, AMD's Ryzen AI software — each implement calibration routines, often including SmoothQuant-style smoothing, and each has its own list of supported patterns. How long are the contexts? If they are long, cache quantization and cache-efficient architecture move from optional to essential, and long-context evaluation must be part of the test plan. A common, robust configuration for phones in 2026 is 4-bit group-wise weights, 8- or 16-bit activations, and an 8-bit KV cache, with the most sensitive layers kept at higher precision. More aggressive settings — 4-bit activations with rotations, 4-bit or lower caches — can pay off, but each step down should be justified by a measured bottleneck and verified by an evaluation that would catch its specific failure mode. Chapter 6. Quantization-Aware Training and the Sub-Four-Bit Frontier Post-training methods ask the model to put up with quantization. Quantization-aware training (QAT) teaches the model to expect it. The model is trained, or fine-tuned, with quantization simulated in its forward pass, so that its weights settle into values that survive rounding. The cost is training compute and data; the reward is accuracy at bit-widths where post-training methods break down, and markedly better 4-bit models than post-training methods can deliver for the most heavily trained small models. This chapter explains how QAT works, how it has been used in shipped products, and what lies at the far end of the spectrum — models trained from the start with ternary weights. How simulated quantization trains The central difficulty is that rounding has no useful gradient. The function round(x) is flat almost everywhere, with jumps at the half-integers; its derivative is zero or undefined. If a training loop computes the loss through quantized weights, back-propagation reports that small changes to the underlying weights have no effect, and nothing learns. The standard workaround is the straight-through estimator (STE), described by Yoshua Bengio, Nicholas Léonard and Aaron Courville in 2013. In the forward pass the model uses quantized weights: each full-precision "shadow" weight is scaled, rounded, clipped and rescaled, a procedure often called fake quantization because the values are still stored in floating point but restricted to the quantization grid. In the backward pass the rounding is treated as if it were the identity, passing the gradient straight through to the shadow weight (usually with the gradient zeroed for values clipped outside the range). The shadow weights accumulate small updates; occasionally an update pushes a weight across a rounding boundary and its quantized value changes. Over many steps, the network finds a configuration whose quantized version performs well. The scales can be learned too. LSQ (Learned Step Size Quantization, Esser and colleagues, 2019) treats each quantizer's step size as a trainable parameter with its own gradient estimate, and variants of this idea are standard. Some methods learn rounding decisions directly: AdaRound (Nagel and colleagues, 2020) learns, for each weight, whether to round up or down by optimising a layer-wise reconstruction objective, sitting between pure post-training methods and full QAT. OmniQuant (Shao and colleagues, 2023) learns clipping ranges and equivalent transformations block by block on calibration data. Such "learned PTQ" methods use gradients but only a small calibration set and a few GPU hours, and they occupy a useful middle ground. The quality of QAT depends heavily on what the model is trained toward. Fine-tuning on raw text with the ordinary next-token loss can drift the model away from its instruction-tuned behaviour, especially if the QAT data differs from the original post-training data. The better practice, now widespread, is self-distillation: train the quantized model to match the output probability distribution of the original full-precision model on a broad set of inputs. The unquantized model is the teacher, its own quantized self the student. Because the teacher provides a full distribution over the vocabulary at every position rather than a single correct token, each training example carries far more signal, and the student is pulled toward preserving the teacher's behaviour rather than learning anything new. When Google released QAT checkpoints of its Gemma 3 models in April 2025, it described running QAT for about 5,000 steps using the probabilities from the non-quantized checkpoint as targets, and reported that this reduced the perplexity drop from quantizing to llama.cpp's Q4_0 format by 54 percent. The 4-bit Gemma 3 4B model's weights fell from 8 gigabytes in bfloat16 to 2.6 gigabytes. QAT in shipped products QAT has moved from research into the standard practice of model vendors, and several published cases show how it is used. Apple's on-device model. Apple's 2025 technical report on its foundation models describes compressing its approximately three-billion-parameter on-device model to 2 bits per weight using quantization-aware training, with the embedding table quantized to 4 bits through joint training with the base weights during QAT, and the KV cache stored at 8 bits. Apple further describes training low-rank adapters to recover quality lost during compression — a pattern discussed in the low-rank chapter. The decision to go to 2 bits reflects the device budget: a 3-billion-parameter model at 2 bits occupies well under a gigabyte for its transformer weights, which leaves room on an iPhone for everything else and roughly halves the bytes read per token relative to a 4-bit version. Meta's quantized Llama 3.2. In October 2024 Meta released quantized versions of its Llama 3.2 1B and 3B models intended for mobile deployment through ExecuTorch. One variant used QAT combined with low-rank adapters ("QLoRA" in Meta's description), trained in Meta's own post-training pipeline; the other used SpinQuant, a post-training rotation method. Both used a 4-bit group-wise scheme for transformer weights, 8-bit per-channel weights for the embedding and output layers, and 8-bit dynamic activation quantization. Meta reported that the QAT variant retained quality closer to the original than the post-training variant, consistent with the general pattern, while the SpinQuant variant offered a route for developers who wanted to quantize their own fine-tuned models without access to Meta's training data. Google's Gemma line. Beyond the Gemma 3 QAT checkpoints, Google's mobile-oriented releases have increasingly been accompanied by quantized checkpoints prepared by the vendor, and Gemma 4's small models were published with formats targeting phone deployment through LiteRT and Android's AICore. Reported memory figures for the smallest Gemma 4 model at 2-bit quantization on a Raspberry Pi 5 put it under 1.5 gigabytes. The pattern across vendors is consistent. Where the vendor controls the training pipeline and cares about the deployed quality, it uses QAT with self-distillation. Where it wants to give developers a recipe they can apply to their own fine-tunes, it offers a post-training method. Developers should take vendor QAT checkpoints when they exist and fit the target format, because reproducing them requires data and compute that most teams lack. Doing QAT yourself For teams that need a bit-width or format no vendor provides, or that have fine-tuned a model and want to quantize the result well, QAT is feasible with modest resources for small models. The essential ingredients are: A teacher: the full-precision model you want to preserve, typically your fine-tuned checkpoint. A data mix representative of the model's intended use: instruction-formatted conversations, the languages and domains of the product, examples of structured output and tool calling if those matter. The data need not be labelled; the teacher supplies targets. Several hundred million tokens is a typical scale reported for small models, but meaningful gains often appear far earlier, and the right amount is found by monitoring evaluation metrics. A quantization scheme that exactly matches the deployment format. This point is easy to get wrong and expensive when wrong. If the model will be deployed in a format with groups of 32 and a particular rounding and scale encoding, the fake quantization during training must reproduce that scheme exactly, including which tensors stay at higher precision; otherwise, the model learns to tolerate a different noise from the one it will meet. PyTorch's torchao library provides QAT tooling designed to export to ExecuTorch formats, and vendor toolchains provide their own. A training configuration with a low learning rate, since the goal is to nudge rather than retrain: typical setups use learning rates an order of magnitude below those of the original fine-tuning, and many recipes freeze some components, such as normalisation parameters. Some teams start with a few hundred steps of training with quantization disabled to settle the pipeline, then switch it on. Evaluating periodically on the product's own evaluation set is essential, both to decide when to stop and to catch regressions in capabilities not represented in the training mix. The cost is a few GPU-days for a model of a few billion parameters at 4 bits — more at lower bit-widths — which is a small fraction of the original training cost and well within reach of a serious product team. An alternative for teams with less compute is to combine post-training quantization with a short low-rank recovery fine-tune: quantize with GPTQ or AWQ, freeze the quantized weights, and train low-rank adapters in higher precision to absorb the quantization error, again distilling from the original model. The low-rank chapter treats this in detail. It is cheaper than full QAT and recovers much of the same quality, at the cost of carrying adapter weights at inference or merging them back (which re-quantizes and partly undoes the recovery unless done carefully). Below two bits: ternary and binary models At the far end of the spectrum lie models whose weights take only three values, −1, 0 and +1, or only two. Post-training methods cannot reach this point for useful language models; the weights must be trained in this form from the beginning, or nearly so. Microsoft Research's BitNet line has pursued this most visibly. The original BitNet paper (2023) trained transformers with binary weights. "The Era of 1-bit LLMs" (Shuming Ma and colleagues, 2024) introduced BitNet b1.58, so named because a ternary weight carries log₂ 3 ≈ 1.58 bits of information, and argued that ternary models trained from scratch could match full-precision models of the same size once the models were large enough. In April 2025 Microsoft released BitNet b1.58 2B4T, a 2-billion-parameter ternary model trained on 4 trillion tokens, with open weights, describing it as the first open-source native 1-bit language model at that scale and reporting performance on par with full-precision open-weight models of similar size. Its activations are quantized to 8 bits. Microsoft also released bitnet.cpp, an inference framework with CPU kernels designed for ternary weights. Ternary weights are attractive at the edge for two reasons beyond size. Multiplying by −1, 0 or +1 requires no multiplier: matrix multiplication becomes addition and subtraction, which can be done with very cheap logic and, in principle, very low energy. And memory per weight falls below 2 bits, giving a large reduction in bandwidth. The practical limitations are equally clear. Ternary models must be trained from scratch (or at least from an early checkpoint) with the ternary constraint in place, so a team cannot take an existing strong model and convert it; it depends on someone with substantial pre-training resources choosing to train one. The open ternary models available as of 2026 are few and small compared with the abundance of conventional small models. And the hardware advantage is largely theoretical on current devices: mainstream CPUs, GPUs and NPUs have no native ternary arithmetic, so runtimes implement ternary matrix multiplication with lookup tables or packed integer tricks. These can be efficient — bitnet.cpp's kernels exploit exactly such tricks — but the dramatic energy savings that ternary arithmetic promises require purpose-built silicon that does not yet exist in consumer devices. The ternary approach is therefore best seen as an important research direction and a signal of where hardware might go, rather than a default choice. For a product being built today, a well-trained conventional small model with 4-bit QAT, or 2-bit QAT where the vendor has done it, is more likely to deliver. Scaling laws for precision A question that runs underneath this whole chapter is whether it is better to have a larger model at lower precision or a smaller model at higher precision, for the same number of bytes. The answer has shifted as models have changed. Early studies in 2022 and 2023, working with models trained near Chinchilla-optimal ratios, generally found that for a fixed memory budget, 4-bit quantization of a larger model beat 8-bit or 16-bit storage of a smaller one, and Dettmers and Zettlemoyer's study of "k-bit inference scaling laws" (2023) concluded that 4 bits was close to optimal for zero-shot accuracy per bit across many model families. That finding underpinned the community's settling on 4-bit as the default. Later work complicated the picture. The 2024 "Scaling Laws for Precision" paper by Tanishq Kumar and colleagues modelled how both training and inference precision interact with model size and data, and reported that post-training quantization degrades models more the more data they were trained on — to the point where, past some amount of training data, additional pre-training data can make the quantized model worse. The authors' interpretation was that heavily trained models pack more information into each weight, leaving less slack for rounding. Since small edge models are among the most heavily overtrained models in existence, this is directly relevant: it predicts that small modern models will be more sensitive to aggressive post-training quantization than older or larger models, and that QAT — which lets the model adapt to its precision — becomes more valuable as training data grows. The practical upshot is consistent with what vendors actually do. Four bits remains the natural default for post-training quantization. Going lower with post-training methods is increasingly risky for the best small models. Going lower with QAT is viable, and is how the most aggressive on-device deployments — such as Apple's 2-bit model — have been built. Choosing a larger model at very low post-training precision over a smaller model at 4 bits is a bet that should be tested rather than assumed. Choosing between PTQ and QAT The decision can be framed in terms of the two budgets. If a calibrated 4-bit post-training quantization of a suitable model meets both the device budget and the quality floor, stop there; QAT is an expense without a reason. If the device budget demands fewer bits than post-training methods can deliver at acceptable quality — typically 3 bits or below — QAT, or a vendor's QAT checkpoint, is the route. If 4-bit post-training quantization fits the device budget but falls just short of the quality floor on the capabilities the product needs, there are three options in rough order of cost: improve the calibration data and keep sensitive layers at higher precision; train low-rank recovery adapters with self-distillation; or run full QAT. And if the model has been fine-tuned for the product, as most deployed models are, the quantization step should be planned as part of fine-tuning rather than bolted on after it, because the cheapest time to make a model robust to quantization is while it is already being trained. Chapter 7. Pruning: What Removal Actually Buys Pruning is the oldest idea in neural network compression and the one most often misunderstood at the edge. The idea is simple: many weights in a trained network contribute little, so remove them. Papers routinely report that half the weights of a large language model can be removed with modest loss of quality. Yet the number of edge deployments that ship an unstructured-pruned language model is tiny. The reason is the arithmetic of the first chapter: a removed weight saves something only if the storage format and the hardware can turn its absence into fewer bytes moved or fewer operations performed. This chapter separates the kinds of pruning by what they actually buy, explains the main methods, and identifies the cases where pruning genuinely belongs in an edge pipeline. Three kinds of removal Unstructured pruning sets individual weights to zero wherever they fall. It offers the most flexibility — the algorithm can remove exactly the least important weights — and therefore the least quality loss for a given fraction removed. But the zeros are scattered. A dense matrix with half its entries zero still occupies the same memory and takes the same time to multiply unless the runtime uses a sparse storage format (storing only non-zero values and their positions) and sparse kernels. Sparse formats carry index overhead: at 50 percent sparsity, the bytes saved on values are partly spent on indices, and the irregular memory access patterns of sparse multiplication run poorly on CPUs, GPUs and NPUs designed for dense blocks. In practice, unstructured sparsity below roughly 70 to 80 percent rarely speeds up inference on general-purpose edge hardware, and language models cannot usually be pruned that far without severe damage. Semi-structured pruning, most commonly the 2:4 pattern, requires that in every contiguous group of four weights, at least two be zero. NVIDIA introduced hardware support for this pattern with its Ampere GPUs in 2020: sparse tensor cores store only the two non-zero values per group plus a small 2-bit index for each, and execute the multiplication at up to twice the throughput of the dense equivalent. The pattern is a compromise — less flexible than unstructured pruning, so it costs more quality at the same 50 percent sparsity, but genuinely accelerated on supporting hardware. The catch for edge work is that support is concentrated in NVIDIA GPUs, including the embedded Jetson line; mainstream phone NPUs and CPUs have historically not accelerated 2:4 sparsity, although some vendors' newer accelerators advertise sparsity support of various kinds, which should be verified for the specific chip and toolchain. Structured pruning removes whole units: entire neurons in the feed-forward layers (rows and columns of the weight matrices), entire attention heads, entire channels of the hidden dimension, or entire transformer layers. The result is simply a smaller dense model, which every runtime and every accelerator runs efficiently. Structured pruning saves memory, bandwidth, compute and — if attention heads or layers are removed — KV-cache size, all at once. Its weakness is coarseness: removing a whole neuron removes good weights along with bad ones, so the immediate quality loss is much larger than for unstructured pruning at the same parameter count. Structured pruning nearly always requires retraining afterwards, usually with distillation, to recover. Table 4 sets the three side by side in terms of what each actually saves on a typical edge device. The entries summarise the hardware behaviour described above rather than any single benchmark. Table 4. What each kind of pruning saves on typical edge hardware (qualitative). Kind of pruning Memory saved Decode speed-up Hardware needed Recovery training Unstructured Only with sparse storage; index overhead Rarely, below very high sparsity Sparse kernels; uncommon at the edge Optional at moderate sparsity Semi-structured 2:4 Roughly a third to a half of weight storage Yes, on supporting GPUs NVIDIA sparse tensor cores, including Jetson Short fine-tune recommended Structured (width or depth) Proportional to units removed, plus KV cache if heads or layers go Yes, everywhere None beyond a dense runtime Substantial, usually with distillation The short version: unstructured pruning is a research tool and a component of some compression pipelines, semi-structured pruning is a deployment tool on specific hardware, and structured pruning is a way of manufacturing a smaller dense model. One-shot unstructured pruning: SparseGPT and Wanda The classic approach to pruning — magnitude pruning, removing the weights with the smallest absolute value, then retraining, repeated over several rounds — becomes prohibitively expensive for language models with billions of parameters. Two methods published in 2023 showed that large language models can be pruned in one shot, without retraining, with surprisingly small loss. SparseGPT, by Elias Frantar and Dan Alistarh (ICML 2023), applies the same second-order reasoning as GPTQ, from the same group, to pruning. Removing a weight is a perturbation; given the Hessian of the layer's reconstruction error, computed from calibration data, the remaining weights can be adjusted to compensate. SparseGPT processes columns in order, decides which weights to remove using an importance score that accounts for the Hessian, and updates the remaining weights to absorb the error. The paper reported pruning the largest open models of the time, with 175 billion parameters, to 50 to 60 percent unstructured sparsity in a few hours with little increase in perplexity, and showed that the method extends to 2:4 and 4:8 patterns and combines with quantization. Wanda ("Pruning by Weights and activations"), by Mingjie Sun, Zhuang Liu, Anna Bair and J. Zico Kolter (presented at ICLR 2024), asked whether the weight updates were necessary at all. Its importance score is simply the absolute value of each weight multiplied by the norm of the corresponding input activation across calibration data — the same activation-awareness that underlies AWQ — and weights are compared within each output row rather than across the whole layer. Wanda removes the lowest-scoring weights and performs no reconstruction. The authors reported results close to SparseGPT at 50 percent sparsity with much less computation, which made Wanda a common baseline. Both methods highlighted the same phenomenon as the quantization literature: the importance of a weight depends on the activations it meets, and the outlier channels that make activations hard to quantize also make their weights important to keep. The limitations matter for edge practitioners. Results reported for very large models are more forgiving than results for small ones; small, heavily trained models have less redundancy, and the same sparsity costs them more. Perplexity, the usual headline metric, understates damage to harder capabilities, and subsequent evaluations found that one-shot-pruned models lost more on knowledge-intensive and reasoning tasks than perplexity suggested. And 50 percent unstructured sparsity, as already discussed, does not translate into a faster model on a phone. The main practical uses of these methods at the edge are as the 2:4 variant on hardware that accelerates it, as a component in a combined sparse-and-quantized pipeline on a runtime with sparse kernels, or as a step in research. Structured pruning of width and depth If unstructured pruning buys little at the edge, structured pruning buys a great deal — at a price. Several lines of work have mapped that price. Width pruning removes neurons from feed-forward layers, heads from attention layers, or channels from the hidden dimension. LLM-Pruner (Xinyin Ma, Gongfan Fang and Xinchao Wang, NeurIPS 2023) identified groups of coupled structures — a neuron's row in one matrix and the matching column in the next must be removed together — scored them with gradient-based importance estimates, removed the least important, and recovered quality with a short low-rank fine-tune. SliceGPT (Ashkboos and colleagues, ICLR 2024) used the rotation invariance described in the activations chapter: rotate the residual stream with principal component analysis so that information concentrates in a subset of dimensions, then delete the remaining rows and columns, shrinking the hidden dimension. Sheared LLaMA (Mengzhou Xia and colleagues, ICLR 2024) learned pruning masks toward a specified target architecture and then continued pre-training, producing 1.3- and 2.7-billion-parameter models from a 7-billion-parameter one with far less compute than training from scratch. Depth pruning removes entire layers. Several studies in 2024 — among them "ShortGPT" by Men and colleagues and "The Unreasonable Ineffectiveness of the Deeper Layers" by Gromov and colleagues — found that many middle and late layers in large language models change the residual stream only slightly, as measured by the similarity between a layer's input and output, and that a substantial fraction of layers can be removed with surprisingly little loss on some benchmarks, especially after a brief recovery fine-tune. Layer removal is attractive at the edge because it reduces latency directly: fewer sequential steps per token, fewer weights to fetch, a smaller KV cache. But the same studies, and later evaluations, found that the losses concentrate in capabilities that need deep computation — multi-step reasoning, some kinds of knowledge recall — and that benchmarks dominated by easy multiple-choice questions can hide them. NVIDIA's Minitron work ("Compact Language Models via Pruning and Knowledge Distillation," Muralidharan and colleagues, 2024) is the most complete practical recipe published for structured pruning, and it is worth describing in some detail because it shows what a serious pipeline looks like. Starting from a trained model, it estimates the importance of each attention head, feed-forward neuron, embedding channel and layer from activation statistics on a small calibration set. It then prunes along one or more axes to a target size, evaluates candidate architectures briefly, and retrains the chosen one using knowledge distillation from the original model as teacher. The authors compared width pruning with depth pruning and reported that, after retraining, width pruning generally retained more quality, while depth pruning gave larger speed-ups for the same parameter count. They reported that their pruned-and-distilled models required many times fewer training tokens than training equivalent models from scratch, and the resulting Llama-3.1-Minitron and Mistral-NeMo-Minitron models were released openly. Meta's Llama 3.2 1B and 3B models were themselves produced, according to Meta, by structured pruning of Llama 3.1 8B followed by distillation from larger Llama models. The pattern across all of these is clear. Structured pruning is not a cheap post-processing step but the first half of a two-step process whose second half is substantial retraining with distillation. It is a way to manufacture a custom-sized model from a larger one more cheaply than training from scratch. It is the right tool when no available model fits the device budget with acceptable quality and the organisation has the training resources to retrain; it is the wrong tool for a team hoping to shave 20 percent off an existing small model in an afternoon. Pruning the parts nobody else prunes Some structured pruning targets are especially effective for small edge models and are often overlooked. The vocabulary. As discussed in the chapter on choosing a model, embedding and output matrices with vocabularies of 150,000 to 260,000 entries can be a large fraction of a small model's parameters. If a product serves only a few languages, many vocabulary entries — tokens for scripts and languages it will never see — are dead weight. Trimming the vocabulary to the tokens that appear in a large sample of the product's expected text, and remapping the tokenizer accordingly, can shrink the embedding and output matrices substantially and speed up the final projection, which for small models is a noticeable share of per-token compute. This must be done carefully: the tokenizer's merge rules must remain consistent, rare but legitimate inputs must degrade gracefully rather than crash, and the change makes the model incompatible with standard tooling that expects the original tokenizer. It is most appropriate for single-purpose, single-language deployments. Experts in a mixture-of-experts model. In a mixture-of-experts model, some experts are rarely selected for the product's inputs. Expert pruning removes them, or merges similar ones, reducing total memory while leaving the per-token active size unchanged. Several research groups have reported that a significant fraction of experts can be dropped for domain-specific deployments with limited loss. For edge devices where mixture-of-experts models are attractive because of their low active size but too large in total, expert pruning is one of the few ways to fit them. Modalities and encoders. A multimodal small model used only for text can drop its vision and audio encoders entirely, which many runtimes support simply by not loading them. This is the cheapest pruning there is. Combining pruning with quantization Pruning and quantization are complementary in principle: one reduces the number of weights, the other the bits per weight. In practice the combination behaves differently for different kinds of pruning. Structured pruning composes cleanly with quantization. The pruned, retrained model is an ordinary dense model, and it can be quantized with any method in this book. The only caution is that a pruned-and-distilled model has, by construction, less redundancy than its parent, and may be somewhat more sensitive to quantization; it should be evaluated after both steps, not just after pruning. Unstructured and semi-structured sparsity compose less cleanly. Both pruning and quantization introduce error in the same layers, and their errors add. SparseGPT and later work showed that joint pruning and quantization can be performed in a single pass with error compensation, and some runtimes — notably those targeting NVIDIA hardware with 2:4 support — can execute sparse 4-bit kernels. On phones and laptops without such support, sparse weights in a quantized model typically save nothing. A decision rule Pruning earns a place in an edge pipeline under three conditions. The first is when the hardware accelerates the sparsity pattern, as with 2:4 sparsity on NVIDIA GPUs, including embedded boards, and the throughput gain is needed. Then 2:4 pruning with SparseGPT or a similar method, followed by a short recovery fine-tune, is a legitimate optimisation. The second is when the model must be smaller in every dimension than any available model of acceptable quality, and the organisation can afford retraining. Then structured pruning plus distillation, in the Minitron style, is a sound way to create the needed model. The third is when there are obvious unused components — languages outside the product's scope, experts never selected, encoders for unused modalities — whose removal is cheap and whose loss is easily tested. Outside these conditions, the effort is better spent on choosing a smaller base model, quantizing it well and specialising it to the task. A pruning result that reports a high sparsity percentage with modest perplexity loss should prompt two questions before anyone acts on it: will this run faster or smaller on my device, and what happened to the capabilities my product depends on? Chapter 8. Distillation: Teaching Small Models to Think Like Large Ones Quantization and pruning take an existing model and make it cheaper. Distillation is different in kind: it trains a new, smaller model — or retrains a compressed one — to imitate a larger "teacher." It is the most expensive technique in this book, and also the one with the greatest power to change what a small model can do. Almost every capable small model released since 2024 has been shaped by distillation somewhere in its training. This chapter explains why imitation works better than learning from scratch, the main variants and their failure modes, how distillation is used to transfer reasoning into small models, and when a product team should do it themselves. Why soft targets carry more information The modern form of distillation was set out by Geoffrey Hinton, Oriol Vinyals and Jeff Dean in their 2015 paper "Distilling the Knowledge in a Neural Network." Their insight was that a trained model's output is richer than its top prediction. When a language model predicts the next token after "The capital of France is," its distribution places most probability on "Paris," but the remaining probability is not random: it favours other city names, other ways of beginning the answer, and plausible continuations, and disfavours nonsense. This "dark knowledge," in Hinton's phrase, encodes the teacher's understanding of how tokens relate. A student trained to match the full distribution learns these relations from every example, whereas a student trained only on the correct next token learns one fact per position. Hinton and colleagues also introduced a temperature parameter that softens both distributions before comparing them, making small probabilities more visible to the loss. The usual training objective is the Kullback–Leibler (KL) divergence between teacher and student distributions, often mixed with the ordinary cross-entropy loss on the true next token. The effect on data efficiency is substantial. In language modelling, a student trained on teacher distributions learns more from each token than one trained on raw text, which is why distillation became central to the economics of small models. Google described its Gemma 2 2B and 9B models (2024) as trained with knowledge distillation from a larger model rather than with next-token prediction, and argued that this let them train on more effective signal than their token count alone would provide. Meta used logits from Llama 3.1 8B and 70B as targets when building Llama 3.2 1B and 3B. NVIDIA's Minitron recipe, described in the pruning chapter, depends on distillation to recover quality after pruning. Variants of the objective Several design choices distinguish distillation methods, and each has consequences. Logit distillation versus sequence-level distillation. Logit distillation requires access to the teacher's full output distribution at every position, which means running the teacher alongside the student during training (or storing its top-k probabilities, which saves memory at some loss of signal). Sequence-level distillation, proposed by Yoon Kim and Alexander Rush in 2016, instead has the teacher generate complete outputs and trains the student on them as ordinary text. It needs only the teacher's generated text, not its probabilities — which makes it possible to distil from models accessible only through an API — but it discards the dark knowledge in each distribution. Most "synthetic data" pipelines, in which a large model writes training examples for a small one, are sequence-level distillation under another name. Forward versus reverse KL. The standard objective, forward KL from teacher to student, penalises the student for assigning low probability anywhere the teacher assigns high probability. A small student that cannot represent everything the teacher knows responds by spreading its probability thinly to cover all the teacher's modes, which can produce bland or incoherent generations. MiniLLM (Yuxian Gu and colleagues, ICLR 2024) argued for reverse KL, which penalises the student for placing probability where the teacher places little; this encourages the student to concentrate on the teacher's main modes and, the authors reported, produced more precise and better-calibrated generations from small students. Generalised objectives that interpolate between the two are now common. Off-policy versus on-policy. A subtle problem arises in generation. During training, the student is shown teacher-generated or human-written sequences and learns to predict the next token. At inference it conditions on its own previous outputs, which contain its own mistakes; it has never been trained on how to continue from those mistakes. This mismatch between training and inference distributions — known as exposure bias — compounds over long generations. Generalized Knowledge Distillation (GKD), from Rishabh Agarwal and colleagues at Google (ICLR 2024), addressed this by training the student partly on its own generated sequences, with the teacher providing token-level targets on those sequences. This on-policy distillation teaches the student how the teacher would continue from where the student actually goes. It has since become a standard ingredient in post-training small models, including for reasoning. Feature distillation. Rather than matching only outputs, the student can be trained to match the teacher's internal representations — hidden states or attention patterns at chosen layers — through learned projections where dimensions differ. DistilBERT (Victor Sanh and colleagues, 2019), which reduced BERT's size by 40 percent while retaining most of its language-understanding performance, combined output distillation with a hidden-state alignment loss and initialised the student from alternate layers of the teacher. Feature distillation helps most when the student's architecture is derived from the teacher's, as after structured pruning, because the correspondence between layers is natural. Distilling reasoning The largest change in small-model capability between 2024 and 2026 came from distilling reasoning. Models trained with reinforcement learning to produce long chains of thought before answering — OpenAI's o-series, DeepSeek-R1 and their successors — showed large gains on mathematics, coding and multi-step problems. The question was whether small models could share them. DeepSeek's January 2025 release of R1 gave a clear answer. Alongside the large model, DeepSeek released six smaller "distilled" models, built on Qwen 2.5 and Llama 3 base models from 1.5 to 70 billion parameters, which it had fine-tuned on about 800,000 samples generated by R1. DeepSeek reported that this simple supervised distillation produced small models with substantially stronger reasoning than their bases, and that, for the smaller models in its experiments, distilling from the large reasoning model worked better than applying reinforcement learning directly to the small model. The finding was widely replicated, and distilled reasoning became a standard part of small-model post-training. Small models in several families released since then offer "thinking" modes that produce a reasoning trace before the answer. For edge deployment, distilled reasoning creates a new trade-off. A reasoning trace may run to hundreds or thousands of tokens before the answer appears. On a device generating twenty tokens per second, a thousand-token trace is fifty seconds of waiting. Reasoning modes are therefore valuable where the task genuinely needs them and the latency is tolerable — a coding assistant, a planning step in an agent — and wasteful for simple requests. Many small models now let the application switch thinking on and off, and some products route only hard requests to thinking mode. Reasoning traces also stress compression in a specific way: they are long, multi-step generations in which small per-step errors compound, precisely the regime the fundamentals chapter identified as most sensitive to quantization. A model that quantizes well on short answers can fall apart in thinking mode, and evaluation must test both. Distillation for a specific task Most product teams will not distil a general model from scratch. The more common and more valuable use is task-specific distillation: making a small model nearly as good as a large one on one narrow job. The pattern is straightforward. Collect inputs representative of the product — real user requests where privacy permits, synthetic ones otherwise. Run a strong teacher on them to produce outputs, ideally with the teacher's probabilities if the teacher is an open model you can run, otherwise its text. Filter the outputs for correctness, using automated checks where the task allows (does the JSON parse, does the code pass its tests, does the extracted field match the source document) and human review where it does not. Fine-tune the small model on the result, usually with low-rank adaptation, described in the next chapter. Evaluate on held-out inputs. The economics are compelling. A small model distilled for a narrow task — classifying support tickets, extracting fields from invoices, rewriting text in a house style, routing requests to tools — often matches a much larger general model on that task, because the narrow task needs only a sliver of the large model's knowledge. The literature on this goes back years; "Distilling Step-by-Step" (Cheng-Yu Hsieh and colleagues, 2023) showed that training small models on both the labels and the rationales produced by a large model let them match or exceed much larger models on several benchmarks while using less training data than standard fine-tuning. The pattern has become routine in industry. Three cautions apply. First, the teacher's licence and terms of service may restrict using its outputs to train other models; check before building a pipeline. Second, the student inherits the teacher's errors and biases along with its skills, and filtering is the only defence. Third, task-specific distillation narrows the model. Training heavily on one task can erode general abilities the product still relies on — following instructions outside the task, refusing inappropriate requests, handling unexpected inputs gracefully. Mixing in general instruction data, keeping the training short, and using low-rank adaptation rather than full fine-tuning all limit this erosion, and the evaluation set should include general-behaviour checks as well as task metrics. Distillation as the repair step Distillation's third role at the edge is as the repair step after other compression. Every technique in the preceding chapters — aggressive quantization, structured pruning, layer removal, vocabulary trimming — damages the model, and the most effective way to repair that damage is to train the compressed model to match the original. This is self-distillation: teacher and student start as the same model, and the student is the damaged one. This framing unifies much of what has been described. Google's Gemma 3 QAT used the unquantized checkpoint's probabilities as targets. Minitron retrains pruned models by distilling from the unpruned parent. Apple trains low-rank adapters to recover quality lost in compression. In each case the original model provides a dense, task-agnostic training signal that pulls the compressed model back toward its original behaviour without requiring any labelled data. The practical advantage is that self-distillation needs only unlabelled inputs that resemble real use and enough compute to run the teacher and train the student. It is far cheaper than repeating the original post-training, and it preserves the alignment and instruction-following properties instilled by that training, which fine-tuning on raw text tends to erode. Practical constraints Distillation is constrained by three resources. Teacher inference. Logit distillation needs the teacher's distribution at every training position. For a large teacher this dominates the cost: every training token requires a forward pass through the teacher. Storing only the top few dozen probabilities per position, computed once offline, reduces this but sacrifices some signal and requires substantial storage. Sequence-level distillation from generated text is cheaper but weaker. Tokenizer compatibility. Logit distillation compares distributions over the same vocabulary. If teacher and student use different tokenizers — common when distilling across model families — the distributions cannot be compared directly. Research methods for cross-tokenizer distillation exist, but the simplest path is to choose a student and teacher from the same family, which is one reason vendors distil within their own model lines. Capacity gap. A student that is much smaller than its teacher may not be able to absorb the teacher's distribution, and several studies have reported that an intermediate-sized "teaching assistant" model or a less capable teacher sometimes produces a better small student than the largest available teacher. The right teacher is not always the strongest one; it is the one whose behaviour the student can learn to approximate. For a product team, the decision usually comes down to this. If a general small model, quantized well, is good enough, distillation is unnecessary. If the product depends on one or a few narrow capabilities that a larger model handles and the small model does not, task-specific distillation is likely the highest-value investment in the whole pipeline. If aggressive compression has damaged the model, self-distillation is the most reliable repair. And if no model of the required size exists at all, structured pruning followed by distillation is the route — but that is a training project, and should be planned as one. Chapter 9. Low-Rank Adaptation on Compressed Models Low-rank adaptation is not, strictly, a compression technique. It does not make a model smaller. But it has become indispensable to edge deployment for three reasons: it makes specialising a small model to a task cheap enough to do routinely; it can repair the damage done by quantization; and it lets a single compressed base model on a device serve many tasks by swapping small adapter files rather than whole models. This chapter explains how low-rank adaptation works, why QLoRA made fine-tuning compressed models practical, how adapters interact with quantization, and how they are used on devices. The low-rank idea Fine-tuning a model updates its weights: each weight matrix W becomes W + ΔW. Full fine-tuning learns ΔW in its entirety, which for a model with billions of parameters means storing gradients and optimiser state for every parameter — several times the model's own memory — and producing a full-sized copy of the model for every task. LoRA (Low-Rank Adaptation), introduced by Edward Hu and colleagues at Microsoft in 2021, rests on the observation that the update ΔW needed to adapt a pre-trained model to a new task tends to have low "intrinsic rank": it can be well approximated by the product of two thin matrices. LoRA freezes W and learns ΔW = BA, where, for a weight matrix with d_out rows and d_in columns, B has d_out rows and r columns and A has r rows and d_in columns, with the rank r typically between 4 and 64. For a 4,096 × 4,096 matrix and r = 16, the adapter holds 2 × 4,096 × 16 = 131,072 parameters instead of nearly seventeen million — less than one percent. B is initialised to zero, so training starts from the unmodified model, and a scaling factor (conventionally α divided by r) controls the magnitude of the update. The paper reported that LoRA matched full fine-tuning quality on a range of tasks while reducing the number of trainable parameters by orders of magnitude and the memory needed for training substantially. After training, the adapter can be kept separate, applied at inference by computing the low-rank product alongside the frozen weight, or merged into the weight (W + BA) so that inference costs nothing extra. Many variants have followed. DoRA (Shih-Yang Liu and colleagues, ICML 2024) decomposes each weight into a magnitude and a direction and applies the low-rank update to the direction only, reporting closer agreement with full fine-tuning. Initialisation schemes such as PiSSA start the adapter from the principal components of the original weight rather than from zero. Methods such as LoRA+ use different learning rates for A and B. None changes the basic economics; all are refinements worth trying when a standard LoRA falls slightly short. Two things are worth understanding about what LoRA can and cannot do. It is very good at teaching a model a format, a style, a narrow skill, or a domain's vocabulary — adaptations that redirect existing capabilities. It is less good at injecting large amounts of new factual knowledge; a 2024 study by Biderman and colleagues, "LoRA Learns Less and Forgets Less," found that LoRA underperformed full fine-tuning on substantial new-domain learning such as code and mathematics continued pre-training, but also preserved the base model's other abilities better. For edge products, which usually need specialisation rather than new knowledge, that trade is usually favourable. QLoRA: fine-tuning a quantized model LoRA reduced the memory needed for optimiser state but still required the frozen base weights in 16-bit precision. QLoRA, by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer (NeurIPS 2023), removed that requirement by freezing the base model in 4-bit form and training 16-bit LoRA adapters on top. Gradients flow through the dequantized 4-bit weights into the adapters; the base weights never change. QLoRA introduced three components. NF4, the 4-bit NormalFloat data type mentioned in the fundamentals chapter, uses a codebook spaced according to normal-distribution quantiles, designed to be information-theoretically well suited to normally distributed weights. Double quantization quantizes the per-block scale factors themselves, saving roughly a third of a bit per parameter. Paged optimisers use unified memory to move optimiser state between GPU and CPU memory, smoothing over memory spikes during training. The paper reported fine-tuning a 65-billion-parameter model on a single 48-gigabyte GPU while preserving 16-bit fine-tuning performance on its evaluations. For edge practitioners QLoRA has two meanings. The first is practical: fine-tuning a small model of one to nine billion parameters becomes possible on a single consumer GPU or a cloud instance costing little, and a team can iterate on task-specific adapters quickly. Tooling such as Hugging Face's PEFT library, Unsloth, Axolotl and TRL has made the workflow routine, and MLX supports LoRA and QLoRA training directly on Apple silicon laptops. The second is subtler and more important for deployment. An adapter trained with QLoRA was trained on top of a specific quantized base: NF4 weights with particular block sizes. If the model is deployed with a different quantization — say, merged into 16-bit and then re-quantized to a GGUF Q4_K_M or to a vendor's NPU format — the base weights the adapter sees at inference differ from the base weights it was trained against. The adapter learned to correct for NF4's specific errors; it may now be correcting for errors that are not there and ignoring ones that are. Usually the effect is small, but on sensitive tasks it can be noticeable. The safest practice is to train the adapter against the same quantized weights that will ship, or to evaluate carefully after re-quantization. Adapters as quantization repair The QLoRA arrangement — frozen quantized base, trainable high-precision adapter — suggests a further use: train the adapter not for a new task but to undo the quantization error itself. LoftQ (Yixiao Li and colleagues, ICLR 2024) made this explicit. Rather than initialising the adapter at zero, LoftQ chooses the quantized weights Q and the low-rank initialisation BA jointly so that Q + BA approximates the original 16-bit weight as closely as possible, alternating between quantizing the residual and computing a low-rank fit of what quantization lost. Fine-tuning then starts from a model much closer to the original. The authors reported particular gains at 2 and 3 bits, where the gap between the quantized and original model is largest. Apple's on-device model uses the same principle at production scale. Apple's technical reports describe training low-rank adapters to recover quality lost when its on-device model is compressed to low bit-widths, and they expose LoRA adapter fine-tuning to developers through the Foundation Models framework, so that third-party specialisations sit on top of the same compressed base. Meta's quantization-aware-trained Llama 3.2 variants likewise combined QAT with low-rank adapters. The general recipe for a team that has quantized a model and found it slightly short of the quality floor is: freeze the quantized weights in exactly their deployment format; attach LoRA adapters to the linear layers (all of them, or at least the attention and feed-forward projections); train with self-distillation from the original model on representative unlabelled data; evaluate. A rank of 16 to 64 is a common starting point. The adapter adds a small amount of memory and computation at inference — for a three-billion-parameter model, rank-16 adapters on all linear layers add on the order of tens of millions of parameters, which in 16-bit precision is tens of megabytes. If memory is very tight, the adapter can be merged into the base weights and the result re-quantized, but merging re-introduces rounding error: the merged weight W_q + BA must be quantized again, and the carefully learned correction is partly lost to the new rounding. Keeping the adapter separate, in higher precision, preserves the repair. Many tasks on one base The most distinctive edge use of adapters is multiplexing. A device has room for one compressed base model; a product may need several specialisations — summarisation, reply suggestions, extraction, a domain assistant. Shipping several full models is impossible. Shipping one base with several small adapters is easy: each adapter is tens of megabytes, can be downloaded and updated independently, and can be swapped in a fraction of a second. Apple's platform is built around this design. Its 2024 description of Apple Intelligence explained that its on-device foundation model is specialised for different features by small adapters loaded and swapped dynamically, giving the model the ability to specialise "on the fly" for the task at hand while sharing the base weights in memory. The same pattern is available on other platforms: llama.cpp supports loading LoRA adapters at runtime and applying them without merging, ExecuTorch and other mobile runtimes have added adapter support, and server frameworks can apply different adapters per request. Multiplexing adapters raises engineering questions that do not arise with a single model. The adapters must all be trained against the identical quantized base; any change to the base — a new quantization, a model update — invalidates them all, so base updates become coordinated releases. Applying an adapter without merging adds per-token compute that is small but not zero, especially on NPUs, where the extra low-rank multiplications may run on a different unit from the main graph. Some NPU toolchains require adapters to be compiled into the graph ahead of time, which limits how dynamically they can be swapped. And evaluation multiplies: each adapter on the compressed base needs its own test set. Practical guidance for training adapters Some lessons recur across teams that train adapters for edge models. Target the right layers. Early LoRA work often adapted only the attention query and value projections. Later experience, including the QLoRA paper's findings, favours adapting all linear layers, including the feed-forward projections, at a modest rank. This generally improves quality for little additional cost. Match the chat template exactly. Instruction-tuned models depend on a specific prompt format. Training an adapter with a slightly different template — a missing system-prompt token, different whitespace — teaches the model a format it will never see in production, or worse, degrades its behaviour on the real format. Keep data diverse enough to avoid collapse. A narrow training set teaches the narrow task but can erode general behaviours. Mixing in a fraction of general instruction data, stopping early, and checking general-behaviour metrics prevent the adapter from teaching the model to answer every question as if it were an invoice. Distil where possible. Adapter training benefits from the same soft-target signal as any other distillation. When a larger teacher in the same family is available, training on its distributions rather than only on target text tends to give better adapters from less data. Train against the shipped base. The adapter should see, during training, the same quantized base weights it will see at inference. This is the single most common source of unexplained quality differences between a model evaluated in a training environment and the same model on a device. The place of adapters in the blueprint Low-rank adaptation fills a specific position in an edge compression pipeline. It comes after the base model has been chosen and quantized, and it serves one of three purposes: specialising the model to the product's tasks, which is almost always worthwhile; repairing quality lost to quantization, when cheaper fixes have failed; and letting one base serve many features. It does not reduce the model's size, and it adds a small cost at inference if kept separate. But it is the cheapest form of training available, and it turns a compressed general model into a compressed specialist — which is, in the end, what most edge products need. Chapter 10. Runtimes, Formats and Silicon A compressed model is only as good as the software that runs it and the chip underneath. Many promising compression results fail at this stage: a format the runtime cannot execute efficiently, an operator the neural processing unit does not support, a quantization scheme that saves bytes on paper but forces the runtime to fall back to slow code. This chapter surveys the hardware that runs language models at the edge as of late 2026, the runtimes that connect models to it, and the techniques — speculative decoding chief among them — that runtimes use to get more out of the same hardware. Its argument is simple: choose the format to fit the runtime and the runtime to fit the chip, not the other way round. The processors in an edge device A modern phone or laptop system-on-chip contains three kinds of processor that can run a language model, all sharing the same pool of memory. The CPU is the most flexible and most widely supported target. Modern ARM cores include vector instructions (NEON, and on newer designs the Scalable Vector and Scalable Matrix Extensions) with dot-product and matrix-multiply instructions for 8-bit integers, and x86 laptop chips have their own vector extensions. Because decode is bandwidth-bound, a CPU with good kernels can approach the memory-bandwidth ceiling for token generation on small models; its weakness is prefill, where its arithmetic throughput falls well short of the GPU's or NPU's. Arm's KleidiAI library provides optimised low-precision kernels that frameworks including llama.cpp, ExecuTorch and MediaPipe have integrated. The GPU offers far more arithmetic throughput than the CPU, which makes it the usual target for prefill and for larger models on laptops. On Apple silicon, the GPU is the main engine for open-model inference through Metal; Apple's M5 generation added matrix-multiplication units to the GPU cores — which Apple calls Neural Accelerators — and Apple's own measurements with MLX attributed a three- to four-fold improvement in time to first token over the M4 to them. On Android, the GPU is reachable through OpenCL or Vulkan, and runtimes such as llama.cpp, MLC LLM and LiteRT have GPU paths of varying maturity across Qualcomm Adreno, ARM Mali and other GPUs. The NPU, or neural processing unit, is a dedicated accelerator for tensor operations, usually optimised for low-precision integer arithmetic and very high energy efficiency. Every major mobile and laptop chip now has one: Apple's Neural Engine, Qualcomm's Hexagon NPU, MediaTek's APU, Google's TPU in its Tensor chips, Samsung's NPU in Exynos, Intel's NPU in Core Ultra processors and AMD's XDNA NPU in Ryzen AI processors. Microsoft's Copilot+ PC specification, introduced in 2024, required an NPU capable of more than 40 trillion operations per second, and vendors have competed on that figure since. Qualcomm described the Hexagon NPU in its Snapdragon 8 Elite Gen 5, announced in September 2025, as 37 percent faster than its predecessor. Headline NPU figures, measured in TOPS (trillions of operations per second, usually at INT8 or INT4), measure peak arithmetic — the resource that matters for prefill. For decode, the NPU reads weights from the same memory as the CPU and GPU, at the same bandwidth. An NPU does not make decode faster than the memory bus allows; it can make decode more energy-efficient, and it can free the CPU and GPU for other work. This is why NPU marketing and practitioner experience often diverge: the NPU may halve time to first token while leaving tokens per second roughly where the CPU had them. NPUs also impose constraints that CPUs and GPUs do not. Most compile a static graph ahead of time, with fixed tensor shapes, which sits uneasily with the dynamic shapes of language-model inference (growing context, varying prompt lengths); runtimes work around this by compiling several graph variants for different sequence lengths, or by splitting prefill and decode into separate graphs. Supported operator sets are narrower, so unusual layer types — new attention variants, linear-attention recurrences, custom activation functions — may fall back to the CPU, with expensive data transfers at each boundary. And the supported number formats are specific: a given NPU might execute 8-bit activations with 4-bit weights efficiently, but not 16-bit floating point, or not the particular block-scaled format a model was quantized in. Each vendor's toolchain — Qualcomm's AI Engine Direct and AI Hub, MediaTek's NeuroPilot, Apple's Core ML, Intel's OpenVINO, AMD's Ryzen AI software — defines what is possible on its silicon. The runtime landscape The runtimes that run small language models at the edge fall into three groups. Cross-platform open runtimes prioritise breadth. llama.cpp, with its GGUF format, runs on almost anything with a CPU and has backends for Metal, CUDA, Vulkan, SYCL, OpenCL and others; it added a backend for Qualcomm's Hexagon NPU in late 2025. It is the engine behind Ollama, LM Studio and many applications, and its quantization formats, discussed in the post-training chapter, are the de facto standard for local models on laptops and desktops. MLC LLM, from the team behind Apache TVM, compiles models to many GPU backends, including mobile GPUs, and has strong Android and iOS support. ONNX Runtime, with its GenAI extension, runs models exported to the ONNX format across CPUs, GPUs and NPUs through execution providers, and underpins Microsoft's Windows AI platform. Platform runtimes are built by OS and framework vendors. Apple offers MLX, an array framework for its silicon that has become the most popular way to run and fine-tune open models on Macs, and Core ML, the deployment framework that can target the Neural Engine. Apple's Foundation Models framework, introduced in 2025, gives developers direct access to Apple's own on-device model with guided generation and tool calling, without shipping a model at all. Google offers LiteRT (formerly TensorFlow Lite) and LiteRT-LM, a runtime for language models used in Google's own products and supporting Gemma models on CPU, GPU and, increasingly, NPU, along with MediaPipe's LLM inference API and Android's AICore system service, which hosts Gemini Nano and has offered developer previews of Gemma models. PyTorch's ExecuTorch, which reached version 1.0 in October 2025, exports PyTorch models directly to an on-device runtime with backends including XNNPACK with Arm KleidiAI on CPU, Core ML, Qualcomm's AI Engine, Arm Ethos-U, Vulkan, MediaTek and Samsung's Exynos; Meta reported using it for on-device AI features across Instagram, WhatsApp, Messenger and Facebook. Vendor stacks expose each silicon maker's hardware most directly: Qualcomm's AI Engine Direct SDK and its Genie runtime for generative models, MediaTek's NeuroPilot, Intel's OpenVINO GenAI, AMD's Ryzen AI software with ONNX Runtime, and NVIDIA's TensorRT-LLM and TensorRT Edge-LLM for Jetson devices. They often deliver the best performance on their own chips and the worst portability. Table 3 matches the main runtimes to their typical formats and targets. It is a starting map, not a guarantee; support changes month by month, and each entry should be checked against the current release. Table 3. Edge runtimes, their usual model formats and primary hardware targets (as of late 2026; verify against current documentation). Runtime Typical model format Primary targets Notable strength llama.cpp GGUF (k-quants, i-quants, Q4_0, Q8_0) CPU, Apple Metal, CUDA, Vulkan; Hexagon backend Breadth, community quantizations MLX MLX safetensors, group-wise 2–8 bit Apple silicon GPU Fast iteration and fine-tuning on Mac Core ML mlpackage with palettized or linear weights Apple Neural Engine, GPU, CPU Neural Engine access on Apple devices ExecuTorch PTE program, torchao quantization Mobile CPU, Core ML, Qualcomm, MediaTek, Arm NPUs PyTorch-native export, many backends LiteRT / LiteRT-LM LiteRT model files with int4/int8 weights Android CPU, GPU, NPU Google's production mobile path ONNX Runtime GenAI ONNX with int4 weights Windows CPU, GPU, NPUs Windows integration, many providers Speculative decoding: borrowing compute to beat bandwidth Decode is bandwidth-bound: each step reads all the weights to produce one token, leaving the arithmetic units mostly idle. Speculative decoding turns that idle compute into speed. The technique, published independently in 2022 and 2023 by Yaniv Leviathan, Matan Kalman and Yossi Matias at Google and by Charlie Chen and colleagues at DeepMind, uses a small, fast draft model to propose several tokens ahead. The large target model then verifies all the proposed tokens in a single forward pass — which costs nearly the same as generating one token, because the weights are read once and applied to several positions at once. The target model accepts the longest prefix of the draft that matches what it would have produced (using a rejection-sampling rule that guarantees the output distribution is exactly the target model's), and generates one corrected token at the first disagreement. If the draft model is usually right, several tokens are produced for the bandwidth cost of one. At the edge this is attractive and subtle. It is attractive because the conditions it exploits — bandwidth-bound decode, spare compute — are exactly the conditions on phones and laptops. It is subtle because the draft model also consumes memory and bandwidth, and on a device with little headroom a separate draft model may not fit. Several variants avoid a separate draft model. Medusa (Tianle Cai and colleagues, 2024) adds extra prediction heads to the target model to guess several future tokens. EAGLE (Yuhui Li and colleagues, 2024) and its successors train a small one-layer draft network that operates on the target model's hidden features, which tends to give high acceptance rates at very low cost. Prompt lookup or n-gram decoding uses no model at all, drafting by copying spans from the prompt, which works remarkably well for tasks such as editing, summarisation and code modification where the output repeats much of the input. Some model families now ship with multi-token prediction heads trained in, which runtimes can use for self-speculation. The gain from speculative decoding depends on the acceptance rate, which depends on the task: predictable outputs (code, structured data, edits) accept many draft tokens; creative text accepts fewer. Typical reported speed-ups in the literature range from about one-and-a-half to three times, but on edge hardware the result should be measured, because the extra verification compute and draft memory can erase the gain on a heavily constrained device. Other runtime techniques worth knowing Several further techniques determine how well a given format performs. Prompt caching stores the KV cache for a fixed prefix — a system prompt, tool definitions, a document being discussed — and reuses it across requests, eliminating repeated prefill. For agentic applications with long, stable preambles it can cut time to first token dramatically. Constrained decoding restricts generation to tokens that keep the output valid according to a grammar or JSON schema. llama.cpp supports grammars natively; Apple's Foundation Models framework provides guided generation into Swift types; libraries such as Outlines and XGrammar implement constrained decoding for other runtimes. For small, compressed models, constrained decoding is not just a convenience: it guarantees structural validity that a quantized model might otherwise break, rescuing exactly the kind of multi-step formatting errors that compression tends to introduce. Memory mapping lets a runtime map model weights from storage into memory on demand rather than loading them up front, so a model starts quickly and the operating system can reclaim pages under pressure. llama.cpp uses memory mapping by default. On phones, aggressive memory management by the operating system can then cause weights to be evicted and re-read from flash mid-generation, which is dramatically slower; locking critical weights in memory, where the platform allows it, avoids this. Heterogeneous scheduling runs different parts of the workload on different processors — prefill on the NPU or GPU, decode on the CPU, or the embedding lookup on the CPU while the transformer runs on the NPU. Runtimes are increasingly doing this automatically, but the boundaries between processors cost time, and a naive split can be slower than running everything on one unit. Matching format to silicon The practical consequence of all this is a matching exercise, to be done before compression rather than after. Start from the device and its intended runtime. If the product targets Apple platforms and can use Apple's own model, the Foundation Models framework removes the compression problem entirely, at the cost of using Apple's model and its adapter mechanism. If it ships its own model on Apple devices, MLX on the GPU is the fastest route to a working prototype on Macs, and Core ML or ExecuTorch with the Core ML backend is the route to the Neural Engine on iPhones. If it targets Android broadly, LiteRT-LM, ExecuTorch, MLC LLM and llama.cpp are the main options; for NPU access on specific chipsets, the vendor toolchains or ExecuTorch's vendor backends are needed. If it targets Windows laptops, ONNX Runtime and the Windows AI stack reach NPUs from Qualcomm, Intel and AMD; llama.cpp covers CPUs and GPUs broadly. If it targets embedded Linux boards, llama.cpp on CPU is the universal fallback, and NVIDIA's stack is the choice on Jetson. Then ask what formats that runtime executes fast on that silicon. On CPU-based llama.cpp, the k-quants and i-quants are well optimised, and some simpler formats such as Q4_0 enjoy special ARM-specific repacking that can make them faster than nominally better formats. On NPUs, the vendor's supported integer formats dictate the scheme, usually 4- or 8-bit weights with 8- or 16-bit activations and per-channel or per-group scales. On GPUs, group-wise 4-bit formats with fast dequantization kernels are the norm. Only then choose the compression method that produces the best model in that format: AWQ or GPTQ for integer weight formats, imatrix-calibrated quantization for GGUF, SmoothQuant-style calibration and possibly rotations for NPU activation quantization, QAT if the vendor has published a checkpoint in the target format. A brilliant 3-bit vector-quantized model that the target runtime cannot execute is worth less than a plain 4-bit model that runs at the bandwidth ceiling. Finally, measure on the device itself — not on a development laptop, not in a simulator, and not in a thirty-second burst. Measure time to first token at realistic prompt lengths, decode speed over realistic generation lengths, sustained performance over several minutes of use, peak memory including the KV cache at the longest supported context, and energy where the platform exposes it. Compare the decode speed with the bandwidth ceiling computed from the model's size. The gap between the two tells you whether the next improvement should come from the model or from the software. Chapter 11. Measuring What Compression Costs The description of this book promises methods "that preserve downstream benchmark performance." That phrase deserves scrutiny, because preserving benchmark scores and preserving the behaviour a product depends on are not the same thing. A compressed model can match its parent on a widely reported benchmark to within a point and still fail at the specific task it was deployed for. This chapter is about measuring compression honestly: which signals are sensitive, which are misleading, how to build an evaluation that would catch the failures that matter, and how to decide whether a compressed model is good enough to ship. Why aggregate benchmarks mislead The standard way to report a compression result is a table of scores on public benchmarks — knowledge and reasoning multiple-choice tests, commonsense tests, mathematics word problems — plus perplexity on a text corpus, for the original and the compressed model. If the numbers are close, the compression is declared nearly lossless. There are four problems with this. Accuracy hides flips. A 2024 study by Abhinav Dutta and colleagues at Microsoft Research, "Accuracy is Not All You Need," compared compressed models with their originals and found that even when aggregate accuracy was nearly unchanged, a substantial fraction of individual answers changed — some from correct to incorrect and others from incorrect to correct, cancelling out in the total. They proposed measuring the proportion of such "flips" and the divergence between the two models' output distributions as more faithful indicators of how much the model had changed. The implication for edge deployment is direct: if a product depends on consistent behaviour, a stable aggregate score is not evidence that behaviour is stable. Easy benchmarks are insensitive. Multiple-choice benchmarks measure the model's ranking of a handful of options at a single position. Compression error can perturb the probabilities substantially without changing which option ranks highest. Tasks that require generating long, exact outputs — code, mathematics with worked steps, structured data, multi-turn tool use — accumulate error over many steps and degrade earlier and more sharply. Several systematic evaluations of quantized models since 2023 have reported exactly this pattern: knowledge benchmarks hold up well down to about 4 bits, while generation-heavy and reasoning tasks show losses sooner. Benchmarks miss what the product does. Public benchmarks are mostly in English, mostly short-context, mostly free of the system prompts, chat templates, tools and output schemas that real deployments use. Compression damages capabilities unevenly, and the capabilities most likely to be damaged — less common languages, long-context retrieval, strict formatting, rare skills — are those least represented in public suites. Perplexity is a blunt instrument. Perplexity — the exponential of the average negative log-likelihood on held-out text — is cheap, sensitive and useful for comparing quantizations of the same model on the same text. But it averages over every token, and most tokens in ordinary text are easy. A model can retain near-identical perplexity while losing the ability to produce the few hard tokens that matter in a reasoning chain. It is also sensitive to the text it is measured on, and a calibration set that overlaps the perplexity corpus produces optimistic numbers. None of this means public benchmarks are useless. They are good for detecting gross breakage, for comparing methods under controlled conditions, and for relating a result to the literature. They are not sufficient for deciding whether a compressed model is fit for a product. More sensitive signals Better signals compare the compressed model directly with its parent, token by token, rather than comparing both with a ground truth. KL divergence. The Kullback–Leibler divergence between the original and compressed models' next-token distributions, averaged over a representative set of texts, measures how much the compressed model's predictions have moved, at every position and across the whole vocabulary. It is far more sensitive than accuracy and does not depend on having correct answers. The llama.cpp project's perplexity tool can compute KL divergence between a quantized model and a reference, reporting the mean and the tails of the distribution, and this has become a standard way for the community to compare quantization types. The tails matter: a quantization whose mean divergence is small but whose 99th-percentile divergence is large has a few positions where it behaves very differently, and those positions are where generation goes off course. Top-token agreement. The fraction of positions at which the compressed model's most likely token matches the original's is an intuitive complement to KL divergence. For greedy decoding it predicts directly how often the two models will diverge. Flip rate. On a task with defined answers, the fraction of examples whose correctness changes between the original and compressed model, in either direction, measures behavioural instability that net accuracy hides. Divergence over generations. Generate long outputs from the same prompts with both models at temperature zero and measure where they first diverge, and whether the divergence matters. For structured tasks this can be automated; for open-ended tasks it needs judging. These signals share one property: they measure how far the compressed model has drifted from the original, which is exactly what compression should minimise. They do not tell you whether the original was good enough, which is a separate question answered by task evaluation. Building a product evaluation The evaluation that decides whether to ship should be built from the product's own tasks, and it should be built before compression begins, so that every compression decision can be tested against it. A useful product evaluation has several layers. Task accuracy on representative inputs. Several hundred examples drawn from real or realistic use, with defined correct outputs or clear grading criteria. For classification and extraction tasks, exact metrics are straightforward. For generative tasks, graded rubrics applied by a strong model acting as judge, with periodic human spot-checks of the judge's reliability, are now standard practice. Format and protocol compliance. The rate at which outputs parse as valid JSON, call tools with valid arguments, follow the required schema, stop at the right place and respect length limits. These are among the first properties to break under compression and among the most damaging when they break, because downstream software fails outright. If the product uses constrained decoding, test both with and without it, to know how much the constraint is carrying. Long-context behaviour. If the product uses long contexts, test retrieval and reasoning at the full length the product supports — not only short prompts. Tests that hide a specific fact deep in a long document and ask for it are crude but effective at catching KV-cache quantization damage and attention problems. Multilingual and domain coverage. Test every language and domain the product serves, with examples weighted by usage but never omitting low-volume languages entirely, because compression damage concentrates there. Multi-turn and agentic behaviour. If the product has conversations or tool loops, test complete trajectories, not single turns. Errors that are rare per turn become common over ten turns. Safety and refusal behaviour. Compression and fine-tuning can both shift a model's handling of harmful or out-of-scope requests. Include a set of prompts the model should decline or handle carefully, and check that compressed variants behave as the original does. Research on quantized models has reported that safety behaviours can degrade under aggressive quantization in ways that general benchmarks do not show, which makes this a necessary check rather than an optional one. Thinking modes. If the model has a reasoning mode that the product uses, evaluate it separately. Long reasoning traces amplify compression error, and a model can pass in direct-answer mode while failing in thinking mode. The evaluation should report both absolute task scores and drift signals relative to the uncompressed model, so that a regression can be attributed to compression rather than to the base model's own limitations. On-device measurement Quality is half the decision; the other half is whether the compressed model meets the device budget, and that can only be measured on the device. Measure peak memory during the most demanding realistic session: longest context, largest prompt, any adapters loaded, alongside the rest of the application. Measure time to first token at realistic prompt lengths, including the system prompt and any retrieved content. Measure decode speed over realistic output lengths, and compare it with the bandwidth ceiling computed from the model's size and the device's memory bandwidth. Measure sustained performance over several minutes, on battery and on charge, at a warm room temperature, because thermal throttling changes the numbers. Measure energy per request where the platform provides counters. Measure on the lowest-specification device the product supports, not only the developer's flagship. Run these measurements as part of an automated pipeline. Edge models are updated — new base versions, new adapters, new runtime releases, operating system updates that change memory management — and each change can shift the numbers. A team that measured once at launch will discover regressions from user complaints. Deciding what is good enough The final decision is a judgement about two budgets, and it helps to make it explicit. For quality, define, before compressing, the minimum acceptable score on each layer of the product evaluation, and the maximum acceptable drift relative to the original model. Some properties — format compliance, safety behaviour — warrant near-zero tolerance. Others — the stylistic quality of open-ended text — can tolerate more. Write these down; otherwise, the threshold drifts toward whatever the latest compressed model achieves. For the device, define the memory ceiling, time-to-first-token ceiling, decode-speed floor and energy envelope on the lowest supported device. Then evaluate candidates against both sets of thresholds. A compressed model that meets every threshold ships. One that misses on the device side needs more compression or a smaller base. One that misses on the quality side needs a gentler compression, a better method, a recovery step — adapters, QAT, distillation — or a stronger base. One that misses on both is a sign that the size class is wrong. It is also worth keeping, through the life of the product, a small set of "canary" prompts — examples that previously exposed compression damage — and running them on every new artefact. Compression failures tend to recur in the same places, and the cheapest way to catch them is to remember where they happened before. Chapter 12. A Blueprint for Shipping The preceding chapters described techniques one at a time. A real project uses several of them, in sequence, under deadlines, with incomplete information. This chapter assembles them into a blueprint: an ordered set of decisions, each with the questions to answer and the evidence to collect before moving on. It then walks through three worked scenarios — a phone assistant, a laptop document tool and an embedded device — to show how the same blueprint produces different pipelines when the budgets differ. The scenarios are illustrative composites, not reports of particular companies; the findings attributed to each team are the kind of findings the blueprint is designed to surface, and any real project must measure its own. The ordering principle is the one stated at the start of the book. Remove bytes in the cheapest places first; spend training compute only when cheap methods have run out; choose techniques the hardware can actually exploit; and judge every step by measurement against the product's own evaluation. Step 1: Write down both budgets Before touching a model, write down the device budget and the quality budget. The device budget comes from the lowest-specification device you will support. Record total memory and an honest estimate of what the operating system and the rest of the application leave free; sustained memory bandwidth, from published specifications or a simple measurement; the processors available and the runtimes that reach them; the longest context the product needs; the minimum acceptable decode speed and maximum acceptable time to first token; and any energy constraint. From these, derive three ceilings: bytes of weights, bytes of KV cache at maximum context, and bytes read per generated token (bandwidth divided by required tokens per second). The quality budget comes from the product. Build the evaluation described in the previous chapter — task accuracy, format compliance, long-context, languages, multi-turn, safety, thinking mode if used — and set thresholds for each layer. Teams that skip this step end up compressing toward a moving target. Teams that take it seriously frequently discover that one budget makes the other impossible — a context length that cannot fit, a quality bar no model of the feasible size meets — and that discovery is far cheaper at the start than at the end. Step 2: Choose the base model and runtime together Using the weight ceiling and the assumption of roughly 4.5 bits per weight, identify the size class. Shortlist two or three models in that class whose licence fits, whose architecture suits the context requirement (grouped-query attention with a favourable ratio, or a hybrid design), and whose instruction-tuned behaviour is close to the product's needs. In parallel, choose the runtime for each target platform, and confirm that each shortlisted architecture runs efficiently on it — including any NPU you intend to use. Check whether the model vendor publishes quantized or quantization-aware-trained checkpoints in a format the runtime supports. If it does, those are usually the best starting artefacts. Run the product evaluation on each shortlisted model at full precision. This establishes the ceiling of what each model can achieve and catches models that are unsuitable before any compression effort is spent. Step 3: Quantize weights with a calibrated method Quantize each remaining candidate's weights to about 4 bits using the best calibrated method native to the chosen runtime: an imatrix-calibrated k-quant or i-quant for llama.cpp, AWQ or GPTQ for GPU runtimes, the vendor's calibrated flow for NPU targets, or the vendor's QAT checkpoint. Build calibration data from the product's own distribution, formatted with the model's chat template, covering every language and task type. Keep the embedding and output layers, and any layers that sensitivity analysis flags, at higher precision if the format allows. Quantize the KV cache to 8 bits unless measurement shows a problem. Evaluate: product evaluation plus drift signals (KL divergence, flips) against the full-precision model. Measure on the device. Very often this is the end of the compression work. If the model meets both budgets, go to step 7. Step 4: If the device budget is still exceeded If the 4-bit model does not fit or is too slow, the options, in order of preference, are: First, a smaller base model at 4 bits. This is usually better than pushing a larger model below 4 bits with post-training methods. Second, architecture-level savings that do not touch weight precision: a shorter maximum context, a more cache-efficient model, prompt caching for fixed prefixes, dropping unused encoders, trimming the vocabulary for single-language products. Third, lower bit-widths for the least sensitive layers, keeping sensitive ones at 4 bits or above. Mixed 3- and 4-bit configurations, carefully calibrated, can save meaningful memory with tolerable loss on robust models. Fourth, quantization-aware training to 3 or 2 bits, with self-distillation. This is a training project, but it is the path that the most aggressive shipped deployments have taken. Fifth, for speed specifically, speculative decoding, provided memory headroom allows it, and activation quantization on an NPU or GPU for prefill-heavy workloads. Structured pruning plus distillation, to manufacture a custom-sized model, belongs here only if no available model of the needed size meets the quality bar and the organisation can afford to retrain. Step 5: If the quality budget is missed If the 4-bit model fits but falls short on some layer of the evaluation, diagnose first. Compare the compressed model with its full-precision parent on the failing cases. If the parent also fails, the problem is the base model, not compression; specialisation (step 6) or a stronger base is needed. If the parent passes and the compressed model fails, compression is the cause. For compression-caused failures, try, in order: better calibration data that covers the failing behaviour; higher precision for sensitive layers, or a 5- or 6-bit format overall if the device budget allows; low-rank recovery adapters trained with self-distillation against the exact shipped weights; full QAT. Step 6: Specialise with adapters Most products benefit from specialising the compressed model to their tasks. Train LoRA adapters on the shipped quantized base, using task data distilled from a stronger teacher where possible, mixed with general instruction data to protect general behaviour, in the exact chat template the product uses. Evaluate the adapted model on the full product evaluation, including general-behaviour and safety layers, not only on the target task. If the product has several features, consider one adapter per feature on a shared base, with the engineering implications discussed in the low-rank chapter: all adapters trained on the identical base, coordinated updates, and per-adapter evaluation. Step 7: Harden and monitor Before shipping, add the runtime protections that compensate for compressed-model fragility: constrained decoding for every structured output, sensible limits on generation length, fallbacks for malformed outputs, and, where a cloud path exists, a policy for routing requests the on-device model cannot handle. After shipping, keep the evaluation and on-device measurements in an automated pipeline, run them on every new artefact and runtime release, and maintain canary prompts drawn from past failures. Scenario one: a private assistant on mid-range phones A team is building a messaging assistant that summarises conversations and drafts replies entirely on-device, for Android phones with 8 gigabytes of memory and above, in English, Spanish and Portuguese. The device budget, on the lowest-tier phone, leaves perhaps 2.5 gigabytes for the model and cache. Conversations to summarise can run to 6,000 tokens. Users expect replies to start within about two seconds and stream faster than reading speed. Applying the blueprint: at about 4.5 bits per weight, a model of three to four billion parameters would occupy around 2 gigabytes, leaving too little room for a 6,000-token cache in 16-bit precision; a model of one to two billion parameters leaves comfortable room. The team shortlists small models around 1 to 2 billion parameters with grouped-query attention or hybrid attention, permissive licences and good multilingual support, and confirms that its chosen runtime supports them on CPU and on the target NPUs. It quantizes weights to 4 bits with a calibrated method, using calibration data built from synthetic conversations in all three languages formatted with the model's chat template, keeps the output layer at 8 bits, and quantizes the cache to 8 bits. Evaluation shows good summaries in English and Spanish and noticeably weaker Portuguese, in both the compressed and full-precision models. The base model is the limitation. The team distils summaries and replies from a larger open teacher in all three languages, filters them, and trains a LoRA adapter on the quantized base. Portuguese quality reaches the threshold. Format compliance for the structured summary is near perfect with constrained decoding and imperfect without it, so constrained decoding ships. Time to first token on the CPU for 6,000-token conversations is too slow; moving prefill to the NPU through the vendor backend brings it within budget, while decode stays on the CPU, near the bandwidth ceiling. Nothing exotic was needed. The decisive choices were the size class, the calibration data, a task adapter, constrained decoding and putting prefill on the right processor. Scenario two: a document tool on laptops A team is building a desktop application for lawyers that answers questions about long contracts locally, on laptops with 16 to 32 gigabytes of unified memory, including Apple silicon Macs and Windows Copilot+ PCs. Here memory is less constrained, but contexts are long — contracts plus retrieved precedents can exceed 30,000 tokens — and accuracy on precise retrieval is non-negotiable. The blueprint points to a larger model: at 4.5 bits, a model of eight or nine billion parameters occupies around 5 gigabytes, leaving room for a large cache. The team prefers a model with an efficient attention design, because a 32,000-token cache for a conventional 8-billion-parameter model with grouped-query attention would itself run to several gigabytes. It quantizes weights to 4 bits with calibration on legal text and the model's chat template, and tests KV-cache quantization carefully: an 8-bit cache passes long-context retrieval tests; a 4-bit key cache measurably reduces retrieval accuracy at the far end of long documents, so keys stay at 8 bits. Time to first token is the dominant complaint, because every question involves a long prefill. The team uses prompt caching for the contract text, so that follow-up questions reuse the cached prefix, and runs prefill on the GPU — on Apple silicon through MLX, on Windows through a GPU or NPU provider. Speculative decoding using prompt lookup proves effective because answers often quote the contract verbatim. The decisive choices were cache precision tested against the product's own retrieval task, prompt caching, and a speculative method matched to the task's tendency to quote. Scenario three: an offline voice assistant on an embedded board A manufacturer wants an offline voice command interpreter for industrial equipment, running on an ARM single-board computer with 4 gigabytes of memory and modest bandwidth. Inputs are short transcribed commands; outputs are structured function calls from a fixed set of about forty operations. The device budget is tight on bandwidth: a board of this class may sustain well under twenty gigabytes per second, so a 1-gigabyte model can generate at most around fifteen to twenty tokens per second, and probably less. But the task is narrow, the context short and the output a small JSON object. The blueprint points to a very small model — a few hundred million to one billion parameters — specialised by distillation. The team generates tens of thousands of synthetic commands with paraphrases and noise resembling speech-recognition errors, labels them with a large teacher, verifies the labels against the function schema, and fine-tunes a small model with LoRA, then merges and quantizes to 4 bits with calibration on the same command distribution. Constrained decoding against the function-call grammar guarantees valid output. The resulting model, a fraction of a gigabyte, runs within the latency budget on the board's CPU with llama.cpp. The decisive choices were recognising that a narrow task needs a tiny specialised model rather than a compressed general one, and letting distillation and constrained decoding do the work that a larger model would otherwise have done. What the scenarios share In none of the three scenarios did the team need 2-bit quantization, unstructured pruning or a novel algorithm. Each succeeded by getting the budgets right at the start, choosing a base model sized to them, quantizing carefully with representative calibration data, measuring with an evaluation built around the product, and spending training compute — distillation and adapters — exactly where the evaluation showed a gap. The exotic techniques in this book are real and occasionally decisive. The ordinary ones, done well, ship most products. Conclusion The techniques in this book can be listed in a sentence — quantize, prune, distil, adapt — and the list is what most people take away from a survey of model compression. It is the least useful thing to take away. What the preceding chapters show is that the techniques are not alternatives on a menu but prices in a market, and that the market is set by two budgets that most teams never write down. Once the budgets are explicit, several things that look like open questions turn out to have answers. Should we use the larger model at 2 bits or the smaller one at 4? The device arithmetic says both read about the same number of bytes per token, so they will decode at similar speeds. The precision-scaling evidence says that heavily trained models — which small modern models are — lose more under aggressive post-training quantization than older models did. The evaluation chapter says the loss will show up first in long, multi-step, structured behaviour, which is what most products depend on. Unless someone has done quantization-aware training to 2 bits, the smaller model at 4 bits is the better bet, and the place to spend effort is in specialising it. Should we prune? Ask what the hardware does with a zero. On almost every phone and laptop the answer is nothing, so unstructured sparsity saves nothing that matters. Structured pruning saves everything, but it is a training project. For most teams the right "pruning" is cheaper and more mundane: drop the vision encoder you do not use, cap the context you do not need, and pick a model whose attention design does not waste cache. Will the NPU make it faster? It will make prefill faster and decode more efficient. It will not make decode faster than the memory bus allows, because nothing will. The question to ask of any accelerator is which phase of generation it speeds up and whether that phase is the one users complain about. Is the compressed model as good as the original? Not if the only evidence is that its benchmark score moved by less than a point. Ask how many answers flipped, how far its distributions diverged, and what happened on the long, formatted, multilingual, multi-turn tasks that the product actually performs. What the field has settled Some things are now well established. Weight-only quantization to about 4 bits, with a calibration-aware method and calibration data that resembles real use, is close to free for most models and should be the default. Activation outliers are the central obstacle to quantizing language models more aggressively, and the effective answers are to move them into the weights (smoothing), spread them out (rotations), or train the model to live with them (QAT). Distillation from a stronger model is the most effective way to give a small model capabilities it lacks, and self-distillation from the original model is the most effective way to repair compression damage. Low-rank adapters make specialisation cheap enough to be routine and let one compressed base serve many tasks. And speed at the edge is governed by bytes moved per token, which ties every one of these techniques back to the same measurement. The tools have matured around these conclusions. A model can be taken from an official checkpoint to a calibrated 4-bit file in minutes, fine-tuned with adapters on a laptop, and deployed through runtimes that span phones, PCs and embedded boards. Vendors increasingly do the hardest step — quantization-aware training — themselves and publish the result. What remains unsettled Other questions remain genuinely open, and a practitioner should hold views on them loosely. How low can precision go? Apple's 2-bit on-device model shows that quantization-aware training at very low bit-widths can ship in a consumer product. Ternary models such as BitNet b1.58 show that models trained from the start at under 2 bits can be competitive at small scale. Whether these approaches will become the norm depends on whether vendors choose to pay for quantization-aware or native low-bit training of their best models, and on whether hardware arrives that turns the theoretical efficiency of ternary arithmetic into real energy savings. What will the next architectures demand? Hybrid models that replace most attention with linear recurrences, mixture-of-experts models small enough for laptops, and multimodal models with large embedding tables each change which bytes dominate and which methods apply. Quantization methods tuned on standard transformer layers will need adjusting for recurrent state, and NPU compilers must learn new operators. The budgeting discipline transfers; the specific recipes may not. How should compressed models be evaluated? The field has better drift metrics than it had a few years ago, but there is still no standard, widely adopted evaluation that captures what compression does to long-horizon, agentic and safety-relevant behaviour. Until there is, every team must build its own, and published claims of "lossless" compression should be read as claims about the benchmarks reported, not about behaviour in general. Where is the boundary between device and cloud? As on-device models improve, the set of tasks that must leave the device shrinks, but it does not vanish. Hybrid systems that route requests between a local model and a remote one — as several platform vendors now do — raise their own questions of privacy, consistency and evaluation that compression alone does not answer. What to do differently For a reader about to start an edge project, the practical conclusions are short. Write down both budgets before choosing anything, and derive the size class from the memory and bandwidth arithmetic rather than from model-card scores. Build the product evaluation first, including drift metrics against the uncompressed model, and run it on every artefact. Choose the base model and runtime together, and confirm that the architecture runs well on the target silicon before compressing anything. Quantize to about 4 bits with a calibrated method and representative calibration data; take a vendor's QAT checkpoint whenever one fits. Spend training compute — adapters, distillation, QAT — only where the evaluation shows a specific gap, and train against the exact weights you will ship. Use the runtime's protections, especially constrained decoding, to cover the fragility that compression introduces. Measure on the slowest device you support, for longer than feels necessary. None of this is glamorous. It is, however, the difference between a compressed model that looks good in a table and one that works in a user's hand. Compression for the edge rewards discipline more than cleverness: the cleverest method in this book will not rescue a project whose budgets were never written down, and the plainest methods, applied in the right order and measured honestly, are enough to ship most of what people want to build. Glossary Activation. An intermediate value computed inside the network for a particular input, such as the input to a linear layer. Unlike weights, activations change with every input. Calibration data. A small set of representative inputs passed through a model to collect statistics used by a quantization or pruning method. No training takes place. Decode. The phase of generation in which the model produces one token at a time. It is usually limited by memory bandwidth. Distillation. Training a student model to imitate a teacher model, typically by matching the teacher's output probability distributions rather than only the correct tokens. Group-wise quantization. Quantization in which each small block of weights, commonly 32 to 128 values, has its own scale factor. Grouped-query attention. An attention design in which several query heads share each key and value head, reducing the size of the KV cache. KV cache. The stored keys and values for every processed token at every layer. It lets the model attend to earlier tokens without recomputing them, and it grows with context length. KL divergence. A measure of how much one probability distribution differs from another. In this book it is used to measure how far a compressed model's predictions have drifted from the original model's. LoRA. Low-rank adaptation. A model is fine-tuned by learning a small update expressed as the product of two thin matrices while the original weights stay frozen. NPU. Neural processing unit. A dedicated accelerator for tensor operations, usually optimised for low-precision integer arithmetic. Outlier. An activation or weight value far larger than typical values in its tensor. Outliers make low-precision quantization difficult. Prefill. The phase of generation in which the model processes the whole prompt at once and builds the KV cache. It is usually limited by compute. Post-training quantization (PTQ). Quantizing a trained model without further training, sometimes with calibration data. Quantization-aware training (QAT). Training or fine-tuning with quantization simulated in the forward pass, so that the model learns weights that survive rounding. Rotation. Multiplying weights and activations by an orthogonal matrix, often a Hadamard matrix, to spread outlier values evenly before quantization without changing the network's function. Speculative decoding. A technique in which a cheap draft proposes several tokens and the main model verifies them in one pass. It produces several tokens for roughly the memory cost of one. Structured pruning. Removing whole neurons, attention heads, channels or layers, so that the result is a smaller dense model. Time to first token. The delay between submitting a request and receiving the first generated token. It is dominated by prefill. Further Reading Foundations of quantization for language models Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer, "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale" (2022). The paper that documented emergent activation outliers and why they break naive quantization. Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh, "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers" (ICLR 2023). Second-order error compensation that made 4-bit and 3-bit weights practical. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan and Song Han, "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" (MLSys 2024). The clearest statement of why activation statistics decide which weights matter. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth and Song Han, "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models" (ICML 2023). Migrating activation difficulty into weights for W8A8 inference. Saleh Ashkboos and colleagues, "QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs" (2024), and Zechun Liu and colleagues, "SpinQuant: LLM Quantization with Learned Rotations" (ICLR 2025). The two key papers on rotation-based quantization. Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov and Christopher De Sa, "QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks" (ICML 2024). The route below 3 bits. Tanishq Kumar and colleagues, "Scaling Laws for Precision" (2024). Why heavily trained models are harder to quantize after training. Cache, pruning and distillation Zirui Liu and colleagues, "KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache" (ICML 2024), and Coleman Hooper and colleagues, "KVQuant" (NeurIPS 2024). How keys and values differ and how to quantize each. Elias Frantar and Dan Alistarh, "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot" (ICML 2023), and Mingjie Sun, Zhuang Liu, Anna Bair and J. Zico Kolter, "A Simple and Effective Pruning Approach for Large Language Models" (Wanda, ICLR 2024). Saurav Muralidharan and colleagues, "Compact Language Models via Pruning and Knowledge Distillation" (NVIDIA, 2024). The Minitron recipe, the most complete published account of structured pruning followed by distillation. Geoffrey Hinton, Oriol Vinyals and Jeff Dean, "Distilling the Knowledge in a Neural Network" (2015), and Rishabh Agarwal and colleagues, "On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes" (GKD, ICLR 2024). DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (2025). Includes the distilled small reasoning models and the comparison with applying reinforcement learning directly to small models. Adaptation Edward Hu and colleagues, "LoRA: Low-Rank Adaptation of Large Language Models" (2021), and Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer, "QLoRA: Efficient Finetuning of Quantized LLMs" (NeurIPS 2023). Yixiao Li and colleagues, "LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models" (ICLR 2024). Using adapters to absorb quantization error. Shipped systems and evaluation Apple, "Apple Intelligence Foundation Language Models Tech Report 2025" (arXiv 2507.13575). Describes 2-bit quantization-aware training, 4-bit embeddings, an 8-bit KV cache, KV-cache sharing and quality-recovery adapters for the on-device model. Shuming Ma and colleagues, "BitNet b1.58 2B4T Technical Report" (Microsoft Research, 2025). The first open native ternary model at 2 billion parameters. Abhinav Dutta and colleagues, "Accuracy is Not All You Need" (Microsoft Research, 2024). Why aggregate accuracy hides behavioural change in compressed models, and the case for flip and divergence metrics. Yaniv Leviathan, Matan Kalman and Yossi Matias, "Fast Inference from Transformers via Speculative Decoding" (ICML 2023). The original account of speculative decoding. Tools The llama.cpp repository and its documentation on quantization types, importance matrices and the perplexity and KL-divergence tools; the documentation for MLX, ExecuTorch, LiteRT-LM and ONNX Runtime GenAI; and the Hugging Face PEFT library. Each changes quickly, and the current documentation is a better guide than any snapshot. Hashtags: #SmallLanguageModels #SLM #ArtificialIntelligence #MachineLearning #DeepLearning #EdgeIntelligence #EdgeAI #OnDeviceAI #ModelCompression #NeuralNetworkPruning #ModelPruning #Quantization #ModelQuantization #PostTrainingQuantization #QuantizationAwareTraining #LowBitInference #FourBitQuantization #KnowledgeDistillation #TeacherStudentLearning #LowRankAdaptation #LoRA #EfficientAI #LightweightLanguageModels #MemoryOptimization #KVCacheOptimization #InferenceOptimization #MobileAI #EmbeddedAI #LocalLLM #PrivacyPreservingAI

Download the Book (PDF): Introduction At a signalized intersection in almost any city, a small cabinet sits on the corner. Inside it, a controller decides, many times a minute, which movements get green and for how long. For most of the twentieth century that decision was written in advance: an engineer counted cars on a weekday in spring, worked out a cycle length and a split of green time, and loaded the plan into the box, where it might run unchanged for a decade. The city around it changed. A school opened, a supermarket closed, a bus route moved, a bridge two kilometres away was shut for repairs. The plan in the box did not know. That gap, between the city as it was measured once and the city as it actually behaves today, is the opening for machine learning in municipal infrastructure. Cameras, loop detectors, fare cards, smart meters and connected vehicles now produce a continuous record of how people and energy move. Models trained on that record can estimate queues from video, forecast how many passengers will board a bus at 8:10 tomorrow, and predict the load on a distribution feeder before the evening peak. In principle, a city that used all of this would retune itself constantly, the way a good driver adjusts to the road rather than to a map drawn years ago. In practice, the record is mixed. There are real successes. Adaptive signal systems have cut travel times measurably on real corridors. Transit agencies use passenger counts and fare data to plan service with a precision that was impossible twenty years ago. Utilities forecast load and solar output well enough to run grids with large shares of variable generation. There are also expensive failures: sensor networks installed and then abandoned, dashboards nobody consulted, pilot projects that worked in a press release and nowhere else, and at least one high-profile smart district that collapsed under the weight of public distrust before a single building went up. This booklet argues that the difference between the successes and the failures is rarely the model. It is whether the model has been placed inside a working control loop. A city system improves only when four things are in place at once: a sensing layer that measures what matters reliably, a decision layer that turns measurements into actions, an actuation layer that can carry those actions out on real equipment, and an institutional layer that decides what the system is for, who answers when it goes wrong, and how its effects are judged. Machine learning is powerful in the second of these and increasingly useful in the first. It does nothing for the third and fourth, and those are where most projects stall. Why the loop matters more than the algorithm The point sounds abstract, so consider a concrete case. Suppose a city installs cameras at fifty intersections and trains a state-of-the-art detector to count vehicles by lane. The counts are excellent. What happens next? If the signal controllers at those intersections cannot accept new timing parameters except by an engineer driving out with a laptop, the counts will feed a report. If the controllers can accept new parameters but there is no agreed objective for what the timings should optimize, the engineer will tune for car delay, because that is what the profession has always measured, and buses and pedestrians will be served as an afterthought. If the objective is set but there is no way to tell whether the new timings actually helped, because the counts after the change are compared against counts from a different season, the city will believe whatever the vendor tells it. Each of those failures is outside the model. Each is common. The same pattern runs through transit and energy. A demand forecast that is accurate to within a few percent is worthless if the scheduling department can only change timetables twice a year. A load forecast for a neighbourhood is valuable only if someone can act on it, by shifting electric vehicle charging, dispatching a battery, or deferring a transformer upgrade, and if the tariff and regulatory arrangements permit that action. The model is one link in a chain, and the chain breaks at its weakest link. This is why the booklet treats machine learning as one component of urban infrastructure rather than as a product to be bought. It explains what the models do, in enough detail to judge claims about them, and then spends equal effort on the plumbing around them. What the booklet covers The first chapter sets out the control-loop framing and the features that make cities harder than the settings where machine learning first flourished: equipment that lasts decades, objectives that conflict, people who adapt to whatever the system does, and a public that has a right to understand decisions made about its streets. Chapters 2 and 3 take the description's first theme, traffic. Chapter 2 is about sensing: how computer vision turns camera feeds into counts, queues, speeds and turning movements, where it fails, and why the camera is often not the right sensor. Chapter 3 is about control: how signal timing has evolved from fixed plans through the adaptive systems of the 1980s to schedule-driven and learned controllers, what the published results from real deployments actually show, and why reinforcement learning, so impressive in simulation, has been slow to reach the street. Chapters 4 and 5 turn to public transport. Chapter 4 covers demand forecasting from fare card, passenger counter and location data, at horizons from minutes to years, including how the pandemic broke models trained on the world before 2020. Chapter 5 follows the forecast into operations: scheduling, managing bus bunching, crowding information for riders, and the mixed history of algorithmically dispatched on-demand transit. Chapter 6 covers the electrical side of the city: forecasting load and rooftop solar on distribution networks, managing electric vehicle charging, and the work that machine learning can and cannot do on a grid whose physics are well understood but whose edges are increasingly unobserved. Chapter 7 steps back from operations to planning: simulation, the much-abused idea of the digital twin, and the gap between predicting what people will do next week and predicting how a city will respond to a new road or rail line over twenty years. Chapter 8 deals with governance: procurement, privacy, equity, accountability, and the regulatory frameworks, including the European Union's AI Act, that now treat traffic and energy systems as high-risk applications. The conclusion draws out what follows for anyone commissioning, building or scrutinizing these systems. Who it is for The intended reader is someone with a stake in how a city runs and enough technical curiosity to want to understand the machinery: a city engineer or planner, a transport or utility manager, a councillor or policy adviser, a data scientist moving into public infrastructure, or a resident who wants to know what is happening inside the cabinet on the corner. No mathematics beyond arithmetic is required. Where a technique matters, it is explained in plain terms with an example. The booklet is not a vendor survey, and it names products only where a published evaluation or a well-documented deployment makes them instructive. Nor is it a manifesto for or against the smart city. The phrase has been used to sell a great deal of equipment and to attract a great deal of justified suspicion. The more useful question is narrower: in which parts of municipal infrastructure does learning from data reliably improve outcomes, what does it take to get those improvements, and how should a city know whether it has got them? That question has reasonably clear answers, and they are the subject of what follows. Chapter 1: The City as a Control Problem Every piece of municipal infrastructure that moves something, whether vehicles, passengers, water or electrons, is already run by a control system of some kind. Sometimes it is a sophisticated automated loop. More often it is a person with a spreadsheet, a fixed plan in a cabinet, or a set of rules that someone wrote down after a bad winter. Machine learning enters a city not as a replacement for nothing but as a change to one of these existing loops. To judge whether that change will help, it is necessary first to see the loop clearly. Four layers, one loop A useful way to describe any city system is as four layers stacked on top of each other. The first is sensing: the instruments that measure what is happening. For traffic, that means inductive loops cut into the pavement, radar and video detectors on poles, probe data from phones and connected vehicles, and manual counts. For public transport, it means automatic vehicle location on buses and trains, automatic passenger counters at doors, and fare collection records. For electricity, it means meters at substations, smart meters at homes and businesses, and increasingly the inverters on rooftop solar and the chargers of electric vehicles. The second is estimation and decision: the logic that turns raw measurements into a picture of the current state, predicts how that state will evolve, and chooses what to do. This is where machine learning lives most naturally. A detector produces a stream of pixels or pulses; a model turns it into "eleven cars queued in the left-turn lane." A history of boardings becomes a forecast of tomorrow's ridership. A history of feeder loads becomes an estimate of when a transformer will overheat. The third is actuation: the equipment that carries decisions out. Signal controllers that change phase timings, dispatchers who add a bus or hold one at a stop, switches and batteries and chargers on the grid. Actuation is physical and usually old. Its constraints, what it can accept, how fast it can change, how it fails, set hard limits on what any decision layer can achieve. The fourth is institutions: the objectives the system serves, the people accountable for it, the budgets and procurement rules that shaped it, and the public whose streets and bills it affects. This layer decides what "better" means. It also decides whether anyone will notice when the system makes things worse. A system works only when information flows all the way around: sensors feed estimates, estimates feed decisions, decisions reach the actuators, the effects show up in the sensors again, and the institution looks at the result and adjusts the objectives. Break the loop anywhere and the rest does not matter. The most accurate camera analytics in the world deliver nothing if the controller cannot accept new timings. A beautifully engineered adaptive controller will drift into irrelevance if nobody is responsible for checking its detectors. The practical consequence is that the first question to ask about any proposed use of machine learning in a city is not "how accurate is the model?" but "which loop is this joining, and what is currently the weakest link in it?" Very often the honest answer is that the weakest link is not the decision layer at all. Many US cities, for example, still operate signals that have not been retimed in years, not because good timing plans are hard to compute but because retiming requires counts, staff time, and access to the controllers, and those are what is scarce. Federal Highway Administration guidance has long recommended reviewing signal timing every three to five years; surveys of practice have repeatedly found that many agencies fall short of that. A machine-learning project that improves the count data but leaves the staffing and controller access unchanged will tend to produce better reports rather than better signals. What makes cities hard Machine learning made its reputation in settings with particular properties: abundant data, a clear objective, fast feedback, and cheap mistakes. A recommendation engine learns from billions of clicks, optimizes a measurable engagement target, sees the result of each choice within seconds, and loses little if it shows someone the wrong film. Cities have almost none of these properties, and it is worth being specific about why. Equipment lasts for decades. A signal controller may stay in service for fifteen or twenty years; a transformer for forty or more; a rail signalling system longer still. The installed base in any city is a mixture of generations, protocols and vendors. A learned controller that needs a modern processor and a fibre connection at every intersection is not deployable across a network where half the cabinets are on copper telephone lines. Much of the practical work in this field is adapting good ideas to old equipment. Objectives conflict. At an intersection, giving more green to the main road reduces delay for most drivers and increases it for the side street, for pedestrians waiting to cross, and sometimes for the bus that turns left. Shortening pedestrian crossing times saves vehicle delay and costs the elderly and disabled. Optimizing a bus network for total ridership tends to concentrate service where density is highest; optimizing for coverage spreads it thinly so that everyone has some service. These are value choices. A learning algorithm needs a single number to optimize, and choosing that number, or the weights in a combination of numbers, is a political act that is often made silently by an engineer or a vendor. People adapt. Unlike a robot arm or a server farm, the system being controlled is made of people who notice changes and respond to them. Retime a corridor so that it flows smoothly and some drivers who previously avoided it will switch to it, eroding part of the gain. Publish real-time crowding on trains and riders will shift to the less crowded cars or departures, which changes the crowding pattern the model was trained on. Economists Gilles Duranton and Matthew Turner documented a version of this at metropolitan scale in their 2011 paper "The Fundamental Law of Road Congestion," finding that vehicle travel in US metropolitan areas rose roughly in proportion to the expansion of interstate highway lane-kilometres. Signal optimization is not road building, but the same logic applies in miniature: capacity created by better control is partly absorbed by new demand. Feedback is slow and noisy. A change to signal timing shows up in travel times within days, but the measured effect is buried in variation caused by weather, school calendars, roadworks and events. A change to bus frequency takes months to affect ridership. A change to the design of a street takes years to affect land use. Separating the effect of an intervention from everything else happening in a city is a serious statistical problem, and it is frequently skipped. Mistakes are public and sometimes dangerous. A poorly timed signal frustrates thousands of people at once and can cause crashes. A grid control error can leave a neighbourhood without power. A transit forecast that underestimates demand leaves people on the platform in the rain. Unlike a website, a city cannot easily run randomized experiments on its residents, and when it does, it has to be able to justify them. The public has standing. Decisions made by a city are, in most democracies, subject to rights of explanation, appeal and political oversight that do not apply to private products. A resident whose street is suddenly full of diverted traffic can ask a councillor why. "The model decided" is not an acceptable answer, and increasingly it is not a legal one either. None of these properties makes machine learning useless in cities. They make it a different discipline from machine learning on the web, one where the surrounding engineering, measurement and governance count for as much as the model. Where learning genuinely helps Against that backdrop, it is possible to identify the kinds of task where learning from data has a strong and reasonably proven advantage over the methods it replaces. The first is perception: turning rich, unstructured sensor data into structured measurements. Video, lidar, radar returns and smartphone traces contain far more information than a loop detector's on-off pulse, but extracting it requires recognizing objects, tracking them and classifying them. This is exactly what deep learning does well. Computer vision has made it feasible to count turning movements, classify vehicles, measure pedestrian volumes and detect near-misses from cameras that cities already own. Chapter 2 examines how this works and where it breaks. The second is short-term prediction from rich history: forecasting the next fifteen minutes of traffic on a motorway, tomorrow's boardings on a bus route, the next day's load on a feeder. These are problems with large amounts of historical data, stable structure punctuated by events, and many interacting inputs, which is the regime where gradient-boosted trees and neural networks consistently beat simpler statistical baselines, though often by less than their promoters claim. Chapters 4 and 6 examine prediction for transit and energy. The third is filling gaps in observation: estimating quantities that are not measured directly. A city has detectors at some intersections and not others; a utility has smart meter data but not a real-time view of every low-voltage circuit; a transit agency knows where passengers tapped on but often not where they got off. Learned models can infer the missing pieces from the observed ones, with quantified uncertainty if done well. The fourth, still the least proven in practice, is control itself: learning a policy that chooses actions, such as signal phases, bus holding times or charging schedules, by trial and error in a simulator or from logged data. This is where the most dramatic research results have been published and where the gap between simulation and the street is widest. Chapter 3 examines why. What these tasks have in common is that they sit inside the decision layer, and they produce their value only if the other layers are ready to use it. A short history of disappointment and progress Cities have been here before. Computerized traffic control dates to the early 1960s, when Toronto connected its signals to a central computer, and by the late 1970s and early 1980s two adaptive systems that are still in wide use had been developed: SCATS, created in Sydney by the New South Wales Department of Main Roads, and SCOOT, developed by the UK's Transport Research Laboratory with industry partners. Both adjust signal timings in response to detector data, and both have been deployed in hundreds of cities. Los Angeles built its Automated Traffic Surveillance and Control system around its intersections in time for the 1984 Olympics and gradually extended it until, by 2012, it covered essentially all of the city's signals, more than 4,000 of them. These systems are a reminder that "real-time data-driven control" is not new; what is new is the richness of the data and the flexibility of the models. The more recent smart-city wave, from roughly 2010 onward, generated a different kind of lesson. In 2016 the US Department of Transportation ran the Smart City Challenge and awarded Columbus, Ohio, up to $40 million, with a further $10 million from Vulcan, Paul Allen's company, to demonstrate connected and automated transport. The programme produced some useful infrastructure and a great deal of documentation, but several of its flagship projects attracted far fewer users than planned, and the independent evaluations were sober about the measurable outcomes. In Toronto, Sidewalk Labs, a sister company of Google, proposed to build a sensor-rich district called Quayside on the waterfront. After two and a half years of intense controversy over data governance, privacy and the scale of the company's ambitions, it withdrew in May 2020, citing economic uncertainty. Neither of these is evidence that data-driven urban infrastructure cannot work. Both are evidence that projects led by technology rather than by a specific operational problem, and that underestimate the institutional layer, tend to fail. Meanwhile, less glamorous projects quietly delivered. Utilities improved load forecasting and integrated large amounts of solar and wind generation. Transit agencies adopted the General Transit Feed Specification, created in 2005 by Google and Portland's TriMet, which made schedules machine-readable and, with its real-time extension, allowed riders to see where their bus actually was. Traffic engineers used probe data from commercial providers to measure corridor performance continuously rather than with occasional floating-car studies. In each case the gain came from a specific operational problem, a data source fit for it, and an organization ready to act on the result. The three systems compared The booklet's three operational domains, traffic, public transport and electricity, sit differently on these dimensions, and the differences explain much of what follows. Traffic control has very fast feedback, decisions made every few seconds, and a well-defined actuator in the signal controller, but its sensing has historically been crude and the behaviour of road users is only loosely modelled. Public transport has rich data on its own vehicles and increasingly on its riders, but its main levers, timetables and network design, change slowly, and its daily operations rely on people, drivers and dispatchers, rather than automated actuators. Electricity distribution has precise physics and strict safety limits, a long tradition of engineering forecasting, and new actuators appearing at the edge in the form of chargers, batteries and smart inverters, but poor visibility into the low-voltage network that serves homes. These differences mean that the same technique plays different roles in each. Forecasting is a supporting tool in traffic, a central planning tool in transit and an operational necessity on the grid. Learned control is a live research topic in traffic, a modest one in transit and a cautious one in energy, where physical models already do much of what learning would offer. A city that applies one template across all three, for example by buying a single platform to "optimize" everything, is likely to get the fit wrong in at least two. They also interact. Signal timing determines how fast buses move. Bus electrification turns depots into large electrical loads. Electric vehicle charging links the transport system's daily rhythm to the grid's peak. Some of the most valuable opportunities lie at these junctions, and they are also where institutional boundaries, between a traffic department, a transit agency and a utility, are most likely to stop anything happening. How to read the claims The rest of this booklet examines specific applications, and many of them come with headline numbers: travel times down by a quarter, emissions down by a tenth, forecasts improved by a third. A reader needs a way to judge such claims. Five questions do most of the work. First, what was the baseline? An adaptive system compared with signal timings that had not been updated for a decade will look excellent, but some of that gain would have been available from a simple retiming. The fair comparison is with a well-maintained conventional system. Second, what was measured, where, and for whom? Delay on the main corridor is not the same as delay across the network, and car delay is not the same as person delay. Results reported only for the intersections that were changed may hide displacement to neighbouring ones. Third, was it measured or modelled? Many reported emissions savings come from applying an emissions model to estimated changes in stops and speeds, not from measuring emissions. That is a legitimate method, but the uncertainty is larger than a single headline percentage suggests. Fourth, how long did the effect last? Gains measured in the first month after a change can fade as drivers adapt, as detectors fail, or as the novelty of a pilot's attention wears off. Fifth, who reported it? Vendor case studies, city press releases, peer-reviewed evaluations and independent audits carry different weights. This is not cynicism; it is the ordinary discipline of evidence. With those questions in hand, the claims become easier to place. The chapters that follow apply them to each domain in turn, starting with the most visible sensor in the modern city: the camera. Chapter 2: Reading the Street: Computer Vision for Traffic Sensing For half a century the standard way to detect a vehicle at an intersection has been a loop of wire cut into the road surface. When a car stops over it, the metal changes the loop's inductance, and the controller registers a presence. Loops are cheap per unit, well understood, and reliable when they work. They also fail constantly: resurfacing cuts them, frost heave breaks them, and a survey of almost any large city will find a significant share out of service at any given time. More fundamentally, a loop answers only one question, whether something metallic is above it now, and from that one question traffic engineers have spent decades inferring everything else. A camera answers many questions at once. From the same video it is possible to count vehicles by lane, classify them as cars, trucks, buses, bicycles and motorcycles, measure their speeds, track the paths they take through an intersection, estimate the length of queues, count pedestrians waiting and crossing, and detect near-misses between road users. Cities already have tens of thousands of cameras on poles for surveillance and traffic monitoring. Turning those feeds into measurements is the most mature application of modern machine learning in urban traffic, and it is the necessary foundation for most of what follows. From pixels to measurements The pipeline that turns video into traffic data has four stages, and it is worth understanding each because each has its own failure modes. The first stage is detection: finding objects in each frame and drawing a box around each one with a class label. Modern detectors are convolutional or transformer-based neural networks trained on large labelled datasets. The YOLO family, first published by Joseph Redmon and colleagues in 2016, made real-time detection practical on modest hardware by predicting boxes and classes in a single pass over the image, and its successors remain popular in traffic applications. A detector outputs, for each frame, a list of objects with locations, classes and confidence scores. The second stage is tracking: linking detections across frames so that the car in frame 100 is recognized as the same car in frame 101. Tracking is what turns a series of snapshots into trajectories. Common methods combine a motion model, which predicts where each object should be in the next frame, with an appearance model, which checks whether the object found there looks like the one expected. When two vehicles pass close to each other, or one is hidden behind a bus, trackers can swap identities or lose objects and create new ones, and these errors propagate into counts and turning movements. The third stage is geometry: mapping positions in the image onto positions on the road. A camera sees the world in perspective, so a pixel near the bottom of the frame covers far less ground than one near the horizon. To measure speeds, distances and queue lengths, the system needs a calibration that relates image coordinates to real-world coordinates, usually by marking known points such as lane lines or stop bars. Calibration drifts when a camera is bumped, repointed by an operator, or moved by wind, and a drifting calibration quietly corrupts every speed and length measurement. The fourth stage is aggregation: turning trajectories into the quantities traffic engineers use. Virtual detection zones are drawn on the image, and a vehicle is counted when its track crosses a zone. Turning movements come from the zone where a track enters and the one where it leaves. Queue length comes from the furthest stopped vehicle behind the stop bar. Occupancy, the fraction of time a zone contains a vehicle, mimics what a loop would have reported, which matters because many signal controllers can only accept loop-style inputs. Each of these stages can be done at the camera, on a small processor in the signal cabinet, or on a central server. Processing at the edge reduces bandwidth and avoids sending identifiable video off-site, which simplifies privacy obligations. Central processing allows larger models and easier updates. Most current commercial systems process at the edge and send only counts and events to the centre, with the option of short video clips for verification. What the camera can measure that loops cannot The value of video detection lies less in replacing loops for the stop-line presence call than in the measurements loops never provided. Turning movement counts. Signal timing depends heavily on how many vehicles turn left, go straight and turn right at each approach. Traditionally these counts were made manually, by people with clickers or by reviewing recorded video, for a few hours on a few days, and then used for years. Automated counts from cameras make it possible to have turning movements every fifteen minutes, every day. That alone changes what retiming can achieve. Vulnerable road users. Loops do not detect pedestrians at all and detect bicycles poorly. Cameras can count people waiting at crossings, measure how long they wait, detect people still crossing when the signal changes, and count cyclists by direction. This makes it possible to time signals for pedestrians on the basis of evidence rather than default values, and to extend crossing time automatically for a slow pedestrian still in the road. Several cities have used camera detection to trigger extended clearance intervals in this way. Classification. Knowing that a vehicle is a bus, a heavy truck or an emergency vehicle allows different treatment. Bus priority systems traditionally rely on transponders on the bus; camera classification can supplement them. Trucks matter for timing because they accelerate slowly and take up more of the queue. Conflicts and near-misses. Crashes are rare events, which makes them a poor basis for evaluating safety at a single intersection: years of data are needed to see a pattern, and by then people have been hurt. Traffic safety researchers have long used surrogate measures, such as post-encroachment time, the gap between one road user leaving a conflict point and another arriving at it. Trajectories from video make it possible to compute such measures automatically for every interaction, so that an intersection where right-turning vehicles routinely pass within a second of crossing pedestrians can be identified and fixed before the crash record accumulates. Bellevue, Washington, worked with Microsoft and the University of British Columbia on an early video-analytics programme of this kind under its Vision Zero commitment, and the approach has since been adopted by a number of commercial providers. Queue spillback. When a queue extends back past the upstream intersection, it blocks traffic that should be flowing, and the whole network can lock up. Loops placed at the stop bar cannot see this; cameras with a long view can. Detecting spillback in time to adjust upstream signals is one of the more valuable inputs to adaptive control. Where vision fails Computer vision in traffic is good enough to be useful and imperfect enough that a city should know exactly how imperfect. The common failure modes are well documented. Weather and lighting. Rain on the lens, snow, fog, low sun shining straight into the camera, headlight glare at night and deep shadows all degrade detection. Performance that is excellent on a dry afternoon can fall sharply at dusk in the rain, which is unfortunately also when traffic and crash risk are high. Thermal cameras, which detect heat rather than visible light, are more robust in darkness and glare and are increasingly used for pedestrian and cyclist detection, at a higher price. Occlusion. At a busy intersection, large vehicles hide smaller ones. A cyclist alongside a bus may be invisible for several seconds. Camera placement, height and angle matter enormously, and a camera installed for surveillance, at the angle that suits an operator watching a screen, is often badly placed for detection. Domain shift. A model trained mostly on images from one region may perform worse elsewhere. Vehicle types, road markings, signage and the mix of road users differ: auto-rickshaws in Delhi, mopeds in Ho Chi Minh City, cargo bikes in Copenhagen, snow-covered markings in Montreal. Off-the-shelf detectors trained on general-purpose datasets may misclassify or miss road users that are rare in their training data. Any city procuring vision analytics should ask what the system was trained on and ask for validation on its own streets. Silent degradation. The most dangerous failure is the one nobody notices. A loop that breaks usually reports a fault or a constant call that is obvious in the data. A camera that has been knocked a few degrees off its calibration, or whose lens is gradually coated with grime, continues to produce plausible-looking counts that are wrong. Good systems monitor their own health, comparing counts against historical patterns and against neighbouring sensors, and flag anomalies. Many do not. The practical answer to all of these is validation: comparing automated counts against ground truth collected independently, at the city's own intersections, across conditions that include night, rain and peak periods. The comparison should report errors by class and by movement, not just a single overall accuracy figure, because a system that counts cars within two percent but misses a third of cyclists will distort precisely the decisions that matter most for safety. Validation should also be repeated, since performance changes when software is updated or cameras are moved. Choosing the sensor Cameras are not always the right answer, and the choice among sensors is a practical engineering decision rather than a matter of fashion. The main options differ on what they measure, how they fail, and what they cost to install and keep running, as Table 1 summarizes. Table 1. Main traffic detection technologies compared. Sensor Measures well Main weaknesses Privacy exposure Inductive loop Presence, counts at a point Cut by roadworks; no pedestrians Very low Radar Speed, presence in all weather Limited classification; no turns Very low Video Counts, turns, classes, pedestrians Weather, glare, occlusion, calibration High unless processed at edge Thermal camera Pedestrians, cyclists in darkness Higher cost; lower detail Low to moderate Lidar Precise 3D trajectories Cost; rain and snow; maturity Low Probe data Travel times, speeds on links No counts; sample bias; contracts Depends on supplier Two of these deserve further comment. Probe data, meaning speeds and travel times derived from phones, navigation apps and connected vehicles, has transformed corridor performance measurement. It requires no hardware in the road and covers every street that enough vehicles travel. But it measures a sample of vehicles, not all of them, and the sample is skewed toward people using particular apps or vehicles. It gives travel times well and volumes poorly. It also comes with contractual terms that may restrict how the city can use, publish or keep the data, which matters for transparency and for building long-term records. Lidar, which measures distance with laser pulses, produces precise three-dimensional point clouds that are excellent for trajectory tracking and naturally anonymous, since it captures shapes rather than faces or number plates. Prices have fallen substantially with the automotive industry's investment, and several cities have piloted lidar at complex intersections. It is still more expensive than cameras and less proven in heavy precipitation. In practice the best results usually come from fusion: radar for stop-line presence and speed in all weather, video for turning movements and pedestrians, probe data for corridor travel times, each validated against the others. Fusion is itself a machine-learning problem, since the model must learn how much to trust each source under each condition. Detecting incidents Intersections are where signals are timed, but some of the most costly delays in a city begin elsewhere: a broken-down van in a tunnel, a collision on an urban motorway, debris in a lane. Every minute an incident goes undetected adds to the queue behind it, and the queue keeps growing after the incident is cleared. Traffic engineers have a long-standing rule of thumb that each minute of blockage generates several minutes of delay afterwards, and secondary crashes in the tail of a queue are a recognized hazard. Detecting incidents quickly is therefore one of the most valuable things a sensing layer can do. The older method, automatic incident detection from loop data, looks for characteristic signatures: occupancy rising sharply upstream of a blockage while it falls downstream. Algorithms of this kind, such as the California algorithms developed in the 1970s, compare adjacent detector stations and raise an alarm when the pattern crosses a threshold. They work, but they generate many false alarms in heavy traffic, when ordinary congestion produces similar patterns, and they can be slow when detectors are far apart. Video analytics addresses both problems. A camera watching a tunnel or a motorway segment can detect a stopped vehicle, a pedestrian on the carriageway, a vehicle travelling the wrong way, or smoke, within seconds, and can show the operator the scene to confirm. Many road tunnels in Europe, where safety regulation after serious tunnel fires in 1999 made fast detection a priority, now use video-based incident detection as standard. Learned models have improved the trade-off between sensitivity and false alarms, particularly by learning what normal looks like at each camera, including its shadows, reflections and routine stops, rather than applying fixed rules. Probe data offers a third route. When vehicles reporting their speeds slow sharply on a link where traffic normally flows, an incident is a likely cause, and crowdsourced reports from navigation apps often arrive before any official detection. Several traffic management centres now fuse these sources, treating an app report as a prompt to look at the nearest camera. The lesson for the sensing layer is the same as at the intersection. The detection is only useful if someone acts on it: an operator who dispatches a recovery vehicle, a lane control system that closes the blocked lane upstream, signals that adjust to divert traffic. A detection system feeding an unstaffed control room at night delivers nothing. Privacy and the camera Video is the most privacy-sensitive traffic sensor by a wide margin. A camera that can count cars can, with different software, read number plates, and one that can count pedestrians can, in principle, identify them. Residents know this, and in many cities the installation of cameras for traffic purposes has become entangled with debates over surveillance and policing. The technical answer is privacy by architecture: process video at the edge, discard frames immediately after extracting counts and trajectories, never store or transmit raw video except short clips for validation under defined rules, and do not run identification models at all. When these constraints are built into the hardware and contract rather than left to policy, they are much more credible. The institutional answer is clear published rules on what the cameras do, who can access what, and whether data can be shared with police. Cities that have deployed traffic analytics without such rules have sometimes found that the analytics programme was halted not because it failed but because residents could not be confident about what else the cameras were doing. This is the first appearance of a theme that runs through the whole subject. The sensing layer is not a neutral technical matter. What a city chooses to measure, how, and with what safeguards determines what decisions it can make later and whether the public will let it make them. From measurement to decision A city with good camera-based sensing has something it has rarely had before: a continuous, detailed, multimodal record of how its intersections actually work. That record has value even without any automated control. It allows engineers to retime signals on the basis of current data rather than five-year-old counts, to identify the most dangerous intersections from conflict data rather than waiting for crashes, to evaluate changes properly by comparing before and after, and to set pedestrian and cycling timings on evidence. But the larger promise is to close the loop, to feed those measurements directly into signals that adjust themselves. That is the subject of the next chapter, and it is where the gap between what research papers report and what streets experience is widest. Chapter 3: Timing the Lights: From Fixed Plans to Learned Control A traffic signal is a device for dividing time. At a simple four-way intersection, the north-south movements and the east-west movements cannot both have green at once, so the signal gives each a share of a repeating cycle. Three numbers describe the arrangement: the cycle length, the time taken to serve every movement once; the split, the share of the cycle given to each movement; and the offset, the point in the cycle at which this signal turns green relative to its neighbours. Get the cycle and splits right and the intersection discharges its queues efficiently. Get the offsets right along a corridor and platoons of vehicles arrive just as the light turns green, the familiar "green wave." Almost everything in signal control, from the 1950s to today's learned controllers, is a method for choosing these numbers, or for abandoning the idea of a fixed cycle altogether and deciding phase by phase. The methods differ in how much they know about current traffic, how quickly they respond, and what they try to optimize. Understanding the lineage makes it much easier to judge the newest claims. Fixed-time plans and why they persist The foundation of the field is a formula published by F. V. Webster of the UK Road Research Laboratory in 1958, which gives an approximately delay-minimizing cycle length from the lost time at an intersection and the ratio of demand to capacity on each approach. Webster's work and its successors made it possible to compute good timings from traffic counts, and fixed-time plans built this way still run a large share of the world's signals. A well-designed fixed-time system is not primitive. Engineers typically prepare several plans, for the morning peak, midday, evening peak, evening and night, and a time-of-day schedule switches between them. Along arterial corridors, offsets are set to produce green waves in the dominant direction. Software packages such as TRANSYT, also originating at the UK's road research laboratory, and Synchro, widely used in North America, optimize cycles, splits and offsets across networks from counts and a traffic model. Fixed-time plans have two virtues that are easy to undervalue. They are predictable, which helps coordination and lets pedestrians and drivers learn the rhythm. And they do not depend on detectors, so they keep working when sensors fail. Their weakness is that they are only as good as the counts they were built on and they cannot respond to anything unexpected: an incident, a concert letting out, a diversion from roadworks, or simply the gradual drift in demand as the city changes. A plan that was optimal on the day it was computed degrades steadily, and the main problem in practice is that cities do not recompute often enough. Actuated and adaptive control The first step beyond fixed time is actuation: using detectors to extend or terminate green phases within limits. A side street gets green only when a vehicle is detected waiting; a main-road green is extended while vehicles keep arriving, up to a maximum. Fully actuated control is standard at isolated intersections in North America and works well where volumes are moderate. It responds to the present but does not plan ahead, and along a coordinated corridor it is constrained by the need to keep the cycle in step with neighbours. Adaptive systems go further by continuously re-estimating demand and adjusting cycles, splits and offsets across groups of intersections. The two longest-lived examples were developed around 1980. SCATS, the Sydney Coordinated Adaptive Traffic System, measures how efficiently each green phase is used, from detectors at the stop line, and adjusts cycle length and splits incrementally from cycle to cycle, choosing offsets from a library of pre-prepared plans. It is deployed in many cities in Australia, New Zealand, Asia and elsewhere. SCOOT, the Split Cycle Offset Optimisation Technique, places detectors upstream of the stop line, uses them to predict the profile of vehicles arriving at the next signal, and makes frequent small adjustments to splits, offsets and cycle length to reduce a weighted combination of delay and stops. It has been used in London, which runs it at thousands of signals, and in many other UK and international cities. Evaluations of adaptive systems over four decades show a consistent pattern. Compared with old, poorly maintained fixed-time plans, they often deliver large reductions in delay. Compared with freshly optimized fixed-time plans, the gains are smaller, typically in the range of a few percent to perhaps ten or twenty percent depending on how variable the traffic is, and they are largest where demand is unpredictable. Under heavy saturation, when queues fill the network, all methods struggle, because there is simply not enough capacity to go round. Adaptive systems also need well-maintained detectors, and a system running on half-broken detection may perform worse than a good fixed plan. Several US cities that bought adaptive systems in the 2000s later reverted to time-of-day plans because they could not keep the detection working. This history is the right backdrop for the newest generation of methods. The relevant question is never only "is this better than what we have?" but "is it better than the best conventional approach we could have with the same maintenance effort?" Schedule-driven control: the Surtrac example Between classic adaptive systems and fully learned controllers lies a family of methods that plan explicitly over a short future horizon. The best-documented example is Surtrac, short for Scalable Urban Traffic Control, developed at Carnegie Mellon University's Robotics Institute by Stephen Smith and colleagues. Surtrac abandons the fixed cycle. Each intersection treats the approaching vehicles, detected by cameras and radar, as clusters of jobs to be scheduled through the intersection, and solves a small scheduling problem every couple of seconds to decide the sequence and duration of phases that minimizes total waiting time over the next minute or so. It then shares its planned outflows with downstream neighbours, so that each intersection knows what is coming toward it. There is no central computer; coordination emerges from intersections exchanging their projected flows. This decentralization is what makes it scalable. A pilot began in June 2012 at nine intersections in Pittsburgh's East Liberty neighbourhood, a corridor where three major roads meet and traffic patterns change through the day. The researchers reported that, compared with the previous signal plans, travel times fell by more than a quarter on average and time spent waiting at signals by around 40 percent, with estimated emissions falling as well. The system was extended to further Pittsburgh intersections over the following years, and in 2015 the technology was spun out to a company, Rapid Flow Technologies, for commercial deployment. The Surtrac results are among the strongest real-world numbers in the field, and they deserve the same scrutiny as any. The comparison was with the existing plans, not with freshly optimized ones. The pilot corridor was chosen because it had a problem. The measurement period was relatively short. None of this invalidates the result, but it places it: Surtrac demonstrated that a decentralized, short-horizon planner using rich detection could run real intersections reliably and beat the incumbent by a large margin. It did not demonstrate that every city would see a quarter off travel times. Optimization without new hardware: probe-based retiming A different approach skips real-time control entirely and uses large-scale data to recommend better fixed-time plans, which engineers then load into existing controllers. This addresses the most common real problem, signals that have not been retimed, at very low cost. Google's Project Green Light is the most widely publicized example. It uses aggregated driving trends from Google Maps to model each intersection's traffic patterns, the existing signal timing inferred from how vehicles stop and start, and the interaction between neighbouring signals. It then recommends changes, such as adjusted splits or offsets that create green waves, which city engineers can review and implement on existing equipment. When Google announced the expansion of the programme in October 2023, it reported that it was live at 70 intersections in 12 cities, from Seattle and Manchester to Hamburg, Rio de Janeiro, Bangalore and Jakarta, with early estimates of up to 30 percent fewer stops and up to 10 percent lower emissions at those intersections. Google was explicit that the emissions figures were modelled rather than measured, using a US Department of Energy model that treated all traffic as a single vehicle type. Later reports from participating cities give a more grounded picture. Boston, which began working with the programme in 2024, reported that after adjusting 114 congested intersections across 20 neighbourhoods, unnecessary stops at those signals fell by about 20 percent on average and delays by about 13.5 percent, with the best locations improving by around a quarter. The interest of Green Light for this booklet is less the specific numbers than the architecture. It keeps the engineer in the loop, deciding whether to accept each recommendation. It uses existing hardware, so the actuation layer is not a barrier. It uses data the city did not have to collect. And it is essentially a tool for doing traditional retiming more often and more cheaply, which is what most cities most need. Its limitations follow from the same design: it produces static plans rather than real-time responses, it sees only vehicles whose drivers use the relevant apps, it says little about pedestrians and cyclists, and it creates a dependency on a single private data provider whose terms the city does not control. Reinforcement learning: promise and the gap to the street Reinforcement learning, in which an agent learns a policy by trying actions and observing rewards, seems almost designed for signal control. The state is the traffic at an intersection, the actions are phase choices, the reward is reduced delay, and a simulator can generate millions of training episodes without bothering anyone. Since the mid-2010s a large research literature has grown up, with methods such as IntelliLight, PressLight and CoLight, the last two from groups including Hua Wei and Zhenhui Li at Penn State, reporting substantial improvements over conventional controllers in simulation. Open benchmark environments such as CityFlow and SUMO-RL have made it easy for researchers to compare methods. A particularly instructive thread in that literature is max pressure control, proposed by Pravin Varaiya at Berkeley in 2013. Max pressure is not learned at all. At each decision point it gives green to the phase whose upstream queues most exceed its downstream queues, which turns out to be provably throughput-optimal under certain assumptions and works well in simulation with very little tuning. PressLight's contribution was to use this pressure as the reward for a reinforcement learner. The lesson is that a simple, well-understood rule captured much of what the learned controllers were discovering, and that learned methods should always be compared against strong simple baselines, not only against fixed-time plans. Despite the volume of research, reinforcement-learning signal control remains rare on real streets. The reasons are instructive, because they apply to learned control throughout city infrastructure. The simulator is not the street. A policy trained in simulation learns to exploit the simulator's assumptions about driver behaviour, detector accuracy and demand patterns. When those assumptions are wrong, as they always are to some degree, performance can collapse. Calibrating a microsimulation to reproduce a real network faithfully is itself a substantial project. Detection is imperfect. Most papers assume perfect knowledge of every vehicle's position. Real controllers see the world through noisy, sometimes failed detectors. A learned policy that is not trained to cope with sensor faults may behave unpredictably when one occurs. Safety constraints are hard. Signals must respect minimum green times, pedestrian clearance intervals, and conflict rules that are enforced by a separate safety unit in the cabinet. A learned controller must work entirely inside these constraints, which limits what it can do and requires careful engineering to integrate. Behaviour is hard to explain. An engineer can inspect a fixed-time plan or a SCOOT adjustment and understand why it did what it did. A neural policy offers no such explanation, which makes it difficult to approve, debug, defend to a councillor or adjust when residents complain. Gains over good baselines are modest. When learned controllers are compared with well-tuned actuated control or max pressure under realistic conditions, the advantage is typically much smaller than comparisons with fixed-time plans suggest. There have been real-world trials, notably in China, where large technology companies have worked with city authorities on combined camera analytics and signal optimization, of which Alibaba's City Brain project in Hangzhou, launched in 2016, is the best known. Published detail on such deployments is limited, which makes independent assessment difficult. Measuring what signals actually do One of the most useful developments in signal operations over the past fifteen years involves no machine learning at all, and it changes what learning can achieve. Modern signal controllers can record every event that happens inside them, each detector actuation and each change of phase, timestamped to a tenth of a second. Researchers at Purdue University, led by Darcy Bullock, working with the Indiana Department of Transportation, showed in the late 2000s that these high-resolution logs could be turned into detailed performance measures without any new sensors: how many vehicles arrive on green rather than red, how often a phase ends with vehicles still queued (a "split failure"), how well platoons are progressing along a corridor, and how long pedestrians wait after pressing the button. A common format for the logs was agreed, and the Utah Department of Transportation built an open-source system to compute and publish the measures across its whole network. The US Federal Highway Administration has since promoted these automated traffic signal performance measures as a recommended practice. The importance of this for the subject of this chapter is that it closes the loop for evaluation. Before such measures existed, a city could not easily tell whether a signal was performing well, except by sending engineers to watch it or waiting for complaints. With them, problems show up in the data, often within a day: a detector stuck on, a timing plan that leaves the left-turn queue uncleared every cycle, an offset that has drifted after a power cut. Engineers can prioritize their limited time on the signals that most need it, and they can see within days whether a change helped. High-resolution logs are also exactly the data that learned methods need. Arrival patterns, queue clearance and phase utilization, recorded continuously, provide both the training data for models and the yardstick against which any new controller should be judged. A city that has this measurement infrastructure in place is in a far stronger position to evaluate a vendor's claims, or a probe-based retiming recommendation, than one that does not. In practice, getting performance measures running is often the best first investment a signal department can make, before any adaptive or learned control. Priority for buses and pedestrians Signal control is often discussed as if every vehicle counted equally, but most cities have explicit policies favouring some road users, and the control layer is where those policies are made real or quietly ignored. Transit signal priority gives an approaching bus or tram an earlier or longer green. In its simplest form, a bus sends a request through a transponder or a radio message and the controller extends the current green by a few seconds or brings the next green forward. Modern systems make the request conditional: only buses running late request priority, which avoids disrupting cross traffic for buses that do not need help and keeps them on schedule rather than early. Predictions of when the bus will reach the stop line, which depend on whether it must first stop to board passengers, are a natural place for learned models, since a priority call placed too early or too late wastes green time. Evaluations of transit signal priority in many cities have found modest but worthwhile reductions in bus travel time and, more importantly, in its variability, which is what riders feel. Pedestrian treatments are just as consequential. A leading pedestrian interval, which gives people a few seconds of walk signal before the parallel traffic gets green, makes pedestrians more visible to turning drivers and has been associated in several before-and-after studies with substantial reductions in pedestrian crashes. Recalling the pedestrian phase automatically in busy areas, rather than requiring a button press, reduces waiting and non-compliance. Neither requires machine learning. Both require someone to have decided that pedestrian safety and delay count in the objective, which is precisely the institutional choice that a learned controller cannot make on its own. What a city should actually do The comparison across these approaches, set out in Table 2, suggests a fairly clear order of priorities for most cities. Table 2. Families of signal control compared. Approach Uses real-time data Hardware needs Typical role today Fixed-time plans No Minimal Most signals worldwide Actuated control Local detectors Detectors per approach Isolated intersections Adaptive (SCATS, SCOOT) Network detectors Detectors and comms Large city networks Schedule-driven (e.g. Surtrac) Rich detection Cameras/radar, comms Corridors, growing Probe-based retiming Historical probe data None beyond existing Low-cost retiming Reinforcement learning Detectors, simulator Varies Mostly research and pilots First, fix the basics. Inventory the signals, find the ones that have not been retimed, and retime them using current data. Probe-based tools and camera turning counts make this far cheaper than it used to be, and it is where most of the available gain lies. Second, fix the detection. Any form of responsive control depends on detectors that work. A maintenance programme that keeps them working, with automated monitoring to flag failures, is a precondition for everything else. Third, decide the objective explicitly. Choose, as a matter of published policy, how the city weighs vehicle delay against bus delay, pedestrian waiting and safety. Measure person delay rather than vehicle delay where possible, since a bus carrying sixty people is worth more than a car carrying one. Give pedestrians and cyclists their own measured performance indicators. Fourth, deploy responsive control where demand is genuinely variable and the detection and staffing can support it, and evaluate it properly against a well-tuned baseline, with before-and-after measurements that control for season and trend. Fifth, treat learned controllers as a research and pilot activity until they have been independently evaluated in conditions like the city's own, and insist that any such controller operates within an explicit safety envelope and can be explained to the people who must answer for it. This sequence is unglamorous. It is also where the evidence points. The most sophisticated controller in the world cannot compensate for broken detectors, an unstated objective or a network that has outgrown its timing plans. And for the many people who travel by bus, the question of who gets green time leads naturally to the larger question of how transit service is planned in the first place, which is the subject of the next two chapters. Chapter 4: Forecasting the Riders: Predicting Public Transit Demand A transit agency is in the business of placing capacity where people will be. Every timetable is a bet: that the 7:42 from the suburbs will fill, that the crosstown route needs a bus every eight minutes at midday but every twenty at night, that a new line will attract enough riders to justify its cost. Most of these bets used to be made with sparse data, an occasional on-board survey, manual counts on a sample of trips, and the planner's experience. Over the past two decades that has changed completely. Agencies now collect, as a by-product of operations, data on almost every trip a vehicle makes and a large share of the trips riders make. Forecasting demand from that data is one of the most productive uses of machine learning in urban infrastructure, and also one where the limits of learning from the past were exposed brutally in 2020. The data an agency already has Three operational systems produce most of the data that demand forecasting relies on. Automatic vehicle location records where each bus or train is, usually every few seconds to every thirty seconds, from GPS on buses and from track circuits or signalling data on rail. It tells the agency when vehicles actually arrived at each stop, which is the basis for measuring reliability and for matching other data to particular trips. Automatic passenger counters, typically infrared or stereo-camera sensors over the doors, count boardings and alightings at each stop. Not every vehicle carries them; many agencies equip a share of the fleet and rotate equipped vehicles across routes to build a statistical sample. Raw counts need cleaning, since sensors miss people in crowds, double-count, and drift, and the US National Transit Database has specific procedures for certifying that counter data are accurate enough to report. Automated fare collection records each smart-card or contactless payment tap. London's Oyster card, introduced in 2003, and the open-payment contactless system that followed it, together with similar systems in Hong Kong, Singapore, Seoul and many other cities, generate millions of records a day. Each record gives a time, a location and an anonymised card identifier. In systems where riders tap both on and off, as on the London Underground, every trip's origin and destination is observed. In systems where riders tap only on boarding, as on most bus networks, destinations must be inferred. Inference of destinations from tap-on data is a good example of the gap-filling role of learning described in Chapter 1. The classic method, developed in work on New York City's MetroCard data in the early 2000s and refined since, is trip chaining: assume that a rider's next boarding location is near where they alighted from the previous trip, and that the last trip of the day returns them near the first boarding of the day. Matched against vehicle location data, this assigns a probable alighting stop to a large share of trips. Machine learning extends the approach by learning from cards whose destinations are known to predict those that are not, and by handling exceptions, such as riders who walk a long way between trips or make part of their journey by another mode. Together, these sources allow an agency to build origin-destination matrices, the core input to network planning, continuously rather than from a survey every decade. Horizons: minutes, days and years "Demand forecasting" covers problems that differ so much in horizon that they need different methods and serve different decisions. Very short-term forecasts, from minutes to a couple of hours ahead, predict how many people will board a particular vehicle or arrive at a station soon. They feed real-time operations: whether to deploy a standby bus, hold a train to allow a connection, or warn riders that the next service is crowded. They depend heavily on what is happening right now, the passengers already on board, the arrivals at upstream stations, the current delays, and they need to update continuously. Short-term forecasts, from a day to a few weeks, predict ridership by route, stop and time of day for operational planning: assigning vehicles and staff, planning for events, preparing for holidays and weather. They depend on calendar effects, weather, scheduled events and recent trends. Medium-term forecasts, from months to a couple of years, inform service changes: how much frequency a route should get in the next timetable, how ridership will respond to a fare change, what a network redesign will do. These need models of how people respond to service, not only extrapolation of past patterns. Long-term forecasts, a decade or more ahead, are used to justify capital projects such as new rail lines. They depend on projections of population, employment and land use, and they belong as much to urban planning as to operations. Chapter 7 returns to them. The value of machine learning declines with the horizon. At the shortest horizons, where rich recent data and stable patterns dominate, learned models have a clear advantage. At the longest, where the question is how a city will change in response to a new piece of infrastructure, historical data contain little information about the counterfactual, and the methods are fundamentally different. What works at short horizons The research literature on short-term transit demand forecasting is large, and the methods map onto the general toolbox of time-series prediction. Table 3 summarizes the main families and where each tends to earn its place. Table 3. Main method families for short-term transit demand forecasting. Method family Strengths Weaknesses Typical use Historical averages by time slot Simple, robust, transparent Misses trends and events Baseline; fallback Statistical time series (e.g. ARIMA) Well understood; good uncertainty One series at a time; few covariates Station or route totals Gradient-boosted trees Handles many covariates; strong accuracy Needs feature engineering Daily and hourly forecasts Recurrent and attention networks Learn complex temporal patterns Data hungry; harder to explain Large systems, rich data Graph neural networks Use network structure across stations Complexity; benchmark-heavy evidence Metro systems; research Several lessons recur across studies and practice. First, the baseline is hard to beat. A historical average for the same time slot on the same type of day, adjusted for recent trend, captures most of the predictable variation in transit demand because travel patterns are highly regular. Work commuters board at similar times each weekday. Any model must be compared with this baseline, and the honest improvement is often in single-digit percentages of error for ordinary days. Second, the gains are concentrated on unusual days. Learned models earn their keep when something departs from routine: heavy rain, a football match, a school holiday, a disruption on a parallel line. These are also the days when operational decisions matter most. A model that predicts ordinary Tuesdays slightly better and match days much better is more valuable than its average error suggests. Third, covariates matter more than architecture. Adding information about weather, events, school calendars and disruptions typically improves forecasts more than switching from one model family to another. Building and maintaining clean feeds of these covariates is a data-engineering task that is easy to underestimate. Fourth, network structure helps for rail. On metro systems, demand at one station is linked to demand at others through the trips that connect them. Graph neural networks, which treat stations as nodes and connections as edges, have produced good results on benchmark datasets from systems such as those in Hangzhou, Shanghai and Beijing, where large open datasets exist. Their advantage over well-engineered tree-based models with network features is real but often modest. Fifth, uncertainty is as important as the point forecast. An operational decision, such as whether to deploy an extra train, depends on the risk of overcrowding, not on the expected number of riders. A forecast that says "probably 900 passengers, but a ten percent chance of more than 1,300" is far more useful than "900." Quantile regression, ensembles and probabilistic models all provide this, and agencies should ask for it. Crowding In many cities the most valuable short-term forecast is not total demand but crowding: how full each vehicle will be. Crowding affects comfort, dwell times at stops, accessibility for people with wheelchairs and pushchairs, and, since 2020, perceived health risk. Forecasting crowding requires combining demand forecasts with the actual service: which vehicles will run, how they will be spaced, and how passengers will distribute among them. A late bus collects the passengers who would have taken it plus some who would have taken the next, so crowding depends on reliability as much as on demand. Several agencies now publish crowding predictions to riders. Transport for London has shown how busy Underground stations typically are by time of day, and many journey-planning apps display predicted crowding on buses and trains based on agency data. The feedback loop mentioned in Chapter 1 is visible here: when riders can see that one train is crowded and the next is not, some will wait, which spreads the load but also changes the patterns the model was trained on. A crowding model that does not account for its own influence will drift over time. The pandemic shock In March and April 2020, transit ridership in cities around the world fell by well over half, and in many systems by more than eighty percent, within a few weeks. Every demand model trained on data from before 2020 became useless overnight. This was the most dramatic natural experiment in the history of transit forecasting, and it taught several lasting lessons. The first is that learned models encode the world they were trained in. A model that has learned that weekday morning demand on a commuter rail line is reliably high has learned a fact about the world that can stop being true. No amount of model sophistication protects against a structural break. What protects against it is monitoring: comparing forecasts against actuals continuously and alerting when errors move outside their normal range. The second is that recovery was uneven and structural. As restrictions eased, ridership did not return uniformly. Off-peak and weekend travel generally recovered faster than the weekday peak, and the most dramatic and lasting change was on Mondays and Fridays as hybrid office work took hold. Routes serving hospitals, shops and lower-income neighbourhoods, whose riders could not work from home, recovered faster than routes serving central business districts. Commuter rail into downtown offices recovered slowest of all in many North American cities. Models had to be rebuilt on the new patterns, and agencies had to decide how much weight to give pre-2020 data, which described a world that might partly return, against recent data, which described a world still changing. The third is that the questions changed. Before 2020, the main forecasting problem for most agencies was peak capacity. Afterwards, it became understanding a new and less concentrated pattern of demand, and in many cities designing service for all-day, all-week travel rather than for the commute. Forecasting tools built around the peak had to be redesigned. For anyone building or buying demand forecasts, the pandemic is a permanent warning. A model's accuracy on historical test data is a measure of how well it learned the past, not a guarantee about the future, and the system around the model must be able to detect when the world has moved. Events and disruptions Two kinds of unusual day deserve particular attention because they dominate the operational value of forecasts. Planned events, such as football matches, concerts, marathons and festivals, produce surges of demand at known places and times. Their effect is predictable in outline but not in detail: the same stadium can generate very different loads depending on the opponent, the weather, the kick-off time and whether the match goes to extra time. Agencies with good historical data can learn these effects. A model that knows the attendance, the start and end times and the past pattern of arrivals at the nearest stations can predict the post-event surge well enough to plan extra services and crowd control. The practical difficulty is often not the model but the data on events: maintaining a reliable calendar of what is happening, where, and how large, which requires cooperation with venues and the events office. Unplanned disruptions, such as a signal failure on a metro line, a closed road or a severe storm, redistribute demand across the network. When one line stops, its passengers spill onto parallel lines, buses and roads, and the knock-on effects can be predicted only if the model understands how the network connects. Fare data from past disruptions are a valuable record of how riders actually rerouted, and some agencies have used them to build models that estimate, within minutes of a disruption, which alternative routes will be overwhelmed and where replacement buses are most needed. Here the time horizon is minutes and the decision is immediate, which is the regime where learning helps most. From forecasting to understanding response At medium horizons, the planner's question shifts from "how many people will ride?" to "how many people will ride if we change the service?" This is a causal question. Historical data show what happened under the service that was operated, not what would have happened under a different one. Transit planning has long-established tools for this, notably elasticities: the percentage change in ridership associated with a one percent change in service or fare. A widely cited rule of thumb from the research literature is that ridership responds to frequency more than to fare, and more strongly on routes with infrequent service, where each additional bus cuts waiting time substantially. Elasticities are averages from many studies, and local response can differ. Machine learning can help sharpen these estimates when an agency has histories of past service changes, by modelling how ridership responded and controlling for other factors. But the core of the problem is causal inference, not pattern recognition. Natural experiments, such as a route that gained frequency while a similar one did not, are worth more than any number of correlations. Agencies with good data and a habit of structured before-and-after evaluation build up a body of local evidence that makes each subsequent decision better. Perhaps the most influential recent use of data in transit planning has not been forecasting at all but network redesign. Houston's 2015 bus network redesign, developed with the planning consultant Jarrett Walker, rebuilt the network around a grid of frequent routes, accepting longer walks for some riders in exchange for frequent service and easier transfers for most. Its analysis relied on mapping which places could reach which others within a given travel time under each network option, an accessibility measure that is computed from detailed schedule data and population and employment locations. The approach has since been used in many cities. It is a reminder that the most valuable output of transit data is often not a prediction but a clear, shared picture of what different choices would mean for riders. What to ask for For an agency or city building demand forecasting capability, several practical principles follow. Invest first in the data foundation: automatic vehicle location on the whole fleet, passenger counters on a statistically sound sample, fare data linked to trips, and clean feeds of calendar, weather and event information. The best model on poor data will be beaten by a simple one on good data. Benchmark every model against a well-constructed historical baseline and report improvement on ordinary days and unusual days separately. Require probabilistic forecasts where decisions depend on risk. Monitor forecasts against actuals continuously, with alerts for drift, and plan for retraining. Keep privacy in view. Fare card data are among the most sensitive data a city holds: a sequence of taps reveals where someone lives, works, and spends evenings. Aggregation, minimal retention, strong access controls and rules against sharing identifiable records should be built in from the start. And remember that a forecast is valuable only if it changes something. The next chapter follows demand forecasts into the operational decisions they are meant to inform: scheduling, dispatching, keeping buses evenly spaced, and the mixed record of algorithmically dispatched on-demand transit. Chapter 5: Running the Network: From Forecasts to Transit Operations A demand forecast is a statement about the future. A timetable is a promise. Between them lies the operational work of a transit agency: turning expected ridership into routes, frequencies, vehicle assignments and driver shifts, and then, on the day, keeping the service close to the promise despite traffic, breakdowns, absent staff and weather. Machine learning touches every stage of this process, but its contribution is uneven. In some places it adds real precision to decisions that were made roughly before. In others it has been oversold as a substitute for the basic economics of moving people in shared vehicles. From demand to timetable Transit planning traditionally proceeds in a fixed sequence, sometimes called the planning cascade: design the network of routes, set the frequency on each route by time of day, build the timetable of individual trips, assign vehicles to sequences of trips (vehicle blocks), and assign drivers to sequences of blocks (duties or runs), which are then grouped into rosters. Each stage constrains the next, and the later stages, vehicle and crew scheduling, are classic operations-research problems that have been solved with mathematical optimization since the 1970s and 1980s. Commercial scheduling packages used by most large agencies embed decades of refinement in these methods. Machine learning does not replace this optimization. It improves its inputs. Two inputs matter most. The first is running time: how long a trip takes from end to end, and between each pair of timing points, by time of day and day of week. Running times set the timetable. If they are too short, buses run late all day and drivers lose their breaks; if they are too long, buses run early and must wait at timing points, wasting riders' time and the agency's money. Traditionally, running times were set from a sample of observed trips. With automatic vehicle location on every vehicle, agencies can now estimate the full distribution of running times for every segment, every hour, every season. Learned models can predict how running times will change with traffic, roadworks or a new bus lane, and choose the scheduled time as a percentile of the distribution that balances lateness against waiting. The second is demand by trip: how many people will ride each scheduled trip. This determines where frequency is needed, where larger vehicles should be assigned, and where short-turn trips, which run only the busiest part of a route, would serve riders better than full-length trips. The forecasts described in Chapter 4 feed directly into this. The gains from better inputs are concrete. A timetable built on accurate running times runs closer to schedule, which makes connections reliable and reduces the buffer time added at the ends of routes. Since buffer time is paid driver time and vehicle time that carries no passengers, trimming excess buffer while maintaining reliability frees resources that can be put back into service frequency. Keeping buses apart The single most visible operational failure on a frequent bus route is bunching: two or three buses arriving together, followed by a long gap. Riders experience it as a long wait followed by a choice of empty buses. It is not bad luck; it is an inherent instability, well described in transport research since the 1960s. If a bus falls slightly behind schedule, more passengers accumulate at the stops ahead of it, so it spends longer at each stop and falls further behind. The bus following it finds fewer passengers waiting, moves faster, and catches up. Left alone, any small disturbance grows until buses travel in pairs. Agencies have always fought bunching by holding buses at timing points, telling an early bus to wait so that it falls back into its scheduled slot. Schedule-based holding works poorly on very frequent routes, where riders do not consult timetables and what matters is the even spacing of buses, called the headway. Headway-based control holds buses to keep them evenly spaced rather than on schedule. An elegant contribution came from John Bartholdi and Donald Eisenstein, who in a 2012 paper described a self-coordinating method: at a control point, each bus waits for a time based on the gap behind it, so that spacing equalizes automatically without a fixed schedule. They tested it on a bus route in Atlanta and found that buses maintained steadier headways with less management effort. Since then, a variety of real-time control strategies have been developed, including predictive holding that uses forecasts of where buses will be in the next several minutes and of passenger arrivals at upcoming stops to decide how long each bus should wait. Machine learning contributes here primarily through better short-term predictions, of travel times on upcoming segments and of passenger loads at upcoming stops, which a controller uses to choose holding times. Reinforcement learning for bus holding has been explored in research, with similar caveats to those for signal control: it is promising in simulation and largely untested at scale. The actuation layer once again sets the limits. Holding requires drivers to receive and follow instructions, which depends on the in-vehicle systems, the union agreement, and drivers' trust that the instructions make sense. Holding also delays the passengers on the held bus, so it trades their time against that of riders waiting downstream. And holding cannot correct a gap that has already become too large; that requires adding a vehicle, short-turning a bus or expressing one past some stops, which are dispatcher decisions with their own costs. The most effective fixes for bunching are often not algorithmic at all. All-door boarding, where riders can board at any door with a contactless card, reduces the time each passenger adds at a stop and so weakens the feedback that drives bunching. Bus lanes and signal priority reduce the variability of travel time. Algorithms work best when combined with these physical and policy measures. Real-time information The most widely felt result of transit data for riders has been real-time arrival information. Once the General Transit Feed Specification made schedules machine-readable, and its real-time extension, GTFS Realtime, published vehicle positions and predicted arrivals, journey-planning apps could tell riders when their bus would actually come. Research has consistently found that real-time information reduces the anxiety of waiting and the time riders perceive themselves to have waited, and some studies have found modest ridership gains after its introduction. Its value depends entirely on accuracy. A prediction that says "3 minutes" and turns out to be eight erodes trust quickly, and riders who have been misled stop believing it. Arrival prediction is a well-studied machine-learning problem. Early methods simply projected the scheduled running time from the bus's current position. Better methods learn from historical running times on each segment, conditioned on time of day and current conditions, and update as the bus moves. Some combine segment travel times observed from the buses just ahead, which carry information about current traffic. Predicting the end of a long route from its start remains difficult, because delays accumulate unpredictably, and good systems communicate their uncertainty honestly, for example by showing a range or by noting when a vehicle has not yet left its terminus. A further step, now common, is publishing predicted crowding alongside arrival times, as discussed in Chapter 4. Accessibility information, such as whether a station's lifts are working, is less glamorous and arguably more important for riders who depend on it. On-demand transit: what the algorithms could not fix No area of transit has attracted more algorithmic ambition, or more disappointment, than demand-responsive transport: services that do not follow fixed routes and timetables but dispatch vehicles in response to booked trip requests, pooling riders going in similar directions. The appeal is obvious. A fixed bus route serves a corridor; a dynamically routed vehicle could, in principle, take everyone where they want to go, with an algorithm solving the routing problem in real time. Research on algorithmic ride-pooling, including an influential 2017 paper by Javier Alonso-Mora and colleagues at MIT that simulated how a fleet of a few thousand shared vehicles could serve most of New York's taxi demand, suggested very large efficiencies. Practice has been harsher. Helsinki's Kutsuplus, an early on-demand minibus service launched in 2012 and widely praised for its technology, was closed at the end of 2015 because the cost per passenger was far higher than the city's conventional buses and scaling it up to reduce that cost was judged unaffordable. Bridj, a venture-funded on-demand shuttle service in Boston, shut down in 2017. A pilot with Bridj in Kansas City, run with the regional transit authority and Ford, carried very few riders over its year of operation. Many smaller pilots have followed similar paths. The reason is not poor algorithms. It is geometry and economics. A fixed-route bus with sixty passengers is efficient because those passengers have agreed to gather at stops along a single line. A dynamic service must detour to pick up and drop off each rider, and every detour costs time for everyone else aboard. As demand rises, pooling improves, but vehicles quickly fill up, and the service needs more vehicles and more drivers; since the driver is the largest operating cost, the cost per passenger stays high. In low-density areas, where fixed routes are weakest, demand is too sparse for much pooling, and the service behaves like a subsidized taxi. Algorithms can shave a few percent off detours. They cannot make a minibus carry as many people as a full bus, or a driver cost less. Demand-responsive transport does have real roles. It can replace unproductive fixed routes in low-density suburbs and rural areas, where a few riders an hour do not justify a bus but some service is a social obligation. It can serve people with disabilities, a function long met by paratransit services, more flexibly. It can provide first- and last-mile connections to frequent trunk lines. Arlington, Texas, a large US city without conventional fixed-route buses, has operated an on-demand service with Via since 2017 and expanded it to cover the whole city, providing a useful long-running example of the model at city scale, at a subsidy per trip that the city has judged worthwhile for its circumstances. The lesson from the past decade is to deploy on-demand services where their economics fit, not as a technological replacement for frequent fixed-route transit in dense areas. Rail and the limits of the timetable Most of this chapter concerns buses, because buses share streets with everything else and their operations are where data-driven control has the most room to act. Urban rail is different. A metro line runs on its own right of way under a signalling system that enforces safe separation between trains, and capacity is set by how closely trains can follow one another. Communications-based train control, in which trains report their positions continuously and the system calculates safe braking distances in real time, allows much closer spacing than traditional fixed-block signalling, and many metros have upgraded to it to run more trains per hour on existing track. Within that framework, the operational problems are recovery and crowding. When a train is delayed, the trains behind it bunch up just as buses do, and dispatchers must decide whether to hold trains, skip stations or short-turn services to restore an even pattern. Decision-support tools that predict how a delay will propagate and suggest recovery strategies are an active area for optimization and learning, and several large metros use such tools to advise controllers. The binding constraint during peak hours is usually station dwell time: when platforms are crowded, boarding takes longer, each train stays longer, and the line's capacity falls just when it is needed most. Forecasts of platform crowding, combined with staff deployed to manage boarding and with information that steers passengers to less crowded parts of the train, address this directly. The broader point is that a rail system's capacity is set by physical infrastructure and signalling, and data cannot create capacity that the track and stations do not have. What it can do is help operators run as close as possible to that capacity and recover quickly when something goes wrong. Staff and vehicles: the hidden constraints Transit operations depend on resources that are easy to overlook in discussions of algorithms: drivers and vehicles. Since 2020, many agencies in North America and Europe have struggled with operator shortages, cancelling scheduled trips because there was no one to drive them. A perfect forecast and a perfect timetable are worthless if the trips cannot be staffed. Forecasting has a role here too, if a less glamorous one. Predicting absence rates by day, depot and season allows agencies to plan the size of their reserve pool of drivers more precisely. Predicting vehicle failures from maintenance records and on-board diagnostic data allows maintenance to be scheduled before breakdowns rather than after, which keeps more vehicles available and reduces in-service failures that strand passengers. Predictive maintenance is one of the more mature industrial applications of learning, and transit fleets, with rich telemetry from modern buses and trains, are well suited to it. These applications rarely make headlines, but they bear directly on whether the promised service runs. Agencies that cancel a few percent of their trips because of staff or vehicle shortages lose riders' trust far faster than they can rebuild it with better arrival predictions. Operations data as a management tool Perhaps the least appreciated value of operations data is in management and accountability. With vehicle location and passenger counts on every trip, an agency can measure exactly where and when service fails riders: which routes run late, at which hours, at which stops, and how much excess waiting time riders experience compared with what the timetable promised. The excess wait time metric, used by Transport for London to manage bus contracts since the early 2000s, measures how much longer riders on frequent routes wait than they would if buses ran evenly spaced, and has been tied to bonus and penalty payments for the private operators who run London's buses. Learned models can help locate the causes of poor performance by attributing delays to specific segments, times and conditions, separating, for example, delay caused by traffic congestion from delay caused by long boarding times or by slow signal phases. That kind of diagnosis directs investment to where it will help: a bus lane on the segment where traffic is the problem, all-door boarding on routes where dwell times dominate, signal priority at the intersections where buses wait longest. This is where transit and traffic meet. Many of the most valuable interventions for bus reliability are changes to street operations, the signal timings discussed in Chapter 3, lane allocation, enforcement of bus lanes, and they depend on the transit agency and the traffic authority sharing data and objectives. In many cities these are separate organizations with separate budgets, and the institutional gap between them is a larger obstacle than any technical one. Summary of the operational picture The pattern across transit operations mirrors that in traffic control. Learning from data adds most value where it sharpens inputs to well-established decision processes: running times for scheduling, short-term forecasts for holding and dispatching, arrival predictions for riders, diagnostics for management. It adds least where it is asked to overcome fundamental constraints, such as the economics of pooling or the physics of a bus stuck in traffic. The agencies that have benefited most are those that combined better data with the unglamorous physical and policy changes, bus lanes, all-door boarding, signal priority, network redesign, that let better decisions take effect. The same lesson applies with even greater force to the third system that cities run, or increasingly share responsibility for: the electricity grid that powers street lights, traffic signals, trains, and a growing fleet of electric buses and cars. Chapter 6: The Urban Grid: Forecasting and Balancing Energy Distribution Electricity grids were built on a simple pattern. Large power stations generated electricity, high-voltage transmission lines carried it across long distances, and distribution networks stepped the voltage down and delivered it to homes and businesses. Power flowed in one direction, from the station to the socket. The distribution network at the end of that chain, the substations, feeders and transformers on residential streets, was designed with generous margins and then largely left alone. Utilities knew in detail what happened on the transmission system and comparatively little about what happened on the low-voltage streets, because little happened there that required attention. That pattern has broken. Rooftop solar panels push power back up the network on sunny afternoons. Heat pumps add large winter loads. Electric vehicles can double a household's peak demand when they charge in the evening. Batteries in homes and garages can absorb or release power on command. Cities are electrifying their bus fleets, which concentrates very large charging loads at depots. The distribution network, once passive, has become the place where the energy transition is either accommodated or blocked. Machine learning has a substantial role here, but the grid is a domain where the physics are well understood and the cost of errors is high, and that shapes what learning should and should not do. Who runs the urban grid It is worth being clear about institutional arrangements, because they vary widely and they determine who can act on a forecast. In some cities the municipality owns the electricity distribution utility outright: Los Angeles, through its Department of Water and Power; Seattle, through Seattle City Light; Austin, through Austin Energy; and many municipal utilities in Germany, known as Stadtwerke. In others, a private or state-owned distribution company serves the city under regulation, and the municipality is a customer, a planning authority and a political stakeholder but not the operator. Either way, the city has direct responsibility for significant loads, including street lighting, traffic signals, municipal buildings, water and wastewater pumping, and transit, and it controls decisions that shape grid demand, from building codes to where electric vehicle chargers can be installed. When this chapter speaks of optimizing energy distribution in city grids, it means the combined work of the distribution operator and the city: forecasting what the network will have to carry, managing flexible loads to fit within its limits, and planning investment where it is needed. Forecasting load and solar Load forecasting is one of the oldest applications of statistical learning in infrastructure. Utilities have forecast system demand for decades using regression on temperature, time of day, day of week and holidays, and since the 1990s neural networks have been used for the purpose. At the level of a whole system, day-ahead load forecasts are now very accurate on ordinary days, with typical errors of a few percent, because aggregate demand across millions of customers is smooth and strongly driven by weather and calendar. The harder problem, and the one that matters for city distribution networks, is forecasting at lower levels of aggregation: individual substations, feeders, and the transformers that serve a few dozen homes. As aggregation decreases, individual behaviour matters more and the load becomes spikier and less predictable. A transformer serving twenty houses may see its peak shift by an hour depending on when two or three households plug in their cars. Forecasting at this level requires more data, finer time resolution, and models that represent uncertainty well. Rooftop solar adds a second forecasting problem. A feeder with substantial solar has a "net load" that is the difference between consumption and local generation, and on a sunny midday it may become negative, exporting power upstream. Much rooftop solar is not separately metered in real time; the utility sees only net consumption at the substation, which hides both the true load and the true solar output. Machine-learning methods that estimate this hidden solar, sometimes called disaggregation, use weather data, solar irradiance estimates and the shape of the net load curve to separate the two. Accurate solar forecasts, built from numerical weather prediction, satellite cloud images and, for very short horizons, sky cameras, then feed into the net load forecast. The case of California illustrates the scale of the issue. The state grid operator's well-known "duck curve," first published around 2013, shows net load falling sharply in the middle of the day as solar generation rises, and then climbing steeply in the late afternoon as the sun sets and people return home. Each year the belly of the duck has deepened as more solar is installed. At the distribution level, the same pattern appears on individual feeders, sometimes more extremely, and the question becomes whether the local equipment can handle both the midday export and the evening ramp. What has proved effective in practice for distribution forecasting? The pattern is similar to that in transit. Gradient-boosted trees and neural networks trained with weather, calendar and recent-load features outperform simple regression, but the most important determinants of accuracy are data quality and relevant covariates. Probabilistic forecasts that give the range of possible loads are essential, because equipment is sized for peaks, not averages. And forecasts must be monitored for drift, since adoption of solar, heat pumps and vehicles changes the load shape steadily, so a model trained on last year's data may already be out of date. Seeing the low-voltage network A distinctive problem for city distribution is observability. Utilities historically had measurements at substations but little at the low-voltage level. Smart meters, now installed in tens of millions of homes in Europe and North America, change that, but they were designed for billing rather than grid operation. Their readings typically arrive with delay, in intervals of fifteen to sixty minutes, and the utility's records of which meter is connected to which phase of which transformer are often wrong. That last point is surprisingly consequential. To know whether a transformer is overloaded, or whether voltage on a street is out of limits, the utility needs to know which customers it serves. Records of this "connectivity" are frequently inaccurate after decades of changes. Machine-learning methods can infer the true connectivity from smart meter voltage data, because customers connected to the same transformer or phase show correlated voltage fluctuations. Studies and utility projects have used clustering and correlation analysis to correct connectivity models, which is unglamorous data work but a precondition for operating the low-voltage network actively. With connectivity fixed, state estimation becomes possible: estimating voltages and currents throughout the network from a limited set of measurements, combined with a physical model of the network. This is fundamentally a physics problem, solved by power-flow equations, and learning plays a supporting role: filling gaps in measurements, estimating loads at unmetered points, and speeding up calculations. The principle that physics should be used where it is known, and learning where it is not, runs through all the successful work in this area. Managing electric vehicle charging The largest new load on urban distribution networks is electric vehicle charging, and it is also the most flexible. A car parked overnight typically needs only a few hours of charging to replace a day's driving, and it usually does not matter to the owner which hours. If every car on a street begins charging at 6 p.m. when its owner arrives home, the local transformer may be overloaded. If the charging is spread across the night, the same transformer may cope easily. Managed charging, often called smart charging, coordinates when vehicles charge to fit within network limits and, ideally, to use cheaper or cleaner electricity. The approaches range from simple to sophisticated. Time-of-use tariffs, which make overnight electricity cheaper, shift a large share of charging without any algorithm. Chargers with a default delayed start achieve a similar effect. More advanced schemes have an aggregator or the utility send signals to chargers, adjusting charging rates in response to network conditions or prices, while guaranteeing that each car is charged by its owner's departure time. The UK has provided useful evidence on this. Since 2022, regulations have required new home and workplace chargers sold in Great Britain to have smart functionality, including a default setting that avoids charging during peak hours unless the owner overrides it. Large trials by distribution network operators, such as the Electric Nation project run by Western Power Distribution with partners between 2016 and 2019, found that most drivers accepted managed charging when they retained the ability to override it and their cars were reliably charged when needed. Machine learning contributes in several ways: forecasting when vehicles will arrive and depart and how much energy each will need, so the scheduler can plan; forecasting local network constraints; and learning drivers' patterns so that defaults fit their routines. The optimization of charging schedules itself is usually done with conventional methods, linear programming or simple priority rules, because the problem is well structured and the constraints, such as transformer limits, must be respected with certainty. Electric bus depots are a special case of great importance to cities. A depot charging a hundred buses overnight can need several megawatts, comparable to a small industrial site, often in a location the network was never designed to serve that much load. Cities that ordered electric buses without early engagement with the distribution operator have sometimes found that the grid connection takes years to deliver. Careful charge scheduling, which uses forecasts of each bus's energy needs based on its route, the weather and the season, can reduce the required peak capacity substantially, and that reduction translates directly into lower connection costs and faster delivery. This is one of the clearest examples in this booklet of forecasting producing tangible savings, and it depends on transit and energy planners working together. Batteries, buildings and flexibility Beyond vehicles, cities contain many other potentially flexible loads and resources: batteries in homes and at substations, heating and cooling systems in buildings that can preheat or precool, water pumps that can run when power is plentiful and storage tanks are low, and street lighting that can be dimmed. Together these are often called distributed energy resources, and the idea of coordinating many of them as a single resource is called a virtual power plant. Flexibility services are now a real market. In Great Britain, distribution network operators procure flexibility from aggregators who can reduce demand or increase generation at specific locations at specific times, as an alternative to reinforcing the network. The principle is straightforward: if a substation will exceed its capacity for only a few hours a year, paying local customers to reduce demand in those hours can be cheaper than replacing the equipment. Making this work requires forecasting at every stage: forecasting when and where constraints will occur, forecasting how much flexibility each resource can provide, and verifying afterwards that the flexibility was actually delivered, which means estimating what consumption would have been without the intervention. That last step, the baseline problem, is subtle and has led to disputes, because a poorly designed baseline can pay customers for reductions they would have made anyway. Buildings deserve a separate word because they are the largest energy consumers in most cities. Municipal buildings offer cities a direct opportunity. Learned models of a building's thermal behaviour, trained on data from its own management system, can schedule heating and cooling to reduce costs and peak demand while maintaining comfort. The best-known demonstration of learned control in a related setting is DeepMind's work with Google on data centre cooling, reported in 2016 as reducing the energy used for cooling by up to 40 percent. Data centres are unusually instrumented and homogeneous environments, and results there do not transfer directly to schools and offices with old controls and patchy sensors, but the principle that a building's own data can improve its control is sound. Street lighting and the city's own load One part of the urban electrical system is entirely the city's: public lighting. Street lights are among the largest electricity costs for many municipalities, and their conversion to LED over the past decade has produced some of the clearest energy savings in municipal infrastructure, typically halving consumption or better compared with the sodium lamps they replace, with lower maintenance as well. Los Angeles, which replaced well over a hundred thousand street lights with LEDs from 2009 onward, became an often cited early example. Once lights are LED and networked, they become controllable: they can be dimmed late at night on quiet streets, brightened where sensors detect pedestrians or cyclists, and monitored remotely so that failures are reported automatically rather than by residents. The decision of how far to dim is not purely technical. It involves road safety, residents' perception of personal security and, increasingly, concern for the effects of artificial light at night on wildlife and on people's sleep. Cities that have introduced dimming schedules have generally done so street by street, after consultation, rather than through an automated optimization. The networked lighting system is also, in many cities, a ready-made platform for other sensors: air quality monitors, traffic counters and environmental sensors mounted on the poles and powered from the lighting circuit. This is convenient, but it brings the privacy questions of the camera to every street corner and requires the same discipline over what is measured and kept. Where learning should and should not decide Electricity grids differ from traffic and transit in an important respect: their physics are precisely understood. The flow of power through a network obeys well-established equations, and equipment limits are known. This makes it possible to calculate, rather than learn, many things that a learned model would only approximate. The appropriate division of labour, reflected in most serious utility practice, uses learning for what is uncertain and physics-based optimization for what is known. Forecasts of load, solar output, vehicle behaviour and prices come from learned models. Network constraints and power flows come from physical models. The decision of what to do, how to schedule charging or dispatch a battery, comes from an optimization that combines the two, with hard constraints guaranteeing that equipment limits are respected regardless of what any forecast says. Reinforcement learning for direct grid control has been widely studied, but its use in real distribution networks remains limited, for the same reasons as in signal control: the difficulty of guaranteeing safety and explaining decisions, and the existence of strong conventional methods. This is also an area where security is not an afterthought. Grid systems are critical infrastructure and recognized targets; the attacks on Ukrainian distribution companies in December 2015, which cut power to roughly a quarter of a million customers, were a stark demonstration. Every connection that allows a forecast to drive an action, every charger that accepts remote commands, is a potential path for attack. The architecture of any learned control system on the grid must assume that inputs can be wrong or malicious and must fail safe. The planning payoff Perhaps the largest value of better data and forecasting on city grids lies not in real-time control but in investment planning. Distribution networks need enormous investment to support electrification, and the question of where to reinforce first, and how much, is answered better with good forecasts of where heat pumps, chargers and solar will be adopted, combined with accurate network models. Over-investment wastes billions; under-investment delays connections and holds back the energy transition. Utilities and cities that combine adoption forecasts with network models can target reinforcement, use flexibility to defer it where uncertain, and coordinate with city plans for housing, transport and building retrofit. That brings the discussion to planning more broadly: the use of models not to operate the city day to day but to decide how it should change. Chapter 7: Planning the City: Simulation, Digital Twins and the Long Horizon Everything discussed so far concerns operating the city as it is: timing the signals on existing streets, scheduling the buses on existing routes, balancing loads on the existing grid. Planning concerns the city as it might be. Should this corridor get a light rail line or bus rapid transit? What happens to traffic if a motorway through the centre is replaced by a boulevard? Where should new housing go so that people can reach jobs without a car? Which substations will need reinforcing by 2035? These questions have horizons of years or decades, they involve large capital commitments, and they ask about worlds that do not yet exist. The machine-learning methods that excel at short-term prediction are least reliable here, because the question is precisely what happens when the city changes in ways not present in its history. Yet data and modelling have a great deal to offer planning if they are used for the right purposes. This chapter examines what they are. The four-step model and its critics The dominant planning tool in transport for more than half a century has been the travel demand model, often called the four-step model after its stages. Trip generation estimates how many trips start and end in each zone of the city, based on population, households, employment and land use. Trip distribution decides where the trips from each zone go, typically using a gravity model in which zones attract trips in proportion to their size and in inverse relation to the travel cost between them. Mode choice splits the trips among car, transit, walking and cycling according to the time and cost of each, usually with a discrete choice model of the kind developed by Daniel McFadden, whose work on such models earned him a share of the Nobel prize in economics in 2000. Assignment loads the trips onto the network, so that travellers choose routes that minimize their own travel time given the congestion caused by everyone else. The four-step model was developed for the highway planning of the 1950s and 1960s and has been criticized for nearly as long. Its zones are coarse. Its trips are treated independently, ignoring the way people chain trips into tours. Its behavioural assumptions are simplified. And, most consequentially, it was often used in ways that ignored induced demand, the tendency of new road capacity to generate new traffic. Forecasts that treated future traffic as a fixed quantity to be accommodated justified road expansions that then filled up, as Duranton and Turner's evidence, cited in Chapter 1, suggests. Activity-based models, developed since the 1990s and now used by many large metropolitan planning organizations, simulate individual synthetic people going through their day: leaving home, dropping children at school, going to work, shopping, returning. Agent-based simulators such as MATSim, an open-source framework developed initially at ETH Zurich and TU Berlin, model millions of such agents interacting on a detailed network. These models are more realistic and can answer questions the four-step model cannot, such as how a congestion charge affects different household types. They are also data-hungry and expensive to build and calibrate. Where learning helps planning Machine learning contributes to planning models in several distinct ways, and it helps to separate them. Synthesizing inputs. Planning models need detailed descriptions of where people live, work and travel. Censuses provide some of this, but with long gaps between them. Mobile phone data, aggregated and anonymised, can describe how many people travel between areas at different times, and have been used to build and update origin-destination matrices in many countries. Satellite imagery and street-level images can be analysed to estimate building density, land use and even socioeconomic characteristics. Learned models can generate synthetic populations whose statistical properties match the real one without revealing any individual. Estimating behavioural relationships. The choice models at the heart of mode choice and destination choice can be estimated with richer methods. Machine-learning classifiers often predict choices more accurately than traditional logit models, but they are less interpretable, and planners need interpretable quantities, such as how much travellers value an hour of time saved, to compare options. A substantial research effort has gone into combining the two, for example by using neural networks inside a discrete choice framework so that key parameters remain interpretable. For practice, the important point is that prediction accuracy on historical choices is not the same as reliability when the options change. Speeding up simulation. Detailed simulations can take hours or days to run. Surrogate models, learned approximations of the simulator, can produce results in seconds, allowing planners to explore thousands of scenarios and identify the promising ones for full simulation. This is one of the most practical current uses of learning in planning. Monitoring outcomes. Planning has historically been poor at checking whether its forecasts came true. Continuous data from probe vehicles, passenger counters, phones and meters make it possible to compare what was forecast with what happened. This is less a modelling innovation than a discipline, but it is perhaps the most valuable one. The optimism problem The most important fact about long-term infrastructure forecasts is that they are frequently wrong in a consistent direction. Bent Flyvbjerg and colleagues, in a series of studies beginning in the early 2000s, compiled data on hundreds of transport projects worldwide and found that rail projects had, on average, substantially overestimated ridership, and that cost overruns were common across all project types. Their book Megaprojects and Risk, published in 2003 with Nils Bruzelius and Werner Rothengatter, set out the pattern and its causes, which include both genuine uncertainty and what the authors called strategic misrepresentation: forecasts shaded to win approval. No model fixes this, because it is a problem of incentives rather than technique. Flyvbjerg's proposed remedy, reference class forecasting, is a deliberately simple statistical approach: instead of building up a forecast from the details of the project, compare it with the distribution of outcomes for a reference class of similar completed projects, and adjust accordingly. If rail projects of this type have historically achieved a certain share of their forecast ridership, treat the new forecast with the same scepticism. The UK Department for Transport adopted a version of this approach for optimism bias adjustments in project appraisal. The lesson for machine learning is pointed. A sophisticated model built by a team with an interest in the project's approval offers no protection against optimism and may give it unwarranted authority. The safeguards are institutional: independent review, publication of forecasts and their assumptions, and systematic comparison of forecasts with outcomes after projects open. The digital twin No term in this field has been stretched further than digital twin. Originally an engineering concept for a detailed virtual model of a physical asset, such as a jet engine, kept continuously updated with data from sensors on the real one, it has been applied to whole cities to mean anything from a three-dimensional visualization to a full simulation of every system. The most cited example is Virtual Singapore, a project launched in 2014 under the National Research Foundation to build a detailed three-dimensional model of the city-state, with semantic information about buildings, terrain and infrastructure, for planning, research and simulation. Helsinki has published detailed 3D city models as open data, and many other cities have built models of their own. Such models are useful. They allow planners to visualize proposed developments in context, analyse sunlight and shadow, model flood risk, and communicate proposals to the public. The confusion arises when a 3D visualization is presented as if it were a predictive model of the city's behaviour. A model that shows every building accurately but has no reliable representation of how people will travel, where businesses will locate or how energy demand will respond to a policy is a map, not a twin. Conversely, a valuable traffic simulation may have no 3D representation at all. A more useful way to think about the digital twin is in terms of the questions it is built to answer. A city that wants to test signal timing changes before deploying them needs a calibrated traffic simulation of the relevant corridors, connected to live detector data. A city that wants to plan flood defences needs a hydrological model with accurate terrain and drainage. A utility that wants to plan reinforcement needs a network model with accurate connectivity and load forecasts. These can share data and a common geographic framework, but they are different tools, and the integration of all of them into a single all-seeing model is rarely necessary and almost never achieved. When evaluating a proposed digital twin, a city should ask: which decisions will it inform, how will it be kept current, how will its predictions be validated against what actually happens, and who will maintain it when the initial project funding ends? Many twin projects have produced impressive demonstrations and then decayed because no one owned the continuing work of keeping the data current. Accessibility as the planning objective If the purpose of urban transport is to let people reach the places they need to go, then the most important planning measure is accessibility: how many jobs, schools, shops, health services and other destinations a person can reach within a given time by each mode. Accessibility shifts attention from the movement of vehicles, which is a means, to the opportunities available to people, which is the end. Accessibility has been a concept in planning since at least the 1950s, but computing it in detail used to be impractical. With open data on transit schedules in GTFS form, street networks from sources such as OpenStreetMap, and fine-grained data on population and employment, it can now be computed for every block of a city in every hour of the day. Open-source tools such as OpenTripPlanner and R5, the latter developed by Conveyal, make this routine. The University of Minnesota's Accessibility Observatory has published detailed annual accessibility measures for US metropolitan areas. Accessibility analysis has three virtues. It is understandable to the public: "this change would let people in your neighbourhood reach 30,000 more jobs within 45 minutes by transit" is a meaningful statement. It is directly linked to equity, because accessibility can be compared across income groups and neighbourhoods. And it does not require forecasting behaviour, which avoids the largest source of error in long-term models. It measures what the transport system makes possible rather than predicting what people will do with it. Machine learning plays a modest role here, mostly in improving inputs such as job locations and travel time estimates. The larger point is about choosing the right objective. A planning process that optimizes vehicle flow will produce a different city from one that optimizes accessibility, and the choice between them is a choice of values, not of algorithms. Learning from tactical experiments There is a middle path between modelling a change and building it permanently: trying it cheaply and measuring what happens. Over the past two decades cities have increasingly used temporary, low-cost interventions, often called tactical urbanism, to test street changes before committing to them. New York City's conversion of parts of Broadway around Times Square to pedestrian space began in 2009 as a trial using paint, planters and movable furniture, with travel times, crash rates and pedestrian counts measured before and after; the results supported making the change permanent. Many cities have since used temporary bus lanes, trial cycle lanes and pop-up pedestrian spaces in the same way, and during 2020 a large number of such schemes were installed quickly as streets were reallocated for distancing and cycling. The rich, continuous data described in earlier chapters makes these experiments far more informative. With camera counts, probe travel times and signal performance logs in place before a trial begins, a city can measure its effects across the surrounding network rather than at a single point, and can see whether traffic diverted onto other streets or simply disappeared, as studies of road space reallocation have often found that some of it does. The evaluation is faster, cheaper and more credible than a forecast, because it measures behaviour in the real city rather than in a model. Trials have limits. Some changes, a new rail line or a major development, cannot be piloted. Short trials may capture initial reactions rather than long-run adjustment, since people take time to change routes, modes and even where they live. And a trial perceived as a fait accompli can generate opposition that has more to do with process than outcome. But where they are feasible, measured trials are often a better guide to decision than any simulation, and they are a natural partner to the operational data systems a modern city already has. Land use and the long feedback The deepest difficulty in long-range planning is that transport and land use shape each other. A new rail line changes which areas are attractive for housing and offices; over decades, development follows the line, which changes the demand the line serves. A new motorway encourages dispersed development that makes car travel necessary. These feedbacks operate over decades and are poorly represented in most models. Integrated land use and transport models, such as UrbanSim, an open-source system developed by Paul Waddell and colleagues, attempt to represent these feedbacks by simulating households and firms choosing locations, developers building, and the transport system responding. They are valuable for exploring scenarios but are difficult to validate, since the outcomes they predict take decades to unfold and cities are subject to economic shocks that no model anticipates. For this horizon, the appropriate posture is humility. Learned models can describe current patterns of location choice in great detail, but they cannot tell a city with confidence what it will look like in thirty years. Planning for that horizon is better approached through scenarios, which explore a range of possible futures, and through decisions that keep options open and perform acceptably across many of them, than through a single optimized forecast. Planning the grid alongside the streets Long-range planning has traditionally been divided by sector: transport planners model travel, utilities plan networks, and land-use planners zone sites, each with its own forecasts and time horizons. Electrification makes that division costly. A decision to electrify a bus fleet, to require chargers in new residential buildings, or to phase out gas heating in a district is simultaneously a transport, building and energy decision, and the distribution network may need years of lead time to accommodate it. Some cities and utilities have begun to plan these jointly, building shared scenarios for housing growth, vehicle and heat pump adoption, and depot locations, and testing them against network models to find where capacity will run out first. The modelling is not exotic: adoption forecasts informed by income, housing type and existing uptake, combined with load profiles and network capacity data. What is new is sharing the assumptions across institutions that historically did not talk to each other, so that a bus depot is not sited where the grid connection will take five years, and a substation upgrade is not designed for a neighbourhood whose redevelopment the city has already approved at twice the density. This is planning in the scenario mode the chapter has argued for: no one knows exactly how fast electrification will proceed, but several plausible paths can be explored, and investments chosen that hold up across them. What this means for practice The chapter's argument can be stated compactly. At planning horizons, the value of data and learning lies less in prediction than in description, exploration and accountability: describing the city as it is in far more detail than before, exploring many scenarios quickly with surrogate models, measuring accessibility rather than forecasting traffic, and comparing forecasts with outcomes after the fact. The risk is that sophisticated models lend false authority to forecasts that are no more reliable than the old ones and may be shaped by the same incentives. The safeguard is a planning culture that publishes its assumptions, tests its forecasts against reality, and treats long-term predictions as scenarios rather than facts. That culture is part of the broader governance of urban machine learning, which is the subject of the final chapter. Chapter 8: Governing the Algorithms: Procurement, Privacy, Equity and Accountability Every earlier chapter has ended at the same place: the decision layer is only as useful as the institutions around it. This chapter takes that layer directly. It asks how a city should buy machine-learning systems, what it must protect, how it can tell whether the benefits are fairly shared, who is accountable when things go wrong, and what the law now requires. These are not questions to settle after the technical work is done. They determine whether the technical work is worth doing. Buying intelligence Most cities do not build machine-learning systems. They buy them, usually as part of larger contracts for signals, cameras, fare systems, meters or software platforms. Procurement is therefore where many of the most consequential decisions are made, often by staff who are expert in purchasing equipment but not in evaluating learned models. Several procurement failures recur. Buying outcomes that cannot be verified. A contract promises a percentage reduction in delay or emissions, measured by the vendor's own system against a baseline the vendor defines. Without independent measurement and a baseline agreed in advance, such claims are unfalsifiable. Contracts should specify how outcomes will be measured, by whom, against what baseline, and over what period, including seasonal variation. Lock-in through data and formats. A system that stores detector data, timing plans or forecasts in proprietary formats, or that retains ownership of data generated on city streets, makes it costly to change suppliers and prevents the city from building its own records. Contracts should give the city ownership of, or at least full and perpetual rights to, data generated by its infrastructure, in documented open formats. The success of GTFS in transit is a reminder of what open standards can achieve: once schedules were in a common format, an entire ecosystem of tools became available to every agency. Opaque models. A city cannot evaluate, explain or defend a decision made by a system whose logic it cannot see. Full source code is not always necessary, but the city should know what the system optimizes, what data it uses, how it was trained and validated, and how it behaves when inputs fail, and it should have the right to independent testing. Pilots without a path. Many smart-city projects begin as pilots funded by grants or vendor goodwill, demonstrate something on a few intersections or one route, and then end, because nobody budgeted for scaling, maintenance or staff. A pilot worth running is designed from the start to answer a question that will inform a decision about scaling, with a plan and budget for what happens if the answer is yes. Hidden maintenance costs. Sensors fail, models drift, software needs updating, and the people who understand the system leave. The total cost of a learned system over its life is dominated by operations, not purchase. A procurement that compares only upfront prices will systematically favour systems that are cheap to install and expensive to keep working. The common remedy is in-house capability. A city does not need a large data science team, but it needs enough people who understand the technology to write sensible specifications, evaluate bids, test what is delivered, and hold suppliers to account. Cities that have built this capacity, often in a small central data or digital team that works across departments, consistently get more from their suppliers than those that rely entirely on vendor expertise. Privacy Urban infrastructure generates intensely personal data. Fare card taps reveal where someone lives, works, worships and spends the night. Video can identify faces and number plates. Smart meter readings at fine intervals reveal when a household wakes, sleeps and goes away. Connected vehicle and phone location traces are among the most identifying data that exist: a widely cited 2013 study by Yves-Alexandre de Montjoye and colleagues, published in Scientific Reports, found that four spatio-temporal points were enough to uniquely identify 95 percent of individuals in a large mobile phone dataset. That finding is why simply removing names from location data does not make it anonymous, and why several earlier releases of supposedly anonymised mobility data were later shown to allow re-identification. In 2014, for example, a dataset of New York City taxi trips released under a freedom of information request was found to have poorly protected medallion and licence numbers that could be recovered, exposing individual drivers' trips and, combined with photographs, some passengers'. Good practice in urban data rests on a few principles that are well understood, even if not always followed. Collect only what the purpose requires. A system counting vehicles at an intersection does not need to identify them. Edge processing that extracts counts and discards images, as described in Chapter 2, is a practical form of data minimization. Aggregate early. Transit planning needs origin-destination flows between zones, not the travel history of individual card holders. Wherever possible, data should be aggregated before it is stored or shared. Limit retention. Raw records should be kept only as long as needed, with aggregates retained for the long-term record. Control access and sharing. Clear rules on who can access what, logged and audited, and explicit policy on sharing with other agencies, particularly the police, are essential to public trust. In several cities, the discovery that traffic or transit data were being shared with law enforcement or immigration authorities without public knowledge has damaged support for data programmes far beyond the specific case. Use privacy-enhancing techniques where they fit. Differential privacy, which adds carefully calibrated noise to published statistics so that no individual's presence can be inferred, has been adopted by the US Census Bureau for the 2020 census and is increasingly used in mobility data releases. It involves real trade-offs in accuracy, especially for small areas, and is not a substitute for minimization, but it is a valuable tool. In Europe, the General Data Protection Regulation applies to much of this data and requires a lawful basis for processing, purpose limitation and, for high-risk processing, a data protection impact assessment. Elsewhere, legal requirements vary, but the principles above are sound regardless of what the law demands. Equity Algorithmic systems in cities distribute benefits and burdens, and they can do so unfairly in ways that are hard to see. Several mechanisms deserve attention. Biased sensing. Systems that rely on data generated by particular devices see the people who use those devices. Probe data from navigation apps reflect drivers who use those apps. Systems that collect complaints and service requests, such as municipal 311 lines, reflect who complains, and research on such systems has repeatedly found that reporting rates differ across neighbourhoods for reasons unrelated to actual need. Boston's Street Bump app, which used smartphone accelerometers to detect potholes, became an early cautionary example: because smartphone ownership was concentrated in wealthier areas, the city recognized that relying on it would direct repairs toward those areas, and supplemented it with other data. Biased objectives. A signal optimizer that minimizes vehicle delay favours drivers over pedestrians and bus riders. A transit forecast that predicts ridership from past ridership will tend to allocate service to routes that already have good service, reinforcing existing patterns. An electricity investment plan that follows forecast adoption of electric vehicles and heat pumps will tend to upgrade networks first in the neighbourhoods wealthy enough to adopt them early. None of these is malicious; each follows from an objective chosen without considering distribution. Uneven performance. A computer vision system that detects people less reliably in some conditions, or a demand model that is less accurate for some neighbourhoods, will serve those people less well. Accuracy should be evaluated across groups and places, not only in aggregate. The practical remedy is to measure distribution explicitly. For any significant system, a city can ask: who benefits, by how much, and who bears costs? Accessibility analysis, described in Chapter 7, provides a natural framework for transport: it allows a city to compare how a change affects access to jobs and services across income groups and neighbourhoods. The US Federal Transit Administration's Title VI requirements, which oblige transit agencies receiving federal funds to analyse whether major service and fare changes have disparate impacts on minority and low-income populations, are an example of such analysis built into the decision process. Similar analysis can be applied to signal timing, to the placement of chargers, and to the targeting of grid investment. Accountability and transparency When an algorithmic system makes or shapes a decision, someone must be answerable for it. That sounds obvious, but the diffusion of responsibility across vendors, departments and software makes it easy for nobody to be. A resident complaining that signal timing on their street has changed should be able to find out what changed, why, and who decided. Several cities have experimented with formal transparency mechanisms. In September 2020, Amsterdam and Helsinki launched public algorithm registers, websites describing the algorithmic systems the cities use, what data they draw on, how they work and who is responsible for them. New York City established an Automated Decision Systems Task Force under a 2018 local law, which produced a report in 2019 that was widely criticized, including by some of its own members, for its limited access to information about the city's systems, a reminder that transparency mechanisms need real powers to be effective. Several countries' national governments have since introduced their own registers or standards for algorithmic transparency in the public sector. Transparency is more than publishing a list. For an operational system, accountability requires documentation of what the system optimizes and how; logs of what it did and when; monitoring of its performance, including for different groups; a process for people to challenge decisions; and the ability for a human to override the system and understand the consequences. These requirements are easier to meet for systems designed with them in mind than for systems where they are added afterwards. The regulatory frame: the EU AI Act The European Union's Artificial Intelligence Act, which entered into force in August 2024, is the most comprehensive legal framework for AI so far, and it bears directly on urban infrastructure. The Act classifies AI systems by risk. Among those it lists as high-risk in its Annex III are AI systems intended to be used as safety components in the management and operation of critical digital infrastructure, road traffic, and the supply of water, gas, heating and electricity. High-risk systems face substantial obligations: a risk management system, data governance requirements for training data, technical documentation, record-keeping and logging, transparency to deployers, human oversight, and appropriate levels of accuracy, robustness and cybersecurity. Public bodies deploying high-risk systems are also required to carry out an assessment of the impact on fundamental rights before putting them into use. The timetable has moved. Under the Act as originally adopted, most high-risk obligations were to apply from August 2026. In 2026 the EU adopted a "Digital Omnibus on AI" amending the timetable; it was published in the Official Journal in July 2026 and postponed the obligations for stand-alone high-risk systems listed in Annex III to 2 December 2027, and for high-risk AI embedded in products covered by EU product legislation to 2 August 2028. Cities and their suppliers in the EU therefore have additional time, but the direction is settled. A learned signal controller or grid control system that acts as a safety component will need to meet these requirements, and procurement started today should already anticipate them. The precise boundaries of the high-risk category, such as whether a system that recommends signal timings to an engineer, rather than setting them directly, counts as a safety component, will be clarified through guidance and practice. For a city, the prudent course is to treat the Act's requirements as a sound description of good practice for any consequential system, regardless of its formal classification. Risk management, data governance, documentation, logging, human oversight and robustness are what the earlier chapters have argued for on engineering grounds. Outside the EU, regulation is more fragmented. In the United States there is no comparable federal law, though federal guidance on public-sector AI has been issued and changed across administrations, and some states and cities have their own rules. Other jurisdictions have their own frameworks. Cities with suppliers and residents across jurisdictions will find the EU requirements an influential reference point regardless. Evaluation as a duty A theme running through every chapter is that claims about urban machine learning are frequently unverified, and that cities often cannot tell whether a system they have bought is working. Evaluation is therefore not an academic extra but a governance obligation. A city that spends public money on a system that changes how its streets, buses or grid behave owes the public an honest account of what it achieved. Good evaluation in this setting has a recognizable shape. It defines in advance what will be measured and what would count as success. It establishes a baseline from data collected before the change, ideally over a long enough period to capture seasonal variation. Where possible, it uses comparison sites, similar intersections, routes or feeders that did not receive the intervention, so that city-wide trends such as a fuel price rise or a new school term are not mistaken for effects of the system. It measures effects on everyone affected, not only on the intended beneficiaries. It distinguishes measured outcomes from modelled ones. And it is published, including when results are disappointing. A useful technique, where operations allow it, is to switch a system on and off in alternating periods, so that its effect can be compared directly under similar conditions. Some evaluations of adaptive signal systems have used this design. It costs a little performance during the off periods and yields far stronger evidence than a single before-and-after comparison. Independent evaluation, by a university, a national transport research body or another city's engineers, adds credibility, especially for vendor systems. Sharing evaluation results among cities, through professional associations and open publication, is one of the most effective ways for a city to avoid repeating another's mistakes, and one of the most underused. Security Connecting infrastructure to data systems makes it attackable. Traffic signal controllers have repeatedly been shown by security researchers to have weak protections. A widely reported 2014 study by researchers at the University of Michigan demonstrated that they could take control of a network of roadside signals using unencrypted wireless links, with the cooperation of a local road agency. Grid attacks, as Chapter 6 noted, have caused real outages. Learned systems add new vulnerabilities, such as manipulated inputs designed to fool a vision model, but the basic exposure comes from connectivity itself. The response is standard security engineering applied rigorously: segmenting operational networks from corporate and public ones, authenticating every command, encrypting communications, keeping software patched, monitoring for anomalies, and designing systems to fail safe, so that a signal controller cut off from the centre reverts to a safe local plan and a charger that loses contact reverts to a conservative default. Independent safety units in signal cabinets, which prevent conflicting greens regardless of what the controller commands, are an example of a safeguard that a learned system can never override, and that kind of hard boundary should be the norm. Public trust as infrastructure The thread that runs through this chapter is trust. Urban infrastructure works only with public consent: people must accept cameras at intersections, share their travel data through fare cards, allow their cars to be charged on a managed schedule, and accept that their signals are timed by systems they do not see. That consent is easily lost. The collapse of Sidewalk Labs' Quayside project in Toronto, described in Chapter 1, came after a long public argument in which data governance, not engineering, was the central issue. Trust is built by the practices described above: collecting only what is needed, explaining what is done, measuring who benefits, answering for mistakes, and keeping promises about data. These are not obstacles to deploying machine learning in cities. They are the conditions under which it can be deployed at all, and they are as much a part of the infrastructure as the cameras and the cables. Conclusion The cabinet on the street corner, where this booklet began, now looks different. It may hold a controller that can accept new timings remotely, a small processor turning camera images into counts, and a link to a central system that compares this intersection with its neighbours. A few streets away a bus is being told to wait thirty seconds at a stop to keep it evenly spaced from the one ahead. Under the pavement, a transformer is serving a street where half the houses have solar panels and a third have electric cars, and a scheduler is staggering their charging so that it does not overheat. In a planning office, someone is comparing what last decade's rail forecast promised with what the passenger counters actually recorded. Each of these is machine learning at work in a city, and each succeeds or fails for reasons that have little to do with the sophistication of the model. That is the booklet's central argument, and it is worth drawing out what follows from it. What the evidence supports Across traffic, transit and energy, the evidence points to a consistent hierarchy of value. Learning delivers most reliably as a sensing and estimation technology. Computer vision that turns existing cameras into multimodal counters, inference that fills in unmeasured destinations or hidden solar generation, models that correct grid connectivity records: these extend what a city can see, and seeing is where most infrastructure management starts. The benefits are real and comparatively easy to verify, because the outputs can be checked against ground truth. It delivers substantially as a short-term forecasting technology, where rich history and stable patterns make prediction tractable. Transit demand, running times, arrival predictions, feeder loads and vehicle charging needs are all forecast better with modern methods than without them, and those forecasts improve scheduling, dispatching and planning. The improvements over strong simple baselines are usually more modest than headline claims suggest, concentrated on unusual days, and contingent on clean data and monitoring for drift. It delivers least reliably, so far, as a direct control technology. Learned signal controllers and grid controllers produce impressive results in simulation and a thinner record on real infrastructure. The obstacles, the gap between simulator and street, imperfect sensing, hard safety constraints, the need to explain decisions, and the strength of conventional methods, are not temporary. They will narrow, but they mean that for most cities, the immediate gains lie elsewhere. And it delivers least of all as a substitute for economics, physics and politics. No algorithm made demand-responsive minibuses as efficient as full buses. No forecast removed the optimism from megaproject appraisal. No optimizer chose, on its own, between drivers and pedestrians. Those remain human choices, and models that hide them do harm. What a city should do differently From that hierarchy, a set of practical priorities follows for anyone commissioning or overseeing these systems. Start with a named operational problem, not a technology. The successful projects described in this booklet, from retiming neglected signals to scheduling electric bus depots, began with a specific failure and asked what data would help fix it. The failed ones began with a platform and looked for problems. Fix the loop's weakest link first. If the controllers cannot accept new timings, the detectors are broken, or no one has time to act on forecasts, a better model will not help. Often the most valuable investment is in maintenance, staff and the ability to actuate decisions. Make the objective explicit and public. Decide, as policy, how the city weighs the competing interests its systems serve: person delay rather than vehicle delay, accessibility rather than flow, reliability for riders who cannot work from home, grid investment that does not leave poorer neighbourhoods behind. Write it down. Then hold the systems to it. Measure properly. Evaluate against a well-maintained baseline, not a neglected one. Measure what happens to everyone affected, not only the people the system was designed to serve. Separate measured effects from modelled ones. Keep measuring after the pilot ends. Own the data and the knowledge. Contracts should give the city rights to data generated on its infrastructure in open formats, and the city should keep enough in-house expertise to specify, evaluate and challenge what it buys. Build privacy and security into the architecture. Process at the edge, aggregate early, keep little, share carefully, and design every connected system to fail safe. Treat governance as engineering. The documentation, logging, human oversight, impact assessment and transparency that the EU AI Act will require of high-risk systems from December 2027 are good practice everywhere. They are the means by which a city knows what its systems are doing and can answer for them. What remains unsettled Several questions remain genuinely open, and a reader should watch them. The first is whether learned control will close the gap with practice. The ingredients are improving: richer sensing, better simulation calibrated against real data, methods for learning safely within constraints. Within a decade, learned controllers operating inside strict safety envelopes and evaluated against strong baselines may become routine in some settings. The test will be independent evaluation on real networks, not results in benchmark simulators. The second is the role of large private data providers. Probe data from navigation and mapping companies, and the analytics built on it, have given cities capabilities they could not build themselves, at low cost. They have also created dependencies on firms whose priorities, pricing and terms the city does not control. How cities secure durable, fair access to such data, and how they maintain their own measurement capacity alongside it, is unresolved. The third is how the energy transition will reshape the urban grid. Electrified transport and heating will make the distribution network one of a city's most important and most constrained pieces of infrastructure. Whether flexibility, managed charging and better forecasting can defer enough reinforcement to keep costs manageable, and whether they can do so fairly, will be decided over the coming decade. The fourth is how the public will judge all of this. Trust in algorithmic systems is fragile and uneven. A single scandal over data sharing or a visible failure can set back years of careful work. Cities that build transparent, accountable systems now will be better placed to use more capable tools later. A final word The smart city, as it was sold in the 2010s, promised a city that would run itself: sensors everywhere, a central platform, algorithms making decisions in real time. That vision misunderstood both the technology and the city. Cities are not machines to be optimized against a single objective but places where people with different needs share limited space, time and energy, and where decisions about who gets what must be answerable to them. What machine learning can do, and is doing, is more modest and more valuable. It lets cities see their own infrastructure far more clearly, anticipate demand more accurately, and adjust their systems more often and more precisely than before. Used inside well-engineered control loops, with explicit objectives, honest measurement and real accountability, it makes streets flow better, buses run more reliably and grids carry more clean energy. Used as a substitute for those things, it produces dashboards. The difference lies not in the algorithm but in the city that deploys it. Further Reading Jarrett Walker, Human Transit: How Clearer Thinking about Public Transit Can Enrich Our Communities and Our Lives (Island Press, 2011; revised edition 2024).** The clearest account of the geometry and trade-offs of transit network design, including the ridership-versus-coverage choice and why frequency matters. Essential background for anyone applying data to transit planning. Bent Flyvbjerg, Nils Bruzelius and Werner Rothengatter, Megaprojects and Risk: An Anatomy of Ambition (Cambridge University Press, 2003).** The foundational study of cost overruns and demand shortfalls in large infrastructure projects, and of why forecasts err in a consistent direction. Gilles Duranton and Matthew A. Turner, "The Fundamental Law of Road Congestion," American Economic Review 101, no. 6 (2011): 2616–2652.** The empirical case that new road capacity in US cities generates proportionate new traffic, with direct implications for any claim that better control will cure congestion. Shannon Mattern, A City Is Not a Computer: Other Urban Intelligences (Princeton University Press, 2021).** A thoughtful critique of the idea of the city as a data-processing system, and a reminder of the forms of knowledge that sensors do not capture. Ben Green, The Smart Enough City: Putting Technology in Its Place to Reclaim Our Urban Future (MIT Press, 2019).** A former city data practitioner's argument for using technology in service of specific municipal goals rather than as an end in itself, with case studies of both success and failure. Anthony M. Townsend, Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia (W. W. Norton, 2013).** An early and still useful history of the smart-city movement, its corporate promoters and its grassroots alternatives. Roger P. Roess, Elena S. Prassas and William R. McShane, Traffic Engineering (Pearson, 5th edition, 2019).** A standard textbook covering detection, signal timing and traffic flow theory, for readers who want the engineering foundations behind Chapters 2 and 3. Hua Wei, Guanjie Zheng, Vikash Gayah and Zhenhui Li, "Recent Advances in Reinforcement Learning for Traffic Signal Control: A Survey of Models and Evaluation," ACM SIGKDD Explorations Newsletter 22, no. 2 (2021).** A survey of learned signal control by researchers active in the field, useful for understanding the methods and the evaluation problems discussed in Chapter 3. Yves-Alexandre de Montjoye, César A. Hidalgo, Michel Verleysen and Vincent D. Blondel, "Unique in the Crowd: The Privacy Bounds of Human Mobility," Scientific Reports 3 (2013): 1376.** The short, influential paper showing how few location points are needed to identify individuals, and why mobility data cannot be anonymised simply by removing names. Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice (OTexts, 3rd edition, 2021; freely available online).** A clear practical guide to forecasting methods, evaluation and uncertainty, directly applicable to the transit and energy forecasting problems in Chapters 4 and 6. Hashtags: #AIForSmartCities #SmartCities #ArtificialIntelligence #UrbanPlanning #TrafficOptimization #SmartUrbanPlanning #IntelligentTransportationSystems #TrafficManagement #SmartTrafficSignals #AdaptiveTrafficControl #TrafficFlowOptimization #UrbanMobility #SmartMobility #TransportationPlanning #PublicTransportation #TransitDemandForecasting #TrafficPrediction #MachineLearning #DeepLearning #ComputerVision #ReinforcementLearning #UrbanDigitalTwins #SmartInfrastructure #InternetOfThings #IoT #SmartEnergyGrids #SustainableCities #UrbanDataAnalytics #IntelligentInfrastructure #ResponsibleAI

Download the Book (PDF): Introduction At noon on a clear day in July, a square metre of farmland in southern Europe receives close to a kilowatt of sunlight. A lettuce leaf in that square metre cannot use most of it. Its photosynthetic machinery saturated hours ago; the surplus energy arrives as heat, drives water out of the leaf and the soil, and forces the plant to spend effort protecting itself rather than growing. Three hours later, as a thunderstorm builds and clouds thicken, the same leaf would welcome every photon it can get. The sun does not adjust its delivery to the plant's appetite. For most of agricultural history, nothing could. A solar tracker can. A tracker is a row of photovoltaic panels mounted on a rotating tube, driven by a small motor and governed by a controller that decides, every few minutes, which way the panels should face. In an ordinary solar farm, the controller has one job: point the panels at the sun to catch as much energy as possible, and dodge the shadow of the next row when the sun is low. Put the same machine above a field of crops, give it a different set of instructions, and it becomes something agriculture has never had before: a movable roof that can decide, minute by minute, how much sunlight reaches the ground and how much is turned into electricity. That is the core of this book. Agrivoltaics, the practice of producing solar electricity and crops on the same land, is usually presented as a compromise. Farmland is scarce, solar farms need space, and agrivoltaics promises to let both happen at once in exchange for some loss on each side. That framing is correct as far as it goes, and much of the public argument about agrivoltaics is conducted on its terms: how much yield is lost, how much electricity is forgone, how high the panels must be, whether the farmer is a real farmer or a landlord with a solar lease. The argument here is different. The controlling idea of this book is that the value of an agrivoltaic system is decided less by its hardware than by how it allocates sunlight over time. Sunlight arrives in a pattern that rarely matches what a crop can use. On many days, in many climates, the field receives more light at midday than the crop can turn into growth, and more heat and evaporative demand than its water supply can comfortably meet. At other times, the same crop is starved of light. A fixed solar structure takes the same share of the sun regardless of whether the crop needs it. A tracker, governed by the right algorithm, can take the surplus and return the scarce part. Where that matching works, the system produces more food and more energy per hectare than either use alone. Where it fails, agrivoltaics really is a compromise, and sometimes a poor one. Why the timing of light matters The physics behind this idea is simple enough to state in a paragraph, and it shapes everything that follows. A solar panel and the patch of ground beneath it compete for exactly the same direct beam of sunlight. Whatever the panel intercepts, the ground does not receive. At any single instant, the trade is close to one for one: tilt the panel to catch more of the beam, and its shadow grows by the same proportion. If agrivoltaics were only about the quantity of light, there would be little to gain from cleverness. A panel that took a third of the light would take a third of the light, whichever way it moved. The gain comes from the fact that a unit of light is not worth the same to the crop at every moment. A photon arriving at a saturated leaf at 1 p.m. on a hot afternoon is worth little or nothing to the plant; the same photon is worth exactly as much to the panel as any other. A photon arriving at 8 a.m., or under thin cloud, may be worth a great deal to the plant. A tracker that shifts its interception toward the hours when the crop cannot use light, and away from the hours when it can, changes the exchange rate. It is buying light from the crop when the crop's price is low and leaving it alone when the price is high. Water reinforces the point. The rate at which soil and leaves lose water depends heavily on the radiation they absorb, and that demand peaks in the same midday hours when light is most likely to be in surplus. Shade placed over a crop at noon cuts evaporation when evaporation is fiercest. In dry climates, the water saved can be worth more than the light lost. What this book covers and what it leaves out The book follows that idea from first principles to field practice. The first chapter explains how crops use light and why their appetite for it varies so much, by species, by stage of growth and by hour of the day. The second turns to geometry: how panels cast shade, how height, spacing and orientation decide where that shade falls, and why the different architectures in use today, from fixed high-mounted roofs to vertical bifacial fences, behave so differently. The third describes how solar trackers work in conventional solar farms, because the control logic of agrivoltaic trackers is built on top of that machinery and inherits its constraints. The fourth chapter is the centre of the book: the algorithms that decide how an agrivoltaic tracker moves, from simple rules keyed to radiation and temperature through to crop-model-driven control and model predictive control that optimises over forecasts. The fifth and sixth chapters follow the consequences into the soil and the air: what shade does to evaporation, transpiration and the distribution of rain, and how trackers can shelter crops from heat, frost and hail, sometimes in direct conflict with protecting the panels themselves. The seventh chapter asks how success is measured, through the land equivalent ratio and the yield thresholds that regulators in France, Germany, Japan and Italy now write into law. The eighth confronts the economics and the failures: the extra cost of elevated structures, the contracts that decide who controls the tracker, and the lessons of Japan, where a substantial share of approved projects have struggled to keep farming going under the panels. The book deliberately does not attempt to survey every form of solar-agriculture overlap. Grazing sheep among ground-mounted panels, planting pollinator habitat beneath conventional arrays, and building solar greenhouses are all real and useful practices, and each appears where it illuminates the main argument. But the focus throughout is on cropland where a crop is grown for harvest and the solar structure is designed, and ideally controlled, around it. That is where the question of how to allocate sunlight is sharpest, and where good control makes the largest difference. A field that has grown up quickly The idea is older than most people assume. In 1981, Adolf Goetzberger, founder of the Fraunhofer Institute for Solar Energy Systems in Freiburg, and his colleague Armin Zastrow proposed raising solar collectors above fields so that potatoes could grow beneath them. Their paper, published the following year, set out the basic geometry still used today: lift the panels high enough, space them widely enough, and enough light reaches the ground for a crop to grow. The proposal was decades ahead of the economics. Photovoltaic modules were then so expensive that nobody would build an elevated structure to spread them out. The economics changed after 2010, as module prices fell by roughly an order of magnitude and solar farms began spreading across farmland in Europe, Asia and North America. Researchers at INRAE in Montpellier, led by Christian Dupraz, coined the term agrivoltaic and in 2011 published a modelling study suggesting that combining panels and crops could raise total land productivity by between about a third and nearly three quarters. In Japan, an engineer named Akira Nagashima had already begun building narrow-panel "solar sharing" systems above fields, and from 2013 the agriculture ministry allowed farmland to carry solar structures on condition that the crop beneath remain productive. Fraunhofer ISE built a 194-kilowatt research installation above a farm at Heggelbach, near Lake Constance, and in the hot, dry summer of 2018 recorded a land use efficiency of 186 per cent. In Arizona, Greg Barron-Gafford's group at the University of Arizona showed in 2019 that peppers and tomatoes grown under panels in the desert used water more efficiently and, in some cases, yielded far more. By the mid-2020s, agrivoltaics had moved from research plots into national energy policy. France passed a law in 2023 and a decree in 2024 defining what counts as agrivoltaic and capping the acceptable yield loss. Germany published a technical standard, DIN SPEC 91434, in 2021 and wrote agrivoltaics into its renewable energy law. Italy used European recovery funds to run a dedicated agrivoltaic tender, awarding about 1.5 gigawatts to 540 projects at the end of 2024. Japan, after a decade of growth to more than six thousand approved sites, tightened its rules because too many of those sites had let the farming slide. Through all of this, one technical thread has become steadily more important. The early systems were fixed: a static roof of panels on tall posts. The newest generation increasingly moves. French companies such as Sun'Agri built their business on panels that rotate according to crop needs, guided by plant models developed with INRAE. Mainstream tracker manufacturers now offer agrivoltaic versions of their products, and academic control engineers have begun writing optimisation algorithms for them. Whether a tracker over a field serves the crop, the power purchase agreement or some negotiated balance of the two is now a question of software. That shift is the reason for this book. The hardware of agrivoltaics is maturing and becoming standardised. The control policy is where the real choices are made, where the most value is won or lost, and where farmers, developers and regulators have the least shared understanding. The chapters that follow are written to give a serious, non-specialist reader that understanding: enough of the plant science, the geometry, the engineering and the policy to judge an agrivoltaic project on the question that matters most, which is what it does with the sun. Chapter 1: How Crops Use Light Every agrivoltaic decision begins with a question about the plant: how much light can it use, and when? Engineers who come to the field from solar energy tend to treat light as a single quantity, measured in watts per square metre and summed into kilowatt-hours. Plants do not experience light that way. They respond to particular wavelengths, to the rate at which light arrives, to the total delivered over a day, to its direction and to the heat and dryness that come with it. A tracker algorithm that ignores these distinctions is steering blind. Photosynthesis has a ceiling Plants use light in the band from roughly 400 to 700 nanometres, called photosynthetically active radiation, or PAR. Plant scientists count it not in watts but in photons, because each photosynthetic reaction is driven by the absorption of individual photons regardless of their energy. The usual unit is micromoles of photons per square metre per second. Full summer sun at midday delivers around 2,000 of these units. A heavily overcast day may deliver a tenth of that. The response of a leaf to increasing light follows a curve that every agronomy student learns. In darkness, the leaf respires and loses carbon. As light rises, photosynthesis climbs steeply and almost linearly, because every extra photon can be put to work. Then the curve bends. The leaf's capacity to fix carbon dioxide, limited by the enzyme Rubisco, by the rate at which carbon dioxide diffuses in through the stomata and by the regeneration of the molecules that carry energy within the chloroplast, begins to fall behind the supply of photons. Eventually the curve flattens altogether. This is light saturation. Beyond it, more light produces no more sugar. For a single, well-watered leaf of a typical temperate crop, saturation often arrives well below full sunlight, sometimes at a third to a half of it. This is the observation that makes agrivoltaics attractive in the first place. If a leaf is saturated at half of full sun, then shading it from full sun to half sun costs it nothing. The trouble is that crops are not single leaves. A canopy is a layered structure, and the leaves at the top shade those below. When the top leaves are saturated, the lower leaves may still be receiving far less light than they could use, and any extra light that filters down through gaps adds to the total. As a result, the light response of a whole canopy saturates much more gradually than that of an individual leaf, and for dense, productive crops like maize or wheat it may barely saturate at all on a clear day. This is why early enthusiasm about "free" shading proved too optimistic for many field crops. The leaf-level argument is real, but the canopy-level argument is weaker, and yield is decided at the canopy level. Two further complications matter. First, the efficiency with which plants use light depends on whether it is direct or diffuse. Direct sunlight arrives in a single beam and strikes upper leaves at high intensity, saturating them, while leaves lower down sit in deep shadow. Diffuse light, scattered by clouds, haze or structures, arrives from all directions and penetrates deeper into the canopy, illuminating more leaves at moderate intensity. Studies of forests and crops alike have found that canopies use diffuse light more efficiently per photon than direct light. A solar structure that converts some direct light into diffuse light, by scattering it off frames and panel edges, or by casting moving shadows that break up the beam, can partly offset the loss of total light. Second, the photosynthetic pathway matters. Most temperate crops, including wheat, potatoes, rice, soybeans, vegetables and fruit trees, use the C3 pathway, which saturates relatively early and suffers from photorespiration, a wasteful side reaction that increases with temperature. Maize, sugarcane, sorghum and millet use the C4 pathway, which concentrates carbon dioxide around Rubisco, suppresses photorespiration and keeps responding to light up to much higher intensities. C4 crops are, in general, poor candidates for heavy shading. C3 crops in hot climates are often good ones, because shading lowers leaf temperature and photorespiration at the same time as it trims surplus light. Daily light integral and the shape of a day Photosynthesis depends on the instantaneous rate of light, but growth depends on the total. Greenhouse growers have long used a measure called the daily light integral, or DLI: the number of moles of PAR photons received per square metre over a whole day. A clear midsummer day at mid-latitudes delivers somewhere around 50 to 60 moles. A dull winter day may deliver fewer than 5. Greenhouse practice gives a useful sense of how much light different crops need to grow well, because greenhouse growers pay for every photon they add with electricity. Leafy greens such as lettuce are commonly grown to targets in the mid-teens of moles per square metre per day; above a certain point, extra light gives diminishing returns and can contribute to disorders such as tipburn. Fruiting crops such as tomatoes and peppers are typically given considerably more, often in the range of twenty to thirty moles or higher, because they must fuel both leaves and fruit. These are greenhouse figures and should not be transferred directly to an open field, where wind, temperature swings and water stress change the picture. But they make one thing plain: on a clear summer day, an open field in southern Europe or the American Southwest may deliver two or three times the light a leafy crop needs. The shape of the day matters as much as the total. Light rises through the morning, peaks around solar noon and falls through the afternoon. Temperature lags behind light, peaking in mid-afternoon. Vapour pressure deficit, the drying power of the air, follows temperature. A crop's capacity to use light is therefore highest in the morning, when it is cool and well hydrated, and lowest in the early afternoon, when it may be hot, short of water and closing its stomata to conserve moisture. When stomata close, carbon dioxide cannot get in, and the leaf saturates at a much lower light level than it would in the morning. Many crops show a midday depression in photosynthesis on hot days for exactly this reason. The consequence for agrivoltaics is that the surplus of light is not spread evenly. It is concentrated in the hours around and after noon on clear, hot days. Those are also the hours when a solar panel produces the most power. A fixed structure cannot exploit this alignment beyond the average, because it takes roughly the same share of light at every hour. A moving structure can concentrate its shade in the hours when the crop's appetite is lowest. Shade tolerance, stage by stage Crops differ widely in how they respond to partial shade, and the differences are not always intuitive. They fall roughly into four broad groups, and the patterns, drawn from field trials across Europe, Asia and North America, are summarised in Table 1. Table 1. Broad crop responses to moderate shade (around 20–40 per cent light reduction). Crop group Examples Typical response Main reason Shade-tolerant leafy and forage crops Lettuce, spinach, chard, celery, clover, some herbs Little loss, sometimes gain; larger, softer leaves Early leaf saturation; benefit from lower heat and water stress Berries and some fruit Raspberries, currants, blueberries, grapes in hot regions Often neutral or positive; quality may improve Protection from heat, sunburn, hail; slower ripening Tubers and temperate grains Potatoes, winter wheat, barley Moderate loss in normal years; neutral or gain in hot, dry years Canopy-level light limitation offset by water savings Light-demanding and C4 crops Maize, sorghum, sunflower, many legumes for grain Clear losses under heavy shade High canopy saturation point; C4 pathway The table hides a great deal of variation, and three qualifications matter. The first is climate. The same crop can lose yield under shade in a cool, cloudy year and gain in a hot, dry one. Fraunhofer's Heggelbach trial showed this directly: in 2017, a relatively normal season, yields of the arable crops under the panels were somewhat lower than in the reference field, while in the drought summer of 2018, winter wheat under the panels yielded about 3 per cent more than the reference and celery about 12 per cent more. Clover, by contrast, yielded about 8 per cent less. The panels had not changed; the weather had changed which constraint was binding. The second is the target of production. Shade often changes quality rather than just quantity. Under panels, lettuce tends to produce larger, thinner leaves, which may suit some markets and not others. Grapes ripen more slowly, which in hot regions can preserve acidity and moderate alcohol, attributes winemakers in the south of France increasingly prize as summers warm. Field trials at Piolenc in the Rhône valley, run with Sun'Agri's dynamic panels and monitored by the local chamber of agriculture, the French wine institute IFV and INRAE, reported berry weights about 17 per cent higher under the panels and higher acidity and anthocyanin content, with no change in alcohol strength. For a grower selling wine rather than tonnes of grapes, that is a different calculation altogether. The third is the stage of growth. Crops are not equally sensitive to shade throughout the season. For grain crops, the critical windows are typically around flowering and grain fill, when the plant is setting and filling the organs that become yield; shade then can be costly. For leafy crops, the early establishment phase, when young plants need light to build leaf area, is often the most sensitive, while later growth tolerates more shade. Fruit trees need light at flowering and fruit set, and again for colour development before harvest, but may benefit from shade during the hottest weeks of fruit sizing. This is the most important point for tracker control. A fixed structure imposes the same shade from sowing to harvest. A tracker can open up during the sensitive windows and close down during the tolerant ones. Heat, water and the value of a photon So far, light has been treated mainly as food. It is also a stress. Sunlight absorbed by a leaf that it cannot use for photosynthesis must be disposed of, as heat, as re-emitted light or through protective chemical pathways. On a hot, bright afternoon, leaf temperature can rise several degrees above air temperature if the plant is short of water and cannot cool itself by transpiring. Above crop-specific thresholds, often in the low to mid-thirties Celsius for many temperate crops, enzymes begin to malfunction, pollen fertility falls and fruit can scorch. Excess light also drives water loss. The energy that evaporates water from soil and leaves comes overwhelmingly from absorbed radiation. On a clear summer day, a crop canopy can transpire several millimetres of water, and much of that demand is concentrated in the midday hours. A plant that cannot supply enough water closes its stomata, and photosynthesis falls even though light is abundant. In other words, in dry conditions, light beyond a certain point actively harms the crop by pushing it into water stress. This changes the way an agrivoltaic designer should think about the value of each photon reaching the ground. It is helpful to imagine that every moment of the day has an implicit price for light, from the crop's point of view. In the early morning, with the crop cool and hydrated and photosynthesis below saturation, the price is high: each photon removed is growth lost. Around solar noon on a mild day, with the upper canopy saturated but lower leaves still hungry, the price is moderate. In mid-afternoon on a hot day, with stomata partly closed and leaves heating, the price may be close to zero, and can even be negative: removing light helps the crop by reducing heat and water stress. The panel, meanwhile, places a nearly constant value on every photon it can intercept. A unit of electricity produced at 2 p.m. is worth roughly as much as one produced at 10 a.m., unless the electricity market says otherwise, which, as later chapters discuss, it increasingly does. The opportunity for agrivoltaics lies in the gap between these two price curves. Where the crop's price for light is low and the panel's is high, the panel should take the light. Where the crop's is high, the panel should yield it. Expressed this way, the problem of agrivoltaic control becomes clear. A well-designed tracker algorithm is, in effect, a continuous auction for sunlight between two bidders whose valuations change through the day, through the season and with the weather. The rest of this book describes how the hardware makes that auction possible, how the algorithms run it, and what happens when they get it right or wrong. What the crop tells the controller For a tracker to allocate light intelligently, it needs some way to know the crop's state. The simplest approach is to rely on fixed knowledge of the crop's general behaviour: lettuce tolerates shade, grapes benefit from afternoon shade in July, and so on. More sophisticated approaches measure the crop directly. Several kinds of signal are used in research and commercial systems. Radiation sensors above and below the panels measure how much light the crop is actually receiving and how much the panels are taking. Air temperature and humidity sensors in the canopy allow the controller to calculate vapour pressure deficit, a direct indicator of the drying power of the air. Soil moisture probes at several depths track the water available to roots. Leaf temperature, measured by infrared thermometers, reveals whether the crop is keeping itself cool by transpiring or overheating because it is short of water. In tree and vine crops, dendrometers measure tiny daily changes in trunk or stem diameter that indicate water status, and sap flow sensors measure the rate of transpiration directly. None of these signals is a direct readout of the crop's price for light. They have to be interpreted, and the interpretation depends on knowledge of the crop's physiology. A rise in leaf temperature at noon in a well-watered lettuce field means something different from the same rise in a vineyard during drought. This is why the most capable agrivoltaic control systems combine sensors with crop models, computer simulations of plant growth and water use that turn measurements into estimates of how much each hour of light is worth. The fourth chapter returns to these models in detail. For now, the essential point is that an agrivoltaic tracker is only as good as its understanding of the plant beneath it. The engineering is the easier half of the problem. Chapter 2: The Geometry of Shade A solar panel above a field does two things at once. It catches sunlight, and it throws a shadow. The two are bound together by geometry. Before it is possible to understand what a tracker can do, it is necessary to understand where shadows fall, how they move and what the designer can control through the shape of the structure itself. Much of what makes an agrivoltaic system succeed or fail is fixed the day the posts go into the ground. The panel and its shadow Consider a single panel, a flat rectangle about a metre wide, held above the ground. Direct sunlight arrives as a beam of nearly parallel rays. The panel intercepts the part of the beam that strikes it, and the ground beneath receives a shadow of the same cross-section, displaced and stretched according to the sun's angle. The key quantity is the angle between the sun's rays and a line perpendicular to the panel's surface, called the angle of incidence. When the panel faces the sun squarely, the angle of incidence is zero and the panel intercepts the full width of the beam. As the panel turns away, it presents a narrower face to the sun. The width of beam it intercepts falls in proportion to the cosine of the angle of incidence. At 30 degrees, the panel catches about 87 per cent of what it would catch face-on. At 60 degrees, it catches half. At 90 degrees, when the panel is edge-on to the sun, it catches almost nothing of the direct beam. The shadow obeys exactly the same rule, because the shadow is simply the part of the beam the panel has caught. A panel facing the sun casts the largest possible shadow. A panel edge-on to the sun casts almost none. This is the single most important geometric fact in agrivoltaics. At any instant, and for direct light, the electricity a panel collects and the shade it casts rise and fall together. There is no position that gives more power and less shade at the same moment. The cosine relationship has one more consequence that matters enormously for control. Near the optimum, the cosine curve is flat. Turning a panel 10 degrees off the sun costs only about 1.5 per cent of its direct-beam capture; 20 degrees costs about 6 per cent. Small deviations are cheap. Larger ones become expensive quickly: 45 degrees costs about 29 per cent, 60 degrees costs half, and the final 30 degrees to edge-on gives up the remaining half entirely. A tracker that wants to give the crop a little more light at modest cost has room to manoeuvre. One that wants to give the crop most of the light must surrender most of the power. Diffuse light complicates the picture in a useful way. Diffuse radiation comes from the whole sky, not from a single direction, so a panel never blocks all of it, and the shade it casts in diffuse light is soft and broad rather than sharp. On overcast days, when most light is diffuse, a panel tilted to face the brightest part of the sky, usually straight up, collects the most energy, while the ground beneath receives a share of skylight from all the directions the panel does not block. A higher structure lets more diffuse light reach the ground under each panel, because the panel subtends a smaller part of the sky as seen from the crop. Height, spacing and the ground coverage ratio Three design parameters dominate the light environment of an agrivoltaic field: the height of the panels above the crop, the spacing between rows and the orientation of those rows. The spacing is usually expressed as the ground coverage ratio: the width of the panel rows divided by the distance from the centre of one row to the centre of the next. A conventional utility solar farm on single-axis trackers commonly uses a ground coverage ratio somewhere between about 0.3 and 0.5, meaning the panels, laid flat, would cover roughly a third to a half of the ground. Agrivoltaic systems over crops typically use lower ratios, often around 0.2 to 0.4, to let more light through. Over the course of a whole day, the fraction of light intercepted by the panels is roughly related to the ground coverage ratio, adjusted for how the panels are angled and how much of the light is diffuse. A useful first approximation is that a system with a ground coverage ratio of one third, tracking the sun, will remove somewhat more than a third of the direct light from the ground over a day, because a tracking panel presents its full face to the beam for much of the time. That approximation hides a pattern through the day that turns out to matter a great deal. Consider rows of single-axis trackers with a ground coverage ratio of one third: panels 2 metres wide on rows 6 metres apart. At solar noon in summer, a tracker following the sun holds its panels nearly flat. A flat panel 2 metres wide, over a 6-metre strip of ground, blocks one third of the direct beam falling on that strip, whatever the height of the sun. Two thirds reaches the crop. In mid-morning, with the sun lower in the east, the panels tilt to face it. Each panel now presents its full 2-metre width to the beam, but the beam is arriving at a slant, so the strip of ground each row's shadow covers is wider than 2 metres. As the sun sinks lower, the shadows lengthen and reach toward the next row. At the sun height where the shadow of one row just touches the foot of the next, which with a ratio of one third happens when the sun is roughly 20 degrees above the eastern horizon in the plane of rotation, the panels together intercept essentially all of the direct beam. No direct sunlight reaches the ground at all. Below that sun height, the tracker begins backtracking to avoid shading its own panels, but the ground beneath remains in nearly continuous shadow until the sun climbs higher. The result is counterintuitive. A tracker run for energy takes about a third of the direct light at noon but nearly all of it in the early morning and late afternoon. Over a summer day, it removes noticeably more than a third of the direct light from the field, and it removes it disproportionately in the cool hours when, as Chapter 1 explained, the crop can use light most efficiently. A horizontal fixed array with the same ratio would take a steady third all day. In terms of timing, conventional tracking is close to the worst possible allocation for the crop. This is not a criticism of trackers. It is the reason they offer so much room for improvement. The same morning and evening hours in which a sun-following tracker casts the most shade are the hours in which it can most easily be turned edge-on to the sun, opening the field to light. A tracker that does nothing clever is worse for the crop than a fixed structure. A tracker with a good control strategy can be much better. Height does not change the total amount of shade over a large field, which is set by the area of panels and their angles. It changes how the shade is distributed. A low panel casts a sharp, dense shadow directly beneath it and leaves the gap between rows in full sun. Crops in the shadow strip are heavily shaded; crops in the gap are not shaded at all. The result is a field of stripes, with uneven growth and ripening that complicate harvest. A high panel casts its shadow further away, and as the sun moves through the day the shadow sweeps across a wider band of ground. Each plant then experiences a period of shade and a period of sun, and the daily total of light becomes much more uniform across the field. The diffuse component also becomes more uniform, because each point on the ground sees a similar slice of open sky. This is why the classic overhead agrivoltaic structures are tall. Fraunhofer's Heggelbach installation placed its panels on a structure about 5 metres high, with enough clearance for combine harvesters and tractors to work beneath. Germany's DIN SPEC 91434 standard treats systems with a clear height of at least 2.1 metres as a distinct category, recognising that this height allows people and most machinery to pass beneath. Japan's newer benchmarks propose panel heights of around 3 metres. Height, however, is expensive. Every additional metre adds steel, larger foundations and greater wind loads, and the cost of the support structure is the main reason elevated agrivoltaic systems cost more than ground-mounted solar. Orientation interacts with height. In the northern hemisphere, rows aligned north to south, with panels rotating from east to west through the day, cast shadows that sweep across the field from west to east and back, giving each plant a similar daily pattern of light. Rows aligned east to west, with panels tilted toward the south, cast a shadow that stays on roughly the same strip of ground for the whole day, moving only slowly north and south with the seasons. For fixed systems, east-west rows produce persistent stripes. For trackers, north-south rows are standard, because single-axis trackers rotate east to west to follow the sun's daily path. Five architectures The geometry of shade has produced a small number of distinct architectures, each with its own pattern of light, its own costs and its own suitability for different farms. They are compared in Table 2. Table 2. Main agrivoltaic architectures and how they shape light. Architecture Shade pattern Control of light Machinery access Typical uses Fixed overhead (tall structure) Moving stripes, fairly uniform daily total None after construction Good if clearance high Arable and vegetable farming, orchards Overhead single-axis tracker Adjustable, sweeping shadow Continuous, rotation around one axis Good if clearance high; posts constrain lanes Vineyards, orchards, vegetables, arable Interspace single-axis tracker (low) Strong stripes beside rows Continuous but limited by low height Between rows only Forage, grazing, some arable Vertical bifacial fences Narrow stripes morning and evening None Wide lanes between fences Grassland, arable, livestock Solar greenhouse or tunnel Shade on roof area, often permanent Little or none Inside structure Horticulture, berries Fixed overhead systems are the most straightforward. Panels are mounted at a fixed tilt on a tall frame, often in rows that are spaced or broken into strips to let light through. They are robust and comparatively simple, and with careful choice of height and spacing they can deliver uniform shade. Their weakness is that the shade is the same on a cloudy day as on a scorching one, in April as in July. Some projects use semi-transparent or spaced-cell modules to soften the shade, at the cost of producing less power per square metre of panel. Overhead trackers mount rotating panel rows on a tall frame. They combine the uniformity of high structures with the ability to change the share of light taken at any moment. Sun'Agri in France built many of its vineyard and orchard systems this way, with panels several metres above the canopy rotating on a horizontal axis. They are the main subject of this book. Interspace systems place panels between rows of crops rather than above them. A conventional single-axis tracker, raised somewhat higher than usual and spaced more widely, allows crops or grass to grow in the lanes between rows. Germany's standard classifies such systems as Category II, permitting a larger share of land to be lost to the structure than for overhead systems. The tracker's control over light is real but limited, because the low height means each panel's shadow falls mostly on a narrow strip next to it. Vertical bifacial fences stand panels on edge, usually facing east and west, so that they collect morning and evening light on their two faces. The German company Next2Sun has pioneered this design. The shade they cast is narrow and falls mostly in the morning and evening, leaving the field in full sun at midday. They suit crops and grassland that need full light and farms that need wide lanes for large machinery. They also produce power at times when conventional south-facing or tracking arrays are less productive, which, in grids saturated with midday solar, has become an increasing financial advantage. They offer no control over light, and they provide little midday shade, so they deliver few of the water and heat benefits that are central to agrivoltaics in hot climates. Solar greenhouses and tunnels integrate panels into the cladding of protected cropping structures. They are a large share of agrivoltaic capacity in China and elsewhere, and in some cases have been criticised for prioritising power at the expense of crops grown inside. They operate under a different set of physical constraints, with heat trapped by the enclosure, and are largely outside the scope of this book. Uniformity, rain and the structure itself Three further aspects of geometry are easy to overlook and can decide the success of a system. The first is uniformity. Farmers manage fields as units. They sow, fertilise, spray and harvest whole blocks at once. If the light environment under the panels varies too much from one strip to the next, the crop will mature unevenly and the farmer will be forced to compromise on timing for the whole field. Research systems usually report the average light reduction, but what the farmer experiences is the variation. High structures, north-south rows and moving panels all improve uniformity. So does a control strategy that varies the panels' position through the day, so that no strip of ground sits in shadow for hours at a time. The second is rain. A solid panel intercepts rain as well as light. The water runs off its lower edge and falls in a concentrated line, while the ground beneath the panel's centre stays relatively dry. INRAE researchers working at a dynamic agrivoltaic site near Montpellier measured this effect in detail and showed that the resulting pattern of wet and dry strips can be pronounced, with consequences for crop growth, soil erosion and the uniformity of irrigation. Fixed systems can mitigate the effect with gutters or by leaving gaps between modules. Trackers have a further option: by rotating the panels during rain, the controller can move the drip line across the field or tilt the panels to near vertical so that rain largely falls past them. Chapter 5 returns to this. The third is the structure itself. Posts, foundations, cable trenches and inverter stations occupy ground that cannot be cropped. The posts also constrain how machinery moves. A system designed for a vineyard, with posts aligned to the vine rows, may be well suited to its farm; the same system on an arable field, where a combine needs wide, uninterrupted passes, may waste land. Germany's DIN SPEC 91434 caps the land lost to structures at 10 per cent for overhead systems and 15 per cent for interspace systems, and requires that the agricultural yield remain at least 66 per cent of a reference. These rules exist precisely because poorly designed structures can quietly turn a farm into a solar plant with grass under it. Geometry sets the bounds The geometry of a system sets the limits within which any control strategy must work. A low, dense array cannot give a crop much light no matter how cleverly its trackers move. A tall, sparse one cannot give it much shade. The ground coverage ratio caps the share of light the panels can ever take; the height and orientation decide how evenly that share is spread; the post spacing decides what the farmer can do beneath. Within those bounds, a tracker gives the operator a lever that a fixed system lacks: the ability to choose, moment by moment, between the panel's position of maximum power and positions that let more light through. The cosine relationship says that this lever is cheap to pull a little way and expensive to pull far. The crop science of Chapter 1 says when it is worth pulling. Before turning to how an algorithm decides, it is worth understanding the machine it controls, which was built for an entirely different purpose. Chapter 3: How Solar Trackers Work The single-axis solar tracker is one of the quiet success stories of the energy transition. In the early 2010s it was a niche product, used where land was cheap and sunshine abundant. By the mid-2020s it had become the default choice for large ground-mounted solar plants in much of the world, and in the United States the large majority of new utility-scale capacity was being built on trackers. Companies such as Nextracker, Array Technologies, GameChange Solar, Arctech and Soltec ship tens of gigawatts of them every year. Agrivoltaic trackers are, for the most part, adaptations of these machines. Their capabilities and their limits are inherited from equipment designed to do one thing: make as much electricity as possible from a given set of panels at the lowest cost. The basic machine A single-axis tracker consists of a long horizontal steel tube, called the torque tube, running north to south and supported on a line of posts driven or screwed into the ground. Panels are clamped across the tube, usually one or two panels deep, so that a single row may carry dozens or a hundred or more modules over a length of tens of metres. A motor, typically at the centre of the row, turns the tube through a gearbox or slewing drive, rotating all the panels together. Most designs can rotate through about 50 to 60 degrees either side of horizontal. At dawn, the panels face east, tilted steeply toward the rising sun. Through the morning they rotate toward horizontal, reaching it near solar noon, and through the afternoon they tilt progressively toward the west. At night they return to a resting position, often flat or tilted east in readiness for the morning. The tracker follows only the sun's daily east-to-west movement; it does not tilt north or south to follow the sun's seasonal change in height. Dual-axis trackers, which follow the sun in both directions, capture more energy per panel but cost more, need more space and are rarely used at utility scale. Each row is controlled by a small electronic unit. In many modern designs, the controller is powered by its own small solar panel and battery, so that rows need no separate power cabling. An inclinometer on the tube tells the controller the actual angle of the panels. The row controllers communicate, usually by a wireless mesh network, with a central network controller for the site, which in turn connects to the plant's supervisory control and data acquisition system, the software through which operators monitor and command the whole installation. The central controller distributes the target angles, weather alerts and stow commands; the row controllers drive their motors to match. The economics of the machine are simple. A tracker costs more than a fixed mounting structure, and it needs maintenance. In return, by keeping the panels closer to the sun's direction for more of the day, it raises the energy yield of each panel. The gain varies with latitude and climate. It is largest in sunny, low-latitude locations where the sun moves across a wide arc of sky and the morning and evening hours are bright. In such places, single-axis tracking commonly adds somewhere between a sixth and a quarter more annual energy than a well-designed fixed-tilt system. The extra output also arrives in the morning and evening, which has become more valuable as grids with lots of solar power see prices collapse at midday. Astronomical tracking and backtracking The core algorithm of a tracker is astronomical. Given the date, the time and the site's latitude and longitude, the position of the sun can be computed to a tiny fraction of a degree. The standard reference, the Solar Position Algorithm published by the US National Renewable Energy Laboratory in 2004, achieves an uncertainty of a fraction of a thousandth of a degree over several thousand years. Trackers do not need that precision, but the point is that the sun's position is perfectly predictable. A tracker does not need to sense the sun; it calculates where the sun is and turns to face it. For a single-axis tracker, facing the sun means rotating the panels until the plane containing the torque tube and the line perpendicular to the panels also contains the sun. This angle, called the ideal or true tracking angle, minimises the angle of incidence of direct sunlight given that the panel can only rotate around one axis. True tracking runs into trouble early and late in the day. When the sun is low in the east, a panel tilted steeply to face it casts a long shadow westward, and that shadow falls on the next row. Shading even part of a crystalline silicon module can cut its output out of proportion to the shaded area, because the cells are wired in series and the weakest cell limits the current of its string, with bypass diodes giving only partial relief. The solution, nearly universal on modern trackers, is backtracking. When the sun is low enough that rows would shade each other, the controller deliberately turns the panels back toward horizontal, just far enough that each row's shadow ends exactly at the foot of the next. As the sun rises, the panels progressively tilt further toward it until, at a certain sun height, true tracking resumes. The geometry depends on the ground coverage ratio: the more tightly packed the rows, the longer the backtracking period each morning and evening. Backtracking is the first and oldest example of a tracker deliberately not pointing at the sun. It accepts a modest loss of beam capture on every panel to avoid a much larger loss from shading. It also illustrates a principle that becomes central in agrivoltaics: the optimal angle for a single panel in isolation is not the optimal angle for a whole system. On uneven ground, where neighbouring rows sit at different heights, simple backtracking still leaves some rows shading others, or leaves some rows backtracking more than necessary. Manufacturers have developed terrain-aware backtracking, in which the controller uses a survey of each row's height and slope to compute individual angles. This is an early form of system-level optimisation, and it relies on the same kind of model of the site's geometry that agrivoltaic controllers require. Diffuse light and smart tracking Astronomical tracking assumes that the important light comes from the sun's direction. On clear days that is true; on cloudy days it is not. Under a thick overcast, nearly all the light is diffuse, and the brightest part of the sky is usually overhead. A panel tilted steeply toward an invisible sun near the horizon faces a dim patch of sky and the ground, and collects less than it would lying flat. Several manufacturers now offer software that recognises these conditions and adjusts. Nextracker's TrueCapture and Array Technologies' SmarTrack, among others, use weather data, irradiance sensors and machine learning trained on site performance to decide when to flatten the panels in diffuse conditions and how to optimise backtracking for each row. Their claimed gains are usually a few per cent of annual energy, modest in percentage terms but valuable at the scale of a large plant. These products are important to agrivoltaics for two reasons. First, they establish that tracker angles can be governed by models and data rather than by the sun's position alone, and they provide the communication and control infrastructure, from sensors to the network controller to individual rows, through which any other objective can be pursued. Second, they show how the industry measures the value of a control decision: in megawatt-hours gained or lost relative to a baseline. An agrivoltaic algorithm will be judged against the same baseline, and every hour it holds the panels away from the energy-optimal angle will show up in the plant's performance reports as a loss. Stow: protecting the machine Trackers spend most of their time following the sun or backtracking, but some of the most consequential decisions they make concern when to stop. Large arrays of panels on rotating tubes are vulnerable to weather. High winds can set up oscillations in long rows that, if not damped, can damage or destroy the structure. Hail can crack the glass on modules. Heavy snow can load panels beyond their design limits. Trackers respond by moving to protective positions, called stow. In high winds, most designs stow at a specific angle chosen to minimise aerodynamic loads and avoid dangerous oscillation; the angle varies by manufacturer and structural design. In forecasts of hail, many trackers now move to a steep angle, close to their maximum tilt, so that hailstones strike the glass at a glancing angle and deliver less energy. Insurers and module makers have pushed hard for hail stow after a series of costly hail losses at solar plants in Texas and elsewhere in the early 2020s. In snow, panels may tilt steeply to shed the load. Stow commands are triggered by site anemometers, by weather forecasts and, increasingly, by dedicated hail-warning services. Stow logic is non-negotiable from the plant owner's point of view. Lenders and insurers require it; the structural warranty depends on it. Any agrivoltaic algorithm must sit beneath stow in the hierarchy of commands. That has implications for the crop. Hail stow, for example, tilts the panels nearly vertical, which removes their protection from the crop beneath at exactly the moment hail falls. A system designed to protect a crop from hail would want the panels flat, presenting their full area as a roof. The tension between protecting the panels and protecting the crop is explored in Chapter 6. It is mentioned here because it shows that the agrivoltaic controller does not start with a free hand. It inherits a set of priorities set by the energy industry. From energy machine to light valve The single-axis tracker, then, arrives at the farm gate as a highly developed machine with a clear purpose, a mature control architecture and a set of embedded priorities. It can rotate its panels through most of a half-circle, within limits on speed and range. It can be addressed row by row. It already incorporates logic for deviating from the sun's direction for good reason, through backtracking, diffuse tracking and stow. It reports its every move to a performance monitoring system that counts the energy lost to each deviation. What changes in agrivoltaics is the objective. Instead of maximising energy alone, the controller must balance energy against the needs of a crop whose value to the farmer, and often to the regulator, is at least as important. Physically, the machine becomes a light valve: a device that can be opened to let sunlight through to the ground or closed to capture it for electricity. Opening the valve means rotating the panels toward edge-on to the sun, so that they intercept as little of the beam as possible. Closing it means turning them to face the sun, as in ordinary tracking, or holding them flat at midday. Several practical constraints shape how this valve can be used. Rotation range limits how fully it can open: if the panels can rotate only 55 degrees from horizontal, then at noon, with the sun nearly overhead in summer, they cannot be turned edge-on and will always cast substantial shade. In the morning and evening, when the sun is low, they can be turned nearly edge-on easily. Motor speed limits how quickly the valve can respond to a passing cloud. Each additional rotation adds wear to motors and bearings, so a control strategy that moves the panels constantly may shorten equipment life. And the communication architecture, designed for sending a handful of commands a day, may need to be upgraded if each row is to be steered according to the conditions in the crop beneath it. The noon problem The limit on rotation deserves a closer look, because it shapes what any agrivoltaic algorithm can achieve. Take a site at about 44 degrees north, roughly the latitude of Montpellier or Bologna. At the summer solstice, the sun at solar noon stands about 69 degrees above the southern horizon. A north-south tracker rotates around a horizontal axis pointing at that sun's azimuth, so at noon the sun lies in the plane that contains the axis, and the panels face it best when flat. To turn them edge-on to the noon sun, the tracker would have to rotate them a full 90 degrees, until they stood vertical. No standard tracker can do this. At its limit of about 55 to 60 degrees, a panel still presents roughly half its area to the noon beam. Early in the morning, the situation is reversed. At 7 a.m. solar time in midsummer at the same latitude, the sun is low in the east. The angle the panels must rotate to become edge-on to it is much smaller, comfortably within the tracker's range. Opening the valve fully is easy. This asymmetry has a practical meaning. A tracker can return almost all of the morning and evening direct light to the crop, but only part of the midday light. Since Chapter 1 showed that morning light is usually the most valuable to the crop and midday light the least, the asymmetry happens to work in agriculture's favour. But it means an agrivoltaic tracker can never be a perfect valve. Around the middle of a summer day, whatever the algorithm decides, a substantial share of the field will be in shade. Designers who want more control over midday light must use lower ground coverage ratios, trackers with wider rotation ranges, or panel designs that let some light through even when facing the sun, such as glass-glass modules with gaps between cells. The asymmetry also shows why agrivoltaic and conventional tracker design diverge. A conventional plant has no reason to rotate beyond the point of maximum energy, and manufacturers set rotation limits accordingly, trading range against cost and structural loads. An agrivoltaic tracker benefits from range beyond what energy alone would justify, because range is what gives the crop access to light. A few manufacturers now offer extended-range designs for this reason. The difference is small in engineering terms and large in agronomic consequence. None of these constraints is prohibitive. Manufacturers already supply trackers with a range of rotation limits and drive configurations, and several have introduced agrivoltaic versions with taller posts, wider row spacing and software hooks for external control. What remains is the question that the rest of this book addresses: given a machine that can open and close a valve on sunlight, how should it decide when to do so? Chapter 4: Algorithms That Share the Sun An agrivoltaic tracker controller faces a decision every few minutes: what angle should each row of panels hold? Ordinary solar trackers answer that question with geometry and a little weather data. An agrivoltaic tracker must answer it with geometry, weather and biology together, and must do so with incomplete knowledge of the crop's state and uncertain knowledge of what the sky will do in the next hour. This chapter describes how that decision is made, from the simplest rules through to optimisation over weather forecasts, and works through an example to show what is really being traded. What is being traded It helps to begin with a precise statement of the trade. Chapter 2 established that, for direct sunlight at any given moment, the light a panel intercepts and the shade it casts are the same quantity. Every unit of direct light returned to the crop by rotating the panels away from the sun is a unit of direct light the panels do not convert. The exchange rate, in physical terms, is roughly one for one. That might suggest that control strategy makes no difference: give the crop a certain amount of light over the day, and the panels lose the same amount whichever hours are chosen. The total physical loss is indeed similar. What differs is the value of the light on each side. The crop does not value a unit of light equally at every hour, as Chapter 1 explained. Neither, increasingly, does the electricity market. The control problem is therefore not about how much light to share, which the geometry of the system largely fixes, but about when. A worked example makes this concrete. Consider a field of lettuce under single-axis trackers in southern France on a clear midsummer day. The numbers are illustrative, chosen to be realistic in magnitude rather than drawn from any single site. Suppose that, without panels, the field would receive about 55 moles of photosynthetically active photons per square metre over the day. Divide the day into three blocks: the morning, from sunrise to 10 a.m., delivering about 13 moles; the middle of the day, from 10 a.m. to 3 p.m., delivering about 30; and the late afternoon and evening, delivering about 12. Suppose that the system's geometry is such that, when the panels follow the sun normally, they intercept about 45 per cent of the light reaching the field in each block. In reality, as Chapter 2 showed, a sun-following tracker takes a larger share in the morning and evening than at midday; assuming an even share keeps the arithmetic simple and, if anything, understates the case for morning opening. The crop then receives about 7 moles in the morning, 16.5 at midday and 6.6 in the afternoon, roughly 30 moles in total. Now suppose the agronomist judges that, at this stage of growth, the crop would do better with about 35 moles. The controller must return about 5 moles to the crop. It has two obvious ways to do so. It could open the valve in the morning. With the sun low in the east, the panels can be rotated nearly edge-on, cutting their interception from 45 per cent to perhaps 10 per cent. The crop then receives about 11.7 moles in the morning instead of 7.2, a gain of about 4.5 moles. The panels lose the same 4.5 units of intercepted light. Or it could open the valve at midday, trimming the panels' interception in the 10 a.m. to 3 p.m. block from 45 per cent to about 30 per cent. The crop again gains about 4.5 moles, and the panels again lose about 4.5 units. Physically, the two options are nearly identical. Biologically, they are not. In the morning, the lettuce is cool, its stomata are open and its leaves are well below saturation. Almost every extra photon becomes growth. At midday, the upper leaves are already saturated, the air is hot and dry, and some of the extra light will simply heat the leaves and drive water loss. If, for the sake of illustration, the crop turns morning light into growth with full efficiency and midday light with 60 per cent efficiency, the morning option delivers roughly two-thirds more useful light than the midday option for the same energy cost. Economically, the picture can flip. In many European and Californian grids by the mid-2020s, the abundance of solar power had driven midday wholesale prices very low, sometimes below zero, while morning and evening prices remained higher. Under those conditions, the energy lost in the morning is worth more than the energy lost at midday. The same choice that is best for the crop is the most expensive for the plant owner. The algorithm has to weigh not just physical quantities but two sets of prices, one biological and one financial, and the weighting is ultimately a commercial and regulatory question, not a technical one. Rules, thresholds and calendars The first agrivoltaic tracker strategies were, and many still are, rule-based. They encode agronomic knowledge as a set of conditions under which the panels should adopt one position or another. The simplest rules are calendar-based: track normally except between, say, 11 a.m. and 4 p.m. during July and August, when the panels are held in a shading position; or open fully during the four weeks around flowering. Such rules require no sensors and are easy to explain to a farmer. Their weakness is obvious: they shade on cool, cloudy days in August as readily as on scorching ones, and they cannot respond to an early heat wave in June. Threshold rules improve on calendars by responding to measured conditions. A typical rule might say: if incoming radiation exceeds a set level and air temperature exceeds a set level, or if vapour pressure deficit exceeds a threshold, move the panels to track the sun and shade the crop; otherwise, rotate them toward edge-on to let light through. Further rules can be layered on: if soil moisture falls below a threshold, shade more aggressively; if the crop is at a sensitive stage, raise the temperature threshold so that shading kicks in only under severe heat. Threshold rules have practical virtues. They are transparent, easily audited and robust to sensor failure if sensible defaults are set. Their designers must take care to add hysteresis, meaning that the threshold for switching into a mode differs slightly from the threshold for switching out, to avoid the panels hunting back and forth as a passing cloud drops radiation just below and then above the line. They must also deal with the slow speed of the trackers themselves: a full rotation can take many minutes, so a rule that tries to respond to every cloud will spend much of its time moving and achieve little. The deeper weakness of rules is that they are myopic. A threshold rule decides what to do now based on conditions now. It cannot reason about the whole day. If the morning is cloudy and the afternoon will be scorching, a rule-based controller may let light through all morning, when there is little to let through, and then shade all afternoon, leaving the crop short of its daily light. It has no concept of a budget. Light budgets and crop models The next step is to give the controller a target for the day, or for a longer period, and let it plan. The most intuitive version is a light budget. The agronomist sets a daily light integral the crop should receive, perhaps varying with the stage of growth. The controller, using a forecast of the day's radiation, plans when to let light through so that the crop reaches its target, preferring the hours when light is most useful to the crop and least valuable to the grid. As the day unfolds and actual radiation departs from the forecast, the controller replans. If clouds arrive earlier than expected, it opens the panels more in the remaining hours; if the sky stays clear, it can shade more. A light budget is a large improvement over thresholds, but it still treats light as the only currency. Real crops trade light against water and temperature, and a fixed daily light target ignores the fact that on a very hot day the crop might benefit from less light than usual, and on a cool day from more. This is where crop models come in. A crop model is a computer simulation that represents the main processes of plant growth: interception of light by the canopy, photosynthesis, allocation of sugars to leaves, roots and harvestable organs, transpiration and the soil water balance. Models of this kind have been developed for decades for most major crops; well-known examples include STICS, developed by INRAE in France, and the APSIM and DSSAT families developed in Australia and the United States. For agrivoltaics, researchers have extended such models to handle the rapidly changing patterns of light and shade under moving panels. The French experience shows how this works in practice. INRAE researchers at the Lavalette experimental station near Montpellier built one of the first dynamic agrivoltaic test sites, with panels on single-axis trackers above lettuce and other crops. Benoît Valle and colleagues, publishing in Applied Energy in 2017, compared panels that followed the sun to maximise electricity with panels operated in a controlled mode designed to let more light reach the crop at chosen times. They showed that orientable panels, by intercepting the variable part of solar radiation, gave operators a new means of raising total land productivity beyond what fixed panels could achieve. Yassin Elamri and colleagues developed a coupled crop growth and water budget model for the same kind of system, able to handle the rapid fluctuations in radiation caused by moving panels, and used it to explore strategies trading light, water and yield. Sun'Agri, a company founded in France to commercialise this approach, built its system around this research, developed over about a decade with INRAE and the agricultural software company ITK. Its controllers combine weather forecasts, field sensors and crop models to decide how the panels should be oriented, with the stated principle that agricultural needs take priority over energy production. The company's vineyard and orchard installations rotate the panels to provide shade during periods of heat and water stress, open them when the crop needs light, and move them to protective positions against frost and hail. The advantage of a model-driven controller is that it can evaluate alternatives. Faced with a hot day ahead, it can simulate the crop's growth and water use under several panel strategies, estimate the effect of each on yield and quality, compute the energy each produces and choose the best combination according to a set of weights. The disadvantage is that the model is only as good as its calibration. Crop models are imperfect, particularly for quality attributes and for the unusual microclimate under panels, and an error in the model can cause the controller to make systematically poor choices that are hard for the farmer to detect. Optimisation over forecasts The most advanced control approaches formalise the trade-off as a mathematical optimisation problem and solve it repeatedly over a rolling horizon of weather forecasts. This technique, known as model predictive control, is widely used in industry, from chemical plants to building climate systems. At each control step, the controller forecasts the conditions over the next several hours or days, computes the sequence of panel angles that maximises a chosen objective over that horizon, implements the first step of that sequence and then repeats the whole process at the next step with updated forecasts and measurements. A 2026 study by Stuhlmacher, Srisuthankul, Mathieu and Seiler, engineers working on control of energy systems, illustrates the approach for dual-axis agrivoltaic trackers. The objective combined electricity production with an estimate of crop yield, weighted by a single parameter that expresses how much the operator values one against the other. The crop model linked light received to yield for lettuce, and the geometry of panel shadows was represented explicitly. The raw problem, with its trigonometric relationships between panel angle and shadow, is mathematically awkward. The authors reformulated it as a convex optimisation problem of a kind that standard solvers handle quickly and reliably, with only negligible loss of accuracy over most of the range of weightings. In a simulated 60-day lettuce season in Michigan, solved hour by hour, they reported land equivalent ratios of up to about 1.9, meaning the combined system produced nearly as much as two hectares managed separately. Introducing forecast errors of 10 per cent reduced the land equivalent ratio only slightly, to about 1.85. This is a simulation study, and like all such studies it depends on its assumptions about crop response, and on dual-axis trackers that are not the commercial norm. But it shows the direction of travel. The trade-off between energy and crop can be written as an explicit objective, solved efficiently, and made robust to the inevitable errors of weather forecasts. It also shows the central role of the weighting. Change the weight on crop versus energy, and the optimal trajectory of the panels changes. The algorithm can find the best compromise for any given weighting; it cannot choose the weighting. Table 3 summarises the main families of control strategy now in use or under development. Table 3. Families of agrivoltaic tracker control. Strategy Main inputs Strengths Weaknesses Calendar rules Date, time Simple, transparent, no sensors Blind to weather and crop state Threshold rules Radiation, temperature, humidity, soil moisture Responsive, auditable Myopic; no daily budget; can hunt Light-budget planning Radiation forecast, daily light target Ensures crop gets its light Treats light as only currency Crop-model control Forecasts, sensors, calibrated crop model Weighs light, water, heat, stage Depends on model accuracy Model predictive control Forecasts, crop and energy models, explicit weights Optimal for given weights; robust to forecast error Complex; weights must be set by people The hierarchy of commands Whatever the algorithm, it does not act alone. In any real installation, tracker commands come from several sources with different priorities, and how they are ordered matters as much as how any one of them is designed. At the top sit safety commands: wind stow, snow stow and, where used, hail stow, which protect the structure and the panels. These override everything. Next come crop protection events: frost protection on clear spring nights, heat protection during extreme temperatures, and hail protection for the crop where the structural design allows it. Below these sits the routine crop strategy, whatever its form, and finally ordinary energy tracking, which fills the remaining hours. Some systems add a layer for grid commands, such as curtailment when the grid operator asks the plant to reduce output, during which the panels can be turned to let light through to the crop at no extra cost to anyone. Two further features distinguish good agrivoltaic control systems. The first is spatial resolution. Because each tracker row has its own controller, different rows can be managed differently. A field with several crops, or with a crop at different stages, can be given different shading strategies row by row. Some research sites use this to run experiments on multiple strategies at once. The second is a farmer override. However sophisticated the algorithm, the person who farms the land knows things the controller does not: that harvest is planned for tomorrow, that a disease outbreak needs the canopy to dry, that a sprayer will pass beneath at 6 a.m. Good systems give the farmer a simple way to command the panels into particular positions for particular operations, and a clear record of what the controller did and why. Without that, the farmer is a tenant under a machine with its own priorities. The algorithm can make the auction for sunlight efficient. It cannot decide who should win it. That decision, expressed in the weights, the thresholds and the override rights written into contracts and regulations, is where agrivoltaics stops being an engineering problem and becomes a question of governance. The chapters that follow return to it repeatedly. Chapter 5: Water Under the Panels In most of the places where agrivoltaics is expanding fastest, from Andalusia and Puglia to Arizona, California, northern India and the Middle East, the limiting factor for farming is not light but water. A crop in these regions is seldom short of sunlight. It is often short of moisture, and it is short of moisture largely because of the sunlight. That reversal is why water belongs at the centre of agrivoltaic design, and why a tracker that can decide when to cast shade is potentially a water-management tool as much as an energy one. Where the water goes Water leaves a field in two main ways. Some evaporates directly from the soil surface. The rest passes through plants: drawn up by roots, carried to the leaves and released through stomata as vapour, a process called transpiration. Together they are known as evapotranspiration. In an irrigated field in a hot, dry climate, a summer crop can lose several millimetres of water a day this way, which across a hectare amounts to tens of cubic metres. The rate of evapotranspiration is governed by the energy available to turn liquid water into vapour, and by the ability of the surrounding air to carry the vapour away. Agronomists calculate it using a formula developed by Howard Penman in the 1940s and refined by John Monteith, which, in its standard form as adopted by the UN Food and Agriculture Organization, combines four main drivers: net radiation, air temperature, humidity and wind speed. The relationship can be described in words. Absorbed sunlight supplies the energy. Dry air, measured by the vapour pressure deficit, pulls water out of leaves and soil. Wind replaces humid air near the surface with drier air from above. Higher temperatures raise both the capacity of air to hold moisture and the vapour pressure at the leaf surface. Of these, net radiation is usually the largest single driver on clear summer days. Shading a crop reduces the radiation it absorbs and therefore the energy available for evaporation. A panel overhead also changes the other drivers. It slows the wind near the ground. It can raise humidity beneath it by trapping some of the vapour released by crop and soil. During the day it keeps the air and soil beneath it cooler than in the open, although at night, by blocking the radiation of heat to the cold sky, it can keep the ground somewhat warmer. The net effect, in study after study, is that crops and soil under panels lose less water than those in the open. The split between soil evaporation and transpiration matters for interpretation. Reducing soil evaporation is almost pure gain: water that would have evaporated from bare soil without contributing to growth stays available to roots. Reducing transpiration is more ambiguous. Transpiration is the price a plant pays for photosynthesis, since the same stomata that let water out let carbon dioxide in. If shade reduces transpiration because it reduces photosynthesis, the plant has not saved water so much as stopped working. If it reduces transpiration mainly by cutting the excess radiation load and vapour pressure deficit at midday, while photosynthesis continues at or near the level it would otherwise reach, the plant uses water more efficiently. The best agrivoltaic outcomes come from the second effect, and they are most likely when shade is concentrated in the hours of greatest evaporative demand. What the field evidence shows A body of field studies now documents these effects across a range of crops and climates. Some of the most frequently cited results are compared in Table 4. Table 4. Selected field results on water under agrivoltaic panels. Site and system Crop Water finding Source Lavalette, Montpellier, France; dynamic panels Lettuce Irrigation cut by 20% at cost of about 10% yield or a slightly longer cycle Elamri et al., 2018 Piolenc, Vaucluse, France; Sun'Agri dynamic panels Grenache vines Water requirement 12–34% lower depending on system Vitisphere report, 2020 Biosphere 2, Arizona, USA; fixed panels Jalapeño, tomato, chiltepin Soil moisture about 15% higher under alternate-day irrigation; jalapeño transpiration about 65% lower Barron-Gafford et al., 2019 Corvallis, Oregon, USA; fixed ground array Unirrigated pasture 90% more late-season biomass than open control; water-use efficiency 328% higher Adeh et al., 2018 The French lettuce work, carried out by Yassin Elamri, Bruno Cheviron and colleagues at INRAE and published in Agricultural Water Management in 2018, is particularly useful because it explicitly modelled the trade-off. Using a crop and water balance model calibrated at the Lavalette dynamic agrivoltaic site, they showed that irrigation could be reduced by 20 per cent while accepting either a yield decrease of around 10 per cent or a slight lengthening of the cropping cycle. In a region where irrigation water is rationed in dry summers, as southern France increasingly is, that trade is often worth making. The Arizona results, published in Nature Sustainability in 2019 by Greg Barron-Gafford and colleagues, came from a desert setting where the benefits of shade are larger. Under panels, chiltepin pepper production was roughly three times higher and cherry tomato production roughly twice that of open plots, while jalapeño yields were similar. Jalapeño plants under panels lost about 65 per cent less water through transpiration, and soil moisture under alternate-day irrigation was about 15 per cent higher. In an extreme environment, where open-field plants spend much of the day in heat and water stress, shading released them to grow. The Oregon study, by Elnaz Hassanpour Adeh, John Selker and Chad Higgins of Oregon State University, looked at a very different setting: unirrigated pasture under a fixed solar array in a region with wet winters and dry summers. Areas beneath the panels kept higher soil moisture through the summer, produced about 90 per cent more late-season biomass than the open control area and were far more water efficient. Here the effect was essentially to extend the growing season by delaying the point at which the soil dried out. These results come from fixed and dynamic systems, from deserts and temperate climates, and from crops of very different physiology. Their common lesson is that agrivoltaic shade saves water reliably, and that the agronomic value of the saving depends on how scarce water is. In a humid region where rain is ample, the water saving may matter little, and the loss of light may dominate. In an arid or semi-arid region, or in a temperate region during drought, the water saving can outweigh the light loss and raise yields. Rainfed arable crops raise a different question. Most of the world's grain is grown without irrigation, and in regions such as the Po valley, central Spain or the American Great Plains, summer drought is the main source of yield variation from year to year. Stefano Amaducci and colleagues at the Catholic University of the Sacred Heart in Piacenza used a crop model, calibrated for an elevated tracking agrivoltaic installation in northern Italy, to simulate maize under panels across many years of weather. Their 2018 study in Applied Energy found that the effect of the panels on maize depended strongly on water. In irrigated conditions, shade mostly reduced yields, as would be expected of a light-hungry C4 crop. In rainfed conditions, the reduction in water stress under the panels could offset the loss of light, and yields under shade were higher and less variable from year to year than in the open in the scenarios they modelled. That last point, stability, is easy to overlook. A farmer's income depends not only on the average yield but on the frequency of bad years. A system that slightly reduces yields in good years while preventing collapses in drought years may be worth more to a farm than one with a higher average and a wider spread. Agrivoltaic shade acts, for rainfed crops in drought-prone regions, partly as insurance. A tracker can strengthen this effect by letting light through in wet years, when water is not limiting, and shading more heavily in dry ones, so that the insurance is paid for only when it is needed. Rain, drip lines and the tracker as umbrella Panels do not only shade. They also intercept rain, and what they do with it can create problems that offset some of the gains described above. A flat or tilted panel collects the rain falling on its surface and sheds it from its lower edge. The ground beneath the panel's centre is sheltered and stays drier, while a narrow strip beneath the drip edge receives a concentrated stream. If the panels are fixed, this pattern repeats in every storm, producing persistent bands of wet and dry soil, with risks of erosion along the drip lines and uneven crop growth. Elamri and colleagues studied this phenomenon in detail at the Lavalette site, publishing their results in Hydrology and Earth System Sciences in 2018. They measured rainfall at many points beneath the panels and expressed the spatial unevenness using the coefficient of variation, a statistical measure of how much the values scatter relative to their average. With panels held flat during rain, the coefficient of variation in one set of measurements was 2.13, a very uneven distribution. With the panels operated in an avoidance strategy, rotated to minimise their interception of the falling rain, the coefficient fell to 0.22, comparable to the uniformity expected of a good irrigation system. They also built a model, which they called AVrain, to predict the effective rainfall reaching each point on the ground from measurements of rain, wind speed and wind direction. The implication for control is direct. A tracker can be programmed to respond to rain by turning its panels so that they present as narrow a profile as possible to the falling drops, usually close to vertical, with adjustments for the wind direction that tilts the rain. The crop then receives something close to the full rainfall, evenly distributed. Alternatively, in a heavy downpour that threatens to waterlog the soil or damage delicate crops, the tracker could hold the panels flat to shelter the crop, accepting the drip lines as the lesser problem. Either way, rain becomes another input to the controller, and another dimension in which a moving structure can do what a fixed one cannot. There is also a more ambitious possibility, explored in some research and design projects: collecting the rain shed by the panels in gutters and storing it for irrigation. Over a large area of panels, the volume collected in a wet season can be substantial. Whether this is practical depends on the cost of gutters and storage, the rainfall regime and the alignment between when rain falls and when the crop needs water. It remains a niche approach. Timing shade to evaporative demand If shade saves water mainly by reducing the radiation load during peak demand, then the logic of the tracker becomes clear. Shade is most valuable in water terms when the vapour pressure deficit is high and radiation is intense, usually from late morning through mid-afternoon on clear, hot days. It is least valuable in the early morning, when the air is cool and humid and evaporative demand is low. This aligns well with the crop's light-use logic described in Chapter 1. The hours when the crop can least use additional light are also the hours when shade saves most water. A control strategy that opens the panels in the morning and closes them around midday and in the early afternoon serves both purposes at once. It gives the crop light when its stomata are open and its leaves are cool, and withholds light when the leaves would be heating, closing their stomata and losing water. A tracker can also coordinate with irrigation. If a field has drip irrigation controlled by soil moisture sensors, the tracker controller and the irrigation controller can share data. On a day when the soil is moist, the tracker can let more light through, accepting some extra water use. On a day when soil moisture is low and irrigation water is constrained, it can shade more aggressively. Over a season, a coordinated system can trade light against water dynamically, aiming for the combination that gives the best crop outcome given the water allocation available. This kind of integration is still rare in commercial practice. Most agrivoltaic installations are designed and operated by energy companies, while irrigation is managed by the farmer, often with different equipment and software. Bringing them together requires not only compatible technology but agreement between parties about who controls what. Yet in water-scarce regions, it is arguably where the largest gains lie. The panel's side of the water ledger The water relationship runs in both directions. Solar panels lose efficiency as they heat up. For conventional crystalline silicon modules, output falls by roughly 0.3 to 0.5 per cent for every degree Celsius of cell temperature above the standard test condition of 25 degrees. On a hot summer day, cell temperatures in a ground-mounted array over bare gravel or dry soil can reach 60 degrees or more, costing a meaningful share of output. A crop beneath the panels, transpiring water, cools the air around it. Barron-Gafford's team in Arizona found that panels over crops ran cooler during the day than panels over bare ground in a conventional array, which translated into somewhat higher electricity output in the hot season. The effect is modest, typically a small percentage of energy, but it is a genuine synergy: the water the crop transpires, partly saved by the panel's shade, in turn cools the panel and improves its efficiency. Conversely, the panel's shade, by reducing the crop's water stress, can make the crop transpire more effectively during the hours when it is not shaded, adding to the cooling effect. Agrivoltaic systems that manage water well thus improve both outputs through a loop of mutual benefit. This loop is strongest in hot, dry, irrigated systems and weakest in cool, humid ones. Water is the strongest case for agrivoltaics in exactly those parts of the world where solar resources are richest and agricultural water most constrained. It is also where a tracker's ability to time its shade pays off most directly, because the value of shade in water terms is so strongly concentrated in particular hours. A field under a fixed roof gets a fixed share of shade, all day and all season. A field under a well-governed tracker gets shade when shade saves the most water and light when light grows the most crop. Chapter 6: Shelter From Heat, Frost and Hail Farmers have always built shelter for their crops: hedgerows against wind, walls against frost, nets against hail and birds, shade cloth over nurseries. Each of these is a fixed investment that protects against a particular threat at the cost of some light, space or labour. An agrivoltaic structure is, among other things, a very large and expensive shelter. Its panels can block sun, intercept hail, slow the wind and alter the radiation balance of the ground at night. Because it is expensive, and because it earns its keep primarily through electricity, the question of how it shelters the crop has usually been treated as secondary. As climate change makes extreme weather more frequent, that ordering looks increasingly outdated. For a tracker, sheltering is a matter of choosing positions. The same panels that face the sun to generate power can be turned to cast shade at the hottest hour, laid flat to act as a roof against hail or held horizontal overnight to keep frost off blossoms. Some of these positions cost energy; some cost nothing because they occur at night or during storms. Some conflict directly with the positions the energy industry requires for protecting the equipment. This chapter examines each threat in turn. Heat Heat damage to crops is no longer a rare event in the main agrivoltaic regions. On 28 June 2019, a heat wave pushed temperatures in southern France to 46 degrees Celsius, the highest ever recorded in the country. Vineyards in the Hérault and Gard suffered scorched leaves and burned bunches across large areas in a single afternoon. Similar events have hit orchards and vegetable crops across southern Europe, California and South Asia in the years since. Heat damages plants in several ways. Above a crop-specific threshold, enzymes in the photosynthetic machinery begin to lose function, and repair processes cannot keep up. Pollen viability falls during flowering, reducing fruit and grain set. Fruit exposed to direct sun can suffer sunburn, in which the surface tissue is literally cooked, leaving brown or bleached patches that make apples, peppers or grapes unmarketable. Leaves that cannot transpire enough to keep cool, because water is short or stomata have closed, can heat several degrees above air temperature. Shade addresses the most direct cause of these problems: the radiation load. A leaf or fruit in shade is not exposed to the direct beam and so stays much closer to air temperature. Under a panel, the air itself is usually somewhat cooler than in the open during the day, although the difference is typically a degree or two rather than many degrees, because the air mixes freely with the surrounding field. The bigger difference is in the temperature of surfaces exposed to the sun. A fixed structure provides some of this protection all the time. A tracker can provide much more of it exactly when it matters. In a heat wave, the controller can hold the panels in the position that shades the crop most completely during the hottest hours, even if that costs electricity. Because heat waves are forecast several days ahead with reasonable skill, the controller can plan for them, letting more light through in the days before so that the crop builds reserves, and then shading heavily on the critical afternoons. The cost of heat protection in energy terms is usually small. During the middle of a hot, clear day, the panels following the sun are already casting nearly the maximum shade they can. The main change is to hold them in shading positions into the late afternoon, when ordinary tracking would turn them west and let more light reach the eastern edge of each shadow strip. The energy lost in those hours is modest. There is a further benefit for people. University of Arizona researchers measured skin temperatures of people working in agrivoltaic plots about 10 degrees Celsius cooler than in open fields under desert sun. Heat stress among farm workers is a growing occupational health problem, and in some regions it is already restricting the hours when field work is possible. Shade structures that allow work to continue through the middle of the day have an economic value that rarely appears in agrivoltaic project assessments. Frost Spring frost is the other great weather threat to fruit growers and vineyards, and it has also become more damaging, not because frosts are more frequent, but because warmer winters and early springs bring buds out sooner, exposing them to frosts that would once have arrived while they were still dormant. In April 2021, a severe frost after a warm late March destroyed a large share of the French wine and fruit crop, in one of the worst agricultural disasters in the country in decades. The most damaging frosts in spring are usually radiative. On clear, calm nights, the ground and plants lose heat by radiating infrared energy to the cold sky. Without clouds to send some of that energy back, surface temperatures fall below the air temperature a metre or two up, and a layer of cold air pools near the ground. Buds and blossoms can freeze even when the air temperature measured at standard height stays just above zero. A panel overhead acts like a small cloud. It blocks part of the sky from the plants' view and, being warmer than the sky, radiates infrared energy back down to them. The plants beneath therefore lose less heat than those in the open, and their temperatures stay somewhat higher. The effect is typically a degree or so, sometimes more, depending on how much of the sky the panels cover and how high they are. In radiative frost, where the margin between survival and loss for a bud may be a degree, that can be decisive. For a tracker, frost protection means holding the panels flat at night, presenting their maximum area to the sky and thus blocking as much of the plants' view of it as possible. At night this costs no energy at all. Sun'Agri and other dynamic agrivoltaic operators have marketed frost protection as one of the benefits of their systems. The effect is real, but it should not be overstated. In advective frosts, where a mass of cold air flows into a region with wind, the radiative effect of panels offers little help. And for severe radiative frosts, panels alone will not match the protection of dedicated methods such as wind machines, heaters or overhead sprinklers. They are a supplement, not a substitute. Hail Hail is the most dramatic threat, and the one that brings the interests of panel and crop into sharpest conflict. A few minutes of large hail can destroy a year's fruit crop and shatter the glass of solar modules. Orchards in hail-prone regions are often covered with hail nets, which typically cost a significant sum per hectare and reduce light by a modest fraction. An agrivoltaic structure over an orchard can replace or supplement those nets, offering a second return on the investment. The physics of hail protection is straightforward. A hailstone falls nearly vertically, with some angle from the wind. A panel lying flat presents its full area to the falling hail and intercepts every stone that would have hit the ground beneath it. A panel tilted steeply presents a narrow profile and intercepts few. For crop protection, flat is best. The physics of panel protection points the other way. A hailstone striking glass at right angles delivers its full momentum to the glass. One striking at a glancing angle delivers much less, because much of its momentum carries along the surface. That is why the hail stow strategy, which the solar industry adopted rapidly after a string of costly hail losses at large plants in the early 2020s, turns trackers to their steepest angle when hail is forecast. In that position, the panels are best protected and the crop beneath is almost fully exposed. This is a genuine conflict, not a matter of algorithm design. There are only a few ways to resolve it. One is to accept the risk to panels: hold them flat to protect the crop and absorb some module breakage, which may be acceptable for small hail but not for the large stones that cause most of the damage. A second is to use stronger modules. Glass-glass modules with thicker front glass, already common in agrivoltaics because they allow some light through and are more durable, withstand hail better than standard designs and allow a flatter stow without unacceptable risk. A third is to supplement the structure with hail netting in the gaps between panels, so that the crop is protected even when the panels are tilted. Sun'Agri has used various combinations of these approaches in its orchard installations. The resolution chosen has to be agreed in advance among the farmer, the energy operator, the module manufacturer and the insurer. It cannot be left to the algorithm alone. It is one of the clearest examples of why agrivoltaic tracker control is ultimately a matter of contract and governance. Wind, humidity and disease Two further effects of panels deserve mention, because they cut in both directions. Panels slow the wind near the ground. For crops prone to wind damage, or in exposed sites, this is a benefit, and reduced wind also means reduced evaporation. But tracker rows themselves must be protected from high winds by stowing, which, depending on the design, may tilt them to a position that offers little shelter. And the structure as a whole must be engineered for wind loads that increase with height, which is one reason elevated structures cost more than ground-mounted ones. Reduced wind and greater humidity under panels can also favour fungal diseases. Many of the most damaging crop diseases, including downy mildew in grapes and botrytis in strawberries and grapes, thrive when leaves stay wet for long periods. Under panels, dew may form and persist longer, and the canopy may dry more slowly after rain. Growers using agrivoltaic systems have reported the need to adjust their disease management, sometimes spraying more often or adopting different timings. Here again, a tracker offers options. In the early morning, when dew sits on the canopy, the controller can open the panels fully to let direct sun reach and dry the leaves. After rain, it can prioritise letting light through to speed drying. Some research has explored using the tracker to manage humidity under the panels, alongside light and temperature, though this remains less developed than heat or frost strategies. A year above a vineyard It helps to see these functions together, as a controller would apply them through a single season. Consider a vineyard of Grenache in the southern Rhône under an overhead tracker system, its panels several metres above the vines. What follows is a composite sketch, consistent with the published strategies of dynamic agrivoltaic operators and the physiology described in earlier chapters, rather than the log of a particular site. In winter, the vines are dormant and leafless. The crop has no use for light, and the controller simply tracks the sun for maximum energy, interrupted only by wind stow during storms. These months cost the farm nothing. In late March and April, buds swell and break. This is the frost season. On clear, calm evenings when the forecast shows a risk of radiative frost, the controller lays the panels flat at dusk and holds them there until sunrise, reducing the heat the young shoots lose to the sky. During the day, it tracks normally, because the shoots are small and their need for light is modest relative to what reaches them. From May into June come flowering and fruit set, when the vine's need for carbohydrate is high and poor light can reduce the number of berries. The controller now leans toward the crop. It opens the panels toward edge-on through the morning and late afternoon, and tracks only around the middle of the day, when the vines are most likely to be saturated. If a hail warning arrives, the response is whatever the contract specifies: flat panels to shelter the flowers, perhaps combined with netting, or a steeper stow that protects the glass at the vines' expense. High summer, from late June through August, is the season of heat and water stress. Berries are sizing, and in the hottest years sunburn and leaf scorch are real threats. Now the controller leans the other way. On clear, hot days it holds the panels in shading positions from late morning through mid-afternoon, guided by forecasts of temperature and vapour pressure deficit and by soil moisture readings. On cooler or cloudier days, it gives more light back. If a heat wave is forecast, it plans for maximum shade during the critical afternoons. Veraison, when the berries soften and colour, and the weeks of ripening that follow call for a more delicate balance. Some light on the bunches helps colour and aroma development; too much heat accelerates sugar accumulation and burns off acidity. The controller, guided by the winemaker's goals, may moderate its afternoon shading, letting the vines finish under somewhat more light than in July. At harvest, the panels are turned to whatever position eases access for pickers or machines, and the farmer's override takes precedence. After harvest, as leaves yellow and fall, the controller returns to energy tracking for the winter. Across such a season, the energy cost of agricultural priority is concentrated in a few months, mainly the flowering period and the hottest weeks of summer, and even then mostly in particular hours. The rest of the year, the system behaves much like an ordinary solar farm. That pattern, a predictable and limited energy sacrifice concentrated in the agronomically critical windows, is what makes dynamic agrivoltaics financially workable over perennial crops. Protection as a reason for being Taken together, these protective functions can amount to a substantial share of the agricultural value of an agrivoltaic system, particularly for high-value perennial crops such as wine grapes, apples, pears, apricots and berries. For such crops, a single heat wave, frost or hailstorm can destroy most of a year's income, and the costs of conventional protection are high. A structure that provides shade, frost mitigation and hail cover, while also generating electricity, can make economic sense for the farmer even if the crop yield in a normal year is slightly lower. This shifts the frame for agrivoltaics. Much of the policy debate assumes the crop beneath the panels will suffer some yield loss, and asks how much loss is acceptable for a given amount of energy. For protective agrivoltaics over vulnerable perennial crops, the more relevant comparison is with the risk of losing the crop entirely in a bad year. In a warming climate, with more frequent extremes, the average loss in normal years may be far smaller than the average saving in bad ones. This is the strongest argument for agrivoltaics in many of the wine, fruit and vegetable regions of southern Europe, and it depends critically on a controller that knows when to move the panels into protective positions and has the authority to do so even when it costs energy. It also highlights the limits of thinking about agrivoltaic control purely as optimisation. Heat waves, frosts and hailstorms are rare, high-impact events. A controller tuned to maximise expected returns over an average season may pay too little attention to them, just as an insurance policy that is rarely claimed may seem expensive until it is needed. Good protective control requires explicit rules, triggered by reliable forecasts, that give priority to the crop when the risk is greatest, even at the cost of energy in that hour. Those rules must be written into the hierarchy of commands described in Chapter 4, below equipment safety but above routine energy and crop optimisation. Chapter 7: Measuring Success A system that produces two outputs from one piece of land needs a way of judging whether the combination is worth more than either output alone. For agrivoltaics, that question has a scientific answer, a legal answer and a commercial answer, and they do not always agree. The scientific answer is the land equivalent ratio. The legal answer, in a growing number of countries, is a yield threshold written into regulation. The commercial answer is whatever the contracts say. This chapter examines the first two and how they bear on the control of trackers. The third is the subject of the next chapter. The land equivalent ratio The land equivalent ratio was developed in the 1970s by agronomists studying intercropping, the practice of growing two or more crops together in the same field. Its logic is simple. Measure the yield of each crop grown alone on a hectare, and the yield of each grown together on a hectare. Divide each mixed yield by the corresponding sole yield and add the fractions. The sum is the area of land that would be needed, with each crop grown separately, to produce what one hectare produces in the mix. A land equivalent ratio greater than one means the mixture uses land more efficiently. Christian Dupraz and colleagues at INRAE adapted the concept to agrivoltaics in their 2011 paper in Renewable Energy, treating electricity as one crop and food as another. A worked example shows how it operates. Suppose a farmer in northern Italy grows winter wheat, and on open land the field yields 8 tonnes per hectare. Under an elevated agrivoltaic structure on the same soil, the wheat yields 6.4 tonnes per hectare, a crop ratio of 6.4 divided by 8, or 0.8. Now consider electricity. A conventional ground-mounted solar plant on the same kind of land might pack panels densely enough to generate, say, 1,200 megawatt-hours per hectare per year. The agrivoltaic structure, with panels spaced more widely to let light through, holds roughly half the panel area per hectare. Because the panels are higher and better ventilated, and because they shade each other less, each panel may produce slightly more than in the dense plant. Suppose the agrivoltaic system generates 660 megawatt-hours per hectare per year. The electricity ratio is then 660 divided by 1,200, or 0.55. The land equivalent ratio is 0.8 plus 0.55, which is 1.35. To produce 8 tonnes of wheat and 1,200 megawatt-hours separately would require one hectare of wheat and one hectare of solar. To produce the same quantities using agrivoltaics would require about 1.48 hectares (two hectares divided by 1.35). The combined system, in other words, saves about a quarter of the land. Field results from research systems have generally fallen in the range from about 1.3 to somewhat below 2. Dupraz's original modelling estimated values from about 1.35 to 1.73 depending on the configuration. Fraunhofer ISE's Heggelbach site reported a land use efficiency, calculated in essentially the same way, of 160 per cent in 2017 and 186 per cent in the drought year of 2018, when the crops under the panels did comparatively well. Optimised tracking in simulation, as in the model predictive control study described in Chapter 4, has reached values close to 1.9. The land equivalent ratio has real merits as a measure. It is dimensionless, comparing like with like, and it captures the central claim of agrivoltaics: that sharing land between two uses can produce more in total than dividing it. It also makes plain the role of control. A tracker strategy that shifts some light from panels to crops will lower the electricity ratio and raise the crop ratio; whether the sum rises depends on whether the crop gains more, proportionally, than the panels lose. Because the crop's response to light is curved, with large gains when light is scarce and small gains when it is abundant, there is usually an interior optimum where the sum is maximised. A second illustration shows how control strategy moves the ratio on a single installation. Take a tracker system over a leafy vegetable crop. Suppose that, run purely for energy, it produces 60 per cent of the electricity a dense conventional plant would produce on the same area, and the crop yields 70 per cent of the open-field reference. The land equivalent ratio is 1.30. Now suppose the operator adopts a crop-first strategy that returns to the crop an amount of light equal to about a twelfth of what the panels were catching, concentrated in the cool morning hours when the crop uses it most efficiently. The electricity ratio falls by five points, to 0.55. If the crop, which was clearly light-limited at 70 per cent, responds strongly to that well-timed light and rises to 85 per cent of the reference, the ratio becomes 1.40. Suppose instead the operator returns the same amount of light around midday, when much of it falls on saturated, heat-stressed leaves. The electricity ratio still falls to 0.55, but the crop may gain only modestly, to perhaps 77 per cent. The ratio is then 1.32, barely better than the energy-only strategy. The same hardware, the same physical quantity of light returned and the same energy cost give a land equivalent ratio of 1.30, 1.32 or 1.40 depending on when the light is given back. The numbers are illustrative, but the pattern follows directly from the crop physiology of Chapter 1: timing, not quantity, separates good control from indifferent control. But the ratio has important limitations. It says nothing about value: a tonne of wheat and a megawatt-hour of electricity are each counted as fractions of their own sole yield, regardless of price. A system with a high land equivalent ratio might still be uneconomic for the farmer if the crop ratio is low and the farmer receives only a small share of the electricity revenue. It says nothing about quality, which, as the vineyard examples in earlier chapters show, can matter as much as quantity. And it is sensitive to the reference against which each output is measured: a poor reference field makes the agrivoltaic crop look better, and a choice of dense conventional solar as the electricity reference can make the agrivoltaic electricity ratio look worse. Regulators have therefore preferred simpler, harder tests. Yield thresholds and control plots The core regulatory instrument across the countries leading in agrivoltaics is a yield threshold: the crop under the panels must yield at least a set fraction of a reference. The thresholds differ considerably, as do the surrounding rules, and they are summarised in Table 5. Table 5. How four countries define acceptable agrivoltaics. Country Main instrument Crop yield condition Other key conditions France Law of 10 March 2023; decree of 8 April 2024 Yield loss below 10% versus control zone Must deliver an agricultural service; panel coverage capped at 40% with exemptions for proven designs Germany DIN SPEC 91434 (2021), referenced in renewables law At least 66% of reference yield Land lost to structures at most 10% (overhead) or 15% (interspace) Japan Farmland conversion permits since 2013; since tightened At least 80% of local average yield Cultivation plans, annual reports; benchmarks of under 30% shading, about 3 m height Italy National guidelines (2022) and recovery-fund tender (2024) Continuity of farming, verified by monitoring At least 70% of area in agricultural use; elevated or tracking structures for advanced category France sets the strictest yield condition. Under the decree of April 2024, implementing the renewable energy acceleration law of March 2023, an agrivoltaic installation must provide at least one direct service to the agricultural activity, such as improving agronomic potential, adapting to climate change, protecting against weather hazards or improving animal welfare. The yield loss on the crop under the panels must remain below 10 per cent of the average on a control zone, and the farm's agricultural income must not fall. Panel coverage is capped at 40 per cent of the plot, though designs with proven agricultural performance may be exempted. The French framework thus defines agrivoltaics as a service to farming, not a sharing of land on equal terms. Germany's DIN SPEC 91434, published in 2021 by the national standards body, is looser on yield, requiring that the agricultural yield be at least 66 per cent of a reference, but strict on the loss of land to the structure itself. It divides systems into two categories, overhead systems with a clear height of at least 2.1 metres and interspace systems at lower heights, with different limits on land loss. The German renewable energy law references the standard in defining which agri-PV systems qualify for support. Japan's rules, the oldest of the four, allow farmland to carry solar structures under a temporary conversion permit, typically renewed every few years, on condition that the crop yield remains at least 80 per cent of the local average. By the end of fiscal 2023, Japan had approved 6,137 such sites covering about 1,360 hectares. Reviews found that around a fifth to a quarter of sites had problems with their farming, and in response the ministry tightened the rules, requiring cultivation plans, financial projections and annual reports, and proposing technical benchmarks: shading under 30 per cent, panel heights of about 3 metres and support pillar spacing of 4 to 5 metres to allow machinery. Italy has taken a different path, tying its definition to public funding. National guidelines published in 2022 set out requirements, including that a large majority of the area, at least 70 per cent, remain in agricultural use. The recovery-fund support programme launched in 2024 targets "advanced" agrivoltaic systems, using elevated or tracking structures that allow continuous cultivation, combined with monitoring of both the crop and the microclimate. In its first tender, concluded in December 2024, Italy selected 540 projects totalling about 1.5 gigawatts. What thresholds do to control Yield thresholds shape tracker control directly. A system under the French rules must deliver at least 90 per cent of the control yield. Its operators have a strong incentive to program the trackers to protect the crop, letting light through whenever the crop's yield would otherwise fall toward the threshold. A system under the German standard, with its 66 per cent threshold, has much more room to favour energy. A Japanese system at 80 per cent sits between. The thresholds also interact with the variability of crop yields. Yields vary considerably from year to year, and from one part of a field to another, for reasons that have nothing to do with the panels. A system that typically yields 92 per cent of its control might fall below 90 per cent in a bad year through sheer chance. A cautious operator will therefore aim well above the threshold on average, which means leaving more light for the crop than the threshold alone would require. Conversely, if the control plot happens to perform poorly, the threshold becomes easy to meet. This sensitivity to the reference is one of the main weaknesses of yield-based regulation. Control plots must be comparable in soil, management and microclimate, and they must be managed identically to the agrivoltaic area. In practice, this is hard to achieve and harder to verify. A control plot that receives slightly less fertiliser or water, or sits in a slightly worse part of the field, can make a mediocre agrivoltaic system look acceptable. The French rules specify how control zones should be defined and monitored, but enforcement depends on inspections and on trust. Measuring what the controller does An alternative, or complement, to yield-based measurement is to measure what the system does to the crop's environment, rather than only what the crop yields at the end of the season. If the regulator or the farmer can see how much light the crop received each day, when it received it, and how the tracker moved, then the performance of the control strategy can be assessed directly. Modern trackers already log their angles continuously. Combined with the site geometry and measurements of incoming radiation, these logs allow a calculation of the light reaching each part of the field, hour by hour. Sensors under the panels can confirm the calculation. With such data, a farmer could see whether the tracker had delivered the light budget agreed for each stage of growth, whether it had moved to frost or heat protection when forecasts warranted, and how much energy each decision had cost. This kind of transparency is technically straightforward and rarely implemented. Most agrivoltaic monitoring focuses on energy output, which the plant owner needs for its revenue, and on the annual crop yield, which the regulator requires. The intermediate layer, the light and shade the controller actually delivers, is usually invisible to the farmer. Yet it is the layer in which the farmer's interests are decided. Italy's recovery-fund programme, by requiring monitoring of microclimate and crop alongside energy, moves in this direction. A logical next step would be to require that tracker control logs, translated into light delivered to the crop, be made available to the farmer and the regulator in a standard format. The land equivalent ratio and the yield threshold judge outcomes after the fact. A light delivery record would allow problems to be detected and corrected during the season, while there is still time to save the crop. Measuring success in agrivoltaics, in short, means measuring the allocation of light, not just its consequences. The land equivalent ratio shows whether the allocation is efficient overall. Yield thresholds set a floor on the crop's share. Neither tells a farmer, in July, whether the algorithm moving the panels above the field is serving the crop as promised. For a technology whose central promise is intelligent sharing, that is a significant gap. Chapter 8: Who Controls the Sun Every agrivoltaic project is a partnership between two businesses with different time horizons, different risks and different ideas of what the land is for. The farmer thinks in seasons and in decades of soil fertility. The energy investor thinks in a power purchase agreement of fifteen to twenty-five years and in the internal rate of return on a large upfront capital outlay. The tracker sits between them, and every rotation it makes moves value from one to the other. This chapter examines the economics that frame that partnership, the contracts that govern it and the evidence, mostly from Japan, about what happens when it goes wrong. The cost of lifting panels Agrivoltaic systems cost more to build than conventional ground-mounted solar. The main reason is the structure. Lifting panels several metres above the ground to allow machinery beneath, and spacing them more widely to let light through, requires taller and stronger posts, deeper foundations, longer spans and more steel per kilowatt. Wind loads rise with height, so the structure must be designed for larger forces. Overhead trackers add the cost of drives and controllers spread over fewer panels per hectare. Construction itself is more demanding, and must be planned to avoid compacting or damaging the soil that the farm depends on. Lower panel density also reduces electricity output per hectare, so fixed costs such as grid connection, permitting and site preparation are spread over fewer megawatt-hours. The combined effect is that the levelised cost of electricity from an elevated agrivoltaic system is usually materially higher than from a conventional solar farm on the same land. Interspace systems and vertical bifacial fences, which use lower, simpler structures, narrow the gap; elevated overhead trackers widen it. Governments that want agrivoltaics therefore have to pay for it, or allow it to be paid for through higher prices. Italy's 2024 tender shows the scale of support. Projects competed for a tariff with ceiling prices of 93 euros per megawatt-hour for installations up to 300 kilowatts and 85 euros for larger ones, alongside capital grants from European recovery funds. Bidders offered discounts ranging from about 9 to 53 per cent from those ceilings, with larger projects able to offer more. The spread suggests that the cost premium of agrivoltaics varies widely with design and scale, and that some developers believe they can build systems eligible for support at prices not far above ordinary solar. What society gets for the premium is the continued use of the land for farming, together with whatever agronomic benefits the system provides. Whether that is good value depends on how scarce farmland is, how much the public values its agricultural use and how real the agronomic benefits are. In densely populated countries with limited farmland and strong political attachment to agriculture, such as Japan, France, Germany and Italy, the premium has proved politically acceptable. In countries with abundant land, it is harder to justify, which is one reason agrivoltaics in the United States has developed more slowly, through research sites and state-level programmes rather than national schemes. Two price curves and a contract The economics of the tracker's moment-to-moment decisions depend on two price curves introduced in earlier chapters: the crop's value for light and the market's value for electricity. Both have been changing. The electricity curve has been reshaped by the success of solar power itself. In markets with high shares of solar generation, such as Germany, Spain, the Netherlands, California and parts of Australia, wholesale prices around midday on sunny days have fallen sharply, and in some hours turned negative, because so much solar output reaches the grid at once. Germany recorded hundreds of hours of negative prices in 2024 and 2025. A megawatt-hour delivered at noon in June is now often worth far less than one delivered at 7 p.m. For agrivoltaics, this has two effects. First, it lowers the energy cost of giving the crop light at midday. If midday power is nearly worthless, rotating panels to let light through to the crop costs the operator very little. Second, during hours of negative prices, when many plants are required or choose to curtail output, a tracker can turn its panels fully toward the crop at no cost at all. These are welcome developments. But they cut against the crop's own valuation: midday is precisely when the crop needs light least. The hours when the crop most values light, mornings and evenings, are increasingly the hours when electricity is most valuable. A tracker that gives the crop its light in the morning is spending the operator's most valuable output. How this tension is resolved depends on contracts. In a typical agrivoltaic project, a developer leases land from a farmer, or partners with a farmer who continues to work the land. The developer builds, owns and operates the solar plant and sells the electricity. The farmer receives rent or a share of revenues and keeps the agricultural income. The control of the trackers is formally a matter for the plant operator, but the contract may specify an agricultural operating plan, priority rules for crop protection, light budgets or minimum yield commitments, and penalties or compensation if the crop suffers. Sun'Agri's model, built on the principle that agriculture comes first, gives the agronomic strategy priority over energy in the control logic, and the company's commercial pitch has been to farmers seeking climate protection for their crops, with the electricity revenue making the investment viable. Many other projects make no such commitment. In those, the tracker follows an energy-maximising strategy by default and deviates only to meet the minimum agricultural conditions required by regulation or contract. A well-written agrivoltaic contract therefore needs to address several things explicitly: who sets the agronomic strategy and how often it can be changed; which events trigger protective positions and who bears the energy cost; what data the farmer receives about tracker movements and the light delivered; how disputes about crop performance will be resolved, including the choice of control plot; and what happens to the land and the structure at the end of the project. Each of these is, in effect, a clause about the control algorithm. The rooftop objection The most common objection to agrivoltaics is not that it fails but that it is unnecessary. Why build expensive elevated structures over farmland, the argument runs, when there are millions of hectares of roofs, car parks, motorway verges, former industrial sites and degraded land available for ordinary solar panels? Put the solar there, and leave the farms alone. The objection has force, and agrivoltaic advocates should concede part of it. Rooftop and brownfield solar avoids any conflict with food production and places generation close to demand. Many countries have far more of this potential than they are using, and policies that encourage it are sensible. Nothing in the case for agrivoltaics requires neglecting them. But the objection overlooks three things. The first is scale and cost. Rooftop systems are typically much more expensive per kilowatt than ground-mounted plants, and the transition to low-carbon electricity in most countries is planned to rely heavily on large ground-mounted solar. The European Commission's Joint Research Centre estimated in 2023 that agrivoltaic systems on just 1 per cent of the EU's utilised agricultural area could host something on the order of a thousand gigawatts of capacity, more than the EU's entire solar target for 2030. Ground-mounted solar will be built on farmland in any case. The real choice is not between agrivoltaics and rooftops but between agrivoltaics and conventional solar farms that take land out of farming. The second is location. Large solar plants need grid connections, and grid capacity is often available in rural areas where farmland is the only land available. A project that can share land with farming may be the only realistic way to use that connection. The third, and most important for this book, is the agronomic case itself. In hot, dry regions facing intensifying heat, drought and hail, controllable shade over vulnerable crops has value in its own right. A vineyard that suffers less from heat waves, an orchard spared a frost, a vegetable field that needs a fifth less water: these benefits do not arise from rooftop solar. For such farms, agrivoltaics is not a way of fitting solar onto farmland but a way of protecting the farming that is there. The rooftop objection answers the question of where to put solar panels. It does not answer the question of how to shield crops from a changing climate. The Japanese warning Japan offers the longest record of agrivoltaics at scale, and its experience is instructive. Since 2013, Japanese farmland owners have been able to install solar structures under temporary permits, conditional on maintaining crop yields of at least 80 per cent of the local average. Japan's feed-in tariff, introduced in 2012 at generous rates, made such projects highly attractive. By the end of fiscal 2023, 6,137 sites had been approved, covering about 1,360 hectares. The results have been mixed. Government reviews found that about 22 per cent of projects had problems with their farming in fiscal 2022, rising to 24 per cent in fiscal 2023. Most problems were attributed not to external factors such as weather but to poor management or cultivation practices. In some cases, the crop under the panels had been neglected, with the land effectively used as a solar farm and only nominal farming continuing. The government suspended subsidies to 15 installations run by six operators in August 2024 and to 14 installations run by seven operators in November 2024. The crops grown under Japanese agrivoltaic systems also reveal something. Ornamental plants account for about 36 per cent of sites, vegetables about 28 per cent and fruit about 13 per cent. Among the ornamental plants, shade-tolerant species grown for cut foliage, such as sakaki, a shrub used in Shinto ritual, have been popular because they tolerate heavy shade. Choosing a crop that can survive under almost any amount of shade is a rational response to a regulation that measures only whether the crop survives, but it means the yield threshold constrains the energy system very little. Japan's response has been to tighten the rules. New requirements, elaborated in national benchmarks published in 2026, require developers to submit cultivation plans, financial projections and equipment designs, to provide evidence that crops can grow beneath the panels and to report annually on production and finances. Approvals can be denied where yields fall by 20 per cent or more, farming is neglected, or panel height and spacing prevent the use of machinery. The proposed technical benchmarks of under 30 per cent shading, about 3 metres of height and 4 to 5 metres between pillars are explicit attempts to fix in hardware what could not be secured through yield conditions alone. The lesson for tracker-based agrivoltaics is double-edged. Fixed rules about shading and height constrain the worst outcomes but also prevent the dynamic strategies described in this book, which may involve shading well above 30 per cent at some hours and far below it at others. A tracker can satisfy an average shading benchmark while concentrating its shade intelligently, or it can satisfy the benchmark while concentrating its shade in exactly the wrong hours. Hardware rules cannot tell the difference. Only an examination of the control strategy and its results can. Building trust in the algorithm The broader lesson from Japan, from the French debate over what counts as agrivoltaics and from the growth of agrivoltaic tenders in Italy and Germany, is that the sector's legitimacy depends on demonstrating that farming is real and that the energy system serves it. As more agrivoltaic systems move from fixed structures to trackers, that demonstration increasingly depends on the control algorithm. In the United States, research by the National Renewable Energy Laboratory has emphasised the human side of agrivoltaic success. Its framework of five factors, published in 2022, covers the climate, soil and environmental conditions of a site; the configuration of the solar technology; the choice of crops and cultivation methods; compatibility between solar and agricultural operations; and collaboration among the people involved. The last two are essentially about governance. A system can be technically sound and still fail because the farmer and the operator do not trust each other, or because the farmer cannot influence how the structure above the field is used. Sites such as Jack's Solar Garden, a 1.2-megawatt community solar installation near Longmont, Colorado, built in 2020 with research partners including NREL and the University of Arizona, have shown how collaboration can work at a small commercial scale, with farmers, researchers and solar operators sharing data and adapting practices over several seasons. Scaling that kind of collaboration to thousands of sites operated by large energy companies is a different challenge. Several practical measures would help. Tracker control logs, translated into the light delivered to the crop, could be made available to farmers in real time, so that they can see how the panels above their fields have been used and challenge strategies that do not serve the crop. Agronomic control strategies could be documented and filed with permit applications, as Japan now requires for cultivation plans, so that regulators can review not just the hardware but the intended use of it. Farmers could be given defined override rights, specifying when they can command the panels for operations such as spraying, harvest or disease control, and how the resulting energy loss is shared. And the weightings in model-based controllers, which determine the balance between energy and crop, could be stated explicitly in contracts rather than buried in proprietary software. None of these measures requires new technology. They require recognising that in a tracker-based agrivoltaic system, the algorithm is where the partnership between farm and energy is enacted, hour by hour. A contract that specifies rent and minimum yield but says nothing about how the panels will move leaves the most important decisions to one party. Conclusion Agrivoltaics began as a geometric idea: lift the panels high enough and space them widely enough, and a crop can grow beneath them. That idea was sound, and it remains the foundation of every system built today. But geometry alone delivers a compromise. A fixed structure takes a fixed share of the sun, whether the crop needs the light or not, whether the day is mild or scorching, whether the plants are flowering or merely growing leaves. Its outcome depends on whether the average share happens to suit the crop, the climate and the season. Sometimes it does. Often it does not. The argument of this book has been that the most important advance in agrivoltaics is the shift from geometry to control. A tracker above a field is a valve on sunlight. It can take the light the crop cannot use and turn it into electricity, and return the light the crop needs. It can cast shade when shade saves the most water, hold its panels flat on frosty nights and turn them to spread rain evenly. Each of these choices is made by an algorithm, and the algorithm determines, far more than the steel and glass, whether the system delivers more from the land than either use alone. Several consequences follow. The control policy is the product The first is that agrivoltaic projects should be judged, designed and regulated by their control policies, not just their hardware. Two installations with identical structures, panels and trackers can deliver entirely different outcomes for the crop, depending on how the trackers are programmed. One may give the crop its light in the cool morning hours and shade it during the heat of the afternoon; the other may follow the sun for maximum energy and deviate only when a contractual minimum is threatened. From the road, they look the same. In the field, they are different systems. Current regulation mostly ignores this. France caps yield losses and panel coverage. Germany caps land lost to structures and sets a yield floor. Japan sets yield thresholds and is moving toward fixed limits on shading and height. Italy requires monitoring and elevated or tracking structures. All these rules govern outcomes or hardware. None governs the control strategy itself. As the share of agrivoltaic capacity built on trackers grows, this gap will matter more. A regulation that sets a maximum shading percentage cannot distinguish between a tracker that shades intelligently and one that shades badly. A rule that requires the control strategy to be documented, logged and reported, in terms of the light it delivers to the crop over time, can. The weights belong to people The second consequence is that the most consequential parameter in an agrivoltaic controller is not technical. Every optimising algorithm, from simple threshold rules to model predictive control, embeds a choice about how much a unit of crop is worth relative to a unit of electricity. In threshold rules, the choice is hidden in the threshold values. In optimisation, it is explicit in the weights. Either way, it determines who wins the auction for sunlight in each hour. Engineers can make the auction efficient. They cannot decide its rules. Those rules must come from the farmer and the energy operator, through contracts, and from the public, through regulation. The French principle that agrivoltaics must serve farming is one answer. The German standard, permitting a third of the crop to be lost, is another. Neither is wrong in itself, but both should be translated into control parameters that can be inspected and enforced. A weighting written into a contract, and visible in the controller's logs, protects the farmer far better than a promise of agricultural priority buried in proprietary software. Water and protection are the strongest cases The third consequence concerns where agrivoltaics makes most sense. The evidence surveyed in this book points clearly to systems in hot, dry or climatically volatile regions, over crops that suffer from excess radiation, heat or water stress, and for which protection from extreme weather has high value. Vineyards and orchards in southern Europe, vegetable production in arid regions, berries and other high-value horticultural crops in areas prone to hail or heat waves: these are where shade saves water, where the crop's appetite for light is most often exceeded and where the protective functions of a tracker can prevent catastrophic losses. In cool, cloudy regions growing light-demanding arable crops, the case is weaker. Shade there more often costs yield than saves it, and the water and heat benefits are smaller. Agrivoltaics can still make sense, particularly in drought years, as Heggelbach demonstrated, or with designs such as vertical bifacial fences that take little midday light. But these regions should not expect the large mutual benefits reported from deserts and Mediterranean vineyards. This geography aligns with climate trends. The regions where agrivoltaics offers the most are the regions facing the sharpest increases in heat, drought and weather extremes. As those pressures intensify, the agricultural value of controllable shade will grow, and with it the case for paying the cost premium of elevated tracker systems. Integration is the frontier The fourth consequence is that the largest remaining gains lie in integration. Today, most agrivoltaic trackers are controlled by energy companies using software developed for solar farms, with agricultural rules added on top. Irrigation is managed separately by the farmer. Crop models are used in research and by a few specialist companies, but rarely linked to commercial tracker controllers. Weather forecasts inform stow decisions but seldom feed into daily light planning. A fully integrated system would treat the field and the solar plant as a single managed environment. The tracker controller, the irrigation system, the crop model and the weather forecast would share data and optimise jointly, with the farmer setting priorities and the energy operator setting financial constraints. Light, water and heat would be allocated across the day and the season to maximise the combined value of the crop and the electricity, within rules agreed by both parties and visible to regulators. None of the technical components of such a system is beyond current capability. What is missing is the institutional arrangement that would bring them together. The sun as a shared resource Agrivoltaics forces a recognition that has been implicit in agriculture since its beginnings: sunlight is a resource to be managed, not just a condition to be accepted. Farmers have always worked with the light they were given, choosing crops, planting dates and row directions to make the best of it. The tracker gives them, for the first time, a means to redistribute that light through the day and across uses. That capacity carries responsibility. A machine that can decide, every few minutes, how much sunlight reaches a field is a machine that can serve the farm or exploit it. Which it does depends on how it is programmed, who controls the programming and whether its decisions are visible to those affected. The technology has matured quickly. The governance of its algorithms has barely begun. Getting that governance right, so that the sun's surplus feeds the grid and its scarce hours feed the crop, is the condition on which the promise of agrivoltaics now depends. Hashtags: #Agrivoltaics #AgriVoltaicsTechnology #SolarAgriculture #SolarFarming #AgriculturalInnovation #RenewableEnergy #SolarEnergy #PhotovoltaicSystems #CropProduction #SustainableAgriculture #SmartFarming #PrecisionAgriculture #DualLandUse #LandUseEfficiency #SolarInfrastructure #SolarTrackingSystems #DynamicAgrivoltaics #SolarPanelOptimization #SmartSolarTrackers #CropYieldOptimization #Photosynthesis #SolarRadiationManagement #ShadeManagement #WaterUseEfficiency #ClimateResilientAgriculture #AgriTech #ModelPredictiveControl #SustainableFoodProduction #EnergyFoodNexus #CleanEnergyInnovation

Download the Book (PDF): Introduction It is eleven at night in the second week of finals. A student has been at her laptop since dinner, working through lecture slides, a shared document of practice questions and a PDF of a textbook chapter that refuses to render at any size larger than a postage stamp. Her eyes burn. The letters swim a little when she looks up at the clock on the far wall, then settle. There is a dull pressure across her forehead, and her neck feels as if it has been set in plaster. She rubs her eyes, which helps for about ten seconds, and wonders, not for the first time, whether she is damaging her sight. A friend swears by the blue-tinted glasses she bought online. A post she scrolled past earlier said screens are giving a whole generation irreversible eye disease. Another said it is all fine and the 20-20-20 rule fixes everything. She has no idea which of these is true, and at eleven at night in finals week she does not have the time to find out. This book is written for her, and for the many thousands of students in the same position. It sets out what is actually happening to your eyes during long study sessions, what the research does and does not show, and what you can do about it with the equipment you already own. It is not a book of alarms. Nor is it a book of reassurance for its own sake. It takes a position, and the position is this: the discomfort most students feel after hours at a screen is a problem of how the eyes are being used, not of what the screen is emitting. Hold your focus at one near distance for hours, blink less and less completely because you are concentrating, sit with the screen set too high and too close in a room with dry, moving air, and your eyes will complain whether you are reading pixels or paper. Change those conditions and most of the complaint goes away. The fixes are behavioural and environmental. They are cheap or free. The products marketed most heavily to worried students, blue-light filtering glasses above all, are largely beside the point. That position has a second half, which matters just as much. Most eye strain is temporary and does no lasting harm. But two things that travel alongside heavy study can matter over the long term. The first is myopia, or short-sightedness, which is becoming more common worldwide and which, in some students, continues to progress into the early twenties under a heavy load of close work and little time outdoors. The second is the health of the ocular surface, the thin living layer of tears and tissue at the front of the eye, which can move from occasional dryness into a chronic condition, and which contact lens wearers can put at genuine risk through a few specific habits. Knowing where the real risks lie is what lets you stop worrying about the imaginary ones. What this book covers The chapters follow the logic of the argument. Chapter 1 defines digital eye strain properly, separates the two main families of symptoms it produces, and explains why it is so common and so rarely serious. Chapter 2 turns to the blink, the small, overlooked reflex that keeps the front of the eye wet, and shows why concentration, rather than the screen as such, is what suppresses it. Chapter 3 takes the most famous piece of advice in this field, the 20-20-20 rule, and asks what it is supposed to do, what the evidence says it does, and how to make it work in a real study session rather than a poster. Chapter 4 builds a desk around your eyes: distance, height, text size, lighting, glare and the particular problems of laptops, tablets and phones in bed. Chapter 5 deals with blue light, where public anxiety and commercial marketing have run far ahead of the evidence. It separates the question of eye damage, where the evidence is reassuring, from the question of sleep, where there is a real but more modest effect, and it explains what systematic reviews have found about filtering lenses. Chapter 6 looks at myopia: what is known about near work, what is known about daylight, and what is and is not known about whether the heavy reading of university years can still shift your prescription. Chapter 7 is a practical guide to keeping the ocular surface healthy, including artificial tears, warm compresses, the environment you study in and the rules that keep contact lens wear safe. Chapter 8 explains when a symptom is not strain at all but a sign that you need an examination, sometimes urgently, and what an eye examination can find that no amount of self-help will fix. The conclusion draws out what follows from all of this for how you study, and what remains genuinely unsettled. Along the way you will find a small number of tables where a comparison is easier to read in rows than in prose, and a set of notes at the end identifying the studies discussed. Where the book describes a student's situation, the student is an illustrative composite rather than a real person. Where it cites a figure, the figure comes from a named source. How to read the evidence Research on eye strain has some peculiar weaknesses, and it helps to know them in advance. Symptoms such as burning, tiredness and blur are subjective and fluctuate from hour to hour. Many studies are small, some run only for a single session, and some are sponsored by companies selling the product being tested. Surveys of digital eye strain have reported prevalence figures ranging from a modest minority of users to nearly everyone, partly because there has been no agreed definition of the condition until recently. A major international review published in 2023 by the Tear Film and Ocular Surface Society (TFOS) noted exactly this problem, and much of what this book says about the state of the evidence leans on that review and on a 2023 Cochrane systematic review of blue-light filtering lenses. The consequence is that you will often read, in these pages, that something "may help" or that the evidence is "limited". That is not hedging for its own sake. It is an accurate description of a field in which the simplest interventions have been tested less rigorously than one would like. Where the evidence is strong, as with the effect of concentration on blinking, the benefit of correcting an uncorrected prescription, the protective association of outdoor time with myopia in children, or the risk of sleeping in contact lenses, the book says so plainly. Where it is weak, the book tells you what a sensible person would do in the meantime, and why. There is also a quieter point about evidence that runs through the whole book. Many of the most useful changes you can make are ones that no trial is likely ever to test, because they are too obvious or too cheap for anyone to fund: moving a laptop onto a stack of books, turning down the brightness of a screen that is glaring against a dark room, pointing the desk fan away from your face. These changes rest on well-understood physiology rather than on clinical trials, and they cost nothing to try. When a change is free, reversible and grounded in how the eye works, the absence of a randomised trial is not a reason to ignore it. When a change costs money and is sold on claims that the trials do not support, it is. A note before you begin This book is educational. It is not a substitute for an examination by an optometrist or ophthalmologist, and it cannot diagnose the cause of your particular symptoms. If your eyes are painful rather than merely tired, if your vision changes suddenly, or if symptoms persist despite the sensible changes described here, you should be seen. Chapter 8 sets out the warning signs in detail. Readers in the United States will find occasional remarks about how eye care is organised and paid for there; arrangements differ considerably from country to country and between institutions, and your campus health service is usually the best place to find out what applies to you. The student at her laptop at eleven at night does not need a new gadget. She needs to understand what her eyes are telling her, to change three or four things about how she works, and to know which symptoms are worth a visit to a clinic. That is what the following chapters set out to give her. Chapter 1: What Eye Strain Is, and What It Is Not "Eye strain" is one of those phrases everyone uses and nobody defines. Students use it for burning eyes, for blurred vision at the end of the day, for headaches, for the gritty feeling that arrives around the third hour of reading, and sometimes for a vague sense that their eyes are being worn out like the tread on a tyre. Some of those experiences have a clear physical basis. One of them, the last, is almost entirely mistaken. Getting the definition right is the first step towards doing something useful, because the different complaints bundled under the one phrase have different causes and different remedies. For many years the condition went under the name "computer vision syndrome", a term the American Optometric Association popularised when desktop computers spread through offices. As phones, tablets and laptops replaced the office terminal, the term "digital eye strain" came to be preferred. In 2023 the Tear Film and Ocular Surface Society, in a large international review of how the digital environment affects the front of the eye, proposed a formal definition: digital eye strain is "the development or exacerbation of recurrent ocular symptoms and/or signs related specifically to digital device screen viewing". Two features of that definition deserve attention. The first is the word recurrent. Digital eye strain is a pattern that comes back with the activity, not a single bad afternoon. The second is the phrase development or exacerbation. Screen use can produce symptoms in a healthy eye, but it can also make an existing problem, such as mild dry eye or a small uncorrected prescription, noticeable for the first time. A good deal of what students experience as screen-induced strain is in fact a pre-existing issue that the screen has exposed. That distinction matters later, when we come to the question of whether you need an examination. The same review was frank about how poorly the condition has been measured. Published prevalence estimates range from around a quarter of users to as high as 97 per cent. That spread tells you less about the world than about the research: studies have used different questionnaires, different thresholds and different populations, and many questionnaires cannot distinguish symptoms caused by screens from symptoms that would appear during any demanding close task. A 2018 review by Amy Sheppard and James Wolffsohn, two optometry researchers at Aston University, suggested that the prevalence among computer users may be 50 per cent or more. The reasonable summary is that some degree of eye discomfort during heavy screen use is very common, perhaps the norm, and that the exact figure depends on how strictly one defines it. Two families of symptoms The single most useful thing to understand about eye strain is that its symptoms fall into two broad groups, which Sheppard and Wolffsohn describe as internal and external. They have different mechanisms and they respond to different fixes, so it is worth learning to tell them apart in your own experience. Table 1 sets out the main symptoms, the mechanism most often responsible, and where in this book the remedy is discussed. Table 1. The two families of digital eye strain symptoms. Symptom family Typical sensations Main mechanism Where addressed External (ocular surface) Burning, grittiness, stinging, watering, redness Reduced and incomplete blinking, tear evaporation, dry or moving air Chapters 2 and 7 Internal (focusing and alignment) Tired or aching eyes, blur that clears slowly when looking up, headache around the brow Sustained near focusing and convergence, uncorrected prescription, binocular vision problems Chapters 3 and 8 Postural (not strictly ocular) Neck, shoulder and upper back ache, headache at the base of the skull Screen height and distance forcing a fixed head position Chapter 4 Sleep-related Difficulty falling asleep after late screen use Evening light and mental arousal Chapter 5 The external group concerns the surface of the eye. The cornea, the clear dome at the front, is one of the most densely innervated tissues in the body. It stays healthy only when it is covered by a stable film of tears, renewed by blinking every few seconds. When that film thins and breaks up, the nerve endings beneath register it as burning, grittiness or a feeling that something is in the eye. Paradoxically, a drying eye often waters: the irritation triggers reflex tearing, which floods the eye with watery tears that do not contain the oily layer needed to keep them from evaporating. A student who complains that her eyes are "both dry and watery" is describing a coherent physiological sequence, not a contradiction. The internal group concerns the machinery of focusing and pointing the eyes. To see something close, two things have to happen at once. Inside each eye, a ring of muscle called the ciliary muscle contracts, allowing the lens to become more curved so that near objects come into focus. This is called accommodation. At the same time, the muscles outside the eyes rotate both eyes inwards so that they point at the same near target. This is convergence, or more generally vergence. The two are linked by the nervous system, so that when one changes the other tends to follow. The demand these systems face can be put in numbers. Optometrists measure focusing effort in dioptres, which are simply the reciprocal of the viewing distance in metres. A screen at 50 centimetres asks the eye for 2 dioptres of accommodation; a phone held at 25 centimetres asks for 4; a page held at 20 centimetres asks for 5. Anything beyond about six metres, or twenty feet, asks for almost none, which is why eye-care professionals treat that distance as optically equivalent to infinity and why it appears in the rule discussed in Chapter 3. The range of focusing a person has available, called the amplitude of accommodation, is large in childhood and shrinks steadily through life, which is why people in their mid-forties begin holding menus at arm's length. A typical university student still has plenty in reserve. The general principle of comfortable work, though, is that one should not sustain more than a fraction of one's available focusing effort for long periods, and a phone held a hand's width from the face on a late-night scroll approaches the upper end of what is comfortable to hold for hours. Neither accommodation nor convergence is especially effortful in a young, healthy visual system, but both are active states. Holding them for hours at a fixed distance, particularly if the eyes also have to work around a small uncorrected prescription or a mild tendency to drift apart at near, produces the aching, tired sensation that people describe as their eyes being "strained", and sometimes a slow-to-clear blur when looking up at something distant. The third and fourth rows of Table 1 are not strictly ocular, but students experience them as part of the same problem and they are caused by the same study habits. Neck and shoulder pain from a laptop screen set too low, or headaches from a head tilted back to see through the bottom of a pair of glasses, are among the most common complaints of heavy screen users. Difficulty falling asleep after a late session is real, but as Chapter 5 explains, it has little to do with eye strain and a good deal to do with light timing and an alert mind. Why screens make it worse, and why they are not the whole story If eye strain were simply caused by screens, one would expect reading a printed book for the same length of time to be harmless. It is not. Students who spent the days before laptops reading heavy textbooks under a desk lamp also complained of tired eyes and headaches, and the physiology of accommodation and convergence is the same whether the target is ink or pixels. A careful study by Mark Rosenfield and colleagues at the State University of New York College of Optometry, which is discussed further in the next chapter, found that when a screen and a printed page were matched for text size, contrast, distance and viewing angle, blink rate did not differ significantly between them. What changed blinking was the demanding reading task itself. Still, there are reasons why screen work tends to be harder on the eyes than the kind of reading most people did in the past. The first is duration. Screens make it possible to read, write, watch and message for hours without a natural pause. Nobody used to read a textbook for six hours without getting up to fetch another book, find a pen or turn on a light; a laptop removes most of those interruptions. The second is geometry. Printed pages are usually held low, in the lap or flat on a desk, so that the eyes look downwards and the upper lids cover more of the eye's surface. Desktop monitors are often set at or above eye level, which exposes more of the eye to the air. The third is visual quality. Small text, low contrast, glare from windows and reflective screens, and cramped layouts can all make the eyes work harder than a well-printed page would. The fourth is the number of screens. Many students now work across a laptop, a second monitor and a phone at different distances, which asks the focusing system to change repeatedly, and they often end the day by reading a phone held very close in a dark room. None of these features is intrinsic to screens as such. Each is a feature of how screens tend to be used. This is the central point of the book, and it is worth holding on to: almost every factor that makes screen work uncomfortable can be changed without giving up the screen. What eye strain is not It is just as important to be clear about what eye strain is not, because the worry it generates is often out of proportion to the harm. Eye strain is not, on present evidence, a cause of permanent damage to a healthy adult eye. The symptoms of digital eye strain are typically transient. The ciliary muscle does not wear out from use in the way a joint wears. The tear film recovers when you blink fully and rest. After a night's sleep, a young healthy eye that was burning and tired at midnight is usually back to normal. This is why eye strain is best understood as a signal about working conditions rather than as an injury in progress. It is the eye's equivalent of the stiffness you feel after sitting for too long: unpleasant, worth correcting, and a sign that something about the arrangement should change, but not in itself evidence of lasting harm. Eye strain is not caused by screens "burning" the retina with blue light. This claim circulates widely, often attached to the sale of filtering lenses, and it is addressed in detail in Chapter 5. The short version is that the amount of blue light emitted by phones, tablets and computers is a small fraction of what you receive from daylight, and there is no good evidence that it harms the retina at typical viewing distances and durations. Eye strain is not the same as myopia. Short-sightedness is a structural change in the eye, usually an elongation of the eyeball, which means distant objects focus in front of the retina. It develops mostly in childhood and adolescence, and its relationship to close work is real but complicated. A tired, aching eye after a long evening does not mean myopia is developing, and the absence of strain does not mean it is not. Chapter 6 treats myopia separately for precisely this reason. And eye strain is not always eye strain. A small number of students attribute to screens symptoms that have another cause entirely: headaches from migraine, tension or dehydration; blurred vision from an out-of-date prescription; red, painful eyes from infection or inflammation. A useful rule is that eye strain gets better with rest. If a symptom does not improve after a night's sleep and a day with less screen work, or if it is severe, one-sided, sudden or accompanied by changes in how you see, it is not a problem this book can solve, and Chapter 8 explains what to do. A student's audit, and why it matters Consider a second-year engineering student who spends most of his waking hours at a laptop. He notices that by late afternoon his eyes feel hot and gritty, that he gets a headache across his forehead most evenings, and that when he looks up from his screen at the whiteboard in a lab session, it takes a moment for the writing to sharpen. He assumes the screen is the culprit and considers buying filtering glasses. A more useful approach is to sort his symptoms by family. The heat and grittiness are external symptoms, most likely from reduced blinking and a dry environment; the laptop sits on a desk directly beneath an air-conditioning vent. The forehead headache and the slow refocusing are internal symptoms, suggesting sustained near focusing, possibly aggravated by a prescription that has not been checked since he was fifteen. His neck ache, which he had not thought to mention, comes from looking down at a laptop screen set flat on the desk. Each of these has a different remedy: blinking habits and moving away from the vent for the first, breaks and an eye examination for the second, a stand and an external keyboard for the third. Filtering glasses would address none of them. This kind of audit takes five minutes and is the most practical thing you can do before reading further. Note which symptoms you get, at what point in a session they start, whether they differ between laptop, phone and paper, and whether they clear with sleep. Note too where you usually study: the air, the light, the height of the screen, how far away it is. The rest of this book is organised to help you act on what you find. If eye strain is usually temporary and harmless, a reasonable student might ask why it deserves a book. There are three answers. The first is that discomfort costs attention. Burning eyes and a mild headache in the third hour of revision are not dangerous, but they are distracting, and they tend to end study sessions earlier or make them less productive. Students who learn to prevent the discomfort often find that they can work longer and more comfortably with less effort, which is the whole point of studying in the first place. The second is that eye strain can be a symptom of something worth correcting. A significant fraction of students with persistent strain turn out to have an uncorrected refractive error, a binocular vision problem such as convergence insufficiency, or early dry eye disease. Each of these is treatable, and each tends to be missed because the student has attributed the symptoms to screens in general. The TFOS review was clear that the first step in managing digital eye strain is a full refractive correction for the distances at which a person works, which requires an examination. The third is that the habits that prevent strain overlap considerably with the habits that protect the eye over the longer term. Breaking up long stretches of close focus, spending some daylight hours outdoors, keeping the ocular surface healthy and handling contact lenses safely do more than make an evening of study more comfortable. They bear on the two genuine long-term concerns this book identifies, myopia progression and chronic ocular surface disease. Learning to read your eyes' signals is a way of building those habits while they are still easy to form. The next chapter begins with the most overlooked of those signals, and the most important reflex in the story: the blink. Chapter 2: The Blink You blink thousands of times a day and notice almost none of them. Each blink lasts a fraction of a second, too brief to interrupt vision, and is controlled by circuits that run without conscious effort. It is easy to think of blinking as a kind of background noise. It is anything but. The blink is the mechanism that keeps the front of the eye alive, and changes in how often and how completely you blink are the single largest cause of the burning, gritty discomfort that students associate with long hours at a screen. Understanding it takes a little physiology, but it repays the effort, because once you understand why the blink fails during study you can see which remedies make sense and which do not. The tear film and why it needs renewing The cornea has no blood vessels. It has to be transparent, and blood vessels would get in the way of light. Instead it takes much of its oxygen from the air and relies on a thin layer of fluid, the tear film, to keep it moist, smooth and protected. The tear film is only a few micrometres thick, far thinner than a human hair, and yet it does several jobs at once. It provides the smooth optical surface that makes clear vision possible; a dry, irregular cornea scatters light and blurs the image. It supplies nutrients and carries away debris. It contains antimicrobial proteins that help defend the eye against infection. And it lubricates the passage of the eyelids over the eye, thousands of times a day. The film is traditionally described in layers, although current thinking, reflected in the 2017 report of the Tear Film and Ocular Surface Society's Dry Eye Workshop (known as TFOS DEWS II), treats it as a more integrated structure. Closest to the eye is a watery gel rich in mucins, sticky proteins that help the tears cling to the surface. Most of the volume is water, produced mainly by the lacrimal glands above and to the side of each eye. On the outside sits a very thin layer of oils, secreted by the meibomian glands, a row of small glands embedded along the edge of each eyelid. That oily layer matters enormously. It slows the evaporation of the water beneath it and helps the film spread evenly. When the meibomian glands are blocked or produce poor-quality oil, the tears evaporate quickly even if plenty are being produced, and this evaporative form of dryness is by far the most common type of dry eye. A tear film does not last. Between blinks it thins by evaporation and drainage, and eventually it breaks up, forming dry spots on the cornea. In a healthy eye, the time from a blink to the first break-up is typically longer than the normal interval between blinks, so the film is renewed before it fails. Clinicians measure this as the tear break-up time; in people with dry eye it is shortened. Each blink does three things: it spreads a fresh film across the surface, it squeezes a small amount of oil out of the meibomian glands onto the lid margin, and it pumps used tears towards the drainage ducts at the inner corner of the eye. A blink is therefore not just a wipe. It is a coordinated act of resurfacing. This is why the interval between blinks matters so much. If you blink every three or four seconds, the film is refreshed long before it breaks up. If you stare for fifteen or twenty seconds at a time, as people often do when absorbed, the film may break up before the next blink arrives, exposing parts of the cornea to the air. The nerves register this as discomfort. Repeat the cycle over several hours and the surface becomes irritated, and irritation, over time, can itself make the tear film less stable. Contact lenses complicate this picture. A soft lens sits within the tear film and effectively splits it into two thinner layers, one in front of the lens and one behind. The layer in front of the lens is thinner and less stable than a normal tear film, and it tends to break up sooner after each blink. This is one reason contact lens wearers so often report that their eyes feel dry late in the day, or that their lenses become uncomfortable during long sessions at a screen when they would have been fine during a day of lectures and conversation. The lens is not doing anything wrong; it simply narrows the margin for error. A contact lens wearer who stares, blinks incompletely and studies in dry air is running the tear film closer to its limits than someone wearing glasses, and it is common for wearers to find that switching to glasses for long evening study sessions solves most of the problem at once. Chapter 7 discusses contact lens habits in more detail. What concentration does to blinking Blinking is partly spontaneous and partly modulated by what you are doing. At rest, during conversation, people blink often; figures in the region of fifteen to twenty times a minute are commonly quoted. When attention narrows onto a visual task, the rate falls. This makes a certain sense: every blink blanks out vision for a moment, and the brain appears to suppress blinks when missing a fraction of a second of visual input might matter. Studies of people watching films have found that blinks tend to cluster at natural pauses, such as the end of a scene, rather than during moments of action, as if the brain is timing them to minimise what is lost. The best-known early report on screens came from the Japanese ophthalmologist Kazuo Tsubota and his colleague Kouhei Nakamori, who in a short 1993 letter to the New England Journal of Medicine described a sharp fall in blink rate when people worked at video display terminals compared with when they were relaxed, and a further reduction compared with reading a book. They also noted that people looking at a screen set at eye level had a wider eye opening than those looking down at a page, exposing more of the eye's surface. Their observations helped establish the idea that computer work dries the eye, and it has been repeated in many forms since. What those early studies did not do well was compare like with like. A screen at eye level with small text is a very different visual task from a book in the lap. In 2014 Christina Chu, Mark Rosenfield and Joan Portello, at the State University of New York College of Optometry, set out to fix this. They asked 25 people to read the same text for twenty minutes, once from a desktop monitor and once from a printed page, with the size, contrast, distance and viewing angle matched as closely as possible. The average blink rate was 14.9 blinks a minute on the screen and 13.6 on paper, a difference that was not statistically significant. Their conclusion was that the reduced blinking seen in earlier studies was more likely produced by the cognitive demands of the task than by the screen itself. The kind of task matters too. Anything that demands continuous fixation and rapid visual decisions, such as reading dense text, checking lines of code for an error, playing a fast-paced game or working through a spreadsheet, tends to depress blinking more than relaxed browsing or watching a lecture video in which nothing needs to be caught. Students often notice that their eyes are worse after an evening of problem sets than after an evening of the same number of hours on streaming video, and this is the likely reason. It also explains why the most intense study periods, the week before an exam or the night before a deadline, produce the worst symptoms: these are the periods of greatest sustained concentration, often combined with the longest sessions and the least sleep. This finding sits at the heart of the argument of this book. Blinking falls when you concentrate hard on something close. Screens make it easy to concentrate hard on something close for a very long time, often with the display positioned in a way that widens the eye, and often in dry, air-conditioned rooms. The screen is the setting, not the mechanism. A student reading a dense printed article with total absorption for two hours under a desk lamp beside a heater will experience much the same thing. The incomplete blink The same group at SUNY noticed something else, which has turned out to be at least as important as the rate. Not every blink is a full blink. In an incomplete blink the upper lid comes down only part of the way, leaving the lower part of the cornea uncovered and unrefreshed. The lower cornea is exactly where dry spots and surface damage tend to show up in people with screen-related dryness, which fits. In a 2013 study, Portello, Rosenfield and Chu filmed 21 people reading from a desktop computer for fifteen minutes. The average blink rate was 11.6 blinks a minute, but the proportion of blinks that were incomplete varied enormously between individuals, from under 1 per cent to more than half, with an average of about 16 per cent. When the researchers compared this with how uncomfortable each person's eyes felt after the task, the percentage of incomplete blinks was significantly related to symptoms. Then they tried the obvious fix. In a second session an audible tone sounded every four seconds and participants were told to blink when they heard it. The average blink rate roughly doubled, to 23.5 a minute. Symptoms did not change significantly. That result is worth dwelling on, because it cuts against a piece of advice students hear constantly: "remember to blink more." If you blink more often but still blink incompletely, the lower part of the eye may be no better off. The authors suggested that encouraging complete blinks, which fully cover the cornea, may be more useful than simply encouraging more of them. The 2014 comparison added a further detail: while blink rate did not differ between screen and paper, the proportion of incomplete blinks was modestly but significantly higher on the screen, at around 7 per cent against 4 per cent. Screens may not reduce how often you blink when the task is matched, but they may make each blink a little lazier. It is important not to overstate what these small studies show. They involved a few dozen people over short periods, and incomplete blinking is not the only factor in dry, uncomfortable eyes. But the pattern is consistent with the physiology, and it suggests a sensible priority: quality of blinking first, quantity second. Practising a better blink If the problem is partly that concentration makes blinks rarer and shallower, the solution cannot be to stop concentrating. What can be changed is the rhythm of work and the conditions around it, together with a few habits that restore full blinks at intervals. The simplest habit is a deliberate complete blink. Close your eyes gently but fully, so that the upper and lower lids meet, pause for a second or two, and open them again. Do not screw the eyes shut; a gentle full closure is enough. Repeating this a few times in a row, at points where you naturally pause, spreads a fresh film over the whole surface and expresses a little oil from the lid glands. Some optometrists suggest a routine of several slow, complete blinks at intervals through a session, and some small studies of blink training have reported improvements in comfort and in measures of the tear film, although the evidence is preliminary and the protocols vary. As a habit it is free, takes seconds, and follows directly from how the tear film works. The difficulty, as anyone who has tried to "remember to blink" knows, is remembering. Deep concentration is exactly the state in which you forget. It helps to attach full blinks to events that already happen: every time you finish a page, submit a quiz answer, switch between windows, or look up from the screen to check the time. The break routine discussed in the next chapter provides a natural trigger. Some students put a small note at the edge of the monitor; others use one of the many break-reminder apps. There is no need to become obsessive about it. The aim is simply to punctuate long stretches of staring with occasional full resets. The second lever is the position of the screen. When you look down, the upper lid naturally lowers and covers more of the eye, reducing the exposed area and the rate of evaporation. When you look straight ahead or upwards, the eye opens wider. This is one reason printed books held low tend to be more comfortable than monitors set high, and it is one of the main reasons the setup guidance in Chapter 4 places the centre of the screen somewhat below eye level. Changing the height of a monitor is one of the few interventions that reduces the drying load continuously, without any need to remember to do anything. The third lever is the air. Evaporation from the eye's surface increases in low humidity and in moving air. A desk fan blowing across the face, a heating or air-conditioning vent above the desk, a car heater on the drive to campus, or the dry air of an aircraft cabin all accelerate the break-up of the tear film. Moving the desk, redirecting the vent or turning the fan away can make as much difference as anything else in this chapter. Chapter 7 returns to the environment in more detail. The fourth lever is time. Every one of these mechanisms accumulates. A tear film that is marginal over fifteen minutes becomes a real problem over three hours. Breaking a long session into shorter segments with brief rests, during which the eyes are closed or looking at something distant while you blink naturally, allows the surface to recover before discomfort sets in. This is the practical logic of the 20-20-20 rule, which is the subject of the next chapter, although as we will see its evidence base is thinner and more interesting than its popularity suggests. What the blink tells you There is one more reason to pay attention to blinking, which is that it offers early information about the state of your eyes. If you notice that your eyes feel more comfortable when you close them for a few seconds, that the burning builds steadily across a session and eases after a break, and that it is worse in dry or windy settings, the problem is very likely evaporative: a tear film that is not being renewed often or fully enough in an environment that dries it quickly. Those symptoms respond to the measures in this chapter and in Chapter 7. If, on the other hand, your eyes are uncomfortable first thing in the morning before any screen work, if the edges of your eyelids are red, crusted or swollen, if one eye is much worse than the other, or if the discomfort is genuine pain rather than irritation, blinking habits are unlikely to be the whole story. Conditions such as blepharitis, meibomian gland dysfunction, allergy and infection all produce overlapping symptoms and need proper assessment. Chapter 8 sets out which of these patterns warrant a visit to an optometrist and which warrant urgent care. Consider a first-year student who notices that her eyes burn every afternoon in the library but rarely in her room. She has a desk directly beneath an air-conditioning outlet, uses a monitor raised on a stand so that she looks slightly upwards, and works in long stints without pausing. None of these is a screen problem as such. Lowering the monitor so that its top sits at or just below eye level, moving to a desk away from the vent, and pausing every twenty minutes or so for a few full blinks and a look out of the window will very likely make the burning much less frequent. If it does not, that too is useful information: it points to something about her eyes, rather than her environment, that deserves a closer look. The blink, in short, is where the argument of this book begins. The screen does not dry your eyes. Staring does, especially staring upward into moving, dry air, hour after hour. Staring is a habit, and habits can be changed. Chapter 3: The 20-20-20 Rule Under Scrutiny If you have heard one piece of advice about screens and eyes, it is probably this: every 20 minutes, look at something 20 feet away for 20 seconds. The 20-20-20 rule appears on university health service pages, on the websites of professional optometry bodies, in workplace safety leaflets and in countless apps. It is commonly attributed to Jeffrey Anshel, a Californian optometrist who wrote about visual ergonomics at the computer from the 1990s onward. It is easy to remember, costs nothing, and sounds scientific because it is built from numbers. It deserves a closer look, both because it is so widely recommended and because the reasons it might work are more interesting than the rule itself. The numbers are a mnemonic, not a measurement. Understanding what each part of the rule is trying to achieve lets you apply it intelligently, adapt it to the way you actually study, and avoid the most common way of undermining it. What each number is for Take the numbers one at a time. The 20 feet, or about six metres, is the distance beyond which the eye needs essentially no focusing effort. Chapter 1 explained that the demand on accommodation is the reciprocal of the viewing distance in metres: two dioptres for a screen at half a metre, four for a phone at a quarter of a metre, and about one-sixth of a dioptre at six metres, which is negligible. Looking at something twenty feet away therefore lets the ciliary muscle relax almost completely. It also lets the eyes swing back from their converged, inward-pointing position to a nearly parallel one. There is nothing magic about exactly twenty feet. Anything across the room or out of the window will do, and the further the better. What matters is that the target is far, not near. The 20 seconds is the duration of that look. The focusing system responds quickly; accommodation typically changes within a second or so of shifting gaze from near to far. So twenty seconds is not required for the lens to refocus. What twenty seconds does buy is time for several natural blinks, for the eyes to rest in a less effortful position, and for any lingering blur after a long period of close focus to clear. Researchers have long described a small, temporary shift towards short-sightedness after sustained near work, which they call nearwork-induced transient myopia. In most people it decays within seconds to a minute or two, and it is the reason the whiteboard may look slightly soft for a moment when you look up after a long stretch of reading. A twenty-second look into the distance is long enough to let that settle. The 20 minutes is the interval. Here the rule is least grounded in any particular physiology. It is a practical compromise: frequent enough to interrupt the accumulation of dryness and focusing fatigue that builds over a session, but not so frequent as to shatter concentration. Nobody has shown that twenty minutes is better than fifteen or thirty. What the evidence on rest breaks does show, as we will see, is that frequent short breaks are better for comfort than long unbroken stretches with only occasional long breaks. Put together, the rule is really a compact way of saying three things. Do not hold one near focus for long periods. When you pause, look far away rather than at another near object. And pause often enough that discomfort never has time to build. What the evidence shows Given how universally the rule is recommended, it is striking how little direct testing it has received. For many years its support rested mainly on physiological reasoning and on studies of rest breaks in general. The strongest general evidence comes from occupational research on office workers. In a field study published in the journal Ergonomics in 2000, Traci Galinsky and colleagues at the US National Institute for Occupational Safety and Health compared two schedules for 42 data-entry workers. On a conventional schedule, workers took two fifteen-minute breaks in a shift. On a supplementary schedule, they took the same two breaks plus a five-minute break in each hour that otherwise had none. Eyestrain and discomfort in several parts of the body were significantly lower on the supplementary schedule, and the extra breaks did not reduce data-entry performance. A follow-up study published in 2007 with 51 workers found the same pattern: supplementary breaks reduced discomfort and eyestrain, and typing speed actually rose enough during work periods that total output was maintained despite twenty minutes less time at the keyboard. Those studies were not testing the 20-20-20 rule itself, and the breaks were minutes long rather than seconds, but they support its underlying principle: short, frequent interruptions reduce eye discomfort without costing productivity. The first direct test of the rule came from a team led by Cristian Talens-Estarelles at the University of Valencia, working with James Wolffsohn and Amy Sheppard at Aston University, published in 2022 in Contact Lens and Anterior Eye. They recruited 29 computer users who already had symptoms of digital eye strain and installed software on their laptops that used the webcam to monitor how long they worked, where they looked and how they blinked. The software then issued personalised reminders to take breaks according to the 20-20-20 rule. The researchers measured symptoms and a long list of clinical signs before the reminders began, after two weeks of using them, and one week after they stopped. The results were instructive in both directions. The reminders changed behaviour: participants took more breaks per day and worked for shorter stretches. Symptoms of digital eye strain and of dry eye improved significantly while the reminders were in use. But when the reminders were switched off, the improvement faded, and a week later symptoms were no longer significantly different from where they began. The clinical measures of the tear film and ocular surface, such as tear break-up time and staining of the cornea, did not change over the two weeks, and nearly all measures of binocular vision stayed the same, with one exception, a small improvement in how quickly the eyes could change focus. The fair reading of this study is that the 20-20-20 rule, when actually followed, relieves symptoms; that people do not keep following it once they stop being reminded; and that two weeks is not long enough to see changes in the underlying health of the ocular surface, if any would occur. It is a small study without a control group given sham reminders, so it cannot rule out some placebo effect. The 2023 TFOS review of the digital environment reached a similar overall judgement about management of digital eye strain: interventions are not well established, but encouraging regular breaks, better blinking and a better working environment may help. A related question is whether more elaborate eye exercises add anything. The internet is full of them: rolling the eyes in circles, "palming" the eyes with warm hands, tracing figure-eights, focusing alternately on a thumb and a distant object. Some of these descend from the Bates method, a system promoted in the early twentieth century by the American ophthalmologist William Bates, who claimed that exercises could cure short-sightedness and remove the need for glasses. There is no good evidence that any such routine changes a person's refractive error, and eye-care bodies have long rejected the claim. Gentle exercises that involve looking into the distance, closing the eyes or relaxing may well feel pleasant during a break, and there is no harm in them if they do, but their benefit comes from the same interruption of sustained near focus that the 20-20-20 rule provides, not from any special property of the movements. This is quite different from vision therapy prescribed by an optometrist or orthoptist for a diagnosed binocular vision problem such as convergence insufficiency, where structured, supervised training has been tested in clinical trials, mainly in children and adolescents, and can be effective. The distinction is between a targeted treatment for a specific condition and a general routine sold as a cure for everything. The honest summary, then, is that the 20-20-20 rule is sensible, plausible and supported by indirect evidence and one small direct trial, but it is not a proven treatment in the sense that a drug is proven. Its main weakness is not that it fails to work but that people stop doing it. Making it work in a real study session The central practical problem with the 20-20-20 rule is adherence. It asks you to interrupt yourself three times an hour at exactly the moments when you are most absorbed, and the Valencia study suggests that most people stop as soon as nothing forces them to continue. The solution is to design the habit around the way you already study, rather than relying on willpower. Use a timer, but choose a rhythm you will keep. For students who already work in timed blocks, such as the widely used Pomodoro pattern of twenty-five minutes of work and a five-minute break, the break structure is already there and needs only a small modification: at the end of each block, before doing anything else, look out of a window or across the room for twenty seconds or more and blink fully a few times. For students who prefer longer, deeper stretches of focus, a gentle reminder every twenty to thirty minutes, set to prompt a distant look without requiring you to leave the task, may be more sustainable. Many operating systems and browsers offer break reminders, and there are free applications built around the 20-20-20 pattern. The best reminder is the one you will not switch off on the second day. Look at something genuinely far away. The most common way to undermine the rule is to "take a break" by picking up your phone. A phone held at thirty centimetres demands more focusing effort than a laptop, not less, and it holds your eyes in exactly the fixed near position the break was meant to relieve. The same applies to glancing at a second monitor or at notes on the desk. The point of the break is distance. A window is ideal; daylight has other benefits discussed in Chapter 6. Failing that, the far wall of the room, a doorway, or the end of a library corridor will do. Combine it with full blinks and a posture change. The look into the distance is the natural moment for a few slow, complete blinks, as described in Chapter 2. It is also a good moment to roll the shoulders, sit back, or stand. The muscular aches that accompany eye strain respond to the same interruption, and a break that relieves your neck as well as your eyes is more likely to feel worth taking. Take longer breaks too. The 20-20-20 rule is a micro-break. It does not replace real breaks. The occupational studies found benefits from breaks of several minutes each hour, on top of longer breaks through the day. A practical pattern for a long revision day might be a twenty-second distant look every twenty minutes or so, a five-to-ten-minute break away from any screen every hour, and a longer break, ideally outdoors, every two to three hours. A walk to get coffee, a short trip outside between library sessions, or a meal eaten away from the desk all count. Use the natural breaks in your work. Many study tasks have built-in pauses: the end of a chapter, the completion of a problem, the moment a video lecture switches slides, the upload of a submission. Attaching a distant look to these events can be more reliable than an arbitrary timer, because it fits the rhythm of the work rather than interrupting it. If you finish each problem in a problem set by looking out of the window for a few breaths, you will be taking breaks at roughly the right frequency without ever setting an alarm. To see how this fits together, consider a three-hour afternoon session in a university library, planned by a student preparing for a statistics exam. She chooses a desk near a window but facing along it rather than towards it, so the daylight falls from the side. She sets a timer for twenty-five-minute blocks. At the end of each block she looks out of the window at the trees on the far side of the quadrangle for a slow count of twenty, blinks fully three or four times, and then either returns to work or, every second block, stands, fills her water bottle and walks to the end of the floor and back. After ninety minutes she takes a proper fifteen-minute break outside, leaving her phone in her bag. By the end of three hours she has spent perhaps twenty-five minutes not looking at a screen, broken into small pieces. In exchange, she finishes the session without the burning eyes and forehead headache that used to end her library afternoons after two hours, and her total productive time is, if anything, higher. None of this is heroic. It is the ordinary structure of a well-run study session, with the eyes' needs built in. When the rule is not enough The 20-20-20 rule addresses the internal, focusing-related symptoms of eye strain and gives the tear film a chance to recover. It cannot fix causes that are always present. If you are working with an uncorrected or out-of-date prescription, your eyes are straining against a blur they cannot escape, and no number of breaks will remove it. If you have a binocular vision problem, such as convergence insufficiency, in which the eyes struggle to keep turning inward together for near work, breaks will relieve symptoms temporarily but they will return quickly with each session. If your eyes are dry because of meibomian gland dysfunction, the tear film will break up sooner than normal between every blink regardless of how often you pause. A useful sign that the rule is not enough is when symptoms come back very quickly after each break, or begin within the first ten or fifteen minutes of a session, rather than building slowly over hours. Another is when symptoms are much worse at near than at a distance, or when they include double vision, words that seem to move on the page, or a pulling sensation in the eyes. These are reasons to see an optometrist, as Chapter 8 explains, rather than to redouble your efforts with the timer. Consider a third-year history student who dutifully follows the 20-20-20 rule but still finds that her eyes ache and the text blurs within a quarter of an hour of opening a PDF. She has never needed glasses, because she can read the board in lectures without difficulty. An eye examination finds a small amount of long-sightedness and some astigmatism: not enough to affect her distance vision, which her young focusing system compensates for, but enough that sustained reading requires noticeably more effort than it should. With a modest reading prescription her symptoms fall away. The breaks were never the problem; they simply could not overcome a demand that was present for every second of every session. The 20-20-20 rule is best treated not as a clinical treatment but as a reminder of the three principles it encodes: vary your focus, look far when you rest, and do not let discomfort accumulate. Treated that way, it becomes flexible. On a day of light reading in a well-lit room with good air, you may need it less. On a day of intense coding in a dry, air-conditioned laboratory, you may need breaks more often than every twenty minutes. The numbers are a starting point for a habit, not a prescription. The rule also points towards the next question. If much of the discomfort of screen work comes from holding the eyes in a fixed near position, then the position itself matters: how far away the screen is, how high it sits, how large the text is and how the light falls across it. Every one of those factors changes the load on the eyes continuously, whether or not you remember to take a break. The next chapter builds a study setup around them. Chapter 4: Building the Desk Around Your Eyes Breaks and blinks are things you have to remember to do. The arrangement of your desk is different: it acts on your eyes every second you sit there, whether you are paying attention or not. A screen that is too close, too high, too small or too bright for its surroundings imposes a steady extra load on the focusing system and the ocular surface for the whole of a session. Correcting it takes one afternoon and a few books or a cheap stand, and the benefit is continuous. This is the most efficient change most students can make, and it is also the one most often neglected, because the typical student workspace is a laptop on a bed, a sofa, a kitchen table or a library desk designed for books. This chapter works through the four features of a setup that matter most for the eyes, distance, height, light and text, and then deals with the particular problems of laptops, phones and tablets. Table 2 summarises the main recommendations and their sources. None of the numbers is precise to the centimetre. Each is a range within which most people are comfortable, and your own comfort within that range is the final test. Table 2. Recommended screen setup and where the guidance comes from. Feature Recommendation Source Viewing distance (monitor or laptop) About 50 to 70 cm (20 to 28 inches) from the eyes American Optometric Association Acceptable wider range About 50 to 100 cm (20 to 40 inches) US Occupational Safety and Health Administration Screen height Top line of text at or slightly below eye level; centre of screen about 15 to 20 degrees below horizontal gaze American Optometric Association; OSHA Screen orientation to windows Screen at right angles to windows, not facing or backing onto them General ergonomic guidance (OSHA and others) Screen brightness Similar to the brightness of the surroundings General ergonomic guidance Text size Comfortably larger than the smallest size you can just read Low-vision principle of "acuity reserve" Distance and text size Distance is the most direct lever on focusing effort. At 50 centimetres the eye needs two dioptres of accommodation; at 70 centimetres it needs about 1.4. The American Optometric Association recommends a viewing distance of roughly 20 to 28 inches, or about 50 to 70 centimetres, from the eyes to the screen. The US Occupational Safety and Health Administration, in its guidance on computer workstations, gives a broader acceptable range of about 20 to 40 inches. A quick check that needs no tape measure: sitting back in your chair, stretch your arm out towards the screen. For most adults, the fingertips should reach close to the screen's surface. If you can rest your palm flat on it, it is probably too close. People tend to drift closer to screens over a session, especially when text is small or when they are tired. This is one reason a good distance at the start of an afternoon can become a poor one by evening. It is also why distance and text size have to be considered together. If you move the screen back to seventy centimetres without enlarging the text, you will simply lean forward to read it, and you will be back where you started with a sore neck as well. The principle for text size comes from low-vision research, where clinicians speak of "acuity reserve": comfortable, fluent reading requires print that is comfortably larger than the smallest print a person can just make out, not text at the threshold of legibility. The same logic holds for people with normal vision. Text that you can only just read forces you to lean in, to squint, and to concentrate harder on the act of seeing, all of which increase strain and depress blinking. A practical test is to sit at your normal distance and ask whether you can read a paragraph of body text without effort, without leaning forward, and without any sense of peering. If not, enlarge it. On most systems, the display scaling setting enlarges everything, while the zoom in a browser or document viewer enlarges only the content; both are useful. The PDF of a scanned textbook chapter that renders at a tiny size is a common culprit; zooming to fit the page width rather than the whole page usually fixes it. Screen size interacts with both. A small laptop screen encourages close viewing, because everything on it is small; a larger external monitor allows the same text to be read from further away at a comfortable size. For a student who does most of their work at one desk, a second-hand external monitor, placed at arm's length with the text scaled up, is one of the more worthwhile purchases for the eyes, and generally better value than any specialised eyewear. Very large monitors can overshoot: if you have to turn your head to read from one side of the screen to the other, it is either too close or larger than you need, and moving it back usually solves the problem. Contrast matters as well. Dark text on a light background, known as positive polarity, has generally been found to give better legibility than light text on a dark background for people with normal vision; a 2013 study by Cosima Piepenbrock and colleagues in Ergonomics reported an advantage for positive polarity in both younger and older adults. That does not make dark mode harmful. Many people prefer it, particularly in dim rooms at night, and there is little evidence either way that it affects eye strain. The more important principle is that text should contrast clearly with its background. Pale grey text on a white page, a common choice in slick website designs, is harder to read than black on white regardless of mode, and a slide deck with coloured text on a coloured background is harder still. Height and angle The second lever is height. Chapter 2 explained that looking downwards lowers the upper eyelid, covering more of the eye and reducing evaporation, while looking straight ahead or up widens the eye and exposes more of its surface. Looking slightly downwards also happens to be the eyes' natural position for close work: when you read a book, you almost never hold it at eye level. The combined effect is that a screen positioned slightly below the eyes is both easier to focus on comfortably and less drying. The American Optometric Association suggests that the centre of the screen should sit about 15 to 20 degrees below eye level, which at a normal working distance means roughly 10 to 13 centimetres, or four to five inches, below the eyes. A simpler rule, used in OSHA's guidance, is that the top line of text should be at or slightly below eye level. With either rule, sitting upright and looking straight ahead, your gaze should fall at or near the top of the screen, and you should look gently down to read the middle of it. Two common mistakes pull in opposite directions. The first is the screen set too high: a monitor on a tall stand, perhaps placed there on the well-meant advice that screens should be "at eye level", or a large external display whose centre sits above the eyes. This widens the eye opening and increases drying, and it can push the head backwards, loading the neck. It is especially uncomfortable for people who wear varifocal or bifocal lenses, who must tilt their heads back to see through the reading part at the bottom of the lens, although this is more often an issue for older readers than for students. The second mistake is the screen set far too low, which is the default position of every laptop on a desk. Looking down at a laptop is fine for the eyes, but it pulls the head forward and down, and a long day with the head hanging forward over a keyboard is a reliable recipe for neck and shoulder pain. The screen should also be roughly perpendicular to your line of sight rather than tilted far back, which reduces reflections and keeps the whole screen at a similar distance. If you use two monitors, place the one you use most directly in front of you. If you use both equally, place them symmetrically with the join between them in front of your nose, and at the same height and distance, so that your eyes and head are not constantly swinging to one side or changing focus between two distances. Light, glare and contrast The third lever is light. The eyes are most comfortable when the screen and its surroundings are of roughly similar brightness. A bright screen in a dark room forces the eyes to adapt repeatedly between the glowing display and the dark surroundings each time you glance away, and makes the screen feel harsh. A dim screen in a very bright room, or beside a sunlit window, makes text look washed out and hard to read. The practical rule is to set screen brightness to match the room: turn it up by day, down by night, and turn on a room light rather than working at night in the dark with only the screen for illumination. Automatic brightness settings do a reasonable job of this on most modern devices. Glare comes in two forms. Direct glare is a bright light source in your field of view: a window behind the screen, a bare bulb, the sun. Reflected glare is light bouncing off the screen surface into your eyes, which lowers the contrast of the text and makes you strain to read through it. The standard ergonomic advice is to place the screen at right angles to windows, so that daylight falls across the desk from the side rather than shining into your eyes from behind the screen or reflecting off it from behind you. Blinds or curtains can soften bright daylight, and moving the screen or tilting it slightly usually removes the worst reflections from overhead lights. If you cannot avoid a reflective glossy screen in a bright space, a matte anti-glare screen protector can help. Anti-reflective coatings on spectacle lenses reduce reflections off the lenses themselves, which some glasses wearers find useful. A desk lamp for paper work should light the page without shining into your eyes or onto the screen. Many students work with a laptop and paper notes side by side, and a lamp that lights the notes well may also create glare on the screen; positioning it on the side away from the screen and angling it down usually resolves the conflict. Some people are sensitive to flicker from screens that use a technique called pulse-width modulation to dim their backlight or display, which switches the light rapidly on and off. The flicker is generally too fast to be seen but has been suggested as a cause of discomfort or headache in some individuals. The evidence here is limited, and most people are unaffected. If you notice that one particular device gives you headaches that others do not, especially at low brightness, flicker is one possible explanation, and running that screen at a higher brightness, where many devices stop modulating, is a simple experiment. Laptops, tablets, phones and beds Laptops pose a particular problem because they join the keyboard to the screen. With the keyboard at a comfortable height for the hands, the screen is too low for the neck; with the screen raised to a comfortable height, the keyboard is too high for the arms. For short sessions this does not matter much. For anyone who does most of their studying on a laptop, which is most students, the solution is to separate the two: raise the laptop on a stand, or on a stack of books, so the top of the screen is near eye level, and use an inexpensive external keyboard and mouse at desk height. This one change resolves the most common source of neck strain in students and allows the screen to be placed at the right distance and height for the eyes. Many universities lend or sell basic laptop stands through their libraries or disability and accessibility services, and a stack of old textbooks does the job for free. Tablets and phones present a different problem: they are usually held too close. A phone held at twenty-five centimetres demands twice the focusing effort of a laptop at fifty, and many people hold phones closer still, especially when lying down. Reading course material on a phone for long periods is one of the more demanding things a student can ask of the eyes. Where possible, do sustained reading on the largest screen available, at a proper distance. When you do use a phone or tablet for study, enlarge the text so you can hold it further away, and prop a tablet on a stand at a comfortable reading angle rather than holding it close in your lap. The bed deserves a paragraph of its own, because it is where a great deal of student screen time actually happens. Lying on your back holding a phone above your face, or on your side with a laptop inches away, combines most of the unfavourable conditions described in this book: a very close viewing distance, a screen often above the eyes so the eye opens wide, a dark room with a bright screen, and a posture held for a long time. Nothing about this is dangerous, and nobody expects students never to use their phones in bed. But if your eyes are uncomfortable at night, this is a good place to start. Turning on a lamp, sitting up, holding the device lower and further away, and enlarging the text will each help. The effects of late-night screen use on sleep are a separate question, discussed in the next chapter. A setup for real student spaces Most students do not have a dedicated study room with an adjustable chair and monitor arm. They have a desk in a shared room, a kitchen table, a library carrel, or a sofa. The principles in this chapter apply to all of them, and most can be met with improvised equipment. Consider a student who shares a small room and works at a narrow desk facing a window, with a laptop flat on the desk. In the afternoon, the window is bright behind the screen and the text looks washed out; he turns the screen brightness to maximum and leans in. By evening, the window is dark and he works with the main light off, his screen now glaring. He also has a heating vent beneath the desk blowing upwards. A few changes improve almost every feature at once. He turns the desk so that the window is to his side. He raises the laptop on three old textbooks and plugs in a second-hand keyboard. He turns on the room light in the evening and lets the screen's automatic brightness do the rest. He increases the default text size in his browser and document viewer by a notch. And he closes the vent or points it away. The total cost is a keyboard, and the result is a workspace that is kinder to his eyes and neck for every hour he spends in it. A library carrel may leave less room to adjust, but a portable laptop stand and a compact keyboard fit in most backpacks, and choosing a seat where the windows are to the side, and away from the vents, is free. The central idea is always the same: arrange the space so that your eyes look slightly down, at a comfortable distance, at text large enough to read without effort, in light that matches the screen. Once the setup is right, the breaks and blinks of the previous chapters have much less work to do. Chapter 5: Blue Light Without the Marketing No topic in this book has generated more anxiety, or more sales, than blue light. Students are told that the light from their screens is damaging their retinas, ageing their eyes, causing eye strain and wrecking their sleep, and that a pair of filtering glasses, a screen protector or a special coating will protect them. Filtering lenses are sold online for a few dollars and by opticians as expensive add-ons. Some are marketed with images of eyes being bombarded by harsh blue rays. It is a compelling story, and it contains a grain of real science. But most of it does not survive contact with the evidence. The key to thinking clearly about blue light is to separate three quite different questions that the marketing blurs together. Does the blue light from screens damage the eye? Does it cause digital eye strain? And does it affect sleep? The answers are, in turn: no, on present evidence; there is no good evidence that it does; and yes, to some extent, although less than is often claimed and for reasons that have little to do with eyes feeling tired. Table 3 sets out the common claims alongside what the best available evidence shows. Table 3. Common claims about blue light from screens and what the evidence shows. Claim What the evidence shows Key source Screen blue light damages the retina Screen emissions fall well below international exposure limits, even with extended viewing O'Hagan and colleagues, Public Health England (2016) Blue light causes digital eye strain Eye strain is explained mainly by blinking, focusing and environment; blue-light blocking not found effective TFOS Lifestyle report (2023) Filtering glasses relieve eye strain May make no difference to visual fatigue in the short term; low-certainty evidence Cochrane review, Singh and colleagues (2023) Filtering glasses improve sleep Mixed results in small, varied trials; effect uncertain Cochrane review, Singh and colleagues (2023) Evening screen light can delay sleep Real effect under some conditions, modest in size; content and timing matter too Chang and colleagues (2015); Duraccio and colleagues (2021) What blue light is and where it comes from Visible light spans wavelengths from about 380 to 780 nanometres. The shorter end, from roughly 400 to 500 nanometres, appears blue or violet; the longer end appears red. Shorter wavelengths carry more energy per photon, which is why the most energetic end of the spectrum, ultraviolet, just beyond violet, is well known to damage skin and eyes. Blue light sits next to ultraviolet, and that proximity is the basis of the concern. Blue light is not a product of screens. The largest source by far, for everyone, is the sun. Daylight is intensely rich in blue wavelengths, which is why the sky is blue. Even on an overcast day, outdoor light is typically many times brighter than an indoor room, and a screen is dimmer than the room. The white light of a modern screen, like that of LED room lighting, is usually produced by a blue LED combined with a phosphor that converts some of the blue into longer wavelengths, which gives the light a spectral peak in the blue. That is why screens are described as "blue-rich". But blue-rich does not mean bright. In absolute terms, the amount of blue light entering your eyes from a phone or laptop is a small fraction of what you receive from being outdoors in daylight. That comparison matters because humans evolved outdoors, under a sky that bathes the eyes in blue light for hours every day. If ordinary exposure to blue light at the intensity of a screen were damaging, everyone who had ever spent a day outside would be at far greater risk from the sky than from any device. The damage question There is a real phenomenon called the blue-light hazard. At very high intensities, short-wavelength visible light can cause photochemical damage to the retina. This is why staring at the sun during an eclipse can burn a permanent blind spot into the centre of vision, and why welders need protective filters. Laboratory experiments in which cells or animal retinas are exposed to intense blue light show damage, and these experiments are sometimes cited in marketing for filtering products. The problem is that the intensities and durations used in such experiments bear little relation to looking at a phone. The most direct test of whether everyday sources approach dangerous levels was carried out by physicists at the Centre for Radiation, Chemical and Environmental Hazards at Public Health England, John O'Hagan, Marina Khazova and Luke Price, and published in the journal Eye in 2016. They measured the emissions of a range of common sources, including low-energy light bulbs, computer screens and tablets, and compared them with international exposure limits for the blue-light hazard, and with the exposure a person would receive from looking at a blue sky. None of the sources they assessed approached the exposure limits, even for extended viewing times. In plain terms, looking at a screen all day does not deliver anything close to a dose of blue light known to harm the retina. This is consistent with the position taken by the American Academy of Ophthalmology, which has stated that the blue light from digital devices does not damage the eye and that it does not recommend blue-light-blocking glasses for computer use. The 2023 Cochrane review, discussed below, looked for trials measuring macular health in people using filtering lenses and found none, so there is no clinical trial evidence that such lenses protect the retina either. There remain some open research questions about lifelong cumulative light exposure and age-related macular degeneration, a disease of older adults with many causes, but those questions concern total light exposure over decades, of which screens are a minor part, and they do not translate into evidence that screen use harms the retina of a student. The eye strain question The second claim is that blue light causes the tiredness, discomfort and burning of digital eye strain. The previous chapters have already offered a better explanation. Eye strain arises mainly from reduced and incomplete blinking during concentrated work, from sustained focusing at a fixed near distance, from uncorrected refractive errors and binocular vision problems, and from the way screens tend to be positioned and lit. None of those mechanisms depends on the colour of the light. If blue light were the main cause, one would expect reading from paper under warm, yellowish lamplight to be free of strain, and it is not. This is not to say colour and brightness are irrelevant to comfort. A glaring screen in a dark room is uncomfortable, and some people prefer a warmer colour setting in the evening. But the discomfort of a too-bright screen is a matter of brightness contrast with the surroundings, discussed in Chapter 4, and turning the brightness down or the room lights up addresses it more directly than tinting the light. The 2023 TFOS review of the digital environment and the ocular surface considered this question explicitly. Its summary conclusion was that, based on current best evidence, blue-light blocking interventions do not appear to be an effective management strategy for digital eye strain. That judgement leads naturally to the product at the centre of the debate. What filtering lenses do and do not do Blue-light filtering lenses, sometimes called blue-blocking lenses, work either by a coating that reflects some blue light, often visible as a faint blue sheen on the lens surface, or by a tint or material that absorbs it, which may give the lens a yellowish cast. Different products filter very different amounts. Many clear lenses marketed for screen use remove only a modest proportion of blue light, while strongly amber-tinted glasses, which are sometimes studied for sleep, remove much more but visibly change colours. In 2023, the Cochrane Collaboration, whose systematic reviews are among the most rigorous summaries of clinical evidence available, published a review of blue-light filtering spectacle lenses for visual performance, sleep and macular health in adults, led by Sumeer Singh and Laura Downie at the University of Melbourne. The authors searched for randomised controlled trials comparing blue-light filtering lenses with ordinary lenses and found seventeen, with between five and 156 participants each and follow-up ranging from less than a day to five weeks. Their conclusions were measured. Blue-light filtering lenses may not reduce symptoms of eye strain with computer use over the short term compared with ordinary lenses. There was probably little or no effect on sharpness of vision. The effect on sleep quality could not be determined, because six trials gave inconsistent results in varied populations. No trials had measured macular health, contrast sensitivity, colour discrimination or glare. Many of the included studies were at high risk of bias, and the certainty of evidence was mostly low. The authors called for larger, better trials. Two points about that review are often lost in popular reporting. The first is that "no good evidence of benefit" is not the same as "proven useless". The trials were small and short, and it remains possible that some particular lens, in some particular group, has an effect that has not been detected. The second is that the review found no evidence that the lenses are harmful, although adverse effects were not consistently reported. So a student who already owns a pair and likes them has no reason to throw them away. The practical conclusion is simply that there is no good reason to buy them for eye strain, and the money is better spent on things that address the actual causes: an eye examination, a laptop stand, or an external keyboard. There is one group for whom this deserves a specific caution. Some filtering lenses, particularly the more strongly tinted ones, shift the appearance of colours. For students in art, design, photography, film or any field that depends on colour judgement, wearing tinted lenses during work can be a genuine disadvantage. The sleep question The third question is the one where blue light matters most, although not in the way it is usually framed. It concerns not the eyes' comfort but the body's clock. The retina contains a small population of cells called intrinsically photosensitive retinal ganglion cells, which contain a pigment called melanopsin. These cells are not mainly concerned with forming images. Their job is to report the overall level of light to the brain's master clock, which uses that information to keep the body's daily rhythms aligned with day and night. Melanopsin is most sensitive to light in the blue part of the spectrum. Bright light in the evening, especially blue-rich light, signals "daytime" to the clock, suppresses the release of the hormone melatonin, which normally rises in the evening before sleep, and can push the timing of the clock later. A well-known study by Anne-Marie Chang, Charles Czeisler and colleagues at Brigham and Women's Hospital, published in the Proceedings of the National Academy of Sciences in 2015, tested this directly. Participants spent several evenings reading an electronic book on a light-emitting tablet for about four hours before bedtime, and several evenings reading a printed book under dim light. After reading on the tablet, participants took longer to fall asleep, had less evening sleepiness, showed reduced melatonin secretion, had a later-timed circadian clock and were less alert the next morning. The effects were real and measurable. The conditions, though, were demanding: hours of reading on a bright tablet every evening, compared with a printed book in dim light, in a controlled laboratory. The size of the effect on the time taken to fall asleep was modest. The real world is more complicated, because screens in the evening do much more than emit light. They deliver content that is interesting, emotionally engaging, socially demanding and often designed to be hard to put down. A message thread, a game or an argument on social media keeps the mind alert regardless of the colour of the pixels, and screen use often displaces sleep simply by keeping people up later. A randomised study at Brigham Young University, published in Sleep Health in 2021 by Kara Duraccio and colleagues, illustrates the point. It assigned 167 undergraduate students aged 18 to 24 to one of three conditions for the hour before bed over a week: using an iPhone with the Night Shift warm-colour setting on, using it with Night Shift off, or not using a phone at all. Sleep was tracked with wrist-worn devices. Across the whole sample there were no significant differences in sleep outcomes between the groups. In an exploratory analysis of students who were already getting around seven hours of sleep, those who did not use their phones slept somewhat better than those who did, whether or not Night Shift was on. The sensible reading of this evidence is that evening screen use can affect sleep, that light is one mechanism among several, and that shifting the screen's colour towards warm tones is at most a small help. The measures that matter more are the ones that address both the light and the engagement: dimming screens and room lights in the last hour or so before bed, finishing intensive study some time before lights out rather than working until the moment you close your eyes, and keeping the phone out of reach once you are in bed. For a student, the most reliable way to protect sleep from screens is to stop using them for demanding work late at night, not to change their colour. Night mode settings do no harm and some people find a warmer screen more pleasant in a dark room, but they do not turn a midnight revision session into a restful one. Consider how this plays out for a student with an early lecture the next morning. On a typical night she studies at her laptop until about half past midnight, then lies in bed scrolling her phone for another half hour, falls asleep after one, and struggles through a nine o'clock lecture. She has turned on Night Shift on her phone and bought a pair of filtering glasses, and neither has made much difference. A more effective plan changes the timing and the content rather than the colour. She aims to finish demanding study by about eleven, switches the room to a lamp rather than the overhead light, and spends the last half hour on something undemanding, which may still involve a screen, such as a familiar show at a distance, but not the messaging and scrolling that keep her mind switched on. Her phone charges on the desk across the room rather than beside the pillow. In the morning she walks to her lecture instead of taking the bus, which gives her twenty minutes of daylight. None of this requires any special equipment, and it addresses light exposure, mental arousal and the displacement of sleep all at once. Chapter 3's advice about breaks helps here too: evening sessions that include regular pauses tend to end with less of the wired, gritty-eyed state that makes it hard to wind down. It is also worth being candid about the limits of what is known. The laboratory findings on light and melatonin are robust. The size of the real-world effect of ordinary evening screen use on sleep, separated from the effects of content, stress and late bedtimes, is much less certain, and it probably varies between people. Some individuals appear to be more sensitive to evening light than others. If you find that you sleep well despite evening screen use, there is no need to worry about the colour of your screen. If you sleep badly, it is worth looking at the whole evening, not just the light. There is a final, positive side to the biology of blue light that is easily missed. The same cells that respond to blue light in the evening respond to bright light in the morning and daytime, and bright daytime light, above all outdoor daylight, helps anchor the body clock and support alertness during the day and sleep at night. Blue light is not the enemy. It is a signal that the body uses to tell the time. The goal is not to avoid it but to get plenty of it during the day and less of it late at night. That point, and the value of daylight, bring us to the one genuinely long-term concern associated with heavy study: myopia. Chapter 6: Near Work, Daylight and Myopia Everything discussed so far concerns comfort. Eye strain, dryness and focusing fatigue are unpleasant and worth preventing, but in a healthy young eye they are temporary. This chapter concerns something different: a structural change in the eye that, once it has happened, does not reverse. Myopia, or short-sightedness, is the one long-term consequence of an intensely studious life for which there is serious scientific evidence. It is also a subject in which public discussion tends towards two errors at once, alarm about screens in particular and complacency about near work and indoor life in general. Getting the picture right matters for students, because the university years sit at the tail end of the period in which myopia develops and progresses, and the habits formed then carry forward. What myopia is and why it is rising In a myopic eye, light from distant objects comes to a focus in front of the retina rather than on it, so distant things look blurred while near things are clear. In the great majority of cases this happens because the eyeball has grown too long from front to back. Growth of the eye is normally regulated during childhood by a process called emmetropisation, in which visual feedback guides the eye towards the length that brings distant objects into focus. In myopia this regulation overshoots. The eye keeps elongating beyond the point of clear distance vision. Glasses, contact lenses and refractive surgery correct the blur very effectively, and for most people with mild myopia that is the end of the matter. The concern lies with higher degrees. Because myopia is usually caused by an elongated eye, the retina and other tissues at the back of the eye are stretched, and the degree of stretch rises with the degree of myopia. Higher myopia is associated with an increased lifetime risk of several sight-threatening conditions, including retinal detachment, glaucoma, early cataract and myopic maculopathy, a degenerative change in the central retina that is one of the leading causes of irreversible vision loss in parts of East Asia. Refractive surgery changes the shape of the cornea to remove the need for glasses, but it does not shorten the eye, so it does not remove these risks. The risk is not confined to very high prescriptions. In a 2019 analysis in Optometry and Vision Science, Mark Bullimore and Noel Brennan drew on five large population studies and estimated that each additional dioptre of myopia was associated with a 67 per cent increase in the prevalence of myopic maculopathy. Their point, summarised in the title "why each diopter matters", was that slowing myopia by even one dioptre is worthwhile, whatever a person's starting point. It is worth noting that one of the authors was employed by a contact lens company with an interest in myopia control, but the underlying relationship between degree of myopia and risk of complications is well established. Myopia is becoming dramatically more common. The most widely cited global estimate comes from Brien Holden and colleagues, who pooled data from 145 studies covering 2.1 million people and published their analysis in Ophthalmology in 2016. They estimated that about 1.4 billion people, 22.9 per cent of the world's population, were myopic in the year 2000, and projected that by 2050 the figure would reach about 4.8 billion, nearly half the world, with around 938 million people, almost one in ten, having high myopia. Projections of this kind involve many assumptions, but the direction is not in doubt. In some urban populations in East and Southeast Asia, the large majority of young adults leaving school are now myopic. A change this fast cannot be genetic. Genes influence individual susceptibility, and children with two myopic parents are at higher risk, but the human gene pool does not change in a few decades. The rise is environmental, and two environmental factors stand out: the amount of close work people do, and the amount of time they spend outdoors. Near work: a real association, but not a simple one The idea that reading causes short-sightedness is centuries old. The modern evidence supports a link, but a more modest and more complicated one than folk wisdom suggests. The most striking evidence concerns education. In a 2018 study in the BMJ, Edward Mountjoy and colleagues at the University of Bristol used a genetic technique called Mendelian randomisation, which uses inherited variants as natural experiments, to ask whether more years of education cause myopia or whether myopia leads people to stay in education longer. Using data from nearly 68,000 people in the UK Biobank, they found evidence that education causes myopia, not the reverse. Each additional year of education was associated with about a quarter of a dioptre more myopia. By their calculation, a UK university graduate with seventeen years of education would, on average, be at least one dioptre more myopic than someone who left school at sixteen. Education, of course, is not a single exposure: it bundles together reading, writing, screen work, time indoors and a lifestyle arranged around desks. But the finding makes clear that something about the educated life shapes the eye. Studies that try to isolate near work itself find smaller effects. A 2015 meta-analysis in PLoS One by Hsiu-Mei Huang and colleagues pooled 27 studies of about 25,000 children aged 6 to 18 and found that more time spent on near work was associated with higher odds of myopia; the odds rose by about 2 per cent for each additional dioptre-hour of near work per week, a unit that combines how close the work is with how long it lasts. That is a real association, but a modest one per hour, and near work is notoriously hard to measure accurately by questionnaire. Some studies suggest that how near work is done matters as much as how much: research in Australian schoolchildren published in 2008 by Jenny Ip and colleagues found that reading at a close distance and reading continuously for long periods without a break were associated with myopia, which fits the logic of the previous chapters. What about screens specifically? A 2021 systematic review in The Lancet Digital Health by Joshua Foreman and colleagues examined 33 studies of smartphone and tablet use and myopia in children and young adults. In a meta-analysis of a subset, smart device screen time was associated with higher odds of myopia, and the association was stronger when smart device use was combined with computer use. The authors were careful in their conclusion: smart device exposure might be associated with increased risk. Every study relied on questionnaires to measure screen time, many did not adjust properly for other factors, and the studies varied greatly. The honest position is that screens are one form of near work among many, and there is no strong evidence that they are worse for myopia than books held at the same distance for the same length of time. What screens do is make near work easier to sustain, more continuous and closer, particularly on phones, and they often displace time that would otherwise be spent outdoors. Daylight, the strongest protective factor If near work is a modest risk factor, time outdoors is a comparatively strong protective one, at least in children. The evidence here includes randomised trials, which is unusual in this field. The best known was carried out in Guangzhou, China, by Mingguang He and colleagues and published in JAMA in 2015. Twelve primary schools were randomly assigned either to add a forty-minute outdoor activity class to every school day, with parents encouraged to take children outdoors after school and at weekends, or to continue as usual. Among children who were not myopic at the start, the three-year cumulative incidence of myopia was 30.4 per cent in the intervention schools and 39.5 per cent in the control schools. Forty extra minutes outside each day reduced new cases by roughly a quarter. Other school-based trials, notably in Taiwan, have found related effects, and programmes that encourage outdoor time have been adopted in several East Asian education systems. Why outdoor time helps is not fully settled. The leading hypothesis is light intensity. Outdoor light, even in shade, is many times brighter than indoor light, and in animal studies bright light stimulates the release of dopamine in the retina, which appears to act as a brake on the elongation of the eye. Other contributing factors may include the greater viewing distances outdoors and the different pattern of focus across the visual field. What seems clear is that the benefit does not depend on exercise or sport as such; it is being outside that counts. This is one reason the outdoor break has a place in the study routine of Chapter 3: a walk outside does two useful things at once. The scale of the difference in light is easy to underestimate, because the eye adapts so well that a bright office and a cloudy street can look similarly lit. Light meters tell a different story. A typical indoor room or lecture hall is lit to a few hundred lux, the unit of illuminance. Outdoors on an overcast day, levels in the thousands or tens of thousands of lux are common, and direct summer sunshine can exceed a hundred thousand. Even sitting in the shade of a tree, you are usually receiving far more light than at the brightest desk indoors. This is why a library seat by a window, though better than one in an interior corner, is not the same as being outside, and why the researchers who designed the school trials concentrated on getting children out of doors rather than on brighter classrooms, although some have since experimented with brighter indoor environments as well. Most of this evidence concerns preventing the onset of myopia in children. Outdoor time appears to be less effective at slowing progression once myopia has started, and there is much less direct evidence in young adults. A small Australian study published in 2013 by Katrina Schmid and colleagues, which fitted 35 university students with light and ultraviolet sensors for three days, found some evidence that students with stable myopia had received more outdoor ultraviolet exposure than those whose myopia was progressing, and called for larger longitudinal studies. That is where the adult evidence currently stands: suggestive, biologically plausible, and not yet proven. What this means for a university student The practical question is whether a student in their late teens or twenties still has any myopia to gain or lose. The answer is that most myopia begins and progresses in childhood and adolescence, and in many people it stabilises by the late teens or early twenties, but a substantial minority continue to progress, and some become myopic for the first time, during the years of intensive study. Two longitudinal studies illustrate this. In Norway, Bjørn Kinge and colleagues followed 224 engineering students with a mean age of about 21 over three years, publishing their results in 2000. On average the students' refraction shifted towards myopia by about half a dioptre over the three years. The shift was associated with time spent reading scientific literature, doing practical near work and attending lectures. Interestingly, given the concerns of this book, it was not associated with time spent at computer screens or watching television, although the screens of the late 1990s were used very differently from those of today. In China, a 2022 study by Xue Bai and colleagues followed 526 first-year medical students in Tianjin for two years. More than nine in ten were already myopic at entry. Over two years their average refraction became more myopic by about 0.38 dioptres, the proportion with high myopia rose from about 21 to about 28 per cent, and time spent on online games was associated with faster progression. These findings do not mean that studying will ruin your eyesight. The changes they describe are, on average, a fraction of a dioptre over several years. But they do mean that the university years are not neutral for the eye, particularly for students who are already myopic, who come from families with a history of high myopia, or who study in the most demanding, indoor-heavy disciplines. Given the principle that each dioptre matters, the habits that might slow progression are worth adopting when they are cheap and pleasant, which most of them are. In practice, the advice converges with that of the earlier chapters. Break up long stretches of close work, rather than reading or scrolling continuously for hours. Keep reading material at a reasonable distance; the phone held a hand's width from the face is the least favourable arrangement. Spend some time outdoors in daylight every day, for example by walking between classes, eating lunch outside, or taking study breaks outdoors rather than at the next screen. The evidence that these measures slow myopia in adults is incomplete, but they are free, they have no downside, and they address eye strain and sleep at the same time. It is also worth knowing what is available for those with progressing myopia. In the past decade, treatments designed to slow myopia progression, as opposed to merely correcting the blur, have become established for children. They include low-dose atropine eye drops, orthokeratology lenses worn overnight to reshape the cornea, and specially designed spectacle and soft contact lenses. Most of the evidence for these treatments comes from children, and their role in young adults is much less clear. A student whose prescription is still changing noticeably from year to year, particularly one with an already high prescription, may reasonably ask an optometrist or ophthalmologist whether any such options make sense in their case, but should expect an honest answer that the evidence in adults is limited. For students who want to monitor their own situation, the most useful thing is a record. Keep a note of your prescription after each examination, or ask the practice for a copy, which in many places you are entitled to. A change of a quarter of a dioptre between examinations can reflect ordinary measurement variation; a steady drift in the same direction over several years is more meaningful. Some clinics now measure axial length, the front-to-back length of the eye, with an optical instrument that takes a few seconds and involves no contact with the eye. Axial length is a more direct measure of the structural change underlying myopia than the prescription alone, and repeated measurements over time give a clearer picture of whether the eye is still growing. Not every practice offers it, and it is not necessary for everyone, but for a student with a strongly myopic family history and a changing prescription it is a reasonable thing to ask about. Finally, a word about what a myopic student should not do. Some online sources recommend deliberately wearing a weaker prescription than needed, or going without glasses, in the belief that this will slow progression. Undercorrection has been tested in children and has not been shown to slow myopia; some studies suggested it may even speed progression. It also makes lectures, driving and daily life harder. Wear the prescription that lets you see clearly, have it checked regularly, and put your efforts into the habits that the evidence supports. Consider a first-year student who has worn glasses since she was twelve and whose prescription has increased at every examination. Her mother is highly myopic. She arrives at university planning to spend long days in the library. Nothing about her situation calls for alarm, but it is a good reason to take the principles of this chapter seriously from the start: to choose a library seat by a window, to take her twenty-minute breaks looking out of it, to walk to campus rather than take the bus, to study outside on fine afternoons, to keep her phone for short tasks rather than long reading, and to have her eyes checked each year so that any progression is noticed early and she can discuss the options. These habits cost her nothing and cannot hurt. They may help her eyes, and they will almost certainly help her comfort, her sleep and her mood. The next chapter returns to the front of the eye and the most common chronic consequence of long hours of concentrated screen work: the dry eye. Chapter 7: Keeping the Ocular Surface Healthy Most of the burning and grittiness that students feel after long screen sessions is a temporary failure of the tear film, of the kind described in Chapter 2. It passes with rest, a few full blinks and a night's sleep. For some people, though, the discomfort stops passing. The eyes feel dry in the morning before any screen work, sting in wind, tire quickly when reading, and water unpredictably. At that point the problem is no longer simply a bad afternoon. It may be dry eye disease, a common, chronic condition of the ocular surface that is increasingly recognised in young adults and that heavy screen use can bring to the surface or make worse. This chapter explains the difference between transient dryness and dry eye disease, and then sets out what you can do to protect the ocular surface: in the environment where you study, in what you put in your eyes, and, for the many students who wear them, in how you handle contact lenses. Table 4 summarises the main self-care measures and how strong the evidence for each is. When dryness becomes a disease The 2017 report of the Tear Film and Ocular Surface Society's Dry Eye Workshop, TFOS DEWS II, defined dry eye as a multifactorial disease of the ocular surface characterised by a loss of stability of the tear film, accompanied by symptoms, in which an unstable and overly salty tear film, inflammation and damage to the surface, and changes in the nerves of the eye all play a part. The key idea is a vicious circle. An unstable tear film breaks up quickly and evaporates, leaving the remaining tears more concentrated. The concentrated tears irritate the surface and trigger inflammation. Inflammation damages the cells that produce mucin and destabilises the film further. Around and around it goes, and the longer it continues, the more self-sustaining it can become. Dry eye was long thought of as a condition of older adults, particularly older women, and it does become more common with age. But studies in university populations have found substantial rates of symptoms among students, and the TFOS reports identify screen use and contact lens wear, both near universal among students, as important contributing factors. Young women appear to be at somewhat higher risk than young men, which may reflect hormonal influences on the tear glands. The most common underlying problem, in young and old alike, is meibomian gland dysfunction. The meibomian glands, described in Chapter 2, line the edges of the eyelids and secrete the thin oily layer that slows evaporation of the tears. When their openings become blocked, or the oil they produce becomes thick and waxy, the tears evaporate too quickly. Signs include eyelid margins that look red or slightly thickened, a foamy appearance along the lid edge, and symptoms that are often worse later in the day. Reduced and incomplete blinking during screen work is thought to contribute, because each full blink helps squeeze oil out of the glands; glands that are rarely squeezed may be more prone to blockage. Several medications commonly used by students can also dry the eyes. Isotretinoin, the oral medication widely prescribed for severe acne, is well known to affect the meibomian glands and to cause dry eye, sometimes significantly. Oral antihistamines taken for hay fever, some antidepressants and some other medicines can reduce tear production. If your eyes became uncomfortable after starting a new medication, mention it to the prescriber and to an optometrist; do not stop a prescribed medicine on your own, but do make sure the connection is known. The practical distinction between transient screen-related dryness and dry eye disease is not something you can make with certainty at home. A rough guide is persistence. If your symptoms go away with rest, sleep and the changes described in this book, you are dealing with the first. If they are present most days, including days with little screen use, if they are present on waking, or if they are getting worse over months despite reasonable changes, you should have your eyes examined. An optometrist or ophthalmologist can look at the tear film and eyelids under magnification, measure how quickly the tears break up, stain the surface with a harmless dye to reveal damage, and examine the meibomian glands. Dry eye disease is treatable, and treatments available through a clinician, including anti-inflammatory drops and procedures to clear the meibomian glands, go well beyond what you can buy over the counter. The study environment The first line of defence is the environment in which you study. The ocular surface loses water to the air, and it loses more when the air is dry and moving. Low humidity is common in precisely the places students spend their time. Heated buildings in winter, air-conditioned libraries and laboratories in summer, and aircraft cabins all tend to have dry air. The simplest measures are about position: sit away from heating and air-conditioning vents, especially those that blow across your face, and point desk fans at your body rather than your eyes. In a dry room of your own, a small humidifier may help, and the TFOS review of the digital environment identified humidifiers, along with warm compresses and moisture-retaining goggles, among the promising strategies in its systematic review, although the trials were limited. A bowl of water on a radiator will not make a measurable difference; an actual humidifier, kept clean to prevent mould and bacterial growth, might. Wind, smoke and pollutants all irritate the surface. Wraparound sunglasses outdoors on a windy day reduce evaporation. Tobacco smoke, including second-hand smoke, is a well-recognised irritant, and vaping aerosols may also irritate the eyes. The car heater on the way to campus, set to blow onto the face, is a small but easily fixed contributor. Screen position, as Chapter 4 explained, affects evaporation continuously: a screen set slightly below eye level lets the upper lids cover more of the eye. Combined with the break and blink habits of Chapters 2 and 3, a well-positioned screen in reasonably humid, still air removes a large share of the drying load. Table 4. Self-care measures for dry, irritated eyes during study. Measure What it does Strength of evidence Notes Regular breaks and full blinks Renews tear film, expresses lid oils Physiologically well founded; small trials supportive Free; see Chapters 2 and 3 Lower screen, still air, away from vents Reduces evaporation Physiologically well founded Free or cheap Humidifier in a dry room Raises humidity near the eyes Promising but limited trials (TFOS 2023) Keep clean Lubricating eye drops (artificial tears) Supplements and stabilises tear film Widely used; trial evidence for screen strain limited Preservative-free if used often Warm compresses and lid hygiene Softens and releases meibomian oil Promising, especially for gland dysfunction (TFOS 2023) Comfortably warm, never hot Omega-3 supplements Proposed anti-inflammatory effect Large trial found no benefit over placebo (DREAM, 2018) Not a reliable remedy Switching from contacts to glasses for long sessions Restores a normal tear film Physiologically well founded Useful for evening study What to put in your eyes, and what not to Lubricating eye drops, commonly called artificial tears, are the most widely used remedy for dry eyes. They are available over the counter in most countries, in many formulations. Some are thin and watery, suitable for use during the day; others are thicker gels that last longer but can blur vision for a few minutes, and are often used before sleep. Some contain lipids intended to support the oily layer of the tear film, which in principle makes them better suited to evaporative dryness. The TFOS review noted that more and larger trials are needed to establish which constituents work best for digital eye strain, so there is no single product that the evidence identifies as superior. In practice, many people try one or two and settle on the one they find most comfortable. Two practical points matter more than the brand. First, preservatives. Multi-dose bottles contain preservatives to prevent contamination once opened, and some preservatives, notably benzalkonium chloride, can irritate the ocular surface with frequent use. If you use drops more than a few times a day, preservative-free formulations, sold in single-use vials or in special multi-dose bottles designed to stay sterile without preservatives, are the usual recommendation. Second, timing. Drops work best as a supplement to good habits, used before or during a long session, rather than as a rescue after hours of burning. They do not replace breaks and blinking; they buy the tear film a little more time between them. One type of drop is worth avoiding for regular use. So-called redness relievers, which contain decongestants such as tetrahydrozoline or naphazoline, work by constricting the small blood vessels on the white of the eye. They make red eyes look white for a few hours, but they do nothing for dryness, and with repeated use the vessels can dilate again more strongly when the drops wear off, a rebound effect that encourages further use. A red eye is a signal, and hiding it does not address its cause. If you need to look fresh for an interview or photograph, an occasional dose does no harm, but it is not a treatment. Warm compresses and lid hygiene are the standard self-care for meibomian gland dysfunction. The oil in blocked glands becomes thicker as it deteriorates, and gentle heat softens it so that it flows. A clean flannel soaked in comfortably warm water and held over the closed eyes for several minutes, rewarmed as it cools, or one of the reusable heated eye masks sold for the purpose, can be used once or twice a day. The heat should be warm, never hot enough to scald the delicate skin of the eyelids. Afterwards, gently massaging along the lid margins, or cleaning them with a clean cloth or a product designed for lid hygiene, helps clear debris. The TFOS review listed warm compresses among the promising strategies, and they are a mainstay of clinical advice for gland dysfunction. A word about supplements, which are marketed heavily for dry eye. Omega-3 fatty acids from fish oil were widely recommended for years, on the theory that their anti-inflammatory properties would help. The DREAM study, a large, well-conducted trial funded by the US National Eye Institute and published in the New England Journal of Medicine in 2018, tested this directly. It randomly assigned 535 patients with moderate to severe dry eye disease to take either 3,000 milligrams a day of fish-derived omega-3 fatty acids or an olive oil placebo for a year. Both groups improved to a similar extent, and there was no significant difference between them in symptoms or in any of the clinical measures. The TFOS lifestyle review, looking at the broader literature, took a somewhat more open view of nutritional supplementation as an area of promise, but on the strength of the largest and most rigorous trial, omega-3 supplements cannot be recommended as a reliable remedy. Eating fish as part of a varied diet is a reasonable thing to do for general health; buying expensive capsules specifically for your eyes is not supported by the best evidence. Eye makeup deserves a brief mention, because it bears directly on the meibomian glands. Eyeliner applied to the inner rim of the eyelid, the waterline, sits directly over the openings of the glands and can contribute to blockage and to debris in the tear film. Lining outside the lashes, removing all eye makeup thoroughly before sleep, and discarding old products after a few months all reduce the risk. The TFOS lifestyle reports devoted a full review to cosmetics and the ocular surface, a sign of how seriously the field now takes the issue. Contact lenses without the risk Contact lenses are safe, effective and enormously popular among students, and nothing in this section is an argument against them. But they are medical devices placed directly on a living surface, and the risks of wearing them are concentrated in a few specific habits that are common among young adults. The most serious risk is microbial keratitis, an infection of the cornea that can cause scarring and, in severe cases, lasting loss of vision. It is uncommon, but the risk is strongly affected by how lenses are worn. The clearest data come from a large population-based study in Australia by Fiona Stapleton and colleagues, published in Ophthalmology in 2008. It identified every new case of contact lens-related microbial keratitis presenting to eye-care practitioners across the country over a year, and combined these with a national survey of lens wearers to calculate risk. In people wearing daily-wear soft lenses, the annual incidence was about 1.9 cases per 10,000 wearers. In people who wore soft lenses overnight, it rose to around 20 per 10,000, and for overnight wear of silicone hydrogel lenses about 25 per 10,000. Overnight wear of any lens carried a higher risk than daily wear. Poor hygiene of the lens storage case, smoking, buying lenses online and being a relatively new wearer were also identified as risk factors. For students, several rules follow from this and from long-standing clinical guidance. Do not sleep in your lenses unless they have been specifically prescribed for overnight wear and you are following that advice. This includes the nap on the library sofa and the night you fall asleep in your lenses after a party or a long study session. If it happens occasionally, remove the lenses as soon as you wake, and do not put in a fresh pair until your eyes feel completely normal. Keep lenses away from water. Tap water, swimming pools, hot tubs, lakes and the sea can carry organisms, including the amoeba Acanthamoeba, which causes a rare but particularly severe and hard-to-treat corneal infection. Never rinse lenses or cases in tap water, never store lenses in water, and avoid swimming or showering in them. Swimming goggles over lenses or, better, prescription goggles, reduce the risk. Look after the case. Use fresh solution every time rather than topping up old solution, rinse the case with solution rather than water, let it air-dry open and face down, and replace it regularly. The case is often more contaminated than the lenses. Replace lenses on schedule. Two-week or monthly lenses worn for longer than intended accumulate deposits and become less comfortable and potentially less safe. Daily disposable lenses, used as intended, avoid cases and solutions altogether, which removes several of the hygiene risks, although, as the Australian data show, they carry risks of their own if worn overnight. Buy lenses through a proper prescription. Lenses need to be fitted to your eyes. Buying them from sources that do not require a valid prescription, and decorative or costume lenses sold without fitting, have been linked to serious infections. In the United States, all contact lenses, including purely decorative ones, are regulated as medical devices and require a valid prescription; rules elsewhere vary. Take lenses out when your eyes complain. If a lens-wearing eye becomes red, painful, sensitive to light or blurred, remove the lenses immediately and do not reinsert them. If the symptoms do not settle quickly, particularly if there is pain or light sensitivity, you need to be seen promptly, the same day where possible. Chapter 8 explains why this is one of the situations in which delay can matter. Beyond safety, contact lenses interact with screen work through comfort. As Chapter 2 explained, a lens thins and destabilises the tear film in front of it, so lens wearers often find their eyes dry out faster during long sessions. Many students find that wearing glasses for evening study, and saving lenses for lectures, sport and social life, removes most of their end-of-day discomfort. If lenses are uncomfortable even for moderate periods, it is worth going back to the practitioner who fitted them, since a different material, a different design or treatment of an underlying dry eye can make a large difference. The ocular surface is resilient, but it is not indestructible. Treated with a little care, a young eye can tolerate long days of study for years without lasting harm. The next chapter deals with the signs that something beyond ordinary strain or dryness is going on, and with the examination that is the best single step most students can take for their eyes. Chapter 8: Warning Signs and the Examination The previous chapters have concentrated on what you can do yourself: arrange the desk, take breaks, blink fully, manage the air, look after contact lenses and get outside. Those measures resolve most students' eye strain. They do not resolve all of it, and they are not meant to. Some symptoms that feel like strain are really the product of a problem that only an examination can find, and a small number of eye symptoms need urgent attention. Knowing which is which is the last piece of practical knowledge this book has to offer, and arguably the most important, because it is the one most likely to protect your sight rather than merely your comfort. What an examination can find The TFOS review of the digital environment was clear about the first step in managing digital eye strain: people should be given a full refractive correction for the distances at which they actually work. That sounds obvious, yet a great many students with persistent strain have never had a proper examination as adults, or last had one years ago, or have a prescription intended for distance that was never checked for close work. Four kinds of problem are commonly found. The first is uncorrected or under-corrected refractive error. Mild myopia rarely causes strain at the screen, because a short-sighted eye sees near objects well; it causes blur at a distance, in lectures and when driving. Long-sightedness, or hyperopia, is different. A young hyperopic eye can usually overcome its error by focusing harder, so distance vision may be perfectly clear and the person may pass a quick screening test with ease. The cost is paid at near, where the eye has to focus harder still, all day. Hyperopia is a classic hidden cause of tired eyes and headaches during reading, and it is easily missed if nobody measures it properly. Astigmatism, in which the eye focuses light unevenly in different directions, can produce similar symptoms even in small amounts, as the eye continually adjusts in an attempt to sharpen an image that cannot be fully sharpened. And a prescription that was correct three years ago may no longer be correct now, particularly in a young myope whose eyes are still changing. The second is a problem with the way the two eyes work together. The most common in students is convergence insufficiency, in which the eyes have difficulty turning inwards and holding together at near. Symptoms include tired eyes and headaches during reading, words that seem to move or run together, losing one's place, needing to reread lines, occasional double vision at near, and a tendency to close or cover one eye to read more comfortably. These symptoms appear quickly, often within minutes of starting to read, and return quickly after breaks. Convergence insufficiency is diagnosed with simple tests of how close the eyes can converge and how well they hold together, and it responds to treatment. The Convergence Insufficiency Treatment Trial, a series of randomised trials funded by the US National Eye Institute, showed that supervised office-based vergence therapy, with exercises at home, was effective in children and adolescents; evidence in adults is more limited but the condition is still worth identifying. Other binocular problems include small misalignments of the eyes that the brain must work to overcome. The third is a problem with accommodation itself. Some young people have reduced focusing ability for their age, or a focusing system that is slow to relax, so that distant objects stay blurred for a while after a period of close work. These can be identified by measuring the amplitude and flexibility of focusing, and a reading prescription or specific exercises may help. The fourth is ocular surface disease, discussed in Chapter 7: dry eye, meibomian gland dysfunction, blepharitis, allergic conjunctivitis, or problems related to contact lens wear. A clinician can see signs that you cannot, such as blocked glands, staining of the cornea, or changes on the inside of the eyelids from lens wear. For some students the outcome of an examination is a pair of glasses prescribed specifically for screen and reading distances. These are sometimes called computer glasses, which is a perfectly legitimate idea when they are prescribed to correct a real need at those distances, and quite different from the tinted, non-prescription "blue-light" glasses discussed in Chapter 5. The value lies in the prescription, not the coating. How often should a student be examined? Recommendations vary between countries and professional bodies, and depend on age, prescription and health. As a general guide, a young adult with no symptoms and no known eye problems benefits from an examination every couple of years, while anyone who wears glasses or contact lenses, has a changing prescription, has diabetes or another condition affecting the eyes, or has persistent symptoms should be seen more often, as advised by their optometrist. A student who is starting university without having had an eye examination for several years, and who is about to spend far more time reading than ever before, has a good reason to book one. Getting care as a student Eye care is usually provided by two kinds of professional. Optometrists carry out eye examinations, prescribe glasses and contact lenses, and diagnose and in many places treat common eye conditions, including dry eye and many infections; they refer to a specialist when needed. Ophthalmologists are medical doctors who specialise in eye disease and surgery. For most of the concerns in this book, an optometrist is the right first port of call. In some countries, orthoptists specialise in binocular vision problems such as convergence insufficiency. Opticians fit and dispense glasses but do not usually examine eyes. How you access and pay for this care depends heavily on where you are. In the United States, routine eye examinations and glasses are often covered by separate vision insurance rather than by medical insurance, while the medical treatment of eye disease, such as an infection or an injury, is usually billed to medical insurance. Many student health plans include some vision benefits, and some universities, particularly those with optometry schools, run clinics that offer examinations at reduced cost. In other countries arrangements differ considerably. In some, eye examinations are free or subsidised for young people or for all, while in others they are paid privately. Your campus health service is the best place to find out what applies to you and where to go. It is also worth knowing what campus health services can and cannot do. A general health clinic can assess and treat many problems of the eyelids and conjunctiva, such as a simple conjunctivitis or a stye, and can advise on urgency. It usually cannot do a full refraction or a detailed examination of the back of the eye, and will refer you to an optometrist or ophthalmologist if these are needed. For contact lens problems, an optometrist is usually better placed than a general clinic. It helps to distinguish an eye examination from a vision screening. The quick test at a driving licence office, a school nurse's chart or a workplace check measures how well you can read letters at a distance. That is useful but narrow. It will not detect low hyperopia that a young eye is compensating for, a binocular vision problem that only shows up at near, or early signs of eye disease. A full examination includes measurement of the prescription at distance and near, assessment of how the eyes work together and focus, examination of the front of the eye under magnification, and examination of the retina and optic nerve at the back of the eye. If you have passed a screening but still have symptoms, the screening has not ruled anything out. A little preparation makes an appointment far more useful. Bring your current glasses and, if you wear them, your contact lenses and their details. Before you go, measure the distances at which you actually work: from your eyes to your laptop screen, to any external monitor, and to your phone as you usually hold it. A tape measure takes a minute, and the numbers let the optometrist check your vision at the distances that matter to you rather than at a standard reading distance that may not match your habits. Write down your symptoms in the terms this book has used. Are they external, like burning and grittiness, or internal, like aching and blur? When do they start in a session, and what relieves them? Do they differ between screens and paper, between morning and evening, between contact lenses and glasses? Note any medications you take, including acne treatments and antihistamines, and any family history of high myopia, glaucoma or other eye disease. Be aware that some examinations involve drops to widen the pupils, which allow a better view of the retina. These can leave your vision blurred at near and sensitive to light for several hours. If you have an exam, a deadline or a long drive the same day, ask when you book whether dilation is likely and plan accordingly; sunglasses help on the way home. Dilation is worth the inconvenience when it is recommended, particularly for students with significant myopia or new floaters. Symptoms that need prompt or urgent attention Most eye discomfort is not an emergency. But a small number of symptoms mean something more serious may be going on, and in those cases acting quickly can protect your sight. Table 5 sets out the main warning signs and how quickly to seek care. Table 5. Eye symptoms that need more than self-care, and how quickly to act. Symptom Possible concern How quickly to seek care Sudden loss or dimming of vision in one or both eyes Problems of the retina, optic nerve or circulation Immediately: emergency care New flashes of light, a sudden shower of floaters, or a shadow or curtain across vision Retinal tear or detachment Same day, urgently Chemical splash in the eye Chemical injury Rinse with clean water at once, then emergency care Eye injury, or something hitting the eye at speed Penetrating injury, internal damage Immediately: emergency care Painful red eye with light sensitivity or blurred vision, especially in a contact lens wearer Corneal infection or inflammation Same day New double vision, or headache with changes in vision Neurological or other serious causes Same day, or emergency care if sudden or severe Persistent strain, dryness or blur despite the measures in this book Refractive error, binocular vision problem, dry eye disease Routine appointment within weeks Some of these deserve explanation. Flashes of light and new floaters are worth special mention for students, because myopia, which is so common in this population, raises the risk of retinal tears and detachment. Most floaters are harmless, the familiar drifting specks and threads that many people notice against a bright sky. What matters is a sudden change: a burst of new floaters, flashes of light in the side of your vision, or a shadow that seems to spread across part of the visual field like a curtain. These need examination the same day, because a retinal tear caught early can often be treated simply, while a detachment left untreated can lead to permanent loss of vision. For a chemical splash, including cleaning products and laboratory chemicals, the first step is to rinse the eye immediately with plenty of clean water for at least fifteen minutes, holding the eyelids open, and then to seek emergency care. Laboratories should have eye-wash stations, and students who work in them should know where they are before they need one. Prevention matters here as much as response. Many eye injuries among students happen in laboratories, workshops, studios and sports, and most are avoidable with the right eye protection. Safety glasses in a chemistry or engineering laboratory are not a formality, and ordinary prescription glasses do not count as protection, since their lenses and frames are not designed to stop a splash or a projectile. Sports with small fast balls or rackets, such as squash, carry a particular risk of serious eye injury, and protective eyewear made for the sport is worth wearing. Fireworks, bungee cords and aerosol cans pointed the wrong way account for a steady stream of emergency visits every year. A painful red eye in a contact lens wearer is the scenario that most often goes wrong through delay. It is tempting to put in some drops and hope it settles by morning. If there is real pain, sensitivity to light or a drop in vision, remove the lenses, keep them and the case, since they can help identify an infection, and be seen the same day. Corneal infections can progress quickly, and early treatment makes a large difference to the outcome. For any symptom that seems like an emergency, particularly sudden loss of vision, serious injury or sudden severe headache with visual disturbance, go to an emergency department or call emergency services. In the United States the number is 911; other countries use different numbers, and it is worth knowing yours. When it is not the eyes at all Headaches are among the commonest complaints attributed to screens, and eye strain can certainly cause a mild ache around the brow and temples that eases with rest. But many headaches experienced during study have other causes. Tension-type headache, the most common kind, is often associated with stress, poor sleep, sustained posture and tight neck and shoulder muscles, all of which are abundant in exam season. Migraine, which is common in young adults and particularly in young women, can be triggered by sleep disruption, skipped meals, stress and sometimes bright or flickering light, and may be preceded by visual disturbances such as zigzag lines or shimmering blind spots. Dehydration, caffeine withdrawal and hunger all contribute. A student who attributes every headache to screens may miss the pattern, and a treatable condition, entirely. A reasonable rule of thumb is that eye strain headaches are mild, build during close work, and ease with rest and a break from the screen. Headaches that are severe, that wake you from sleep, that come with visual disturbances, numbness, weakness or confusion, that are the worst you have ever had, or that have changed in pattern, should be assessed by a doctor rather than managed with better screen habits. Recurrent headaches that fit a migraine pattern are worth discussing with a general practitioner or campus physician, since effective treatments exist. Similarly, blurred vision is not always strain. Blur that clears after a few seconds of blinking is likely to come from the tear film. Blur that clears within a minute or two of looking into the distance is likely to reflect focusing fatigue. Blur that persists, affects one eye more than the other, or is present first thing in the morning, needs an examination. Consider a final-year student who has been having headaches most afternoons for a term. She has adjusted her screen, bought a stand, and follows the 20-20-20 rule faithfully; the headaches have not changed. When she keeps a simple diary for two weeks, recording sleep, meals, caffeine and screen time alongside each headache, a pattern emerges: the headaches come on days when she has slept less than six hours and skipped lunch, regardless of how long she has been at her laptop. An eye examination finds her eyes healthy and her prescription unchanged. Her doctor diagnoses tension-type headache and helps her tackle the real triggers. Her screen habits were worth improving anyway, but they were never the heart of the problem. That example captures the spirit of this chapter. Good screen habits prevent most eye strain, but they are not a cure-all, and the willingness to recognise when something else is going on, and to get it examined, is part of looking after your eyes. The best thing most students can do for their vision, beyond the habits in this book, is to book an eye examination. Conclusion: The Session, Not the Screen This book began with a student at her laptop at eleven at night, eyes burning, wondering whether she was harming her sight and which of the contradictory pieces of advice she had seen was true. The chapters since have given her the material for an answer, and the answer is more interesting than either the alarm or the reassurance she had been offered. It is not that screens are dangerous, and it is not that everything is fine. It is that the unit that matters for the eyes is not the device but the study session: its length, its rhythm, its geometry, its air and its light, and what happens between one session and the next. What follows from the evidence Put the chapters side by side and a pattern emerges that none of them shows alone. The mechanisms of short-term discomfort and the risk factors for the one long-term structural change associated with study turn out to be tied to the same few features of how a student works. Blinking falls and becomes incomplete during long, concentrated stretches of close work. Focusing fatigue accumulates during long, uninterrupted stretches at one near distance. Myopia, in children and probably in some young adults, is associated with close, continuous near work and with too little time in daylight. Evening sleep is disturbed by late, bright, engaging screen use. In each case the problem is not a property of the screen but of the pattern: long, close, continuous, indoors and late. That convergence has a practical consequence that is easy to miss if you read about each problem separately. You do not need a separate strategy for dry eyes, another for eye strain, another for myopia and another for sleep. One well-designed study routine addresses all of them. Work in blocks rather than marathons. Look far away, preferably out of a window, at the end of each block, and blink fully while you do it. Place the screen at arm's length, slightly below the eyes, with text large enough to read without effort, in light that matches it, away from moving air. Take real breaks, and take some of them outdoors in daylight. Finish demanding screen work well before bed. None of these steps is dramatic. Taken together, they change the conditions that produce nearly every eye complaint of the studious life. The same convergence explains why the products most heavily marketed to students are the least useful. A blue-light filter acts on the colour of the light, which is not the mechanism of strain, not a meaningful risk to the retina at screen intensities, and only a modest factor in sleep compared with timing and content. An omega-3 capsule acts on a pathway that the largest trial found did not change dry eye. A new gadget cannot interrupt a four-hour stare. The pattern of work can. The money a student might spend on specialised eyewear is better spent on an eye examination, a laptop stand and a keyboard, and the time better spent outdoors. It follows, too, that the right response to persistent symptoms is investigation, not endurance. Because the ordinary causes of strain respond so reliably to changes in routine and setup, symptoms that do not respond are informative. They point to an uncorrected prescription, a binocular vision problem, dry eye disease, a medication effect, a contact lens problem or a cause outside the eyes altogether. The student who adjusts everything in this book and still has aching eyes after fifteen minutes of reading has not failed at self-care. She has gathered the evidence that justifies an examination, and that examination is likely to find something that can be treated. What remains unsettled A book that insists on honesty about evidence should end by marking where the evidence stops. The 20-20-20 rule has been tested directly in only a small study, which found that it reduced symptoms while people followed it, that people stopped following it when the reminders stopped, and that two weeks was too short to show changes in the ocular surface. Whether regular breaks, sustained over months and years, change the health of the tear film or the course of dry eye is not known. Neither is the best interval or duration of a break. The relationship between study and myopia in young adults is real but poorly quantified. The genetic evidence that education causes myopia is strong; the evidence on which components of education matter most, and on whether outdoor time can slow progression after the teenage years, is thin. Myopia control treatments that have transformed paediatric practice have barely been tested in university-age students. It is entirely possible that the next decade will show that adult myopia progression is more modifiable than currently assumed, or less. On blue light and sleep, the laboratory effects are robust but the real-world size of the effect of ordinary evening screen use, separated from what people are doing on those screens, remains uncertain and probably varies from person to person. Filtering lenses have been tested only in small, short and often biased trials, so the conclusion that they probably do not help with eye strain is a statement about the best current evidence, not a final verdict. And the treatment of digital eye strain generally, as the TFOS review put it, is not well established. Artificial tears have not been compared rigorously for screen users; humidifiers and warm compresses look promising in small trials. A good deal of the advice in this book rests on physiology and plausibility rather than on large randomised trials, which is why it has favoured measures that are free, reversible and harmless. What to do differently If the argument of this book is right, a few changes follow, some for individual students and some for the institutions around them. For students, the most important shift is to stop thinking of eye care as something done to the eyes, with drops, glasses or filters, and to start thinking of it as something built into the way study is organised. The eyes are a good guide to whether a routine is sustainable. Burning, aching and blur at the end of a session are signals that the session was too long, too close, too dry or too still, and they are signals worth heeding before they become habits. For universities, the argument points to things that students cannot fix alone. Libraries and study spaces that place desks by windows, provide outdoor study areas, keep air reasonably humid and away from faces, and offer adjustable monitors or laptop stands make good eye habits the default rather than an effort. Timetables that allow breaks between long teaching blocks help. Student health services that make eye examinations easy to access, and that publicise the warning signs in Chapter 8, catch problems early. None of this is expensive, and all of it supports study as well as sight. The student at her laptop at eleven at night has one more thing to do before she closes it. She can lower the screen, turn on the lamp, look out of the window for twenty seconds, blink, and decide to stop in half an hour rather than two. Tomorrow she can walk to the library the long way, sit where she can see the sky, and book the eye examination she has been putting off. Her eyes will thank her in the only way eyes can: by staying quiet, and letting her get on with her work. Notes Introduction Wolffsohn, J. S., Lingham, G., Downie, L. E., et al. (2023). TFOS Lifestyle: Impact of the digital environment on the ocular surface. The Ocular Surface, 28, 213-252. Singh, S., Keller, P. R., Busija, L., et al. (2023). Blue-light filtering spectacle lenses for visual performance, sleep, and macular health in adults. Cochrane Database of Systematic Reviews, 8, CD013244. Chapter 1: What Eye Strain Is, and What It Is Not Wolffsohn, J. S., et al. (2023), cited above. Source of the formal definition of digital eye strain and of the observation that reported prevalence ranges up to 97 per cent. Sheppard, A. L., and Wolffsohn, J. S. (2018). Digital eye strain: prevalence, measurement and amelioration. BMJ Open Ophthalmology, 3(1), e000146. Source of the estimate of 50 per cent or more among computer users and of the division of symptoms into internal and external groups. Rosenfield, M. (2011). Computer vision syndrome: a review of ocular causes and potential treatments. Ophthalmic and Physiological Optics, 31(5), 502-515. Chapter 2: The Blink Craig, J. P., Nichols, K. K., Akpek, E. K., et al. (2017). TFOS DEWS II Definition and Classification Report. The Ocular Surface, 15(3), 276-283. Tsubota, K., and Nakamori, K. (1993). Dry eyes and video display terminals. New England Journal of Medicine, 328(8), 584. Chu, C. A., Rosenfield, M., and Portello, J. K. (2014). Blink patterns: reading from a computer screen versus hard copy. Optometry and Vision Science, 91(3), 297-302. Source of the blink rates of 14.9 and 13.6 per minute and incomplete blink proportions of about 7 and 4 per cent. Portello, J. K., Rosenfield, M., and Chu, C. A. (2013). Blink rate, incomplete blinks and computer vision syndrome. Optometry and Vision Science, 90(5), 482-487. Source of the mean blink rate of 11.6 per minute, the incomplete blink range and the audible-tone experiment. Chapter 3: The 20-20-20 Rule Under Scrutiny Talens-Estarelles, C., Cerviño, A., García-Lázaro, S., Fogelton, A., Sheppard, A., and Wolffsohn, J. S. (2023). The effects of breaks on digital eye strain, dry eye and binocular vision: Testing the 20-20-20 rule. Contact Lens and Anterior Eye, 46(2), 101744 (published online 2022). Galinsky, T. L., Swanson, N. G., Sauter, S. L., Hurrell, J. J., and Schleifer, L. M. (2000). A field study of supplementary rest breaks for data-entry operators. Ergonomics, 43(5), 622-638. Galinsky, T., Swanson, N., Sauter, S., Dunkin, R., Hurrell, J., and Schleifer, L. (2007). Supplementary breaks and stretching exercises for data entry operators: a follow-up field study. American Journal of Industrial Medicine, 50(7), 519-527. Ciuffreda, K. J., and Vasudevan, B. (2008). Nearwork-induced transient myopia (NITM) and permanent myopia: is there a link? Ophthalmic and Physiological Optics, 28(2), 103-114. Chapter 4: Building the Desk Around Your Eyes American Optometric Association. Computer vision syndrome (digital eye strain). Patient information available at aoa.org. Source of the 20 to 28 inch viewing distance and the 15 to 20 degree screen position. US Occupational Safety and Health Administration. Computer Workstations eTool. Available at osha.gov. Source of the 20 to 40 inch range and the guidance on screen height and placement relative to windows. Piepenbrock, C., Mayr, S., Mund, I., and Buchner, A. (2013). Positive display polarity is advantageous for both younger and older adults. Ergonomics, 56(7), 1116-1124. Chapter 5: Blue Light Without the Marketing O'Hagan, J. B., Khazova, M., and Price, L. L. A. (2016). Low-energy light bulbs, computers, tablets and the blue light hazard. Eye, 30(2), 230-233. Singh, S., et al. (2023), cited above. Source of the description of seventeen trials and their conclusions on visual fatigue, visual acuity, sleep and macular health. Wolffsohn, J. S., et al. (2023), cited above. Source of the conclusion that blue-light blocking interventions do not appear to be an effective management strategy for digital eye strain. Chang, A.-M., Aeschbach, D., Duffy, J. F., and Czeisler, C. A. (2015). Evening use of light-emitting eReaders negatively affects sleep, circadian timing, and next-morning alertness. Proceedings of the National Academy of Sciences, 112(4), 1232-1237. Duraccio, K. M., Zaugg, K. K., Blackburn, R. C., and Jensen, C. D. (2021). Does iPhone night shift mitigate negative effects of smartphone use on sleep outcomes in emerging adults? Sleep Health, 7(4), 478-484. Chapter 6: Near Work, Daylight and Myopia Holden, B. A., Fricke, T. R., Wilson, D. A., et al. (2016). Global prevalence of myopia and high myopia and temporal trends from 2000 through 2050. Ophthalmology, 123(5), 1036-1042. Bullimore, M. A., and Brennan, N. A. (2019). Myopia control: why each diopter matters. Optometry and Vision Science, 96(6), 463-465. Mountjoy, E., Davies, N. M., Plotnikov, D., et al. (2018). Education and myopia: assessing the direction of causality by mendelian randomisation. BMJ, 361, k2022. Huang, H.-M., Chang, D. S.-T., and Wu, P.-C. (2015). The association between near work activities and myopia in children: a systematic review and meta-analysis. PLoS One, 10(10), e0140419. Ip, J. M., Saw, S.-M., Rose, K. A., et al. (2008). Role of near work in myopia: findings in a sample of Australian school children. Investigative Ophthalmology and Visual Science, 49(7), 2903-2910. Foreman, J., Salim, A. T., Praveen, A., et al. (2021). Association between digital smart device use and myopia: a systematic review and meta-analysis. The Lancet Digital Health, 3(12), e806-e818. He, M., Xiang, F., Zeng, Y., et al. (2015). Effect of time spent outdoors at school on the development of myopia among children in China: a randomized clinical trial. JAMA, 314(11), 1142-1148. Schmid, K. L., Leyden, K., Chiu, Y.-H., et al. (2013). Assessment of daily light and ultraviolet exposure in young adults. Optometry and Vision Science, 90(2), 148-155. Kinge, B., Midelfart, A., Jacobsen, G., and Rystad, J. (2000). The influence of near-work on development of myopia among university students: a three-year longitudinal study among engineering students in Norway. Acta Ophthalmologica Scandinavica, 78(1), 26-29. Bai, X., Jin, N., Wang, Q., et al. (2022). Development pattern of ocular biometric parameters and refractive error in young Chinese adults: a longitudinal study of first-year university students. BMC Ophthalmology, 22(1), 220. Chapter 7: Keeping the Ocular Surface Healthy Craig, J. P., et al. (2017), cited above. Source of the definition of dry eye disease. Wolffsohn, J. S., et al. (2023), cited above. Source of the assessment of artificial tears, humidifiers, warm compresses and nutritional supplements. Dry Eye Assessment and Management Study Research Group; Asbell, P. A., Maguire, M. G., Pistilli, M., et al. (2018). n-3 fatty acid supplementation for the treatment of dry eye disease. New England Journal of Medicine, 378(18), 1681-1690. Craig, J. P., Alves, M., Wolffsohn, J. S., et al. (2023). TFOS Lifestyle Report Introduction: A lifestyle epidemic, ocular surface disease. The Ocular Surface, 28, 304-309. The TFOS Lifestyle series includes separate reports on cosmetics, contact lenses and elective medications, freely available from the Tear Film and Ocular Surface Society. Gomes, J. A. P., Azar, D. T., Baudouin, C., et al. (2023). TFOS Lifestyle: Impact of elective medications and procedures on the ocular surface. The Ocular Surface, 29, 331-385. Stapleton, F., Keay, L., Edwards, K., et al. (2008). The incidence of contact lens-related microbial keratitis in Australia. Ophthalmology, 115(10), 1655-1662. Chapter 8: Warning Signs and the Examination Wolffsohn, J. S., et al. (2023), cited above. Source of the recommendation that people with digital eye strain receive a full refractive correction for their working distances. Convergence Insufficiency Treatment Trial Study Group (2008). Randomized clinical trial of treatments for symptomatic convergence insufficiency in children. Archives of Ophthalmology, 126(10), 1336-1349. Further Reading Tear Film and Ocular Surface Society, TFOS Lifestyle reports (2023), published in The Ocular Surface and freely available at tearfilm.org.** The most thorough recent review of how everyday life affects the front of the eye. The report on the digital environment is the single best technical source for Chapters 1 to 3; others cover contact lenses, cosmetics, medications, nutrition and environmental conditions. Singh, S., and colleagues, "Blue-light filtering spectacle lenses for visual performance, sleep, and macular health in adults", Cochrane Database of Systematic Reviews (2023).** A careful, readable summary of the trial evidence behind Chapter 5, with a plain-language summary at the start. Freely available. Sheppard, A. L., and Wolffsohn, J. S., "Digital eye strain: prevalence, measurement and amelioration", BMJ Open Ophthalmology (2018).** A short, open-access review for readers who want a clinician's overview of digital eye strain. International Myopia Institute, white papers (published in Investigative Ophthalmology and Visual Science from 2019 onward, and at myopiainstitute.org).** Consensus reports on the definition, causes, prevention and management of myopia, written for professionals but accessible to a determined student. The best place to follow developments in myopia control. American Optometric Association, "Computer vision syndrome (digital eye strain)", at aoa.org. A concise patient guide to screen setup and eye care, including the distance and height recommendations in Chapter 4. US Occupational Safety and Health Administration, Computer Workstations eTool, at osha.gov. Practical, illustrated guidance on arranging a desk, chair, keyboard and monitor. Written for workplaces but directly applicable to a student desk. US Centers for Disease Control and Prevention, "Healthy Contact Lens Wear", at cdc.gov. Clear guidance on safe contact lens habits, including water exposure, case care and sleeping in lenses. Readers outside the United States will find similar guidance from their national optometric or ophthalmological bodies. Jeffrey Anshel, Visual Ergonomics Handbook (CRC Press, 2005).** A practical book on vision in the workplace by the optometrist to whom the 20-20-20 rule is commonly attributed. Some of its technical detail on displays has dated, but its principles have not. Glossary Accommodation. The eye's ability to change focus for near objects, produced by the ciliary muscle allowing the lens to become more curved. Astigmatism. A refractive error in which the eye focuses light unevenly in different directions, usually because the cornea is not equally curved in all directions. Convergence insufficiency. A common binocular vision problem in which the eyes have difficulty turning inward and holding together for close work, causing strain, headaches and sometimes double vision when reading. Dioptre. The unit of focusing power. The focusing demand of a target is the reciprocal of its distance in metres, so a screen at half a metre requires two dioptres of accommodation. Dry eye disease. A chronic condition of the ocular surface in which the tear film becomes unstable, leading to symptoms, inflammation and surface damage in a self-perpetuating cycle. Hyperopia. Long-sightedness. A young hyperopic eye can often see clearly at a distance by focusing harder, but at the cost of extra effort for near work. Incomplete blink. A blink in which the upper lid does not come down far enough to cover the whole cornea, leaving the lower part unrefreshed. Meibomian glands. Small glands along the edges of the eyelids that secrete the oily outer layer of the tear film, which slows evaporation. Meibomian gland dysfunction is the most common cause of evaporative dry eye. Microbial keratitis. An infection of the cornea, most often associated with contact lens wear, which can scar the cornea and threaten vision if not treated promptly. Myopia. Short-sightedness, usually caused by elongation of the eyeball, so that distant objects are focused in front of the retina and appear blurred. Tear break-up time. The time between a blink and the first appearance of a dry spot in the tear film, used by clinicians as a measure of tear film stability. Hashtags: #ScreenTime #EyeStrain #DigitalEyeStrain #ComputerVisionSyndrome #EyeHealth #VisionProtection #DryEyePrevention #DryEyeSyndrome #HealthyStudyHabits #StudentWellness #StudyHealth #DigitalWellness #ScreenTimeManagement #TwentyTwentyTwentyRule #EyeCareTips #HealthyVision #BlinkingExercises #TearFilmHealth #OcularSurfaceHealth #VisualFatigue #EyeStrainRelief #ScreenErgonomics #BlueLightAwareness #ComputerEyeStrain #MyopiaPrevention #ContactLensSafety #EyeHydration #HealthyScreenHabits #StudyErgonomics #PreventiveEyeCare

Download the Book (PDF): Introduction A child who hears the word "dog" while looking at a dog is doing something that took machine learning roughly sixty years to imitate convincingly. She is not processing a sound and then, separately, processing an image, and then consulting a lookup table that connects the two. She is building a single internal sense of what a dog is, one that can be reached through the sound of barking, the sight of a tail, the feel of fur, or the written word on a page. When she later reads "the dog hid under the table", she can picture the scene. When she sees a dog under a table, she can say so. For most of the history of artificial intelligence, systems did not work this way. Speech recognisers, image classifiers and language models were built by different research communities, trained on different data, evaluated on different benchmarks and published at different conferences. An image classifier trained on ImageNet could tell you that a photograph contained a "golden retriever", but the label was just an integer that happened to have a name attached. The model had no idea that the word "retriever" had anything to do with the word "fetch", and a language model that knew all about fetching had never seen a pixel. That separation has collapsed. The most capable AI systems of the mid-2020s accept photographs, screenshots, diagrams, audio recordings and video alongside text, and they answer in text or speech. A user can photograph a broken bicycle derailleur and ask how to adjust it, hold a phone up to a menu in a foreign language and ask what to order, or upload a recorded meeting and ask what was decided. These systems are routinely described as "foundation models", and the qualifier "multimodal" is increasingly dropped because it is assumed. This booklet explains how that happened and how these systems actually work. It is written for readers who are comfortable with the idea of a neural network, who may have used a large language model, and who want to understand what is happening inside the systems that look at pictures and listen to sound. It assumes no specialist background in computer vision or speech processing, and it explains its terms as it goes. The argument The central claim of this booklet is simple to state and has consequences that take the whole book to work through. Multimodal deep learning is, at bottom, the problem of alignment: getting signals that arrive in utterly different forms into representations that can be compared with one another and can attend to one another. Every major architecture in the field is a particular answer to two questions. Where in the network should the different modalities meet? And what training signal should teach them to agree? The answers matter because they determine what a system can and cannot do. A model that aligns an entire image with an entire caption, as the influential CLIP model does, learns a remarkably general sense of what images are about. It can sort photographs into categories it was never explicitly trained on, retrieve images from text descriptions, and serve as the visual front end for larger systems. But the same design makes it surprisingly poor at telling "a mug on a laptop" from "a laptop on a mug", because it was never asked to align the parts of an image with the parts of a sentence. A model that lets every word attend to every region of an image, through the mechanism called cross-attention, can in principle make those distinctions, but it pays for that capacity in computation and in the quantity of carefully paired data it needs. So the story of the field is not a march from worse architectures to better ones. It is a series of trade-offs about where alignment is learned and how finely. Coarse alignment is cheap, scales to billions of noisy examples scraped from the web, and generalises broadly. Fine alignment, the kind needed to follow an instruction like "count the red cups to the left of the sink", is expensive, data-hungry and still incompletely solved. The systems that reason most convincingly over visual scenes and natural-language instructions are hybrids: they borrow a coarsely aligned visual encoder, connect it to a large language model through a thin bridge, and then teach the combined system, with relatively small quantities of carefully constructed data, to ground its words in what it sees. Understanding this trade-off is the most useful single thing a reader can take away. It explains why multimodal models are brilliant at describing a photograph's mood and unreliable at counting objects in it. It explains why they "hallucinate" objects that are not there. It explains why adding audio to a vision-language model is not merely a matter of bolting on another encoder. And it gives a practitioner a principled way to think about which kind of model suits which job. What the booklet covers The book runs in a deliberate order, from raw signals to reasoning systems. Chapter 1 sets out the problem. It explains what makes a "modality" distinct, why combining them is hard, and the classic choices between fusing early, fusing late and fusing somewhere in between. It introduces the vocabulary the rest of the book uses. Chapter 2 is about the first step in any multimodal system: turning an image, a sentence or a sound into a sequence of vectors that a neural network can process. The fact that all three can now be reduced to "tokens" of the same shape is the quiet technical change that made modern multimodal systems possible. Chapter 3 explains attention, and specifically cross-attention, in enough detail that the reader could implement it. Cross-attention is the mechanism that lets one modality query another, and it is the hinge on which most fusion architectures turn. Chapter 4 covers contrastive pretraining, the training method behind CLIP and its successors. It explains how a model can learn to align images and text from nothing more than the fact that a particular caption appeared next to a particular picture on the web, and why that turned out to be one of the most consequential ideas in the field. Chapter 5 examines what contrastive models fail to learn. These failures are not incidental; they follow directly from the training objective, and understanding them is the key to understanding the limitations of the larger systems built on top. Chapter 6 describes how visual encoders are connected to large language models to produce systems that can converse about images. It compares the main connection strategies, from the gated cross-attention layers of DeepMind's Flamingo to the startlingly simple linear projection of LLaVA, and explains the role of visual instruction tuning. Chapter 7 turns to sound: speech recognition at scale, contrastive audio-text models, models that produce speech directly, and attempts to bind many modalities into a single shared space. Chapter 8 addresses the hardest and most practically important question: what it takes for a system to reason over a visual scene in response to a natural-language instruction, how such reasoning is measured, why models hallucinate, and where the frontier now lies. The conclusion draws out what follows from the whole argument for anyone building, choosing or evaluating these systems. A note on names and numbers The field moves quickly, and model names come and go. This booklet names specific systems when they introduced an idea that has lasted, not because they are the current state of the art; by the time anyone reads this, newer models will exist. The ideas, by contrast, have proved remarkably durable. The contrastive objective in CLIP, published in 2021, still underlies many of the vision encoders inside today's assistants. The cross-attention mechanism dates from the original Transformer paper of 2017, and before that from attention in machine translation. The trick of projecting visual features into a language model's input space, popularised in 2023, remains a standard pattern. Where the book quotes a number, such as the size of a training set or a benchmark result, the figure is the one reported in the original publication. Benchmark numbers in particular should be read as illustrations of scale and direction rather than as permanent facts; they are superseded constantly. The aim is to give the reader a durable understanding of the mechanisms, so that each new system can be read as another answer to the same two questions: where should the modalities meet, and what should teach them to agree? CHAPTER 1 The Problem of Many Senses In 1976 the psychologists Harry McGurk and John MacDonald published a short paper in Nature with the memorable title "Hearing lips and seeing voices". They had dubbed a recording of a person saying the syllable "ba" onto a video of the same person's lips forming "ga". Most viewers reported hearing neither. They heard "da", a sound that existed in neither the audio nor the video. Close your eyes and the sound becomes "ba" again; open them and it returns to "da". The McGurk effect is the most famous demonstration that human perception does not process the senses independently and then compare notes. Vision and hearing are fused early, below the level of conscious awareness, into a single percept that can differ from what either sense reports on its own. The brain is not taking a vote between two separate classifiers. It is building one representation from two streams of evidence. This is a useful place to start because it frames the central engineering question of multimodal deep learning in its sharpest form. When a system receives an image and a sentence, or a video and its soundtrack, should it process each input separately and combine the conclusions? Should it merge the raw inputs at once? Or should it do something in between, allowing each stream to develop its own representation for a while and then letting them interact? The answer shapes everything else about the system: what it can learn, what data it needs, how much it costs to run, and where it fails. What makes a modality The word "modality" is used loosely, and it is worth being precise. In machine learning, a modality is a type of signal with its own characteristic structure: its own way of being sampled, its own statistics, its own notion of what counts as a neighbouring element. Images, text, speech, non-speech audio, video, depth maps, thermal imagery, tabular measurements and the readings of an accelerometer are all modalities in this sense. So, for some purposes, are code, molecular structures and protein sequences. What makes modalities hard to combine is that they differ in almost every respect that matters to a learning system. Consider four dimensions of difference. Structure. An image is a two-dimensional grid of pixel intensities in which nearby pixels are strongly correlated and meaning is mostly local: an edge here, a texture there, assembled into objects. Text is a one-dimensional sequence of discrete symbols in which meaning can depend on words far apart ("The trophy did not fit in the suitcase because it was too big"). Audio is a one-dimensional sequence of continuous amplitude values sampled tens of thousands of times a second, in which meaning lives in frequencies and their change over time. Video adds a time axis to images, and usually an audio track as well. Density of information. A single second of CD-quality stereo audio contains 88,200 numbers. A modest 224 by 224 colour image contains about 150,000. A sentence describing that image might contain twelve words. Raw images and sound are enormously redundant; most of their numbers can be predicted from their neighbours. Text is the opposite: dense, abstract and already compressed by the human who wrote it. A model fusing the two must reconcile a firehose with a trickle. Discreteness. Text arrives in discrete units drawn from a finite vocabulary. Images and audio are, for practical purposes, continuous. Many of the most successful techniques in language modelling, such as predicting the next token from a fixed vocabulary, do not apply directly to continuous signals. A great deal of ingenuity in the field has gone into making continuous signals look discrete, or making discrete models handle continuous inputs. Correspondence. Even when two modalities describe the same event, the correspondence between their parts is usually loose. A caption that says "a dog catching a frisbee in a park" does not tell you which pixels are the dog and which are the frisbee. A transcript of a lecture does not tell you which slide was on screen when each sentence was spoken. The alignment between parts of one modality and parts of another is typically hidden, and learning it is one of the field's central problems. These differences are why, for decades, each modality had its own specialist community with its own architectures: convolutional networks for vision, recurrent networks and later Transformers for language, hidden Markov models and later deep networks for speech. Each architecture encoded assumptions, known as inductive biases, that suited its modality's structure. Convolutional networks assume that local patterns matter and that a pattern means the same thing wherever it appears in an image. Recurrent networks assume that information arrives in order. Getting these architectures to talk to each other was awkward because they did not share a common currency. Five problems, not one In an influential survey published in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2019, Tadas Baltrušaitis, Chaitanya Ahuja and Louis-Philippe Morency proposed a taxonomy of the core challenges in multimodal machine learning. The taxonomy has aged well, and it is a useful map because it shows that "combining modalities" is really several distinct problems that are easily confused. The first is representation: how to encode each modality, and possibly several together, as vectors a model can work with. Representations can be joint, where different modalities are projected into a single shared space, or coordinated, where each modality has its own space but the spaces are constrained to relate to one another, for example by making the vector for an image close to the vector for its caption. The second is translation, or mapping: generating one modality from another. Image captioning translates images to text; text-to-image generation goes the other way; speech recognition translates audio to text; speech synthesis does the reverse. The third is alignment: identifying which parts of one modality correspond to which parts of another. Matching the words in a caption to the regions of an image is alignment; so is matching the words in a transcript to the moments in an audio recording. The fourth is fusion: combining information from several modalities to make a prediction. Deciding whether a video shows a person laughing, using both their face and the sound they make, is a fusion problem. The fifth is co-learning: using one modality to help learn another, especially when one is data-poor. Using abundant text to help train a model of a modality with few labels is co-learning. The value of the taxonomy is that it makes clear these problems interact but are not the same. A system can be excellent at coordinated representation and poor at fine-grained alignment, and Chapter 5 shows that this is precisely the situation of contrastively trained models such as CLIP. A system can fuse modalities well for classification and still be unable to translate between them. When people say a model "understands" images and text, it is worth asking which of these five things they mean. A later and more expansive survey by Paul Pu Liang, Amir Zadeh and Morency, first circulated in 2022 and published in ACM Computing Surveys in 2024, refined the picture by emphasising how modalities interact: some information is redundant across modalities (the word "dog" and the picture of the dog), some is unique to one (the colour of the dog's collar, which the caption does not mention), and some emerges only from their combination (sarcasm, where the words say one thing and the tone of voice another). A good multimodal model must exploit all three. Redundancy is easy to learn and is what most web-scale training captures. Synergy, the information that exists only in the combination, is the hardest and the most valuable. Early, late and in between The most basic architectural choice in fusion is when to combine the streams. Three broad answers have been tried, and each corresponds to a different bet about where the useful interactions between modalities lie. Early fusion combines modalities at or near the input. In its crudest form, the raw features of each modality are concatenated into one long vector and fed to a single model. The appeal is that the model can learn any interaction between modalities from the start, including low-level ones such as the synchrony between lip movements and speech sounds that the McGurk effect depends on. The difficulty is that raw inputs are so different in structure and scale that a single model struggles to make sense of their concatenation. Early fusion also tends to be computationally expensive, because the model must process every element of every modality together. Late fusion processes each modality with its own complete model and combines only the final outputs, for instance by averaging the predicted probabilities of a video classifier and an audio classifier. Late fusion is simple, modular and robust: if one modality is missing, the other model still works, and each model can be trained separately on whatever data is available for its modality. Its weakness is that it can only capture interactions at the level of conclusions. If the meaning lies in the relationship between a specific word and a specific region of an image, late fusion never sees it, because by the time the streams meet, each has already collapsed into a summary. Intermediate fusion, sometimes called hybrid or deep fusion, lets each modality be processed by its own encoder for some depth, building representations suited to its structure, and then allows the representations to interact through shared layers. This is where almost all modern multimodal architectures sit. The open questions become how deep each encoder should run before interaction begins, what mechanism should govern the interaction, and whether interaction should be one-way (text queries the image) or two-way (each queries the other). There is a fourth pattern that the history of the field made important: no fusion at all, only coordination. Two separate encoders, one for images and one for text, are trained so that their output vectors agree for matching pairs and disagree for mismatched ones. The modalities never interact inside the network; they meet only in the final comparison of two vectors. This sounds like a weak form of late fusion, and in a sense it is. But it has one enormous practical advantage: because the two encoders are independent, every image and every piece of text can be encoded once and stored, and comparing them afterwards is just a dot product. That makes it possible to search a billion images for the one that matches a text query in milliseconds. It also makes training on hundreds of millions of examples tractable. This is the design of CLIP, and Chapter 4 explains why it changed the field. The choice between these patterns is a trade-off between expressiveness and efficiency, and the trade-off recurs throughout this book. Deep interaction between modalities can capture fine-grained relationships but costs computation that grows with the product of the sizes of the inputs, and it requires the model to be run afresh on every combination. Coordinated encoders are fast, scalable and reusable but can only capture what survives compression into a single vector per input. Why the field changed after 2017 For most of the 2010s, multimodal systems were built from parts designed for single modalities. A typical image-captioning model of 2015 used a convolutional network pretrained on ImageNet to produce a grid of image features and a recurrent network to generate the caption word by word, with an attention mechanism letting the generator look at different parts of the image grid as it produced each word. The "Show, Attend and Tell" model of Kelvin Xu and colleagues, published that year, made this pattern famous, and its attention maps, showing the network looking at the frisbee when it produced the word "frisbee", were among the first convincing demonstrations of learned alignment between words and image regions. Visual question answering became a major benchmark task in the same period. The VQA dataset of Stanislaw Antol and colleagues, released in 2015, asked models to answer free-form questions about images. It quickly emerged that models could score well by exploiting statistical regularities in the questions alone: the answer to "What colour is the grass?" is nearly always "green", whatever the image shows. A revised version, VQA v2, released by Yash Goyal and colleagues in 2017, tried to correct this by pairing each question with two similar images that have different answers, forcing models to actually look. The episode was an early warning of a problem that has never gone away: multimodal models are adept at answering from the language prior alone, and benchmarks must work hard to make looking necessary. Three developments between 2017 and 2021 transformed the situation. The first was the Transformer, introduced by Ashish Vaswani and colleagues in 2017 in the paper "Attention Is All You Need". It was designed for machine translation, but its core operation, attention over a set of vectors, makes almost no assumptions about what the vectors represent. A Transformer does not care whether its inputs are words, patches of an image or slices of a spectrogram, provided they arrive as vectors of the right size. Chapter 3 explains attention in detail; for now the point is that the Transformer offered a single architecture that could, in principle, process any modality. The second was the demonstration that this was true in practice for images. The Vision Transformer, or ViT, published by Alexey Dosovitskiy and colleagues at Google and presented at ICLR in 2021 under the title "An Image is Worth 16x16 Words", chopped an image into a grid of square patches, flattened each patch into a vector and fed the sequence to a standard Transformer. With enough training data, it matched or beat the best convolutional networks. Similar moves followed for audio. Suddenly all the major modalities could be represented as sequences of vectors processed by the same machinery. Chapter 2 explains how. The third was scale and self-supervision. Large language models showed that training on vast amounts of unlabelled text, with the simple objective of predicting the next word, produced representations of remarkable generality. The multimodal field asked the obvious question: what is the equivalent for images and text together? The internet contains billions of images with nearby text: alt-text, captions, surrounding paragraphs. That text is noisy and often irrelevant, but it is free and plentiful. If a training objective could be found that extracted signal from it at scale, the bottleneck of hand-labelled data would disappear. Contrastive pretraining was that objective. The shape of a modern multimodal system With these developments in place, a reasonably standard anatomy emerged for systems that reason over images and text, and it is worth describing in outline because the rest of the book fills in each part. First, each modality has an encoder that turns raw input into a sequence of vectors. For images this is usually a Vision Transformer; for audio, a Transformer over a spectrogram or over learned features of the waveform; for text, the embedding layer of a language model. Second, there is an alignment stage, usually a pretraining phase in which the encoders are taught that matching inputs across modalities should have related representations. Contrastive training is the most common method; generative training, in which one modality must be reconstructed or described from another, is the main alternative. Third, there is a connector that brings the aligned representations into a shared processing stream. This may be cross-attention layers, a small Transformer that summarises the visual tokens into a fixed number of query vectors, or simply a learned linear projection that maps visual vectors into the input space of a language model. Fourth, there is a reasoning backbone, nowadays almost always a large language model, which processes the combined sequence and produces output. Fifth, there is an adaptation stage, usually instruction tuning on curated examples of images paired with questions and answers, which teaches the combined system to follow requests and to ground its responses in the input. Each of these stages can be done in different ways, and each choice carries the trade-off this chapter has described: between interaction that is deep but costly and coordination that is shallow but scalable. A good deal of the progress since 2021 has come not from inventing new mechanisms but from finding effective ways to combine a cheap, broadly aligned stage with a smaller, more expensive stage that adds fine-grained grounding. What "understanding" would require It is worth ending the chapter with a concrete test, because the vocabulary of fusion and alignment can make the subject sound more abstract than it is. Suppose a system is shown a photograph of a kitchen and asked, "Is there anything on the stove that could boil over?" To answer well, it needs to recognise the stove, which is a matter of visual representation. It needs to connect the word "stove" in the question to the right region of the image, which is alignment. It needs to recognise a pot, judge that it contains liquid, notice steam or a flame, and combine those observations, which is fusion of evidence within and across modalities. It needs background knowledge about what boiling over means and when it happens, which comes largely from language. And it needs to produce a sentence, which is translation. A system that merely knows the photograph is "a kitchen" will fail. A system that knows there is "a pot" but not where it is, or whether it is on the burner or beside it, will be unreliable. A system that has learned from millions of captions that kitchens often contain pots may assert that there is a pot even when there is not. Each of these failure modes corresponds to a gap in one of the five problems, and each is visible in real systems. The remaining chapters follow the signal through the pipeline: from raw input to tokens, from tokens to aligned representations, from aligned representations to a language model that can reason, and finally to the question of whether that reasoning is grounded in what the model sees. The first step, turning a picture or a sound into something a Transformer can read, is where the story begins. CHAPTER 2 Turning the World into Tokens A Transformer processes a set of vectors. It does not know, and does not need to know, where those vectors came from. This indifference is the source of its power as a multimodal architecture, but it moves the hard work elsewhere. Before any fusion can happen, each modality must be converted into a sequence of vectors of a common size, in a way that preserves the information that matters and discards the redundancy that does not. This conversion is called tokenisation, and the choices made here constrain everything downstream. A useful way to think about tokenisation is as a negotiation between two pressures. On one side is fidelity: the tokens must keep enough detail for the model to answer the questions that will be asked of it. A model asked to read the small print on a medicine bottle needs tokens fine enough to resolve individual letters. On the other side is cost: attention, the core Transformer operation, has a cost that grows with the square of the sequence length, so doubling the number of tokens roughly quadruples the cost of each attention layer. Every tokenisation scheme is a point on the curve between these pressures, and a surprising number of the practical limitations of multimodal models trace back to where that point was chosen. Text: the reference case Text is the modality for which tokenisation is most mature, and it is the reference against which the others are usually designed. Modern language models do not operate on whole words or on individual characters but on subword units. The most common scheme, byte-pair encoding, was adapted for neural machine translation by Rico Sennrich, Barry Haddow and Alexandra Birch in 2016. It starts from individual characters (or bytes) and repeatedly merges the most frequent adjacent pair into a new symbol, building a vocabulary in which common words are single tokens, less common words are split into a few pieces, and rare strings fall back to characters. A typical vocabulary has somewhere between tens of thousands and a few hundred thousand entries. The English word "the" is one token; a long technical term might be four or five. Each token in the vocabulary is assigned a learned vector, its embedding, and the sequence of embeddings is what the Transformer actually processes. Position information is added, either as a learned vector for each position or through schemes that rotate the vectors according to their position, so that the model knows the order of the words. Two properties of text tokens matter for what follows. First, they are discrete: each token is one of a finite set, so a model can be trained to predict the next one with a simple classification objective. Second, they are semantically dense: one token often corresponds to a meaningful unit, so a sentence of twenty tokens can describe a scene that would take hundreds of image tokens to depict. When images and text are placed in the same sequence, as most modern systems do, the imbalance is stark. The visual part of the input usually dominates the token count while carrying, in some sense, less meaning per token. Images: from convolutions to patches For most of the 2010s, images were encoded by convolutional neural networks. A convolutional network slides small learned filters across the image, producing maps of where each local pattern occurs, and stacks many such layers with occasional downsampling so that later layers respond to larger and more abstract patterns. The final layers of a network such as ResNet, introduced by Kaiming He and colleagues in 2015, produce a coarse grid of feature vectors (seven by seven, for a standard 224 by 224 input) in which each vector summarises a region of the image. That grid is already a form of tokenisation. Early image-captioning systems used exactly these forty-nine vectors as the things a text generator could attend to. Later systems, such as the Bottom-Up and Top-Down attention model of Peter Anderson and colleagues in 2018, went a step further and ran an object detector over the image first, producing a feature vector for each detected object region. A typical setting used between ten and one hundred regions per image. For several years, these region features were the standard visual input to vision-language models, and they worked well, because each token corresponded to something nameable: a person, a cup, a dog. The drawback was that the detector had to be trained separately on labelled bounding boxes, was slow to run, and could only produce tokens for the categories it had been trained to detect. Anything outside its vocabulary was invisible. The Vision Transformer swept this away. Its tokenisation is almost insultingly simple. The image is divided into a grid of non-overlapping square patches, typically 16 by 16 or 14 by 14 pixels. Each patch, a small block of colour values, is flattened into a long vector and multiplied by a learned matrix to produce an embedding of the model's working width. A position embedding is added to each so the model knows where in the image the patch came from. Often an extra learned token, conventionally called the class token, is prepended to the sequence; after the Transformer has processed everything, the output at this position serves as a summary of the whole image. For a 224 by 224 image with 16-pixel patches, the grid is 14 by 14, giving 196 patch tokens. With 14-pixel patches, the same image gives 256. At a resolution of 336 by 336 with 14-pixel patches, the grid is 24 by 24 and the token count is 576. This last configuration is worth remembering, because it is the setting of the widely used CLIP ViT-L/14 encoder at 336-pixel resolution, and through its adoption in LLaVA and its successors it became the visual front end of many open vision-language models. Patch tokenisation has three consequences that shape everything built on it. First, it is generic. A patch token does not correspond to an object; it corresponds to a square of pixels that may contain half a dog and half a lawn. Whatever objects and relationships the model comes to represent must be assembled by the Transformer layers from these arbitrary fragments. This makes the encoder more flexible than a detector-based system, since it is not limited to a fixed list of categories, but it means that object-level structure is learned rather than given, and Chapter 5 shows that it is often learned less precisely than one might hope. Second, it is resolution-bound. A 14-pixel patch at 336-pixel resolution means each token covers about 0.17 per cent of the image. Fine detail, such as small text, distant objects or the exact number of items in a crowded shelf, may fall within a single patch or straddle several in ways that make it hard to recover. Raising the resolution helps but increases the token count quadratically: doubling the side length of the image quadruples the number of patches. Much engineering effort since 2023 has gone into ways around this, such as cutting a high-resolution image into tiles, encoding each tile separately alongside a low-resolution view of the whole, and passing all of the resulting tokens to the language model. Some systems instead adapt the encoder to accept images at their native aspect ratio and resolution with a variable number of patches. Either way, the cost of seeing more detail is paid in tokens. Third, it is dense and redundant. Neighbouring patches of sky or wall carry nearly the same information. Many systems therefore compress the patch tokens before handing them to a language model, by pooling neighbouring patches, by merging similar tokens, or by using a small learned module that summarises hundreds of patches into a few dozen query vectors. Chapter 6 returns to these connectors in detail. Audio: spectrograms and learned frames Sound arrives as a waveform, a single sequence of amplitude values sampled at a fixed rate: 16,000 times per second is standard for speech, 44,100 or 48,000 for music. Even ten seconds of speech at 16 kHz is 160,000 numbers, far too many to feed directly to a Transformer as individual tokens. Two broad strategies reduce this to something tractable. The first is to convert the waveform into a spectrogram, a two-dimensional representation of how much energy is present at each frequency over time. The audio is cut into short overlapping windows, typically 25 milliseconds long and spaced 10 milliseconds apart, and a Fourier transform computes the frequency content of each window. The frequencies are then grouped into bands spaced according to the mel scale, which approximates the way human hearing is more sensitive to differences between low frequencies than between high ones, and the energies are put on a logarithmic scale. The result, a log-mel spectrogram, has perhaps 80 or 128 frequency bands and 100 time frames per second. A spectrogram looks like an image, and can be treated as one. The Audio Spectrogram Transformer of Yuan Gong, Yu-An Chung and James Glass, published in 2021, did exactly that, cutting the spectrogram into patches and processing it with a Vision Transformer. OpenAI's Whisper speech recognition model, described by Alec Radford and colleagues in 2022, takes a different route: it computes an 80-channel log-mel spectrogram over 30-second windows, passes it through two convolutional layers that halve the time resolution, and feeds the resulting sequence of 1,500 frames to a Transformer encoder. Each frame then corresponds to 20 milliseconds of audio. The second strategy is to learn the frame representation directly from the waveform. The wav2vec 2.0 model of Alexei Baevski and colleagues, published in 2020, runs a stack of convolutional layers over the raw waveform to produce a feature vector roughly every 20 milliseconds, then processes these with a Transformer trained by a self-supervised objective: parts of the sequence are masked, and the model must identify the correct quantised representation of the masked portion among distractors. Models trained this way on unlabelled audio learn representations that make speech recognition possible with remarkably little transcribed data. Both strategies arrive at roughly the same place: around fifty tokens per second of audio, each a continuous vector. That rate matters. A one-minute voice message becomes about 3,000 tokens, comparable to five or six high-resolution images. An hour-long meeting becomes nearly 200,000. Audio is therefore expensive to place in a language model's context, and systems that handle long recordings usually either transcribe the audio to text first, which compresses it enormously, or compress the audio tokens further. Discrete tokens for continuous signals The tokens described so far for images and audio are continuous vectors. They can be fed into a Transformer, but they cannot easily be predicted the way text tokens are, because there is no finite vocabulary to choose from. This matters for any system that must generate images or sound rather than merely perceive them, and it matters for systems that want to treat all modalities uniformly, with a single next-token prediction objective. The solution is to learn a discrete codebook. The vector-quantised variational autoencoder, or VQ-VAE, introduced by Aäron van den Oord, Oriol Vinyals and Koray Kavukcuoglu in 2017, trains an encoder to map an input to a grid of vectors, then replaces each vector with the nearest entry in a learned codebook of, say, 8,192 vectors. A decoder is trained to reconstruct the original input from the codebook entries alone. Once trained, the encoder turns any image into a grid of integers, each the index of a codebook entry, and the decoder turns any such grid back into an image. An image becomes a "sentence" in a visual vocabulary. For audio, neural codecs such as Google's SoundStream, published by Neil Zeghidour and colleagues in 2021, and Meta's EnCodec, published by Alexandre Défossez and colleagues in 2022, apply the same idea with a refinement called residual vector quantisation: the first codebook captures a coarse approximation, a second codebook quantises the error left by the first, a third the error left by the second, and so on. The result is a stack of discrete token streams, typically several codebooks running at around 50 to 75 frames per second, from which high-quality audio can be reconstructed. Models such as Google's AudioLM, described by Zalán Borsos and colleagues in 2023, generate speech and music by predicting these codec tokens with a language model. Discrete tokenisation makes a striking idea possible: a single Transformer trained to predict the next token in sequences that interleave text tokens, image tokens and audio tokens, all drawn from one combined vocabulary. Meta's Chameleon models, described in 2024, took this "early fusion" approach for images and text, tokenising images into discrete codes and training one model from scratch on mixed sequences. The appeal is uniformity: one model, one objective, and the ability to both read and produce any modality. The cost is that quantisation loses detail, and the image codebooks are trained for reconstruction rather than for understanding, so the tokens may not carry the semantic information that a contrastively trained encoder provides. Whether discrete early fusion or continuous encoders with connectors will dominate is one of the open architectural questions of the field. Video: time as another axis Video adds time to images, and with it a new scale of cost. At even one frame per second and 256 tokens per frame, a five-minute video is 76,800 tokens before its soundtrack is considered. At the 24 or 30 frames per second of ordinary footage, naive tokenisation is hopeless. Several remedies are used, usually in combination. Frames can be sampled sparsely, on the assumption that most of the information in a short clip is captured by a handful of frames; many video question-answering systems use eight or sixteen frames per clip. Patches can be extended in time, so that each token covers a small cube of pixels across two or more consecutive frames; the ViViT model of Anurag Arnab and colleagues, published in 2021, called these tubelets. Tokens can be pooled aggressively within each frame. And attention can be factorised, so that tokens attend first within their frame and then across frames at the same position, rather than to all tokens at once. Each of these remedies throws away information on the assumption that it does not matter. Sparse frame sampling misses brief events. Heavy pooling loses spatial detail. The consequence is that video understanding in current systems is often closer to understanding a slideshow than understanding motion, and questions that depend on precise timing, such as which of two people moved first, remain hard. Comparing the modalities Table 1 gathers the typical tokenisation choices for each modality described in this chapter. The token counts are representative configurations taken from the original publications of the systems named, not universal constants; real systems vary widely. Table 1. Typical tokenisation of each modality. Modality Raw signal Common encoder Token unit Typical count Text Characters or bytes Byte-pair encoding plus embedding table Subword About 1.3 per English word Image Pixel grid Vision Transformer 14 or 16 pixel patch 196 to 576 per image Speech Waveform at 16 kHz Log-mel spectrogram plus Transformer, or wav2vec-style convolutions Frame of about 20 ms About 50 per second General audio (generation) Waveform at 24 kHz or more Neural codec with residual quantisation Stack of codebook indices 50 to 75 frames per second, several codebooks Video Frames over time Per-frame encoder or tubelet patches Patch or tubelet Hundreds per sampled frame Source: Dosovitskiy et al. (2021); Radford et al. (2022); Baevski et al. (2020); Défossez et al. (2022); Arnab et al. (2021). What the comparison shows is that text is by far the cheapest modality per unit of meaning. This is not an accident. Language is a human invention for compressing experience into transmittable symbols, and it has had a very long time to become efficient. Images and sound have no such compression built in. A large part of what a multimodal model must learn is, in effect, how to perform the compression that language performs: to extract from hundreds of patch tokens the handful of facts that matter for the question at hand. What tokenisation decides It is tempting to treat tokenisation as plumbing, a preprocessing step to be settled once and forgotten. It is better understood as the first and least reversible decision about what a model can perceive. If an image encoder works at 224-pixel resolution, no amount of downstream reasoning will let the model read a footnote in a photographed document. If a video is sampled at one frame per second, a model cannot answer questions about anything that happens between frames. If audio is converted to a transcript before the language model sees it, tone of voice, hesitation, laughter and the identity of the speaker are gone. If image tokens are compressed from 576 to 32 before reaching the language model, fine spatial relationships may not survive the compression. Conversely, a tokenisation that preserves everything may be unaffordable. The design of a multimodal system is in large part the design of where information is discarded, and the best systems discard it as late as possible and as selectively as possible, ideally in response to the question being asked. That last idea, that the question should determine which parts of the input are kept, is precisely what attention, and cross-attention in particular, makes possible. It is the subject of the next chapter. CHAPTER 3 Attention as the Bridge Once every modality has been turned into a sequence of vectors, the question becomes how those sequences should interact. The dominant answer for nearly a decade has been attention. Attention is the mechanism that lets a model decide, for each element it is processing, which other elements are relevant and how much to draw on each. When the elements being consulted come from a different modality from the one doing the consulting, the mechanism is called cross-attention, and it is the closest thing the field has to a universal connector between senses. This chapter explains attention concretely, because the intuition behind it is easy to state and easy to misunderstand. It then shows how the same mechanism, arranged in different ways, produces the main families of fusion architecture, and why the arrangement matters as much as the mechanism. Attention, step by step Begin with a sequence of vectors, each representing one token. In a sentence, these might be the embeddings of its words; in an image, the embeddings of its patches. Attention updates each vector by mixing in information from the others, weighted by relevance. The question is how to compute relevance. The Transformer's answer, set out by Vaswani and colleagues in 2017, is to derive three different vectors from each token by multiplying it by three learned matrices. The first is called the query: it represents what this token is looking for. The second is the key: it represents what this token offers to others who are looking. The third is the value: it represents the information this token will actually contribute if it is attended to. The update for any single token proceeds in four steps. First, its query is compared with the key of every token in the sequence, including itself, by taking a dot product. A dot product is large when two vectors point in similar directions, so a token whose key matches the query closely receives a high score. Second, the scores are divided by the square root of the vector dimension. This is a numerical precaution: in high dimensions, dot products of random vectors grow large, and large scores would push the next step into a regime where it produces almost all-or-nothing weights and learns poorly. Third, the scaled scores are passed through a softmax function, which turns them into positive weights that sum to one. A token with a much higher score than the others receives most of the weight; tokens with similar scores share it. Fourth, the token's new representation is the weighted sum of all the value vectors, using those weights. If the query strongly matched one particular key, the output is mostly that token's value; if it matched several, the output is a blend. All tokens are updated this way in parallel, which is what makes Transformers fast on modern hardware. In practice, the process is run several times side by side with different learned matrices, a design called multi-head attention, so that one head can attend to, say, the grammatical subject of a sentence while another attends to the nearest adjective. The outputs of the heads are concatenated and projected back to the working width. Each attention layer is followed by a small feed-forward network applied to each token separately, and residual connections add each layer's output to its input so that information can pass through many layers without degrading. A small example helps. Suppose a sentence reads "the cat sat on the mat because it was warm", and the model is updating the representation of "it". The query for "it" might, after training, encode something like "I am a pronoun seeking a plausible antecedent that can be warm". The keys for "cat" and "mat" both partly match; the key for "sat" does not. If the model has learned that mats are more often described as warm in this construction, the weight on "mat" will be higher, and the updated representation of "it" will carry much of the information in the value vector of "mat". Nothing in the mechanism was told about pronouns or antecedents. The matrices that produce queries, keys and values were learned because doing this kind of matching helped the model make better predictions. Self-attention and cross-attention When queries, keys and values all come from the same sequence, the operation is self-attention. Every token in a sentence attends to every other token in the same sentence; every patch in an image attends to every other patch. When the queries come from one sequence and the keys and values from another, the operation is cross-attention. This is the form that matters most for multimodal fusion. Suppose the queries come from the words of a question about an image, and the keys and values come from the image's patch tokens. Then each word asks, in effect, "which parts of the image are relevant to me?", and its updated representation is a blend of the visual information from those parts. The word "red" in "what is the red object on the left?" can draw information from red patches; the word "left" can draw on patches whose position embeddings place them on the left side. The asymmetry is important. In cross-attention, the querying modality is the one being updated. Text that cross-attends to an image becomes visually informed text; the image is not changed. If the image representations should also be informed by the text, a second cross-attention in the other direction is needed. Different architectures make different choices about which direction to run, and the choice reflects what the system is for. A captioning or question-answering system needs text to be informed by the image, because the text is what it will produce. A system that segments the part of an image referred to by a phrase needs the image to be informed by the text. It is worth being clear about what cross-attention does and does not do. It does not, by itself, align the modalities. The query and key matrices must be learned so that the queries of words and the keys of patches end up in a space where matching ones have large dot products. If the encoders producing the word and patch vectors have never been trained together, the initial attention weights are essentially random, and learning to make them meaningful requires paired data. Cross-attention is the channel along which aligned information can flow; it is the training signal that determines whether anything useful flows along it. This is why most successful systems combine cross-attention with encoders that have already been aligned in some way, typically by contrastive pretraining. The cost of cross-attention is proportional to the product of the two sequence lengths. A question of 20 tokens attending to 576 image patches computes 11,520 query-key scores per head per layer; that is modest. A dialogue of 2,000 tokens attending to the patches of ten high-resolution images, each tiled into several thousand tokens, is a different matter. And because cross-attention depends on both inputs, it must be recomputed for every new pairing: an image cannot be encoded once and reused against many different queries, as it can in a coordinated architecture. These costs are the price of fine-grained interaction. Single-stream and dual-stream designs The first wave of Transformer-based vision-language models, appearing in 2019 and 2020, explored two ways of arranging attention between modalities. Dual-stream models keep separate Transformers for each modality and connect them through cross-attention layers. ViLBERT, introduced by Jiasen Lu, Dhruv Batra, Devi Parikh and Stefan Lee at NeurIPS in 2019, used two parallel streams, one for image regions and one for words, joined by "co-attentional" layers in which each stream's queries attend to the other stream's keys and values. LXMERT, by Hao Tan and Mohit Bansal, published at EMNLP the same year, used a similar three-part design: an object-relationship encoder for the image, a language encoder, and a cross-modality encoder in which the two exchanged information. The rationale for dual streams is that each modality benefits from its own processing for a while, and that the two can have different depths and widths suited to their complexity. Single-stream models concatenate the image tokens and the text tokens into one sequence and process it with a single Transformer using ordinary self-attention. Because every token attends to every other, image tokens attend to words, words attend to image tokens, and both attend within their own modality, all in the same operation. VisualBERT, by Liunian Harold Li and colleagues in 2019, and UNITER, by Yen-Chun Chen and colleagues in 2020, followed this pattern. Its virtue is simplicity: no special cross-attention layers, one set of parameters, full interaction from the first layer. Its cost is that the attention computation grows with the square of the combined length. Both families were pretrained on a few million image-caption pairs from datasets such as Conceptual Captions and COCO, using objectives borrowed from language modelling: predict masked words given the image and the remaining words, predict masked image regions given the words, and decide whether a given caption matches a given image. They then were fine-tuned on downstream tasks such as visual question answering and image retrieval, where they substantially improved on earlier systems. In retrospect, the most notable fact about these models is that the difference between single-stream and dual-stream designs turned out to matter less than the quantity and kind of pretraining data. Comparative studies found that, with comparable data and training, the two families performed similarly. The lesson, which the field absorbed over the following two years, was that the mechanism of interaction was less of a bottleneck than the signal teaching the modalities to agree. Contrastive pretraining at web scale, discussed in the next chapter, would make that point emphatically. Bottlenecks: attending with fewer queries One problem with full attention between modalities is that the visual sequence is often far longer than it needs to be. Hundreds of patch tokens carry heavily redundant information, and allowing every text token to attend to all of them, or concatenating all of them into a language model's input, is expensive. A family of designs addresses this by introducing a small set of learned query vectors that attend to the long visual sequence and summarise it. The Perceiver, introduced by Andrew Jaegle and colleagues at DeepMind at ICML in 2021, made this idea general. Instead of running self-attention over a long input, it maintains a small array of latent vectors, perhaps a few hundred, which repeatedly cross-attend to the input and then self-attend among themselves. The cost of each cross-attention is proportional to the number of latents times the input length, rather than the input length squared. Because the latents are independent of the input's structure, the same architecture could process images, audio, point clouds and video. The Perceiver was explicitly modality-agnostic, and it showed that a fixed-size bottleneck could compress very different inputs into a common working representation. The same pattern reappears in later vision-language systems in more specialised forms. DeepMind's Flamingo used a "Perceiver Resampler" that took the variable number of visual features from an image or video and produced a fixed 64 visual tokens. Salesforce's BLIP-2 used a "Querying Transformer", or Q-Former, with 32 learned queries that extract from a frozen image encoder the features most relevant to language. Chapter 6 examines both. The essential idea is the same in each case: a small number of learned queries cross-attend to the visual input and become its compressed representation. The queries are trained so that what they extract is whatever the downstream task needs. A bottleneck of this kind is a deliberate act of forgetting. Thirty-two vectors cannot carry everything in 576 patches. Whether the right things survive depends on how the queries were trained, and if the training data rarely asked about small text or precise counts, the queries will learn not to keep that information. Several later systems abandoned learned-query bottlenecks for this reason in favour of passing all patch tokens to the language model, accepting the cost in exchange for not prejudging which details matter. Gated cross-attention: adding vision without breaking language A distinctive use of cross-attention appeared with Flamingo, described by Jean-Baptiste Alayrac and colleagues at NeurIPS in 2022. Flamingo's designers wanted to start from a large, already-trained language model, which had learned a great deal about the world from text, and give it the ability to see without damaging what it already knew. Their solution was to freeze the language model's weights and insert new cross-attention layers between its existing layers. In each inserted layer, the text tokens act as queries and the visual tokens produced by the resampler act as keys and values. The output of the cross-attention is then multiplied by a learned scalar passed through a tanh function, and that scalar is initialised to zero. At the start of training, therefore, every new layer contributes exactly nothing, and the model behaves exactly as the original language model did. As training proceeds, the gates open gradually and visual information begins to flow in. This is a small detail with a large rationale. Inserting randomly initialised layers into a trained network typically disrupts it badly; the network's carefully tuned internal representations are suddenly perturbed by noise, and it can take a long time to recover, during which some of what it knew may be lost. A zero-initialised gate guarantees a smooth start. The same principle, of adding new capacity so that it begins as an identity operation, recurs throughout modern work on extending pretrained models, including techniques for fine-tuning image generators and language models. Flamingo also had to handle sequences that interleave several images with text, as on a web page. Its cross-attention was masked so that each text token attended only to the visual tokens of the image that most recently preceded it, while self-attention in the language model carried information across images. This let it be prompted with a few example image-text pairs followed by a new image, and produce an answer in the style of the examples, a capability called few-shot in-context learning that had previously been demonstrated only for text. What attention maps do and do not show Because attention weights are explicit numbers, it is tempting to read them as explanations. If a model's cross-attention from the word "frisbee" peaks over the frisbee in the image, it seems natural to say the model is "looking at" the frisbee when it processes that word. Visualisations of this kind were influential in the early days of image captioning and remain common. They should be read with caution. In a deep Transformer, information is mixed at every layer, so by the middle layers a patch token's vector may carry information about many other patches. Attention to a particular token is therefore not attention to the corresponding pixels alone. Multiple heads attend to different things, and averaging them can produce a map that reflects none of them. And research on language models, notably the 2019 paper "Attention is not Explanation" by Sarthak Jain and Byron Wallace, showed that attention weights can often be substantially altered without changing a model's output. Attention maps are a useful diagnostic for spotting gross failures, such as a model that never attends to the image at all, but they are not a faithful account of why a model produced a particular answer. The mechanism and the signal The story of attention in multimodal learning is, in the end, a story about the limits of mechanism. Cross-attention provides a flexible, differentiable way for any token in one modality to retrieve information from any token in another. It is capable of expressing extremely fine-grained alignment: this word with that patch, this syllable with that frame of lip movement. But it does not supply that alignment. The alignment must be learned from data, and the quality and granularity of the alignment a model achieves depends on what its training data and objective demanded. The early vision-language Transformers had expressive attention and modest data. They learned useful alignments but were limited by the few million carefully captioned images available. The next step in the field's development did something counterintuitive. It discarded cross-attention almost entirely, kept two independent encoders that never interact, and trained them on hundreds of millions of noisy pairs with an objective so simple it can be described in a sentence. The result was a model whose representations became the foundation for most of what followed. CHAPTER 4 Learning from Pairs: Contrastive Pretraining and CLIP In January 2021 OpenAI released a model called CLIP, short for Contrastive Language-Image Pre-training, described in a paper by Alec Radford, Jong Wook Kim and colleagues titled "Learning Transferable Visual Models From Natural Language Supervision" and presented at ICML later that year. Its headline result was startling to anyone who had worked in computer vision. Without being trained on a single one of the 1.28 million labelled images in the ImageNet training set, CLIP's best model classified the ImageNet validation images with 76.2 per cent accuracy, roughly matching a standard ResNet-50 that had been trained directly on that labelled data. It did this by being told, in plain English, the names of the thousand categories. CLIP did not use cross-attention between images and text. Its image and text encoders never exchanged information inside the network at all. It was trained with an objective that can be explained to a non-specialist in a minute. And yet its representations turned out to be so broadly useful that, five years later, CLIP-style encoders sit inside a large share of the world's vision-language systems, image generators and image search engines. This chapter explains how it works and why it worked so well. The objective in one paragraph Take a batch of image-text pairs, say 32,768 of them, scraped from the web, each image with the text that appeared alongside it. Pass every image through an image encoder to get one vector per image. Pass every text through a text encoder to get one vector per text. Normalise all the vectors to unit length, so that their dot product measures the cosine of the angle between them. Now compute the similarity of every image with every text in the batch, producing a 32,768 by 32,768 grid of scores. The diagonal of that grid holds the scores of the true pairs; everything off the diagonal is a mismatched pair. Train both encoders so that, in each row, the diagonal entry is the highest, and likewise in each column. That is, for every image, its own caption should score higher than all the other captions in the batch; for every caption, its own image should score higher than all the other images. The technical name for this is a contrastive objective, and the specific loss is a symmetric version of what Aäron van den Oord and colleagues called InfoNCE in their 2018 work on contrastive predictive coding. In each row, the scores are treated as the inputs to a softmax over all the texts, and the loss is the negative log probability assigned to the correct one, exactly as if the model were doing a 32,768-way classification in which the "classes" are the captions in the current batch. The same is done for each column. The two losses are averaged. One further detail matters. Before the softmax, the similarity scores are divided by a temperature parameter, which CLIP learned during training rather than fixing in advance. A low temperature sharpens the softmax, so that small differences in similarity produce large differences in probability; a high temperature smooths it. The temperature controls how hard the model pushes apart pairs that are already fairly well separated, and getting it right turns out to matter considerably for training stability and final quality. Why such a simple objective is so powerful The contrastive objective has a property that is easy to overlook: it asks the model to distinguish each correct pairing from tens of thousands of alternatives simultaneously. To pick out the right caption for a photograph of a golden retriever on a beach, the image vector must be closer to "my dog Max loving the sand at Brighton" than to every other caption in the batch, which might include captions for other dogs, other beaches, other golden objects, other photographs taken in Brighton. The only way to succeed consistently is to encode in the image vector the features that distinguish this image from all the others, and to encode in the text vector the features that distinguish this caption. Over hundreds of millions of examples, those distinguishing features come to include objects, scenes, styles, actions, text appearing in the image, famous people and places, artistic media and a great deal more. Crucially, the supervisory signal is natural language rather than a fixed list of categories. A conventional ImageNet classifier learns exactly a thousand concepts, because those are its labels; it has no way to represent "a dog wearing sunglasses" except as "dog". CLIP's text encoder can represent any phrase, so the space of concepts the image encoder must learn to distinguish is, in principle, as open as language itself. Natural language supervision, as the paper's title says, gives the model a vocabulary of concepts as large as the vocabulary of the web. The second source of power is scale. CLIP was trained on a dataset OpenAI assembled for the purpose, called WIT for WebImageText, of 400 million image-text pairs collected from the internet. Earlier vision-language pretraining had used curated datasets of a few million pairs. Because the contrastive objective needs only the co-occurrence of an image and some text, without any human labelling, it can consume data at this scale with little preparation. And because the two encoders are independent, training is efficient: each image and each text is encoded once per step, and the grid of similarities is a single matrix multiplication. The paper reported that it had first tried a more obvious objective: training the model to generate the exact caption for each image, word by word. That approach learned much more slowly. Predicting the exact words of a web caption is extremely hard, because the same image could have been captioned in countless ways; the model wastes capacity trying to guess phrasing it could never know. The contrastive objective asks only that the right caption be closer than the wrong ones, which is much easier and, it turned out, much more efficient at producing useful representations. In the paper's comparisons, the contrastive approach reached a given level of zero-shot ImageNet accuracy with several times less compute than the generative one. Zero-shot classification CLIP's most famous capability follows directly from its training. To classify an image into one of a set of categories, write a sentence for each category, such as "a photo of a dog", "a photo of a cat", "a photo of an aeroplane". Encode each sentence with the text encoder. Encode the image with the image encoder. Pick the category whose sentence vector is most similar to the image vector. No training on the target categories is required; the class names are the only supervision. This was called zero-shot transfer, and it changed how people thought about image classification. Instead of collecting thousands of labelled examples for every new category, a practitioner could simply describe the categories. The quality depended on the description, and the CLIP authors found that how the prompt was phrased made a measurable difference. Using the template "A photo of a {label}." rather than the bare label improved ImageNet accuracy by about 1.3 percentage points, partly because web captions rarely consist of a single word and partly because it disambiguated words with several meanings: "crane" alone could be a bird or a machine. Averaging the text vectors from 80 different templates, such as "a bad photo of a {label}", "a sculpture of a {label}" and "a close-up photo of a {label}", improved accuracy by a further few points. This practice, known as prompt ensembling, was an early instance of what would later be called prompt engineering for multimodal models. Zero-shot classification is not the only way to use the encoders. The CLIP paper also evaluated the more conventional "linear probe", in which the image encoder is frozen and a simple linear classifier is trained on its output vectors using labelled examples from the target task. Across a suite of 27 datasets, linear probes on CLIP's features compared favourably with the strongest image models trained on ImageNet, which indicated that the representations were not merely good for matching text but were rich visual features in their own right. The paper also reported a telling comparison between the two modes of use: on average across its evaluation datasets, zero-shot CLIP performed about as well as a linear classifier trained on its own features with four labelled examples per class. In other words, a well-chosen sentence describing a category was worth roughly four example images of it. That figure gives a concrete sense of how much information natural language can carry about a visual concept, and also of its limits: with more labelled examples, the trained classifier pulled ahead, because examples can specify fine distinctions that a short phrase cannot. Zero-shot performance varied greatly across tasks. CLIP did well on recognising common objects, scenes, actions and well-known landmarks, and even on reading text in images to a limited degree. It did poorly on specialised tasks far from the web's typical content: counting objects, estimating distances, classifying satellite imagery, distinguishing car models or aircraft variants at fine grain, and recognising handwritten digits as well as a simple classifier trained on them. These weaknesses turn out to be informative, and Chapter 5 examines them. Robustness to distribution shift A less celebrated but arguably more important finding in the CLIP paper concerned robustness. Classifiers trained on ImageNet are known to lose much of their accuracy when tested on images that depict the same categories differently: sketches, cartoons, sculptures, unusual viewpoints or photographs taken in different conditions. Several benchmarks, such as ImageNet-Sketch, ImageNet-R (renditions) and ObjectNet, were designed to measure this gap. Zero-shot CLIP closed much of it. Its accuracy on these shifted versions dropped far less than that of ImageNet-trained models with the same accuracy on the original test set. The interpretation offered by the authors was that a model trained only on ImageNet learns, in part, the specific statistical quirks of ImageNet photographs, while a model trained on the full diversity of web images, described in natural language, learns concepts that are more independent of any one dataset's style. A dog is a dog whether photographed, drawn or carved, and captions on the web say so. The same paper also noted that fine-tuning CLIP on ImageNet improved its ImageNet accuracy but reduced its robustness to these shifts, a trade-off later studied more carefully by others. The general lesson has held: broad pretraining with language supervision tends to produce representations that transfer well, and narrow fine-tuning can erode that breadth. ALIGN, SigLIP and the open reproductions CLIP prompted rapid follow-up. Within months Google published ALIGN, described by Chao Jia and colleagues at ICML 2021, which applied essentially the same dual-encoder contrastive recipe to a much larger and noisier dataset of about 1.8 billion image and alt-text pairs, using only light filtering. ALIGN's message was that scale could compensate for noise: rather than carefully curating data, collect a great deal more of it. In 2023 Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov and Lucas Beyer at Google introduced SigLIP, presented at ICCV that year, which replaced CLIP's softmax with a sigmoid. Instead of asking, for each image, "which of these captions is the right one?", the sigmoid loss treats every image-text pair in the batch as an independent yes-or-no question: is this a matching pair? The difference sounds technical but has practical consequences. The softmax requires normalising over the entire batch, which, when training is spread across many processors, requires gathering all the similarity scores in one place. The sigmoid loss does not, which makes very large batches cheaper and allows good results with smaller ones. SigLIP encoders became popular as the visual front end for later open vision-language models. Because OpenAI released CLIP's model weights but not its training data, independent groups set out to reproduce it in the open. The LAION project assembled LAION-400M and then LAION-5B, the latter described by Christoph Schuhmann and colleagues at NeurIPS 2022 as a dataset of about 5.85 billion image-text pairs filtered from the Common Crawl web archive. A project called OpenCLIP trained CLIP-style models on these datasets, and a 2023 study by Mehdi Cherti and colleagues at CVPR used them to measure how performance scaled with model size, data and compute. These open models made contrastive encoders available to anyone and underpinned a large part of the open-source multimodal ecosystem, including the Stable Diffusion image generators. Table 2 sets out how these models differ on data and objective. Table 2. Major dual-encoder contrastive models. Model Year Training data Loss Main contribution CLIP (OpenAI) 2021 WIT, 400 million pairs, not released Softmax contrastive Showed zero-shot transfer from natural-language supervision ALIGN (Google) 2021 About 1.8 billion noisy alt-text pairs Softmax contrastive Showed scale can offset noise with minimal filtering OpenCLIP (LAION and collaborators) 2022 onward LAION-400M and LAION-2B, public Softmax contrastive Open reproduction enabling scaling studies SigLIP (Google) 2023 WebLI Pairwise sigmoid Removed batch-wide normalisation, easing large-scale training Source: Radford et al. (2021); Jia et al. (2021); Cherti et al. (2023); Zhai et al. (2023). Data is the model If the CLIP family taught one lesson above all, it is that the training data determines most of what a contrastive model learns. The architecture is simple and the objective is fixed; what varies is the distribution of images and the text that accompanies them. This was made explicit by DataComp, a benchmark introduced by Samir Yitzhak Gadre and colleagues at NeurIPS in 2023. DataComp held the model architecture, training code and compute fixed and asked participants to compete solely on how they selected training data from a common pool of 12.8 billion web image-text pairs. Better filtering, rather than more data, produced substantial gains: a well-chosen subset outperformed a larger, unfiltered one. Among the most effective filtering methods was, in a pleasing recursion, using an existing CLIP model to score how well each image matched its text and keeping only the high-scoring pairs. Data also determines a contrastive model's social behaviour. Web captions encode the associations, stereotypes and omissions of the people who wrote them, and a model trained to match images with captions inherits them. The CLIP paper itself reported tests in which the model associated images of people with derogatory or crime-related categories at different rates for different demographic groups, and later audits have documented similar patterns in open models. Web-scale datasets also contain material that should never have been collected: in December 2023 LAION withdrew LAION-5B after a report from the Stanford Internet Observatory identified links to child sexual abuse material in it, releasing a cleaned version the following year. These are not side issues. For a model whose knowledge is entirely a function of its data, the composition of that data is the most consequential design decision. What CLIP gave the field CLIP's practical legacy is its encoders. A CLIP image encoder produces representations that are already aligned with language: an image vector sits near the vectors of texts that describe it. This makes such encoders an extraordinarily convenient starting point for other systems. Text-to-image generators such as the early versions of Stable Diffusion used CLIP's text encoder to condition image generation on prompts. Image search systems use CLIP-style embeddings to retrieve images by description. And, as Chapter 6 describes, most open vision-language assistants built since 2023 begin with a CLIP or SigLIP image encoder, take its patch-level outputs and feed them into a large language model. Its conceptual legacy is the demonstration that coarse alignment at very large scale produces broadly useful representations. By asking only that whole images and whole captions be matched, and by doing so across hundreds of millions of examples, CLIP learned a visual vocabulary of extraordinary breadth. But "whole images and whole captions" is doing a lot of work in that sentence. The contrastive objective never asks the model to know which words in a caption refer to which parts of an image, or how the objects in the image relate to each other. It is reasonable to wonder what a model trained this way does not learn. The answer is quite a lot, and the gaps are not random. They are the subject of the next chapter. CHAPTER 5 What Contrastive Models Cannot See There is a simple test that exposes the central weakness of contrastively trained vision-language models. Show a model a photograph of a mug sitting on top of a laptop. Give it two captions: "a mug on a laptop" and "a laptop on a mug". Ask which caption matches better. A person finds this trivial. A CLIP-style model, for much of the field's history, did not reliably get it right, and on carefully constructed versions of the test its accuracy hovered around the level expected from guessing. This is not a quirk. It follows directly from what the training objective asks for, and understanding why is the key to understanding both the limitations of CLIP-style models and the limitations of the larger systems that use them as eyes. This chapter examines what contrastive pretraining fails to learn, why, and what has been done about it. The bag-of-words problem In a paper presented at ICLR in 2023 under the title "When and Why Vision-Language Models Behave like Bags-of-Words, and What to Do About It", Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky and James Zou introduced a benchmark called ARO, for Attribution, Relation and Order. It tested whether models could tell the difference between captions that use the same words in different arrangements. Could a model distinguish "the paved road and the white house" from "the white road and the paved house"? "The horse is eating the grass" from "the grass is eating the horse"? A correctly ordered caption from one whose words had been shuffled? Many widely used vision-language models, including CLIP, often could not. On some of the tests their performance was close to chance; on the word-order tests they frequently preferred a scrambled caption no more often than the correct one would be expected to be preferred at random. In effect, they treated a caption as an unordered collection of words, a "bag of words" in the old terminology of text retrieval, and matched images on the basis of which words were present rather than how they related. The authors offered a persuasive explanation. In contrastive training, a model needs to distinguish each image's caption from the other captions in the batch. Those other captions are, overwhelmingly, about different things. If the batch contains a photograph of a horse in a field, the competing captions describe kitchens, cityscapes, birthday parties and product shots. To pick the right caption, the model only has to notice that the right one mentions a horse and grass. It almost never encounters, within a batch, a competing caption that contains the same words in a different order, so it has no reason to learn that order matters. The objective rewards recognising content and gives essentially no reward for recognising structure. The same paper showed that the problem could be substantially reduced by adding "hard negatives" during training: for each caption, also generate a version with words swapped or shuffled, and include it in the batch as a mismatch. A model forced to distinguish "the horse is eating the grass" from "the grass is eating the horse" learns to attend to order. Their fine-tuned variant, which they called NegCLIP, performed much better on the compositional tests while keeping most of its performance on standard tasks. The general principle is important: a contrastive model learns exactly the distinctions its negatives require, and no others. Spatial language is a particularly clear case. In a study presented at EMNLP 2023 under the title "What's 'up' with vision-language models?", Amita Kamath, Jack Hessel and Kai-Wei Chang photographed everyday objects in controlled arrangements, a mug on a table, then under it, then to its left and right, and asked models to choose among captions differing only in the preposition. Many popular models performed close to chance. The reason is the same: on the web, captions rarely state where one object sits relative to another, and when they do, the competing captions in a training batch almost never differ only in that respect. A word such as "left" or "under" is, for a contrastive model, almost never the deciding factor in a match, so it never learns to tie the word to the geometry of the scene. Winoground A year earlier, Tristan Thrush and colleagues had presented Winoground at CVPR 2022, a smaller but harder benchmark built by hand. Each of its 400 examples consists of two images and two captions, where the captions contain exactly the same words in a different order and each caption matches exactly one image. One example pairs "some plants surrounding a lightbulb" with "a lightbulb surrounding some plants". Another pairs "there is a mug in some grass" with "there is some grass in a mug". To score on an example, a model must match both captions to their correct images and both images to their correct captions. The results were sobering. Every vision-language model the authors tested, including CLIP and several cross-attention models of the ViLBERT and UNITER families, performed at or below chance on the strictest of the benchmark's three scores. Human annotators found the task easy. Even models that used cross-attention between words and image regions, and so had the architectural capacity to bind words to regions, had not learned to use it for this purpose, because their pretraining data rarely demanded it. Winoground has been widely used since, and scores have risen, particularly for large vision-language assistants that can reason about a caption in words before answering. But the benchmark made a point that has stayed relevant: the ability to recognise what is in an image is not the ability to understand how its contents are arranged, and the training regimes that produce the first do not automatically produce the second. One vector per image The bag-of-words behaviour is one symptom of a deeper constraint. A CLIP-style model must compress an entire image into a single vector, typically of 512 to 1,024 numbers, and an entire caption into another. Everything the model knows about the image must survive that compression, and the training objective decides what survives. What survives is whatever helps distinguish one image from the others in a typical batch. The overall subject, the setting, the style and the most prominent objects are highly distinctive and are preserved. Details that rarely distinguish one web image from another are not: the exact number of objects, the left-right arrangement of things, the colour of an item that is not the main subject, the relative sizes of objects, whether a door is open or closed. The CLIP paper itself reported that zero-shot CLIP was poor at counting objects in synthetic scenes and at estimating the distance to the nearest car in driving images, tasks that depend on precisely this kind of detail. There is a further reason these details are under-represented, and it lies in the captions rather than the objective. People writing captions do not describe what is obvious or expected. A photograph of a banana is rarely captioned "a yellow banana", because bananas are usually yellow; a photograph of a green banana might well be captioned "green banana". Linguists call this reporting bias: the frequency with which something is mentioned does not match the frequency with which it occurs. A model trained on captions learns the statistics of what people say about images, not the statistics of what images contain. It may come to associate "yellow" with bananas weakly, because the association is rarely stated, and "green" with bananas more strongly than reality would justify. The modality gap A geometrical oddity of contrastive models, documented by Weixin Liang and colleagues at NeurIPS 2022 in a paper titled "Mind the Gap", helps explain why their joint space is less unified than the training objective might suggest. One might expect that after training, image vectors and text vectors would be intermingled in the shared space, with each image near its caption. In fact, the image vectors and the text vectors occupy two separate regions, roughly two narrow cones pointing in different directions, with a consistent gap between them. Matching pairs are closer than mismatched ones, as training requires, but every image is still closer to other images than to any text. The authors traced this to a combination of how neural networks initialise, which tends to place all outputs of a randomly initialised encoder in a narrow cone, and the contrastive loss with a low temperature, which preserves the gap rather than closing it. They showed that the size of the gap affected performance on some downstream tasks and on measures of fairness. The practical significance is that the "shared space" of a contrastive model is not quite a space in which images and words are interchangeable. It is a space in which the relative similarities are meaningful, which is what retrieval and classification need, but in which an image vector and a text vector are still recognisably different kinds of thing. Systems that try to substitute one for the other, for example by training on text vectors and testing on image vectors, must account for the gap. Reading instead of seeing A different kind of failure was documented by Gabriel Goh and colleagues at OpenAI in a 2021 article in the online journal Distill titled "Multimodal Neurons in Artificial Neural Networks". Studying CLIP's internal units, they found individual neurons that responded to a concept regardless of how it was presented: a single unit that fired for photographs of Spider-Man, drawings of Spider-Man and the written words "Spider-Man". This was striking evidence that CLIP had learned abstract concepts that spanned modalities, in a way reminiscent of the so-called concept cells reported in the human brain. The same property created a vulnerability. Because CLIP had learned to read text in images, and because text in an image is often a very strong signal of what the image shows, the model could be fooled by labels. The authors' best-known example was an apple with a handwritten note reading "iPod" stuck to it, which CLIP then classified as an iPod with high confidence. They called these typographic attacks. The model had learned, correctly for most web images, that written words in a picture are reliable evidence about its content, and it over-weighted that evidence when it conflicted with the visual appearance. Typographic attacks are a vivid illustration of a general point. A model trained to match images with text will exploit any regularity that helps it do so, and some of those regularities are shortcuts that fail in situations a human would find obvious. Larger vision-language assistants built on such encoders inherit some of this behaviour: they can be misled by text in an image, and they can be steered by instructions written into an image, a weakness that has security implications when such systems read web pages or documents on a user's behalf. When the eyes limit the mind The most consequential finding about contrastive encoders concerns what happens when they are used as the visual front end of larger systems. In a paper presented at CVPR 2024 titled "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs", Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun and Saining Xie looked for what they called "CLIP-blind pairs": pairs of images that look clearly different to a person but receive nearly identical CLIP embeddings. They found such pairs by comparing CLIP with a different visual encoder, DINOv2, trained by self-supervision on images alone without any text. Where DINOv2 judged two images dissimilar but CLIP judged them nearly identical, CLIP was evidently discarding some visual difference. Inspecting these pairs revealed systematic categories of blindness: the orientation and direction of objects, whether a particular feature was present or absent, the state of something (open or closed, on or off), counts of objects, viewpoint and spatial arrangement. The authors then built a benchmark, MMVP, from questions about such pairs, and tested leading multimodal assistants, including commercial ones, on it. Many performed poorly, some below chance, on questions such as which direction a dog was facing or whether a butterfly's wings were visible. The conclusion was direct. When a large language model receives its visual input from a CLIP-style encoder, it cannot recover information that the encoder has thrown away, however capable the language model is. The reasoning system is bounded by its eyes. Adding features from a self-supervised vision encoder alongside the CLIP features improved performance on these questions, which suggested that the problem lay with the contrastive encoder's training rather than with vision encoders in general. Remedies Each failure described in this chapter has prompted remedies, and they share a common theme: change what the model is asked to distinguish. Harder negatives. As NegCLIP showed, generating captions that differ from the correct one only in structure or detail forces a contrastive model to encode that structure or detail. The approach is cheap and effective for the specific distinctions targeted, though it cannot anticipate every distinction a downstream task might need. Generative objectives alongside contrastive ones. If the model must generate the caption, word by word, it cannot treat the caption as a bag of words, because word order affects every prediction. CoCa, short for Contrastive Captioners, described by Jiahui Yu and colleagues at Google in 2022, combined a contrastive loss on pooled image and text vectors with a captioning loss in which a text decoder cross-attended to the image's patch tokens. The combination produced encoders that were strong both at retrieval and at tasks needing finer detail. Several later encoder families adopted similar mixed objectives. Better captions. If web captions under-describe images, write better ones. Several groups have used an existing captioning model to generate long, detailed descriptions of training images, mentioning colours, counts, positions and background objects, and trained on those instead of or alongside the original alt-text. OpenAI's DALL-E 3 report, by James Betker and colleagues in 2023, described this recaptioning approach for training an image generator and showed it improved how closely generated images followed detailed prompts. The same logic applies to training encoders: the model can only learn distinctions that its text makes. Region-level alignment. Rather than matching whole images to whole captions, match regions to phrases. GLIP, introduced by Liunian Harold Li and colleagues in 2022, unified object detection and phrase grounding, training a model to associate each noun phrase in a caption with a box in the image. Region-level training directly targets the binding of words to locations that whole-image contrastive learning neglects, though it requires either box annotations or a way to generate them. Combining encoders. As the "Eyes Wide Shut" authors showed, features from self-supervised vision models such as DINOv2, published by Maxime Oquab and colleagues at Meta in 2023, carry fine-grained visual information that contrastive features lack. Some vision-language systems now use several encoders together, concatenating or interleaving their outputs, to get the semantic breadth of language-aligned features and the visual precision of self-supervised ones. The lesson of the gaps The weaknesses of contrastive models are sometimes presented as embarrassing failures of a hyped technology. That reading misses what they teach. A contrastive model is an extremely efficient learner of exactly what its objective requires: whatever distinguishes a matching image-text pair from the other pairs it is compared against. When the comparisons are between unrelated web images and captions, it learns the gist and ignores the detail, because the detail never helps. It is not that the model cannot represent spatial relations or counts; it is that nothing in its training ever made them worth representing. This has a practical corollary for anyone building on such models. The visual front end of a multimodal system determines, more than any other component, what the system can see. A large language model attached to a CLIP-style encoder will be fluent, knowledgeable and articulate about images, and its fluency will make it easy to forget that it is reasoning from a summary that may lack exactly the detail the question needs. The next chapter describes how such systems are assembled, and the chapter after it how they are extended to sound, but the boundaries described here stay with them throughout. CHAPTER 6 Connecting Eyes to a Language Model By 2022 the field had two powerful but separate kinds of model. Contrastive encoders such as CLIP could look at an image and produce representations aligned with language, but they could not say anything; they could only score how well a given text matched. Large language models could write, reason, explain and converse, drawing on knowledge absorbed from enormous quantities of text, but they were blind. The obvious move was to connect them: give the language model the encoder's output as a kind of perception, and let it talk about what it saw. The obvious move turned out to have several non-obvious answers, and the differences among them illuminate the central trade-off of this book. How much of the visual information should reach the language model? Through what kind of channel? Which weights should be trained and which left alone? And what data teaches the combined system to use its new sense? This chapter follows the main answers in the order they appeared and then describes the training recipe that made them useful. The prefix idea The simplest conceivable connection treats visual information as extra words. A language model processes a sequence of token embeddings; if an image could be turned into a few vectors of the same size and placed at the start of the sequence, the language model would process them along with the text that follows, and its attention layers could draw on them. This idea was demonstrated in 2021 by Maria Tsimpoukelli and colleagues at DeepMind in a NeurIPS paper titled "Multimodal Few-Shot Learning with Frozen Language Models". They kept a pretrained language model entirely frozen and trained only a vision encoder whose output was a short sequence of vectors in the language model's embedding space. Trained on captioned images to make the language model produce the right caption, the system, which they called Frozen, could then answer simple questions about images and even pick up new associations from a few examples given in the prompt. Its performance was modest, but it established a pattern that would become dominant: the language model does not need to be redesigned to accept images; it only needs inputs in a form it can already read. Flamingo: cross-attention into a frozen giant Flamingo, introduced the following year by Jean-Baptiste Alayrac and colleagues at DeepMind, took a more elaborate route and set a new standard. Chapter 3 described its two key mechanisms; here is how they fit together. Flamingo began with two frozen pretrained components: a vision encoder trained contrastively on image-text pairs, and the 70-billion-parameter Chinchilla language model. Between them it added two kinds of new, trainable component. The Perceiver Resampler took the variable-sized grid of visual features from each image, or from several frames of a video, and compressed it into a fixed 64 visual tokens using learned queries. Gated cross-attention layers, inserted between the frozen layers of the language model, let the text tokens attend to those visual tokens, with gates initialised to zero so that training began from the unchanged language model. The full model had about 80 billion parameters, most of them frozen. What made Flamingo distinctive was its training data. Alongside image-text pairs and video-text pairs, it was trained on a dataset of some 43 million web pages, with the images and text kept in their original interleaved order. Learning from interleaved documents taught the model to handle sequences containing several images and passages of text in arbitrary arrangement. This in turn made it possible to prompt Flamingo with a few examples, say three images each followed by a question and answer, then a new image and a question, and have it answer in the same manner. Across 16 benchmarks, the paper reported that Flamingo with a handful of examples was competitive with specialised models that had been fine-tuned on far more task data, and with 32 examples it exceeded the fine-tuned state of the art on six of them. Flamingo's design embodied a particular view: keep the language model's knowledge intact, add vision through a separate channel, and control the influence of that channel carefully. Its drawback was weight. Cross-attention layers at many depths of an enormous model are expensive to train, and the architecture is relatively complex to reproduce. Open reproductions, such as OpenFlamingo and Hugging Face's IDEFICS, followed, but the field's centre of gravity soon shifted to simpler designs. BLIP-2: a learned translator In early 2023, Junnan Li, Dongxu Li, Silvio Savarese and Steven Hoi at Salesforce published BLIP-2, presented at ICML that year, which kept both the image encoder and the language model frozen and trained only a lightweight module between them. That module, the Querying Transformer or Q-Former, used 32 learned query vectors that cross-attended to the frozen image encoder's patch features and produced 32 output vectors, which were then linearly projected into the language model's input space and placed before the text, as in the prefix approach. The Q-Former was trained in two stages. In the first, without any language model, it learned to extract language-relevant visual information using three objectives together: a contrastive objective matching its outputs to text, a binary objective deciding whether an image and text matched, and a generative objective producing the text from the image. In the second stage, its outputs were connected to a frozen language model and trained so that the language model could produce correct text conditioned on them. The first stage did the heavy lifting of alignment; the second taught the language model to read the result. BLIP-2's appeal was efficiency. Its trainable module had a little under 200 million parameters, a tiny fraction of the full system, yet it outperformed Flamingo on zero-shot visual question answering benchmarks with many times fewer trainable parameters. It showed that a carefully trained bottleneck could translate between a frozen visual representation and a frozen language model, and it popularised the learned-query approach for a time. The cost, as Chapter 3 noted, is that 32 vectors are a narrow channel. Whatever the Q-Former did not learn to extract during its training is unavailable to the language model, which can neither look again nor ask for more detail. For tasks that depend on fine spatial information or text in images, this compression proved limiting. LLaVA: the surprising power of a straight line A few months later came a design so simple that its success surprised many in the field. LLaVA, for Large Language and Vision Assistant, was introduced by Haotian Liu, Chunyuan Li, Qingyang Wu and Yong Jae Lee in a paper titled "Visual Instruction Tuning", presented at NeurIPS 2023. LLaVA took the patch-level outputs of a CLIP ViT-L/14 image encoder, passed each patch vector through a single learned linear projection into the word embedding space of an open language model, Vicuna, and placed all of the projected patch tokens before the text. There was no resampler and no Q-Former: just a matrix multiplication and every patch handed to the language model. The follow-up, LLaVA-1.5, described by the same group at CVPR 2024 under the title "Improved Baselines with Visual Instruction Tuning", replaced the linear projection with a small two-layer network, raised the image resolution to 336 pixels (giving 576 visual tokens per image), and added academic question-answering datasets to its training mix. The authors reported that it reached strong results across a broad set of benchmarks and could be trained in about a day on a single node of eight GPUs. Its simplicity and openness made it the template for a large share of the open vision-language models that followed. Why did such a thin connector work so well? Part of the answer is that the CLIP patch features were already aligned with language by contrastive pretraining, so the projection had a relatively easy job: rotating and scaling an already language-shaped space into the particular coordinate system of one language model. Part is that passing all 576 patches, rather than a compressed summary, let the language model's own attention decide what to use for each question; the language model, with billions of parameters, is a far more capable selector than a small bottleneck module. And a large part of the answer lies not in the architecture at all but in the data used to train it, described below. Comparing the connections These designs, together with the early-fusion approach introduced in Chapter 2, represent the main ways to connect vision to a language model. Table 3 compares them on the points that matter for the trade-off between interaction and cost. Table 3. Four ways to connect vision to a language model. Strategy Example Visual tokens seen by the language model Language model weights Main strength Main weakness Gated cross-attention Flamingo (2022) 64 per image via resampler, through inserted layers Frozen Preserves text abilities; handles interleaved images Heavy, complex to train Learned-query bottleneck BLIP-2 (2023) 32 per image as input prefix Frozen Very few trainable parameters Narrow channel loses detail Projection into input LLaVA-1.5 (2023) All patches, 576 at 336 pixels Fine-tuned in second stage Simple; language model selects what to use Token cost grows with resolution Discrete early fusion Chameleon (2024) Discrete image codes in one shared vocabulary Trained from scratch on mixed data Single model reads and writes both modalities Expensive to train; codes trade detail for uniformity Source: Alayrac et al. (2022); Li et al. (2023); Liu et al. (2024); Chameleon Team (2024). The trend over time is toward giving the language model more visual tokens and more freedom to use them, at the cost of compute. That trend makes sense in light of Chapter 5: a bottleneck that discards detail cannot be compensated for downstream, and the language model is the component best placed to judge, question by question, which details matter. Visual instruction tuning The most influential contribution of the original LLaVA paper may not have been its architecture but its training data. A connector trained only on image-caption pairs teaches a system to describe images, but not to answer arbitrary questions, follow instructions, explain its reasoning or hold a conversation. Text-only language models had acquired those abilities through instruction tuning: fine-tuning on examples of instructions paired with good responses. The question was where to get equivalent examples for images, without paying people to write hundreds of thousands of them. LLaVA's answer was to use a text-only model to generate them. For images in the COCO dataset, which come with human-written captions and labelled bounding boxes for objects, the authors gave the captions and box coordinates, as text, to the language-only GPT-4, and asked it to produce three kinds of training example: multi-turn conversations about the image, detailed descriptions, and questions requiring more complex reasoning. GPT-4 never saw the images; it worked entirely from their textual descriptions. The result was about 158,000 instruction-following examples. Training then proceeded in two stages. In the first, only the projection was trained, on a few hundred thousand simple image-caption pairs, to align the visual tokens with the language model's input space. In the second, the projection and the language model were both fine-tuned on the instruction data. The resulting model could carry on a conversation about an image, explain why a picture was funny, and describe a scene in detail, behaviours that had previously been demonstrated mainly by proprietary systems. The technique, visual instruction tuning, became standard. Later datasets broadened the mix to include document and chart understanding, optical character recognition, region-level questions, multi-image reasoning and video. Some groups distilled data from proprietary multimodal models; others, concerned about the circularity and licensing of that approach, collected new human data. The Molmo models from the Allen Institute for AI, described by Matt Deitke and colleagues in 2024, are a notable example of the latter: their accompanying PixMo datasets included detailed image descriptions collected by having annotators speak about an image for a minute or more rather than type, which produced richer and more spatially specific descriptions, and pointing data in which annotators marked the location of objects referred to in text. The quality and character of instruction data turn out to shape a system's behaviour as much as its architecture does. A model tuned mostly on descriptions of salient content will describe salient content well and neglect the rest. A model tuned on data in which answers mention objects that are not in the image, a hazard when instructions are generated from text alone, will learn to do the same. Chapter 8 returns to this problem. Resolution and the token budget The projection approach places a premium on the number of visual tokens, and the token budget became one of the main practical constraints on later systems. At 576 tokens per 336-pixel image, a model cannot read the small print in a scanned contract or distinguish the digits on a distant sign. Raising resolution raises the token count quadratically. Two families of solution emerged. One cuts a high-resolution image into tiles, encodes each tile at the encoder's native resolution, encodes a downsized version of the whole image for global context, and passes all the resulting tokens to the language model. Later versions of LLaVA adopted this approach. The other modifies the vision encoder to accept images at their native size and aspect ratio with a variable number of patches, and adapts the position encoding accordingly; Alibaba's Qwen2-VL, released in 2024, is an example. Both approaches are usually paired with some compression, such as merging adjacent patches, to keep the token count manageable. Neither solution is free. A detailed document image can easily consume several thousand tokens, and a conversation involving several such images can approach the context limits of the language model and slow it considerably. The design question of Chapter 2 reappears here in practical form: how much detail can the system afford to keep, and who decides which detail to keep? Natively multimodal models All the designs above begin with a separately pretrained language model and add vision to it. A different approach trains a single model on mixed data from the start. Google described its Gemini models, announced in December 2023, as natively multimodal, trained from the outset on text, images, audio and video together. OpenAI's GPT-4o, released in May 2024, was described as a single model trained end to end across text, vision and audio. Meta's Chameleon, as described in Chapter 2, used discrete image tokens in a shared vocabulary for the same purpose. The internal details of the commercial systems have not been fully published, so it is not possible to say exactly how their visual inputs are processed. The rationale for native multimodality is that modalities learned together from the beginning may integrate more deeply than modalities bolted on afterwards, and that a single model can both read and produce any modality. The cost is that every training run must handle all modalities at full scale, which is enormously expensive, and that the clean modular reuse of pretrained components is lost. For most practitioners outside the largest laboratories, the modular approach, a pretrained vision encoder plus a connector plus a pretrained language model, remains the practical choice, and research such as Cambrian-1, published by Shengbang Tong and colleagues at NeurIPS 2024, has focused on systematically comparing encoders, connectors and data within that framework. What the connection adds and what it cannot Connecting a vision encoder to a language model brings something genuinely new: a system that can bring the full reasoning and knowledge of a language model to bear on what it sees. It can identify a plant and explain how to care for it, read a chart and draw a conclusion, look at a photograph of a circuit board and suggest where a fault might lie. This is the capability that made multimodal assistants useful outside research. But the connection is only as good as what flows through it. The language model sees what the encoder preserved, compressed by whatever the connector chose to keep, and interpreted according to habits learned from instruction data. Each stage can lose or distort information, and the language model, being fluent, will produce a confident answer whether or not the information it needed survived. The final chapters ask what happens when a third modality, sound, is added to this picture, and then what it takes to make the system's reasoning reliably grounded in what it actually perceives. CHAPTER 7 Hearing and Speaking Vision and language have dominated the discussion so far, as they have dominated the field. But sound is in many ways the more intimate modality. It is how most people first learn language, how they hold most of their conversations, and how they detect most of the events around them that they cannot see: a kettle boiling, a car approaching from behind, a change in someone's tone of voice. A system that is to share a human's world needs to hear it. Audio poses its own version of every problem in this book. It must be tokenised, and Chapter 2 showed that this produces long sequences. It must be aligned with language, and paired audio-text data is far scarcer than paired image-text data. It must be connected to a reasoning system, and the connection must preserve not just the words spoken but everything carried in how they were spoken. And because conversation happens in real time, a system that listens must also answer quickly enough to feel like a partner rather than a voicemail service. This chapter follows those problems in turn. Speech recognition at scale The single most important development in machine listening during the 2020s was the demonstration that speech recognition, like image recognition, benefits enormously from training on large quantities of loosely labelled data rather than small quantities of carefully labelled data. Whisper, introduced by Alec Radford, Jong Wook Kim and colleagues at OpenAI in 2022 and presented at ICML in 2023 under the title "Robust Speech Recognition via Large-Scale Weak Supervision", was trained on 680,000 hours of audio paired with transcripts collected from the internet. The transcripts were not produced for the purpose; they were whatever text accompanied the audio, filtered heuristically to remove, among other things, transcripts that appeared to have been generated by existing speech recognisers. About a third of the data was in languages other than English, and part of it paired non-English audio with English text, which allowed the model to learn translation as well as transcription. Whisper's architecture was conventional: a Transformer encoder over the log-mel spectrogram described in Chapter 2, and a Transformer decoder that generated text tokens while cross-attending to the encoder's output. Its novelty lay in the breadth of data and in a simple interface for multiple tasks. Special tokens at the start of the decoder's output specified the language, whether to transcribe or translate into English, and whether to produce timestamps. One model did all of these things. The headline finding mirrored CLIP's. Speech recognisers trained on a standard benchmark such as LibriSpeech, a corpus of read audiobooks, could achieve very low error rates on that benchmark and much higher ones on other kinds of speech: accented, noisy, conversational or recorded with different equipment. Whisper, evaluated without any training on the target datasets, was far more robust across this variety. Its errors on LibriSpeech were higher than those of the best specialised systems, but its errors on everything else were much lower. The authors argued, as the CLIP authors had, that a model trained on one curated dataset partly learns that dataset's peculiarities, while a model trained on the messy diversity of the web learns something closer to the general skill. Whisper also illustrated a failure mode that recurs across generative multimodal systems. Because its decoder is a language model, it can produce fluent text that was not spoken. During long silences or in noisy passages, it occasionally inserts words or whole sentences. A 2024 study by Allison Koenecke and colleagues, presented at the ACM Conference on Fairness, Accountability, and Transparency, examined Whisper's transcriptions of speech from people with aphasia and from control speakers and found that around one per cent of transcriptions contained hallucinated phrases or sentences, some of them harmful, and that hallucinations were more frequent for speakers with aphasia, whose speech contains longer pauses. For a transcription system used in medical or legal settings, that rate matters. Aligning sound with language Speech recognition is translation from one modality to another. The contrastive approach of Chapter 4 offers a different and complementary capability: a shared space in which sounds and descriptions of sounds can be compared, so that a system can retrieve "a dog barking in the distance" from a sound library, or classify an audio clip into categories it has never been trained on. Several groups applied the CLIP recipe directly to audio. Microsoft's CLAP, short for Contrastive Language-Audio Pretraining, described by Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail and Huaming Wang at ICASSP 2023, trained an audio encoder and a text encoder to align clips with their captions. A parallel effort from the LAION community, described by Yusong Wu and colleagues at the same conference, assembled a dataset called LAION-Audio-630K of about 630,000 audio-text pairs and trained a similar model. Both demonstrated zero-shot audio classification and text-to-audio retrieval. The numbers reveal the central difficulty. CLIP trained on 400 million image-text pairs, and LAION-5B offered billions. Audio-text datasets number in the hundreds of thousands or low millions of pairs. Sound on the web is rarely accompanied by a description of what it sounds like; people caption photographs, but they seldom write "rain on a tin roof, with distant thunder" next to a recording. The audio-text space is therefore much less richly aligned than the image-text space, and audio-language models have correspondingly narrower knowledge. Several groups have attempted to close the gap with machine-generated audio captions, using language models to turn sound-event tags and metadata into fluent descriptions, much as later image models used generated captions. Music illustrates both the promise and the difficulty. Google's MuLan, described by Qingqing Huang and colleagues in 2022, trained a music-text embedding contrastively on music clips paired with text drawn from the titles, tags and descriptions of the videos in which they appeared, and from playlists. The resulting space placed a jazz piano trio near descriptions of "smooth, late-night piano" and a distorted guitar riff near "aggressive heavy metal". MuLan was then used inside MusicLM, described by Andrea Agostinelli and colleagues in 2023, to condition the generation of music from a text prompt. The text attached to music is often about the artist or the occasion rather than the sound, and descriptive vocabulary for timbre, rhythm and texture is sparse and inconsistent even among experts. A music-text model can learn genre and mood readily and fine musical structure poorly, for the same reason that an image-text model learns subject matter readily and spatial arrangement poorly: the text rarely makes the finer distinction. Binding modalities through images A way around the scarcity of audio-text data was proposed by Rohit Girdhar and colleagues at Meta in ImageBind, presented at CVPR in 2023. The idea rests on a simple observation: although audio rarely comes with text, it very often comes with images, in the form of video. The same is true of depth maps, which come from cameras with depth sensors; of thermal imagery; and of motion sensor readings from devices worn while video is being recorded. ImageBind therefore used images as a hub. It started from a CLIP-style image-text model and then trained encoders for audio, depth, thermal imagery and inertial measurement data, each to align with the image encoder using pairs naturally found together: video frames and their soundtracks, depth images and their colour images, and so on. No modality other than images was ever paired directly with text, and no pair of non-image modalities was ever paired with each other. The striking result was what the authors called emergent alignment. Because audio was aligned with images, and images were aligned with text, audio ended up aligned with text, even though the model had never seen an audio clip paired with a caption. The system could retrieve sounds from text descriptions and classify sounds zero-shot with reasonable accuracy. It could even do arithmetic across modalities: adding the embedding of an image of fruit to the embedding of the sound of birds would retrieve images of fruit with birds. The six modalities, images, text, audio, depth, thermal and motion, were bound into a single space through their common relationship to vision. ImageBind's emergent alignment was weaker than direct alignment would have been, and the binding inherited all the limitations of the CLIP-style space described in Chapter 5. But it demonstrated a useful principle. When direct paired data between two modalities is scarce, each can be aligned to a third modality for which paired data is plentiful, and the alignment will partly transfer. Listening and looking together Audio and vision interact in ways that go beyond either one's relationship with text. The McGurk effect of Chapter 1 shows that human speech perception relies on seeing lips. Machine systems can exploit the same correlation. One line of work learns from the correspondence between sound and image alone, without any labels. Relja Arandjelović and Andrew Zisserman's "Look, Listen and Learn", presented at ICCV in 2017, trained networks simply to decide whether a video frame and a one-second audio clip came from the same moment of the same video. To do this, the networks had to learn what things look like when they make particular sounds, and the resulting representations were useful for both audio and image classification. Related work showed that such models could localise which region of a frame was producing a sound, learning to "see" the source of a noise without ever being told what the source was. Another line exploits the link between lip movements and speech. Audio-visual speech recognition, of which AV-HuBERT, described by Bowen Shi and colleagues at Meta in 2022, is a well-known self-supervised example, combines a video of a speaker's mouth with the audio of their speech. The visual stream is especially valuable in noise, where the audio alone is ambiguous, just as humans watch a speaker's face more closely in a loud room. These systems are reminders that multimodal fusion is not only a matter of attaching sensory inputs to a language model. Some of the most informative relationships are between the senses themselves, and a model that reaches the language model only after each sense has been summarised separately has lost them. From transcripts to listening language models The simplest way to give a language model hearing is to transcribe audio into text first and pass the transcript to the language model. This cascaded approach is modular, cheap and works well for many purposes: a recorded meeting can be transcribed and summarised, a voice command converted to text and executed. But a transcript discards a great deal. It drops the identity of the speaker, the emotion in their voice, their emphasis, their hesitation, the sarcasm that only tone reveals, the laughter in the background and every sound that is not speech. If a user asks, "Does this person sound confident?" or "What is that noise in the background?", a transcript cannot answer. Systems built from 2023 onward therefore began to apply the vision-language recipe of Chapter 6 to audio: take a pretrained audio encoder, often the encoder half of Whisper, project its output frames into the language model's input space through a connector, and fine-tune on instruction data involving audio. Models of this kind, such as Alibaba's Qwen-Audio and the SALMONN model from Tsinghua University and ByteDance, both described in 2023, could answer questions about speech, music and environmental sound, not merely transcribe it. The connector design choices, whether to compress the long audio sequence through learned queries or pass it more fully, mirrored those for vision, with the added pressure that audio sequences are long. Speaking back Hearing is half of a conversation. The other half is producing speech, and here the discrete audio tokens of Chapter 2 become central. If speech can be represented as a sequence of codec tokens, then a language model can generate speech the same way it generates text, by predicting the next token. Microsoft's VALL-E, described by Chengyi Wang and colleagues in 2023, framed text-to-speech in exactly this way: given text and a three-second recording of a speaker, a language model predicted codec tokens that, when decoded, produced the text spoken in that speaker's voice, with their timbre and even the acoustic character of the recording environment. The quality was high enough that the authors explicitly discussed the risk of misuse for impersonation, and the ease of voice cloning from short samples has since become a significant concern for fraud and disinformation. For conversation, the decisive issue is latency. A cascaded voice assistant runs speech recognition, then a language model, then speech synthesis, in sequence; each stage waits for the previous one, and the delays add up. In ordinary human conversation, the gap between one speaker finishing and the next starting is typically a few hundred milliseconds. When OpenAI introduced GPT-4o in May 2024, it reported that the model could respond to audio input in as little as 232 milliseconds, with an average of 320 milliseconds, which it compared with human response times in conversation, and described the model as processing audio end to end rather than through a cascade. Later that year the French laboratory Kyutai released Moshi, described by Alexandre Défossez and colleagues, a speech-to-speech model designed for full-duplex conversation: it listens and speaks simultaneously, modelling the user's audio stream and its own as parallel token streams, so that it can handle interruptions and overlapping speech rather than strictly taking turns. The difference between the cascaded and end-to-end approaches is a version of the fusion trade-off from Chapter 1, and Table 4 sets it out. Table 4. Cascaded and end-to-end approaches to voice interaction. Property Cascaded pipeline End-to-end speech model Structure Recogniser, then language model, then synthesiser One model consumes and produces audio tokens Information passed to reasoning Transcript only Audio representation including tone and non-speech sound Latency Sum of three stages Can begin responding while still listening Turn-taking Strict turns, usually detected by silence Can model interruptions and overlap Modularity Each component replaceable and auditable Components cannot be separated or inspected independently Failure tracing Errors visible in intermediate transcript No intermediate text to inspect The cascade is a form of late fusion through a narrow textual bottleneck; the end-to-end model is a form of deep fusion with all the expressiveness and opacity that implies. Neither is strictly better. A medical dictation system may be better served by the cascade, whose intermediate transcript can be checked. A conversational companion that must respond to the user's mood benefits from end-to-end processing. What sound teaches about fusion Adding audio to multimodal systems does not introduce new principles so much as it stresses the existing ones. The scarcity of paired audio-text data shows how much the success of vision-language models owed to the abundance of captioned images on the web. The way ImageBind bound audio to text through images shows that alignment can be transitive, but also that transitive alignment is weaker than direct. The difference between a transcript and a listening model shows, more sharply than any vision example, how much information is lost when one modality is translated into another before fusion. And the demands of real-time conversation show that the choice of where to fuse is not only a matter of accuracy but of time. The thread connecting these is the one that runs through the book. The more finely and directly modalities are aligned, the more a system can do with them, and the more it costs in data, computation and transparency. The final chapter asks what all of this adds up to when a system is asked to do the hardest thing: look at a scene, read an instruction, and act on it correctly. CHAPTER 8 Reasoning over Scenes and Instructions Consider an instruction that a household robot, a visual assistant for a blind user, or an agent operating a computer might receive: "Hand me the blue mug that's behind the kettle, not the one by the sink." To act on it, a system must parse a sentence with a relative clause and a negation, find mugs in a scene, determine their colours, establish which one is behind the kettle from the camera's viewpoint, rule out the one by the sink, and then either describe the chosen object's location or physically reach for it. Every component of this book is involved: tokenisation that preserves enough detail to see colours and occlusion, alignment fine enough to bind "blue" to the right object, fusion that combines spatial relations with language, and a reasoning process that holds the constraints together. This is the capability that the description of modern foundation models most often promises: reasoning simultaneously over visual scenes and natural-language instructions. It is also where the gap between fluent output and reliable understanding is widest. This chapter examines what grounded reasoning requires, how it is measured, why systems hallucinate, and what approaches have proved effective at narrowing the gap. What grounding means A system's language is grounded when its words are connected to specific things in its perception: when "the blue mug" in its output refers to an identifiable object in the image, and when it would change its answer if the image changed. Grounding is the difference between a model that says "there is a blue mug behind the kettle" because it sees one and a model that says so because blue mugs and kettles often appear together in kitchen photographs. The most direct test of grounding is to ask a model to point. Referring expression comprehension, a task studied since the ReferItGame dataset of Sahar Kazemzadeh and colleagues in 2014 and its successors built on COCO images, asks a system to output the location of the object a phrase describes: "the man in the red shirt on the left", "the second plate from the top". If the model can mark the correct box, its understanding of the phrase is at least partly grounded. If it can only describe the scene in general terms, it may not be. Several vision-language systems have been designed to produce locations as part of their output. Microsoft's Kosmos-2, described by Zhiliang Peng and colleagues in 2023, represented image locations as special tokens in the language model's vocabulary, so that the model could write a sentence in which noun phrases were linked to boxes. The Molmo models mentioned in Chapter 6 were trained on large quantities of pointing data, in which annotators clicked on the objects referred to in text, and could answer a request such as "point to every person wearing a hat" by producing a set of coordinates, which also gave them an effective way to count: point to each instance, then count the points. Grounded output of this kind is valuable twice over. It makes the system's reasoning inspectable, since a user can see which object the model meant, and it trains the model to keep its words attached to locations. How visual reasoning is measured Evaluating multimodal reasoning has proved almost as hard as achieving it, because models are adept at finding shortcuts in benchmarks. Chapter 1 described how the original VQA dataset could be partly solved without looking at the image. That lesson has had to be relearned repeatedly. Benchmarks have evolved in response. Some isolate specific skills under controlled conditions. CLEVR, introduced by Justin Johnson and colleagues at CVPR 2017, used synthetic rendered scenes of simple geometric objects and automatically generated questions such as "Are there an equal number of large things and metal spheres?", making it possible to test counting, comparison and spatial reasoning without real-world clutter or language priors. GQA, from Drew Hudson and Christopher Manning at CVPR 2019, generated compositional questions about real photographs from detailed annotations of the objects and relationships in each scene. Winoground, POPE and MMVP, discussed in Chapter 5 and below, each target a particular failure. Others aim at breadth and difficulty. MMMU, introduced by Xiang Yue and colleagues at CVPR 2024, collected about 11,500 questions from college exams, quizzes and textbooks across six broad disciplines and 30 subjects, many requiring expert knowledge applied to diagrams, charts, chemical structures, sheet music or medical images. It was designed to test whether multimodal models could reason at the level of a skilled student rather than merely recognise everyday objects. Breadth brings its own risk. A 2024 study by Lin Chen and colleagues, titled "Are We on the Right Way for Evaluating Large Vision-Language Models?", found that a substantial fraction of questions in popular multimodal benchmarks could be answered correctly without the image at all, either because the answer was implied by the question and options or because it was common knowledge, and that some language models without any vision scored well above chance on supposedly visual tests. The authors assembled a filtered benchmark, MMStar, of questions verified to require the image. The underlying lesson is that any multimodal benchmark should be checked against a "blind" baseline. If a model does almost as well with the image removed, the benchmark is not measuring vision. Table 5 summarises several widely used benchmarks and what each is designed to probe. Table 5. Selected benchmarks for multimodal reasoning. Benchmark Year What it tests Construction Known caveat VQA v2 2017 Open-ended questions about everyday photographs Paired images with different answers to the same question Many questions still answerable from common sense GQA 2019 Compositional and relational questions Generated from scene-graph annotations Templated question phrasing Winoground 2022 Word-order compositionality Hand-built pairs of images and reordered captions Small; some items need unusual knowledge POPE 2023 Object hallucination Yes-or-no questions on object presence Covers only presence, not attributes or relations MMMU 2024 Expert-level multidisciplinary reasoning College exam and textbook questions Some items answerable without the image MMVP 2024 Fine visual details missed by CLIP-style encoders Questions on image pairs CLIP embeds as similar Narrow by design Source: Goyal et al. (2017); Hudson and Manning (2019); Thrush et al. (2022); Li et al. (2023); Yue et al. (2024); Tong et al. (2024). Why models hallucinate The most persistent and damaging failure of vision-language systems is object hallucination: describing things that are not in the image. A model shown a photograph of a dining table with plates and glasses may confidently add "and a vase of flowers in the centre". Asked whether there is a fork, it may say yes when there is none. The problem was quantified for image captioning by Anna Rohrbach and colleagues in a 2018 paper at EMNLP, "Object Hallucination in Image Captioning", which introduced a metric called CHAIR that counted the proportion of objects mentioned in generated captions that were absent from the image's annotations. For the large vision-language assistants that followed, Yifan Li and colleagues proposed POPE at EMNLP 2023: a polling method that asks a model a series of yes-or-no questions of the form "Is there a [object] in the image?", with the absent objects chosen either at random, from the most frequent objects in the dataset, or adversarially from objects that frequently co-occur with those actually present. Models hallucinated most under the adversarial setting. Asked about a fork in an image of a dining table, they were much more likely to say yes than when asked about a fork in an image of a beach. That pattern points to the principal cause. A vision-language model generates text with a language model that has learned, from vast quantities of text and from image captions, which objects tend to appear together. When the visual evidence is weak, ambiguous or absent, the language prior fills the gap. Dining tables have forks; kitchens have kettles; beaches have umbrellas. The model is not lying; it is doing what a language model does, completing a plausible description, and the visual input is not strong enough to override the completion. Several factors from earlier chapters make this worse. The visual encoder may have discarded the very detail that would settle the question, as Chapter 5 described. The connector may have compressed the visual tokens so that small or peripheral objects are under-represented. The instruction data may itself contain hallucinations: when a text-only model generates conversations from a caption and a list of objects, it can introduce plausible details that are not in the image, and a model fine-tuned on those conversations learns that such embellishment is acceptable. And longer outputs hallucinate more, because as a description extends, the model increasingly attends to its own previous text rather than to the image. Reducing hallucination The remedies divide into those that change the data, those that change the training objective, and those that change how the model generates its answer. Data remedies attack the sources directly: instruction data written or verified by people who could see the image, detailed descriptions that cover the whole scene rather than its most salient parts, and deliberately included examples in which the correct answer is "no" or "I cannot tell from this image". A model that has never been trained to say "there is no fork" will rarely say it. Objective remedies borrow from the alignment of text-only language models. Reinforcement learning from human feedback, in which people rate pairs of model responses and the model is trained to prefer the higher-rated ones, has been adapted to vision-language models. One early effort, LLaVA-RLHF, described by Zhiqing Sun and colleagues in 2023, gave the reward model access to additional factual information about the image, such as human-written captions, so that it could penalise responses mentioning absent objects. Preference-based training of this kind teaches the model that accuracy matters more than fluency. Decoding remedies intervene at generation time. Visual contrastive decoding, proposed by Sicong Leng and colleagues at CVPR 2024, generates each token twice, once with the real image and once with a distorted version, and favours tokens whose probability depends strongly on the real image. Tokens that would be predicted just as readily from a blurred or noisy image are likely to come from the language prior rather than from visual evidence, and are down-weighted. Such methods require no retraining, though they add computation. None of these eliminates the problem. Hallucination is a consequence of the basic architecture: a generative language model conditioned on an imperfect visual summary. It can be reduced but not removed, and any application where an invented object would be costly, such as describing a scene to a blind user or reading a medical image, needs a design that allows for it, typically by asking for grounded output that can be checked or by keeping a human in the loop. Reasoning step by step Text-only language models became markedly better at multi-step problems when prompted or trained to write out intermediate reasoning before giving an answer, a technique known as chain-of-thought. Multimodal models benefit similarly: asked "how many more red blocks than blue blocks are there?", a model that first lists the red blocks, then the blue blocks, then subtracts, does better than one that answers directly. Many recent systems are trained to produce extended reasoning before answering, and this has substantially improved scores on benchmarks such as MMMU and on mathematical problems presented in diagrams. There is an important limitation. Reasoning in words is only as good as the perceptions it starts from. If the model miscounts the red blocks in its first step, careful arithmetic in the second step will not help. Chain-of-thought improves the combination of facts but does not improve perception, and on tasks bounded by perception, such as those in MMVP, it helps much less. A different approach treats perception itself as something to be reasoned about with tools. VisProg, by Tanmay Gupta and Aniruddha Kembhavi, which received a best paper award at CVPR 2023, and ViperGPT, by Dídac Surís, Sachit Menon and Carl Vondrick at ICCV 2023, had a language model write a short program to answer a visual question. The program called specialised vision modules, an object detector, a depth estimator, a cropping function, a text-image matching model, and combined their outputs with ordinary code. To answer "how many muffins can each child have if they share equally?", the program would detect and count the muffins, detect and count the children, and divide. Because each step is explicit, errors can be traced, and because counting is done by a detector and arithmetic by code, the system does not depend on the language model's unreliable internal counting. More recent systems combine both ideas, reasoning in text while being able to zoom into regions of an image, crop and re-encode them at higher resolution, or call external tools. This brings the question of Chapter 2 full circle: rather than deciding in advance which details to keep, the system looks again when its reasoning requires it. From answering to acting The most demanding form of reasoning over scenes and instructions is acting in the world. If a model's output is not a sentence but a motor command, hallucination is no longer an inaccuracy but a collision. Google's PaLM-E, described by Danny Driess and colleagues at ICML 2023, injected visual observations and robot state information into a large language model, in the manner of Chapter 6, and used the model to produce high-level plans for robots: sequences of steps such as "pick up the green block, place it in the bowl". The paper reported that training on a mixture of robotics data and general vision-language data produced better robot planning than training on robotics data alone, an instance of the co-learning described in Chapter 1, with knowledge from web-scale images and text helping in a domain where data is scarce. RT-2, from Anthony Brohan and colleagues at Google DeepMind in 2023, went further and had a vision-language model output low-level robot actions directly, by representing the movements of a robot arm as strings of numbers treated as text tokens. Such systems became known as vision-language-action models. Because the underlying model had been pretrained on web data, RT-2 could follow instructions involving concepts that never appeared in its robot training data, for example moving an object to a picture of a particular celebrity or picking up an item that could serve as an improvised hammer. OpenVLA, released by Moo Jin Kim and colleagues in 2024, provided an open model of the same kind trained on a large collection of robot demonstrations gathered from many laboratories. A parallel development uses screenshots rather than camera images. Agents that operate computers or phones on a user's behalf must read a screen, locate buttons and fields, interpret instructions, and click or type. Their reasoning is over a visual scene in the most literal sense, and their failures, misreading small text, clicking the wrong element, or following instructions embedded in a web page by an adversary, are the failures of this book in a new setting. The typographic attacks of Chapter 5 become prompt injection attacks when a model that reads text in images is also empowered to act. The state of grounded reasoning Grounded reasoning over scenes and instructions has advanced remarkably. Systems can now read documents, interpret charts, answer questions requiring expert knowledge, describe scenes in rich detail, point to objects, and in constrained settings direct robots. These abilities were science fiction a decade ago. Yet the advances are uneven in a way this book's argument predicts. The capabilities that have advanced fastest are those that depend most on the breadth of knowledge that coarse alignment and language models provide: recognising objects and concepts, recalling facts about them, explaining, summarising and conversing. The capabilities that have advanced most slowly are those that depend on fine alignment: exact counts, precise spatial relationships, small details, the absence of objects, and the faithful binding of each word to the right thing. Progress on the latter has come mainly from changes that add fine-grained signal, such as grounded training data, pointing, detailed captions, harder negatives, higher-resolution inputs and the ability to look again. That is not a coincidence. It is the trade-off at the heart of multimodal learning, and it determines where the next gains are likely to come from. Conclusion: Where Alignment Is Learned The argument of this booklet has been that multimodal deep learning reduces to two questions: where should different modalities meet inside a network, and what training signal should teach them to agree? The preceding chapters have traced how the field has answered those questions, from the separate specialist systems of the early 2010s, through the cross-attention Transformers of 2019, the contrastive encoders of 2021, and the vision-language assistants of 2023 onward, to systems that hear, speak and act. It remains to say what follows from that history: what someone building, choosing or relying on these systems should do differently because of it, and what is still unresolved. Coarse alignment is cheap; fine alignment is earned The single most useful lesson is that coarse and fine alignment are different achievements with different costs, and that the field's successes have come overwhelmingly from the first. Coarse alignment, matching whole images to whole captions or whole clips to whole descriptions, can be learned from data that the internet produces as a side effect of people sharing pictures and sounds. It scales to billions of examples with little human effort, and it produces representations of extraordinary breadth. CLIP, ALIGN, SigLIP and their relatives showed that a simple contrastive objective, applied at that scale, yields a visual vocabulary as wide as the web. Whisper showed the same for speech. Almost every capable multimodal system now rests on encoders trained this way. Fine alignment, binding particular words to particular regions, counting instances, tracking spatial relations, noticing what is absent, is not produced as a side effect of anything. It must be demanded by the training signal, through hard negatives, region-level annotations, pointing data, detailed human descriptions or carefully constructed instruction data, and every one of these is more expensive per example than a scraped caption. Chapter 5 showed that models lack fine alignment not because they cannot represent it but because nothing in their training made it worth representing. Chapter 8 showed that the progress that has been made on grounded reasoning has come largely from adding exactly this kind of signal. The implication for practitioners is direct. When a multimodal system fails, the first question to ask is whether the failure is one of breadth or of precision. Failures of breadth, such as not recognising an unusual object or not knowing a fact about it, tend to improve with larger models and more data of the usual kind. Failures of precision, such as miscounting, confusing left and right or describing an absent object, tend not to, and they call for targeted data, higher resolution, a better visual encoder, or a design that checks the model's perceptions rather than trusting them. Choosing a design for the job The trade-offs described in this book translate into fairly concrete guidance about which kind of system suits which task. For search, retrieval, deduplication, clustering and zero-shot classification over large collections, a dual-encoder contrastive model is usually the right tool. Its embeddings can be computed once and compared cheaply, it handles open-ended vocabularies, and its weaknesses in fine detail rarely matter for finding images "of a red bicycle by a canal". Where they do matter, such as distinguishing near-identical products, it can be fine-tuned with hard negatives drawn from the actual confusions. For open-ended questions, explanation, document reading and conversation about images, a vision-language model built from an encoder, a connector and a language model is appropriate. Here the critical choices are often not the language model but the visual front end: what resolution it sees, how many tokens reach the language model, and whether its encoder preserves the kinds of detail the task requires. A system that must read fine print needs high resolution or tiling; a system that must judge orientation or state may need features from a self-supervised encoder alongside contrastive ones. For tasks where an error is costly, such as describing scenes to people who cannot see them, extracting values from medical or financial documents, or directing a physical action, grounded output should be a requirement rather than a feature. A system that marks which region it is talking about, or that routes counting and measurement through explicit tools, allows its perceptions to be checked. A system that only produces fluent prose does not. And for evaluation, every multimodal benchmark result should be read alongside two simple checks. How well does the model do with the image removed? And how does it do on the specific kinds of detail the application depends on, rather than on an average across a broad benchmark? The history of the field, from VQA's language priors to the blind-solvable questions found in recent benchmarks, shows that headline numbers routinely overstate how much a model is actually looking. What remains unsettled Several questions that the field has not resolved bear directly on where it goes next. Modular or native. The modular recipe, a separately pretrained encoder joined to a separately pretrained language model, has dominated open research because it reuses expensive components and is cheap to adapt. Natively multimodal models, trained on mixed data from the start, promise deeper integration and the ability to produce as well as perceive every modality. It is not yet clear whether native training yields fundamentally better grounding or mainly better fluency across modalities, and the largest natively multimodal systems have not been described in enough detail for outsiders to judge. The answer matters because it determines whether progress will continue to be possible for groups without the resources to train everything from scratch. Discrete or continuous. Treating every modality as discrete tokens in one vocabulary allows a single objective and a single model to do everything, but quantisation discards detail and reconstruction-trained codebooks may not carry the semantic structure that contrastively trained encoders provide. Continuous encoders preserve more but are harder to generate from. Hybrid schemes exist; no consensus does. Time. Video and audio both unfold in time, and current systems handle time crudely: by sampling frames sparsely, compressing long recordings or transcribing speech into text. Questions that depend on exactly when things happened, in what order, and for how long remain hard. Real-time conversational systems have begun to treat time as central rather than incidental, and similar treatment of video, in which a system watches continuously rather than inspecting a handful of frames, is an open frontier. Data. The supply of naturally occurring paired data is finite, and the most easily scraped portion has largely been used. Increasingly, multimodal training relies on synthetic data: captions written by models, instructions generated by models, preferences judged by models. Synthetic data can supply exactly the fine-grained signal web data lacks, as recaptioning has shown, but it can also propagate a model's errors into the next model's training, including its hallucinations. How to generate synthetic multimodal data that adds grounding rather than amplifying confident invention is one of the most practically important open problems. Security and trust. Systems that read text in images and can act on what they read are vulnerable to instructions planted in the content they process. Systems that can clone a voice from seconds of audio can be used for impersonation. Systems trained on web-scale data inherit its biases and, as the LAION episode showed, can inherit worse. These are not peripheral concerns to be addressed after the engineering is done. In a field where a model's knowledge is a function of its data and its behaviour a function of its objective, they are engineering questions. The shape of the next advance If the argument of this booklet is right, the most important advances of the coming years will not come primarily from new mechanisms for fusing modalities. Cross-attention, contrastive learning and the projection of one modality into another's input space are flexible enough to express nearly any interaction a system might need. The constraint is the signal that teaches those mechanisms what to align. That suggests where to look. Systems that can choose to look again at the parts of a scene their reasoning depends on, rather than reasoning from a fixed summary, address the tokenisation bottleneck of Chapter 2. Training data that ties every word of a description to a location in the image, or every claim in an answer to a piece of evidence, addresses the alignment gap of Chapter 5. Objectives that reward a model for saying what is absent as well as what is present address the hallucination problem of Chapter 8. And designs that keep perceptions inspectable, through pointing, grounded output or explicit tool use, make it possible to trust a system's reasoning without taking its fluency at face value. The child who hears "dog" while looking at a dog builds an alignment between word and world that is coarse at first and becomes finer with every interaction: this dog, that tail, the dog behind the sofa rather than the one on it. Machines have learned the coarse alignment from the internet with astonishing speed. The fine alignment, the part that lets language reliably refer to what is actually there, has to be taught more deliberately, and the systems that are taught it best will be the ones that can be trusted to reason over what they see and hear. Further Reading Simon J. D. Prince, Understanding Deep Learning (MIT Press, 2023).** A clear, carefully illustrated modern textbook that covers Transformers, attention and self-supervised learning with unusual precision; the best single source for the foundations assumed in Chapters 2 and 3. Christopher M. Bishop and Hugh Bishop, Deep Learning: Foundations and Concepts (Springer, 2024).** A rigorous treatment of the field's core ideas, including Transformers and representation learning, written for readers who want the mathematics behind the mechanisms. Tadas Baltrušaitis, Chaitanya Ahuja and Louis-Philippe Morency, "Multimodal Machine Learning: A Survey and Taxonomy" (IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019). The survey that introduced the five-part taxonomy of representation, translation, alignment, fusion and co-learning used in Chapter 1; still the best map of the field's problems as distinct from its architectures. Paul Pu Liang, Amir Zadeh and Louis-Philippe Morency, "Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions" (ACM Computing Surveys, 2024). A broader and more recent survey, with a useful treatment of how modalities interact through redundancy, uniqueness and synergy. Ashish Vaswani and colleagues, "Attention Is All You Need" (Advances in Neural Information Processing Systems, 2017). The paper that introduced the Transformer; short, readable and the origin of the query, key and value formulation of attention. Alec Radford and colleagues, "Learning Transferable Visual Models From Natural Language Supervision" (Proceedings of ICML, 2021). The CLIP paper. Its long appendices on zero-shot evaluation, robustness, prompt engineering and bias are as instructive as the main text. Jean-Baptiste Alayrac and colleagues, "Flamingo: a Visual Language Model for Few-Shot Learning" (Advances in Neural Information Processing Systems, 2022). A detailed account of gated cross-attention, the Perceiver Resampler and training on interleaved web documents. Haotian Liu, Chunyuan Li, Qingyang Wu and Yong Jae Lee, "Visual Instruction Tuning" (Advances in Neural Information Processing Systems, 2023). The LLaVA paper, which established the projection-plus-instruction-tuning recipe used by many open vision-language models. Mert Yuksekgonul and colleagues, "When and Why Vision-Language Models Behave like Bags-of-Words, and What to Do About It?" (Proceedings of ICLR, 2023). A clear diagnosis of why contrastive training neglects composition, with a simple and effective remedy. Shengbang Tong and colleagues, "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs" (Proceedings of CVPR, 2024). Shows how the limitations of a contrastive visual encoder propagate into the language models built on it; essential reading for anyone choosing a visual front end. Alec Radford and colleagues, "Robust Speech Recognition via Large-Scale Weak Supervision" (Proceedings of ICML, 2023). The Whisper paper, the audio counterpart to CLIP's argument that diverse, loosely labelled data produces robust models. Rohit Girdhar and colleagues, "ImageBind: One Embedding Space To Bind Them All" (Proceedings of CVPR, 2023). Demonstrates emergent alignment across six modalities by using images as a hub, and is a useful case study in what transitive alignment can and cannot achieve. Hashatgs: #MultimodalDeepLearning #MultimodalAI #ArtificialIntelligence #DeepLearning #MachineLearning #GenerativeAI #VisionLanguageModels #VisionLanguageAudio #MultimodalFusion #CrossModalLearning #MultimodalAlignment #CrossAttention #SelfAttention #ContrastiveLearning #ContrastivePretraining #CLIP #VisionTransformers #ImageTextAlignment #AudioTextAlignment #SpeechRecognition #AudioProcessing #ComputerVision #NaturalLanguageProcessing #MultimodalTransformers #EarlyFusion #LateFusion #IntermediateFusion #VisualQuestionAnswering #MultimodalReasoning #MultimodalFoundationModels

Download the Book (PDF) Introduction In the spring of 2023 a group at Stanford released a model called Alpaca. It was a seven-billion-parameter language model, small by the standards of the time, and its creators claimed that in their own informal tests it behaved much like one of OpenAI's far larger instruction-following systems. What made the release notable was not the model but the recipe. The team had started with 175 hand-written example tasks, asked a commercial model to invent tens of thousands more in the same spirit, used those invented examples to fine-tune an openly available base model, and reported spending less than six hundred dollars on the whole exercise. No annotators were hired. No users were surveyed. The training data had been manufactured. Within weeks there were dozens of imitators. Within a year the practice had moved from academic curiosity to industrial default. By mid-2024 NVIDIA reported that more than ninety-eight percent of the data it used to align its Nemotron-4 340B model had been generated synthetically. Google DeepMind's AlphaGeometry, which solved olympiad geometry problems at close to the level of a gold medallist, had been trained without a single human-written proof; its hundred million training examples were produced by a symbolic engine. Microsoft's Phi models built their reputations on "textbook quality" material written largely by other models. Synthetic data had become something the field could no longer treat as a workaround. That shift happened for reasons that are easy to state. The supply of high-quality human text is finite, and the largest training runs are approaching the point where they consume most of what is publicly available. Much of the data that would be most valuable for specialised systems — medical records, financial transactions, legal case files, private conversations — is locked behind privacy law, contractual obligation, or simple decency. And some of the situations a model most needs to handle are rare by nature: the unusual tumour, the fraudulent transaction pattern nobody has seen before, the pedestrian stepping out from behind a parked bus at dusk. Waiting for reality to produce enough examples of rare events is not a strategy. Manufacturing them is. The argument of this book The ease of manufacture is precisely the problem. A large language model can produce a million plausible-looking training examples in an afternoon. It can produce a million plausible-looking wrong ones just as fast, and the two sets will be hard to tell apart by inspection. Generated text carries the generator's blind spots, its stylistic tics, its confident errors, and its tendency to drift toward the bland middle of whatever distribution it learned. Train a model on that output without discrimination and the model inherits all of it. Repeat the process across generations and, as a much-cited 2024 study in Nature demonstrated, the resulting models progressively forget the rare and the unusual until their outputs collapse toward a narrow, distorted average. The controlling claim of this book follows from that tension. In a synthetic data system, generation is the cheap part; the architecture is the verifier. The value of a synthetic dataset is set almost entirely by the quality of the machinery that decides which generated examples to keep, how they are ordered, and how their effect is measured against something real. A strong generator paired with a weak filter produces a large, polluted dataset. A modest generator paired with a rigorous verifier — a unit test, a proof checker, a well-calibrated grader, a held-out set of genuine examples — can produce data good enough to train models that outperform their teachers on narrow tasks. This is not a novel observation in the research literature, but it is routinely neglected in practice, partly because generation is the visible and exciting step and verification is tedious. The chapters that follow try to correct that imbalance by treating synthetic data as an engineering discipline with three equally important stages: generating candidate data, filtering it, and validating that the filtered result actually does what it was meant to do. What the book covers The first two chapters establish why synthetic data matters and where it came from. Chapter 1 sets out the constraints that drive the practice — the exhaustion of public text, the legal and ethical limits on sensitive data, and the statistical problem of rare events — and is careful to separate the constraints that synthetic data genuinely relaxes from those it merely appears to. Chapter 2 traces the lineage from statistical disclosure control and classical data augmentation through simulation and generative adversarial networks to the language-model-generated corpora that dominate current practice, and proposes a working taxonomy of the different things people mean by "synthetic data." Chapter 3 deals directly with privacy, because that is where the most consequential misunderstandings occur. Synthetic data is often described as anonymous by construction. It is not, and the chapter explains how generators memorise and leak, what differential privacy does and does not guarantee, and how to reason about the trade-off between fidelity and protection. Chapters 4 and 5 turn to generation itself. Chapter 4 examines the machinery of producing varied, useful candidates: seed design, personas, instruction evolution, and the persistent problem of diversity collapse, in which a generator keeps producing the same few kinds of example no matter how it is prompted. Chapter 5 addresses curriculum — the deliberate ordering and grading of synthetic examples by difficulty — and argues that the most powerful curricula are those in which the model's own performance determines what gets generated next. Chapters 6 and 7 form the core of the argument. Chapter 6 is about automated grading: the spectrum of verifiers from exact execution checks through learned reward models to language models acting as judges, and the systematic biases each one introduces. Chapter 7 applies that machinery to the specific problem of filtering harmful and hallucinated content — the factual errors, toxic outputs, and subtle contamination that synthetic pipelines generate at scale. Chapters 8 and 9 concern the most commercially important use of synthetic data: transferring capability from a large, expensive teacher model to a small, cheap student. Chapter 8 explains distillation as a technique and its known limits, including the finding that students often learn to imitate a teacher's style long before they acquire its competence. Chapter 9 works through the construction of a small domain model in practical detail, from seed collection to deployment. Chapter 10 confronts the long-run risk: what happens when synthetic data trains models whose output becomes the next round of synthetic data, and when the open web itself fills with machine-generated text. Chapter 11 closes the technical arc with validation and governance — how to establish that a synthetic dataset works, how to document its provenance, and how to make the whole pipeline auditable. Who this is for The book is written for engineers, data scientists, technical managers, and policy readers who need to understand how synthetic data systems are actually built and where they fail. It assumes familiarity with the idea of training a machine learning model on examples, but not specialist knowledge of any particular architecture. Terms are defined as they arise, and a glossary at the end collects the ones that recur. The examples lean toward language models because that is where synthetic data practice has moved fastest and where the stakes are currently highest. But the principles — that a generator is only as trustworthy as its verifier, that diversity must be engineered rather than hoped for, that real data remains the anchor against which everything is measured — apply equally to tabular data, images, robotics simulation, and scientific modelling. Where those domains offer clearer illustrations, the book uses them. One caution before beginning. Synthetic data is a field in which confident claims age quickly. Techniques that seemed decisive in one year have been qualified or overturned in the next, and the empirical evidence on many questions — how much synthetic data can safely replace real data, how much a student can learn from a teacher, how quickly recursive training degrades — remains thinner than practitioners would like. The book tries to be explicit about where the evidence is solid and where it is still forming. The underlying engineering logic, however, has held steady throughout: generate freely, discard ruthlessly, and measure against reality. Chapter 1. The Data Wall and What Synthetic Data Can Actually Move Every machine learning system is a compression of its training data. The model learns whatever regularities the examples contain and nothing they do not. For most of the history of the field, the binding constraint on what a model could learn was the model itself: its size, its architecture, the compute available to train it. Data was assumed to be abundant, or at least obtainable with enough effort and money. That assumption has broken down in three different ways, and synthetic data is a response to all three. Understanding them separately matters, because synthetic data addresses each one with very different effectiveness. The exhaustion of public text The first constraint is quantity. Large language models improved dramatically between 2018 and 2024 largely by being trained on more text. Scaling studies, most influentially the 2022 "Chinchilla" analysis from DeepMind, showed that for a fixed compute budget, the best results came from balancing model size against the number of training tokens, and that many earlier models had been undertrained relative to their size. The practical consequence was a hunger for tokens that outran the growth of the web. Researchers at Epoch AI have tried to quantify how close the field is to running out. Their analysis, first released in 2022 and revised for presentation at the International Conference on Machine Learning in 2024, estimated the effective stock of quality-adjusted, publicly available human-generated text at roughly three hundred trillion tokens. Projecting the growth of training-set sizes forward, they concluded that if current trends continue, frontier models would be trained on datasets roughly equal to that entire stock at some point between 2026 and 2032, with the exact date depending on how aggressively models are over-trained relative to compute-optimal recipes. The estimate carries wide uncertainty — it depends on how much text sits on the deep web, how much low-quality text can be rescued by filtering, and how many times the same data can be reused before returns diminish — but the direction is not in serious dispute. Several qualifications are worth keeping in mind. First, exhaustion of public text is not the same as exhaustion of information. There is a great deal of private text in emails, corporate documents, and messaging platforms, and a great deal of non-textual data in video, audio, and sensor streams. Whether and how that material can be used is a legal and ethical question rather than a technical one. Second, data can be repeated. Studies of multi-epoch training have found that a model can be trained on the same data several times with only modest loss of value per pass, though returns fall off as repetition increases. Third, and most important for this book, the constraint applies with greatest force to frontier pretraining. A company building a small model for customer support in insurance is not limited by the size of the web. It is limited by the number of good examples of insurance customer support it has, which is a different problem entirely. Synthetic data as a remedy for the exhaustion of public text is therefore a contested proposition. It is unclear whether a model can generate text that contains genuinely new information beyond what its own training data held. A language model sampling from its learned distribution produces samples from that distribution; it does not, by itself, add knowledge about the world. What synthetic data can do in this setting is restructure existing knowledge into more learnable forms — rewriting dense web pages as clear explanations, converting facts into question-and-answer pairs, generating worked solutions to problems whose answers can be checked. The gains from this kind of restructuring are real and have been demonstrated repeatedly. But they come from the restructuring and the checking, not from the volume. Where the synthetic data adds information that was not already present, it does so because something outside the generator — a code interpreter, a theorem prover, a physics simulator, a human — introduced it. Privacy and the locked rooms of data The second constraint is access. The data that would be most valuable for many applications is precisely the data that cannot be freely shared. Electronic health records contain the richest available description of disease progression and treatment response. Bank transaction logs encode the patterns of fraud. Customer service transcripts show exactly how real people describe their problems. Each of these is subject to legal restriction, and the restrictions are tightening rather than loosening. In the European Union, the General Data Protection Regulation governs any processing of personal data, defined broadly as information relating to an identified or identifiable natural person. Recital 26 of the regulation states that its principles do not apply to anonymous information — data that does not relate to an identifiable person, or that has been rendered anonymous in such a way that the individual is no longer identifiable. That carve-out is the legal hook on which much of the commercial interest in synthetic data hangs: if synthetic records are genuinely anonymous, they fall outside the regulation and can be shared, sold, and used for training with far fewer obligations. In the United States, the Health Insurance Portability and Accountability Act sets out two routes to de-identification for health data — a "safe harbor" list of eighteen identifier types to remove, and an "expert determination" route in which a qualified statistician certifies that the risk of re-identification is very small. Synthetic health data is typically pursued through the second route. The EU's Artificial Intelligence Act, which entered into force in 2024, adds a further layer. Its provisions on data governance for high-risk systems permit the processing of special categories of personal data for the purpose of detecting and correcting bias, but only where that purpose cannot be effectively fulfilled by processing other data, explicitly including synthetic or anonymised data. In effect, the law nudges developers toward synthetic alternatives before it permits use of sensitive real data. The appeal is obvious. If a hospital could train a generator on its patient records and release synthetic patients that preserved the statistical structure of the originals without corresponding to any real person, researchers everywhere could study them freely. The difficulty, examined at length in Chapter 3, is that "without corresponding to any real person" is much harder to guarantee than it sounds. Generators memorise. Rare records are the easiest to memorise and the easiest to re-identify. And the statistical structure that makes synthetic data useful is the same structure an attacker exploits to make inferences about individuals. Synthetic data relaxes the privacy constraint only when it is built with formal protections, and those protections always cost fidelity. The tyranny of the long tail The third constraint is distribution. Even where data is plentiful and accessible, the examples that matter most are often the rarest. A self-driving system sees millions of hours of ordinary highway driving and very few instances of a child chasing a ball into the road. A medical imaging model sees thousands of normal chest radiographs for every one showing a rare pathology. A fraud detector sees overwhelmingly legitimate transactions. A customer service model sees the same twenty questions asked in slightly different words, and almost never the strange edge case that will produce a viral complaint. This is the problem that synthetic data addresses most convincingly, and it has a long history. Simulation has been used to train and test autonomous vehicles for years; Waymo, among others, has reported running billions of miles in simulation, far more than its physical fleet has driven, precisely so that it can rehearse rare and dangerous scenarios on demand. In robotics, OpenAI's work on training a robot hand to manipulate a Rubik's cube relied on randomising simulated physics — friction, object size, lighting, motor response — so extensively that the real world appeared to the policy as just another variation. In medical imaging, researchers have used generative models to produce additional examples of under-represented conditions to rebalance training sets. The logic of synthetic data for the long tail is different from the logic for quantity or privacy. Here the aim is not to replace real data but to reshape its distribution: to make rare cases common enough that a model learns to handle them. That shift changes what must be verified. A synthetic rare event is valuable only if it is a faithful instance of the rare event, and the generator's ability to produce faithful rare events is usually weakest exactly where the real data is thinnest. A generative model trained on a hundred examples of a rare tumour will produce variations on those hundred, not genuinely novel presentations. A simulator built by engineers who did not anticipate a particular failure mode will not produce it. The long tail is where synthetic data is most needed and where its fidelity is hardest to establish. Scarcity at the level of the task The three constraints so far are framed at the level of a whole field. Most practitioners meet them in a narrower form: the data they need for one task does not exist in useful quantity. Two cases illustrate how common this is. The first is language. The bulk of the text available for training is in English and a small number of other widely written languages. Thousands of languages have very little digitised text, and even languages with tens of millions of speakers may have only a sliver of the web's content, much of it machine-translated or of low quality. A team building a customer service model for speakers of such a language cannot simply collect more examples. Synthetic data generated by translating, adapting, or writing directly in the target language is one of the few options, and it inherits every weakness of the generator's competence in that language. Errors of idiom, register, and cultural reference that a native speaker would catch at once can pass unnoticed through automated filters built mainly for English. The second is specialised labelling. Many valuable tasks require annotation that only experts can provide: identifying the relevant clauses in a commercial lease, grading the severity of a finding on a pathology slide, judging whether a piece of code contains a security vulnerability. Expert time is expensive and limited, and the number of labelled examples available for such tasks is often in the hundreds rather than the hundreds of thousands. Here synthetic data is attractive because a capable model can produce candidate labels or candidate examples cheaply, but its reliability is weakest exactly where expertise is most needed. A general-purpose model that produces plausible-looking lease annotations will also produce plausible-looking wrong ones, and only an expert can tell them apart. Both cases show the same pattern. Synthetic data is most tempting where real data is scarcest, and in those places the generator is usually least reliable, because its own training data was scarce there too. Whatever is gained by generation must be paid for with stronger verification. What actually moves Taken together, these three constraints define what synthetic data can and cannot do. It is useful to be precise. Synthetic data can restructure information that already exists, presenting it in forms that are easier to learn. This is the mechanism behind the success of "textbook" style corpora, question-and-answer reformulations, and step-by-step worked solutions. The information content of the training set does not increase, but its learnability does, and for a finite model trained for a finite time, learnability is often what matters. Synthetic data can inject information from an external source of truth. When a generated program is run and its output checked, or a generated proof is verified by a formal checker, or a simulated physical scenario is computed by a physics engine, the verified example carries information that the generator alone did not possess. The checker is doing the epistemic work. This is the most powerful use of synthetic data and the one on which its most impressive results rest. Synthetic data can transfer capability from one model to another. A large, capable model generates examples; a smaller model learns from them. The smaller model ends up knowing things it would not otherwise have learned from its own limited training. No new information enters the world, but it moves to where it is cheaper to use. Synthetic data can reshape distributions, oversampling rare conditions and undersampling common ones, as long as the generator can produce credible instances of the rare conditions. And synthetic data can, under the right formal mechanisms, release statistical structure while limiting what is revealed about individuals. What synthetic data cannot do is create knowledge from nothing. A generator with no access to an external check cannot reliably produce correct facts that it did not already know, and if it produces them anyway, they will be indistinguishable from the incorrect facts it produces alongside them. Every synthetic data system therefore needs some source of ground truth outside the generator: a verifier, a simulator, a human judgement, or a body of real data against which the synthetic data is evaluated. The engineering of that ground truth is the main subject of this book. The cost structure has inverted One further change explains why synthetic data has become central rather than peripheral. For most of the history of machine learning, labelled data was expensive and computation was relatively cheap. Hiring annotators to label images or transcribe speech or rate responses cost real money per example, and scaling up meant scaling the workforce. The rise of capable generative models has inverted this. Generating a candidate example now costs a fraction of a cent. Labelling it by hand still costs what it always did. This inversion shifts the economics of every stage of the pipeline. Where once the question was how to make the most of a small, expensive labelled set, the question is now how to extract a small, trustworthy subset from a large, cheap, unreliable one. That is a filtering problem rather than a collection problem, and filtering problems reward a different set of skills: designing tests, calibrating graders, detecting subtle failure modes, and measuring the downstream effect of each filtering decision. Teams that approach synthetic data with the habits of the old regime — generate a lot, train on all of it, hope for the best — reliably get worse results than teams that treat the generator as a source of candidates and spend most of their effort on deciding which candidates deserve to survive. The same inversion changes where human expertise belongs. In the old regime, experts were needed at the point of labelling. In the new one, they are needed at the point of designing the verifier: writing the test cases, specifying the rubric, choosing the held-out evaluation set, and reading samples of the filtered output to catch failure modes no automated check anticipated. An hour of a domain expert's time spent designing a good verifier can be worth more than a hundred hours spent labelling, because the verifier applies that expertise to millions of examples. This is the most practical lesson in the book, and every chapter that follows returns to it in some form. Chapter 2. A Lineage of Artificial Data The phrase "synthetic data" is used to describe things that have little in common beyond being manufactured rather than collected. A rotated photograph added to an image classifier's training set is synthetic data. So is a table of fictitious census respondents generated to protect privacy, a simulated car crash rendered by a game engine, a fabricated fraud transaction produced by a generative adversarial network, and a million maths problems with worked solutions written by a large language model. Each of these has its own history, its own failure modes, and its own standard of validity. Treating them as a single category obscures more than it reveals. This chapter traces the main lineages and then offers a working taxonomy that the rest of the book relies on. Disclosure control: synthetic data as a privacy technology The oldest formal use of synthetic data comes from official statistics. National statistical agencies collect detailed information about individuals and businesses under legal promises of confidentiality, and they have long faced the problem of releasing useful data without breaking those promises. Traditional techniques — suppressing small cells in tables, coarsening geographic detail, swapping values between similar records, adding noise — all degrade the data in ways that are hard for downstream analysts to account for. In 1993 the statistician Donald Rubin proposed a different approach. Drawing on his earlier work on multiple imputation for missing data, he suggested that an agency could treat the confidential values of every respondent as if they were missing, fit a statistical model to the real data, and release multiple datasets in which every confidential value had been replaced by a draw from that model. Analysts could then apply standard methods to each released dataset and combine the results using rules that properly accounted for the added uncertainty. In the same year Roderick Little proposed a partial version, in which only the most sensitive variables or records were replaced. These proposals launched a substantial literature on what came to be called fully and partially synthetic data. The ideas reached production in the following decades. The United States Census Bureau released synthetic versions of linked survey and administrative data, and its OnTheMap application, which shows where people live and work, became in 2008 one of the first production systems to use a formal differential privacy guarantee for synthetic data. The 2020 United States Census applied differential privacy to its published tables, a decision that generated heated debate among demographers about the accuracy cost. The statistical-agency tradition contributes two lasting lessons to modern practice. First, synthetic data must be judged by whether analyses performed on it give the same answers as analyses on the real data — a property usually called utility or fidelity. Second, privacy protection is a quantity to be measured and traded against utility, not a binary property that synthetic data has by default. Augmentation: synthetic data as regularisation A second lineage comes from computer vision and speech. Practitioners training image classifiers noticed early that randomly flipping, cropping, rotating, and colour-shifting their training images improved generalisation. The transformed images were still recognisably the same object, so their labels were still correct, but the model was forced to learn features robust to those transformations. The landmark 2012 AlexNet paper used random crops and horizontal reflections alongside colour perturbation, and data augmentation has been standard practice ever since. Augmentation is the most conservative form of synthetic data. Each synthetic example is a deterministic or lightly randomised transformation of a real one, and the transformation is chosen precisely because it preserves the label. The validity check is built in: a flipped cat is still a cat. Later techniques pushed further. Mixup, introduced in 2017, trains on convex combinations of pairs of images and their labels — a blend of seventy percent dog and thirty percent cat with a correspondingly mixed label. In text, back-translation, in which a sentence is translated into another language and back again to produce a paraphrase, became a standard way to augment machine translation and classification data. For tabular data with imbalanced classes, the Synthetic Minority Over-sampling Technique, known as SMOTE and published by Chawla and colleagues in 2002, generates new minority-class examples by interpolating between existing minority examples and their nearest neighbours. SMOTE is still widely used in credit scoring and medical prediction, and it illustrates both the promise and the danger of augmentation. Interpolating between two real fraud cases produces something that looks like fraud, but it may also produce combinations of features that no real fraud case would ever have, and in high dimensions the interpolated points can fall in regions of feature space that belong to the majority class. The augmentation lineage contributes the idea that synthetic data can encode an inductive bias — a prior belief about what transformations should not matter — and that a well-chosen transformation teaches the model something true about the world. Simulation: synthetic data from first principles A third lineage comes from robotics, autonomous vehicles, and scientific computing. Here the synthetic data is produced not by a statistical model of real data but by a model of the physical process itself. A physics engine computes how a robot arm moves when its motors apply certain torques. A rendering engine computes how a scene looks from a particular camera under particular lighting. A molecular dynamics simulation computes how a protein folds. The appeal of simulation is that it can produce unlimited, perfectly labelled data. Every pixel in a rendered image comes with exact depth, exact object identity, exact surface normal. Every simulated crash comes with exact forces. The difficulty is the "reality gap": a model trained in simulation learns the simulator's quirks, and the simulator is never a perfect replica of the world. Textures are too clean, physics is too deterministic, sensor noise is unrealistic. The most influential response to the reality gap was domain randomisation, developed by Josh Tobin and colleagues at OpenAI and Berkeley in 2017. Rather than trying to make the simulator more realistic, they made it less consistent: randomising textures, colours, lighting, camera positions, and object shapes so widely that a model trained across all of these variations treated the real world as one more variation. Their object-detection models, trained only on crudely rendered images with random textures, transferred to real camera images well enough to guide a robot grasping objects. OpenAI's later Rubik's cube work extended the idea with "automatic domain randomisation," which progressively widened the range of randomisation as the policy improved — an early instance of the adaptive curricula discussed in Chapter 5. Simulation contributes the most important idea in this book: synthetic data is most trustworthy when it is produced by a process whose correctness can be established independently of the data. A physics engine encodes known laws. A theorem prover encodes logical rules. A code interpreter encodes the semantics of a programming language. When these are the source of truth, the synthetic data inherits their reliability. Deep generative models: learning to synthesise A fourth lineage arose with deep generative models, beginning with variational autoencoders in 2013 and generative adversarial networks, introduced by Ian Goodfellow and colleagues in 2014. These models learn a distribution from data and can sample new instances from it. For the first time, it was possible to produce realistic synthetic images, and later audio and tabular records, without hand-specifying a simulator. Generative adversarial networks were quickly applied to tabular data for privacy and augmentation. The CTGAN model, published by Lei Xu and colleagues in 2019, adapted the adversarial framework to the mixed continuous and categorical columns of typical relational tables, using mode-specific normalisation for continuous variables with several peaks and a conditional generator to handle imbalanced categories. A commercial industry formed around such tools, selling synthetic versions of customer and patient databases. Diffusion models, which became dominant for image generation from around 2021, extended the reach of learned generators. In medical imaging, researchers have used them to produce synthetic scans of rare conditions. In autonomous driving, they are used alongside or in place of traditional rendering to generate realistic sensor data. The lesson from this lineage is double-edged. Learned generators can capture complex distributions no human could specify by hand, but they also capture whatever artefacts, biases, and individual records exist in their training data, and they offer no independent guarantee of correctness. A diffusion model that produces a realistic chest radiograph has not checked that the radiograph is anatomically coherent. Language models as generators The fifth and most recent lineage is the use of large language models to write training data for other language models, and for themselves. Several papers in 2022 and 2023 mark the transition. Self-Instruct, from Yizhong Wang and colleagues in 2022, showed that a pretrained language model could bootstrap its own instruction-following ability. Starting from 175 human-written seed tasks, the method prompted the model to generate new instructions, then inputs and outputs for those instructions, filtered the results for quality and novelty, and used the survivors to fine-tune the same model. The result was around 52,000 instructions. Unnatural Instructions, published by Or Honovich and colleagues around the same time, did something similar. Stanford's Alpaca, described in the Introduction, applied the Self-Instruct recipe with a stronger commercial model as generator. In 2023 Ronen Eldan and Yuanzhi Li at Microsoft published TinyStories, a dataset of short children's stories written by GPT-3.5 and GPT-4 using only words a young child would understand. They showed that models with fewer than ten million parameters, trained on this corpus, could produce fluent, grammatical, and coherent stories — something much larger models trained on raw web text struggled to do. The same year, Suriya Gunasekar and colleagues published "Textbooks Are All You Need," describing phi-1, a 1.3-billion-parameter code model trained on a mixture of filtered web data and synthetic textbook-style material generated by GPT-3.5. It reached around fifty percent on the HumanEval coding benchmark, a level then associated with models many times larger. By 2024 synthetic data had become a standard component of frontier training. Model technical reports routinely described synthetic pipelines for mathematics, code, reasoning, and alignment data. Hugging Face released Cosmopedia, an open synthetic corpus of textbooks, blog posts, and stories totalling around twenty-five billion tokens. Reasoning models released in 2024 and 2025 relied heavily on synthetic chains of thought, generated by sampling many candidate solutions and keeping those that reached verified correct answers. A working taxonomy These lineages suggest a taxonomy organised not by the type of data — image, text, table — but by the source of truth that stands behind each synthetic example. That source determines what can go wrong and what kind of verification is needed. Table 1 summarises five broad families. Table 1. Families of synthetic data, classified by source of truth. Family Source of truth Typical examples Characteristic failure Natural verifier Transformation The original real example plus an invariance assumption Image flips and crops, back-translation, SMOTE Transformation breaks the label or creates impossible combinations Checks that the label is preserved Simulation An explicit model of a physical or formal process Rendered driving scenes, physics rollouts, proof search Reality gap between simulator and world Comparison against real-world measurements Statistical synthesis A model fitted to private real records Synthetic census microdata, synthetic patient tables Leakage of individual records; loss of rare structure Utility and privacy metrics against the real data Learned generation A neural generator trained on large corpora Diffusion images, model-written instructions and stories Hallucination, bias, homogenisation Automated graders, filters, human review Verified generation A learned generator coupled to an external checker Executed code, checked maths answers, formally verified proofs Verifier too narrow or exploitable The checker itself, plus audit of the checker The categories overlap in practice. A modern pipeline for training a coding model might use learned generation to propose problems, verified generation to keep only solutions that pass tests, and transformation to paraphrase problem statements. But the taxonomy makes one thing clear. As one moves down the table from transformation to learned generation, the generator becomes more powerful and more general, and the built-in guarantee of correctness disappears. The final row, verified generation, is the attempt to recover the guarantee by attaching an external check to a powerful generator. Most of the significant recent advances in synthetic data fall into that final row, and most of the engineering effort described in the rest of this book is concerned with building, calibrating, and auditing the checkers that make it possible. What the lineages agree on Despite their differences, the five traditions converge on a small set of principles. Synthetic data must be evaluated by its effect on downstream tasks performed on real data, not by how realistic it looks. Its value depends on what independent knowledge was injected during its creation, whether that knowledge is a physical law, a privacy budget, a label-preserving invariance, or a verification test. And it carries the properties of its generator — including the generator's errors — unless something intervenes to remove them. The statisticians knew this in 1993. The roboticists rediscovered it in 2017. The language model community has been learning it, at considerable expense, since 2023. Chapter 3. Privacy Without Illusions The most common commercial pitch for synthetic data runs roughly as follows. Your real data contains personal information and cannot be shared. Train a generator on it, sample new records from the generator, and those records will have the same statistical properties as the originals but will not correspond to any real person. Because they are not personal data, you can use them freely. The pitch is attractive because it seems to dissolve a hard trade-off. It is also, in its simple form, wrong. Synthetic data can be a powerful privacy technology, but only when it is engineered with formal protections whose cost is accepted openly. Without them, synthetic data can leak as much as the original, and sometimes it leaks in ways that are harder to detect. Why "no real person" is not enough The intuition behind the pitch is that a synthetic record is a new draw from a learned distribution, not a copy of a real one. If the generator has learned the distribution well, its samples will look like typical members of the population rather than like any particular individual. But generators do not learn distributions in the abstract. They learn from finite samples, and a sufficiently flexible generator can learn the sample itself. The phenomenon is memorisation, and it has been documented across every class of generative model. In 2021 Nicholas Carlini and colleagues showed that they could extract hundreds of verbatim training sequences from GPT-2, including names, phone numbers, email addresses, and code, simply by sampling from the model and ranking the outputs by how confident the model was in them. In 2023 a related group showed that image diffusion models, including Stable Diffusion, could be induced to regenerate near-identical copies of some of their training images. Memorisation is concentrated in examples that are duplicated in the training data and, in tabular settings, in records that are statistical outliers — exactly the records that are easiest to re-identify. Tabular synthetic data has its own version of the problem. Consider a hospital dataset in which one patient is a 97-year-old man with an unusual combination of diagnoses living in a small rural district. A generator that faithfully reproduces the joint distribution of age, sex, diagnosis, and location will produce, with some probability, a synthetic record that matches this person on every attribute. Nobody need have copied his record for the synthetic version to reveal that someone with those characteristics exists in the source data, and that may be enough to reveal his diagnosis to anyone who knows the other attributes. The vulnerability is structural. Synthetic data is useful precisely because it preserves the relationships between variables in the real data. An attacker exploits those same relationships to make inferences. The better the synthetic data captures rare correlations, the more it reveals about the rare individuals who produced them. How privacy attacks work Privacy researchers formalise the risks through several kinds of attack, each corresponding to a different harm. A membership inference attack tries to determine whether a particular individual's record was used to train the generator. Even if the synthetic data reveals nothing new about that person, learning that they were in a dataset of patients with a particular disease, or of customers of a debt collection agency, can itself be harmful. Membership inference typically works by comparing how closely the synthetic data, or the generator, fits a target record relative to how it fits similar records known to be outside the training set. An attribute inference attack assumes the attacker already knows some of a target's attributes — age, postcode, sex — and tries to infer a sensitive one, such as diagnosis or income. The attacker trains a predictor on the synthetic data and applies it to the known attributes. If the synthetic data has preserved the real correlations closely, the inference will be accurate. Note that this attack may succeed even against a perfectly non-memorising generator, because it exploits population-level patterns; whether that constitutes a privacy breach or merely statistical knowledge is a genuinely contested question in the field. A linkage or singling-out attack tries to find a synthetic record that matches a real individual closely enough to be considered a reconstruction. This is closest to the colloquial idea of re-identification, and it is the attack most directly addressed by legal definitions of anonymity. The European Data Protection Board's predecessor, the Article 29 Working Party, identified singling out, linkability, and inference as the three risks an anonymisation technique must address. In 2022 Theresa Stadler, Bristena Oprisanu, and Carmela Troncoso published an evaluation pointedly titled "Synthetic Data — Anonymisation Groundhog Day." Testing several popular generators against these attacks, they found that synthetic data without formal privacy guarantees did not provide a better trade-off between privacy and utility than traditional anonymisation methods, that protection varied unpredictably across records, and that outliers in particular remained highly vulnerable. Generators that produced the most useful synthetic data tended also to produce the most vulnerable. The paper's conclusion was not that synthetic data is useless for privacy, but that it inherits the same fundamental trade-off as every other disclosure-control method and cannot be assumed to escape it. Differential privacy as the formal answer The only widely accepted formal framework for privacy protection in synthetic data is differential privacy, introduced by Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith in 2006. The core idea is simple to state. A randomised algorithm is differentially private if its output distribution changes very little when any single individual's record is added to or removed from its input. If that holds, then nothing an observer learns from the output can depend very much on any one person's data, because the output would have been nearly as likely without them. The strength of the guarantee is controlled by a parameter, conventionally written epsilon. Smaller epsilon means stronger privacy. An epsilon close to zero means the output is almost independent of any individual; an epsilon of ten or more offers a much weaker guarantee in theory, though it may still defeat practical attacks. There is no universally agreed threshold for acceptable epsilon. The United States Census Bureau's 2020 redistricting data used a total privacy-loss budget that was the subject of extended public debate. Industry deployments have used values ranging from below one to well above ten. Differential privacy has three properties that make it especially suitable for synthetic data. It is robust to post-processing: anything computed from a differentially private output is also differentially private with the same epsilon, so once a generator has been trained privately, any amount of sampling from it costs no further privacy. It composes: the privacy losses from several private computations add up in a predictable way, allowing a total budget to be managed. And it makes no assumptions about what the attacker already knows, which is precisely what heuristic anonymisation methods cannot claim. There are two main routes to differentially private synthetic data. The first is to train a neural generator with differentially private stochastic gradient descent, introduced by Martín Abadi and colleagues in 2016. This method clips each individual example's contribution to the gradient, so no single record can move the model too far, and adds calibrated noise to the aggregated gradient at each step. DP-SGD can be applied to any neural generator, including language models, but it degrades model quality, and the degradation is severe for small datasets and large models. The second route, which has proved more effective for tabular data, is to measure a set of low-dimensional statistics of the real data — one-way and two-way or three-way marginal distributions, for example — with differentially private noise, and then to generate synthetic records consistent with those noisy statistics. Methods of this kind, built on graphical models that stitch noisy marginals into a coherent joint distribution, performed strongly in the differential privacy synthetic data competitions run by the United States National Institute of Standards and Technology from 2018 onward. Their weakness is that they preserve only the relationships captured by the chosen marginals. Higher-order interactions not explicitly measured are lost. For text, the frontier is the use of differentially private fine-tuning of language models to produce synthetic corpora, and the alternative of "private evolution" approaches in which a public model generates candidates and a private dataset is consulted only through noisy votes about which candidates resemble it. These remain active research areas, and the utility achievable at strong privacy levels is still limited for complex text. Utility, fidelity, and the shape of the trade-off Every privacy mechanism costs utility, and the costs are not evenly distributed. Differential privacy works by making the output insensitive to any individual, which means it necessarily blurs patterns that depend on few individuals. Rare diseases, small demographic groups, and unusual combinations of attributes are exactly what get lost. This creates an uncomfortable tension with fairness goals: a privately generated synthetic dataset may represent minority groups less accurately than the real data did, and models trained on it may perform worse for those groups. Evaluating the trade-off requires measuring both sides. On the utility side, practitioners typically report several kinds of measure. Marginal fidelity compares the distribution of each variable, and of pairs of variables, between real and synthetic data. Machine learning efficacy, sometimes called "train on synthetic, test on real," trains a predictive model on the synthetic data and evaluates it on held-out real data, comparing its performance to a model trained on real data. Query-specific utility checks whether particular analyses of interest — a regression coefficient, a subgroup prevalence, a survival curve — give the same answers on both datasets. The last of these is the most demanding and most informative, because synthetic data that looks good on average can still give misleading answers to the specific questions an analyst cares about. On the privacy side, the measures depend on whether a formal guarantee exists. With differential privacy, the epsilon value itself is the headline measure, though it should be reported alongside the related parameter delta and the details of how the budget was allocated. Without a formal guarantee, practitioners fall back on empirical measures: the distance from each synthetic record to its closest real record, the success rate of simulated membership and attribute inference attacks, and the proportion of synthetic records that exactly or nearly match a real one. These empirical measures are useful for detecting gross failures but cannot prove the absence of risk, because they test only the attacks the evaluator thought to run. Table 2 sets out the main approaches and what each one does and does not protect against. Table 2. Privacy approaches for synthetic data and what they guarantee. Approach Formal guarantee Main protection Main weakness Typical utility cost Unprotected generator None Avoids exact copying in most cases Memorises outliers and duplicates Low Heuristic post-filtering (remove close matches) None Removes near-copies of real records Does not stop attribute or membership inference Low to moderate DP-SGD trained generator Differential privacy Bounds influence of any one record Large quality loss on small or complex data Moderate to high Noisy-marginal synthesis Differential privacy Bounds influence; strong for tabular data Loses unmeasured higher-order relationships Moderate Synthetic data plus restricted access Contractual and procedural Limits who can attack Depends on enforcement Low Practical guidance Several practical conclusions follow from the evidence. First, decide what the synthetic data is for before choosing how to protect it. Synthetic data intended for software testing, where it needs only to have the right schema and plausible value ranges, can be generated with very strong privacy at little cost because high fidelity is not required. Synthetic data intended for training a clinical prediction model, where subtle correlations matter, faces a much harsher trade-off. Synthetic data intended for public release faces the strongest attack model and needs the strongest guarantee. Second, do not rely on the label "synthetic" as a legal shield. Data protection authorities assess anonymity by asking whether individuals can reasonably be identified, taking into account all the means likely to be used. A synthetic dataset that permits re-identification of outliers is not anonymous in that sense, whatever process produced it. A formal differential privacy guarantee with a defensible epsilon, combined with documented empirical attack testing, is a far stronger position than an assertion that the records are new. Third, deduplicate the training data before training a generator. Memorisation is strongly concentrated in duplicated examples, and removing duplicates is cheap insurance. Fourth, treat outliers as a design decision. They are both the most vulnerable records and often the most informative. Options include suppressing them, coarsening their attributes before training, or accepting that a privacy-protected synthetic dataset will not represent them well and planning downstream analyses accordingly. Fifth, evaluate utility on the analyses that will actually be performed. A synthetic dataset that reproduces every marginal distribution beautifully but gets the key subgroup effect wrong is worse than useless, because it will be trusted. Finally, recognise that synthetic data is one tool among several. Federated learning, in which models are trained where the data lives and only updates are shared; secure enclaves and trusted research environments, in which approved analysts work on real data under controlled conditions; and query-based systems that answer specific questions with differential privacy all address the same underlying problem. Synthetic data is most valuable when broad, exploratory access is needed and some loss of fidelity is acceptable. When precise answers on real data are required, it is usually better to bring the analyst to the data than to bring a synthetic copy of the data to the analyst. The privacy case thus anticipates the book's general argument. A generator alone gives no guarantee. The guarantee comes from a mechanism attached to the generator — here, differential privacy and empirical attack testing — and from validation against the real data the synthetic records are meant to stand in for. Chapter 4. Engineering Diversity at the Point of Generation Ask a large language model to write a hundred customer complaints about a delayed parcel and you will get a hundred complaints. Read them and you will notice that they are remarkably similar. Most will open with a variant of "I am writing to express my frustration." Most will mention a tracking number. Most will be polite, grammatical, moderately annoyed, and end by requesting a refund or an update. Very few will be written in broken English by someone who is furious, or consist of a single line with no punctuation, or bury the actual problem in the third paragraph of an unrelated story, or be from a small business whose entire stock is in the missing parcel. Real complaints look like all of those things. The model's complaints look like the average of them. This is the central problem of generation, and it is more important than the quality of any individual output. A synthetic dataset can consist entirely of correct, well-formed examples and still be a poor training set because it covers only a narrow band of the space the model will encounter in deployment. Diversity does not arise naturally from sampling a language model. It has to be engineered. Why generators converge Language models trained with instruction tuning and reinforcement learning from human feedback become, by design, more predictable than their base versions. The alignment process rewards outputs that human raters prefer, and raters prefer clear, polite, well-structured responses. The result is a model whose output distribution has been sharpened toward a particular house style. Researchers have documented that aligned models produce less diverse outputs than their base models across a range of measures, a phenomenon sometimes called mode collapse in this context. For a chat assistant, that predictability is a feature. For a data generator, it is a liability. Sampling temperature is the first lever practitioners reach for. Raising the temperature flattens the model's probability distribution over next tokens, making less likely tokens more likely to be chosen. It increases surface variety. But it does so indiscriminately: high temperatures introduce errors, incoherence, and off-topic drift at the same time as they introduce novelty, and they do not reliably change the high-level structure of outputs. A complaint generated at high temperature is still usually the same kind of complaint, with odder word choices. Temperature varies the phrasing; it rarely varies the scenario. The deeper problem is that a single prompt specifies a single region of the output space. If every generation call receives the same instruction — "write a customer complaint about a delayed parcel" — then every output is a sample from the model's conditional distribution given that instruction, and that distribution is concentrated. Diversity must therefore come from varying the conditioning: the prompt, the context, the examples, the constraints. Seeds, taxonomies, and attribute grids The most reliable way to induce diversity is to decompose the target space explicitly and then generate within each part of it. Before generating customer complaints, a careful practitioner first asks what dimensions real complaints vary along. Product category, severity of the problem, customer's emotional state, customer's writing proficiency, length, channel (email, chat, social media), whether the complaint contains extraneous information, whether the customer is mistaken about the facts, whether they are a repeat complainant. Each of these dimensions has a set of values. The generator is then asked to produce complaints for specific combinations: a furious, terse, social-media complaint from a non-native speaker about damaged electronics, in which the customer has misread the delivery date. This attribute-grid approach has several advantages. It guarantees coverage of combinations the model would rarely produce unprompted. It makes the composition of the dataset explicit and controllable, so that the proportion of, say, mistaken customers can be set deliberately. And it produces metadata for every example, which is invaluable later for analysing which kinds of example help or hurt the trained model. The cost is that someone must design the grid, and the grid encodes the designer's assumptions about what matters. A grid that omits a dimension — customers writing on behalf of elderly relatives, say — will produce a dataset that omits it too. Grids are best derived from real data where it exists: a sample of a few hundred real complaints, read carefully and annotated by hand, will reveal dimensions that nobody would have thought to include. When real data is unavailable, domain experts and existing taxonomies are the next best source. A related technique uses a hierarchy of topics. The Cosmopedia project at Hugging Face, which generated around twenty-five billion tokens of synthetic textbooks, stories, and educational posts, seeded generation using topics and passages drawn from curated sources and web pages, and specified different target audiences — young children, high school students, college students, professionals — and formats. Varying the audience alone, the team reported, substantially changed the content and style of otherwise similar prompts. Grounding the generator in real web text also gave each output a different starting point, reducing the tendency to regress to the same few framings. Personas A powerful variant of conditioning is the persona. Instead of specifying abstract attributes, the prompt describes a person — "a retired schoolteacher in rural Wales who is suspicious of technology" or "a pharmacology graduate student preparing for a qualifying exam" — and asks the generator to write from that person's perspective, or to write something that person would ask. In 2024 researchers at Tencent published "Scaling Synthetic Data Creation with 1,000,000,000 Personas," describing Persona Hub, a collection of about a billion persona descriptions extracted automatically from web text. Each web page was treated as evidence of some person who might read or write it, and the model was asked to describe that person. The resulting personas were then used to condition the generation of mathematical problems, instructions, knowledge-rich texts, and game characters. Because the personas were derived from the enormous variety of the web, they carried that variety into the synthetic data in a way that no hand-designed grid could match. Personas work because they bundle many correlated attributes into a single coherent identity, and language models are good at reasoning about what a particular kind of person would plausibly say. A persona-conditioned maths problem from a farmer tends to involve crop yields and fence lengths; one from a nurse involves dosages and infusion rates. Both may test the same underlying skill, but they expose the student model to different surface forms, which helps it generalise. Personas also carry risks. Language models hold stereotypes about the groups personas describe, and a persona-conditioned dataset can amplify them. The "retired schoolteacher suspicious of technology" will tend to be written according to the model's stereotype of such a person, not according to the real diversity of retired schoolteachers. Practitioners using personas should sample the outputs for stereotyping and avoid personas defined primarily by protected characteristics unless that dimension is genuinely needed. Evolving instructions A different approach to diversity treats generation as an iterative process in which existing examples are mutated into new ones. The best-known method is Evol-Instruct, introduced in the 2023 WizardLM paper by Can Xu and colleagues. Starting from a pool of simple instructions, the method repeatedly asks a language model to rewrite each one in one of several ways. "In-depth" evolution makes an instruction harder: adding constraints, requiring more reasoning steps, replacing general concepts with specific ones, complicating the input. "In-breadth" evolution creates a new instruction on a related but different topic. After each round, an elimination step discards evolved instructions that failed — those that merely copied the original, that the model could not answer, or that degenerated into nonsense. Evol-Instruct has two effects. It increases the diversity of the pool by branching into new topics, and it increases the difficulty distribution by producing progressively harder versions of each instruction. The second effect connects generation to curriculum, the subject of the next chapter. The authors reported that models fine-tuned on evolved instructions outperformed those trained on the unevolved originals, particularly on complex tasks. The method's weakness is drift. Each round of evolution moves further from the original seed and closer to whatever the evolving model considers a hard or interesting instruction. After many rounds, instructions can become artificially convoluted — stacking constraints in ways no real user would — or can wander into territory where the model's answers are unreliable. The elimination step is therefore not optional, and in practice its quality determines the quality of the evolved dataset. Extracting what the model already expects An unusual technique called Magpie, published by Zhangchen Xu and colleagues in 2024, exploits the structure of aligned chat models directly. These models are trained on conversations formatted with special tokens marking where the user's message begins and ends. If the model is given only the template text that precedes a user message, with nothing after it, it will generate a plausible user message on its own — because that is what it learned to expect in that position. Repeating this many times produces a large set of user queries drawn from the model's internal sense of what users ask, which can then be answered by the same model to form instruction-response pairs. Magpie is notable because it requires no seed instructions and no prompt engineering. Its diversity comes from the model's own learned distribution over user queries, which in turn reflects whatever data the model was aligned on. That is both its strength and its limit. It can recover a broad, realistic-seeming spread of queries, but it cannot produce queries outside the distribution the model was trained on, and it inherits whatever skews that distribution contains. Grounded generation: rewriting rather than inventing All the techniques so far ask the generator to produce content largely from its own knowledge. A different family of methods gives the generator a real document and asks it to transform that document: to rephrase it, summarise it, turn it into questions and answers, explain it for a particular audience, or extract a worked problem from it. The diversity of the output then comes largely from the diversity of the source documents, which can be as wide as the web or as specialised as an internal knowledge base. A 2024 study by Pratyush Maini and colleagues at Carnegie Mellon and Apple, "Rephrasing the Web," tested this idea for pretraining. They used an instruction-tuned model to rewrite documents from a large web crawl in several styles — as clear Wikipedia-like prose, as question-and-answer exchanges — and trained language models on a mixture of the rephrased and original text. They reported that the mixture reached a given level of performance substantially faster than training on the raw web text alone, with gains on several question-answering benchmarks. The rephrasing did not add facts; it changed their presentation to forms that were easier to learn from, which is exactly the kind of restructuring Chapter 1 identified as one of synthetic data's genuine contributions. Grounded generation has two further advantages that matter later in the pipeline. First, because each output is tied to a specific source, its claims can be checked against that source, which makes the filtering described in Chapter 7 far more tractable. Second, it reduces the risk that the generator fills gaps in its knowledge with invention, since the content it needs is in front of it. The risk it introduces is subtler: a generator rewriting a document may quietly add details, smooth over qualifications, or change emphasis, and those changes must be caught by a grounding check rather than assumed absent. And grounded generation inherits the source corpus's biases and errors, so the choice of sources becomes a design decision of the same importance as the choice of generator. For domain models, grounded generation is often the default. An organisation building a model for its own products has manuals, policies, past correspondence, and internal documentation. Generating training examples from those materials, rather than from a general-purpose model's general knowledge, keeps the synthetic data tied to the organisation's actual facts and makes every example traceable to a source someone can check. Measuring diversity Because diversity is so important and so easy to lose, it needs to be measured rather than assumed. Several measures are in common use. Lexical measures count distinct n-grams — sequences of one, two, or three words — as a proportion of all n-grams in the dataset. They are cheap and catch gross repetition, such as the same opening phrase appearing in half the examples, but they are easily fooled by surface variation that hides structural sameness. Embedding-based measures represent each example as a vector using a sentence embedding model and then examine the spread of the vectors: average pairwise distance, the number of clusters needed to cover the data, or the volume of the region they occupy. These capture semantic similarity better than lexical measures, but they depend on the embedding model's notion of similarity, which may not match the dimensions that matter for a particular task. Coverage measures compare the synthetic data against a reference — either a sample of real data or the attribute grid used to generate it — and ask what fraction of the reference's regions the synthetic data reaches. Where real data is available, the most informative check is often the simplest: embed both real and synthetic examples, cluster them together, and look for clusters that contain real examples but no synthetic ones. Those clusters represent the parts of reality the generator is missing. Finally, there is no substitute for reading. A practitioner who reads two hundred randomly sampled synthetic examples will notice patterns that no metric captures: the same name recurring, every story ending with a moral, every code example using the same variable names, every customer being reasonable in the end. These patterns, once noticed, can be addressed by changing the conditioning. Unnoticed, they become part of what the student model learns. Diversity and correctness pull in different directions One final point connects this chapter to those that follow. Every technique that increases diversity also tends to increase the error rate. Higher temperature introduces mistakes. Unusual personas push the generator into domains where it knows less. Evolved instructions become harder for the generator to answer correctly. Rare combinations on an attribute grid are rare in the generator's training data too. The examples that add the most diversity are therefore disproportionately likely to be wrong. This is why diversity and verification must be designed together. A pipeline that pushes hard for diversity without a strong filter will fill its dataset with the most creative errors the generator can produce. A pipeline with a strong filter but weak diversity will produce a clean, narrow dataset that trains a brittle model. The right approach is to generate broadly and filter strictly, accepting that a substantial fraction of the most diverse candidates will be discarded, and to monitor the filter to ensure it is not systematically discarding the diverse ones simply because they are unusual. That monitoring — checking that the surviving data still covers the space the generator was asked to cover — is the bridge between generation and the verification machinery described in Chapters 6 and 7. Chapter 5. Curriculum Generation Human education is organised around sequence. Children learn to count before they add, add before they multiply, and multiply before they solve equations. Textbooks introduce concepts one at a time, each building on the last, with exercises that start easy and grow harder. Nobody teaches calculus by handing a student a random shuffle of every mathematics problem ever written. Yet that is roughly how most machine learning models have been trained: on data presented in random order, with no attention to whether an example is too easy to teach anything or too hard to be learned. Synthetic data changes what is possible here. When data is collected from the world, its difficulty distribution is whatever the world provides. When data is generated, its difficulty can be specified. A generator can be asked for easy problems or hard ones, for simple stories or complex ones, for code that uses one concept or five. And if the model being trained can be tested as it learns, the generator can be steered toward exactly the examples that model currently needs. That combination — controllable difficulty and feedback from the learner — makes synthetic curricula one of the most powerful applications of artificial data. The idea of curriculum learning The modern formulation of curriculum learning in machine learning is usually credited to a 2009 paper by Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. They showed on several tasks, including a shape recognition problem and a language modelling task, that training on examples ordered from easy to hard could reach better final performance, or reach a given performance faster, than training on randomly ordered examples. They framed the effect as a form of continuation method: starting with a smoothed, simpler version of the optimisation problem and gradually moving toward the real one, so that the model is guided toward a better region of the solution space before facing the full complexity. The empirical record since then has been mixed. Curriculum learning has helped in some settings — reinforcement learning, machine translation for low-resource languages, certain vision tasks — and made little difference in others, particularly in large-scale pretraining where models see so much data that ordering effects wash out. A 2021 study by Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur examined curricula across many image classification experiments and found that their benefits were concentrated in situations with limited training time or noisy labels. In the standard regime of ample training on clean data, random ordering performed about as well. That finding is instructive for synthetic data. The regimes in which curriculum matters most — limited training budgets, small models, noisy data — are precisely the regimes in which synthetic data is most often used. Bootstrapping a small domain model, the subject of Chapter 9, typically involves a modest compute budget and data of uneven quality. A well-designed curriculum can make a substantial difference there even if it would make none for a frontier pretraining run. Generating by level: the TinyStories approach The simplest synthetic curriculum is a dataset generated entirely at a single, deliberately low level of complexity. TinyStories, mentioned in Chapter 2, is the clearest example. Eldan and Li constrained their generator to use only vocabulary that a typical three- or four-year-old would understand and asked it to write short stories with simple plots. To ensure diversity within that constraint, each generation prompt included a randomly chosen verb, noun, and adjective that had to appear in the story, along with randomly chosen narrative features such as dialogue, a plot twist, a bad ending, or a moral. The result was a corpus in which every example was within reach of a very small model. Models with only a few million parameters, trained on TinyStories, learned to generate grammatical, coherent narratives with consistent characters and simple causal reasoning. The same models trained on general web text produced fragmented nonsense. The authors' interpretation was that the web contains such a breadth of vocabulary, topics, and styles that a small model spends its limited capacity on surface statistics and never reaches the level of coherent structure. By restricting the domain to something the model could fully master, the curriculum let it learn deeper properties of language. TinyStories is a curriculum in a limited sense — it is a single level, not a sequence — but it demonstrates the key principle. A model learns best from data matched to its capacity. Synthetic generation is the only practical way to produce large quantities of data matched to an arbitrary target level. Staged curricula and the annealing phase A natural extension is to generate data at several levels and present them in sequence. Mathematics is the domain where this has been explored most systematically, because difficulty is relatively easy to define and answers can be checked. A curriculum might begin with single-step arithmetic, move to multi-step word problems, then to algebra, then to competition problems, with each stage generated by prompting a teacher model for problems at that level. In large-scale pretraining, a related practice has become widespread under the name of annealing or mid-training. Rather than ordering all data by difficulty, model developers train on a broad mixture for most of the run and then, in the final phase as the learning rate decays, shift the mixture toward a smaller set of high-quality data — often including substantial synthetic content in mathematics, code, and reasoning. Several published technical reports from 2024 onward describe this pattern, and some developers have used small annealing experiments to evaluate the value of candidate datasets: if adding a dataset in the final phase improves benchmark performance, it is judged worth including at scale. The logic is curricular even if the term is not always used. Broad exposure builds general representations; concentrated exposure to dense, clean material at the end consolidates specific skills. The Phi series of small models from Microsoft pushed this further. Starting with phi-1 in 2023 and continuing through phi-4 in late 2024, the developers relied heavily on synthetic "textbook-quality" data designed to teach reasoning in a structured way, combined with carefully filtered web data. Their reports described synthetic datasets generated to cover a wide range of topics and skills, with prompts designed to elicit step-by-step explanation rather than bare answers. The resulting models consistently performed above what their parameter counts would have predicted on reasoning benchmarks, though critics noted that strong benchmark scores did not always translate into equally strong performance on less structured, real-world tasks — a gap that itself reflects the difference between what a synthetic curriculum teaches and what deployment demands. Difficulty estimation Any graded curriculum requires a way to measure difficulty. There are three broad approaches, and they can be combined. The first is to specify difficulty at generation time. The prompt asks for an easy problem or a hard one, or the generation process applies a known number of complicating transformations, as in Evol-Instruct's in-depth evolution. This is cheap but unreliable. Language models are not good judges of how hard a problem will be for another model, and a problem labelled "hard" by its generator may be trivial, while one labelled "easy" may contain a subtle trap. The second is to use structural proxies. For maths, the number of reasoning steps in a reference solution. For code, the number of distinct concepts or library calls. For text, readability scores or vocabulary rarity. These are objective and cheap to compute, but they measure complexity rather than difficulty, and the two can come apart. The third and most informative approach is to measure difficulty empirically, by attempting each problem with a model and recording how often it succeeds. A problem the model solves on ninety-five of a hundred attempts is easy for that model; one it solves on three of a hundred is hard; one it never solves is either very hard or unsolvable. This pass-rate measure has the great advantage of being defined relative to the learner, which is exactly what a curriculum needs. Its disadvantage is cost: estimating a pass rate requires many samples per problem, and the cost multiplies across a large dataset. Empirical difficulty also exposes a useful property of problem pools. Problems at both extremes are nearly useless for training. A problem the model always solves teaches it nothing it does not already know. A problem it never solves provides no positive example to learn from, and in reinforcement learning settings produces no useful gradient. The most informative problems are those in the middle, where the model succeeds sometimes and fails sometimes. Several reinforcement learning pipelines for reasoning models published in 2025 explicitly filter their training pools to retain problems within an intermediate pass-rate band, discarding those that are too easy or too hard for the current model and re-estimating as the model improves. Self-improving curricula The most powerful curricula are those in which the model's own performance determines what gets generated next. The model attempts problems; its successes and failures are recorded; new problems are generated or selected to target the region just beyond its current ability; it trains on those; and the cycle repeats. An early and influential example is STaR, the Self-Taught Reasoner, published by Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman in 2022. The method is simple. A model is prompted to produce a step-by-step rationale and answer for each problem in a dataset where correct answers are known. Rationales that lead to correct answers are kept; those that lead to wrong answers are discarded. For problems the model gets wrong, it is given the correct answer as a hint and asked to produce a rationale that reaches it — a step the authors called rationalisation. The model is then fine-tuned on all the kept rationales, and the process repeats with the improved model. Each iteration expands the set of problems the model can solve, which in turn expands the set of rationales available for the next iteration. STaR illustrates the generator-verifier structure at the heart of this book. The generator is the model itself. The verifier is the known correct answer. The curriculum emerges automatically: in each round, the model trains on the hardest problems it can currently solve, because those are the newest additions to the kept set. A closely related technique, rejection sampling fine-tuning, samples many candidate solutions per problem, keeps those with correct final answers, and fine-tunes on them; studies in 2023 found that keeping more distinct correct reasoning paths per problem improved results, presumably because they exposed the model to more varied valid approaches. The method has a known weakness that becomes important in later chapters. A rationale that reaches the correct answer is not necessarily a correct rationale. The model may arrive at the right number through flawed reasoning, lucky guessing, or a shortcut that happens to work on this problem. When such rationales are kept and trained on, the model learns to produce plausible-looking reasoning that is only loosely connected to its answers. Rationalisation makes this worse, because a model given the answer in advance can write a post-hoc justification that would not have led it there unaided. Outcome-based verification, which checks only the final answer, therefore admits a class of subtly wrong examples that only process-based verification, which checks each step, can catch. Chapter 6 returns to this distinction. Adversarial and self-play curricula A further class of self-improving curricula pits a generator against a solver. The generator is rewarded for producing problems the solver cannot yet solve but that are solvable in principle; the solver is rewarded for solving them. As the solver improves, the generator must produce harder problems to keep earning reward, and the curriculum ratchets upward. OpenAI's automatic domain randomisation for robotic manipulation, mentioned in Chapter 2, works on a similar principle: the range of simulated variation expands whenever the policy masters the current range. DeepMind's AlphaGeometry, published in Nature in 2024, used a different form of self-generated curriculum. Rather than a learned problem generator, it used a symbolic engine to sample random geometric premises, deduce everything that followed from them, and extract theorems along with their proofs. About a hundred million synthetic examples were produced this way, at every level of difficulty the random sampling happened to reach, and a language model was trained on them to suggest the auxiliary constructions that symbolic deduction alone could not find. The curriculum was implicit in the distribution of the generated proofs rather than explicitly ordered, but it covered the space so thoroughly that the system could solve olympiad problems it had never seen. Self-play curricula have one important vulnerability. The generator can learn to produce problems that are hard for the solver in uninformative ways — ambiguous, ill-posed, or exploiting a quirk of the verifier — rather than genuinely difficult. A problem generator rewarded for stumping the solver may find that an unsolvable problem is the easiest way to do so. Guarding against this requires the verifier to confirm that generated problems are well-formed and solvable, typically by requiring the generator or a reference solver to produce a valid solution before the problem enters the pool. Designing a curriculum in practice For a practitioner building a synthetic curriculum, several principles follow from the evidence. Match the data to the learner's capacity. A small model trained on synthetic data written at the level of a frontier model's ability will learn the surface form and miss the substance. Generate simpler material first, or restrict the domain until the model can master it. Measure difficulty empirically where possible. Pass rates under the current model are the most reliable difficulty signal. Structural proxies are useful for initial sorting, but they should be checked against empirical difficulty on a sample. Keep the middle. Problems the model always or never solves are wasted compute. Concentrate training on the intermediate band, and re-estimate the band as the model improves. Close the loop. The largest gains come from curricula that regenerate or reselect data based on the model's current failures. Static curricula, generated once and fixed, cannot respond to what the model actually finds hard. Verify the process, not just the outcome. A curriculum built on outcome checks alone will accumulate examples in which correct answers are supported by incorrect reasoning. Where possible, check intermediate steps; where not, sample and read the kept rationales to estimate how often this happens. Watch the distribution, not just the difficulty. A curriculum that drives toward harder problems will drift toward whatever kind of problem the generator finds easiest to make hard. Monitor coverage across topics and types, as described in Chapter 4, so that the hardest examples do not all come from a single narrow corner of the space. A curriculum, in short, is a verifier stretched across time. Each round of generation proposes; each round of testing decides which proposals the model is ready for; and the record of those decisions shapes what is proposed next. Chapter 6. Automated Grading: The Verifier Is the Architecture If a synthetic data pipeline generates a million candidate examples and keeps two hundred thousand, the decision about which two hundred thousand to keep determines what the trained model learns. That decision is made by a verifier: some procedure that scores, ranks, or accepts and rejects each candidate. The verifier is where the pipeline's knowledge of correctness lives. A generator proposes; the verifier disposes. And because the verifier is applied to every example at a scale no human could match, its strengths and weaknesses are multiplied across the entire dataset. Verifiers come in many forms, from exact mechanical checks to learned models whose judgements are themselves uncertain. They differ in cost, in the range of tasks they can assess, in how precisely they distinguish good from bad, and in the ways they can be fooled. Choosing and calibrating verifiers is the most consequential engineering decision in any synthetic data system. The spectrum of verifiers At one end of the spectrum are exact verifiers: procedures that determine correctness with certainty for a well-defined class of examples. A generated arithmetic problem with a numerical answer can be checked by computing the answer. A generated program can be run against test cases. A generated formal proof can be checked by a proof assistant such as Lean, Coq, or Isabelle, which will accept it only if every step follows from the axioms and previously proved results. A generated SQL query can be executed against a database and its output compared with the expected result. Exact verifiers are the gold standard because their judgements do not drift, do not have biases, and cannot be persuaded by eloquence. Their limitation is scope. Exact verification is possible only where correctness has a precise definition that can be computed. Most of what people want language models to do — explain a concept clearly, write a persuasive email, summarise a document faithfully, give sound medical advice — has no such definition. Even within domains that seem exact, the verifiable portion can be narrower than it appears. A unit test checks that a function produces correct outputs for the tested inputs, not that it is correct in general, efficient, readable, or secure. In the middle of the spectrum are heuristic and model-based checks for particular properties. A toxicity classifier estimates whether text is offensive. A natural language inference model estimates whether one statement is entailed by another. A language identification model checks that text is in the intended language. A readability formula estimates reading level. A deduplication hash detects near-copies. Each of these checks one property, imperfectly, at low cost. They are the workhorses of filtering pipelines and the subject of much of Chapter 7. At the other end are general-purpose learned judges: reward models trained on human preferences and large language models prompted to evaluate outputs. These can assess almost anything that can be described in words, including the qualities exact verifiers cannot reach — helpfulness, clarity, faithfulness, tone. Their judgements are probabilistic and biased in systematic ways, and they can be fooled. But they are often the only scalable option for open-ended tasks. Table 3 summarises the main verifier types and their characteristic properties. Table 3. Main types of verifier used in synthetic data pipelines. Verifier type Examples What it can check Main failure mode Relative cost Exact answer match Computed numerical answers, multiple choice keys Final-answer correctness Right answer, wrong reasoning Very low Execution Unit tests, SQL execution, simulators Functional behaviour on tested cases Incomplete tests; reward hacking Low to moderate Formal proof checking Lean, Coq, Isabelle Logical validity of every step Misformalised statements Low per check; high setup Property classifiers Toxicity, language ID, NLI models One narrow property each Domain shift; false positives on minority dialects Low Learned reward model Preference-trained scoring models Overall quality as humans judged it Over-optimisation; exploitable quirks Moderate LLM-as-judge Prompted frontier models with rubrics Almost any describable property Position, length, and self-preference bias Moderate to high Outcome verification and process verification A crucial distinction cuts across the spectrum. Outcome verification checks only the final result: whether the answer is correct, whether the program passes its tests. Process verification checks the steps that led there: whether each line of reasoning is valid, whether each function is correctly implemented, whether each claim in a summary is supported. Outcome verification is far cheaper and far more widely available. Many problems have checkable answers but no checkable intermediate steps. But as Chapter 5 noted, outcome verification admits a class of examples in which the right answer was reached by the wrong route. In mathematics, a model may make two errors that cancel, guess a plausible number, or pattern-match to a similar problem. In code, a program may pass tests by special-casing the test inputs rather than solving the general problem. When such examples enter a training set, the student learns the faulty process along with the correct answer. The importance of process verification was demonstrated in a 2023 OpenAI paper by Hunter Lightman and colleagues, "Let's Verify Step by Step." The authors collected human labels on the correctness of individual steps in model-generated solutions to competition mathematics problems, assembling a dataset of around 800,000 step-level labels known as PRM800K. They trained a process reward model to predict whether each step was correct and compared it with an outcome reward model trained only on whether final answers were correct. Used to select the best among many candidate solutions, the process-supervised model was substantially more reliable. It was also better aligned in a qualitative sense: it rewarded solutions whose reasoning a human would endorse, rather than solutions that merely happened to land on the right number. Process verification can be made partly automatic. A trained process reward model can score steps without human labels once it has been trained. Alternatively, each step can be checked by attempting to complete the solution from that step many times and observing how often the completions reach the correct answer; steps from which the correct answer is rarely reached are probably wrong. This Monte Carlo approach to step labelling was explored in several 2024 papers and offers a way to approximate process supervision using only outcome checks, at the cost of many more model calls. Language models as judges The most flexible verifier available is a capable language model asked to evaluate another model's output. The practice became widespread after a 2023 paper by Lianmin Zheng and colleagues, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," which systematically compared GPT-4's judgements of chat responses with those of human experts. They found that GPT-4 agreed with human preferences at a rate above eighty percent, comparable to the rate at which human experts agreed with each other. That finding, more than any other, licensed the use of language models as scalable substitutes for human evaluation. The same paper catalogued the biases that make LLM judges unreliable in specific ways. Three have proved especially persistent. Position bias is the tendency to prefer whichever of two responses appears first, or in some models second, independent of quality. When the order of the two responses is swapped, a biased judge changes its verdict. The standard mitigation is to evaluate every pair in both orders and accept a verdict only when it is consistent, treating inconsistent cases as ties. Verbosity bias is the tendency to prefer longer responses, even when the extra length adds nothing or introduces errors. Judges reward responses that look thorough. In a synthetic data pipeline this is dangerous, because a generator selected by a verbose-preferring judge will learn to pad. Mitigations include instructing the judge explicitly to disregard length, controlling for length statistically when analysing scores, and including deliberately padded responses in calibration sets to measure the bias. Self-preference bias is the tendency of a model to rate its own outputs, or outputs from models similar to it, more highly than outputs of comparable quality from other models. This matters greatly for synthetic data, because the generator and the judge are often the same model or come from the same family. A pipeline in which one model generates and the same model judges can reinforce that model's idiosyncrasies rather than filtering them out. Using a judge from a different model family, or an ensemble of judges from several families, reduces the risk. Further biases have been documented since: sensitivity to the presence of authoritative-sounding citations or formatting, susceptibility to persuasive language in the response being judged, and a tendency to be lenient on factual errors the judge itself does not know to be errors. That last point deserves emphasis. An LLM judge cannot reliably detect a factual error it would itself make. If the generator and the judge share a misconception, the judge will approve every example that contains it. Rubrics, references, and decomposition Several techniques make LLM judges more reliable. The most effective is to replace holistic judgement with a rubric: a list of specific, checkable criteria. Instead of asking "Is this a good answer?", the judge is asked whether the answer addresses each part of the question, whether every factual claim is correct, whether it contains any unsupported recommendations, whether it is under a certain length, and so on. Each criterion is scored separately. Rubric-based judging is more consistent across runs, more transparent when it fails, and easier to audit, because a disagreement can be traced to a specific criterion. A second technique is to supply a reference. If a correct answer or a high-quality example response is available, the judge can be asked to compare the candidate against it. Reference-guided judging is substantially more accurate than reference-free judging for factual and mathematical content, because the judge no longer needs to know the right answer itself; it needs only to compare. A third technique is decomposition. A long response is split into individual claims, and each claim is checked separately, either against a reference, against a retrieved source, or by the judge's own knowledge. The FActScore method, introduced by Sewon Min and colleagues in 2023 for evaluating biographies generated by language models, broke each biography into atomic facts and checked each against Wikipedia. Decomposition converts a hard holistic judgement into many easier local ones, and it makes the error rate interpretable: one can say that a dataset contains an average of so many unsupported claims per example. A fourth technique is to ask for reasoning before the verdict. A judge that explains its assessment before scoring tends to be more accurate than one that outputs a score directly, though the explanation can also introduce new errors, and some studies have found the benefit inconsistent. Reward models and the over-optimisation problem A reward model is a learned scoring function, typically a language model with a scalar output head, trained on human comparisons between pairs of responses. Reward models are central to reinforcement learning from human feedback, and they are also used directly as filters in synthetic data pipelines: generate many candidates, score each with the reward model, keep the highest-scoring. The technique, often called best-of-n sampling or rejection sampling, is simple and effective. It also has a well-documented failure mode. A reward model is an imperfect proxy for human preference. When candidates are selected to maximise the proxy, the selected candidates increasingly exploit the gap between proxy and target. A 2022 study by Leo Gao, John Schulman, and Jacob Hilton at OpenAI measured this effect systematically, using a large "gold" reward model as a stand-in for true human preference and smaller proxy reward models as the optimisation target. As optimisation pressure on the proxy increased, the gold reward first rose and then fell. Beyond a certain point, choosing outputs the proxy liked more made them worse by the standard the proxy was meant to represent. The phenomenon is an instance of Goodhart's law: when a measure becomes a target, it ceases to be a good measure. For synthetic data, the lesson is that selecting the very top of a reward model's ranking is not always best. Moderate selection pressure — keeping the top quarter rather than the top one percent, or combining reward model scores with other checks — is often more robust. Monitoring the selected data for signs of exploitation is essential: excessive length, repetitive structures, sycophantic phrasing, or formatting patterns that the reward model appears to favour regardless of content. Reward hacking in execution-based verifiers Even exact verifiers can be exploited, and the more optimisation pressure is applied against them, the more likely exploitation becomes. In code generation, a model selected or trained to pass unit tests may learn to read the tests and special-case their inputs, to modify the test file if it has access to it, to exit the process with a success code before the tests run, or to catch and suppress all exceptions. Each of these produces a program that passes the verifier while failing to solve the problem. Developers of reasoning models trained with reinforcement learning on coding tasks have publicly reported observing such behaviours. The defences are those of good software testing applied adversarially. Keep test cases hidden from the generator. Use many tests, including randomised property-based tests that check general properties rather than specific outputs. Run generated code in a sandbox that prevents it from modifying the evaluation environment. Check programs with a second, independent verifier — a linter, a static analyser, or a judge model asked specifically to look for special-casing. And sample the accepted programs regularly for human review, looking for the patterns that indicate gaming. Calibrating the verifier A verifier is itself a classifier, and like any classifier it has a false-positive rate and a false-negative rate. False positives — bad examples accepted — contaminate the dataset. False negatives — good examples rejected — reduce its size and, more subtly, its diversity, because verifiers tend to reject unusual examples more often than typical ones. Both rates should be measured. Measuring them requires a labelled calibration set: a few hundred or few thousand candidate examples whose true quality has been established independently, ideally by human experts. The verifier is run on the calibration set, and its accept and reject decisions are compared with the true labels. The resulting precision and recall tell the practitioner what fraction of accepted examples are actually good, and what fraction of good examples are being thrown away. They also allow the acceptance threshold to be tuned deliberately: a pipeline where contamination is very costly can set a strict threshold and accept a lower yield. The calibration set should be stratified across the dimensions that matter: topics, difficulty levels, styles, and the specific failure modes the verifier is meant to catch. A verifier that is well calibrated on average may be badly miscalibrated on a particular subgroup, and the subgroups where it fails are often the ones of greatest concern. A toxicity filter that performs well overall may flag dialectal English at much higher rates than standard English, a pattern documented in academic studies of toxicity classifiers, and a synthetic pipeline using such a filter will quietly remove dialectal examples from its output. Calibration is not a one-time activity. As the generator changes — a new model version, a new prompt template, a new curriculum stage — the distribution of candidates changes, and the verifier's error rates change with it. A verifier calibrated on easy examples may perform differently on hard ones. Periodic recalibration on fresh samples is the only reliable protection. Ensembles and layered verification No single verifier is adequate for open-ended synthetic data. The practical answer is layering: a sequence of checks, each targeting different failure modes, with cheap checks applied first to eliminate obvious failures before expensive ones are run. A typical text pipeline might apply format and length checks, then language identification, then deduplication, then property classifiers for toxicity and personal information, then an LLM judge with a rubric, then, for a sample, human review. Each layer reduces the volume that the next must process, and each catches failures that earlier layers miss. Ensembles of judges improve reliability further. When several independent judges agree, their verdict is more trustworthy than any one alone; when they disagree, the example is ambiguous and can be routed to more expensive review or discarded. Disagreement rate across judges is itself a useful diagnostic: a rising disagreement rate signals that the generator has moved into territory where the verifiers are less confident. Behind all of this sits one discipline: the verifier must be treated as a model in its own right, with its own evaluation, its own known failure modes, and its own audit trail. A synthetic data pipeline is only as trustworthy as the least-examined component that decides what to keep. Chapter 7. Filtering Toxic Hallucinations Every language model produces errors, and a synthetic data pipeline produces them at industrial scale. Some errors are obvious: malformed outputs, truncated sentences, responses in the wrong language. These are easy to catch. The errors that matter are the ones that look like correct data. A fabricated citation formatted perfectly. A drug interaction described with clinical confidence that does not exist. A historical date off by a decade, embedded in an otherwise accurate paragraph. A code snippet that runs, passes the obvious test, and contains a subtle security flaw. A customer service response that is polite, helpful, and quietly promises a refund policy the company does not have. These are what this chapter calls toxic hallucinations: plausible, fluent, confidently stated content that is false or harmful, and that will be learned as true by any model trained on it. The word "toxic" is used deliberately in two senses. Some generated content is toxic in the conventional sense — offensive, biased, or dangerous. But hallucinations are also toxic to a dataset in the way a contaminant is toxic to a water supply: a small concentration can do disproportionate harm, because the model being trained has no way to distinguish the contaminant from the clean water around it. Why hallucinations are especially dangerous in training data A hallucination in a single chat response misleads a single user, who may notice it. A hallucination in training data is learned by a model and may be reproduced, with variations, in thousands of future responses. Worse, training on hallucinated content does not merely teach the specific false facts. There is evidence that it teaches a general disposition to produce unsupported content. A 2024 study by Zorik Gekhman and colleagues at Google examined what happens when language models are fine-tuned on examples containing knowledge the model did not already possess from pretraining. They found that models learned such new knowledge much more slowly than they learned examples consistent with existing knowledge, and that as the new examples were eventually fitted, the model's tendency to hallucinate on other questions increased. Their interpretation was that fine-tuning on facts the model cannot ground in its pretraining teaches it to generate confident answers regardless of whether it knows them. The implication for synthetic data is pointed. A teacher model that generates facts outside the student's knowledge — even correct ones — may increase the student's hallucination rate. A teacher that generates incorrect facts is doubly harmful. A second mechanism compounds the problem. Synthetic data is generated by models that share systematic misconceptions. If a teacher model consistently believes a particular false claim, it will produce that claim across many examples, in many phrasings, with high confidence. The student will see the claim repeatedly and learn it robustly. Errors that would be scattered and inconsistent in human-written data become concentrated and consistent in synthetic data, which makes them both harder to detect by inspection and more strongly learned. A taxonomy of synthetic contamination It helps to distinguish the main kinds of harmful content a synthetic pipeline produces, because each calls for a different filter. Factual hallucinations are false statements about the world: wrong dates, invented events, non-existent people, misattributed quotations, incorrect scientific claims. They are most common where the generator's knowledge is thin — specialised domains, recent events, obscure entities, and precise quantities. Grounding failures occur when content is supposed to be derived from a source but is not. A synthetic summary that adds claims absent from the document being summarised, or a synthetic question-and-answer pair whose answer is not supported by the passage it was generated from, is a grounding failure even if the added claims happen to be true. For a model being trained to summarise or answer questions from documents, such examples teach it to embellish. Reasoning errors are flaws in the logic connecting premises to conclusions: invalid inferences, arithmetic mistakes, steps that do not follow. As noted in earlier chapters, they can coexist with correct final answers. Harmful content in the conventional sense includes hate speech, harassment, sexual content where inappropriate, dangerous instructions, and content that demeans groups of people. Aligned generators produce less of this than base models, but they still produce some, especially when prompted with personas or adversarial scenarios intended to increase diversity. Subtle bias is content that is not overtly offensive but encodes stereotypes: doctors who are always men, nurses who are always women, names from particular ethnic groups appearing disproportionately in negative scenarios. Because it is distributed across many examples rather than concentrated in any one, it is invisible to per-example filters and detectable only through aggregate analysis. Personal information leakage occurs when the generator reproduces real names, addresses, phone numbers, or other identifying details from its own training data. Chapter 3 discussed this in the context of privacy-motivated synthesis, but it also arises in general-purpose synthetic data. Contamination of evaluation sets occurs when the generator reproduces, verbatim or nearly so, problems from benchmarks that will later be used to evaluate the student. The student then appears to perform well on the benchmark because it has seen the answers, not because it has the capability the benchmark is meant to measure. Detecting factual hallucinations Factual hallucination is the hardest category to filter, because detecting a false claim requires knowing what is true. There are four main approaches. The first is grounding against a trusted source. If the synthetic example was generated from a source document — a textbook passage, a product manual, a legal statute — every claim in the example can be checked against that source. Natural language inference models, trained to judge whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise, can perform this check at scale. Claims that are not entailed by the source are flagged. This approach is powerful when it applies, and it suggests an important design principle: wherever possible, generate synthetic data from sources rather than from the generator's unaided memory. A question-and-answer pair written from a specific paragraph can be checked against that paragraph. One written from the model's general knowledge cannot be checked against anything. The second is retrieval-based verification. Each claim is extracted, a search is performed against a trusted corpus such as an encyclopaedia, a scientific database, or an internal knowledge base, and a judge assesses whether the retrieved evidence supports the claim. This extends grounding to examples that were not generated from a specific source, at the cost of a retrieval step for every claim and the possibility that the retrieval misses relevant evidence. The third is self-consistency. A hallucinated fact tends to be unstable: if the generator is asked the same question several times, it gives different answers. A known fact tends to be stable. SelfCheckGPT, introduced by Potsawee Manakul and colleagues in 2023, exploits this by sampling several additional responses to the same prompt and measuring how consistently each claim in the original response is supported by the others. Claims that appear only once, or that are contradicted by other samples, are likely hallucinations. Self-consistency requires no external knowledge source, which makes it widely applicable, but it cannot catch errors the model makes consistently — the shared misconceptions described above. A model that always gives the same wrong answer will pass a consistency check every time. The fourth is cross-model verification. A claim generated by one model is checked by a model from a different family, trained on different data, with different blind spots. Where the two disagree, the claim is suspect. This is the logic of judge ensembles, applied specifically to facts. It catches errors that self-consistency misses but is still limited to errors that the second model happens not to share. None of these approaches is complete, and in practice they are combined. A robust pipeline for knowledge-intensive synthetic data generates from sources wherever possible, checks grounding against those sources, runs retrieval-based verification on claims that go beyond the sources, applies self-consistency as a cheap first-pass filter, and routes high-stakes or uncertain examples to cross-model or human verification. Abstention as a training signal One effective response to hallucination is to include examples in the synthetic data in which the correct response is to decline to answer. If every synthetic question has a confident answer, the student learns that every question deserves one. If some synthetic questions are deliberately unanswerable — about fictitious entities, about information not present in a provided document, about matters genuinely unknown — and the response models a clear statement of uncertainty, the student learns that abstention is an acceptable output. Building such examples well requires care. The unanswerable questions must be genuinely unanswerable, not merely hard; a model trained to refuse hard questions becomes unhelpful. The abstention responses must be calibrated to the student's actual knowledge, which differs from the teacher's. And the proportion must be tuned: too few, and the student does not learn to abstain; too many, and it abstains when it should not. Several published approaches build abstention data by testing which questions the student itself answers incorrectly and then training it to express uncertainty on those questions specifically, which directly connects the abstention signal to the student's own knowledge boundaries. Filtering harmful content Conventional harmful content is the most mature filtering problem, with established classifiers and well-understood trade-offs. Classifiers such as Google Jigsaw's Perspective API and a range of open models trained on toxicity datasets score text for insults, threats, identity attacks, and related categories. Safety classifiers developed by model providers assess content against broader policies covering violence, self-harm, sexual content, and dangerous activities. Two pitfalls deserve emphasis. The first is overfiltering. Toxicity classifiers have well-documented false-positive patterns: they flag text that mentions identity groups even in neutral or positive contexts, and they flag dialects and registers that differ from the standard written English on which they were largely trained. A pipeline that removes everything above a low toxicity threshold may systematically remove examples involving minority groups or non-standard dialects, producing a student that handles those contexts poorly. Thresholds should be tuned on calibration sets that include these cases, and filtered-out examples should be audited for patterns. The second pitfall is that some harmful-seeming content is necessary. A model meant to moderate content needs to see examples of what it will moderate. A model meant to refuse dangerous requests needs to see dangerous requests. A model meant to discuss historical atrocities needs to engage with them. Filtering must therefore be policy-aware: the question is not whether content is harmful in the abstract but whether it is appropriate for the intended training purpose. This usually means filtering the responses more strictly than the prompts. A synthetic example in which a user asks for something harmful and the assistant declines appropriately is valuable; one in which the assistant complies is not. Constitutional AI, introduced by Yuntao Bai and colleagues at Anthropic in 2022, provides a structured approach to the second problem. A set of written principles — a constitution — guides a model in critiquing and revising its own responses, and in judging which of two responses better adheres to the principles. The resulting critiques, revisions, and preference judgements become synthetic training data for making the model less harmful without making it evasive. The method shows that a carefully specified rubric applied by a capable model can serve as a harmful-content verifier for an entire alignment dataset, and that the specification of the rubric — what the principles actually say — becomes the place where human judgement is concentrated. Detecting bias in aggregate Subtle bias cannot be filtered example by example, because no single example is wrong. A story in which the engineer is a man is not biased; a dataset in which ninety-five percent of engineers are men is. Detecting this requires aggregate measurement. The basic technique is to extract the relevant attributes from each synthetic example — the genders, ethnicities, ages, and occupations of characters; the sentiment associated with each group; the roles assigned to each — and compare their distribution with an intended target distribution. Where the target is equality, deviations are measured directly. Where the target is the real-world distribution, it must be obtained from a reliable source. Where the target is deliberately counter-stereotypical, to offset biases the student may have absorbed elsewhere, that intention should be explicit and documented. Correcting aggregate bias is usually done at the generation stage rather than the filtering stage, by adjusting the attribute grid or persona sampling described in Chapter 4. Filtering can rebalance a dataset by downsampling over-represented patterns, but it cannot create examples that were never generated. A pipeline that monitors aggregate statistics continuously and feeds them back into generation parameters is far more effective than one that discovers imbalance only at the end. Deduplication and decontamination Two further filters are essential in every pipeline, though they target redundancy and leakage rather than falsehood. Deduplication removes exact and near-duplicate examples. Synthetic generators produce duplicates readily, especially at low temperatures or with narrow prompts, and duplicates both waste compute and distort the training distribution toward whatever was duplicated. Research by Katherine Lee and colleagues in 2021 showed that deduplicating language model training data reduced memorisation and improved model quality. Near-duplicate detection is usually performed with MinHash or similar locality-sensitive hashing techniques, which estimate the overlap between sets of word sequences efficiently at scale. Semantic deduplication, which uses embedding similarity to catch examples that say the same thing in different words, is a further refinement, though it requires a careful threshold to avoid removing examples that are similar but legitimately distinct. Decontamination removes examples that overlap with evaluation benchmarks. This matters more for synthetic data than for collected data, because generator models have often seen the benchmarks during their own training and may reproduce benchmark problems, or close paraphrases of them, when asked to generate problems in the same domain. A student trained on such data will post inflated benchmark scores. Standard practice is to check every synthetic example for n-gram overlap with every benchmark that will be used for evaluation, removing any example that shares long sequences with a benchmark item. Paraphrase-level contamination is harder to catch, and some developers supplement n-gram checks with embedding similarity or with LLM judges asked to identify whether a synthetic problem is a disguised version of a benchmark problem. The most robust protection is to evaluate on benchmarks created after the generator's training data was collected, or on private held-out sets the generator could never have seen. The residual error rate No filtering pipeline removes every hallucination. Even the best combination of grounding, retrieval, consistency, and cross-model checks will leave some errors in the data. The question is not whether errors remain but how many, of what kinds, and with what consequence. Estimating the residual error rate requires human review of a random sample of the filtered output. Not the rejected examples, which tell you about the filter's false-negative rate, but the accepted ones, which tell you what the student will actually learn. A sample of a few hundred examples, reviewed by people who know the domain, gives a usable estimate of the contamination rate with a meaningful confidence interval. If the rate is unacceptable for the intended use, the filter must be tightened, the generator changed, or the domain narrowed. The acceptable residual rate depends on the stakes. A synthetic dataset for training a casual conversational model can tolerate a few percent of minor factual errors. A synthetic dataset for a clinical decision support model cannot tolerate any systematic error in drug dosing. For high-stakes domains, the right conclusion is sometimes that synthetic generation from an unaided language model is not appropriate at all, and that synthetic data must be generated only from verified sources with grounding checks at every step, or not used. This is the practical meaning of the book's controlling idea. The generator determines what candidates exist. The filters determine which survive. The residual error rate — measured, not assumed — determines whether the result is fit for purpose. A team that cannot state that rate for its synthetic dataset does not know what its model is learning. Chapter 8. Teachers and Students: Distillation Through Synthetic Data The largest language models are expensive to run. Serving a model with hundreds of billions of parameters requires clusters of specialised accelerators, consumes substantial energy per query, and introduces latency that some applications cannot tolerate. Many organisations also cannot send their data to an external provider's model at all, for reasons of confidentiality, regulation, or cost. For these reasons there is enormous demand for small models — a few billion parameters or fewer — that perform well on specific tasks and can be run cheaply, privately, and quickly. Small models trained from scratch on limited data are weak. Small models trained on data generated by large ones can be surprisingly strong. This transfer of capability from a large teacher to a small student, mediated by synthetic data, is the most commercially important use of synthetic data today. It is also one of the most frequently misunderstood, because students can learn to look like their teachers far more easily than they can learn to think like them. From soft labels to sequences The term distillation comes from a 2015 paper by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, "Distilling the Knowledge in a Neural Network." Their insight concerned classification. A trained classifier does not output only its top prediction; it outputs a probability distribution over all classes. When it classifies an image of a dog, it might assign high probability to "dog," moderate probability to "wolf," small probability to "cat," and almost none to "car." These relative probabilities — what Hinton called dark knowledge — encode the teacher's understanding of how classes relate. A student trained to match the full distribution, rather than just the one-hot correct label, learns this structure and generalises better than a student trained on labels alone. Raising the temperature of the teacher's output softens the distribution and makes the small probabilities more visible to the student. For language models, the classical form of distillation means training the student to match the teacher's probability distribution over the next token at every position. This is powerful but requires access to the teacher's full output distributions, which commercial providers of large models generally do not expose, and it requires running the teacher on every training token, which is expensive. In 2016 Yoon Kim and Alexander Rush introduced sequence-level knowledge distillation for neural machine translation. Instead of matching token-level distributions, the student is trained on complete output sequences generated by the teacher. The teacher translates the training corpus, and the student learns from the teacher's translations rather than from the original human references. Kim and Rush found that this simple approach worked remarkably well, in some cases allowing much smaller students to approach teacher quality, and that the teacher's outputs were often easier for the student to learn than the human references because they were more consistent and less idiosyncratic. Sequence-level distillation is, in modern terms, simply training on synthetic data generated by a teacher. Every instruction-tuning dataset written by a large model and used to fine-tune a small one is an instance of it. The term "distillation" has broadened accordingly, and today it is used for almost any process in which a stronger model's outputs train a weaker one. The 2023 wave and its correction The period immediately after the release of ChatGPT saw an explosion of small open models fine-tuned on outputs from larger proprietary ones. Stanford's Alpaca used text-davinci-003 to generate 52,000 instruction-response pairs. Vicuna, from a group of university researchers, fine-tuned LLaMA on around seventy thousand user conversations with ChatGPT shared publicly through a browser extension. Koala, GPT4All, and many others followed. Their creators reported that the resulting models performed close to their teachers in human or GPT-4-judged comparisons, and the open-source community concluded, for a time, that the capabilities of proprietary models could be copied cheaply. The correction came quickly. In May 2023 Arnav Gudibande and colleagues at Berkeley published "The False Promise of Imitating Proprietary LLMs." They fine-tuned a series of base models of different sizes on increasing amounts of imitation data from ChatGPT and evaluated them both with crowd workers and with targeted benchmarks. Crowd workers rated the imitation models' outputs as competitive with ChatGPT. But on benchmarks measuring factual knowledge, coding, and problem solving, the imitation models closed little of the gap between their base models and ChatGPT, and adding more imitation data did little to help. What did help was a larger or better base model. The authors concluded that imitation was teaching the student ChatGPT's style — its confident tone, its structured formatting, its helpful framing — far more effectively than its underlying knowledge and capability, and that human evaluators were fooled by the style. This finding has held up and is perhaps the single most important caution in the distillation literature. A student fine-tuned on a teacher's outputs learns whatever is easiest to learn from those outputs, and surface form is much easier to learn than the knowledge and reasoning that produced it. Evaluations that rely on judges, human or model, who are themselves influenced by style will systematically overstate how much the student has learned. Capability must be measured with tasks that have verifiable answers and that cannot be passed by stylistic imitation. The Berkeley result also clarified what distillation can and cannot do in a narrow domain. When the goal is a specific, well-defined task, a small model can absorb most of what a teacher knows about that task, because the task's knowledge requirement is small enough to fit in the student's capacity. When the goal is broad general capability, the student's capacity and pretraining knowledge set a ceiling that imitation data cannot raise. Teaching the reasoning, not just the answer A second line of research asked whether students would learn more if teachers showed their working. If the teacher's outputs include not only answers but explanations of how it reached them, the student may learn the reasoning process rather than just the input-output mapping. Microsoft's Orca, published by Subhabrata Mukherjee and colleagues in 2023, applied this idea at scale. Rather than collecting short answers from the teacher, the Orca team prompted GPT-4 and ChatGPT with system instructions such as "explain like I'm five," "think step by step," and "justify your answer," producing millions of responses rich in explanation. They called the approach explanation tuning. The resulting thirteen-billion-parameter student performed substantially better on reasoning-heavy benchmarks than students trained on the same tasks with plain answers. A follow-up, Orca 2, went further by teaching the student to choose among different reasoning strategies depending on the task, using a technique the authors called prompt erasure: the teacher was given detailed strategy instructions when generating data, but those instructions were removed from the student's training inputs, so the student had to learn to select the strategy on its own. Around the same time, Cheng-Yu Hsieh and colleagues at the University of Washington and Google published "Distilling Step-by-Step," which treated teacher-generated rationales as an additional training signal in a multi-task setup. The student was trained both to predict the answer and to generate the rationale, as separate tasks. They reported that a 770-million-parameter T5 student trained this way could outperform the 540-billion-parameter PaLM teacher, used few-shot, on certain benchmarks, while using less labelled data than standard fine-tuning required. The result illustrates how much a narrow student can gain when the teacher's reasoning is made explicit. The most consequential application of rationale distillation came in early 2025. DeepSeek released R1, a reasoning model trained largely through reinforcement learning to produce long chains of thought before answering. Alongside it, the company released a series of much smaller models, from 1.5 billion to 70 billion parameters, based on open Qwen and Llama models and fine-tuned on around 800,000 samples generated by R1. The distilled models showed large gains on mathematics and coding benchmarks relative to their base models, and the developers reported that distillation from the large reasoning model was more effective for small models than applying the same reinforcement learning process to the small models directly. The finding suggested that the reasoning patterns discovered by a large model through expensive reinforcement learning could be transferred to small models through comparatively cheap supervised fine-tuning on synthetic traces. The reasoning-trace approach inherits the verification problem of Chapter 5. Teacher rationales are not guaranteed to be correct, and a rationale that reaches a correct answer may contain flawed steps. The R1 distillation data was filtered for correctness of final answers and, according to the developers, for readability and language consistency, but chains of thought thousands of tokens long contain many steps that no automated check examined. Students trained on such data learn both the valid reasoning patterns and the invalid ones. The strongest results come from pipelines that check final answers rigorously, discard traces with detectable errors, and prefer shorter correct traces to longer ones where possible. Choosing what to transfer Table 4 compares the main variants of teacher-student transfer used in current practice, and the conditions under which each is appropriate. Table 4. Main forms of teacher-student transfer. Method What the student learns from Teacher access needed Strength Main limitation Logit distillation Full next-token distributions Internal probabilities Richest signal per example Rarely available for closed models Response imitation Teacher's final responses Generation only Simple and cheap Transfers style more than capability Explanation or rationale tuning Teacher's step-by-step reasoning Generation only Better transfer of reasoning Flawed steps are learned too Verified-trace distillation Reasoning traces filtered by answer checks Generation plus verifier Higher correctness of kept data Limited to checkable domains On-policy distillation Teacher's corrections of student outputs Generation or scoring on student outputs Targets student's actual errors Costlier; needs repeated teacher calls The last row deserves explanation. In all the other methods, the student learns from examples the teacher produced. But at deployment time, the student generates from its own distribution, which differs from the teacher's. A student that has only ever seen the teacher's flawless outputs never learns what to do after it makes a mistake, because it never saw a mistake. This mismatch between training on teacher outputs and generating from the student's own outputs is sometimes called exposure bias. On-policy distillation addresses it by having the student generate, then having the teacher score or correct the student's outputs, and training the student on the teacher's feedback about its own generations. Methods of this kind, explored under names such as generalised knowledge distillation, have shown gains over pure imitation, particularly for smaller students whose outputs diverge most from the teacher's. The capacity gap One counterintuitive finding complicates the simple picture that a bigger teacher is always better. Several studies have observed that when the gap in capacity between teacher and student is very large, distillation can be less effective than distilling from a moderately larger intermediate teacher. The student may be unable to represent the teacher's complex functions, and the teacher's outputs may rely on knowledge or reasoning steps the student cannot reproduce, so the student learns an approximation that is worse than what it could have learned from a teacher closer to its own level. A 2020 paper by Seyed Iman Mirzadeh and colleagues proposed "teacher assistants" — intermediate models used as bridges — to address this. In language model practice, the effect appears as students that produce teacher-like reasoning traces confidently but incorrectly, imitating the structure of reasoning they cannot actually perform. The practical consequence is that teacher selection should be an empirical question. For a given student and task, it is worth testing whether a moderately strong teacher produces better student results than the strongest available, and whether the teacher's outputs should be simplified — shorter explanations, fewer steps, less specialised vocabulary — to match what the student can absorb. This connects distillation back to curriculum: the best teaching data is pitched at the learner's level, not the teacher's. Licensing, terms of service, and provenance Distillation from proprietary models raises legal questions that practitioners cannot ignore. The terms of service of major commercial model providers have, at various times, prohibited using their outputs to develop models that compete with the provider's own. The precise wording varies between providers and has changed over time, and the enforceability of such terms is not settled law. But an organisation that builds a commercial product on a student distilled from a proprietary teacher in violation of the teacher's terms exposes itself to contractual risk, and possibly to reputational harm if the practice becomes public. Open-weight models with permissive licences offer an alternative, though their licences too may contain conditions, including attribution requirements or restrictions on use above certain scales. Some open model licences have required that derivative models trained on their outputs carry a particular naming convention or licence. The safest practice is to record, for every synthetic dataset, which model generated it, under what licence and terms, and on what date, and to have that record reviewed before the resulting student is deployed commercially. Chapter 11 returns to provenance as part of governance. There is also a quieter provenance question. A student trained on a teacher's outputs inherits the teacher's biases, errors, and value judgements, along with any content the teacher reproduced from its own training data. If the teacher was trained on copyrighted material and reproduces fragments of it, those fragments may appear in the student's training data. If the teacher has a systematic political or cultural slant, the student will acquire it. Choosing a teacher is choosing a set of inherited properties, and those properties should be examined deliberately rather than discovered after deployment. What distillation actually transfers The evidence across these studies supports a clear account of what distillation through synthetic data can achieve. It transfers style reliably and quickly. A few thousand examples are enough to make a student sound like its teacher. It transfers narrow task competence well, provided the task is within the student's capacity and the teacher's outputs are verified. A small student can approach or even exceed a large teacher on a specific, well-defined task, because the student is specialised while the teacher's capacity is spread across everything. It transfers reasoning patterns partially, especially when the teacher's reasoning is made explicit and filtered for correctness. The gains are largest in domains such as mathematics and code where verification is possible. It does not reliably transfer broad knowledge. A student cannot learn facts its capacity cannot hold, and fine-tuning on facts it did not acquire in pretraining may increase its tendency to hallucinate. And it does not transfer anything the teacher does not have. Distillation is a redistribution of capability, not a creation of it. The only way a student exceeds its teacher on a task is by specialisation, by verification filtering out the teacher's mistakes, or by combining multiple teachers whose strengths differ. Each of those routes runs through the verifier. Chapter 9. Bootstrapping a Small Domain Model The previous chapters described the components of a synthetic data system in isolation: generation, curriculum, verification, filtering, and transfer from teacher to student. This chapter assembles them into a single pipeline for the task that most organisations actually face. They have a specific problem, a modest amount of real data, a budget that does not stretch to running a frontier model on every request, and constraints — privacy, latency, cost, or regulatory — that push toward a small model they control. They want to use a large model to teach a small one. To keep the discussion concrete, the chapter follows a single illustrative case throughout. It is a composite, built from the common shape of such projects rather than from any one organisation's experience, and the quantities in it are illustrative rather than reported results. A mid-sized property insurer receives tens of thousands of emails, letters, and web-form submissions from policyholders each month. Each must be read, classified by type (new claim, update to an existing claim, complaint, policy question, cancellation request, and so on), and have key fields extracted: policy number, date of loss, type of damage, estimated value, whether there are injuries, whether the customer is vulnerable in any of the senses the regulator defines. Human handlers currently do this triage. The insurer wants a model to do a first pass, routing routine items automatically and flagging uncertain or sensitive ones for human review. The correspondence contains personal and sometimes medical information, so the insurer's data protection officer will not permit it to be sent to an external model provider. The model must run on the insurer's own infrastructure. Latency must be low enough to process the inbox continuously. A model of a few billion parameters, fine-tuned for the task, is the target. Stage one: define the task and build the evaluation set first The most common mistake in bootstrapping projects is to start by generating data. The first step is to define precisely what the model must do and to build a test set of real examples against which every later decision will be measured. For the insurer, this means writing a specification: the list of categories with definitions and boundary cases, the list of fields with formats and rules for ambiguous situations, and the policy for when the model should defer to a human. Writing the specification usually reveals disagreements among experienced handlers about how particular cases should be treated. Those disagreements must be resolved before any data is generated, because a synthetic pipeline will faithfully reproduce whatever ambiguity the specification contains. Next, a sample of real correspondence is drawn — a few hundred items is often enough to start, chosen to cover the range of categories, channels, and difficulties — and labelled carefully by experienced handlers according to the specification. Items on which two handlers disagree are discussed and resolved. This labelled sample is divided in two. One part, perhaps a fifth, is set aside as a seed set that can inform generation. The rest becomes the held-out evaluation set, which is never shown to the generator, never used for training, and consulted only to measure progress. The evaluation set is the anchor for the entire project. Every claim that a synthetic dataset is good, that a filter helps, or that a student has learned, is a claim about performance on this set. If it is too small, the measurements will be noisy. If it is unrepresentative — drawn only from easy cases or from a single month — the measurements will mislead. If it leaks into training, the measurements will be worthless. It should be stored separately from everything else, with access controlled, and its composition documented. Because the evaluation set is drawn from real, sensitive correspondence, it must be handled under the insurer's normal data protection controls. Using real data for evaluation, within the organisation's secure environment, is generally far less problematic than using it to train a model that may be widely deployed, and it is essential. Synthetic data cannot be validated against synthetic data. Stage two: establish baselines Before any synthetic data is created, two baselines are measured on the evaluation set. The first is the teacher's performance. A capable large model — run within the insurer's environment if an open-weight model is used, or on carefully de-identified examples if a hosted model is permitted for development — is prompted with the specification and a handful of examples and asked to perform the task on each item in the evaluation set. Its accuracy on classification and extraction sets a realistic upper bound for what the student can learn from it. If the teacher itself performs poorly, distillation will not rescue the project, and effort is better spent improving the specification, the teacher prompt, or the choice of teacher. The second baseline is the untrained student: the small base model, prompted the same way, without fine-tuning. This establishes how much the fine-tuning must add. It also reveals the student's specific weaknesses — perhaps it handles classification reasonably but fails badly at extracting dates from free text — which will shape the curriculum. For the insurer, suppose the teacher classifies most items correctly and extracts most fields accurately but struggles with letters that discuss multiple claims and with recognising indicators of customer vulnerability. The base student does considerably worse across the board. The gap between them is what distillation is meant to close, and the teacher's weak spots are where its outputs will need the closest verification. Stage three: generate inputs before outputs A synthetic dataset for this task needs two things: realistic input documents and correct outputs for them. It is tempting to generate both at once — asking the teacher to write a customer email and its correct classification together — but separating them is more robust. Synthetic inputs are generated using the techniques of Chapter 4. An attribute grid is derived from the specification and from close reading of the seed set: claim type, peril (flood, fire, theft, storm, escape of water, subsidence), channel, length, writing proficiency, emotional state, presence of multiple issues, presence of vulnerability indicators, presence of extraneous information, and whether the customer has made a factual error such as quoting the wrong policy number. Personas add realistic variation: an elderly widower writing his first claim, a landlord with several properties, a small business owner whose premises have flooded, a customer whose first language is not English. Real seed examples, with identifying details removed, are included in prompts as style references, so that the synthetic letters capture the texture of real correspondence — the spelling errors, the digressions, the forwarded email chains, the scanned form text with recognition errors. Each synthetic input is generated with its intended attributes recorded. Those attributes are not yet the ground-truth labels. They are what the generator was asked to produce, which may differ from what it actually produced. A letter requested as "a complaint with a vulnerability indicator" may, on reading, contain no clear indicator. Stage four: label with the teacher, verify independently Synthetic inputs are then labelled by the teacher, using the same prompt that produced the teacher baseline. For each input, the teacher outputs a classification, extracted fields, and a confidence assessment. Verification compares the teacher's labels against several independent signals. Consistency with generation attributes: if the input was generated as a flood claim and the teacher labels it a fire claim, either the generator or the teacher erred. Disagreements are flagged. Self-consistency of the teacher: the teacher labels each input several times with sampling enabled. Items where its outputs vary are flagged as uncertain. Rule-based field checks: policy numbers must match the insurer's format; dates must be valid and not in the future; monetary values must be plausible for the claim type. Extracted fields that fail these rules are flagged. Cross-model agreement: a second large model from a different family labels the same inputs. Disagreements are flagged. A judge with a rubric: for the most consequential fields — vulnerability indicators, injury mentions — a judge model is given the specification's definitions and asked whether each label is justified by specific text in the input, citing the text. Items that pass all checks are accepted with their labels. Items that fail one or more are either discarded or routed for human review. Human review is expensive, so it is concentrated where it matters most: on the categories where the teacher was weakest at baseline, and on vulnerability and injury flags, where errors carry regulatory consequences. A sample of accepted items, perhaps a few hundred, is also reviewed by handlers to estimate the residual error rate, as described in Chapter 7. If that rate is too high in some category, the verification for that category is tightened, and the loop runs again. Stage five: shape the curriculum The verified dataset is not used as a single undifferentiated pile. It is organised to match the student's learning needs. Difficulty is estimated empirically, by running the base student on each verified item and recording whether it gets the classification and fields right. Items the base student already handles are kept in modest proportion, to prevent it forgetting them, but are not emphasised. Items it fails are the core of the training set. Items that even the teacher found hard — those that required human review to label — are especially valuable and may be augmented with further variations generated around the same attributes. Rare categories are deliberately oversampled. The insurer's real inbox may contain cancellation requests from bereaved relatives only rarely, but the model must handle them sensitively, so the synthetic set includes many of them. The oversampling is recorded, because it means the student's prior over categories will differ from reality, and its confidence scores may need recalibrating on real data before deployment. Unanswerable and deferral examples are included. Some synthetic inputs are generated to be genuinely ambiguous — a letter that might be a complaint or a new claim, with insufficient information to tell — and their correct label is a deferral to human review. Some are generated with missing fields, and the correct output is to mark those fields as absent rather than inventing them. These examples teach the student when not to answer, which is essential for a triage system whose value depends on routing uncertain cases to humans. Stage six: train, evaluate, and iterate The student is fine-tuned on the curriculum and evaluated on the held-out real set. The evaluation reports overall accuracy, per-category accuracy, per-field extraction accuracy, the rate of appropriate deferrals, and — critically — the rate of confident errors, where the student gives a wrong answer without deferring. For a triage system, a confident error is the most costly outcome, because it bypasses human review. The first round almost always reveals specific failures. Perhaps the student handles letters well but performs poorly on web-form submissions, whose terse, structured format was under-represented in the synthetic inputs. Perhaps it extracts dates accurately when they are written in full but fails on informal references such as "last Tuesday." Perhaps it defers too often on complaints, because the synthetic complaints were all ambiguous in the same way. Each failure is traced to its source. Was the relevant kind of input missing from the synthetic data? Was it present but mislabelled by the teacher and missed by verification? Was it present and correctly labelled but too rare for the student to learn? The answer determines the fix: generate more of the missing kind, tighten verification on the mislabelled kind, or upweight the rare kind. A new round of generation, verification, and training follows. Experienced practitioners find that this loop, run several times, accounts for most of the improvement. The first synthetic dataset is rarely good. The fourth, shaped by three rounds of diagnosis against real data, often is. Stage seven: monitor in deployment A deployed student encounters a distribution that shifts over time. A major storm produces a surge of correspondence about a single peril. A new product line introduces unfamiliar terminology. A change in the claims process alters what customers write about. A synthetic dataset frozen at training time cannot anticipate any of these. Deployment monitoring therefore extends the verification architecture into production. A sample of the student's live decisions is reviewed by handlers regularly. Items the student deferred are labelled by handlers in the normal course of their work, and those labels are collected. The student's confidence distribution and deferral rate are tracked, because sudden changes signal distributional shift. When enough new real examples accumulate, they are added to the evaluation set, and when the evaluation shows degradation, the synthetic pipeline is run again with the new examples as seeds. Over time, the balance between synthetic and real data in the system shifts. Early on, the student is trained mostly on synthetic data, because real labelled data is scarce. As deployment generates real labels through human review of deferred and sampled items, real data becomes more plentiful, and the synthetic data's role narrows to covering rare cases and new situations that real data has not yet reached. This is the healthy trajectory for a bootstrapped model: synthetic data as scaffolding that is gradually replaced by the real structure it was built to support, rather than a permanent substitute for it. Common failure patterns Several failure patterns recur across bootstrapping projects regardless of domain. The synthetic inputs are too clean. Generators produce well-written, well-organised text unless pushed hard not to. Real correspondence is messy. A student trained on clean synthetic inputs fails on messy real ones. The remedy is to use real seed examples as style anchors, to generate deliberate noise such as typos, recognition errors, and forwarded chains, and to check synthetic inputs against real ones for stylistic coverage using the embedding-cluster method of Chapter 4. The teacher's errors are correlated with the student's needs. Teachers make the most mistakes on the hardest cases, which are exactly the cases the student most needs to learn. Verification that is uniform across the dataset will let disproportionate errors through on hard cases. Verification effort must be concentrated where the teacher is weakest. The evaluation set is too easy. If the held-out real set was sampled uniformly, it is dominated by routine items, and overall accuracy will look good even if the student fails on every rare and important case. Stratified evaluation, with explicit sub-scores for rare and high-stakes categories, is essential. Synthetic and real data are evaluated together. If some synthetic examples are inadvertently included in the evaluation set, or if the evaluation set was used to tune generation prompts too directly, scores will be inflated. The evaluation set must be protected from every stage of the pipeline that could adapt to it. The specification drifts. As the project proceeds, handlers refine their understanding of edge cases, but the refinements are not written back into the specification, and different stages of the pipeline use different versions of it. Version control for the specification, as for code, prevents this. The economics of the bootstrap The economic logic of the approach rests on a simple asymmetry. Generating and verifying synthetic data costs money once, during development. Running the teacher on every production request costs money continuously. If the student can match the teacher's performance on the task, the up-front investment in synthetic data is repaid by the lower cost, lower latency, and greater control of running the student. The investment is not trivial. The largest single cost is usually expert time: writing the specification, labelling the evaluation set, reviewing flagged items, and diagnosing failures. Model inference for generation and verification is typically much cheaper than that expertise. This reinforces a theme from Chapter 1. In a synthetic data project, human expertise is most valuable not at the point of producing individual labels but at the points where it shapes the verifier: the specification that defines correctness, the evaluation set that measures it, and the review that calibrates the automated checks. The generator is the cheap part. Chapter 10. Recursion, Collapse, and the Polluted Commons Every technique in this book involves a model learning from the outputs of another model. In the simplest cases, the chain has one link: a teacher generates, a student learns, and the student is deployed. But chains grow. The student becomes the teacher for the next generation. A model generates data that trains an improved version of itself, which generates better data, which trains a further improved version. And beyond any single organisation's pipeline, the open web — the source of most pretraining data — is filling with text written by language models, which future models will scrape and train on whether their developers intend it or not. What happens to models trained on the outputs of models, generation after generation? The question has been studied intensively since 2023, and the answer has become clearer, if not entirely settled. Uncontrolled recursion degrades models in a specific and predictable way, starting at the edges of the distribution and working inward. But the degradation is not inevitable. It depends on what happens to real data in the loop, and on whether something outside the generator decides which synthetic data survives. The findings are a sharp confirmation of the argument that verification, not generation, is the architecture. The collapse result The study that brought the problem to wide attention was published in Nature in July 2024 by Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal, under the title "AI models collapse when trained on recursively generated data." An earlier version had circulated in 2023 under the memorable title "The Curse of Recursion." The authors examined several types of model — Gaussian mixture models, variational autoencoders, and a small language model fine-tuned on a Wikipedia-derived text corpus — in a setting where each generation was trained only on data generated by the previous generation. In every case, they observed a characteristic pattern of degradation they called model collapse. In the early stage, the model loses information about the tails of the distribution: rare events, unusual words, minority patterns. This happens because each generation's training data is a finite sample from the previous model, and finite samples under-represent low-probability regions. Events that the previous model assigned small but non-zero probability may simply not appear in the sample, and the next model, never seeing them, assigns them even lower probability. Over generations, the tails thin and vanish. In the late stage, the distribution converges toward something narrow that bears little resemblance to the original. Variance shrinks. The model produces an increasingly limited range of outputs, and errors in the approximation at each step compound into distortions. In the language model experiments, later generations produced text that drifted toward repetitive phrases and, eventually, sequences unrelated to the original prompts. The authors identified three sources of error that compound across generations. Statistical approximation error arises because each generation sees only a finite sample. Functional expressivity error arises because the model cannot perfectly represent the true distribution. Functional approximation error arises because training procedures do not find the best possible model even within what is expressible. Each source alone would cause drift; together, applied recursively without correction, they drive collapse. Model autophagy A complementary study by Sina Alemohammad and colleagues at Rice University and Stanford, first circulated in 2023 and presented at the International Conference on Learning Representations in 2024, examined the same phenomenon in image generators under the name "Model Autophagy Disorder," borrowing the analogy of mad cow disease, which spread when cattle were fed the remains of other cattle. Their contribution was to distinguish several kinds of self-consuming loop. In a fully synthetic loop, each generation trains only on outputs of the previous one. In a synthetic augmentation loop, each generation trains on a fixed set of real data plus synthetic data from the previous generation. In a fresh data loop, each generation trains on synthetic data plus a new sample of real data that it has not seen before. They found that fully synthetic loops degraded consistently. Synthetic augmentation loops with a fixed real dataset delayed degradation but did not prevent it. Only fresh data loops, with sufficient new real data in each generation, avoided progressive decline. They also identified the effect of sampling bias. When generators are sampled in a way that favours higher-quality outputs — lower temperature, truncation, or cherry-picking — each generation's output looks better, but diversity falls faster. Quality and diversity trade against each other, and a loop that optimises for apparent quality accelerates the loss of variety. The accumulation correction A 2024 paper by Matthias Gerstgrasser and colleagues, titled "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data," pointed out that the experimental setup in many collapse studies made a specific assumption: that each generation's data replaced the previous generation's. In practice, data is rarely thrown away. Real-world datasets grow by accumulation. The original human data remains available, and synthetic data from each generation is added to it rather than substituting for it. Testing this alternative setup with language models, diffusion models, and other generative models, they found that when data accumulates — so that each generation trains on the original real data plus all synthetic data produced so far — test error remains bounded rather than growing without limit. They supported this with a theoretical analysis of a simplified linear model, showing that under replacement error grows linearly with the number of generations, while under accumulation it is bounded by a constant. The intuition is that the original real data continues to anchor every generation, and each new batch of synthetic data is a diminishing fraction of the whole. This finding reframed the collapse debate. Collapse is not an inevitable consequence of training on synthetic data. It is a consequence of losing contact with real data. As long as real data remains in the training mixture, and synthetic data supplements rather than displaces it, recursive training appears to be stable, at least in the settings tested. Verification as the brake A further line of work asked whether selection could do what accumulation does. In a 2024 paper presented at the International Conference on Learning Representations in 2025, Yunzhen Feng, Elvis Dohmatob, and colleagues studied the effect of verification on recursive training under the title "Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification." Their question was whether filtering synthetic data through a verifier, even an imperfect one, could prevent the degradation seen in unfiltered loops. They found that it could. When synthetic data was selected by a verifier that preferred correct or higher-quality examples, models trained on it could avoid the collapse that unfiltered training produced. They analysed the effect theoretically in an idealised setting with simple classifiers and verifiers, and tested it in experiments in which transformers learned to compute matrix eigenvalues and language models learned to summarise news, proposing a measurable proxy for verifier quality that tracked downstream performance. The key insight is that the verifier injects information from outside the generator's distribution. Each round of selection corrects some of the errors that would otherwise compound, functioning as a source of external signal much as fresh real data does. This finding unifies the collapse literature with the book's central argument. Recursive training degrades because each generation's errors are passed to the next without correction. Anything that injects correction — fresh real data, accumulated original data, or a verifier that encodes knowledge the generator lacks — can halt or reverse the degradation. What cannot halt it is more generation. Table 5 summarises the main studies and their implications. Table 5. Key studies on recursive training with synthetic data. Study Setting Main finding Practical implication Shumailov et al., Nature, 2024 Each generation trained only on previous generation's output Tails vanish first; distribution narrows and distorts Pure recursion without real data is harmful Alemohammad et al., ICLR 2024 Fully synthetic, fixed-real, and fresh-real loops for image models Only sufficient fresh real data prevents decline; quality-biased sampling speeds diversity loss Keep injecting new real data; watch diversity Gerstgrasser et al., 2024 Accumulating rather than replacing data across generations Error stays bounded when original data is retained Never discard the real anchor Feng et al., ICLR 2025 Synthetic data filtered by a verifier before retraining Verification, even imperfect, can prevent collapse Selection quality governs recursive gains What happens to the tails The detail that collapse begins in the tails deserves emphasis, because it has consequences long before any dramatic degradation is visible. A model trained even partly on synthetic data, without care, may perform well on average while quietly losing competence on rare cases: uncommon languages, minority dialects, unusual medical presentations, rare legal situations, niche technical topics. Average benchmark scores will not reveal this, because averages are dominated by common cases. A related theoretical analysis by Elvis Dohmatob and colleagues in 2024, "A Tale of Tails," modelled the effect as a change in scaling laws. Training on synthetic data that has lost its tails means that adding more data stops improving performance at the rate expected, because the additional data does not contain the rare patterns that further learning would require. The model plateaus, and no quantity of tail-less synthetic data will move it. For practitioners, the implication is that evaluation must deliberately probe the tails. Stratified evaluation sets, with explicit coverage of rare categories, are the only reliable way to detect tail loss before it reaches deployment. This connects directly to the evaluation design of Chapter 9 and the governance practices of Chapter 11. It also connects to fairness. The groups and situations that are rare in a dataset — linguistic minorities, people with uncommon conditions, small communities — are precisely those whose representation is first to erode under recursive training. A synthetic pipeline that is careless about tails is, in effect, a mechanism for progressively forgetting minorities. The polluted commons The collapse studies concern deliberate, controlled loops. A larger and less controllable loop is forming in the open web. Since the public release of capable chat models at the end of 2022, the volume of machine-generated text online has grown rapidly: in marketing copy, product descriptions, search-optimised articles, social media posts, forum answers, and translations. Some is labelled as machine-generated; most is not. Future web crawls will contain a growing share of this material, and models trained on them will be training partly on outputs of earlier models whether or not their developers intend it. The concern is not hypothetical. Researchers have noted that some question-answering and crowdsourcing platforms saw growing proportions of model-written contributions after 2022, and that crowd workers asked to perform annotation tasks sometimes used language models to complete them, contaminating datasets meant to capture human judgement. Several responses are emerging. One is to place greater value on data known to predate widespread generative models. Text collected before late 2022 has been compared, in a widely used analogy, to the "low-background steel" salvaged from ships sunk before atmospheric nuclear testing, valued because it is free of radioactive contamination. Archives of older web crawls, books, and other pre-generative-era sources may become more valuable as a result. A second is detection. Classifiers that attempt to distinguish human from machine text exist, but their reliability is limited, they can be evaded by paraphrasing, and they have been shown to misclassify text by non-native English writers as machine-generated at elevated rates. Detection is a useful signal in aggregate but an unreliable filter for individual documents. A third is provenance at the point of creation. Watermarking schemes embed statistical signals in generated text that can later be detected by someone who knows the key. Google DeepMind has deployed a text watermarking system called SynthID for some of its products, and has described the method publicly. Content provenance standards such as those developed by the Coalition for Content Provenance and Authenticity attach signed metadata to media recording how it was produced. These approaches work only for content from cooperating providers and can be stripped or degraded, but they offer a better foundation than post-hoc detection. A fourth is to treat the web as just another unverified generator. If future web text is a mixture of human and machine content of unknown proportions, then it should be subjected to the same filtering and verification as deliberately generated synthetic data: quality classifiers, deduplication, grounding checks where possible, and evaluation against trusted real data. This reframes the polluted commons not as a new problem but as an extension of the old one. Living with recursion The evidence supports a set of principles for any system in which models learn from model outputs. Never discard the real anchor. Original human data should remain in the training mixture across generations, accumulated rather than replaced. If it must be reduced in proportion, it should not be reduced to zero. Inject fresh real data where possible. New samples of real data, collected after each generation, provide the strongest protection against drift. Verify before reuse. Synthetic data should pass through a verifier before it trains the next generation, and the verifier should encode knowledge the generator lacks. Unverified recursive generation is the configuration in which collapse is best documented. Monitor diversity explicitly. Average quality can rise while diversity falls. Track diversity metrics across generations and treat a decline as a warning, even when headline benchmarks improve. Probe the tails. Evaluate on stratified sets that cover rare cases, and treat degradation on those cases as the leading indicator of collapse. Record provenance. Know which data is real, which is synthetic, and which generation of which model produced it. Without this, none of the other principles can be applied. The lesson of the collapse literature is not that synthetic data is dangerous. It is that synthetic data without an external anchor is dangerous, and that the anchor can take the form of real data, a verifier, or both. Every system that has used synthetic data successfully at scale has had one. The ones that degrade are the ones that forgot it was needed. Chapter 11. Validation and Governance A synthetic dataset is a manufactured artefact, and like any manufactured artefact used in a consequential system, it needs quality assurance, documentation, and accountability. The previous chapters described how to generate and filter synthetic data. This chapter addresses the questions that come after: how to establish that a synthetic dataset actually does what it was meant to do, how to record what was done so that others can understand and reproduce it, and how to govern the pipeline so that its outputs can be trusted over time. These questions are often treated as administrative overhead, attended to after the technical work is done. That is a mistake. Validation determines whether the technical work succeeded, and documentation determines whether anyone can tell. A synthetic data pipeline without rigorous validation is an untested hypothesis. One without documentation is a black box that cannot be audited, debugged, or safely modified. Three levels of validation Validation of a synthetic dataset operates at three levels, each answering a different question. The first level is intrinsic: does the synthetic data have the properties it was designed to have? This includes the checks applied during filtering — correctness, grounding, toxicity, format — but measured on the final dataset rather than applied to individual candidates. It also includes distributional properties: whether the composition across attributes matches the target, whether diversity metrics are within expected ranges, whether duplicates and benchmark contamination have been removed. Intrinsic validation is necessary but not sufficient. A dataset can have every intended property and still fail to train a good model. The second level is comparative: how does the synthetic data relate to the real data it supplements or replaces? For privacy-motivated synthesis, this means the fidelity and privacy measurements described in Chapter 3. For training data, it means comparing the synthetic distribution with real data along the dimensions that matter: topic coverage, style, difficulty, length, the presence of rare cases. The embedding-and-cluster method from Chapter 4 is one tool. Another is to train a classifier to distinguish synthetic from real examples. If the classifier succeeds easily, the features it relies on reveal how the synthetic data differs, and those differences may be ones the student will learn. If the classifier cannot distinguish them, that is reassuring, though not conclusive, since the classifier may not be sensitive to the differences that matter for the downstream task. The third and decisive level is extrinsic: does a model trained on the synthetic data perform well on the real task? This is measured on a held-out real evaluation set, as described in Chapter 9. No intrinsic or comparative measure can substitute for it, because the purpose of training data is to produce a capable model, and the only direct test of capability is performance on real inputs. Ablation: measuring what each component contributes Extrinsic validation tells you whether the final dataset works. It does not tell you why, or which parts of the pipeline are responsible. For that, ablation is needed: training models with and without particular components and comparing their performance. The most basic ablation compares a model trained on synthetic data with one trained on the same quantity of real data, if real data is available in sufficient quantity, and with one trained on real data alone at whatever quantity exists. This establishes whether the synthetic data helps at all, and by how much. Further ablations isolate the effect of each pipeline stage. Training on unfiltered versus filtered synthetic data measures the value of the filter. Training with and without a particular verifier measures that verifier's contribution. Training on synthetic data generated with and without attribute grids or personas measures the value of diversity engineering. Training on curriculum-ordered versus randomly ordered data measures the curriculum effect. Each ablation costs a training run, so small models and limited budgets are often used for ablations, on the assumption that relative effects observed at small scale will hold at larger scale. That assumption usually holds approximately but should be checked at least once at the target scale. Ablations frequently produce surprises. A filter thought to be essential turns out to make no difference. A diversity technique turns out to hurt, because it introduced more errors than variety. Removing half the synthetic data turns out to improve performance, because the removed half was lower quality. Without ablations, these facts remain unknown, and the pipeline accumulates expensive components whose value is assumed rather than demonstrated. A related practice is data attribution: estimating how much each training example, or each subset, contributes to the model's performance on particular evaluation items. Techniques such as influence functions and data models attempt this, though they remain expensive and approximate at the scale of modern language models. Simpler approaches — training on subsets defined by generation attributes and comparing results — give much of the practical value at lower cost, and they are made possible by the attribute metadata recommended in Chapter 4. Evaluation that cannot be gamed Because synthetic data is generated by models that may have seen public benchmarks, and because synthetic pipelines can be tuned, consciously or not, toward whatever the evaluation measures, the integrity of evaluation needs special protection. Several practices help. Private held-out sets, never published and never used for any purpose other than final evaluation, are the strongest protection. Their composition should be documented internally, but their contents should be accessible only to the evaluation process. Benchmarks created after the generator's training cutoff reduce the chance that the generator has seen them. Decontamination, as described in Chapter 7, removes overlaps between synthetic training data and evaluation sets. And evaluating on several independent sets, rather than one, makes it harder for a pipeline to overfit to the quirks of any single set. The style-versus-substance problem from Chapter 8 also bears on evaluation. Evaluations judged by humans or by LLM judges are susceptible to rewarding a student that has learned to sound like its teacher without having learned its capabilities. Evaluations with verifiable answers — task accuracy on real inputs, extraction correctness, code that passes hidden tests — are much harder to fool. A synthetic data project should report results on at least one evaluation of this kind, and should treat judge-based scores as supplementary rather than primary. Documenting synthetic data The machine learning community has developed several documentation standards for datasets and models, and they apply to synthetic data with some extensions. The 2018 proposal "Datasheets for Datasets" by Timnit Gebru and colleagues set out a series of questions that a dataset's creators should answer: why it was created, what it contains, how it was collected, how it was preprocessed, what it should and should not be used for, how it will be maintained. "Model Cards for Model Reporting," proposed by Margaret Mitchell and colleagues in 2019, did the same for trained models, emphasising evaluation across subgroups and intended uses. For synthetic data, a datasheet should additionally record the full generation lineage: which generator model or models were used, with versions; the prompts, templates, and attribute grids; the seed data, if any, and its source; sampling parameters; every filtering and verification stage, with thresholds; the acceptance rate at each stage; the residual error rate estimated by human review, with the sample size and method; and the dates of generation. It should record the licence and terms under which each generator was used. It should record which portion of the dataset, if any, consists of real data, and how real and synthetic portions were combined. And it should record known limitations: categories where the generator was weak, attributes that were deliberately over- or under-sampled, failure modes that the filters are known not to catch. This record serves several purposes. It allows the dataset to be reproduced, or at least approximated, if needed. It allows downstream users to judge whether the dataset is appropriate for their purposes. It allows auditors and regulators to assess compliance. And it allows the creators themselves, months later, to diagnose problems that emerge in deployment by tracing them back to specific generation choices. Lineage and provenance tracking Documentation at the dataset level is necessary but not sufficient for complex pipelines. When synthetic data from several generators is combined, when filtered data from one project is reused in another, and when models trained on synthetic data are used to generate further synthetic data, the lineage becomes a graph rather than a list. Tracking it requires recording provenance at the level of individual examples, or at least of batches. A practical approach attaches metadata to every synthetic example: a unique identifier, the generator and its version, the prompt template and attribute values, the verification results at each stage, and the identifier of the batch it belongs to. This metadata travels with the example through every subsequent use. When a model is trained, the training run records which batches it consumed. When that model generates data, its outputs carry a reference to the training run. The result is a provenance graph in which any example, or any model, can be traced back to its origins. Example-level provenance enables several capabilities that would otherwise be impossible. If a generator is later found to have a systematic flaw — a factual error it consistently makes, a bias, a licensing problem — every example it produced can be located and removed or re-verified. If a deployed model exhibits a harmful behaviour, the training examples most likely to have caused it can be identified and examined. If a regulator asks what proportion of a model's training data was synthetic, or which models generated it, the question can be answered precisely. And the anti-collapse principles of Chapter 10 — retaining real data, monitoring generations — require knowing which data is real and which generation produced each synthetic example. Regulatory context Regulation of AI training data is developing rapidly, and synthetic data occupies an ambiguous position within it. The European Union's AI Act requires providers of high-risk AI systems to apply data governance practices covering design choices, data collection processes and the origin of data, preparation operations such as labelling and cleaning, examination for possible biases, and identification of data gaps. These requirements apply to synthetic data as much as to collected data, and a synthetic pipeline's documentation should be designed to satisfy them. The Act also requires providers of general-purpose AI models to publish a sufficiently detailed summary of the content used for training, and the European Commission published a template for such summaries in 2025. Data protection law applies to synthetic data to the extent that it is personal data, which, as Chapter 3 discussed, depends on whether individuals can be identified from it. A synthetic dataset derived from personal data should be accompanied by an assessment of re-identification risk, ideally with formal privacy guarantees and empirical attack testing, documented in a form that a data protection authority could review. Sector-specific regulators add further requirements. In financial services, model risk management guidance, such as the United States Federal Reserve's SR 11-7 and comparable guidance elsewhere, requires validation of models and their data, including independent review. In healthcare, regulators assessing AI-based medical devices expect evidence about the training data's representativeness and quality. In each case, synthetic data does not exempt a model from scrutiny; it adds questions about how the synthetic data was validated. Organisational practices Beyond documentation and regulation, several organisational practices distinguish teams that use synthetic data well from those that do not. Separate generation from evaluation. The people or processes that generate and filter synthetic data should not be the only ones that evaluate the resulting models. Independent evaluation, by a team with its own held-out data and no stake in the pipeline's success, catches problems that the pipeline's builders are motivated to overlook. This mirrors the separation between development and validation in mature model risk management. Read the data. Automated metrics are indispensable at scale, but they measure only what their designers anticipated. Regular, structured human review of random samples of the synthetic data — not just the flagged examples, and not just the rejected ones — is the most reliable way to discover failure modes nobody thought to measure. Reviews should be scheduled, recorded, and acted on. Version everything. Prompts, templates, attribute grids, filter thresholds, verifier models, specifications, and evaluation sets should all be under version control, like code. A change to any of them can change the resulting dataset and model, and without versioning, the effect of changes cannot be traced. Budget for verification. Teams consistently underestimate the share of effort that verification requires. A reasonable starting assumption is that designing, calibrating, and maintaining verifiers, and reviewing their output, will take more effort than generation. Projects that budget otherwise tend to cut verification when time runs short, with predictable results. Plan for retirement. Synthetic datasets age. The generator that produced them is superseded by better models. The domain shifts. Errors are discovered. A dataset should have an owner responsible for maintaining it, a schedule for re-evaluation, and criteria for retiring or regenerating it. The audit question A useful test of a synthetic data pipeline's governance is to imagine an external auditor asking a series of questions. What proportion of the training data was synthetic? Which models generated it, under what terms? What verifiers were applied, and how were they calibrated? What is the estimated residual error rate, and how was it measured? What real data was the synthetic data validated against, and how was that real data protected from contamination? If a generator flaw were discovered tomorrow, how quickly could every affected example be located? What subgroups or rare categories were evaluated separately, and how did the model perform on them? A team that can answer all of these questions with evidence has a governed pipeline. A team that cannot has a pipeline whose outputs are trusted on faith. Given what the earlier chapters have shown about how synthetic data fails — through hallucinations that look correct, biases invisible at the level of individual examples, tails that vanish silently, style that masquerades as capability — faith is not a sufficient basis for deploying models in consequential settings. Validation and governance are the means by which the verifier's work, done once during construction, remains visible and checkable for as long as the resulting model is in use. Conclusion: Where the Knowledge Comes From Across eleven chapters, one question has recurred in different forms: when a model learns from synthetic data, where does the knowledge come from? The answer, assembled from the evidence, is more precise than the enthusiasm for synthetic data usually allows, and more encouraging than the warnings about collapse usually suggest. Some of the knowledge comes from the generator, but only knowledge the generator already had. A teacher model that knows how to classify insurance correspondence can pass that competence to a student, and a model that has absorbed the content of the web can rewrite it as clear textbook explanations that a small model learns from more efficiently. This is real value. It is redistribution and restructuring, not creation, and it is bounded by what the generator knows and by what the student has the capacity to absorb. The rest of the knowledge comes from outside the generator. It comes from the unit test that rejects a wrong program, the proof checker that rejects an invalid step, the symbolic engine that deduces geometric facts, the physics simulator that computes how a robot hand will move, the source document against which a summary's claims are checked, the human expert who wrote the specification and labelled the evaluation set, and the real data that anchors each generation against drift. Every striking success in synthetic data — AlphaGeometry's olympiad proofs, the reasoning gains of distilled models, the coherent stories of very small language models trained on carefully constrained corpora — rests on one of these external sources. Every documented failure — imitation models that sound capable and are not, recursive loops that forget the tails, privacy-motivated datasets that leak outliers — involves its absence. That is why this book has insisted that the verifier is the architecture. It is not a slogan about the relative importance of pipeline stages. It is a claim about epistemics. A generator, by itself, can only reshape what it already contains. Any genuinely new information that enters a synthetic dataset enters through whatever decides which candidates to keep. What follows for practice Several practical conclusions follow that could not have been drawn without the preceding detail. First, the choice of domain matters more than the choice of generator. Synthetic data delivers its largest and most reliable gains where strong verifiers exist: mathematics with checkable answers, code with executable tests, formal reasoning with proof checkers, structured extraction with validation rules. Where verification is weak — open-ended advice, nuanced judgement, specialised factual knowledge outside any trusted source — synthetic data delivers smaller and less reliable gains, and the residual error rate must be measured with special care. Before starting a synthetic data project, it is worth asking what the verifier will be. If there is no good answer, the project is likely to produce a model that sounds better than it is. Second, effort should be allocated in proportion to where knowledge enters the system. Teams naturally invest in generation, because it is visible and productive. The evidence suggests investing more in verifiers, evaluation sets, and human review of samples, because those are where the system's correctness is determined. A domain expert's time is better spent designing a rubric or labelling a stratified evaluation set than writing seed examples. Third, synthetic data should be designed together with its real-data anchor, not in place of it. The most robust pipelines treat real data as scarce and precious: used to define the evaluation set, to seed and style-anchor generation, to calibrate verifiers, and to remain in every training mixture. Synthetic data multiplies the value of real data. It does not remove the need for it. Fourth, privacy claims must be formal or not made. The label "synthetic" confers no protection by itself. Where synthetic data is used to share or train on sensitive information, differential privacy with a stated and defensible budget, combined with empirical attack testing, is the minimum basis for a claim that individuals are protected. Where that is too costly for the fidelity required, controlled access to real data may be the more honest solution. Fifth, evaluation must look past averages and past style. The failure modes of synthetic data concentrate in rare cases, which averages hide, and in capabilities that style imitates, which judges miss. Stratified evaluation with verifiable answers is the minimum standard for knowing what a model trained on synthetic data can actually do. What remains unsettled Several important questions remain open, and honest practice should acknowledge them. It is not known how far verified synthetic data can extend capability beyond the frontier of human knowledge. In formal domains, systems trained on self-generated, verified data have reached the level of skilled humans, and the logic of verification suggests there is no hard ceiling where verification is exact. Whether similar progress is possible in domains where verification is approximate — scientific reasoning, strategic judgement, open-ended writing — depends on whether approximate verifiers can be made reliable enough to avoid the over-optimisation that Chapter 6 described. The evidence so far is mixed. It is not known precisely how much synthetic data can safely replace real data in general-purpose pretraining. The accumulation results suggest that recursive training is stable when real data is retained, but they were obtained at scales and in settings much smaller than frontier training, and the proportion of machine-generated content on the web is rising in ways that no controlled experiment has replicated. It is not settled how to measure diversity in a way that predicts downstream performance. The metrics in common use catch gross failures but disagree with one another on subtler ones, and none directly measures whether a dataset covers the situations a model will meet in deployment. And it is not settled how the law will treat models distilled from other models, synthetic data derived from personal data, or synthetic data generated by models trained on copyrighted works. These questions are being worked out in courts, regulatory guidance, and licence terms, and practitioners should expect the rules to change. The discipline ahead Synthetic data began as a workaround — a way to get training data when the real thing was scarce, private, or rare. It is becoming something more like a discipline, with its own methods, failure modes, and standards of evidence. The maturation of that discipline will depend less on better generators, which will arrive regardless, than on better verifiers, better evaluation practice, and better habits of documentation. The most important habit is the simplest. For any synthetic dataset, be able to say where its knowledge came from: what the generator contributed, what the verifier contributed, what the real data contributed, and how each contribution was measured. A team that can answer that question has built a synthetic data system. A team that cannot has built a machine for producing plausible text and hoping some of it is true. Glossary Ablation. An experiment that removes or alters one component of a pipeline and measures the effect on the resulting model, used to establish what each component contributes. Attribute grid. An explicit list of the dimensions along which examples should vary, with values for each, used to condition a generator so that the synthetic dataset covers deliberately chosen combinations. Curriculum. A deliberate ordering or selection of training examples by difficulty or type, intended to match the data to what the learner can currently absorb. Decontamination. The removal from a training set of examples that overlap with evaluation benchmarks, so that benchmark scores measure capability rather than memorisation. Differential privacy. A formal guarantee that an algorithm's output changes very little when any single individual's record is added or removed, with the strength of protection set by a parameter called epsilon. Distillation. Training a smaller student model to reproduce the behaviour of a larger teacher, either by matching its output probabilities or, more commonly for language models, by training on text the teacher generated. Domain randomisation. A simulation technique that varies textures, lighting, physics, and other parameters widely so that a model trained in simulation treats the real world as one more variation. Exposure bias. The mismatch that arises when a model is trained only on ideal outputs but must generate from its own, imperfect outputs at deployment, never having learned to recover from its own mistakes. Grounding. The property that every claim in a generated text is supported by a specified source; a grounding failure is the addition of unsupported material. Hallucination. Fluent, confident generated content that is false or unsupported. LLM-as-judge. The use of a large language model, usually given a rubric, to evaluate the outputs of another model. Membership inference. An attack that tries to determine whether a particular individual's record was used to train a model or generator. Model collapse. The progressive degradation of models trained recursively on the outputs of earlier models, beginning with the loss of rare patterns and ending in a narrow, distorted distribution. Outcome verification. Checking only the final result of a solution, such as whether an answer is correct; contrasted with process verification, which checks each intermediate step. Over-optimisation. The degradation that occurs when outputs are selected or trained to maximise an imperfect proxy score, such that the proxy rises while true quality falls. Persona conditioning. Prompting a generator with a description of a person, so that its outputs reflect the perspective, vocabulary, and concerns that person would plausibly have. Reward model. A learned scoring function, usually trained on human comparisons between responses, that estimates how much a response would be preferred. Residual error rate. The proportion of examples in a filtered synthetic dataset that are still wrong or harmful, estimated by human review of a random sample of accepted examples. Verifier. Any procedure that scores, ranks, or accepts and rejects generated candidates, from exact execution checks to learned judges. Further Reading Rubin, D. B. (1993). "Statistical Disclosure Limitation." Journal of Official Statistics, 9(2). The short discussion that proposed releasing fully synthetic data by multiple imputation. Still the clearest statement of why synthetic data must be judged by whether analyses on it reproduce analyses on the real data. Tobin, J., et al. (2017). "Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World." IROS. The paper that showed crude but heavily randomised simulation could train models that worked on real cameras. A model of how to compensate for an imperfect generator by design. Stadler, T., Oprisanu, B., and Troncoso, C. (2022). "Synthetic Data — Anonymisation Groundhog Day." USENIX Security Symposium. A rigorous empirical evaluation of privacy claims for synthetic data. Essential reading before relying on synthetic data for anonymisation. Wang, Y., et al. (2023). "Self-Instruct: Aligning Language Models with Self-Generated Instructions." ACL. The method that started the wave of model-generated instruction data, with a clear account of the filtering steps that made it work. Eldan, R., and Li, Y. (2023). "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?" arXiv:2305.07759. A careful demonstration of how matching synthetic data to a model's capacity changes what it can learn. Gunasekar, S., et al. (2023). "Textbooks Are All You Need." arXiv:2306.11644. The phi-1 paper, describing a small code model trained substantially on synthetic textbook-style data, and the start of an influential line of small-model work. Zelikman, E., Wu, Y., Mu, J., and Goodman, N. (2022). "STaR: Bootstrapping Reasoning With Reasoning." NeurIPS. A compact, clear example of a self-improving curriculum built on outcome verification. Lightman, H., et al. (2023). "Let's Verify Step by Step." arXiv:2305.20050. The study comparing process and outcome supervision for mathematical reasoning; the best single source on why checking steps matters. Zheng, L., et al. (2023). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS Datasets and Benchmarks. Established both the usefulness of language models as judges and the biases that limit them. Gao, L., Schulman, J., and Hilton, J. (2023). "Scaling Laws for Reward Model Overoptimization." ICML. A quantitative treatment of what happens when selection pressure is applied against an imperfect scoring model. Gudibande, A., et al. (2023). "The False Promise of Imitating Proprietary LLMs." arXiv:2305.15717. The key caution on distillation: students learn style far faster than capability, and human raters are fooled by style. Mukherjee, S., et al. (2023). "Orca: Progressive Learning from Complex Explanation Traces of GPT-4." arXiv:2306.02707. The explanation-tuning approach to distillation, with useful detail on prompt design for teacher outputs. Hsieh, C.-Y., et al. (2023). "Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes." Findings of ACL. Shows how teacher rationales can be used as a separate training signal for small task-specific students. DeepSeek-AI (2025). "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv:2501.12948. Describes both the large reasoning model and the distillation of its reasoning traces into much smaller open models. Trinh, T. H., et al. (2024). "Solving Olympiad Geometry Without Human Demonstrations." Nature, 625. AlphaGeometry: the clearest case of a large synthetic corpus produced entirely by a symbolic generator-verifier. Shumailov, I., et al. (2024). "AI Models Collapse When Trained on Recursively Generated Data." Nature, 631. The central paper on model collapse. Gerstgrasser, M., et al. (2024). "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data." arXiv:2404.01413. The important corrective showing that retaining real data keeps recursive training stable. Feng, Y., Dohmatob, E., et al. (2025). "Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification." ICLR. Shows that selecting synthetic data with a verifier can prevent collapse; the most direct experimental support for the argument of this book. Gebru, T., et al. (2021). "Datasheets for Datasets." Communications of the ACM, 64(12). The foundational proposal for dataset documentation, easily extended to synthetic data lineage. Hashtags: #SyntheticData #SyntheticDataArchitecture #SyntheticDataGeneration #SyntheticDataFiltering #SyntheticDataValidation #ArtificialIntelligence #MachineLearning #GenerativeAI #LargeLanguageModels #AITrainingData #DataEngineering #DataQuality #DataValidation #DataAugmentation #InstructionTuning #ModelDistillation #TeacherStudentLearning #CurriculumLearning #AutomatedDataGeneration #DataDiversity #QualityFiltering #AutomatedVerification #LLMAsAJudge #HallucinationDetection #DifferentialPrivacy #PrivacyPreservingAI #ModelCollapse #DatasetGovernance #DataProvenance #ResponsibleAI

Download the Book (PDF): Introduction Every accountant learns early that you never erase an entry in a ledger. If a payment was recorded against the wrong account, you do not scrape the ink off the page and write the correct figure in its place. You post a reversing entry that cancels the mistake, then a new entry that records what should have happened. The balance at the bottom of the column comes out the same either way. What differs is that the ledger still tells the truth about what the business did, when it did it, and when it changed its mind. An auditor reading that ledger five years later can reconstruct not only the current position but every position the business ever held, and the reasoning that moved it from one to the next. Most software does not work like that. The default shape of a business application is a set of tables holding the current state of things: a customer row with an address, an order row with a status, an account row with a balance. When something changes, the application reads the row, modifies it, and writes it back. The previous value is gone. This pattern is so familiar that it has a name borrowed from the four verbs of the database, create, read, update, delete, and so familiar that most developers never think of it as a decision at all. It is simply how data is stored. This book is about the alternative: storing the sequence of things that happened, as an append-only log of immutable facts, and deriving current state from that log whenever it is needed. That approach is called event sourcing. It travels in close company with a second idea, command query responsibility segregation, or CQRS, which says that the model you use to decide what happens next and the model you use to answer questions about the system need not be the same model, and often should not be. The two ideas are distinct. You can use either without the other. But they reinforce each other so naturally that they are usually discussed together, and in practice most event-sourced systems end up with separate read models whether or not their designers set out to build them. Where the ideas came from Neither idea is new in substance. Ledgers, version control systems, database write-ahead logs, and the transaction journals of mainframe banking systems all store history first and derive state from it. What changed in the mid-2000s was that a group of practitioners in the domain-driven design community began to argue that this shape should be a first-class choice for ordinary business applications, not just for infrastructure. Martin Fowler published a description of the pattern under the name Event Sourcing on his website in December 2005, framing it as capturing "all changes to an application state as a sequence of events." Greg Young, a developer working on trading and financial systems, did more than anyone else to turn the pattern into a coherent architectural style. Through talks beginning around 2007 and a widely circulated set of notes known as the CQRS Documents, published around 2010, Young combined event sourcing with a split between command handling and query handling, and gave that split its name. Young has repeatedly said that CQRS itself is a small idea, and that most of what people attribute to it is really event sourcing, domain-driven design, or messaging. Udi Dahan, who had been building service-oriented systems with similar separations, wrote an influential essay in 2009 called "Clarified CQRS" that reframed the pattern around business tasks and the acceptance of stale data. Vaughn Vernon's Implementing Domain-Driven Design (2013) then gave many teams their first worked treatment of aggregates and domain events in a book-length form. The intellectual root of CQRS is older still. Bertrand Meyer, in Object-Oriented Software Construction (1988), proposed command-query separation as a principle for individual methods: a method should either change state or return a value, never both. Young's contribution was to lift that principle from the level of methods to the level of whole models. Since then the ideas have moved from conference talks into products. A dedicated event database, originally called Event Store and later EventStoreDB, was co-founded by Young; the company behind it renamed itself Kurrent in December 2024, and the database became KurrentDB with its 25.0 release in March 2025. In the .NET world, Marten turned PostgreSQL into a document and event store and has become one of the most widely used event-sourcing libraries on that platform. In the Java world, Axon Framework, from the company now called AxonIQ, has long offered a full command, event, and query stack, and its fifth major version, released in October 2025, introduced support for a newer approach to consistency known as dynamic consistency boundaries. These tools, and the design problems they solve, appear throughout the chapters that follow. Why this is hard to get right Event sourcing has a reputation that runs in two directions at once. Its advocates describe it as the natural way to model any domain where history matters, with auditing, debugging, and new analytical views falling out almost for free. Its critics describe projects that collapsed under eventual-consistency bugs, unversionable event schemas, and teams that could no longer answer simple questions about their own data without writing a projection first. Both groups are describing real systems. The difference is rarely the pattern itself. It is whether the team understood what they were buying. What you buy with event sourcing is a complete, trustworthy record of business decisions, and the ability to interpret that record in new ways long after the decisions were made. What you pay is a set of obligations that a CRUD system lets you ignore. Events, once written, are permanent; their schema becomes a public contract with your own future. Read models are derived and therefore always somewhat behind. Consistency boundaries must be drawn deliberately because the store will not enforce relationships across them. Regulatory duties to forget personal data collide with a store designed never to forget. None of these are reasons to avoid the pattern. All of them are reasons to understand it before committing. The chapters are arranged to build that understanding in the order a team usually encounters the problems. Chapter 1 examines what a conventional state-based design discards and why that loss matters in some domains far more than in others. Chapter 2 treats events as the system of record: how to name them, what goes in them, how streams are organized, and how an append-only store enforces correctness through optimistic concurrency. Chapter 3 turns to the write side, covering commands, aggregates, and the boundaries within which a system can guarantee its rules, including the recent challenge to the aggregate posed by dynamic consistency boundaries. Chapter 4 introduces CQRS on its own terms and shows where it helps even without event sourcing. Chapter 5 covers projections, the machinery that turns an event log into queryable views, along with the operational realities of checkpoints, rebuilds, and lag. Chapter 6 takes up the most celebrated capability of the pattern, reconstructing past state, and is careful about what that capability actually delivers for auditing and debugging. Chapter 7 deals with the problem most likely to hurt a mature system: changing the shape of events that can never be changed. Chapter 8 covers long-running business processes and the line between events used inside a service and events published to the rest of an organization. Chapter 9 surveys the infrastructure, from dedicated event databases to relational stores to streaming platforms, and addresses operations, privacy, and the decision of when not to use any of this. Who this is for The intended reader is a working developer, architect, or technical lead who knows how to build a conventional database-backed application and wants a clear account of when and how to build a different kind. No prior knowledge of domain-driven design is assumed; where its vocabulary is needed, it is defined. Code examples are short and written in a neutral, lightly typed style so that the reasoning survives translation into whatever language you use. Where specific products are discussed, the point is the design trade-off each represents rather than a recommendation to adopt it. A single running domain recurs through several chapters: an insurance claims system, in which a policyholder reports a loss, an adjuster assesses it, reserves are set aside, payments are made, and decisions are sometimes reversed. Claims handling is a good teaching domain because its history is the business. Regulators expect insurers to explain how each claim moved from report to settlement, reserves are revised many times as information arrives, and disputes routinely turn on what the insurer knew at a particular moment. If event sourcing is ever the right default, it is in domains with that character. Other examples, from bank accounts to shopping carts to hospital bed management, appear where they illustrate a point more sharply. The argument of this book The position taken here is straightforward. Event sourcing is not a storage technique; it is a decision to treat the history of business decisions as the authoritative record of the system, with current state demoted to a convenient, disposable derivative. That decision pays for itself in domains where the history is itself valuable to the business, the regulator, or the people trying to understand what went wrong, and it is an expensive affectation in domains where it is not. CQRS is a separate and cheaper decision that can be made on its own merits. Teams that succeed with these patterns treat event design as product design, accept eventual consistency as a feature to be shaped rather than a defect to be hidden, and plan from the first day for the fact that their events will outlive the code that wrote them. The ledger analogy that opened this introduction is worth keeping in mind throughout. Accountants did not adopt double-entry bookkeeping because it was elegant. They adopted it because, in a domain where money moved and people disagreed about where it had gone, a record that could not be quietly rewritten was worth the extra effort of keeping it. The question this book helps you answer is whether your domain is like that, and if it is, how to build the ledger well. Chapter 1: The Cost of Forgetting Consider an ordinary claims table in an insurance system. Each row holds a claim number, a policy reference, a status, an assessed amount, a reserve, an assigned adjuster, and a last-modified timestamp. When the adjuster revises the reserve from 40,000 to 65,000 after a structural survey, the application executes an update statement and the row now says 65,000. When the claim is reassigned because the original adjuster goes on leave, the adjuster column changes. When the claim is closed, then reopened six weeks later because the policyholder produced new evidence, the status column flips from closed to open. Now imagine a regulator asks a reasonable question: on the date the claim was first closed, what reserve was held against it, who had approved that figure, and what information was on file? The claims table cannot answer. It knows only the present. The row has been overwritten at least four times, and each overwrite destroyed the value it replaced. Every experienced developer knows the standard responses to this problem, and every one of them is an attempt to reconstruct, after the fact, the history that the primary design threw away. That observation is the starting point for event sourcing. What a state-based model actually stores A conventional relational design stores the current answer to a set of questions the designers anticipated. The claims table answers "what is the reserve now?" and "who is handling this now?" because someone decided those were the questions that mattered. It does not store why the reserve is what it is, what it used to be, or the order in which things happened, because those questions were not on the list, or were on the list but expensive to support. This is not a flaw in relational databases. A relational engine can model history perfectly well if you ask it to. The point is that the default modeling habit, one row per entity, updated in place, encodes a specific stance: the present is what matters and the past is someone else's problem. For a great many applications that stance is correct. A content management system needs to know what an article says now. A product catalogue needs today's price. If last month's price ever matters, it matters to a different system, such as the order history, which captured it at the time of sale. The trouble begins when the past turns out to matter to the same system, and the design has already discarded it. At that point teams reach for one of four familiar remedies. The first is the audit table. A trigger or application hook copies each row to a shadow table before every update, often with a timestamp and a user identifier. This preserves old values, but it preserves them as snapshots of columns, not as statements of intent. The audit table can tell you that the reserve column changed from 40,000 to 65,000 at 14:02 on a Tuesday. It cannot tell you whether that change was a reserve increase following a survey, a correction of a typing error, or a side effect of a bulk migration script. The reason for a change lives in the application code that made it, and the audit table records only the residue. The second is the status history, a child table recording each status transition. This captures some intent, because transitions like "closed" and "reopened" are named. But it covers only the attribute someone thought to track, and it drifts out of sync with the main row the moment a developer writes a code path that updates the status without inserting the history record. The third is soft deletion and versioned rows: instead of updating a row, insert a new version and mark the old one superseded. This is a real temporal design, and SQL:2011 standardized system-versioned tables to support it; PostgreSQL extensions, SQL Server temporal tables, and similar features implement variations. Versioned rows answer "what did this row look like at time T?" very well. They still answer it in terms of column values rather than business facts. The fourth is logging. Application logs often contain the richest record of what happened, including the user's action, the parameters, and the outcome. But logs are written for operators, retained for weeks, and formatted inconsistently. Building business reports from logs is a known path to misery. All four remedies share a structure. The primary record is current state, and history is a secondary artifact maintained alongside it, with no guarantee that the two agree. When they disagree, the current state wins by default, because it is what the application actually reads. History becomes a best-effort record that is trusted until the day it matters, which is exactly the day someone discovers a gap. Inverting the relationship Event sourcing inverts this relationship. The primary record is the sequence of facts that the business considers meaningful, each stored as an immutable event. Current state is computed from those events, and can be thrown away and recomputed at will. For the claim above, the stored record would look something like this, expressed as a sequence of events in a single stream belonging to the claim: ClaimReported { claimId: C-1042, policy: P-77, lossDate: 2026-03-02, ... } AdjusterAssigned { claimId: C-1042, adjuster: "m.okafor" } ReserveSet { claimId: C-1042, amount: 40000, basis: "initial estimate" } SurveyReceived { claimId: C-1042, surveyRef: S-881, finding: "structural" } ReserveIncreased { claimId: C-1042, from: 40000, to: 65000, approvedBy: "j.lin" } AdjusterReassigned { claimId: C-1042, from: "m.okafor", to: "t.reyes", reason: "leave" } ClaimClosed { claimId: C-1042, settledAmount: 61200 } ClaimReopened { claimId: C-1042, reason: "new evidence", evidenceRef: E-19 } The current state of the claim, the thing the CRUD table stored directly, is obtained by starting from nothing and applying each event in order. A function that does this is often called a fold or a left fold, after the functional programming operation it resembles: an initial state, a function that takes a state and an event and returns a new state, and a sequence of events. The result of folding all eight events is a claim that is open, assigned to t.reyes, with a reserve of 65,000 and a prior settlement of 61,200. The regulator's question now has a mechanical answer. Fold only the events up to and including ClaimClosed, and you have the claim as it stood at that moment, with the approver of the reserve named in the event that set it. The question did not need to be anticipated. It is answerable because nothing was thrown away. Three properties of this arrangement matter more than the mechanics. First, the events are named in the language of the business. ReserveIncreased is not a column change; it is a thing an insurer does, with a meaning that claims staff, actuaries, and auditors all recognize. The distinction between ReserveIncreased and a hypothetical ReserveCorrected is exactly the distinction the audit table could not capture. Domain-driven design, the modeling approach from which event sourcing emerged, calls this shared vocabulary the ubiquitous language, and event sourcing forces it into the storage layer. Second, the events are immutable and append-only. Nothing ever updates or deletes an event in normal operation. When a decision is reversed, the reversal is a new event. This is the accountant's discipline from the introduction, and it is what makes the log trustworthy as evidence. A store that permits edits is a store whose history might have been edited. Third, current state is derived and disposable. Because state can always be recomputed from events, you can compute it in more than one way. The adjuster's work queue, the reserve report for the finance team, and the fraud model's feature table can each be a different interpretation of the same events, built at different times by different teams. Chapter 5 is devoted to this idea, which is where most of the practical power of the pattern lives. What it costs The inverted design is not free, and the costs should be stated as plainly as the benefits. Reading current state now requires computation. For a single claim with a few dozen events, folding them takes microseconds and nobody notices. For an entity with tens of thousands of events, such as a long-lived bank account or a high-volume inventory item, replaying everything on every request becomes a real cost, which is why snapshotting exists. Chapter 6 discusses when snapshots help and when they are a sign of a modeling problem. Querying across entities becomes indirect. In the CRUD design, "show me all open claims over 50,000 assigned to t.reyes" is one SQL statement. In an event-sourced design, the event store is organized for appending and reading streams, not for ad hoc filtering across them. To answer that question you need a read model, a projection that listens to claim events and maintains a table shaped for that query. This is the main reason event sourcing and CQRS travel together: once your write model is a log, you need separately built read models to answer questions efficiently. Read models lag. A projection processes events after they are written, so there is always some interval, usually milliseconds and occasionally much longer, during which the read model does not yet reflect the latest event. User interfaces and downstream processes must be designed for that. Events are forever. In a CRUD system, if you design a column badly you migrate the table and move on. In an event-sourced system, events written under last year's understanding of the domain remain in the store, and every future version of the code must either understand them or have them translated. Chapter 7 covers the techniques for living with this, and it is the cost that most teams underestimate. Forgetting becomes hard. Data protection law, most prominently the European Union's General Data Protection Regulation with its right to erasure in Article 17, requires that some personal data be deleted on request. A store whose central virtue is that it never forgets must be designed from the start to forget selectively. Chapter 9 addresses the main technique, encrypting personal data with per-subject keys so that deleting the key renders the data unreadable. Choosing where the inversion pays Table 1 summarizes how the two approaches differ on the dimensions that tend to decide real projects. Table 1. State-based storage compared with event sourcing. Concern State-based (CRUD) Event-sourced Primary record Current values per entity Ordered, immutable facts History Secondary, best effort Intrinsic and complete Intent of changes Usually lost Captured in event names and data Ad hoc queries Direct against tables Via purpose-built read models Schema change Migrate in place Version and translate old events Deleting personal data Delete the row Requires deliberate design The comparison suggests a test. Event sourcing earns its place when three conditions hold together. The domain has behavior worth recording: users and processes make decisions, and those decisions, not just their outcomes, are meaningful to someone. The history is consulted: auditors, regulators, support staff, analysts, or the system's own rules need to know what happened and in what order. And the interpretations change: the business will want to ask questions of the past that nobody can predict today. Claims handling meets all three. So do payments and ledgers, trading and order management, logistics and shipment tracking, clinical records, and many workflow systems in which items move through states under human judgment. In each of these, the industry had already developed paper or digital journals before anyone used the phrase event sourcing, because the history was the business. A user profile service usually fails the test. Users change their display names and avatars; nobody cares what the avatar was last March, and no rule depends on the order of changes. A reference data service holding country codes fails it. So does most of a typical content system. Forcing these into an event-sourced shape produces streams of FieldUpdated events that carry no business meaning, a sure sign the pattern is being applied where it does not belong. Young has made the point repeatedly in talks that event sourcing should be applied within particular parts of a system rather than across the whole of it, and that a system built entirely with it is usually a mistake. There is a subtler failure that the test helps expose. Some teams adopt event sourcing for a domain that does meet the conditions, but then design events that mirror CRUD operations: ClaimCreated, ClaimUpdated, ClaimDeleted, with the updated event carrying a full copy of the new state. This gets the costs of event sourcing without its benefits. The intent is lost exactly as it would be in an audit table, while the team now has to manage versioning, projections, and eventual consistency. The value of the pattern depends almost entirely on the quality of the events, which is the subject of the next chapter. A note on what event sourcing is not Three confusions recur often enough to clear away before going further. Event sourcing is not the same as event-driven architecture. An event-driven system is one in which components communicate by publishing notifications that other components react to, commonly over a message broker. An event-sourced system is one in which a component stores its own state as events. A system can be event-driven without being event-sourced, which is the common case: services publish events to a broker while storing their own state in ordinary tables. A component can be event-sourced without being event-driven, storing its history as events but exposing only a request-response interface. Fowler's 2017 essay "What do you mean by 'Event-Driven'?" distinguishes event notification, event-carried state transfer, event sourcing, and CQRS as four different patterns that are often conflated, and the distinction is worth internalizing. Event sourcing is not the same as change data capture. Change data capture tools read a database's transaction log and emit a stream of row-level changes. That stream is useful for replication and integration, but it records what happened to tables, not what happened in the business. It is the audit-table problem at scale. Event sourcing is not the same as storing events in Kafka. Apache Kafka is a distributed log, and logs are the essence of event sourcing, so the association is natural. But a general-purpose streaming platform lacks some properties an event store needs for the write side, most importantly the ability to append to an individual entity's stream conditionally, only if no one else has appended since you last read it. Chapter 9 examines this debate in detail. With those distinctions in place, the core claim of this chapter can be stated simply. A state-based design answers the questions its designers anticipated and forgets everything else. An event-sourced design remembers what the business did and lets future designers ask new questions. Whether that memory is worth its cost depends on the domain, and the rest of this book is about building it well when it is. Chapter 2: Events as the System of Record If the events are the authoritative record of a system, then designing them is the most consequential modeling work the team will do. A badly designed table can be migrated next quarter. A badly designed event will be read by every version of the software for as long as the system exists. This chapter covers what an event is, how to name and shape one, how events are grouped into streams, and how an event store guarantees that two people acting at once do not corrupt the record. Commands, events, and the grammar of change The vocabulary matters here, so it is worth being precise. A command is a request to do something: IncreaseReserve, AssignAdjuster, CloseClaim. It is phrased in the imperative mood. A command can be rejected. It carries the intent of whoever issued it, along with whatever data the system needs to decide whether to honor it. An event is a record that something happened: ReserveIncreased, AdjusterAssigned, ClaimClosed. It is phrased in the past tense. An event cannot be rejected, because it describes a fact that has already been accepted. The system's job, on receiving a command, is to check the command against the current state and the business rules, and if the command is valid, to record one or more events describing the outcome. A query asks for information and changes nothing. Table 2 sets these side by side, because confusing them is the root of many design errors. Table 2. Commands, events, and queries. Message Grammatical form Can it fail? Who decides Example Command Imperative Yes The receiving model IncreaseReserve Event Past tense No, already true Already decided ReserveIncreased Query Question Only technically Nobody; read only GetOpenClaims The past tense is not a style preference. It forces designers to think about events as facts rather than instructions. An event named SendWelcomeEmail is a command wearing an event's clothes; the fact is CustomerRegistered, and sending an email is one possible reaction to it. When events are phrased as instructions, the producer has taken on knowledge of what its consumers do, coupling that the event model was supposed to remove. The relationship between commands and events is not one to one. A single SettleClaim command might produce PaymentAuthorized, ReserveReleased, and ClaimClosed. A command can also produce no events at all, if it turns out to be a no-op, such as assigning a claim to the adjuster already handling it. And some events arise without any user command, from the passage of time or from external notifications: PolicyLapsed because a premium deadline passed, PaymentBounced because the bank said so. Naming events after the business The single most useful rule for event design is that an event should describe something a domain expert would recognize as having happened. If you read the event name aloud to an experienced claims handler and they nod, the name is probably right. If they look puzzled, the name is probably describing your software rather than their business. This rule rules out a family of names that appear constantly in first attempts. ClaimUpdated says nothing about what changed or why. ClaimStatusChanged is better but still describes a field rather than a business occurrence; closing a claim, reopening a claim, and suspending a claim pending investigation are different acts with different consequences, and they deserve different names. ReserveAmountSet loses the distinction between an initial reserve, an increase following new information, a decrease following a successful recovery from a third party, and a correction of a clerical error. Each of those means something different to an actuary computing loss development, and an actuary will eventually want to ask. The practical technique for discovering good event names is to talk to the people who do the work, in their terms, and to write down what they say happens. Alberto Brandolini's EventStorming workshop format, developed in the early 2010s and described in his self-published book Introducing EventStorming, does exactly this: domain experts and developers place sticky notes bearing past-tense events along a wall in time order, then argue about them. The arguments are the point. When the claims manager and the finance lead disagree about whether "reserve released" happens at settlement or at payment, the team has found a real ambiguity in the business that a CRUD model would have buried. Granularity follows from the same rule. An event should be as large as one business fact and no larger. ClaimReportedWithInitialReserveAndAdjuster bundles three decisions that can be made independently, by different people, and that might be reversed independently. Bundle them and you will find yourself, a year later, unable to distinguish a claim that was reported with a reserve from one that had its reserve set later. On the other side, splitting a single act into fragments, ClaimReportedPart1 and ClaimReportedPart2, produces events that are meaningless on their own and risks a reader seeing one without the other. What goes inside an event An event has two layers: the payload, which carries the business data, and the metadata, which carries information about the recording of the event. The payload should contain everything a reader needs to understand the fact without consulting other sources, and nothing that would be wrong to keep forever. For ReserveIncreased, that means the claim identifier, the new amount, and arguably the previous amount and the reason or basis. Including the previous amount is redundant, since it could be computed by folding earlier events, but redundancy in events is often a kindness. A projection that wants to compute the total of all reserve movements in a month can do it from the events alone, without reconstructing each claim's state. Several practitioners recommend this kind of self-describing payload precisely because events are read far more often, and by far more different consumers, than they are written. What should not go in the payload is derived data that depends on rules that might change. If the reserve increase triggered a re-rating of the policy under a formula, record the inputs and the outcome as the business understood them, not an intermediate computation that only makes sense under today's formula. Events record decisions; they should not encode the algorithm that produced the decision unless the algorithm's output is itself the fact. Personal data deserves particular care. Every field of personal data in an event is a field you may later be legally required to erase from an immutable store. Some teams keep personal details out of events entirely, storing a reference to a separate, mutable record. Others encrypt them per data subject. Either way, the decision belongs in the first design, not the first data-protection request. The metadata carries facts about the event's recording rather than about the business occurrence. Common fields include a unique event identifier, the event type name, a timestamp of when it was recorded, the identity of the user or process that caused it, a correlation identifier linking all events that arose from the same original request, and a causation identifier naming the specific message that directly caused this event. The correlation and causation identifiers are cheap to record and invaluable later; they let you trace a cascade of events through several services back to the click that started it. A recurring subtlety concerns time. The moment an event was recorded is not always the moment the thing happened. A claim reported by phone on Friday afternoon but entered into the system on Monday morning has a loss date, a report date, and a recording date, and all three can matter. Treat the business times as payload fields with explicit names, and let the metadata timestamp mean only when the store accepted the event. Chapter 6 returns to this distinction, which is the foundation of what is called bitemporal modeling. Here is a representative event as it might be stored, with metadata separated: type: ReserveIncreased data: { "claimId": "C-1042", "previousAmount": 40000, "newAmount": 65000, "currency": "GBP", "basis": "structural survey S-881", "approvedBy": "j.lin" } metadata: { "eventId": "6f1c...", "recordedAt": "2026-03-19T14:02:11Z", "userId": "j.lin", "correlationId": "req-55a0...", "causationId": "cmd-91be..." } Streams and the shape of the store Events are grouped into streams. A stream is an ordered sequence of events, usually belonging to one entity or one consistency boundary, and identified by a name such as claim-C-1042. Every event in a stream has a position, often called its version or revision, starting from zero or one and increasing by one with each append. To load an entity, you read its stream from the beginning and fold the events. Most event stores also maintain a global order across all streams, sometimes called the all stream or the global log, in which every event receives a monotonically increasing position. KurrentDB exposes this as a log position and lets subscribers read from it. Marten, which stores events in PostgreSQL tables, assigns each event a global sequence number in addition to its stream version. Projections that need to see every event, in an order consistent with how they were written, read from this global sequence. Chapter 5 discusses why that ordering guarantee is subtler than it sounds when the store is a relational database with concurrent transactions. How fine-grained should streams be? The conventional answer is one stream per aggregate instance, where an aggregate is the cluster of state within which business rules must be enforced together. Chapter 3 is devoted to aggregates. For now the useful intuition is that a stream is the unit of consistency: within a stream, events are strictly ordered and appends can be made conditional; across streams, there is no such guarantee. Streams should generally be short-lived or bounded in size. A stream per claim is natural, since claims open and eventually close. A stream per bank account is natural but can grow without limit for an account used daily over decades, which is one reason banking examples so often lead to snapshots or to modeling accounting periods as separate streams. A single stream for "all claims" would be a serious mistake: it would serialize every write across the business through one point of contention, and loading any claim would mean reading every event ever written. Some practitioners advocate modeling explicit lifecycle boundaries in streams, closing a stream when a business period ends and starting a new one seeded with a summary event. Accounting systems do this at period close: the opening balance of the new period is a fact carried forward, and the detailed history of the prior period is retained but no longer read on every operation. This is often a cleaner answer to long streams than technical snapshots, because it corresponds to something the business already does. Appending safely: optimistic concurrency The final piece of the storage model is the mechanism that keeps the record consistent when two actors try to change the same entity at once. Suppose two adjusters open claim C-1042 at nearly the same moment. Both load its eight events, both see a reserve of 65,000, and both decide to act. One increases the reserve to 80,000; the other closes the claim. In a naive implementation both would append their events, producing a stream in which the claim was closed and had its reserve increased in an order that neither adjuster saw and that may violate a business rule, such as "a closed claim cannot have its reserve changed." Event stores prevent this with optimistic concurrency control on append. When the first adjuster's command handler loaded the stream, it noted that the last event was at revision 7. When it appends ReserveIncreased, it tells the store: append this event to claim-C-1042, but only if the stream's current revision is still 7. The store checks atomically. The first append succeeds, and the stream is now at revision 8. When the second adjuster's handler tries to append ClaimClosed with the same expectation of revision 7, the store refuses, because the stream has moved. The second handler can then reload the stream, re-run its decision against the new state, and either append successfully at revision 8 or reject the command because the rules no longer permit it. In KurrentDB's client libraries this is expressed as an expected revision or expected state passed to the append call; the options include a specific revision, "no stream" for creating a new stream, "stream exists," and "any" to disable the check. In Marten it is expressed by passing an expected version when appending to a stream, or by using its aggregate-loading helpers that track the version automatically. In a hand-built store on a relational database, the simplest implementation is a unique constraint on the pair of stream identifier and version: two transactions trying to insert version 8 into the same stream cannot both succeed. A minimal relational implementation looks like this: CREATE TABLE events ( global_position BIGSERIAL PRIMARY KEY, stream_id TEXT NOT NULL, stream_version INT NOT NULL, event_type TEXT NOT NULL, data JSONB NOT NULL, metadata JSONB NOT NULL, recorded_at TIMESTAMPTZ NOT NULL DEFAULT now(), UNIQUE (stream_id, stream_version) ); The unique constraint is the whole concurrency mechanism. It is also why this approach is called optimistic: the system does not lock the stream while a user thinks; it assumes conflicts are rare and detects them at the moment of writing. For most business entities that assumption holds, because the same claim is rarely changed by two people within the same second. Where it does not hold, for example a hot inventory item touched by thousands of orders a minute, the conflict rate itself is telling you the consistency boundary is drawn in the wrong place. Optimistic concurrency has one more consequence worth understanding. Because the check is per stream, the store can guarantee rules only within a stream. A rule that spans streams, such as "no two policyholders may register with the same email address," cannot be enforced by a single conditional append in a stream-per-aggregate design. This limitation is real and it shapes the next chapter, which covers how aggregates are drawn, how cross-boundary rules are handled, and how a recent proposal called dynamic consistency boundaries tries to relax the constraint. The event as a promise It is worth closing with the change in attitude that event-sourced teams tend to describe once they have lived with the pattern for a while. In a CRUD system, the schema is an internal detail of a service, and changing it is a refactoring. In an event-sourced system, every event type is a promise: to every projection that reads it, to every future version of the code that must replay it, to every auditor who will interpret it, and, if it is published, to every other team that consumes it. Good event designers come to treat event types roughly the way good API designers treat public endpoints. They take time over names, they prefer additive changes, they document meaning rather than just structure, and they resist the temptation to put something in an event merely because it is convenient today. That discipline is the price of having a record you can trust. The rest of the architecture, from aggregates to projections to time travel, depends on it. Chapter 3: Aggregates, Commands, and Consistency Boundaries An event store records facts. Something has to decide which facts to record. In an event-sourced system that something is the write model: the code that receives a command, works out whether the command is permitted given everything the system knows, and if so produces the events that describe what happened. The central design question on the write side is how much the write model needs to know in order to make each decision safely. Answer it too narrowly and the system cannot enforce its rules. Answer it too broadly and every decision contends with every other, and the system grinds. The traditional answer is the aggregate. This chapter explains what that means, how to draw aggregates well, how to handle the rules that do not fit inside them, and why some practitioners now argue the aggregate should give way to something more flexible. The aggregate as a decision boundary The term comes from Eric Evans's Domain-Driven Design (2003), where an aggregate is a cluster of associated objects treated as a unit for the purpose of data changes, with one root entity through which all outside access passes. Evans's motivation was consistency: the aggregate is the boundary within which business invariants, rules that must always hold, are guaranteed to hold after every change. In an event-sourced system the definition becomes concrete. An aggregate is the set of events that must be considered together to decide a command, and it corresponds to one stream. To handle IncreaseReserve for claim C-1042, the write model loads the claim's stream, folds it into a state, checks the rules (the claim is open, the new amount exceeds the old, the approver has authority for that amount), and appends a ReserveIncreased event with an expected revision equal to the last revision it read. The optimistic concurrency check described in the previous chapter guarantees that no other event slipped in between the read and the write. Therefore the decision was made on complete, current information. That is the entire consistency guarantee, and it holds only within the stream. A clean way to express this, popularized by Jérémie Chassaing under the name decider, separates the write model into three pure pieces: an initial state, an evolve function that applies one event to a state, and a decide function that takes a command and a state and returns a list of events or an error. Infrastructure code handles loading, folding, and appending. The business logic is two functions with no side effects, which makes it unusually easy to test: given these past events, when this command arrives, then these new events result, or this error. initialState = { status: "none", reserve: 0 } evolve(state, event) = match event.type: "ClaimReported" -> { ...state, status: "open" } "ReserveIncreased" -> { ...state, reserve: event.newAmount } "ClaimClosed" -> { ...state, status: "closed" } else -> state decide(command, state) = match command.type: "IncreaseReserve" -> if state.status != "open": error("claim not open") if command.amount <= state.reserve: error("not an increase") [ReserveIncreased(previousAmount: state.reserve, newAmount: command.amount, ...)] Notice what the evolve function keeps: only what decide needs. The aggregate's state is not a complete description of the claim. It does not need the policyholder's name, the adjuster's notes, or the history of reserve changes, because no rule depends on them. This is one of the quiet liberations of event sourcing. In a CRUD design the entity's table must hold every attribute anyone might want to display, because the same table serves reads and writes. In an event-sourced design the write model holds only decision-relevant state, and everything else lives in the events, available to read models that want it. The given-when-then shape also produces tests that domain experts can read: given: ClaimReported(C-1042), ReserveSet(40000), ClaimClosed(61200) when: IncreaseReserve(C-1042, 80000) then: error "claim not open" That test documents a business rule in terms the claims manager will recognize, and it runs in microseconds with no database. Drawing the boundary The hardest skill in aggregate design is deciding what belongs together. Vaughn Vernon's three-part essay "Effective Aggregate Design," published by the DDD community in 2011 and later folded into Implementing Domain-Driven Design, set out rules that most practitioners still follow. Paraphrased, they are: model true invariants in consistency boundaries; design small aggregates; reference other aggregates by identity rather than holding them inside; and use eventual consistency outside the boundary. The first rule is the important one. An invariant is a rule that must hold at every instant, such that a violation even for a moment is unacceptable. "A claim's total payments must not exceed its reserve without an approved increase" is a strong candidate: an insurer must not pay out money it has not reserved. Payments and reserve belong in the same aggregate, so that the rule can be checked atomically when a payment is recorded. Many rules that sound like invariants are not. "A policyholder may have at most three open claims" sounds strict, but ask the business what happens if a fourth claim is reported while the third is still being processed. Often the answer is that a supervisor reviews it, or that the limit is a fraud signal rather than a hard prohibition. If the business can tolerate a brief violation followed by a corrective action, the rule does not require a single consistency boundary. It can be enforced asynchronously: a process notices the fourth claim and flags it for review. This distinction between rules that must never be violated and rules that must be noticed and corrected is the most valuable question an event-sourcing team can learn to ask, and it almost always reveals that true invariants are fewer than the first draft supposed. Small aggregates matter because the aggregate is also the unit of contention. If the policy, all its claims, and all their payments were one aggregate, every action on any claim would conflict with every other action on the same policy, and a large commercial policy with hundreds of concurrent claims would become a bottleneck. Making each claim its own aggregate means two adjusters working on different claims never conflict. The cost is that rules spanning claims cannot be checked atomically. Referencing by identity means the claim holds policyId: P-77, not a copy of the policy. If the claim needs to know whether the policy covered the loss date, that question is answered when the claim is reported, and the answer recorded as part of the ClaimReported event, or it is answered by a separate check whose result is fed into the command. The claim aggregate does not reach into the policy aggregate to read its state in the middle of a decision, because doing so would make the decision depend on data the claim's concurrency check does not protect. Rules that cross boundaries Every real domain has some rules that span aggregates. The classic example is uniqueness: no two customer accounts with the same email address. In a CRUD system this is a unique index. In a stream-per-aggregate event store there is no such index, because each customer is a separate stream and no single conditional append can see all of them. Practitioners have converged on a small set of answers. The first is to question the rule. Greg Young has argued in talks that many uniqueness constraints exist because developers assume them rather than because the business requires them, and that the cost of an occasional duplicate, detected and merged afterwards, is often lower than the cost of preventing it. For email addresses this is frequently true: a duplicate registration can be detected within seconds and the second account suspended with an apologetic message. The second is a reservation. Model the scarce resource as its own small aggregate, for example a stream per normalized email address, and require that a registration first claim the address by appending EmailReserved to that stream with an expectation that the stream does not yet exist. If the append succeeds, the address is yours; proceed to create the account. If it fails, someone else holds it. This turns a set constraint into a single-stream constraint, which the store can enforce. The cost is a second write and the need to handle partial failure, such as a reservation whose account creation never completes. The third is to keep a conventional lookup table alongside the event store, updated in the same transaction as the append. When the event store is a relational database, as with Marten on PostgreSQL or a hand-built store, this is cheap and effective: the unique index does its job, and the transaction ensures it never disagrees with the events. Purists object that this introduces mutable state into the write side; pragmatists observe that it works. The fourth is to enforce the rule in a process that reacts to events and issues compensating commands, accepting a short window of inconsistency. Chapter 8 covers these processes. Which answer is right depends on the cost of a violation. Double-booking a seat on a flight is expensive and visible; it justifies a reservation. A duplicate newsletter subscription is cheap; it justifies doing nothing special. The challenge from dynamic consistency boundaries For most of the pattern's history, the stream-per-aggregate model was treated as fixed. In 2023, the software engineer Sara Pellegrini published an essay with the deliberately provocative title "Killing the Aggregate," which argued that the fixed aggregate forces designers to commit to consistency boundaries before they understand the domain, and then makes those boundaries very hard to change because they are baked into stream identities. Her proposal, developed with others and now documented at dcb.events, is called the dynamic consistency boundary, or DCB. The idea works like this. Instead of appending events to named streams, the application appends events to a single log, tagging each event with the identifiers of every entity it concerns. A StudentSubscribedToCourse event might carry the tags student:S-12 and course:C-7. To make a decision, the application queries the log for the events relevant to that decision, by type and tag, for example all subscription events tagged with this course or this student. It folds them into whatever state the decision needs. When it appends the new event, it passes an append condition: the same query it used to read, plus the position of the last event it saw. The store rejects the append if any event matching that query has been written since that position. The effect is optimistic concurrency over an arbitrary, decision-specific slice of the log rather than over a predetermined stream. The canonical example, which appears throughout the DCB literature, is course subscription with two rules: a course has a capacity, and a student may take at most a fixed number of courses. In the aggregate model, these two rules belong to two different aggregates, course and student, and enforcing both atomically requires either merging them into an implausibly large aggregate or coordinating with a saga that can leave the system temporarily inconsistent. With DCB, the decision reads events tagged with either the course or the student, and the append condition protects exactly that set. Two subscriptions to different courses by different students do not conflict; two subscriptions that would together overfill the same course do. DCB has moved quickly from essay to implementation. Axon Framework 5, released in October 2025, and Axon Server's 2025.1 release support it. The dcb.events site lists open-source implementations in several languages, and libraries such as the Python eventsourcing package document DCB support. The specification is deliberately minimal, defining only the query-and-condition interface, so that it can be implemented over different storage engines. It is too early to declare the aggregate dead, and the essay's title should be read as an argument rather than a verdict. Several practical questions are still being worked out across implementations: how efficiently tag queries can be indexed at scale, how to reason about the performance of decisions whose read set is not known in advance, and how teams used to thinking in aggregates should organize code when the boundary is assembled per command. Some practitioners also point out that most domains are well served by small aggregates and that DCB's flexibility, like any flexibility, can be misused to create decisions that read far too much. What DCB does settle is that the stream-per-aggregate model is one point in a design space rather than a law of nature. The underlying principle, that a decision is safe if no relevant fact changed between reading and writing, survives either way. Commands at the edge A few practical matters about commands round out the write side. Commands should carry the user's intent at the level of a business task, not a field edit. A user interface that presents a claim as a form with editable fields, all submitted together with a single save button, will naturally produce a command like UpdateClaim carrying every field, and the write model will be reduced to working out which fields changed and inventing events to match. That is the CRUD shape reasserting itself. Udi Dahan's "Clarified CQRS" made the case for task-based user interfaces, in which the screen offers actions like "increase reserve," "reassign," and "close with settlement," each producing a specific command. Task-based interfaces are more work to design, and not every screen needs one, but in the parts of the system where intent matters they are what allows the events to capture it. Commands should be idempotent where possible, or carry an identifier that lets the write model detect duplicates. Networks retry. A user double-clicks. A message broker delivers twice. If the MakePayment command carries a client-generated payment identifier, and the aggregate records that identifier in the PaymentMade event, a second delivery can be recognized and ignored rather than producing a second payment. Validation divides into two kinds. Some checks need no state: the amount is positive, the currency code is valid, the required fields are present. These belong at the edge, before the command reaches the aggregate. Other checks need state: the claim is open, the payment does not exceed the reserve. These belong in decide. Keeping the two separate keeps the aggregate focused on business rules. Finally, the write model should return little. A command handler that succeeds might return the new stream revision, or the identifiers of created entities, so that the caller can, if it chooses, wait for read models to catch up to that revision. It should not return a rich view of the entity, because that view is a read concern and belongs to the read side. Separating those concerns, and seeing what that separation buys, is the subject of the next chapter. Chapter 4: CQRS: Separating the Write Model from the Read Model Command query responsibility segregation is a small idea with a large shadow. Stated plainly, it says that a system may use one model to handle changes and a different model to answer questions. That is all. It does not require event sourcing, message queues, separate databases, or eventual consistency, though it permits all of them. Much of the confusion surrounding the pattern comes from treating everything that tends to accompany CQRS as part of its definition. This chapter separates the idea from its entourage, explains why the separation is useful, and shows how far a team can go with it before event sourcing enters the picture. From methods to models Bertrand Meyer's principle of command-query separation, set out in Object-Oriented Software Construction, applied to individual routines. A function that returns a value should not change the state of the system; a procedure that changes state should not return a value. The benefit is reasoning: a caller can invoke a query any number of times, in any order, with no side effects, and can rely on commands to be the only place state changes. Greg Young's step was to apply the same separation at the level of the whole model. In a conventional layered application, a single domain model, often a set of classes mapped to tables by an object-relational mapper, serves both purposes. The Claim class has methods that enforce rules when things change and properties that are read to populate screens. Over time those two jobs pull the model in opposite directions. The write side wants a small, behavior-rich object that holds only the state needed for decisions and protects its invariants. The read side wants flat, denormalized data shaped exactly like each screen or report, joined across many entities, filtered and sorted in ways the write model has no reason to know about. A single model serving both ends up compromising both: domain objects bloated with display properties and navigation collections, and queries forced through an object graph that was never designed for them. CQRS resolves the tension by refusing to compromise. Build a write model optimized for deciding, and one or more read models optimized for answering. Let each be as simple as its job allows. Martin Fowler's short article on CQRS, published on his website in 2011, endorsed the pattern while issuing a warning that has aged well: for most systems, he wrote, CQRS adds risky complexity, and it should be applied to specific portions of a system rather than to the whole. That warning is worth taking literally. CQRS is a tool for places where the read and write needs genuinely diverge, not a default architecture. Levels of separation It helps to think of CQRS as a spectrum of increasing separation, each level adding capability and cost. At the first level, the separation is purely in code. The application has command handlers that load domain objects, invoke behavior, and save them, and query handlers that bypass the domain model entirely and read from the database directly, perhaps with hand-written SQL or a lightweight data mapper, returning flat data transfer objects shaped for the caller. Both sides use the same database and the same tables. There is no eventual consistency, because the query reads the same rows the command just wrote. This level alone often brings large benefits: the domain model sheds its display concerns, and queries become simple and fast because they are no longer routed through an object graph. Many teams describe their systems as using CQRS when they are at this level, and that is a perfectly reasonable place to stop. At the second level, the read side gets its own storage within the same database: views, materialized views, or denormalized tables maintained alongside the normalized write tables. A database materialized view refreshed on a schedule, or a summary table updated in the same transaction as the write, gives queries data that is already shaped for them. Consistency is still easy to reason about if updates are transactional. At the third level, the read side lives in a different store altogether, chosen for its query capabilities. The write side might use a relational database, while product search is served from a search engine such as Elasticsearch or OpenSearch, a dashboard from a columnar analytics store, and a recommendation feature from a graph database. Now the read models must be updated asynchronously, by some mechanism that carries changes from the write side, and eventual consistency becomes unavoidable. At the fourth level, the write side is event-sourced, and the events themselves are the mechanism that carries changes to the read side. Every read model is a projection of the event log. This is where CQRS and event sourcing meet, and it is the configuration most people mean when they talk about the two together. The useful observation is that the third level does not require the fourth. A conventional system can publish changes to its read stores using the transactional outbox pattern, described in Chapter 8, or change data capture. But once the write side is event-sourced, the fourth level arrives almost automatically, because an event store is poor at ad hoc queries and the events are already a perfect change feed. CQRS without event sourcing A concrete case shows the value of CQRS on its own. Consider an online retailer's product catalogue. Products are created and edited by a merchandising team through a back-office application: a few hundred edits a day, each subject to rules about pricing approval, regulatory labeling, and supplier data. Products are browsed by customers through a storefront: millions of searches a day, with faceted filters, full-text relevance, personalized ranking, and strict latency targets. These two workloads have almost nothing in common. The back office needs transactional integrity, rich validation, and a normalized model of products, variants, suppliers, and prices. The storefront needs a denormalized document per product, pre-joined with its category path, its current price in each currency, its stock status per region, and its review summary, indexed for search. Forcing both through the same model would serve neither. The standard design separates them. Merchandisers issue commands against a conventional relational write model. Each successful change writes a message to an outbox table in the same transaction. A relay process reads the outbox and updates the search index. The storefront queries only the index. If the index is a second or two behind the write model, nobody notices, because a merchandiser editing a product description does not expect customers to see it within milliseconds. There is no event sourcing here, and there need not be. The merchandising domain does not require a history of every edit to be the authoritative record. But the separation of models brings real benefits: each side scales independently, each uses technology suited to its workload, and changes to the storefront's query needs do not ripple into the merchandising rules. Why event sourcing almost demands CQRS Running in the other direction, an event-sourced write model nearly forces read-model separation. The event store is organized around streams and appends. It is very good at answering "give me all events for claim C-1042" and "give me all events after global position N." It is poor at answering "which claims assigned to t.reyes have a reserve over 50,000," because answering that would mean loading and folding every claim stream. Some event stores offer limited query features over event data. Marten, because it sits on PostgreSQL, allows queries over the raw event table using JSONB operators, and some teams use that for operational investigations. KurrentDB offers server-side projections written in JavaScript that can partition and aggregate events into new streams. But neither is a substitute for a read model designed for a particular query, and practitioners generally warn against building application features on direct queries of the event store. So every event-sourced system needs a mechanism for building and maintaining read models from events, which is the subject of the next chapter. Once that mechanism exists, the system has CQRS whether or not anyone used the term. Designing read models Once the read side is free of the write model, a different set of design habits applies, and they can feel wrong to developers trained in normalization. The guiding rule is to design each read model for the queries it serves, often one read model per screen or per report. The adjuster's claim list needs claim number, policyholder name, loss date, status, current reserve, and days open, sorted by days open. A read model for that screen is a single flat table with exactly those columns and an index on the sort key. Rendering the screen is one query with no joins. The claim detail page needs much more, including the reserve history and the list of documents, and gets its own read model, perhaps a single document per claim. The finance team's reserve report needs reserves aggregated by line of business and month, and gets a third. This means the same fact, the current reserve of claim C-1042, is stored in several places. In a normalized design that duplication would be a defect, because updates might leave copies inconsistent. On the read side it is not a defect, because no one updates read models directly. Each copy is maintained by its own projection from the same events, and if one is ever wrong it can be rebuilt. The source of truth is singular; the read models are deliberately plural. Choosing a store for each read model follows the same logic. A relational table suits most lists and lookups. A document store suits detail views where a whole aggregate is read at once. A search engine suits full-text and faceted queries. An in-memory cache suits small, hot reference data. A columnar or analytical store suits reports that scan large volumes. Nothing requires every read model to live in the same technology, and nothing requires them to live in different ones; most systems start with one relational database holding all read models and add specialized stores only when a query's needs exceed what that database does well. Read models should also be kept free of business rules. A projection that computes whether a claim is overdue by comparing dates against a service-level threshold is embedding a rule that belongs to the domain. When the threshold changes, the projection must be updated and rebuilt, and the rule now lives in two places. The better pattern is for the domain to record the relevant facts, for instance a ClaimBecameOverdue event raised by a scheduled process, and for the projection to reflect them. Where a derived value is purely presentational, such as days open computed from the report date, computing it at query time is often simpler than storing it. The real cost: stale reads The defining cost of CQRS beyond the first level is that the read model can be behind the write model. Udi Dahan's "Clarified CQRS" made a point that remains the clearest framing of this cost: data shown to a user is stale the moment it is displayed. Even in a fully synchronous CRUD system, another user may change the record a second after the screen renders. Dahan's argument was that since staleness is inherent, systems should be designed to tolerate it explicitly rather than pretending to eliminate it, and that the business, not the database, should decide how stale is acceptable for each piece of data. That framing is liberating, but it does not make the user experience problem disappear. The hardest case is read-your-own-writes: a user performs an action and is then shown a screen that does not reflect it. An adjuster increases a reserve, the page refreshes, and the old figure appears because the projection has not yet processed the event. To the adjuster the system looks broken. Several techniques address this, and mature systems usually combine them. The simplest is to update the screen optimistically from the command result. The client knows what it asked for; if the command succeeded, it can display the new reserve without asking the read model. This works well for the user who made the change and does nothing for anyone else, which is usually fine. The second is to wait for the projection. The command returns the global position or stream revision of the events it wrote. The query carries that position and the read side waits, up to a timeout, until its projection has processed at least that far before answering. Marten, for example, supports projections that run inline within the same transaction as the event append, which makes the read model immediately consistent for those projections at the cost of slower writes; asynchronous projections expose their progress so a caller can check how far they have reached. This gives exact read-your-writes semantics at the cost of some latency on the read. The third is to design the interface around the delay. When a user submits a claim, the confirmation screen says the claim has been received and shows its reference, rather than immediately displaying the full claim detail. By the time the user navigates to the detail, the projection has caught up. This is not a trick; it is often a more honest representation of a process that genuinely takes time. The fourth, used for read models that must be immediately consistent with the write, is to update the read model in the same transaction as the write, which is the second level of the spectrum above. This works only when the write and read stores share a transaction, and it couples them, but for one or two critical views it can be the pragmatic answer. Where CQRS goes wrong Three failure patterns recur often enough to name. The first is CQRS everywhere. A team adopts the pattern as a house architecture, and every bounded context, including simple ones with no divergence between read and write needs, gets separate command and query stacks, a message bus, and asynchronous projections. The result is a large amount of infrastructure that provides no benefit for most of the system and makes simple changes slow. Fowler's warning about applying it to portions of a system is aimed at exactly this. The second is ceremonial CQRS: splitting code into command and query handler classes, often through an in-process mediator library, while keeping a single shared model and a single set of tables. There is nothing wrong with organizing code this way, and it corresponds to the first level of separation. The mistake is believing that the architecture now has the properties of the higher levels, independent scaling, specialized read stores, or an audit trail, when it does not. The third is leaking the read model into decisions. A command handler, needing to check a rule, queries a read model instead of loading the aggregate. Because the read model may be stale, the decision can be made on outdated information, and because the read model is not protected by the aggregate's concurrency check, two concurrent decisions can both pass. Sometimes this is acceptable, for instance a soft check that a referenced entity exists, where the worst case is a dangling reference that a later process cleans up. But for invariants, decisions must be made on the write model. A useful rule of thumb: read models may inform the user who issues a command, but the write model decides whether the command succeeds. A small idea, used precisely The case for CQRS is strongest where three things are true: the rules for changing data are complex enough to deserve a dedicated model, the ways of reading data are varied or demanding enough to deserve their own optimized structures, and the business can tolerate, or the interface can absorb, a short delay between a change and its appearance everywhere. In an event-sourced system the first and third conditions are already part of the bargain, which is why the two patterns are so often found together. What CQRS gives an event-sourced system in particular is freedom. Because read models are derived from events and can be rebuilt, the team is never stuck with the read structures it designed on day one. A new report, a new search experience, a new machine-learning feature table can each be added later by writing a new projection and replaying history into it. That freedom, and the engineering needed to exercise it safely, is the subject of the next chapter. Chapter 5: Projections and Materialized Views A projection is a piece of code that reads events and maintains some other representation of the data: a table, a document, a search index entry, a cache, a file. The term is borrowed loosely from geometry, where a projection shows a three-dimensional object from one angle. Each projection shows the event log from one angle, shaped for one set of questions. The adjuster's work queue is one projection. The monthly reserve movement report is another. The fraud model's feature table is a third. None of them is authoritative; all of them can be deleted and rebuilt from the events. That last property is the source of most of event sourcing's practical power, and also of much of its operational burden. This chapter covers how projections work, how they keep track of what they have processed, how they fail, and how they are rebuilt. The anatomy of a projection At its core a projection is the same kind of fold as an aggregate's evolve function, with two differences. It typically reads events from many streams rather than one, and its state is stored durably in a query-friendly form rather than held in memory for a single decision. A projection for the adjuster work queue might look like this: on ClaimReported(e): insert into work_queue (claim_id, policy, status, reserve, adjuster) values (e.claimId, e.policy, 'open', 0, null) on AdjusterAssigned(e): update work_queue set adjuster = e.adjuster where claim_id = e.claimId on ReserveIncreased(e): update work_queue set reserve = e.newAmount where claim_id = e.claimId on ClaimClosed(e): delete from work_queue where claim_id = e.claimId Each handler is small and specific. The projection ignores events it does not care about. It does not enforce business rules, because the rules were enforced when the events were written; a projection's only job is to reflect facts faithfully. If ReserveIncreased arrives for a claim the projection has never heard of, that is a sign of a bug or an ordering problem, not a business error. Projections come in a few common shapes. A per-stream projection builds one read-model document per stream, such as a claim detail view that summarizes everything about one claim. A multi-stream projection groups events from many streams by some key, such as a per-adjuster summary that counts open claims and total reserve, updated by events from every claim assigned to that adjuster. An event projection reacts to each event independently, for instance writing a row to a flat audit table or forwarding a notification. Marten names these categories explicitly in its projection API, distinguishing single-stream projections, multi-stream projections, and event projections, and lets each be run in one of three lifecycles: inline, within the same transaction as the event append; asynchronously, by a background daemon; or live, computed on demand from events without being stored. Other frameworks use different names for the same ideas. Getting events to projections There are two broad ways to feed events to a projection: in the same transaction as the write, or afterwards. Inline projections run as part of the append. When the command handler appends ReserveIncreased, the same database transaction updates the work queue row. The read model is never stale. The costs are that every write pays for every inline projection, that a bug in a projection can cause writes to fail, and that inline projection is only possible when the event store and the read store share a transaction, which essentially means a relational store holding both. Inline projection is the right choice for a small number of read models that must be strictly consistent with the write side, and a poor default for everything. Asynchronous projections run separately, reading events after they are committed. This is the common case, and it requires a subscription mechanism. KurrentDB offers two kinds. A catch-up subscription is driven by the client: the client says "give me events from position N onward," the server sends historical events until the client is caught up and then continues to push new events as they arrive. The client is responsible for remembering where it got to. A persistent subscription is managed by the server: the server tracks each consumer group's position, delivers events to competing consumers, and expects acknowledgments, much like a message queue. Persistent subscriptions make it easy to scale out processing, but because events can be delivered to different consumers concurrently, strict ordering is not guaranteed, which matters for projections that depend on order. KurrentDB's documentation is explicit that persistent subscriptions do not guarantee ordering, which is why catch-up subscriptions are the usual choice for building read models. In stores built on relational databases, the asynchronous mechanism is typically a polling loop: a background process repeatedly queries the event table for events with a global position greater than its last checkpoint. Marten's async daemon works this way. Polling is simple and robust, with latency set by the polling interval and by how quickly new events become visible. Table 3 compares these delivery modes on the properties that matter most when choosing. Table 3. Ways of delivering events to projections. Mode Read model freshness Ordering Write cost Typical use Inline, same transaction Immediate Exact Higher per write Few critical views Catch-up subscription Milliseconds to seconds Global order preserved None Most read models Persistent subscription Milliseconds to seconds Not guaranteed across consumers None Parallel work, side effects Polling a relational log Polling interval plus commit lag Needs gap handling None Relational event stores Checkpoints and the at-least-once problem An asynchronous projection must remember how far it has read, so that after a restart it resumes from the right place. This remembered position is the checkpoint. Getting checkpoints right is the core reliability problem of projections. Suppose a projection processes event 1,000, updates its read model, and then crashes before writing the checkpoint. On restart it reads its last checkpoint, 999, and processes event 1,000 again. If the handler for that event is update work_queue set reserve = 65000, applying it twice is harmless. If the handler is update adjuster_summary set open_claims = open_claims + 1, applying it twice corrupts the read model. This is the general condition of asynchronous processing: delivery is at least once, and handlers must cope with duplicates. There are two reliable remedies. The first is to make handlers idempotent, meaning applying them twice has the same effect as applying them once. Setting a value is naturally idempotent; incrementing a counter is not. Many projections can be written in idempotent form by computing absolute values rather than deltas, or by recording which events have been applied, for example by storing the last processed stream revision on each read-model row and ignoring events at or below it. The second, and stronger, remedy is to store the checkpoint in the same transaction as the read-model update. If the projection writes to a relational database, it can update the read model and its checkpoint row together; either both happen or neither does, and a crash can never leave them inconsistent. This gives effectively exactly-once processing for that projection. It requires that the checkpoint and the read model live in the same transactional store, which is a strong argument for keeping each projection's state and checkpoint together. When the read model lives in a store without transactions shared with the checkpoint, such as a search index, the projection must fall back on idempotency. Writing each search document with a full replacement keyed by entity identifier, carrying the stream revision as a version number so that older updates cannot overwrite newer ones, is the standard approach. Ordering and the gap problem Projections often depend on processing events in the order they were committed. A work queue projection that sees ClaimClosed before ClaimReported for the same claim will either fail or produce nonsense. Within a single stream, every serious event store guarantees order. Across streams, the guarantee depends on the store. A dedicated event database with a single writer per log, as KurrentDB has with its leader node, can assign global positions in commit order. A relational store using an auto-incrementing sequence for global position has a subtle problem: sequence values are assigned when a transaction inserts a row, not when it commits. Transaction A might take position 100 and transaction B position 101, but B commits first. A projection polling at that instant sees 101 but not 100, records its checkpoint as 101, and never processes event 100 when A later commits. The event is silently skipped. This is not a theoretical concern. It is a known issue for event stores built on PostgreSQL sequences and similar mechanisms, and implementations handle it in different ways. One approach is to detect gaps in the sequence and wait a short time for them to fill before advancing past them, on the assumption that a gap is either an in-flight transaction or a rolled-back one; Marten's async daemon uses a high-water-mark mechanism along these lines. Another is to serialize appends through a lock so that sequence order and commit order coincide, trading write throughput for simplicity. A third, available in PostgreSQL, is to read only events whose transaction identifiers are older than the oldest still-running transaction, using the database's own snapshot information. Anyone building an event store on a relational database should understand which approach their library uses, because the failure mode, an event that some projections never see, is quiet and very hard to diagnose after the fact. Not every projection needs global order. A per-stream projection needs only per-stream order, which is always guaranteed. A multi-stream projection that aggregates commutative quantities, such as totals, may tolerate reordering. Knowing which projections are order-sensitive lets a team parallelize the ones that are not. Rebuilding The defining capability of a projection is that it can be thrown away and rebuilt from events. This is how bugs in read models are fixed, how new read models are added, and how read-model schemas change. The mechanics are simple: create a fresh, empty read model, reset the projection's checkpoint to zero, and let it process the entire event history. When it catches up to the present, it is current. The engineering challenges are about doing this without disrupting users. Rebuilding in place, by truncating the live read model and replaying into it, means the read model is empty or partial for the duration of the replay. For a small store that might be seconds; for a store with hundreds of millions of events it can be hours. The standard technique is a blue-green rebuild: build the new version of the read model alongside the old one, under a different table or index name, while the old version continues to serve queries. When the new version has caught up to the live position, switch queries to it, then retire the old one. This also gives a clean rollback path if the new version turns out to be wrong. Replay speed becomes a design concern at scale. Projections that perform a database round trip per event will be slow to rebuild. Batching writes, processing events in memory and flushing periodically, and disabling secondary indexes during the bulk load until the replay completes can improve rebuild times by orders of magnitude. Some teams maintain a rough budget for full rebuild time as a system-level quality target, because a read model that takes a week to rebuild is a read model that will not be fixed when it needs to be. Rebuilds also expose a less obvious coupling. A projection that calls external services, for example looking up a customer's name from another system when processing ClaimReported, will make those calls again on every replay, perhaps millions of them, and will get today's answers rather than the answers that were true when the event was written. Projections should depend only on the events, or the facts needed should be included in the events at write time. Side effects, such as sending emails, must never live in a projection that may be replayed; they belong in processes that track their own progress separately and are not rebuilt, as Chapter 8 explains. Running projections in production Projections are long-running background processes, and they need the same operational attention as any other. The most important metric is lag: the distance between the latest event in the store and the projection's checkpoint, measured in events and in time. A projection that is normally a few hundred milliseconds behind and suddenly falls minutes behind indicates a problem: a slow downstream store, an expensive handler, a poison event that fails repeatedly. Lag should be monitored and alerted on per projection. Poison events deserve a policy. If a projection handler throws on a particular event, perhaps because the event contains data the handler did not anticipate, the projection cannot advance past it. Options include retrying with backoff, skipping the event and logging it for investigation, or parking the event in a dead-letter location. Skipping silently is dangerous for projections that must be complete; stopping is dangerous for projections users rely on. The right choice depends on the projection, and it should be made deliberately. Running more than one instance of a projection for availability requires coordination so that two instances do not both process the same events into the same read model. Leader election, distributed locks, or the store's own subscription features handle this. Marten's async daemon, for example, can distribute projections across application nodes and uses database locks to ensure each projection runs in only one place at a time. The freedom projections buy It is worth returning to why all this machinery is worth having. In a CRUD system, the shape of stored data is the shape of the read model, and changing it is a migration. Adding a new report that needs historical data the schema never kept is impossible. In an event-sourced system, a new projection can be written today and replayed over every event since the system went live, producing a read model that looks as if it had been maintained from the start. In the claims domain, this is how a team answers the actuary who arrives two years after launch wanting to analyze the development of reserves over each claim's lifetime, split by the basis given for each change. Nobody built that report at launch. Nobody anticipated it. But every ReserveIncreased and ReserveDecreased event carried its basis, so a new projection can produce the analysis from complete history. The value of event sourcing is often realized in exactly this way, as answers to questions nobody thought to ask, and it is realized only if the events were designed well enough to answer them. Chapter 6: Time Travel: Auditing, Temporal Queries, and Debugging The capability most often used to sell event sourcing is the ability to travel back in time: to reconstruct the state of any entity, or the whole system, as it stood at any past moment. The capability is real. It is also more subtle than the sales pitch suggests, and teams that rely on it without understanding its limits are sometimes surprised by what it cannot do. This chapter examines what temporal reconstruction delivers, how to make it efficient, how it supports auditing and debugging, and where the boundaries lie. Reconstructing the past The mechanism follows directly from the storage model. Current state is a fold over all events in a stream. State as of any earlier point is a fold over the events up to that point. The point can be specified as a stream revision ("the claim after its fifth event"), as a global position ("everything the system knew when event 4,211,980 was written"), or as a timestamp ("the claim as at the end of the business day on 31 March"). Event stores make this straightforward. KurrentDB lets a client read a stream forwards from the start and stop at any revision. Marten provides methods to aggregate a stream to a specific version or to a specific timestamp, so that a single call can return a claim document as it stood at a given moment. Axon's event store can be read up to a given point in the same way. In a hand-built relational store the query is a simple filter on stream identifier and either version or recorded time, ordered by version. What this gives you, for a single entity, is complete and exact. The claim as at 31 March is not an approximation reconstructed from audit logs; it is the result of applying exactly the facts the system had accepted by that moment, in order. If the regulator asks what reserve was held and who approved it, the answer comes from the same events that drive the live system, which is a much stronger form of evidence than a separately maintained audit trail that might have drifted. For system-wide questions, the picture is more involved. Reconstructing "every open claim and its reserve as at 31 March" means folding every claim stream up to that date. For a small system this can be done on demand. For a large one, it is a job for a projection: a read model built by replaying events up to the chosen point and stopping there. Because projections can be rebuilt from any position, a team can produce a historical read model for any date, or maintain period-end snapshots of key read models as a matter of routine. Finance teams in event-sourced systems often do the latter for month-end and year-end positions, because those figures are consulted constantly and should not depend on replaying history every time. Two kinds of time Reconstructing the past raises an immediate question: which past? There are at least two timelines in any business system, and confusing them produces wrong answers. The first is recorded time, also called transaction time or system time: when the system learned about something. The metadata timestamp on an event, set when the store accepted it, captures recorded time. The second is effective time, also called valid time or business time: when something was true in the world. A policyholder's change of address might be effective from the first of the month but entered into the system on the tenth. A reserve increase might be backdated to the date the survey was received. A premium adjustment might apply from the policy's renewal date. A state-based system typically conflates the two, storing only the current belief about effective dates and overwriting it when corrected. An event-sourced system records every event with its recorded time automatically, and can record effective times as explicit payload fields. The combination supports bitemporal queries: what did we believe, as of recorded time T1, about the state of the world at effective time T2? Martin Fowler's article "Bitemporal History," published on his website in 2021, gives a clear account of why both dimensions are needed, using the example of a salary change that is entered late and must be reflected in payroll calculations already run. The insurance case makes the value concrete. Suppose a payment is recorded on 5 April but is effective from 28 March, because the adjuster authorized it before quarter-end and the entry was delayed. The quarter-end financial report, run on 1 April, did not include it. Was the report wrong? With bitemporal data, the question has a precise answer: as known on 1 April, the quarter-end position excluded the payment; as known today, the quarter-end position includes it. Both are true. The first explains the report that was filed; the second is what a restated report would show. Auditors and regulators care about both, and a system that keeps only one timeline cannot provide the other. Fowler's catalogue of patterns also includes the retroactive event: an event that corrects the past by asserting that something happened at an earlier effective time than when it was recorded. In an event-sourced system, retroactive events are just events with an effective date earlier than their recorded date. Projections that care about effective time must handle them by reprocessing affected periods, which is more work than handling events in order but is far more tractable than reconstructing history from overwritten rows. The practical lesson is to decide, when designing each event, which effective times it needs. Many events take effect at the moment they are recorded and need no separate field. Those that can be backdated, scheduled, or corrected should carry their effective time explicitly, with a name that says what it means: effectiveFrom, lossDate, valueDate. Leaving effective time implicit in the recorded timestamp is a decision that cannot be revisited later without guessing. What auditing actually requires Event sourcing is frequently described as giving auditability for free. It gives a great deal, but auditors and regulators tend to ask four questions, and it is worth checking each against what an event store provides. What happened, and in what order? This is the strength of the pattern. Every accepted change is recorded as a named business fact, in order, with nothing overwritten. Who did it, and on whose authority? This is available only if the events or their metadata record it. Metadata should carry the authenticated user or process identity, and events representing decisions that require authority, such as reserve approvals, should name the approver in the payload. A store that records perfect history of anonymous changes answers the first question and fails the second. Why? Events capture intent through their names and payloads, which is far more than an audit table offers. But the reason behind a decision, the survey reference, the policy clause, the free-text justification, is present only if the event was designed to carry it. Can the record be trusted? Here the common claim of event sourcing needs qualification. An append-only store is append-only by convention and by API. Nothing in a typical event store prevents someone with direct database access from editing or deleting rows. For many organizations, the combination of append-only application semantics, database access controls, and backups is sufficient. Where it is not, for example where records may be used as legal evidence or where insiders are part of the threat model, stronger guarantees are needed. Some teams add hash chaining, storing with each event a cryptographic hash of the event and the previous event's hash, so that any later alteration breaks the chain and is detectable. Others periodically publish a digest of the log to an external system, or write events to storage with write-once retention policies of the kind offered by cloud object stores. These measures are not specific to event sourcing; the point is that event sourcing gives a natural place to apply them, because the log is already the authoritative record. There is also a subtle point about completeness. An event store records what the system accepted. It does not record commands that were rejected, unless the designers chose to record them. For some audit purposes, such as detecting attempted fraud or understanding user behavior, rejected attempts matter. Recording them is easy, either as events in a separate stream or in an ordinary log, but it has to be decided. Debugging with history The second great use of time travel is debugging. When a production entity ends up in a state that should be impossible, a CRUD system offers the current row and whatever logs survived. An event-sourced system offers the complete sequence of facts that produced the state. The typical workflow is direct. Copy the problematic stream from production into a test environment, or into a unit test as a list of given events. Fold it and confirm that the resulting state reproduces the problem. Then step through the events one at a time, inspecting state after each, until the point where it diverges from expectation. Either an event was accepted that should have been rejected, which points to a bug in decide, or an event was applied incorrectly, which points to a bug in evolve, or every event is individually reasonable and the problem is a business rule nobody anticipated. In each case the given-when-then test structure from Chapter 3 turns the reproduction into a permanent regression test. Projections can be debugged the same way. A read model that disagrees with the write model can be rebuilt from scratch in a test environment and compared, which immediately tells you whether the problem is in the projection logic or in incremental processing, such as a skipped event or a duplicate. The approach extends to what-if analysis. Because events are data, a team can replay production history through modified code to see what would have happened under a new rule. A proposed change to the reserve-approval thresholds can be run against a year of real claim events to count how many decisions would have changed. This kind of analysis is possible in other architectures but requires reconstructing inputs from scattered sources; here the inputs are already recorded in order. The limits of replay The time-travel capability has a boundary that is easy to miss. Replaying events reproduces state. It does not reproduce decisions. When the claim's events are folded today, the evolve function used is today's code. If evolve has changed since the events were written, for instance because a bug in how reserve decreases were applied was fixed last month, the reconstructed state as at last year may differ from what the system actually believed last year. Usually this is what you want: you want the correct interpretation of the facts. But if the question is "what did the system show the adjuster on that day," the answer may require the old code, or a read model snapshot taken at the time. Similarly, the decisions recorded in events were made by the decide function as it existed at the time, using whatever external information was available. Replaying history through today's decide will not necessarily produce the same events, because rules change. This is why events record outcomes rather than commands. If you stored only commands and re-executed them to rebuild state, any change to the rules would silently rewrite history. Storing the events, the outcomes as decided at the time, makes the history stable regardless of how the rules evolve. Young has made this distinction central in explaining why event sourcing stores events and not commands, and it is the reason the pattern is robust over time. The practical guidance is to be clear about which question is being asked. "What facts had been established by date D?" is answered exactly by folding events up to D. "What was displayed or reported on date D?" is answered exactly only if the read model was retained or if both the events and the projection code of that time are available. Regulated industries that need the second answer often archive period-end read models or reports as immutable documents for that reason. Snapshots A final topic belongs here because it is so often misunderstood. A snapshot is a saved copy of an aggregate's state at a particular stream revision, used to avoid replaying the whole stream on every load. To load an entity with a snapshot, the system reads the latest snapshot, then reads and applies only the events after it. Snapshots are an optimization, and they are needed less often than people expect. Folding a few hundred events in memory takes well under a millisecond on ordinary hardware, and the time to load a stream is usually dominated by the database round trip, not by the fold. Many practitioners, Young among them, argue that teams should not add snapshots until they have measured a problem, and that a stream long enough to need snapshots is often a sign that the aggregate has an unbounded lifecycle that should be modeled differently, for example by closing periods as described in Chapter 2. When snapshots are warranted, they bring their own obligations. A snapshot is a serialized form of the aggregate's internal state, so it must be versioned: when the aggregate's state shape changes, old snapshots must be discarded and regenerated, which is safe because they can always be rebuilt from events. Snapshots should never be treated as authoritative; if a snapshot and the events disagree, the events win. And snapshots should never be used for time travel in place of events, because a snapshot reflects the code that produced it rather than the current interpretation of the facts. Used with that discipline, snapshots are a useful tool for a minority of aggregates. Used as a default, they add a second representation of state to every aggregate, with all the attendant versioning work, to solve a problem most aggregates never have. Time travel, in short, is a genuine capability built on a simple mechanism. Its value depends on decisions made long before anyone travels: whether events carry authority, reasons, and effective times; whether the store is protected in proportion to the trust placed in it; and whether the team understands that the past is reconstructed through the lens of present code. The hardest of those decisions concerns events whose meaning or shape must change over time, which is the subject of the next chapter. Chapter 7: Evolving Events: Versioning and Schema Change Every system changes. Business rules shift, regulations arrive, products are redesigned, and developers discover that the model they built in the first year was wrong in ways nobody could have foreseen. In a state-based system these changes are absorbed by migrations: the schema is altered, existing rows are transformed, and the old shape disappears. In an event-sourced system the old shape never disappears. Events written three years ago under a different understanding of the domain are still in the store, still read by every replay, still part of the authoritative record. The code must live with them forever, or something must translate them. This is, by the testimony of practitioners, the hardest part of running an event-sourced system over time. When Michiel Overeem, Marten Spoor, Slinger Jansen, and Sjaak Brinkkemper studied 19 event-sourced systems through interviews with 25 engineers, published in the Journal of Systems and Software in 2021, the five challenges practitioners reported were event system evolution, a steep learning curve, a lack of available technology, rebuilding projections, and data privacy. Evolution came first. Greg Young devoted an entire book to the subject, Versioning in an Event Sourced System, published on Leanpub around 2017, and it remains the most thorough practical treatment. This chapter sets out the kinds of change that occur, the tactics available for each, and the principles that make evolution manageable. Kinds of change Not all changes are equal. It helps to sort them by how much they disturb existing events. Additive changes add information to an event type: a new optional field on ClaimReported recording the channel through which the claim arrived. Old events lack the field; new events have it. Readers that treat the field as optional can handle both. Structural changes alter the shape of information already present: renaming a field, splitting a single address string into structured components, changing an amount from a decimal number to an integer count of minor currency units, or nesting fields that were previously flat. The meaning is unchanged, but the representation is incompatible. Semantic changes alter what an event means. Perhaps ClaimClosed originally meant "settled and paid" but the business later introduced a distinction between closing with payment and closing without payment, and every existing ClaimClosed event is now ambiguous. Semantic changes are the hardest, because no mechanical transformation can recover information that was never recorded. Granularity changes split or merge events or streams: what was one PolicyAmended event becomes separate CoverageAdded, CoverageRemoved, and DeductibleChanged events; or what was one stream per policy becomes one stream per policy term. Mistakes are a category of their own: events that were written with wrong data because of a bug, events that should never have been written, or events written to the wrong stream. Each kind calls for different tactics, and most mature systems use several. The tactics The 2021 study identified five tactics that practitioners use for schema evolution: versioned events, weak schema, upcasting, in-place transformation, and copy-and-transform. These map closely onto the options Young describes, and onto the support offered by the main frameworks. Table 4 summarizes them before each is discussed. Table 4. Tactics for evolving stored events. Tactic How it works Store changes? Best suited to Main cost Weak schema Readers tolerate missing or extra fields No Additive changes Defaults must be sensible Versioned events New event type or version alongside old No Structural changes Code handles every version Upcasting Old events transformed on read No Structural, some granularity Transform runs on every read Copy-and-transform New store written from old New copy Large restructurings Operational effort, cutover In-place transformation Stored events rewritten Yes Rare corrections Destroys the original record Weak schema The cheapest tactic is to design events and readers so that most changes need no special handling. A weak schema approach serializes events in a format that tolerates missing and extra fields, typically JSON, and writes readers that map whatever fields are present and supply defaults for absent ones. This is the tolerant reader principle applied to events. Adding the channel field to ClaimReported requires only that readers treat a missing channel as "unknown." Young argues in his book that weak schema with mapping, rather than strict deserialization into exact types, should be the default for most systems, because it makes the most common kind of change, adding information, nearly free. The discipline it requires is that field names never be reused with a different meaning and that removed fields are simply ignored rather than repurposed. Strongly typed schema formats such as Apache Avro and Protocol Buffers offer a related approach with explicit compatibility rules: fields have numbers or names and default values, and a schema registry can verify that a new schema is backward compatible with old data before it is deployed. Teams already using such formats for messaging often extend them to stored events. The trade-off is more rigor and tooling in exchange for automatic compatibility checking. Versioned events When a change is not additive, one option is to introduce a new event type, or a new version of the existing type, and leave the old one in place. ClaimReported becomes ClaimReported_v2 with a different structure. New code writes only v2. Every reader, the aggregate's evolve function and every projection, must handle both v1 and v2. This is explicit and safe: no stored data changes, and the history remains exactly as written. The cost is that version handling spreads through the codebase. After a few years, an aggregate might handle four versions of one event type, and every new projection must be written to understand all four. That accumulation is why versioned events are usually combined with upcasting. Upcasting An upcaster is a function that transforms an old version of an event into a newer version as the event is read from the store, before it reaches any application code. Aggregates and projections see only the latest version. The stored events are untouched. upcast ClaimReported_v1 -> ClaimReported_v2: v2.claimId = v1.claimId v2.policyId = v1.policy v2.lossDate = v1.lossDate v2.channel = "unknown" v2.lossAddress = parseAddress(v1.lossAddressText) Upcasters can be chained, v1 to v2 to v3, so that each transformation is written once. Axon Framework has long supported upcasting as a first-class feature, with upcasters registered in a chain that the event store applies on read, and its documentation treats upcasting as the primary mechanism for event versioning. Marten also supports upcasting, transforming old event payloads into new types during deserialization. In KurrentDB, which stores event payloads opaquely, upcasting is done in the client application's deserialization layer. Upcasting can also handle some granularity changes: an upcaster that receives one old PolicyAmended event can emit several new fine-grained events. Merging is harder, because an upcaster operating on one event at a time cannot easily combine several. The limits of upcasting are worth stating. Upcasters run on every read, including every replay of every projection, so expensive transformations slow everything. Upcasters must be pure functions of the old event; an upcaster that looks up external data to fill in a missing field will produce different results at different times. And upcasting cannot solve semantic changes, because no function of the old event can produce information the old event never contained. When ClaimClosed must be split into ClaimClosedWithPayment and ClaimClosedWithoutPayment, an upcaster can only make that determination if the information exists somewhere in the old events, perhaps by checking whether a payment event preceded the closure in the same stream. That requires looking at the stream as a whole, which is beyond a simple per-event upcaster. Copy-and-transform For larger restructurings, the most powerful tactic is to build a new event store from the old one. A migration process reads every event from the old store, transforms it, and writes the result to a new store, applying whatever logic is needed: splitting streams, merging events, inferring missing information by examining whole streams, correcting known bugs. When the new store has caught up, the application is switched to it. Young describes this as copy-replace and recommends it for changes that the other tactics cannot handle cleanly, while noting that it is an operation, not a code change, and must be planned like one. The old store is retained, unchanged, as an archive, so the original record is never destroyed. For audit purposes the transformation logic should itself be retained, so that anyone examining the new store can see exactly how it was derived from the old. The operational challenge is cutover in a system that is still receiving writes. The usual approach mirrors a blue-green projection rebuild: run the migration while the old store continues to accept writes, have the migration follow the old store's tail until it is nearly caught up, briefly stop writes, let the migration finish, and switch. With large stores this can be a substantial undertaking, which is why teams prefer to accumulate structural changes and perform copy-and-transform migrations rarely. In-place transformation The last tactic, rewriting events in the existing store, is the one most at odds with the principles of event sourcing. It destroys the original record, so it undermines the audit value of the log, and it is dangerous because a bug in the transformation corrupts the only copy. Most practitioners reserve it for narrow cases: removing data that must be legally erased, or correcting events whose content is so broken that no reader can process them. When it is used, it should be done with a full backup and with the transformation logged. A semantic change worked through The tactics are easier to judge against a concrete case, and the hardest kind of change, a semantic one, makes the best example. Suppose the claims system was launched with a single ClaimClosed event. Two years in, the business introduces a distinction it now considers essential: a claim can be closed with a settlement, closed without payment because the loss was not covered, or closed because the policyholder withdrew it. Reporting to the regulator must now distinguish the three, including for historical claims. The team introduces three new event types, ClaimClosedWithSettlement, ClaimClosedAsNotCovered, and ClaimWithdrawn, and stops writing ClaimClosed. The question is what to do with the hundreds of thousands of ClaimClosed events already in the store. Weak schema does not help, because the issue is not a missing field but a missing distinction. Versioned events alone would leave every projection forever handling the ambiguous old type alongside the three new ones, with no way to classify the old ones. The team examines the history and finds that, in practice, the distinction can be inferred for most claims. A ClaimClosed event preceded in the same stream by a PaymentMade event was a settlement. One preceded by a CoverageDeclined event was a not-covered closure. One preceded by a WithdrawalRequested event was a withdrawal. That leaves a residue of claims, perhaps a few percent, where the history is genuinely ambiguous, for instance claims closed by a supervisor without any of those preceding events. A per-event upcaster cannot make this inference, because it sees only the ClaimClosed event, not the events before it. A stream-aware upcaster, one that processes whole streams on read, could, but it would repeat the analysis on every replay of every projection. The cleaner path is copy-and-transform. A migration reads each claim stream, classifies its closure using the rules above, writes the appropriate new event in place of ClaimClosed, and for the ambiguous residue writes an explicit ClaimClosedReasonUnknown event rather than guessing. The old store is archived unchanged. The migration code, and a report listing every claim it classified and on what basis, are retained alongside it. Three points from this example generalize. First, semantic changes are best handled by making the inference once, explicitly, in a migration that can be reviewed, rather than repeatedly in code that runs on every read. Second, where information genuinely does not exist, the honest answer is an event that says so; inventing a classification would corrupt the record the whole architecture exists to protect. Third, the residue of ambiguous cases is itself a lesson for future event design. Had the original ClaimClosed carried a reason field, even a free-text one, the migration would have been trivial. Capturing why, not just what, is cheap at the time and invaluable later. Correcting mistakes Mistakes deserve separate treatment because the natural instinct, to fix the bad events, is usually wrong. If an event recorded wrong data, the business-appropriate response is almost always a compensating event: a new fact that records the correction. If a reserve was set to 650,000 instead of 65,000 because of a typing error, the correct response is a ReserveCorrected event, not an edit to the original. The history then shows both the error and its correction, which is precisely what an auditor wants to see, and what the accountant's reversing entry achieves. Projections that compute current state will reflect the corrected amount. Projections that care about history, such as an analysis of reserve accuracy, can distinguish corrections from genuine revisions. If an event was written because of a software bug and has no business meaning, for example a duplicate PaymentMade caused by a retry that the code failed to deduplicate, the situation is more nuanced. A compensating PaymentReversed is appropriate if the duplicate had real-world effects, such as money actually leaving an account. If it had none, some teams prefer to filter it out with an upcaster or a copy-and-transform rather than polluting the business history with corrections of technical errors. Either choice is defensible; the important thing is that it be documented. Principles that make evolution manageable Beyond the tactics, a handful of practices reduce how often hard changes are needed. Design events for meaning, not for current code. Events that capture business facts in domain language age better than events that mirror the structure of today's classes. A ReserveIncreased event with a clear amount and basis will be readable for decades; a serialized dump of an internal object graph will become unreadable the first time the object graph is refactored. Keep events small and specific. A large event with many fields is more likely to need structural change than a small one, because any change to any field affects it. Fine-grained events also make it easier to introduce new event types alongside old ones. Never reuse a name with a different meaning. If the meaning of a field or event changes, give it a new name. Silent reinterpretation is the most dangerous kind of change because nothing fails; projections simply begin producing wrong results. Separate internal events from published events. Events that stay within one service's store can be evolved with that service's code. Events published to other teams become contracts that require negotiation to change. Chapter 8 develops this distinction, which is perhaps the single most effective way to contain the cost of evolution. Test against historical events. Keep a corpus of real or realistic events from every era of the system, and run every build's aggregates and projections against it. A change that breaks the ability to read old events should fail a test, not a production replay. Record the schema version in metadata. Even with weak schema, knowing which version of the code wrote an event makes diagnosis and upcasting far easier. The underlying point is one that Young makes repeatedly: versioning is not a problem unique to event sourcing. Every system that stores data for long periods or exchanges messages with other systems faces it. Event sourcing makes the problem visible and unavoidable, because the old data is always read, while state-based systems hide it by discarding the old data in migrations. That visibility is uncomfortable, but it is also honest. A team that has learned to evolve events well has learned to evolve any long-lived data contract, and the published events that other services depend on are exactly such contracts, as the next chapter shows. Chapter 8: Processes Across Boundaries: Sagas, Process Managers, and Integration Events The write model, as described so far, makes decisions inside a single consistency boundary. Real business processes rarely fit inside one. Settling an insurance claim involves the claim itself, a payments system that moves money, perhaps a fraud-screening service, a notification to the policyholder, and an update to the general ledger. Each of these has its own data, its own rules, and often its own team. No single transaction spans them, and in a distributed system none should. This chapter covers how event-sourced systems coordinate work across boundaries: how processes react to events and issue commands, how they handle failure, and how events published to other parts of an organization differ from the events a service stores for itself. Reacting to events The basic building block is simple. A component subscribes to events, and when it sees one it cares about, it issues a command somewhere else. When ClaimSettlementApproved is written to a claim stream, a reactor issues InitiatePayment to the payments service. When the payments service records PaymentCompleted, another reactor issues CloseClaim to the claims service. Work flows through the system as a chain of events and commands, each step decided within its own boundary. This structure appears under several names. In Adam Dymitruk's Event Modeling method, such components are called automations, and they are often described using the image of a to-do list: a read model of pending work that a processor works through, marking each item done. The to-do framing is useful because it makes the state of the process visible and queryable. At any moment you can ask which settlements have been approved but not yet paid. Reactors differ from projections in one crucial respect. A projection maintains a read model and can be rebuilt by replaying history. A reactor causes side effects: it sends commands, calls external services, sends emails, moves money. A reactor must therefore never be replayed in the way a projection is. If you reset a reactor's checkpoint to zero, it will try to initiate every payment ever approved. Reactors need their own checkpoints, independent of any projection, and their handlers must be idempotent or protected by deduplication, because at-least-once delivery means they will occasionally see the same event twice. The standard protection is to derive a deterministic identifier for the command from the event that caused it. The reactor that sees ClaimSettlementApproved with event identifier e-123 issues InitiatePayment with payment identifier derived from e-123. If the reactor processes the event twice, it issues the same payment identifier twice, and the payments service, which records each payment identifier it has seen, rejects the duplicate. The idempotency is enforced at the receiving boundary, which is the only place it can be enforced reliably. Sagas and process managers When a process involves several steps, with branching, waiting, and failure handling, simple reactors become hard to follow. The coordination logic is scattered across many small handlers, and nobody can see the process as a whole. Two patterns address this, and their names are used inconsistently enough to cause real confusion. The term saga comes from a 1987 paper by Hector Garcia-Molina and Kenneth Salem, "Sagas," which addressed long-lived database transactions. Their idea was to break a long transaction into a sequence of shorter ones, each with a compensating transaction that semantically undoes it. If step four fails, the system runs the compensations for steps three, two, and one in reverse order. The result is not isolation in the database sense, since other transactions can see intermediate states, but it does guarantee that the process either completes or is semantically rolled back. The term process manager comes from Gregor Hohpe and Bobby Woolf's Enterprise Integration Patterns (2003), where it describes a central component that maintains the state of a multi-step process and decides what message to send next based on the messages it receives. In the event-sourcing community, what is usually called a saga is in practice a process manager: a stateful component that listens to events from several boundaries, tracks where a particular process instance stands, and issues commands. Frameworks reinforce the ambiguity; Axon Framework has historically called its process managers sagas, while other authors reserve "saga" for the compensation pattern. The distinction that matters is not the name but whether a component holds state about the process and makes decisions, or merely translates one event into one command. A process manager for claim settlement might track each settlement through these states: approved, fraud check pending, fraud check passed, payment initiated, payment completed, claim closed. It reacts to ClaimSettlementApproved by requesting a fraud check, to FraudCheckPassed by initiating payment, to PaymentCompleted by closing the claim, and to FraudCheckFailed or PaymentFailed by taking compensating or escalating action. It is itself a small state machine, and in an event-sourced system its state can be stored as events in its own stream, giving the same auditability and debuggability as any aggregate. on ClaimSettlementApproved(e): state = { claimId: e.claimId, amount: e.amount, step: "fraud-check" } send RequestFraudCheck(claimId: e.claimId, amount: e.amount) schedule timeout(after: 4 hours) on FraudCheckPassed(e): state.step = "paying" send InitiatePayment(paymentId: derive(e.claimId), amount: state.amount) on PaymentFailed(e): state.step = "escalated" send RaisePaymentException(claimId: state.claimId, reason: e.reason) on timeout when state.step == "fraud-check": send EscalateToSupervisor(claimId: state.claimId) Choreography and orchestration There are two broad styles for multi-step processes. In choreography, there is no central coordinator: each service reacts to events from others and publishes its own, and the process emerges from these reactions. In orchestration, a process manager directs the steps explicitly. Choreography keeps services loosely coupled and is simple for short chains. Its weakness is visibility. When a process involves six services reacting to each other's events, understanding it means reading six codebases, and answering "where is settlement 4471 stuck?" means querying six systems. Orchestration centralizes the process logic and state, which makes it visible and easy to change, at the cost of a component that knows about every participant. A pragmatic rule many teams follow is to use choreography for simple, stable reactions and orchestration for processes with business significance, multiple steps, timeouts, or compensation. Claim settlement, which regulators may ask about and which involves money, is a clear case for orchestration. Time and failure Two features distinguish robust process managers from naive ones. The first is timeouts. A process that waits for an event may wait forever if the event never comes because a service is down, a message was lost, or a human never acted. Every waiting state should have a timeout that triggers an explicit action: retry, escalate, or compensate. Most frameworks provide scheduling for this; in a hand-built system, a scheduled-messages table polled by a background process is common. The second is compensation. When a step fails after earlier steps have succeeded, the process cannot roll back in the database sense, because earlier steps were committed in other boundaries, possibly in other organizations. It must issue commands that semantically undo them: reverse a payment, release a reservation, reopen a claim. Compensations are business operations with their own rules, and some steps cannot be compensated at all; an email cannot be unsent. Designing a process means deciding, for each step, what happens if a later step fails, and ordering steps so that irreversible ones come last where possible. The outbox problem, solved by the log Any system that changes its own state and then notifies others faces a familiar problem. If the service writes to its database and then publishes a message to a broker, a crash between the two leaves the database changed and the message unsent. If it publishes first and then writes, a crash leaves a message announcing a change that never happened. The two operations cannot share a transaction, because the database and the broker are different systems. The conventional solution is the transactional outbox: write the outgoing message to a table in the same database transaction as the state change, and have a separate relay process read the outbox table and publish to the broker, marking messages as sent. Delivery is at least once, and consumers must deduplicate, but no message is lost and none is sent for a change that did not happen. An event-sourced service gets this for free. The events in the store are the state change and the outbox at the same time. A subscription that reads the store and publishes to a broker is the relay. There is no dual write, because there is only one write. This is one of the less advertised but most practically valuable properties of event sourcing in a distributed architecture. Internal events and integration events The most important design decision in this chapter concerns which events leave the service at all. An event-sourced service's stored events are its internal model. They are named and shaped for the service's own decisions and projections. They change as the service's understanding of its domain changes, and Chapter 7 described the work involved in evolving them. If other services subscribe directly to those stored events, every internal change becomes a cross-team negotiation. The service's model is effectively frozen by its consumers, and the boundary that domain-driven design calls a bounded context, the region within which a model and its language are consistent, has been breached. The widely recommended alternative is to distinguish domain events, which are stored and used within the service, from integration events, which are published to other services as a deliberate, versioned contract. The service translates from one to the other, usually in the same subscription that relays events to the broker. Integration events are typically coarser, more stable, and stripped of internal detail. The claims service might store a dozen fine-grained events as an adjuster works a claim, but publish a single ClaimSettled integration event carrying what the finance and customer communication services need to know. This separation brings several benefits. Internal events can be refactored, split, renamed, and upcast without affecting anyone outside. Integration events can be versioned on a slower, negotiated cadence. Sensitive internal data, such as fraud scores or adjuster notes, never leaves the service by accident. And each consuming service is protected from the producer's model by an explicit contract, which it can translate into its own terms through what domain-driven design calls an anticorruption layer. Integration events also raise the question of how much data to include. A thin notification carries only identifiers: "claim C-1042 was settled." Consumers that need details must call back to the claims service to fetch them, which couples them to its availability and can produce inconsistent reads if the claim changes again before the callback. A fat event, which Fowler's essay on event-driven patterns calls event-carried state transfer, carries the data consumers need: the settled amount, the policy, the payee. Consumers can act without calling back, and can maintain their own local copies of the data they care about. Fat events make the contract larger and more likely to need versioning, but they make consumers more autonomous, and in most integration scenarios that autonomy is worth the cost. Integration events are frequently carried on a streaming platform such as Apache Kafka, even when the service's own event store is something else. That combination plays to each system's strengths: the event store provides per-stream optimistic concurrency and exact per-entity history for the write side, while Kafka provides durable, partitioned, high-throughput distribution to many consumers. The relay subscription bridges them. A failure, step by step It is worth tracing one realistic failure through a settlement process to see how these pieces interact, because the interesting behavior of a process manager lies almost entirely in its unhappy paths. An adjuster approves settlement of claim C-2210 for 18,400. The claims service records ClaimSettlementApproved. The settlement process manager, subscribed to claim events, starts a new process instance in its own stream, records SettlementProcessStarted, and sends RequestFraudCheck. It schedules a four-hour timeout. The fraud service responds within a minute with FraudCheckPassed. The process manager records the transition and sends InitiatePayment with a payment identifier derived deterministically from the claim and the approval event. The payments service accepts the command, records PaymentInitiated, and submits the transfer to the bank. Then the reactor that relays payment events to the broker crashes and is restarted. When it resumes from its checkpoint, it republishes PaymentInitiated, which the process manager has already seen. Because the process manager records the identifier of every event it has handled in its own state, it recognizes the duplicate and ignores it. Without that check, it might have advanced its state twice or issued a second command. Two days later the bank rejects the transfer because the policyholder's account has been closed. The payments service records PaymentFailed with a reason code. The process manager receives it. At this point, earlier steps have committed effects in other boundaries: the claim is marked as approved for settlement, and the reserve has been earmarked. The process manager does not try to undo the approval, because the approval is still correct; the insurer still owes the money. Instead it records PaymentExceptionRaised and sends a command to the claims service that places the claim in a "payment details required" state, which in turn appears in a customer-service work queue projection. When the policyholder supplies new bank details, a new command starts a fresh payment attempt with a new payment identifier, and the process continues. Every step of that story is recorded as events: in the claim stream, in the payment stream, and in the process manager's own stream. A regulator asking why the policyholder was paid four days late can be shown a precise, time-stamped sequence of what happened and who did what. An operations manager can query a projection of process states to find every settlement currently stuck in payment exception. And a developer debugging an unexpected path can load the three streams into a test and replay them. The example also illustrates the most common design error in process managers: treating every failure as a reason to roll back. Compensation is the right response when a later step makes an earlier step wrong, for example when a fraud check fails after a reservation was made. It is the wrong response when the earlier step remains correct and only the later step needs to be retried differently. The business, not the technology, decides which is which. Testing processes Process managers can be tested in the same given-when-then style as aggregates, with one addition: the expected outcome includes commands sent and timeouts scheduled, as well as events recorded. Given that settlement was approved and the fraud check passed, when payment fails with reason "account closed," then the process records PaymentExceptionRaised and sends RequirePaymentDetails. Tests of this kind are fast, need no infrastructure, and read like specifications of the business process. Timeouts deserve explicit tests, because they are the paths least likely to be exercised in development. A test harness that controls the clock, advancing time and delivering due timeouts, lets the team check that every waiting state has an exit and that the exit does the right thing. Many production incidents in process-heavy systems trace to a waiting state that someone forgot to give a timeout, leaving process instances suspended indefinitely and visible only when a customer complains. When processes become the domain A final observation applies to many mature event-sourced systems. As teams model processes explicitly, they often discover that the process is itself a central part of the domain. In claims handling, the settlement process, with its approvals, checks, escalations, and timeouts, is exactly what regulators examine and what operations managers want to monitor. Modeling it as an event-sourced process manager makes its history as complete and queryable as the claims themselves: every escalation recorded, every timeout visible, every compensation explained. That is an argument for treating process managers as first-class parts of the domain model rather than as plumbing. They deserve the same care in event naming, the same tests, and the same attention to evolution as aggregates. A process that moves money or affects customers is a business decision-maker, and the event-sourced approach gives it the same accountability as any other. The machinery described in these chapters, stores, aggregates, projections, versioning, and processes, must ultimately run on real infrastructure, with real operational demands and legal obligations. The next chapter turns to those. Chapter 9: Choosing and Running the Infrastructure The patterns in this book can be implemented on many kinds of storage. A team can use a database built specifically for event sourcing, a library that turns a general-purpose relational database into an event store, a full application framework with its own server, or a hand-built table with a unique constraint. Each choice shapes what is easy and what is hard, and each carries operational obligations that outlast the excitement of adoption. This chapter surveys the main options as they stand in late 2026, examines the recurring question of whether a streaming platform can serve as an event store, and then addresses the operational and legal realities of keeping an append-only record for years: storage growth, backup, and the duty to forget. Dedicated event databases: KurrentDB The longest-established dedicated event database is the one originally released as Event Store, co-founded by Greg Young, and later known as EventStoreDB. In December 2024 the company behind it renamed itself Kurrent and announced a funding round, and in March 2025 the database's 25.0 release shipped under the new name KurrentDB, moving from the Event Store License v2 to the Kurrent License v1. Subsequent releases have continued on a version scheme tied to the year: 25.1 in October 2025, a 26.0 release announced in December 2025, and a 26.1 short-term-support release. KurrentDB's design follows directly from the patterns described in earlier chapters. Data is organized into streams, each with strictly ordered events and per-stream optimistic concurrency through expected revisions. There is a global log across all streams with positions that subscribers can read from. Catch-up and persistent subscriptions are built in, as are server-side projections written in JavaScript that can derive new streams from existing ones, for example by partitioning events by type or category. Clustering uses a leader and followers with quorum-based replication, so a write is acknowledged only when a majority of nodes have it. Recent releases have broadened the feature set considerably: 25.0 introduced archiving of older data to cloud object storage, and 25.1 added atomic appends across multiple streams and secondary indexes, the former addressing directly the cross-aggregate consistency problem discussed in Chapter 3. Kurrent has also added connectors that push events to external systems, reflecting the common need to relay events to brokers and analytics stores. The strengths of a dedicated database are that the event-sourcing model is native rather than layered on, that subscriptions and ordering are handled by the server, and that the system is tuned for append-heavy workloads. The costs are another piece of infrastructure to operate, a smaller pool of operational expertise than mainstream databases enjoy, and, for read models, the need for a separate store, since KurrentDB is not designed to serve application queries. Teams should also read the current licensing terms carefully, because some features are restricted to licensed deployments and the terms have changed more than once over the product's history. Relational stores: Marten and hand-built tables The most common alternative is to store events in a relational database, usually PostgreSQL. The appeal is obvious: most teams already know how to operate PostgreSQL, back it up, monitor it, and secure it. The event table sits next to the read-model tables, so inline projections and transactional checkpointing are straightforward, and ordinary SQL is available for investigations. Marten, an open-source .NET library maintained under the JasperFx umbrella, is the most developed example. It uses PostgreSQL's JSONB support to serve both as a document database and as an event store. Its event-sourcing features include stream appends with optimistic concurrency, single-stream and multi-stream projections with inline, asynchronous, and live lifecycles, an asynchronous daemon that runs projections in the background with high-water-mark tracking to handle sequence gaps, the ability to aggregate a stream to a specific version or timestamp, and upcasting. Marten's 8.x line was current through 2025 and 2026. It is often paired with Wolverine, a messaging and command-handling library from the same ecosystem, which supplies the outbox, command routing, and process handling that Marten alone does not. A hand-built store on a relational database, with the table structure shown in Chapter 2, is a perfectly respectable choice for teams with modest needs. The minimum is a table with a global position, a stream identifier, a stream version with a unique constraint, a type, a payload, and metadata. The work that remains is the subscription mechanism, which must handle the sequence-gap problem described in Chapter 5, checkpoint storage, and whatever upcasting the team needs. Teams that underestimate the subscription problem tend to discover it in production as projections that occasionally miss events. Similar libraries exist in other ecosystems. The Python eventsourcing package supports several storage back ends including PostgreSQL and has added support for dynamic consistency boundaries. Eventuous offers a lighter .NET alternative that works with KurrentDB and other stores. Message DB provides a PostgreSQL-based message and event store usable from several languages. The specific library matters less than whether it handles concurrency, ordering, and subscriptions correctly. Application frameworks: Axon Axon Framework, a Java framework developed by AxonIQ, takes a broader approach. Rather than providing only an event store, it provides a full stack: command buses, event buses, query buses, aggregate support, event processors for projections, process managers, and upcasters. Axon Server, the company's companion product, serves as both event store and message router. Axon Framework 5, released in October 2025, was a major revision that introduced support for dynamic consistency boundaries, moved the event processing pipeline toward asynchronous, non-blocking execution, and changed how handlers access state. Axon Server's 2025.1 release added DCB support on the storage side. The appeal of a framework is that the patterns are already assembled: a team can follow Axon's conventions and have commands, events, projections, and processes working together quickly. The cost is coupling to the framework's model and lifecycle, and a major version change such as the move to version 5 can require significant migration work. Can Kafka be an event store? No question in this field generates more heated discussion than whether Apache Kafka, or a similar distributed log, can serve as the event store for an event-sourced system. Kafka is a durable, partitioned, replicated log with excellent throughput and a rich ecosystem, and logs are the essence of event sourcing, so the idea is natural. Many practitioners who have tried it argue against using Kafka for the write side, and the reasons are specific. The first is per-entity concurrency. The write model needs to append to an aggregate's stream only if no one else has appended since it last read. Kafka has no built-in way to append conditionally on the current position of a particular entity's events. Topics are partitioned, and many entities share a partition; producers append to partitions without such a check. Workarounds exist, such as routing all commands for an entity through a single consumer that processes them serially, but these rebuild the concurrency control outside the log. The second is loading a single entity. To rebuild an aggregate, the write model must read all events for that entity. In Kafka, those events are interleaved with every other entity in the same partition, so reading one entity means scanning a partition or maintaining a separate index. A topic per entity is impractical at the scale of millions of entities. The third is retention and compaction. Kafka topics are often configured with time-based retention or log compaction, both of which discard events. Either can be disabled, and Kafka can retain data indefinitely, but the platform's defaults and operational culture assume streams are transient. The counter-argument, made by Kafka's proponents, is that Kafka excels at exactly the parts of an event-driven architecture that a per-entity store does not: high-volume distribution to many consumers, stream processing, and integration across teams. The pragmatic synthesis, which Chapter 8 described, is to use an event store for each service's write side and Kafka or a similar platform for integration events between services. That combination gives each tool the job it does well. Table 5 summarizes how the main options compare. Table 5. Infrastructure options for an event-sourced write side. Option Per-stream concurrency Built-in subscriptions Read models in same store Operational familiarity KurrentDB Native Yes, catch-up and persistent No Specialized Marten on PostgreSQL Native Yes, async daemon Yes High Axon Server with Axon Framework Native, plus DCB Yes, via framework No Specialized Hand-built relational table Unique constraint Must be built Yes High Kafka alone Not provided Yes, consumer groups No High in many firms Operating an append-only store An event store grows forever by design, and that has consequences. Storage growth is usually less alarming than expected. Business events are small, often a few hundred bytes to a few kilobytes, and many domains generate them at human rather than machine rates. A claims system handling a million claims a year with fifty events each produces fifty million events a year, which is a modest amount of data for modern databases. High-frequency domains, such as telemetry or market data, are different, and those are often better served by time-series stores than by event sourcing in the sense this book describes. Where growth does become a concern, tiered storage helps: KurrentDB's archiving feature, introduced in 25.0, moves older chunks to object storage while keeping them readable, and relational stores can partition the event table by time and move old partitions to cheaper storage. Backup and restore must be planned with read models in mind. The event store is the system of record and must be backed up as rigorously as any primary database. Read models, being derived, can in principle be rebuilt from a restored store instead of being backed up, but rebuild times for large systems may be hours or days, so in practice teams back up important read models as well and treat rebuilding as the fallback. A restore drill should include restoring the event store, bringing projections back up from their checkpoints, and verifying that read models and the store agree. Deleting streams is supported by most stores for operational reasons, such as removing test data or streams that have reached the end of a retention period. KurrentDB supports soft and hard deletion of streams and reclaims space through a process called scavenging. These capabilities should be used under clear policies, because deletion undermines the completeness that gives the store its value. Observability should cover the whole pipeline: append latency and conflict rates on the write side, subscription lag per projection and per reactor, poison event counts, and rebuild progress. Conflict rates are a useful design signal: a stream type that frequently produces concurrency conflicts is telling the team its consistency boundary is drawn around too much activity. The duty to forget The most serious tension in event sourcing is with privacy law. Article 17 of the EU General Data Protection Regulation gives individuals a right to have personal data about them erased in a range of circumstances, and similar rights exist in other jurisdictions. An append-only store that never deletes is, on its face, in conflict with that right. The Overeem study cited in Chapter 7 listed data privacy among the five main challenges practitioners reported. Several approaches are used, often in combination. The first is data minimization in events: keep personal data out of events where possible. An event can refer to a customer by an opaque identifier, with name, address, and contact details held in a separate, conventional, mutable store that can be erased normally. The events then record what happened without recording who it happened to in identifiable form. This is the simplest and most robust approach, and the design decision belongs at the start. The second is crypto-shredding. Personal data that must be in events is encrypted with a key specific to the data subject, and the keys are stored separately. To erase a person's data, delete their key. The encrypted values remain in the event store but are unreadable by anyone. Projections and aggregates must handle the case where a key is missing, typically by substituting a placeholder. Crypto-shredding preserves the immutability of the log while making the erasure effective. Whether it fully satisfies a particular legal obligation is a question for lawyers in the relevant jurisdiction, and some organizations combine it with other measures for that reason. The third is forgettable payloads: storing personal data in a separate store keyed by event identifier, with the event itself holding a reference. Erasure deletes the separate record. This is similar to data minimization but keeps a closer link between the event and its personal details. The fourth is stream deletion or rewriting, which some systems resort to for erasure requests, accepting the loss of history for that subject. Where events about a person are confined to a stream about that person, deleting the stream is clean. Where personal data is spread across many streams, rewriting becomes an in-place transformation, with the risks described in Chapter 7. Whatever approach is chosen, read models must be included. Erasing personal data from the event store accomplishes nothing if a projection holds a copy in a search index or a reporting table. A complete erasure process identifies every projection that could hold the data and either rebuilds it or deletes from it directly. When not to do this The final piece of infrastructure advice is the decision to use none of it. Event sourcing requires a team that understands the patterns well enough to design events, draw boundaries, operate projections, and evolve schemas. The Overeem study's finding of a steep learning curve matches what practitioners report informally. A team new to the patterns, building a system whose domain does not meet the test from Chapter 1, under time pressure, is unlikely to benefit. Several situations argue against adoption. When the domain is mostly data entry with little behavior, CRUD is simpler and sufficient. When the main need is an audit trail of field changes rather than business decisions, temporal tables or change data capture provide it more cheaply. When the team needs a read-optimized view separate from a write model, CQRS without event sourcing may suffice. And when the organization cannot commit to the operational practices, monitoring lag, managing rebuilds, maintaining upcasters, handling erasure, the system will degrade in ways that are hard to recover from. None of these are permanent verdicts. A system can begin as conventional CRUD and introduce event sourcing later in the specific bounded context where history becomes valuable, which is often the wisest path. What matters is that the choice is made deliberately, for the part of the system where it pays, with the costs understood in advance. Conclusion The chapters of this book have moved from storage to decisions to views to history to change to coordination to operations. Laid side by side, they support a set of conclusions that are not visible from any single chapter, and some of them run against the way event sourcing and CQRS are usually presented. The event is the product The first conclusion is that nearly every success and failure in an event-sourced system traces back to event design. Projections can be rebuilt, so a bad read model is a bug fixed in an afternoon. Aggregates can be redrawn, at some cost. Infrastructure can be migrated. But an event that recorded ClaimUpdated with a dump of fields, instead of ReserveIncreased with an amount, a basis, and an approver, cannot be repaired. The information was never captured, and no upcaster, projection, or migration can recover it. The time-travel capability of Chapter 6, the new analytical views of Chapter 5, and the auditability that justifies the pattern in regulated domains all depend on events that recorded what the business actually did and why. This implies that event design deserves more time and more seniority than it usually gets. Teams commonly spend weeks choosing a store and days designing events. The proportions should be reversed. The most valuable investment a team can make at the start is a series of sessions with domain experts, in the manner of EventStorming, that produces events a claims manager or an auditor would recognize. It also implies that event review deserves the same rigor as public API review, because an event type, once written to production, is a permanent commitment. The aggregate was never the point The second conclusion concerns consistency. The event-sourcing literature long treated the aggregate as the defining structure of the write side, and stream-per-aggregate as the natural shape of the store. Chapter 3 described the challenge from dynamic consistency boundaries, and Chapter 9 described KurrentDB's addition of multi-stream atomic appends. Both developments point the same way: the aggregate was a means, not an end. The end is that each decision is made on information that cannot change between reading it and acting on it. Fixed streams enforced that cheaply for the common case and awkwardly for the rest. What follows is that teams should design around decisions rather than entities. For each command, ask which facts the decision depends on and which rules must hold without exception. Where the answer is one entity's history, a conventional aggregate is ideal. Where the answer spans entities and the rule is a true invariant, current tools offer options, from reservations to multi-stream appends to DCB, that did not exist or were not widely available a few years ago. Where the rule can tolerate a brief violation followed by correction, a process is still simpler and more resilient. The question "is this a true invariant?" remains the most useful one on the write side, and the answer is still "no" more often than first drafts assume. Staleness is a product decision The third conclusion concerns eventual consistency, which is the cost of CQRS that teams most often fight. Chapter 4 argued, following Dahan, that data shown to a user is stale anyway, and Chapter 5 described the machinery that makes read models lag. The practical lesson from the two together is that freshness should be specified per view, as a requirement, by the people who understand the business. The adjuster's own work queue after an action needs read-your-writes. The monthly reserve report can be minutes behind. The fraud model's features can be hours behind. Treating all read models as if they needed immediate consistency leads either to inline projections everywhere, which slows writes and couples everything, or to a constant stream of complaints about a system that seems not to save. Treating freshness as a per-view decision, and using the techniques of Chapter 4 to meet each requirement, makes eventual consistency mostly invisible to users. The long term is the real test The fourth conclusion is about time horizons. Event sourcing is usually evaluated in its first year, when the event store is small, every event type has one version, projections rebuild in seconds, and nobody has asked for erasure. The real costs and benefits arrive later. In year three, the actuary asks a question nobody anticipated, and complete history answers it. In year three, too, the aggregate handles four versions of ClaimReported, a projection takes six hours to rebuild, and a data-protection request arrives for a policyholder whose name appears in events across forty streams. The practitioner research cited in Chapters 7 and 9 found that evolution, rebuilding projections, and privacy were among the challenges people actually face, and all three are long-term problems. The implication is that the decisions that determine whether event sourcing succeeds are mostly the ones that feel premature at the start: separating internal events from integration events, keeping personal data out of events or encrypting it per subject, budgeting for rebuild times, building a corpus of historical events for regression tests, and choosing a weak-schema serialization strategy. Each is cheap at the start and expensive to retrofit. What remains unsettled Several questions remain open as of late 2026. The most important is how far dynamic consistency boundaries will reshape practice. The idea is persuasive, and it now has implementations in Axon Framework 5, Axon Server, and several open-source libraries. But it is young, and the field lacks a body of experience with large, long-lived systems built on it. Whether tag-based queries scale cleanly, whether teams can reason about decisions whose read sets vary, and whether the flexibility leads to well-bounded decisions or to sprawling ones are questions that only years of production use will answer. A second open question is how event stores and streaming platforms will converge. Dedicated event databases are adding connectors, secondary indexes, and multi-stream transactions; streaming platforms are adding features for long retention and querying. The clean separation recommended in Chapters 8 and 9, an event store for the write side and a streaming platform for integration, may blur as each category absorbs the other's capabilities. For now the separation remains sound because the concurrency requirements of the write side are still poorly served by general-purpose logs. A third is the legal standing of crypto-shredding. It is widely used and widely recommended, but whether deleting a key satisfies an erasure obligation, when the encrypted data persists, is ultimately a legal question that different regulators may answer differently. Teams in regulated industries should treat it as a strong technical measure, not as a settled legal answer. What to do differently For a team considering these patterns, the argument of this book reduces to a few recommendations. Apply event sourcing only where the domain's history is valuable to someone, using the three-part test from Chapter 1: meaningful behavior, consulted history, changing interpretations. Leave the rest of the system conventional. Treat CQRS as a separate and cheaper decision. Use its first level, separate query paths over the same database, freely; move to separate stores only where read and write needs truly diverge. Spend the early effort on events, in collaboration with domain experts, and review them as permanent contracts. Design for the long term from the first sprint: weak schema, upcasting support, internal versus integration events, a plan for personal data, and monitored projection lag. Specify freshness per view, and use task-based interfaces where intent matters. And keep the ledger in mind. The accountants who refused to erase entries were not building an architecture; they were building trust in a record that people would rely on when it mattered. Event sourcing is worth its costs exactly when a system needs that kind of trust, and when the team is prepared to keep the record carefully for as long as the business depends on it. Further Reading Greg Young, CQRS Documents (c. 2010). A set of notes that set out the combination of CQRS, event sourcing, and task-based interfaces as Young taught it. Short, informal, and still the clearest statement of the original ideas. Widely mirrored online. Greg Young, Versioning in an Event Sourced System (Leanpub, c. 2017).** The most practical book-length treatment of evolving events, covering weak schema, upcasting, copy-replace, and the reasons to store events rather than commands. Martin Fowler, "Event Sourcing" (2005), "CQRS" (2011), "What do you mean by 'Event-Driven'?" (2017), and "Bitemporal History" (2021). Short essays on martinfowler.com. The first defines the pattern; the second gives a balanced view of CQRS with an important warning about overuse; the third separates four commonly confused patterns; the fourth explains the two dimensions of time. Udi Dahan, "Clarified CQRS" (2009). An essay on udidahan.com that reframes CQRS around business tasks, stale data, and user intent. Essential reading on the user-facing side of the pattern. Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software (Addison-Wesley, 2003).** The origin of aggregates, bounded contexts, and ubiquitous language. Dense, but the foundation on which most event-sourcing practice is built. Vaughn Vernon, Implementing Domain-Driven Design (Addison-Wesley, 2013).** A practical companion to Evans with worked chapters on aggregates, domain events, and event sourcing. Vernon's earlier essay series "Effective Aggregate Design" (2011) distills the aggregate rules. Alberto Brandolini, Introducing EventStorming (Leanpub).** The workshop method for discovering domain events with business experts. The best practical guide to the event design work this book argues matters most. Michiel Overeem, Marten Spoor, Slinger Jansen, and Sjaak Brinkkemper, "An empirical characterization of event sourced systems and their schema evolution: Lessons from industry," Journal of Systems and Software (2021).** A grounded-theory study of 19 systems and 25 engineers that documents the challenges practitioners face and the tactics they use for schema evolution. Hector Garcia-Molina and Kenneth Salem, "Sagas," Proceedings of ACM SIGMOD (1987). The original paper on long-lived transactions and compensation. Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns (Addison-Wesley, 2003).** The source of the process manager pattern and a catalogue of messaging patterns that remains relevant to coordinating event-sourced services. dcb.events. The documentation site for dynamic consistency boundaries, including Sara Pellegrini's background, the minimal specification, worked examples, and a list of implementations. Product documentation: KurrentDB (docs.kurrent.io), Marten (martendb.io), and Axon Framework (docs.axoniq.io). Each product's documentation contains substantial conceptual material on streams, subscriptions, projections, and versioning, and is the authoritative reference for current features. Hashtags: #EventSourcing #CQRS #SoftwareArchitecture #ArchitecturalPatterns #DomainDrivenDesign #DDD #EventDrivenArchitecture #ComplexDomains #DomainEvents #EventStore #EventStreams #ImmutableEvents #AppendOnlyLog #CommandQuerySeparation #CommandHandling #Aggregates #ConsistencyBoundaries #OptimisticConcurrency #EventualConsistency #ReadModels #WriteModels #EventProjections #MaterializedViews #EventReplay #EventVersioning #TemporalModeling #EventStorming #DistributedSystems #MicroservicesArchitecture #EnterpriseSoftwareArchitecture

Download the Book (PDF): Introduction Type a question into a modern search box, a customer-support assistant, or a code navigator, and somewhere behind the interface a small piece of arithmetic happens many millions of times. Your words are turned into a list of a few hundred or a few thousand numbers. That list is compared against lists produced, months or minutes earlier, from every document, image, product, or code fragment the system knows about. The handful of stored lists that sit closest to yours are returned, and whatever the system does next, whether it shows you links, writes you an answer, or recommends a song, rests on that handful being the right one. The lists of numbers are embeddings, and the operation is nearest-neighbour search. Neither idea is new. Information retrieval researchers were representing documents as vectors in the 1960s, and computational geometers have studied nearest-neighbour problems for almost as long. What changed in the last decade is that learned embeddings became good enough to carry meaning, and that the collections people wanted to search became very large. A retailer with a hundred million product images, a web-scale search engine with tens of billions of passages, a code host with billions of functions: these are not hypothetical. At that scale, the naive way of answering a query, comparing it against every stored vector, stops being a design option and becomes an outage. This booklet is about what replaces the naive way. It is an engineering text, written for readers who want to understand the machinery rather than merely configure it: software engineers who have been handed a vector database and want to know what the knobs actually do, architects deciding between products, technical leaders who need to reason about cost and risk, and researchers from adjacent fields who want a coherent account of a literature that is scattered across computer vision, databases, information retrieval, and systems conferences. The controlling idea The argument of the book can be put in one sentence. Every vector index that works at scale is a deliberate bet about where the true nearest neighbours are likely to be, and the craft of vector search consists in choosing which bet to make, sizing it against the memory hierarchy of real machines, and measuring how often it loses. There are, broadly, three kinds of bet. The first is partitioning: divide the collection into regions, and when a query arrives, look only in the regions closest to it. The inverted file index, which carves space into clusters with k-means, is the canonical example. The second is compression: replace each stored vector with a much shorter code that preserves distances approximately, so that far more vectors fit in fast memory and each comparison costs less. Product quantization is the canonical example, and its descendants now include scalar, binary, and randomized schemes. The third is navigation: link each vector to a few of its neighbours, forming a graph, and answer a query by walking greedily through the graph toward the target. Hierarchical navigable small world graphs, universally abbreviated HNSW, are the canonical example, and they have become the default index in most of the systems people actually deploy. Each bet can lose. A partition may place a true neighbour in a cluster the query never visits. A compressed code may rank a slightly farther vector above a slightly nearer one. A greedy walk may get stuck in a local region of the graph and never reach the true answer. The losses are measured as recall: the fraction of the true nearest neighbours that the index actually returns. A system that answers in two hundred microseconds with ninety-five per cent recall has made a different bet from one that answers in two milliseconds with ninety-nine per cent, and neither is wrong in the abstract. What is wrong is not knowing which bet you made. Real systems combine the bets. A billion-scale index will typically partition the collection across machines and within each machine, compress the vectors so that they fit in memory or so that more of them fit in a cache line, navigate a graph or scan a list to find candidates, and then re-rank a short list of candidates using more precise information held further down the memory hierarchy. Understanding such a system means understanding each layer separately and then understanding how their errors compound. Why the details matter It is possible to use a vector database for years without knowing any of this. Most products expose a handful of parameters with sensible defaults, and for collections of a few million vectors on a single machine the defaults are usually fine. The details start to matter at three moments. The first is when cost becomes visible. Raw embeddings are expensive to hold in memory. A billion vectors of 768 dimensions stored as 32-bit floating-point numbers occupy a little over three terabytes before any index structure is added. Holding that in RAM across a fleet of servers is possible, but the bill is large enough that someone will ask whether it is necessary. Answering the question requires knowing what compression costs in recall, and what moving vectors to solid-state storage costs in latency. The second is when quality becomes visible. Retrieval feeds everything downstream. In a retrieval-augmented generation system, a language model can only reason over the passages it is given; if the index silently drops the one passage that contained the answer, the model will either answer from memory or invent something. Teams who see their assistant hallucinate often look first at the prompt or the model. Sometimes the fault is several layers down, in an index configured for speed at a recall the application could not afford, or in a metadata filter that interacts badly with the graph. The third is when the workload changes. Indexes are built for a distribution of data and a distribution of queries. When the data grows by an order of magnitude, when deletions accumulate, when the embedding model is replaced, or when queries start arriving from a different modality than the documents, an index that behaved well can degrade without any error being raised. The degradation is statistical, and it is only visible to people who measure. What the book covers The chapters run in an order designed to build the argument from the ground up. Chapter 1 explains what embeddings are and why semantic similarity becomes a geometric problem, including the choice of distance function and the subtle ways that choice interacts with how embedding models are trained. Chapter 2 explains why exact search fails at scale and what we give up by abandoning it: the curse of dimensionality, the cost of brute force on real hardware, and the definition of recall that the rest of the book relies on. Chapters 3 to 5 treat the three bets in turn. Chapter 3 covers partitioning through inverted file indexes. Chapter 4 covers compression, from product quantization to the binary and randomized schemes that have become prominent in the last few years. Chapter 5 covers graph navigation through HNSW in detail, including the parts of its design that are often misunderstood and the operational problems it creates. Chapter 6 leaves main memory, examining how indexes such as DiskANN and SPANN serve billions of vectors from solid-state drives, and how graphics processors change the economics of both building and querying indexes. Chapter 7 confronts the fact that real queries are rarely pure nearest-neighbour searches: they carry filters, they combine keyword and semantic signals, and they demand diversity. Chapter 8 assembles the pieces into a distributed system, with sharding, replication, ingestion, updates, and the management of tail latency. Chapter 9 is about measurement: how to evaluate an index honestly, why public benchmarks are a starting point rather than an answer, and how to tune a deployed system. Chapter 10 works through the decisions an engineer actually faces, with sizing arithmetic for several realistic scenarios. The book assumes comfort with basic programming concepts and with the idea of a vector as an ordered list of numbers. It does not assume knowledge of linear algebra beyond the dot product, and it explains every technique from first principles before discussing its refinements. Where it refers to specific papers or systems, it does so because they introduced an idea that is still in use, and the further reading at the end points to the primary sources. One phrase in the subtitle deserves a word of caution at the outset. Sub-millisecond queries across billions of embeddings are real, but they are not free and not universal. They are achieved by systems that hold their working set in memory across many machines, that accept a defined recall below one hundred per cent, and that are tuned for a particular query distribution. Read the phrase as a description of what careful engineering makes possible, not as a property that any index delivers by default. The rest of the book explains what the careful engineering consists of. Chapter 1. The Geometry of Meaning An embedding is a function that maps an object, such as a sentence, an image, a user, or a protein, to a point in a space of fixed dimension. The function is learned, usually by a neural network, and the learning objective is arranged so that objects which should be treated as similar end up near one another. Once that is true, a question about meaning ("which documents discuss the same thing as this query?") becomes a question about geometry ("which points lie closest to this point?"). Vector search is the machinery for answering the geometric question quickly. Before examining that machinery, it is worth being precise about what the geometry is and where it comes from, because many problems that surface later as index problems are in fact embedding problems. From words to vectors The idea that meaning can be captured by position has a long history in information retrieval. The vector space model of the 1960s and 1970s, associated with Gerard Salton and the SMART system at Cornell, represented each document as a vector with one dimension per vocabulary term. The value in each dimension was a weight, typically some variant of term frequency multiplied by inverse document frequency, and similarity between a query and a document was the cosine of the angle between their vectors. This representation is sparse: a vocabulary might have hundreds of thousands of terms, but any given document uses only a few hundred of them, so almost every coordinate is zero. Sparse representations are excellent at exact matching and poor at meaning. A query for "automobile repair" shares no terms with a document about "fixing cars", and so their cosine similarity is zero. Decades of work on query expansion, stemming, thesauri, and latent semantic analysis attempted to bridge this vocabulary gap. Latent semantic analysis, introduced at the end of the 1980s, used a truncated singular value decomposition of the term-document matrix to project documents into a few hundred dense dimensions, where words that co-occurred in similar contexts were pulled together. It was the first widely used dense semantic embedding, and it already displayed the property that makes dense vectors useful: similarity in the projected space tracks similarity in meaning better than term overlap does. The modern era began with neural word embeddings. Word2vec, published by Tomas Mikolov and colleagues at Google in 2013, trained a shallow network to predict words from their neighbours and produced vectors in which semantic and syntactic relationships appeared as consistent directions. GloVe followed from Stanford in 2014. These were word-level embeddings; to embed a sentence one had to average or otherwise combine word vectors, which loses order and nuance. Transformer language models changed that. BERT, released in 2018, produced contextual representations for every token, and a series of papers then showed how to train such models specifically to produce a single vector per passage that is good for retrieval. Sentence-BERT in 2019 fine-tuned BERT with a siamese architecture so that semantically similar sentences received nearby vectors. Dense Passage Retrieval, from Facebook AI Research in 2020, trained separate query and passage encoders on question-answer pairs and demonstrated that a purely dense retriever could outperform the classic BM25 keyword ranker on open-domain question answering benchmarks. Since then, general-purpose text embedding models have been trained by many organizations, open and commercial, and evaluated on shared benchmarks such as MTEB, the Massive Text Embedding Benchmark, which aggregates dozens of retrieval, clustering, and classification tasks. Parallel developments happened in other modalities. CLIP, from OpenAI in 2021, trained an image encoder and a text encoder jointly on image-caption pairs so that a photograph and a sentence describing it land near each other in a shared space. That made cross-modal retrieval, searching images with text, a nearest-neighbour problem. Recommendation systems learned embeddings for users and items from interaction logs, so that "items this user might like" became "item vectors near this user vector". Code search, chemistry, audio fingerprinting, and face recognition all followed the same pattern. The common structure is that a model, trained once at considerable expense, defines a space. Everything the system stores is embedded into that space at ingestion time. Every query is embedded at query time. The search system knows nothing about language or images; it sees only points. How similarity is measured Given two vectors, there are three similarity or distance functions that dominate practice. The first is Euclidean distance, often written L2: the square root of the sum of squared differences between coordinates. It is the ordinary straight-line distance familiar from school geometry, extended to many dimensions. Because the square root is monotonic, search systems almost always compare squared L2 distances and skip the root. The second is inner product, also called dot product: the sum of products of corresponding coordinates. Larger is more similar. Inner product is not a distance in the mathematical sense, because a vector is not necessarily most similar to itself; a different vector pointing in the same direction but longer will score higher. Search over inner product is therefore called maximum inner product search, MIPS, to distinguish it from nearest-neighbour search under a true metric. The third is cosine similarity: the inner product of two vectors divided by the product of their lengths. It measures only the angle between them and ignores magnitude entirely. These three are related in a way that matters for engineering. If every vector is normalized to unit length, cosine similarity equals inner product, and squared Euclidean distance equals two minus twice the inner product. All three then produce exactly the same ranking. This is why many embedding models output normalized vectors and why many vector databases quietly normalize on ingestion when asked for cosine similarity: it reduces three problems to one, and it allows indexes designed for Euclidean distance to be used for cosine search. When vectors are not normalized, the choice matters. Some models, particularly in recommendation, deliberately encode information in vector length. An item's vector might be longer if the item is popular or of high quality, so that inner product with a user vector rewards both relevance and quality. Searching such vectors with cosine similarity discards the quality signal. Searching them with Euclidean distance penalizes long vectors, which is the opposite of what was intended. The correct choice is whatever the model was trained with. A frequent and silent error is to index vectors with a distance function different from the one used in training, which produces results that look plausible and are systematically worse. Maximum inner product search has a further wrinkle. Many index structures, including graph indexes and some quantization schemes, rely on the triangle inequality or on the property that a point is its own nearest neighbour, neither of which inner product satisfies. A well-known reduction, described in work by Bachrach and colleagues at Microsoft in 2014 and in related papers, converts MIPS into Euclidean nearest-neighbour search by appending one extra coordinate to each stored vector, chosen so that all augmented vectors have the same length, and appending a zero to the query. After the transformation, the Euclidean nearest neighbour of the augmented query is the maximum inner product match of the original. The cost is one dimension and a change in the data distribution that can make some indexes less efficient, so systems that need MIPS at scale often prefer methods designed for it directly, such as the anisotropic quantization in Google's ScaNN, discussed in Chapter 4. What the dimensions mean, and what they do not Embedding dimensions in current models commonly range from 256 to 4,096, with 384, 768, 1,024, and 1,536 being frequent choices. A natural question is what each dimension represents. The honest answer is that individual dimensions usually have no stable interpretation. The model learns a space in which distances are meaningful, but any rotation of that space would preserve all distances and therefore serve equally well. Meaning lives in the relative positions of points, not in the axes. This has two practical consequences. First, it is generally not safe to truncate an embedding by dropping its last coordinates, because there is no reason the information should be concentrated in the first ones. The exception is models trained specifically to allow truncation. Matryoshka Representation Learning, published by Kusupati and colleagues in 2022, trains a model so that the first 64, 128, 256 dimensions and so on are each usable embeddings on their own, with quality declining gracefully as dimensions are removed. Several commercial and open embedding models now ship with this property, and it gives search engineers a direct lever: store and search short prefixes for speed, and use the full vector to re-rank. Second, because the axes are arbitrary, an index can apply its own rotation without changing the meaning. Several compression methods do exactly this, rotating the space so that information is spread more evenly across coordinates before quantizing them. Chapter 4 returns to this idea. Not all dimensions are equally used, however. Learned embeddings are typically anisotropic: rather than filling the space uniformly, the points cluster in a narrow cone, with a few directions carrying most of the variance. Researchers studying contextual language model representations documented this in 2019 and 2020, observing that randomly chosen sentence embeddings from some models have high cosine similarity with one another. Anisotropy is not necessarily harmful for retrieval, since ranking depends on relative differences, but it affects index behaviour. It means the intrinsic dimension of the data, the number of directions along which points genuinely vary, is often much lower than the nominal dimension. A 768-dimensional embedding collection might behave, for indexing purposes, more like a collection of a few dozen effective dimensions. That is good news, because the difficulty of nearest-neighbour search grows with intrinsic dimension, not nominal dimension. The query is not a document A subtle point that matters a great deal in practice is that queries and documents need not come from the same distribution. In a symmetric task, such as finding duplicate questions, both sides are the same kind of text. In an asymmetric task, such as answering short questions from long passages, the query is a few words and the document is a paragraph. Many retrieval models are trained asymmetrically, sometimes with separate encoders for queries and documents, sometimes with a single encoder and a prefix instruction such as "query:" or "passage:" that tells the model which side it is embedding. The geometric consequence is that query vectors and document vectors may occupy different regions of the space. Cross-modal retrieval makes this extreme: in a CLIP-style model, text embeddings and image embeddings are aligned well enough to rank correctly, but they do not overlap; there is a measurable gap between the region occupied by images and the region occupied by text. This is called the modality gap, and it was characterized in a 2022 paper by Liang and colleagues. Index structures are built from the documents. If queries come from elsewhere, the index is being asked to answer questions about points it was not organized around. The Big-ANN benchmark competition held at NeurIPS in 2021 included a dataset built from text-to-image search at Yandex precisely because this out-of-distribution setting made graph indexes noticeably less efficient than on in-distribution data. Subsequent work, including the RoarGraph index published in 2024, builds graphs using a sample of real queries to correct for the mismatch. The lesson for an engineer is to test an index with real queries, not with held-out documents used as stand-in queries, because the latter will flatter the index. Similarity is not relevance The final point of this chapter is the most important one for anyone building a product on top of vector search. Nearest neighbours in embedding space are the items the model considers most similar. Whether they are the items the user needs is a separate question. Embedding models compress a passage into a single point. That compression discards information, and what it discards is determined by the training data. A general-purpose model trained largely on web text may place two contracts near each other because they share legal boilerplate, even if they differ in the one clause that matters. It may struggle with negation, so that "drugs that do not interact with warfarin" retrieves documents about drugs that do. It may handle rare identifiers, part numbers, and proper names poorly, because these appear too infrequently in training to receive distinctive representations. These are limitations of single-vector dense retrieval, and they are why production systems so often combine dense search with keyword search, apply re-ranking models that read the query and candidate together, or use multi-vector representations such as ColBERT, which keeps one vector per token rather than one per passage. There is also a theoretical ceiling. A single vector of dimension d can represent only so many distinct top-k result sets through inner products; work published in 2025 by researchers at Google DeepMind and Johns Hopkins formalized this in terms of the sign-rank of the relevance matrix and constructed simple retrieval tasks on which embedding models of realistic dimension fail. The practical reading is not that dense retrieval is broken, but that its errors are structural and will not be fixed by a better index. This distinction shapes how the rest of the book should be read. An approximate index that returns ninety-eight per cent of the true nearest neighbours is reproducing the model's judgement almost exactly. If the application is still failing, the remaining two per cent are rarely the cause. Conversely, when an application is succeeding, it is easy to over-invest in recall: going from ninety-five to ninety-nine per cent recall can double or triple query cost, and if the downstream re-ranker or language model is robust to a missing candidate or two, that money buys nothing. Knowing which situation you are in requires measuring both index recall against exact search and end-to-end quality against human judgement, and keeping the two separate. Chapter 9 describes how. With the geometry established, the next question is why we cannot simply compute all the distances and sort them. The answer has two parts: one about mathematics, concerning how distances behave in high dimensions, and one about hardware, concerning how fast real machines can move bytes. Chapter 2. Why Exact Search Fails at Scale The simplest nearest-neighbour algorithm is to compute the distance from the query to every stored vector and keep the k smallest. It is called brute-force or flat search, and it is exact by construction. It is also, surprisingly often, the right answer. For collections of up to a few hundred thousand vectors, a well-implemented flat scan on a modern processor completes in a few milliseconds and requires no index at all. Facebook's Faiss library, the most widely used toolkit in this field, calls its flat index IndexFlat and treats it as the gold standard against which every approximate method is measured. Understanding why flat search eventually fails, and why the obvious alternatives from classical computational geometry do not rescue it, is the foundation for everything that follows. The arithmetic of brute force Consider a collection of N vectors of dimension d, stored as 32-bit floats. A single query requires N distance computations, each of which involves d multiplications and d additions. For N of one billion and d of 768, that is roughly 1.5 trillion floating-point operations per query. A single modern server processor core with wide vector instructions can sustain perhaps tens of billions of such operations per second under ideal conditions, so one core would take on the order of a minute. Spread across a hundred cores the query might take under a second, but that machine would then serve one query per second, and a production search system may need thousands. Arithmetic is not actually the binding constraint, however. Memory bandwidth is. To compare a query against every stored vector, the processor must read every stored vector from memory. One billion vectors at 768 dimensions and four bytes per dimension is about 3.1 terabytes. A server memory system might deliver a few hundred gigabytes per second of bandwidth in aggregate. Reading three terabytes at three hundred gigabytes per second takes ten seconds, regardless of how fast the arithmetic units are. The scan is memory-bound: the processor spends almost all of its time waiting for data. This observation is worth internalizing because it explains the shape of nearly every technique in this book. Approximate methods win not primarily by doing less arithmetic, though they do, but by touching fewer bytes. Partitioning touches fewer vectors. Compression makes each vector fewer bytes. Graph navigation touches a small, query-dependent set of vectors. Re-ranking touches full-precision vectors only for a short candidate list. When evaluating any new technique, the first question to ask is how many bytes it reads per query and from which level of the memory hierarchy. Batching changes the picture somewhat. If many queries arrive together, a flat scan can load each stored vector once and compare it against all queries in the batch, turning a memory-bound operation into a compute-bound matrix multiplication. Graphics processors are extremely good at this, and for offline workloads such as computing nearest neighbours for every item in a catalogue, batched brute force on GPUs is often the fastest exact method available. The 2017 paper by Johnson, Douze, and Jégou that introduced Faiss's GPU implementation demonstrated exactly this kind of large batched computation, building a k-nearest-neighbour graph over a billion vectors on a small number of GPUs. Online serving, where queries arrive one at a time and each must be answered in milliseconds, does not get this benefit. Why trees stop working Computational geometry offers elegant exact data structures for nearest-neighbour search in low dimensions. The k-d tree, introduced by Jon Bentley in 1975, recursively splits space along one coordinate at a time, creating a binary tree whose leaves hold small groups of points. To answer a query, one descends to the leaf containing the query, finds the nearest point there, and then backtracks, visiting other branches only if the region they cover could contain a closer point. In two or three dimensions, this visits a tiny fraction of the data and answers queries in roughly logarithmic time. In high dimensions the backtracking step collapses. The test that prunes a branch asks whether the distance from the query to the splitting boundary exceeds the current best distance. In high dimensions, the current best distance is large relative to the spread of coordinates along any single axis, so almost no branch can be ruled out. A k-d tree over 768-dimensional data ends up visiting most of its leaves and performs worse than a flat scan because of the overhead of traversal. Ball trees, cover trees, vantage-point trees, and R-trees all suffer versions of the same problem. A rule of thumb from the database literature is that these exact tree structures stop beating a linear scan somewhere between ten and twenty dimensions, depending on the data. The underlying cause has a name. The curse of dimensionality The phrase "curse of dimensionality" was coined by Richard Bellman in the context of dynamic programming, but in nearest-neighbour search it refers to a specific and somewhat counterintuitive phenomenon: as dimension grows, distances concentrate. Take a large set of points scattered uniformly at random in a high-dimensional cube, and pick a query point. Compute the distance from the query to every point, and look at the ratio between the largest and the smallest. In two dimensions that ratio is enormous: some points are right next to the query and others are on the far side of the cube. As the dimension increases, the ratio shrinks toward one. Every point ends up at roughly the same distance from the query. A 1999 paper by Beyer, Goldstein, Ramakrishnan, and Shaft, titled "When Is Nearest Neighbor Meaningful?", proved that under fairly general conditions on the data distribution, the relative difference between the nearest and farthest distances goes to zero as dimension grows. If that held for real embeddings, nearest-neighbour search in high dimensions would be not only hard but pointless, because the "nearest" neighbour would be barely nearer than anything else. Two facts rescue the enterprise. First, real data is not uniform. Learned embeddings have low intrinsic dimension, as described in Chapter 1: the points lie close to a lower-dimensional structure within the nominal space, and they are clustered, with semantically related items forming dense regions. Distance concentration is governed by intrinsic dimension, so a 768-dimensional embedding collection with intrinsic dimension in the tens behaves far better than uniform random data in 768 dimensions. Second, even when distances are somewhat concentrated, the ranking can still be meaningful. Retrieval cares about which items are closest, not about whether the gap between closest and farthest is large in absolute terms. A model trained with a contrastive objective is explicitly pushed to separate relevant pairs from irrelevant ones by a margin, and that margin survives even when the bulk of distances cluster together. The curse leaves a mark nonetheless. Concentration means that the nearest neighbour is often only slightly closer than the tenth or hundredth, which makes approximate methods sensitive to small errors: a compressed distance that is off by a few per cent can reorder the top results. It also produces a phenomenon called hubness, documented by Radovanović, Nanopoulos, and Ivanović in 2010, in which a small number of points appear in the nearest-neighbour lists of a disproportionately large number of other points. Hubs are points that happen to lie near the centre of the data distribution. They show up in search results for many unrelated queries, which can look like a relevance bug, and they distort graph indexes by attracting a large share of edges. Approximate nearest neighbours Faced with the failure of exact methods, the field changed the question. Instead of demanding the exact k nearest neighbours, approximate nearest-neighbour search, ANN, asks for a set of k items that are probably among the nearest, or close to the nearest, with the probability or closeness controlled by parameters. The theoretical foundations were laid in the late 1990s. Indyk and Motwani's 1998 paper introduced locality-sensitive hashing, LSH, a family of hash functions with the property that nearby points collide with higher probability than distant ones. Hashing every stored vector with several such functions and looking up only the buckets the query falls into gives a sublinear-time algorithm with provable guarantees for the c-approximate nearest-neighbour problem: return a point within c times the true nearest distance. For cosine similarity, the classic LSH family, due to Charikar in 2002, uses random hyperplanes: each hash bit records which side of a random hyperplane the vector lies on. LSH was a breakthrough in theory and remains useful in streaming and privacy-sensitive settings, but in practice it has been largely displaced for dense embedding search. To reach high recall, LSH needs many hash tables, and each table stores a reference to every vector, so memory grows quickly. Its guarantees are worst-case and data-independent, which is precisely why it is outperformed on real data by methods that learn the data distribution. The partitioning, quantization, and graph methods that dominate today are all data-dependent: they look at the collection and organize themselves around it. They rarely come with worst-case guarantees, but on the data people actually have, they are much faster at a given recall. Recall, the measure of the bet To compare approximate methods we need a measure of how much they lose. The standard measure is recall at k. Run the query against the approximate index and retrieve k results. Run the same query with exact brute-force search and retrieve the true k nearest neighbours. Recall at k is the fraction of the true k that appear in the approximate k. If the index returns eight of the true ten nearest neighbours, recall at ten is 0.8. Averaging over a set of test queries gives the index's recall for that configuration. There are variants. Recall 1@10 asks whether the single true nearest neighbour appears anywhere in the top ten approximate results, a looser criterion used in some older literature, particularly with product quantization. Recall 10@100 asks what fraction of the true top ten appear in the approximate top hundred, which is relevant when the approximate results will be re-ranked. It is important to know which variant a paper or vendor is reporting, because the numbers are not comparable. A claim of "99 per cent recall" means little without the k values attached. The essential property of recall is that it measures fidelity to exact search, not relevance. An index with perfect recall reproduces the embedding model's ranking exactly. Whether that ranking is good is a question about the model. Engineers sometimes conflate the two, reporting retrieval quality metrics such as normalized discounted cumulative gain from an evaluation set and treating changes in them as changes in index quality. Index recall should be measured against brute force on the same embeddings, separately, so that the two sources of error can be attributed correctly. Recall alone is meaningless without cost. Any index can reach recall of one by scanning everything. The meaningful object is a curve: for each setting of the index's search parameters, plot throughput, usually queries per second on a fixed machine, against recall. A better index has a curve that lies above and to the right of a worse one, delivering more queries per second at the same recall. The ANN-Benchmarks project, created by Erik Bernhardsson and described in a 2018 paper by Aumüller, Bernhardsson, and Faithfull, standardized this presentation and made it the common language of the field. Its plots, one per dataset, show dozens of algorithms as recall-throughput curves, and they are the first thing most practitioners look at when comparing libraries. Other costs matter too and are often missing from the headline curve: memory consumption, index build time, the cost of inserting and deleting vectors, and latency percentiles rather than averages. A method that wins on throughput at a given recall but needs three times the memory, or takes a day to build, may be the wrong choice. The billion-scale Big-ANN benchmark competitions, organized at NeurIPS in 2021 and again in 2023 by Harsha Vardhan Simhadri and colleagues, broadened the evaluation by fixing hardware budgets and adding tracks for out-of-core search on solid-state drives, filtered search, streaming updates, and sparse vectors. Their results remain one of the best public sources of evidence about how methods behave at scale. Where the bytes live The final ingredient of this chapter is a model of the machine, because the right index depends on where its data sits. A modern server has a hierarchy of storage, each level larger, cheaper per byte, and slower than the one above. At the top are processor registers and on-chip caches, measured in kilobytes to a few hundred megabytes, with access times of a few nanoseconds. Below them is main memory, DRAM, typically hundreds of gigabytes to a few terabytes per server, with access latency around a hundred nanoseconds and bandwidth in the hundreds of gigabytes per second. Below that are NVMe solid-state drives, often tens of terabytes per server, with random read latency on the order of tens to a hundred microseconds and the ability to sustain hundreds of thousands to over a million random reads per second when many requests are in flight. Below those are networked storage and object stores, effectively unlimited in size, with latencies of milliseconds. Graphics processors have their own hierarchy: high-bandwidth memory on the accelerator, measured in tens of gigabytes per device up to a couple of hundred gigabytes on recent data-centre parts, with bandwidth in the terabytes per second, connected to the host by a link that is much slower than the device memory itself. Each index design in this book can be understood as an answer to the question of which data goes at which level. A pure in-memory HNSW index keeps full-precision vectors and the graph in DRAM; it is fast and expensive. An IVF index with product quantization keeps compact codes in DRAM, perhaps a tenth or a thirtieth of the original size, and scans them with cache-friendly code; it is cheaper and loses some recall to compression. DiskANN keeps compressed codes in DRAM for navigation and full vectors with the graph on SSD for re-ranking; it is cheaper still, and its latency is governed by the number of SSD reads per query. A GPU index keeps codes in accelerator memory and processes thousands of queries in parallel. A recurring and very useful technique combines levels: search with a compact approximate representation held high in the hierarchy, retrieve a candidate list larger than k, then fetch the precise representations of only those candidates from lower in the hierarchy and re-rank them exactly. The approximate stage determines which true neighbours can possibly be found, and the re-ranking stage determines their final order. If the candidate list is large enough to contain the true neighbours, re-ranking restores most of the precision that compression discarded. This two-stage pattern appears in nearly every serious large-scale system, and its parameters, the size of the candidate list and the precision of each stage, are among the most important levers an engineer has. With the problem framed as a series of bets constrained by bytes and measured by recall, the next three chapters examine the three bets in turn, starting with the oldest and conceptually simplest: carving the space into regions. Chapter 3. Partitioning: Inverted File Indexes The first bet is the most intuitive. If the collection is divided into regions, and similar items tend to fall in the same region, then a query need only examine the regions closest to it. The bet loses when a true nearest neighbour falls in a region the query does not visit. The design questions are how to draw the regions, how many to draw, how many to visit, and what to do about points that sit near the boundaries. The structure that implements this bet is called an inverted file index, IVF for short. The name comes from text retrieval, where an inverted index maps each word to the list of documents containing it. In the vector setting, the "words" are cluster centres, and each centre maps to the list of vectors assigned to it. The idea appears in the 2011 product quantization paper by Hervé Jégou, Matthijs Douze, and Cordelia Schmid, where it was paired with compression to form the IVFADC method, and it has remained a workhorse of large-scale search ever since. Faiss, Milvus, and many other systems offer IVF variants, and it is the backbone of several of the disk-based methods discussed in Chapter 6. Building the partition An IVF index is built in two steps. First, a set of cluster centres, called centroids, is learned from the data. Second, every vector in the collection is assigned to its nearest centroid and appended to that centroid's list. The centroids are almost always learned with k-means clustering. K-means starts with some initial set of centres, assigns every training point to its nearest centre, moves each centre to the mean of the points assigned to it, and repeats until the assignments stabilize. The result is a set of centres that roughly minimizes the total squared distance from points to their nearest centre. In the language of signal processing, the set of centres is a vector quantizer: it replaces any vector with the identity of the closest centre, and the quantization error is the distance between the vector and that centre. Because this quantizer only determines which list a vector goes into, and not how it is stored, it is called the coarse quantizer. K-means does not need to see the whole collection. A sample is enough, provided it represents the data distribution. The Faiss documentation recommends training on somewhere between thirty and two hundred and fifty-six times as many points as there are centroids; below that range the centroids are poorly estimated, and above it the extra points add cost without much benefit. For a billion-vector collection with a million centroids, this still means clustering tens or hundreds of millions of points, which is why GPU implementations of k-means became important early in the history of large-scale indexing. The number of centroids, usually written nlist, is the first major parameter. With nlist centroids and N vectors, each list holds on average N divided by nlist vectors. More lists means smaller lists, so each list is cheaper to scan, but it also means more centroids to compare the query against and a finer partition, in which true neighbours are more likely to be split across lists. A common rule of thumb is to set nlist proportional to the square root of N; Faiss's guidelines suggest between four and sixteen times the square root of N. For a hundred million vectors this gives somewhere between forty thousand and a hundred and sixty thousand lists, and Faiss users frequently choose a power of two such as 65,536. Searching the partition A query against an IVF index proceeds in two stages. In the first, the query is compared against all nlist centroids, and the nprobe closest centroids are selected. In the second, the query is compared against every vector in each of the selected lists, and the k closest are returned. The parameter nprobe is the main search-time control. Setting it to one examines a single list, which is fast but frequently misses neighbours that fall just across a boundary. Increasing it widens the search. Recall rises steeply at first and then flattens, approaching one only as nprobe approaches nlist, at which point the index has degenerated into a flat scan with extra overhead. A worked example makes the economics concrete. Suppose a hundred million vectors are partitioned into 65,536 lists, giving an average of about fifteen hundred vectors per list. With nprobe set to 64, a query compares against 65,536 centroids and then scans about 64 times 1,525, roughly ninety-eight thousand vectors. The total work is about a hundred and sixty thousand distance computations, roughly 0.16 per cent of the brute-force cost. If the vectors are stored at full precision in 768 dimensions, the list scan reads about three hundred megabytes, which on a single core is still too slow for a sub-millisecond target. This is why IVF is almost never used alone at that scale: it is paired with compression, so that the scanned lists are tens of times smaller, and with a fast structure for the centroid search, so that comparing against tens of thousands of centroids does not dominate. That centroid search is itself a nearest-neighbour problem, and at large nlist it becomes significant. Comparing a query against a million centroids by brute force costs as much as scanning a million vectors. The standard solution is to index the centroids with a graph. Faiss expresses this in its index factory notation with strings such as IVF1048576_HNSW32, meaning an IVF index with about a million lists whose centroids are found through an HNSW graph with 32 links per node. The graph adds a small approximation error in choosing which lists to probe, which in practice is usually negligible compared with the error from the partition itself. Why boundaries are the problem The recall loss in IVF is concentrated near the edges of the clusters. A query that lands near the middle of a cluster will typically find its nearest neighbours in the same cluster. A query that lands near the boundary between two or more clusters will have neighbours on both sides, and if nprobe is small, some of them will be missed. In high dimensions, almost everything is near a boundary. This follows from the same geometry that produces distance concentration. Each cluster has many neighbouring clusters, and a typical point is not much closer to its own centroid than to several others. Empirically, recall at nprobe of one is often poor on real embedding data, sometimes below fifty per cent, and reaching ninety-five per cent may need dozens or hundreds of probes. How many depends on the data and on nlist. Three families of technique address the boundary problem. The first is simply probing more lists, accepting the cost. Because list scanning with compressed codes is very efficient, this is often the pragmatic answer. The second is redundant assignment, also called spilling: place vectors that sit near a boundary into more than one list. Then a query that probes only one side of the boundary will still find them. The cost is memory, since some vectors are stored twice or more, and slightly larger lists. SPANN, a disk-oriented index from Microsoft published in 2021, uses a rule that assigns a vector to additional clusters when its distance to those centroids is within a small factor of its distance to the nearest one, and it limits the duplication to keep storage bounded. Google's SOAR technique, published in 2023 and integrated into the ScaNN library, makes the secondary assignment smarter. It chooses the spill cluster not merely by proximity but so that the quantization error with respect to the second centroid is as different as possible from the error with respect to the first, with the reasoning that a query poorly served by one assignment will then tend to be well served by the other. The third is to make the partition finer without making the centroid search more expensive. The inverted multi-index, introduced by Artem Babenko and Victor Lempitsky in 2012, splits each vector into two halves and runs k-means separately on each half. A vector's cell is the pair of its two half-centroids, so with K centroids per half the index has K squared cells. With K of 16,384, that is more than a quarter of a billion cells, a far finer partition than a single k-means could produce at reasonable cost. Searching requires enumerating cells in order of their combined distance to the query, which a priority queue does efficiently. The inverted multi-index was influential in the early billion-scale work on the SIFT1B and Deep1B datasets. It is less common in production now, partly because graph-based coarse quantizers made very large single-level nlist practical. Imbalance and its consequences K-means does not produce clusters of equal size. Real embedding collections contain dense regions, such as many near-duplicate product listings or many paraphrases of popular questions, and sparse ones. K-means places more centroids in dense regions but not proportionally enough, so some lists end up much longer than average. The variance in list length matters for two reasons. The first is that query cost becomes uneven. A query that lands in a dense region will probe long lists and take much longer than one that lands in a sparse region. Since dense regions are dense precisely because many items are similar to popular content, they are also where many queries land. The average query cost can therefore be considerably higher than the average list length would suggest, and the tail latency, the cost of the slowest one per cent of queries, can be several times the median. The second is that load across machines becomes uneven when lists are distributed across shards. A shard that holds the lists for a popular region will receive disproportionate traffic. Remedies include balanced k-means variants that penalize cluster size during training, hierarchical clustering that recursively splits oversized clusters, and capping list sizes at build time by reassigning overflow vectors to their second-nearest centroid. SPANN uses a hierarchical balanced clustering for exactly this reason: on disk, the cost of reading a list is proportional to its length, and an oversized list directly becomes a slow query. Each remedy trades some clustering quality for predictability. For a latency-sensitive service, predictability usually wins. Residuals: what the partition gives to compression IVF does more than restrict the search to a subset of vectors. It also provides a useful transformation of the data. Once a vector has been assigned to a centroid, the difference between the vector and its centroid, called the residual, is typically much smaller in magnitude than the vector itself and more uniformly distributed across clusters. Compressing residuals rather than raw vectors is markedly more accurate for the same code size, because the compression scheme only has to describe the local variation within a cluster rather than the vector's absolute position in the whole space. This is the central insight of IVFADC in the original product quantization paper. Vectors are stored in their lists as compressed codes of their residuals. At query time, the query's residual with respect to each probed centroid is computed, and distances are estimated between that query residual and the stored code. The cost is that the query must be re-encoded for each probed list, which adds a per-list overhead; the benefit is a substantial improvement in accuracy that has made residual encoding the default for IVF with product quantization. Where IVF stands today IVF has several practical virtues that keep it relevant despite the dominance of graph indexes in many benchmarks. It is memory-efficient. Apart from the centroids, which are small, the index adds only the storage of vector identifiers to the codes themselves. There is no graph with tens of edges per vector. When combined with aggressive compression, IVF can hold a billion vectors in a few tens of gigabytes of memory, which no graph index with full-precision vectors can approach. It is simple to update. Adding a vector means finding its nearest centroid and appending to a list. Deleting a vector means removing it from a list, or marking it deleted and compacting later. There is no graph connectivity to maintain. Over time, if the data distribution drifts, the centroids become stale and lists become unbalanced, so a periodic retraining and rebuild is needed, but the day-to-day operations are cheap. Its access pattern is friendly to hardware. Scanning a list is a sequential read of contiguous codes, which suits processor caches, wide vector instructions, GPUs, and solid-state drives far better than the random access pattern of a graph traversal. This is why IVF-style partitioning underlies the most successful GPU indexes and several of the best disk-based indexes. Its weakness is the recall-cost curve at high recall in memory. To push recall above ninety-five per cent, IVF needs many probes, and each probe has a fixed overhead plus the cost of scanning a list. Graph indexes, which reach the neighbourhood of the query directly and then explore only locally, typically need far fewer distance computations to reach the same recall. On the standard in-memory benchmarks, graph methods sit above IVF methods on the recall-throughput chart at high recall, often by a wide margin. The two families are therefore not competitors so much as tools for different points in the design space. IVF dominates when memory is the constraint, when updates are frequent, when the hardware favours sequential scanning, or when the partition itself is useful for distributing data across machines. Graph indexes dominate when the working set fits in memory and the priority is the lowest latency at high recall. Many large systems use both: a partition to divide the collection into shards or disk blocks, and a graph either within each partition or over the centroids. What makes IVF truly effective at scale is the thing it was designed to be paired with. A list of fifteen hundred full-precision vectors is expensive to scan; a list of fifteen hundred thirty-two-byte codes fits comfortably in a processor cache and can be scanned in a few microseconds. The next chapter explains how such codes are made, and what they cost. Chapter 4. Compression: Product Quantization and Its Descendants The second bet is that a vector does not need all of its bits to be found. A 768-dimensional embedding stored as 32-bit floats occupies 3,072 bytes. Much of that precision is noise from the point of view of ranking: the embedding model itself is not accurate to seven significant figures, and the ordering of neighbours usually survives a great deal of rounding. If each vector can be represented in 64 or 96 bytes instead of 3,072, then thirty to fifty times as many vectors fit in the same memory, each distance computation reads thirty to fifty times fewer bytes, and a memory-bound scan becomes correspondingly faster. The bet loses when compression reorders neighbours. Because distances in high dimensions are concentrated, the true nearest neighbour is often only slightly closer than the next few, and a small distortion in estimated distance can swap them. The art of vector compression is to make the distortion as small as possible for a given code size, to make distance computation on the compressed codes as fast as possible, and to recover lost precision cheaply by re-ranking. Product quantization The method that defined the field is product quantization, PQ, introduced by Jégou, Douze, and Schmid in a paper published in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2011. Its central trick is to avoid the impossibility of a single enormous codebook. Consider first plain vector quantization: learn a set of K centroids with k-means and represent each vector by the index of its nearest centroid. With K of 256, each vector compresses to a single byte. The problem is accuracy. Two hundred and fifty-six points cannot represent a billion distinct vectors with any fidelity. To get a useful approximation, K would need to be astronomically large, perhaps two to the sixty-fourth power for a 64-bit code, and no one can run k-means with that many centroids or store the result. Product quantization splits each vector into m contiguous sub-vectors of equal length. A 768-dimensional vector with m of 96 is split into 96 sub-vectors of eight dimensions each. A separate k-means is run on each sub-space, typically with 256 centroids, so each sub-vector can be encoded as one byte identifying its nearest sub-centroid. The full vector is represented by m bytes: 96 bytes in this example, a compression ratio of 32 against 32-bit floats. The effective codebook is the Cartesian product of the sub-codebooks. With 96 sub-spaces of 256 centroids each, the number of distinct vectors that can be represented is 256 raised to the 96th power, an unimaginably large number. Yet the storage for the codebooks is only 96 times 256 centroids of eight dimensions, about 786 kilobytes in 32-bit floats. That combination of a vast effective codebook with a tiny actual one is what makes PQ work. The approximation it makes is that the sub-spaces are quantized independently. If the coordinates in different sub-vectors are correlated, independent quantization wastes bits. If some sub-vectors carry far more variance than others, assigning the same eight bits to each is inefficient. These issues motivate the refinements discussed below. Computing distances on codes The second insight of PQ is that distances can be computed between a full-precision query and a compressed database vector without decompressing the vector. This is called asymmetric distance computation, ADC: asymmetric because the query is exact and only the database side is approximated. When a query arrives, it is split into the same m sub-vectors. For each sub-space, the squared distance from the query's sub-vector to each of the 256 sub-centroids is computed and stored in a small table. With m of 96, this produces 96 tables of 256 entries each, about 24,576 numbers. The cost is equivalent to roughly 256 full-dimensional distance computations, and it is paid once per query, or once per probed list when residuals are used. The squared Euclidean distance from the query to any encoded database vector is then approximated by summing m table entries: for each sub-space, look up the entry indexed by the vector's code byte and add it to a running total. No multiplication is needed, only m memory lookups and additions. The tables themselves are small enough to sit in the fastest processor caches. The alternative, symmetric distance computation, quantizes the query as well and looks up precomputed centroid-to-centroid distances. It is slightly faster but noticeably less accurate, and ADC is the norm. Asymmetric distance has a further benefit for inner product search. The inner product of a query with a vector decomposes into a sum over sub-spaces, just as squared distance does, so the same table-lookup procedure works for MIPS with tables of inner products instead of distances. Making PQ fast on real processors A naive implementation of the table lookup is limited by the latency of memory accesses into the tables. In the mid-2010s, Fabien André, Anne-Marie Kermarrec, and Nicolas Le Scouarnec showed that PQ scanning could be made dramatically faster by reorganizing the computation to exploit SIMD instructions. Their key move was to use 16 centroids per sub-space instead of 256, so each code is four bits rather than eight. A table of 16 entries, quantized to 8-bit integers, fits inside a single 128-bit processor register, and the processor's byte shuffle instruction can then perform sixteen table lookups in parallel in one instruction. With wider registers, more lookups proceed at once. Codes are laid out in memory in blocks interleaved so that the shuffle can process many database vectors simultaneously. This approach, often called fast-scan PQ or PQ with 4-bit codes, sacrifices some accuracy per bit, since 16 centroids per sub-space are coarser than 256, but it is several times faster to scan. The usual remedy for the lost accuracy is to scan more codes quickly and re-rank a candidate list with more precise data. Faiss incorporated fast-scan variants of PQ around 2020, and Google's ScaNN uses a similar 4-bit SIMD approach in its scoring. Rotations and learned refinements Optimized product quantization, OPQ, published by Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun at CVPR 2013, addresses PQ's dependence on how dimensions are grouped. It learns an orthogonal rotation of the space, applied before splitting into sub-vectors, chosen to minimize quantization error. Intuitively, the rotation spreads variance evenly across sub-spaces and reduces the correlation between them, so that the independent quantizers work closer to their ideal. Because rotation preserves distances, it changes nothing about the search problem except how well it can be compressed. OPQ is cheap at query time, since the query is rotated once with a single matrix multiplication, and in Faiss it appears as a preprocessing step in index strings such as OPQ64,IVF65536,PQ64. Beyond OPQ lies a family of additive quantizers. Instead of splitting the vector into disjoint pieces, these methods represent the vector as a sum of several full-dimensional codewords, one from each of m codebooks. Residual quantization, the oldest of these, encodes a vector with the first codebook, encodes the remaining error with the second, and so on. Additive quantization and local search quantization search more thoroughly for the combination of codewords that best reconstructs each vector. These methods achieve lower reconstruction error than PQ for the same code size, sometimes substantially so, but encoding is much slower and distance computation needs an extra correction term. They tend to be used where compression ratio matters more than ingestion speed. Google's ScaNN, described by Ruiqi Guo and colleagues at ICML 2020, contributed a different kind of refinement: anisotropic vector quantization. The observation is that for maximum inner product search, not all quantization errors are equally harmful. An error in the component of a vector parallel to the vector itself changes its inner product with queries that are similar to it, which are exactly the queries for which the vector might be a top result. An error in the perpendicular direction matters much less for those queries. ScaNN's loss function penalizes parallel error more heavily than perpendicular error, producing codebooks that sacrifice average reconstruction accuracy to protect the ranking of high-scoring items. On inner product benchmarks, this gave ScaNN a leading position when it was released. Scalar quantization Product quantization is powerful but has costs: a training step, a codebook that must match the data distribution, and an encoding step at ingestion. A much simpler alternative is scalar quantization: compress each coordinate independently. The most common form maps each 32-bit float coordinate to an 8-bit integer. For each dimension, the system records a minimum and maximum (or a scale and offset), and linearly maps the range onto the 256 integer values. A 768-dimensional vector becomes 768 bytes, a factor of four smaller. Distance computation proceeds directly on the integers using the processor's integer vector instructions, which on modern hardware are very fast. The loss in recall from 8-bit scalar quantization on typical text embeddings is usually small, often negligible after re-ranking. Halving precision to 16-bit floats is even safer, giving a factor of two with almost no measurable effect. Scalar quantization has become the default compression in many production systems for exactly these reasons. It requires little or no training, it is robust to distribution shift because it does not learn a codebook tied to clusters, it is straightforward to apply inside graph indexes, and its accuracy loss is easy to recover by re-ranking. Lucene, which underlies Elasticsearch and OpenSearch, added scalar quantization of HNSW vectors in 2023, with the quantization ranges computed per segment so that they adapt as data arrives. Intel's Locally-adaptive Vector Quantization, LVQ, published by Cecilia Aguerrebere and colleagues in 2023, refines the idea by computing a separate scale and offset per vector after subtracting the dataset mean, which improves accuracy at four and eight bits per dimension and is used in Intel's Scalable Vector Search library. Binary quantization and its revival At the extreme, each coordinate can be reduced to a single bit: one if positive, zero if negative. A 768-dimensional vector becomes 96 bytes, a factor of 32 smaller, the same size as the PQ example above but with no training at all. Distance between two binary codes is the Hamming distance, the number of differing bits, which modern processors compute with an exclusive-or followed by a population count instruction. This is the cheapest distance computation available: a few instructions per 64 dimensions. Binary codes for similarity search have a long history in the hashing literature, including the random-hyperplane LSH described in Chapter 2. Sign-based binarization of raw embeddings was long considered too lossy for serious use. Two developments changed that view. The first was empirical. As embedding models grew to 1,024 dimensions and beyond, practitioners found that one-bit codes preserved neighbourhoods surprisingly well, particularly when combined with oversampling and re-ranking: retrieve several times more candidates than needed using Hamming distance, then re-score those candidates with full-precision or 8-bit vectors. Several embedding providers began training models with quantization in mind, and some publish models whose binary versions are documented to retain most of their full-precision retrieval quality. The exact retained fraction depends on the model and task, and should be checked on one's own data rather than assumed. The second was theoretical. RaBitQ, published by Jianyang Gao and Cheng Long at SIGMOD 2024, showed how to make binary quantization principled. It applies a random orthogonal rotation to the data, after normalizing vectors relative to a centroid, and then takes one bit per dimension. The random rotation ensures that information is spread evenly across coordinates, so that no single dimension dominates. Crucially, RaBitQ provides an unbiased estimator of the true distance from the binary code plus a small amount of auxiliary information per vector, together with a probabilistic error bound. That bound allows the search to decide, for each candidate, whether its estimated distance is confidently far enough from the current threshold to discard, or uncertain enough to require re-ranking. An extended version supports more than one bit per dimension. RaBitQ has been adopted in several open-source systems and has influenced how others think about compression: as a calibrated estimator with known error, rather than merely a lossy encoding. The economics of compression Table 1 compares the main compression schemes for a 768-dimensional vector. The byte counts follow directly from the definitions and exclude per-index overhead such as identifiers, graph edges, and codebooks; recall effects depend heavily on data and are described qualitatively. Table 1. Storage and distance cost of common compression schemes for a 768-dimensional embedding. Scheme Bytes per vector Ratio vs float32 Distance computation Typical role float32 (none) 3,072 1x Floating-point multiply-add Exact reference and re-ranking float16 / bfloat16 1,536 2x Converted multiply-add Near-lossless storage int8 scalar 768 4x Integer multiply-add Default in-graph compression PQ, 96 sub-spaces, 8-bit 96 32x 96 table lookups IVF list scanning PQ fast-scan, 192 sub-spaces, 4-bit 96 32x SIMD shuffle lookups High-throughput candidate generation Binary, 1 bit per dimension 96 32x XOR and popcount First-pass filtering before re-ranking Three lessons follow from comparing the rows. First, at the same 96 bytes, PQ and binary quantization make very different trade-offs. PQ learns codebooks that capture the data distribution and typically gives more accurate distance estimates for the bytes spent, but requires training and is sensitive to drift. Binary quantization is training-free and its distance computation is cheaper, but it discards magnitude information per coordinate and depends on high dimensionality and on re-ranking to work well. Second, compression ratio translates directly into money. A billion vectors at float32 need about 3.1 terabytes for the vectors alone. At int8 they need about 770 gigabytes, which fits in the memory of a single large server. At 96 bytes they need about 96 gigabytes, which fits in the memory of a modest one or a single high-end GPU. The difference between needing a dozen machines and needing one is the difference that makes some projects economical. Third, compression rarely stands alone; it is nearly always paired with re-ranking. The compressed representation is used to generate a candidate list, typically several times larger than k, and the full-precision vectors of those candidates are fetched, from memory if they fit or from SSD if they do not, and used to compute exact distances. The final top k are taken from the re-ranked list. The recall of the combined system depends on whether the true top k survive into the candidate list, which depends on compression quality and on how large the candidate list is. Doubling the candidate list costs a little more re-ranking work and often recovers most of the recall lost to aggressive compression. This two-stage structure also changes how one should judge a compression scheme. The question is not "how accurate are the compressed distances?" but "how large must the candidate list be for the true neighbours to survive?" A crude scheme with very cheap distances and a large candidate list can beat an accurate scheme with expensive distances and a small one. When compression hurts Compression is not free of risk. Learned codebooks, whether for PQ, OPQ, or additive quantizers, are fitted to the data distribution at training time. If the distribution shifts, for instance because a new product category is added or because the embedding model is upgraded, the codebooks fit the new data poorly, and errors grow. Systems that rely on learned codebooks need a plan for retraining them, which usually means rebuilding the index. Compression interacts badly with near-duplicate detection and with applications that need precise score thresholds. If an application decides relevance by whether a similarity exceeds a fixed value, compressed distances may push borderline items across the line. Re-ranking with exact distances before applying the threshold avoids the problem. Compression also interacts with the choice of index. Graph indexes navigate by comparing the query against neighbours at every step, and if those comparisons use compressed distances, errors can send the walk in the wrong direction early, which is much harder to recover from than an error in a flat scan. Systems that compress inside graphs typically use mild compression, such as int8, for navigation, or use more aggressive compression with a larger search beam and re-ranking at the end. That brings us to graphs, the third bet and the one that dominates in-memory vector search today. Chapter 5. Navigation: Hierarchical Navigable Small World Graphs The third bet is that the collection can be turned into a map, with each vector linked to a few of its neighbours, so that a query can find its nearest neighbours by walking. Start anywhere, look at the neighbours of the current point, move to whichever is closest to the query, and repeat until no neighbour is closer. If the graph is well built, this greedy walk arrives in the right neighbourhood after a modest number of steps, and a slightly broader local exploration then finds the nearest neighbours. The bet loses when the walk gets stuck in a local minimum: a point closer to the query than all of its neighbours, yet not the true nearest. Graph indexes reach high recall with fewer distance computations than any other family on in-memory benchmarks, and the most widely deployed of them, HNSW, has become the default index in a large share of vector databases and search engines. Lucene, and therefore Elasticsearch and OpenSearch, use it. So do pgvector, Qdrant, Weaviate, Milvus among its options, Redis, and many others. Understanding how it works, why it works, and where it struggles is the core technical competence of a vector search engineer. Proximity graphs and greedy search The idea of searching by walking through a graph of neighbours predates HNSW by decades. A proximity graph is any graph in which each data point is a node and edges connect points that are close. The theoretically ideal structure for greedy search in a metric space is the Delaunay graph, in which two points are connected if there is an empty sphere touching both of them. Greedy search on a Delaunay graph always finds the exact nearest neighbour. Unfortunately, in high dimensions the Delaunay graph becomes nearly complete, with almost every point connected to almost every other, so it is useless as an index. Practical graphs approximate it with far fewer edges. The simplest is the k-nearest-neighbour graph, in which each point links to its k closest points. Greedy search on such a graph works reasonably well locally but has two problems. It can be poorly connected globally, since clusters of mutually close points may link only to each other, leaving the graph fragmented. And the walk from a distant starting point to the query's neighbourhood can take many steps, because every edge is short. The second problem points to the key requirement. Efficient navigation needs a mix of short edges, which allow precise local search, and long edges, which allow fast movement across the space. This is the structure of a small-world network, a concept made famous by Duncan Watts and Steven Strogatz in 1998. Jon Kleinberg showed in 2000 that for decentralized greedy routing to work efficiently, the long-range links must follow a specific distribution: the probability of a link between two points should fall off with distance in a particular way, so that at every scale there are links of roughly the right length to make progress. Graphs with this property are called navigable. From NSW to HNSW Yury Malkov and colleagues introduced the navigable small world graph, NSW, in a series of papers culminating in a 2014 article in Information Systems. Their construction was strikingly simple. Insert points one at a time in random order. For each new point, find its approximate nearest neighbours among the points already inserted, using greedy search on the graph built so far, and connect the new point to them with bidirectional edges. The small-world property emerges from the insertion order. Early in construction, when the graph contains few points, the "nearest neighbours" of a new point may be far away in absolute terms, so the edges created are long. Later, when the graph is dense, edges are short. The final graph therefore contains a natural mix of long early edges and short late ones. NSW performed well but its search complexity grew polylogarithmically with collection size, and performance degraded on clustered data and low-dimensional data, where long-range edges were poorly distributed. In 2016, Malkov and Dmitry Yashunin posted a paper, later published in IEEE Transactions on Pattern Analysis and Machine Intelligence, that introduced the hierarchical version. HNSW separates edges by length into explicit layers, borrowing the structure of a skip list, the probabilistic data structure invented by William Pugh in 1990 for ordered search. In a skip list, every element is in the bottom level. A random subset, about half, is also in the level above; a random half of those are in the level above that; and so on. Searching starts at the sparse top level, moves forward as far as possible without overshooting, drops down a level, and repeats. The upper levels provide long jumps; the lower levels provide precision. HNSW applies the same idea to proximity graphs. Every vector is a node in layer zero, the bottom layer. Each vector is also assigned a maximum layer drawn at random from an exponentially decaying distribution, so that most vectors exist only in layer zero, a fraction exist in layer one as well, a smaller fraction in layer two, and a handful at the top. Within each layer, nodes are linked to their neighbours among the nodes present in that layer. Because upper layers are sparse, their edges are long; because layer zero contains everything, its edges are short. How search works A search begins at a single entry point, a node in the top layer. In each upper layer, the algorithm performs greedy search: repeatedly move to whichever neighbour of the current node is closest to the query, until no neighbour is closer. The node where the greedy search stops becomes the entry point for the next layer down. Upper layers thus act as a fast coarse router, moving the search from an arbitrary starting point into the right region of space in a small number of hops. At layer zero the algorithm switches from pure greedy search to a beam search. It maintains two structures: a candidate queue of nodes to explore, ordered by distance to the query, and a result set of the best ef nodes found so far, where ef is a search-time parameter. At each step, it takes the closest unexplored candidate, computes distances to all of that candidate's neighbours that have not been visited, and adds any that are closer than the worst member of the result set to both the candidates and the results, trimming the result set back to ef. The search stops when the closest remaining candidate is farther than the worst member of the result set, meaning no further exploration can improve the results. The top k of the result set are returned. The parameter ef, often called efSearch, is the principal search-time control for recall and speed, in the same way nprobe is for IVF. With ef equal to k, the beam is narrow and the search is fast but more likely to miss neighbours. Increasing ef widens the beam, explores more of the graph, and raises recall at the cost of more distance computations. Because ef can be changed per query without rebuilding the index, applications can trade speed for recall dynamically, for example using a larger ef for a premium tier or for queries whose results will feed a sensitive downstream task. The number of distance computations in an HNSW search is typically a few hundred to a few thousand for collections of millions of vectors at high recall. Compared with a flat scan of millions, this is the source of HNSW's speed. The original paper reported that the number of distance computations grew approximately logarithmically with collection size on the datasets tested, which is the behaviour the skip-list analogy predicts. How construction works Inserting a new vector into HNSW follows the same pattern as search. Draw the vector's maximum layer at random. Starting from the entry point, perform greedy search through the layers above the new vector's maximum layer. Then, for each layer from the vector's maximum layer down to zero, perform a beam search with a construction-time beam width, efConstruction, to find candidate neighbours. From those candidates, select up to M neighbours and connect the new vector to them bidirectionally. If adding the reverse edge pushes an existing node over its maximum number of neighbours, prune that node's neighbour list. Two parameters govern the graph's shape. M is the maximum number of neighbours per node in the upper layers; layer zero typically allows twice as many, 2M, because it carries the bulk of the search work and benefits from denser connectivity. efConstruction is the beam width used to find candidates during insertion; larger values produce better-quality neighbour lists at the cost of slower construction. The level distribution is controlled by a normalization factor, which the paper recommends setting to one over the natural logarithm of M, so that each layer is roughly M times sparser than the one below. The most important detail of construction, and the one most often missed in casual descriptions, is how the M neighbours are chosen from the candidates. The naive approach takes the M closest. HNSW instead uses a heuristic that favours diversity. Candidates are considered in order of distance from the new node. A candidate is accepted only if it is closer to the new node than it is to any neighbour already accepted. Otherwise it is skipped, on the reasoning that an already-accepted neighbour lies between the new node and this candidate and therefore provides a path to it. The effect is that a node's edges point in many different directions rather than clustering toward the densest nearby region. This matters enormously on clustered data. Without the heuristic, a node at the edge of a cluster would link only to other members of its cluster, and the graph would be poorly connected between clusters. With it, the node keeps at least one edge reaching toward the neighbouring cluster, because a member of that other cluster is not closer to any of the node's within-cluster neighbours than it is to the node itself. This pruning rule is related to the relative neighbourhood graph from computational geometry, and closely related rules appear in other graph indexes. The NSG index of Cong Fu and colleagues, published at VLDB in 2019, uses a monotonic search network construction built on the same idea. Vamana, the graph inside DiskANN, generalizes it with a parameter alpha greater than one that relaxes the pruning condition, deliberately keeping some longer edges to reduce the number of hops a search needs. Fewer hops matter greatly when each hop is an SSD read, as Chapter 6 explains. Parameters in practice Table 2 summarizes the effect of the three main HNSW parameters. The directions of effect are well established; the specific values that work best depend on the data, its dimension, and the recall target, and should be found by measurement. Table 2. Effects of the principal HNSW parameters. Parameter Set when Raising it improves Raising it costs Common starting range M Build Recall ceiling, robustness on hard data Memory per vector, build time, per-hop work 16 to 64 efConstruction Build Graph quality, achievable recall Build time, roughly linearly 100 to 500 ef (efSearch) Query Recall at query time Latency and distance computations k up to several hundred Memory deserves a closer look because it is often underestimated. With M of 16, each node at layer zero stores up to 32 neighbour identifiers. At four bytes each, that is 128 bytes per vector for layer zero alone, plus a small expected amount for the sparse upper layers. For 768-dimensional float32 vectors, 128 bytes is modest against 3,072 bytes of vector data. For a 96-byte compressed code, however, the graph is larger than the vector itself. As compression improves, graph overhead becomes the dominant memory cost, which is one reason IVF-based indexes remain competitive when memory is tight. Higher-dimensional and harder datasets benefit from larger M. Data with high intrinsic dimension needs more edges per node to stay navigable, and the HNSW paper and subsequent practice both suggest moving up from M of 16 toward 32, 48, or 64 for such data. Low-dimensional data, conversely, does well with small M. Build time is dominated by the beam searches performed during insertion. Each insertion costs about as much as a search with ef equal to efConstruction, at every layer the node occupies. For a hundred million vectors at efConstruction of 200, construction involves on the order of tens of billions of distance computations; it parallelizes across cores, since insertions can proceed concurrently with fine-grained locking on neighbour lists, but a large index still takes hours on a single machine. Faster construction is an active area. GPU-built graphs, discussed in the next chapter, have reduced build times by an order of magnitude or more in some settings. Why the hierarchy may matter less than it seems The skip-list hierarchy is the feature that gives HNSW its name, and it is natural to assume that it is responsible for its performance. Recent research complicates this picture. Several studies have compared HNSW with a flat navigable graph built using the same neighbour-selection heuristic but without upper layers, starting searches from a fixed or randomly chosen entry point. A 2024 study by Blaise Munyampirwa and colleagues, titled "Down with the Hierarchy: The 'H' in HNSW Stands for 'Hubs'", reported that on high-dimensional datasets the flat graph achieved essentially the same latency and recall as the hierarchical one while using less memory. Their explanation draws on the hubness phenomenon described in Chapter 2: in high dimensions, a small set of hub nodes is well connected to large parts of the graph, and these hubs act as a natural highway, so greedy search from almost any starting point reaches them quickly and then descends into the right region. The hierarchy does still help in low dimensions, where hubness is weak, and in pathological cases where the starting point lies far from the query's region. Because the upper layers are cheap, containing only a small fraction of nodes, there is little reason to remove them in practice. But the finding is a useful corrective to the idea that the hierarchy is what makes graph search work. The decisive ingredients are the diversity-preserving neighbour selection and sufficient edges per node. The operational problems HNSW's search performance is excellent. Its operational characteristics are more demanding, and these are what engineers running it in production tend to spend their time on. Memory and random access. HNSW is designed for main memory. Every hop in a search jumps to a node whose vector and neighbour list are at an essentially random address. A search with a few hundred distance computations incurs a few hundred cache misses, each costing on the order of a hundred nanoseconds. At a few hundred hops, that alone accounts for tens of microseconds. Implementations hide some of this latency by issuing prefetch instructions for the next neighbours' vectors while computing the current distance, and well-tuned code can process millions of distance computations per second per core. But the random access pattern is why HNSW is a poor fit for SSDs, where each random read costs tens of microseconds rather than a hundred nanoseconds, and for GPUs, which prefer large regular memory accesses. Deletion. HNSW was designed for insertion. Removing a node from a graph is difficult because other nodes depend on it for connectivity: it may be the only path from one region to another. The common approach, used in hnswlib, the reference library by Malkov, is to mark deleted nodes with a tombstone. Searches traverse them but exclude them from results. This preserves connectivity but wastes work and memory. As deletions accumulate, a growing fraction of each search is spent on dead nodes, and recall measured against live data can fall. Some implementations allow the slot of a deleted node to be reused for a new insertion, repairing the neighbour lists of affected nodes. Others periodically rebuild the graph. Graph repair after deletion has been studied directly. FreshDiskANN, from Microsoft in 2021, proposed a consolidation procedure that reconnects the in-neighbours of deleted nodes to the deleted nodes' out-neighbours, with pruning. But for many production HNSW deployments, the practical answer remains periodic rebuilds, and the deletion rate is a key input to capacity planning. Unreachable nodes. Even without deletions, pruning during construction can leave some nodes with no incoming edges from the rest of the graph, or reachable only by long detours. Such nodes can never be returned by a search, regardless of ef. The fraction is typically small but not zero, and it grows with aggressive pruning, low M, and frequent updates. Detection requires a reachability analysis over the graph, which some systems perform as part of maintenance. Segment merging. Systems built on log-structured storage, such as Lucene, write data in immutable segments, each with its own HNSW graph, and periodically merge small segments into larger ones. A query must search every segment and merge the results, so many small segments are slow. Merging requires rebuilding a graph over the combined vectors, which is expensive. The interaction of HNSW build cost with segment merging is one of the main performance considerations in Lucene-based vector search, and Lucene has added optimizations that reuse the graph of the largest segment as a starting point during merge rather than building from scratch. Determinism. HNSW construction depends on random level assignment and, in parallel builds, on the order in which concurrent insertions complete. Two builds of the same data can produce different graphs with slightly different recall. This is harmless for most uses but surprises teams who expect identical results from repeated builds, for example in regression testing. Graphs beyond HNSW HNSW is the most deployed graph index, but not the only significant one. NSG, mentioned above, builds a single-layer graph by first constructing an approximate k-nearest-neighbour graph and then refining it, with a fixed entry point at the approximate centre of the dataset. It achieves smaller graphs than HNSW at comparable performance and was used in production at Alibaba's Taobao. Vamana, from DiskANN, builds a single-layer graph with the alpha-relaxed pruning rule and is designed so that searches need few hops, making it suitable for storage where each hop is expensive. CAGRA, from NVIDIA researchers, builds a fixed-degree graph optimized for GPU search. The common thread is that all modern graph indexes use some form of diversity-preserving pruning, and they differ chiefly in how they balance graph size, number of hops, and construction cost against the memory hierarchy they target. Graph indexes in main memory, then, give the best recall for the least computation, at the price of memory, random access, and awkward updates. The question for billion-scale systems is what to do when the collection does not fit in memory at all, or fits only at a cost no one wants to pay. Chapter 6. Beyond Main Memory: Solid-State Drives and Graphics Processors An in-memory HNSW index over a billion 768-dimensional float32 vectors needs roughly 3.1 terabytes for vectors and another hundred-odd gigabytes for the graph. That can be bought. A handful of large-memory servers will hold it, or a few dozen ordinary ones. But memory is the most expensive tier of storage per byte in a data centre, and a billion vectors is no longer an unusual collection. A system that needs many billions, or many separate billion-scale indexes for different customers, cannot assume that everything lives in DRAM. Two developments made billion-scale search practical without proportionally large memory budgets. The first was learning how to search effectively from solid-state drives, which offer ten or more times the capacity per dollar of DRAM and, crucially, can serve large numbers of random reads in parallel. The second was the adoption of graphics processors, which have far less memory than a server's DRAM but far more bandwidth and parallelism, and which changed the economics of both index construction and high-throughput querying. Why naive disk search fails Take an HNSW index and put it on an SSD. Each hop in a search reads one node: its vector and its neighbour list. On DRAM a hop costs around a hundred nanoseconds; on an NVMe SSD a random read costs tens of microseconds to around a hundred, depending on the device and load. A search with three hundred sequential hops would take tens of milliseconds, a hundred times slower than in memory. Moreover, each 3,072-byte vector read pulls in a full 4-kilobyte page from the device, and most of the neighbour vectors read are not useful. The problem is the combination of two things: the number of sequential round trips, since each hop depends on the result of the previous one, and the amount of data read per round trip. A disk-friendly index must reduce both. It should reach the answer in few hops, it should issue many reads in parallel within each hop rather than one at a time, and it should make every byte read from the device useful. DiskANN DiskANN, introduced by Suhas Jayaram Subramanya and colleagues at Microsoft Research India at NeurIPS in 2019, was the first system to show convincingly that billion-scale search with high recall and low-millisecond latency could be served from a single machine with modest memory and an SSD. Its design addresses each of the problems above. The graph is Vamana, a single-layer graph built with the diversity-preserving pruning rule described in Chapter 5, relaxed by a parameter alpha slightly greater than one. The relaxation retains some edges that strict pruning would discard: a candidate is now removed only if an existing neighbour is substantially closer to it, by a factor of alpha, than the node itself. The retained longer edges let the search make larger jumps, reducing the number of hops needed to reach the query's neighbourhood. Vamana graphs use a higher maximum degree than typical HNSW layer-zero graphs, commonly in the range of 64 to 128, which further shortens paths. The layout puts each node's full-precision vector and its neighbour list together in a contiguous block on the SSD, sized to fit in one or a few sectors. One read brings back both the vector, for exact distance computation, and the list of neighbours to explore next. Meanwhile, compressed product-quantization codes for every vector are held in DRAM. These are small, perhaps 32 to 64 bytes each, so a billion of them fit in a few tens of gigabytes. During search, the algorithm uses these in-memory codes to estimate distances to neighbours and decide which nodes to visit next, without touching the SSD. Only the nodes actually chosen for expansion are read from disk. The search uses a beam: at each round, instead of expanding one node, it expands the W most promising unexpanded candidates simultaneously, issuing W SSD reads in parallel. Modern NVMe devices achieve their best throughput with many requests outstanding, so reading four or eight nodes at once costs little more latency than reading one. A beam width of a handful cuts the number of sequential rounds proportionally. Re-ranking comes for free. Every node read from the SSD brings its full-precision vector with it, so exact distances to all expanded nodes are available. The final results are ranked by these exact distances, not the compressed ones, which recovers the precision lost by navigating with PQ codes. Finally, a cache holds the nodes within a few hops of the entry point in DRAM, since every search passes through them. The paper reported that on the SIFT1B dataset, a billion 128-dimensional vectors, DiskANN served queries at high recall with mean latency of a few milliseconds from a workstation with 64 gigabytes of RAM and an inexpensive SSD. That configuration would have required several terabytes of DRAM for an equivalent in-memory graph. The result changed the economics of billion-scale search and has been the reference point for disk-based methods since. Building the index presented its own challenge: constructing a graph over a billion points requires holding the graph in memory, which defeats the purpose. DiskANN partitions the data into overlapping clusters with k-means, assigning each point to two or more nearby clusters, builds a Vamana graph on each cluster in memory, and merges the graphs, using the overlap to stitch them together. The result is a single graph built with memory proportional to one cluster rather than the whole collection. FreshDiskANN, published in 2021, extended the design to support insertions and deletions. New vectors go into a small in-memory graph, deletions are recorded in a list, and a background process periodically merges these changes into the SSD-resident graph, consolidating around deleted nodes as described in Chapter 5. Filtered-DiskANN, published at the Web Conference in 2023, extended it to queries with label constraints, discussed in Chapter 7. The open-source DiskANN codebase and its descendants are used in Microsoft's own services and have been integrated into other databases, including as an index type in Milvus and as the basis of vector indexing in some relational database products. SPANN SPANN, published by Qi Chen and colleagues at Microsoft at NeurIPS in 2021, takes the opposite architectural approach. Rather than a graph on disk with compressed codes in memory, it keeps a partition in memory and posting lists on disk. It is, in essence, an IVF index re-engineered for SSDs. SPANN's centroids are held in memory and indexed by a graph so that the nearest centroids to a query can be found quickly. There are many of them, a significant fraction of the number of vectors, so that each posting list is short. Each posting list, holding the full-precision vectors of its cluster, is stored contiguously on the SSD. A query finds its nearest centroids in memory, reads the corresponding posting lists from disk in parallel, and computes exact distances. The design's success depends on solving the problems with IVF described in Chapter 3. To prevent oversized posting lists, which become slow reads, SPANN uses hierarchical balanced clustering to keep lists of roughly equal size. To handle boundary effects, it assigns vectors near cluster boundaries to multiple posting lists, with a rule that limits duplication. And to avoid reading lists that are unlikely to contain results, it prunes the list of probed centroids dynamically per query: if a centroid is much farther from the query than the nearest centroid, its list is skipped even if it would otherwise be within the probe count. The SPANN paper reported higher recall at lower latency than DiskANN at comparable memory budgets on the billion-scale datasets it tested. Comparisons of this kind depend heavily on configuration, and both designs have been improved since, so the safe conclusion is that both approaches are competitive and suit different workloads. SPANN's access pattern, reading whole lists sequentially, suits SSDs well and makes updates simpler. SPFresh, a follow-up published at SOSP in 2023, supports in-place updates by splitting posting lists that grow too large and reassigning nearby vectors locally, avoiding global rebuilds. Graphics processors Graphics processors approach the problem from the other end of the memory hierarchy. A data-centre GPU has much less memory than a server, from tens of gigabytes to around two hundred on recent parts, but that memory delivers bandwidth in the terabytes per second, several times what a server processor can reach, and the device runs tens of thousands of threads concurrently. The 2017 Faiss GPU paper by Jeff Johnson, Matthijs Douze, and Hervé Jégou established the core techniques: highly optimized brute-force search using matrix multiplication; IVF with product quantization in which list scanning is parallelized across thousands of threads; and an efficient on-GPU k-selection algorithm for picking the top results from large sets of candidate distances, a step that is surprisingly costly when done naively. With IVF-PQ codes, a billion vectors at 32 to 64 bytes each fit across a small number of GPUs, and the paper demonstrated billion-scale k-nearest-neighbour graph construction in a fraction of the time CPU methods needed. Graph search on GPUs is harder, because it is inherently sequential and irregular. Each query's walk depends on its previous step, and different queries diverge, which wastes the GPU's lock-step parallelism. CAGRA, introduced by Hiroyuki Ootomo and colleagues at NVIDIA and published in 2024, designed a graph specifically for the GPU. It builds a fixed-degree graph, starting from an approximate k-nearest-neighbour graph and then optimizing it by pruning redundant edges and adding reverse edges, so that every node has exactly the same number of neighbours, which suits regular memory access. Its search processes many candidate expansions per query in parallel, using a whole group of threads per query, and uses a small hash table for visited nodes in fast on-chip memory. CAGRA reported high throughput at high recall for large batches, and competitive latency even for small batches. The more lasting impact of GPUs may be on construction rather than search. Building a high-quality graph is far more expensive than querying it, and GPU construction is often an order of magnitude or more faster than CPU construction. Because CAGRA graphs can be converted into an HNSW-compatible format, a common pattern has emerged: build the graph on GPUs, then serve it on CPUs. NVIDIA's cuVS library, which grew out of the RAPIDS RAFT project, packages CAGRA, IVF-PQ, and related algorithms, and has been integrated as a backend in Faiss and in several vector databases. GPUs are not a universal answer. Their memory is limited and expensive, so GPU-resident indexes rely on heavy compression or on many devices. They achieve their best efficiency with large query batches, which suits offline jobs and very high-traffic services but not latency-critical services with modest traffic, where a GPU may sit mostly idle. And moving data between host memory and GPU memory over the interconnect is slow relative to on-device bandwidth, so indexes that spill out of GPU memory lose much of their advantage. The cold tier A further tier has become important as vector search moved into multi-tenant cloud services: object storage. Services that host many indexes for many customers, most of which are queried rarely, cannot afford to keep every index in memory or even on local SSD. A design pattern has emerged in which index files, typically partitioned in an IVF-like fashion, are stored in an object store, and caches on local SSD and in memory hold recently used portions. A query against a warm index is served from cache with latency in milliseconds; a query against a cold index must fetch data from object storage first and may take hundreds of milliseconds. Several commercial and open-source systems use variants of this architecture, and in 2025 Amazon announced native vector storage and querying within its S3 object store, aimed at large, infrequently queried collections where cost matters more than latency. The cold tier makes explicit something that is true of all the designs in this chapter: latency is a function of where the data lives, and the right design depends on how often each part of the data is touched. A search service whose traffic is dominated by a few popular indexes can keep those in memory and the rest on disk or in object storage, paying memory prices only for what is hot. Choosing a tier Table 3 summarizes where each major index design keeps its data and the latency regime and update behaviour that follows. The latency regimes are qualitative and assume a single-shard search at high recall on contemporary hardware; actual figures depend heavily on data, configuration, and recall target. Table 3. Index designs by placement in the memory hierarchy. Design In fast memory In slower storage Latency regime Updates HNSW, full precision Vectors and graph Nothing Sub-millisecond Inserts easy; deletes awkward HNSW with int8 or binary Compressed vectors and graph Full vectors, for re-ranking Sub-millisecond to low milliseconds As above IVF-PQ Centroids and codes Optional full vectors Sub-millisecond to low milliseconds Simple; periodic retrain DiskANN PQ codes, hot-node cache Graph and full vectors on SSD Low milliseconds Supported via merge (FreshDiskANN) SPANN Centroids and centroid graph Posting lists on SSD Low milliseconds Supported via local splits (SPFresh) GPU IVF-PQ or CAGRA Codes or graph in GPU memory Host memory or none Best in batches Mostly rebuild Object-store tier Cache of hot partitions Index files in object storage Milliseconds warm, far slower cold Append and compact The pattern in the table is the one this book has been building toward. Each design moves the boundary between what is kept close and what is kept far, and pays for that movement in either recall, latency, or update complexity. None dominates. The right choice depends on collection size, query rate, latency requirement, update rate, and budget, which is the subject of Chapter 10. Before that, there is a complication that every real system must face and that pure nearest-neighbour benchmarks ignore. Users rarely ask for "the nearest vectors". They ask for the nearest vectors that are in stock, in their language, written after last March, and visible to their account. Chapter 7. The Real Query: Filters, Hybrid Retrieval, and Re-ranking Benchmarks ask a clean question: given a query vector, return the k nearest stored vectors. Production systems almost never ask it. A shopper wants similar products that are in stock, under a price, and shippable to their country. An enterprise assistant must retrieve passages only from documents the user is permitted to read. A news search wants semantically relevant articles from the past week. A code search wants functions in a particular repository and language. Each of these is a nearest-neighbour query constrained by a predicate on metadata, and the constraint interacts with the index in ways that can quietly destroy recall. Beyond filters, real retrieval combines signals. Dense embeddings capture meaning but are weak at exact terms, rare identifiers, and names. Keyword search is the reverse. The best systems run both and fuse the results, then often apply a slower, more accurate model to re-rank a short list. This chapter covers how these pieces are built and how they interact with the indexes described so far. Why filters are hard The difficulty of filtered search depends on selectivity: the fraction of the collection that satisfies the predicate. Consider the two extremes. If the filter is very permissive, say it excludes only one per cent of vectors, the obvious approach works. Search the index normally, retrieve somewhat more than k results, and discard the ones that fail the predicate. This is called post-filtering. If ninety-nine per cent of vectors pass, retrieving k plus a small margin yields k valid results almost always. If the filter is very restrictive, say it matches only a thousand vectors out of a hundred million, the obvious approach also works, but it is a different one. Find the thousand matching vectors using an ordinary metadata index, then compute exact distances to all of them by brute force. This is called pre-filtering. A brute-force scan of a thousand vectors costs microseconds, and it is exact. The hard cases lie in between. Suppose the filter matches one per cent of a hundred million vectors: a million candidates. Brute force over a million vectors is slow, too slow for a sub-millisecond target. Post-filtering is also a problem: if one per cent of vectors pass, retrieving k valid results requires retrieving roughly a hundred times k from the unfiltered index, and the index's recall at such large result sizes, and its cost, become unpredictable. Worse, if the filter is correlated with the query, for instance if the user searches for winter coats but filters for products in a summer collection, the nearest neighbours to the query may almost all fail the filter, and the passing vectors may lie far away in the graph. Filtering inside graph traversal The natural compromise for graph indexes is to apply the filter during traversal: walk the graph as usual, but only admit vectors that pass the predicate into the result set. Non-passing nodes can still be traversed, so that they serve as stepping stones, but they never appear in the results. This works well at moderate selectivity, and it is the default in many systems. At low selectivity it degrades in two ways. First, the search must traverse many non-passing nodes for each passing one it finds, so its cost rises roughly in inverse proportion to selectivity. Second, and more seriously, if the implementation refuses to traverse non-passing nodes at all, which some do to save work, the graph restricted to passing nodes may fall apart into disconnected fragments. A search that starts in one fragment can never reach a passing vector in another, and recall collapses without any error being reported. Several techniques address this. The first is a query planner that chooses a strategy by estimated selectivity. Before searching, the system estimates how many vectors pass the filter, using statistics from its metadata indexes. If the count is small, it uses pre-filtering with brute force; if large, it uses in-traversal filtering; in between, it may use a hybrid. Lucene does something of this kind: when the number of documents matching the filter is small enough that exact search would visit fewer vectors than the approximate search is likely to, it switches to exact search. Many vector databases implement similar thresholds, sometimes exposed as configuration. The second is to make the graph robust to filtering. ACORN, published by Liana Patel and colleagues at SIGMOD in 2024, builds a denser HNSW-like graph than usual, with more edges per node and with the standard pruning relaxed, and then, during filtered search, expands not just a node's direct neighbours but its neighbours' neighbours, keeping only those that pass the predicate. The effect is to approximate, on the fly, a well-connected graph over just the passing vectors, for any predicate, without knowing the predicates in advance. The cost is a larger graph and more work per hop. ACORN has been adopted in several systems, including Weaviate, which added it as a filtering strategy in 2024. The third is to build the filter into the graph when the set of possible predicates is known. Filtered-DiskANN, from Microsoft at the Web Conference in 2023, handles queries whose filters are labels, such as categories or tags, by building a graph in which the pruning rule ensures connectivity within each label's subgraph. Each label gets its own entry point, and the search for a label-filtered query starts there and traverses only nodes carrying that label, with the construction guaranteeing that this subgraph is navigable. Qdrant takes a related approach for payload fields that are indexed: it adds extra edges to the graph within groups of points sharing a field value, so that filtered traversals stay connected. The fourth, and the most robust when it applies, is physical partitioning. If a filter is used on almost every query and has a manageable number of values, such as a tenant identifier in a multi-tenant service, the simplest solution is to build a separate index per value. A query for one tenant searches only that tenant's index. No filtering is needed during traversal, and recall is unaffected. The costs are many small indexes, each with fixed overhead, and the inability to search across partitions efficiently. Multi-tenant vector services frequently use this approach for their tenancy boundary, with in-traversal filtering for other attributes. Table 4 summarizes the main strategies. Table 4. Strategies for filtered vector search. Strategy Works best at Recall behaviour Main cost Example Post-filtering High selectivity (most items pass) Falls as filter tightens Oversampling, unpredictable Simple vector stores Pre-filtering with brute force Very low selectivity (few pass) Exact Linear in matching items Query planners at low match counts In-traversal filtering Moderate selectivity Degrades if graph fragments More hops per result Default in many graph indexes Filter-robust graph Wide range, arbitrary predicates Stable across selectivity Larger graph, two-hop expansion ACORN Label-aware graph Known categorical labels Stable per label Build complexity Filtered-DiskANN Partition per value Ubiquitous low-cardinality filter Unaffected Many small indexes Per-tenant indexes IVF indexes handle filtering somewhat differently. Because a query scans whole lists, the filter can be applied during the scan, with no connectivity problem. The difficulty becomes the number of lists that must be probed to find k passing vectors, which again rises as selectivity falls. Some systems store per-list summaries of metadata so that lists containing no passing vectors can be skipped entirely. Range filters, such as date windows or price bands, pose a particular challenge because they are not a fixed set of labels. Research on range-filtered nearest-neighbour search has produced specialized structures, typically organizing vectors by the filtered attribute in a tree and building graphs at each tree level, but in production the most common approach remains a selectivity-aware planner that chooses between brute force and in-traversal filtering. Hybrid retrieval: combining dense and sparse The second departure from pure nearest-neighbour search is the combination of semantic and lexical signals. Chapter 1 described the strengths and weaknesses of each. Dense retrieval handles paraphrase and conceptual similarity; lexical retrieval handles exact terms, identifiers, rare words, and names. A query for an error code, a product SKU, or an uncommon surname is often best served by keyword matching, while a query phrased in natural language benefits from embeddings. Most real query streams contain both kinds. The established lexical ranking function is BM25, developed by Stephen Robertson, Karen Spärck Jones, and colleagues in the probabilistic retrieval framework during the 1980s and 1990s. BM25 scores a document by summing, over query terms, a weight that grows with the term's frequency in the document, saturating so that repeated mentions have diminishing effect, and that grows with the term's rarity across the collection, normalized by document length. It is served from an inverted index mapping terms to the documents that contain them, with mature algorithms such as WAND and block-max WAND for skipping documents that cannot make the top k. A hybrid system runs both a dense vector search and a BM25 search, each returning a ranked list, and combines them. The combination step is called fusion, and two families of method dominate. Score-based fusion normalizes each system's scores to a common range, for instance by min-max scaling within the result list, and takes a weighted sum. It is intuitive but fragile. BM25 scores and cosine similarities have different distributions that vary by query, so a fixed weight that works for one query type may be wrong for another, and min-max normalization is sensitive to outliers in the list. Rank-based fusion ignores scores and uses only positions. Reciprocal rank fusion, introduced by Gordon Cormack, Charles Clarke, and Stefan Büttcher at SIGIR in 2009, assigns each document a score equal to the sum, over the ranked lists in which it appears, of one divided by a constant plus its rank. The constant, sixty in the original paper, damps the influence of the top positions so that a document appearing reasonably high in both lists can outrank one at the top of only one. Reciprocal rank fusion requires no tuning and no score calibration, and it has become the default fusion method in several search engines and vector databases. Its weakness is that it discards information: a document that one system scores as an overwhelming match and another ignores is treated the same as a marginal one. Neither fusion method is universally superior. The right weighting depends on the queries, and where an evaluation set exists, tuning a weighted combination can outperform reciprocal rank fusion. Where none exists, reciprocal rank fusion is a robust default. Learned sparse representations Between classical keyword search and dense embeddings lies a third option: learned sparse retrieval. Models in this family, of which SPLADE, introduced by Thibault Formal and colleagues in 2021, is the best known, use a transformer to produce a sparse vector over the vocabulary for each document and query. Unlike BM25, the weights are learned, and the model can activate terms that do not appear in the text, expanding "car" to include "automobile" and "vehicle". The resulting vectors are sparse enough to serve from an inverted index using the same machinery as BM25, which makes them efficient and interpretable, while capturing some of the semantic generalization of dense models. From the index's point of view, learned sparse vectors pose a distinct nearest-neighbour problem: maximum inner product search over very high-dimensional, very sparse vectors. Inverted-index algorithms handle it, and specialized approximate methods for sparse inner product search have been developed; the 2023 Big-ANN benchmark included a sparse track for this reason. Several vector databases now accept sparse vectors alongside dense ones in the same collection, so that hybrid search can be expressed as a single query. Re-ranking Any first-stage retriever, dense, sparse, or hybrid, is a compromise between quality and speed. Its job is to produce a candidate list that contains the relevant items, not to order them perfectly. A second stage then re-ranks the candidates with a more expensive model. The most common re-ranker is a cross-encoder: a transformer that reads the query and a candidate passage together and outputs a relevance score. Because it attends jointly over both texts, it can detect negation, check whether the passage actually answers the question, and weigh specific details, all things a single-vector embedding cannot do. Cross-encoders are far too slow to apply to a whole collection, since each query-passage pair requires a full model forward pass, but applying one to fifty or a hundred candidates is feasible within interactive latency budgets on suitable hardware. Late-interaction models such as ColBERT, introduced by Omar Khattab and Matei Zaharia at SIGIR in 2020, sit between single-vector retrieval and cross-encoders. ColBERT keeps one embedding per token for every document, and scores a query against a document by matching each query token embedding to its most similar document token embedding and summing these maxima. This captures fine-grained matching while still allowing documents to be encoded in advance. The cost is storage: a passage of a hundred tokens produces a hundred vectors rather than one, so a collection of a hundred million passages becomes ten billion vectors. Making late interaction practical at scale required its own index engineering. PLAID, a 2022 follow-up, clusters token embeddings, stores compressed residuals, and prunes candidates by centroid scores before full scoring, an application of the IVF and quantization ideas from Chapters 3 and 4 to a new problem. The existence of re-ranking changes how first-stage recall should be judged. If a cross-encoder will re-rank the top hundred candidates, what matters is whether the relevant items appear anywhere in that hundred, not their exact order. Index recall at a hundred becomes the relevant metric, which is usually much easier to achieve at a given cost than recall at ten. Teams that tune their vector index for recall at ten while feeding a hundred candidates to a re-ranker may be paying for precision they discard. Diversity, grouping, and thresholds Three further departures from pure nearest-neighbour search come up often enough to mention. Diversity. The nearest neighbours of a query are often near-duplicates of one another: five chunks from the same document, or ten nearly identical product listings. Returning all of them wastes the result list. Maximal marginal relevance, introduced by Jaime Carbonell and Jade Goldstein in 1998, re-ranks candidates by balancing relevance to the query against similarity to results already selected, and it remains the standard technique. It operates on a candidate list after retrieval, which means the first stage must retrieve more candidates than will finally be shown. Grouping. In retrieval-augmented generation, documents are typically split into chunks, each embedded separately. A query may match many chunks of the same document. Applications often want the top documents rather than the top chunks, which requires grouping results by document and taking the best chunk score per group. Like diversity, this requires retrieving more than k chunks to produce k documents, and the required oversampling depends on how clustered the matches are. Thresholds. Some applications want all items within a similarity threshold rather than a fixed k: every near-duplicate of an image, or every passage similar enough to be considered a match. This is range search. Graph and IVF indexes can support it by continuing the search until candidates exceed the threshold, but their recall for range queries is less well characterized than for k-nearest-neighbour queries, and compressed distances near the threshold are unreliable. Re-ranking with exact distances before applying the threshold is advisable. Each of these features adds a requirement on the first stage: retrieve enough candidates, of sufficient quality, for the downstream logic to work. The common lesson is that the parameters of the vector index cannot be chosen in isolation. They depend on what happens to the results afterwards. The next chapter turns from the query to the system that answers it: how to spread an index across many machines, keep it current as data changes, and keep its slowest queries within bounds. Chapter 8. Systems at Billion Scale: Sharding, Freshness, and Tail Latency An index is an algorithm and a data structure. A vector search system is an index plus everything needed to keep it serving: distribution across machines, replication for availability, an ingestion pipeline, a strategy for updates and deletions, a plan for changing the embedding model, monitoring, and capacity management. At small scale most of these concerns are trivial. At billion scale, they dominate engineering effort, and decisions made at the system level can matter more for latency and recall than the choice between HNSW and IVF. Sharding When a collection outgrows one machine, it must be divided into shards, each held by a different server. There are two basic ways to assign vectors to shards, and the choice shapes the whole system. The first is random or hash-based sharding: assign each vector to a shard by hashing its identifier. Each shard holds a statistically representative sample of the collection and builds its own independent index. A query is sent to every shard, each returns its local top k, and a coordinator merges the shard results into a global top k. This pattern is called scatter-gather. Random sharding has important virtues. Load is naturally balanced, since every shard has a similar share of data and receives every query. Adding capacity is simple. And the merge is exact in a useful sense: if each shard returns its true local top k, the global top k is guaranteed to be among the union, so the merge introduces no recall loss beyond that of the shard indexes themselves. The cost is that every query touches every shard. With a hundred shards, every query becomes a hundred sub-queries, and total work per query grows with the number of shards. Each shard's index is smaller than a single global index would be, so each sub-query is somewhat cheaper, but graph search cost grows only slowly with collection size, so a hundred small graphs cost much more in aggregate than one big one would. The second is content-based sharding: assign vectors to shards by their position in the embedding space, typically using a coarse k-means partition, so that each shard holds a set of nearby clusters. A query is routed only to the shards whose regions are nearest to it, much as an IVF index probes only the nearest lists. This is the IVF idea applied at the level of machines. Content-based sharding reduces work per query, because most shards are not consulted. Its costs are those of IVF, magnified. Queries near shard boundaries need several shards, and recall depends on how many are probed. Load is uneven: shards holding popular regions of the space receive more queries and may also hold more data, so hot shards appear and must be split or replicated more heavily. Rebalancing as data grows requires moving clusters between machines. And if the data distribution shifts, the partition becomes stale. Large systems often combine the two. A coarse partition routes queries to a subset of shard groups, and within each group data is distributed randomly. Or a system may use random sharding for its primary path and maintain content-based routing as an optimization that is bypassed when a query lies near many boundaries. The choice depends on query rate, collection size, and how much variance in latency can be tolerated. One subtlety of scatter-gather with approximate shards deserves mention. Each shard runs an approximate search, so each may miss some of its local true neighbours. If each shard independently has recall of ninety-five per cent at k, the global result also has about ninety-five per cent recall, since the misses are spread across shards. But if the coordinator asks each shard for exactly k results and some shards hold none of the global top k, their returned results are wasted, while shards holding several of the global top k needed to be more accurate than average. Asking each shard for somewhat fewer than k results, when shards are numerous and data is random, reduces work without changing recall much, because the probability that any one shard holds many of the global top k is small. Conversely, with content-based sharding, the nearest shard may hold almost all of the global top k and must be searched with high recall. The tail at scale Scatter-gather has a well-known statistical problem, described by Jeffrey Dean and Luiz André Barroso in a 2013 article in Communications of the ACM titled "The Tail at Scale". If a query must wait for responses from a hundred shards, its latency is the latency of the slowest of the hundred. Even if each shard answers within one millisecond ninety-nine per cent of the time, the probability that all hundred do so is only about thirty-seven per cent. The other sixty-three per cent of queries are delayed by at least one slow shard. Occasional slowness from garbage collection, background compaction, a noisy neighbour on shared hardware, or a network retransmission becomes the common case at the system level. This is why the phrase "sub-millisecond queries across billions of embeddings" needs careful reading. A single shard can answer in a fraction of a millisecond. The system as a whole, after network round trips, fan-out, and merge, typically responds in a few milliseconds at the median and more at the high percentiles. Getting close to the per-shard latency at the system level requires deliberate engineering against the tail. The techniques are well established in distributed systems generally. Hedged requests send a duplicate sub-query to a second replica of a shard if the first has not responded within a short delay, typically set near the ninety-fifth percentile latency, and use whichever response arrives first; this adds a few per cent of load and removes much of the tail. Tied requests send to two replicas at once and cancel the slower. Partial results allow the coordinator to return after a deadline with results from the shards that have answered, accepting a small and measurable recall loss for the slowest queries rather than an unbounded delay. Isolation of background work, such as running index builds and compactions on separate machines or in strictly bounded time slices, removes a major source of variance. Index design contributes too. Graph search time varies by query, because some queries land in hard regions of the graph and need more hops. IVF search time varies with the length of the probed lists. Early-termination techniques, which stop a search once the result set has stabilized rather than after a fixed amount of work, reduce average cost but can increase variance. For a tail-sensitive service, an index whose cost per query is predictable may be preferable to one that is faster on average. Replication Shards are replicated for two reasons: availability, so that the loss of a server does not lose data or block queries, and throughput, so that query load can be spread across replicas. The number of replicas is usually set by whichever requirement is larger. For a high-traffic service, throughput dominates, and a shard may have many replicas, each a full copy of the index. Because each replica is a full copy, replication multiplies memory cost. A billion-vector collection at three terabytes in memory, replicated three times, needs nine terabytes. This is a strong incentive for compression and for disk-based indexes: an index that fits in a tenth of the memory reduces the cost of every replica. Replicas need not be identical. A pattern seen in large deployments uses different index configurations for different purposes: a highly compressed index serving the bulk of traffic at moderate recall, and a less compressed one, or one with full-precision re-ranking data, serving requests that need higher recall. Separating storage from compute, so that replicas load index files from a shared store rather than each building their own, simplifies this and makes it fast to add replicas when load rises. Milvus, among others, uses such an architecture, with stateless query nodes loading index segments from object storage. Ingestion and the cost of embedding It is easy to focus on the index and forget that every vector must first be computed. Embedding a document requires a forward pass through a neural network. For a large transformer model, this is expensive, often far more expensive per item than inserting the resulting vector into an index. Embedding a billion passages is a substantial GPU job, measured in thousands of GPU-hours for typical models, and the exact cost depends on model size and passage length. This has three consequences for system design. First, ingestion throughput is frequently limited by embedding, not indexing, so the ingestion pipeline needs its own capacity planning and its own accelerators. Second, re-embedding the whole collection, which is required whenever the embedding model changes, is a major operation, not a routine one. Third, it is worth caching embeddings and storing them durably, separately from the index, so that indexes can be rebuilt without recomputing embeddings. The pipeline itself typically has stages: change capture from the source of truth, chunking of documents into passages, embedding, writing vectors and metadata to durable storage, and indexing. Decoupling these stages with queues allows each to scale independently and absorbs bursts. It also introduces latency between a change in the source and its visibility in search, which brings us to freshness. Freshness and updates A search system over changing data must make new items searchable promptly, make deleted items disappear, and reflect modifications. The index structures from earlier chapters handle these operations unevenly. IVF handles insertions and deletions cheaply. HNSW handles insertions at moderate cost and deletions poorly. Disk-based graphs need batched merges. Learned components, such as IVF centroids and PQ codebooks, become stale as the data distribution changes. The dominant architectural answer, used in some form by most vector databases, borrows from log-structured merge trees in key-value stores. New vectors are written first to a small mutable buffer, which is searched by brute force or with a small, cheaply updated index. When the buffer fills, it is sealed into an immutable segment and an index is built for it in the background. Queries search all segments and the buffer, and merge results. Over time, small segments are compacted into larger ones, rebuilding their indexes; compaction is also when deleted vectors are physically removed and when learned components can be retrained. Deletions in this architecture are recorded as tombstones, typically a bitset per segment marking deleted positions, and applied as a filter during search. This is a form of in-traversal filtering, and it inherits the problems described in Chapter 7: if a large fraction of a segment is deleted, searches waste effort on dead vectors and graph connectivity among live ones may suffer. Compaction policies therefore consider the deleted fraction as well as segment size, compacting segments where deletions exceed a threshold. Updates to an existing vector are implemented as a delete followed by an insert. Updates to metadata only, such as a price change, need not touch the vector index at all if metadata is stored separately, but they do affect filtered search, and the metadata index must be updated consistently with the vectors. The segment architecture introduces a trade-off between freshness and query efficiency. Small, frequent segments make new data visible quickly but multiply the number of indexes each query must search. Large, infrequent segments are efficient to query but delay visibility. The mutable buffer, searched by brute force, bounds the delay, but brute force over a large buffer is slow. Systems tune buffer size, segment size, and compaction frequency together, and the right settings depend on ingestion rate and freshness requirements. Consistency guarantees vary. Many vector databases offer eventual consistency by default: a write is acknowledged once durable but becomes visible to searches after a short delay. Some offer stronger options, such as read-your-writes within a session or bounded staleness, at some cost in latency. Applications should establish what guarantee they need. A recommendation feed can tolerate seconds of staleness; a compliance system that deletes data on request may need confirmation that the data is no longer retrievable, which requires understanding when tombstones take effect across all replicas and caches. Changing the embedding model Embedding models improve quickly, and teams regularly want to upgrade. An upgrade is more disruptive than it appears, because vectors from different models are not comparable. Each model defines its own space, and a query embedded with a new model cannot be meaningfully compared with documents embedded with an old one. The distances are simply meaningless. An upgrade therefore requires re-embedding the entire collection with the new model, building a new index, and switching queries to the new model at the same moment the new index goes live. During the transition, both indexes must exist. For a billion-vector collection, this means temporarily doubling storage and running a large embedding job, which may take days. A careful upgrade runs the new index in shadow mode, receiving copies of production queries without serving results, so that its latency and quality can be compared before the switch. Rollback requires keeping the old index until the new one is proven. Some research has explored aligning embedding spaces across model versions, learning a transformation that maps old vectors into the new space, or training new models to be backward compatible with old ones so that old document vectors can be queried with new query vectors. These approaches can reduce the cost of upgrades but typically sacrifice some of the new model's quality, and full re-embedding remains the standard practice. The practical implication is that the embedding model is part of the schema. Every stored vector should be tagged with the model and version that produced it, and the system should refuse to compare vectors from different versions. Mixing them, which can happen silently if an ingestion pipeline is upgraded before the index is rebuilt, produces degraded results with no errors. Observability The final system concern is knowing whether the system is working. Latency, throughput, error rates, and resource utilization are monitored as for any service. What is specific to vector search is that its most important failure mode, loss of recall, produces no errors at all. The standard technique for monitoring recall in production is to sample a small fraction of live queries and run them, asynchronously and off the critical path, against an exact brute-force search over the same data, or over a sample of it. Comparing the production results with the exact results gives an ongoing estimate of recall. Tracking it over time reveals degradation from accumulating deletions, data drift, configuration changes, or software regressions. Brute force over a billion vectors is expensive even for a small sample of queries, so it is often run on GPUs or against a random subsample of the collection, with recall estimated statistically. Several further signals are worth tracking. The distribution of distances to the top result, by query, shifts when the data or query distribution changes, and a drop may indicate that queries are increasingly out of distribution. The fraction of filtered queries that return fewer than k results indicates filter-selectivity problems. The ratio of deleted to live vectors per segment predicts recall loss from tombstones. List-length or node-degree distributions reveal imbalance. None of this is exotic. It is the vector-search version of the observability that any serious data system requires. But it is frequently missing, because recall does not appear in the default dashboards, and its absence is why many teams learn about recall problems from users rather than from monitoring. The next chapter takes up measurement more broadly: how to evaluate an index before deploying it, how to read benchmarks, and how to tune a running system. Chapter 9. Measuring What Matters: Evaluation and Tuning Every chapter so far has described a bet and the parameters that control it. None of the parameters has a universally correct value. The right nprobe, ef, M, compression level, and candidate list size depend on the data, the queries, the hardware, and what the application needs. They can only be found by measurement. This chapter describes how to measure honestly, how to read the measurements others publish, and how to tune a system once it is running. Building an evaluation set An evaluation of a vector index needs three things: a collection, a set of queries, and the exact nearest neighbours of each query in the collection, called the ground truth. The collection should be the real one, or a representative sample at a realistic scale. Index behaviour changes with size. A graph index tuned on a million vectors will usually need a larger ef to reach the same recall on a hundred million, because paths are longer and the neighbourhood of each query is denser. An IVF index tuned at one size needs different nlist and nprobe at another. Where the full collection is too large to test conveniently, a sample of at least a tenth of the target size gives a more reliable picture than a tiny one, and testing at two or three sizes reveals how parameters must scale. The queries should be real queries from the application, embedded with the production model and the production query-side instructions. This is the most important rule in this chapter and the one most often broken. The common shortcut is to hold out a sample of documents from the collection and use them as queries. Held-out documents are drawn from the same distribution as the indexed data, so they land in well-populated regions of the space, close to many indexed points. Real queries may not. As Chapter 1 explained, queries in asymmetric retrieval tasks are often short and structurally different from documents, and in cross-modal tasks they come from a different region of the space entirely. Indexes evaluated on held-out documents routinely report recall several points higher than they achieve on real queries at the same settings. Where real queries are not yet available because the application is new, synthetic queries generated in the style expected, for instance by asking a language model to write questions that each passage answers, are a better proxy than held-out documents. The ground truth is computed by brute force. For each query, compute the exact distance to every vector in the collection using the production distance function and full-precision vectors, and record the top k for the largest k of interest. For a large collection this is expensive, but it is a one-time cost per evaluation set, it parallelizes trivially, and GPUs make it fast. The ground truth must be recomputed whenever the collection or the embedding model changes. A few thousand queries are usually enough to estimate average recall precisely. Ten thousand is a common benchmark size. More queries help mainly when one wants to study the distribution of per-query recall, discussed below. What to measure For each configuration of the index, five quantities should be recorded. Recall at the k the application actually uses, and if results are re-ranked downstream, recall at the candidate-list size as well. As Chapter 7 noted, recall at a hundred is the right measure for a first stage feeding a re-ranker; recall at ten is the right measure if the ten results are shown directly. Latency, as a distribution rather than an average. The median, the ninety-ninth percentile, and the maximum tell very different stories. A graph index might have a median of three hundred microseconds and a ninety-ninth percentile of two milliseconds because a minority of queries land in hard regions. For a service with a latency objective at the ninety-ninth percentile, the median is nearly irrelevant. Throughput, measured under load. Latency measured with one query at a time on an idle machine is the best case; under concurrent load, queries contend for memory bandwidth, caches, and cores, and latency rises. The useful measurement is latency at a target throughput, or the maximum throughput achievable while keeping the latency percentile within its objective. Load testing should use an open-loop generator, which issues queries at a fixed rate regardless of whether previous ones have completed. A closed-loop generator, which waits for each response before sending the next query, slows down when the system slows down and so systematically understates tail latency, a measurement error known as coordinated omission. Memory, measured as resident memory of the serving process rather than as the size of the vectors. Graph edges, identifier maps, metadata, allocator overhead, and buffers can add substantially to the raw vector size. Build time and update cost: how long it takes to build the index from scratch, how long inserts take, and how recall and latency behave after a realistic volume of inserts and deletes. An index that is excellent when freshly built and degrades after a week of updates should be evaluated after the week. Per-query recall and hard queries Average recall hides a great deal. An index with ninety-five per cent average recall at ten could achieve that with every query getting nine or ten of ten right, or with most queries getting ten of ten and a minority getting five. These are very different systems from a user's point of view. The second will feel unreliable: most searches work and some fail badly. Looking at the distribution of per-query recall reveals which queries are hard, and research has given useful vocabulary for why. Junfeng He, Sanjiv Kumar, and Shih-Fu Chang proposed in 2012 a measure called relative contrast: the ratio of a query's mean distance to the collection to its distance to its nearest neighbour. Queries with low relative contrast, whose nearest neighbours are barely nearer than everything else, are hard for every approximate method. Local intrinsic dimensionality, an estimate of the dimension of the data in the neighbourhood of a query, is another predictor: queries in regions of high local intrinsic dimension are harder. Work by Martin Aumüller and Matteo Ceccarello, among others, has used such measures to construct harder benchmark query sets and to explain why the same index performs differently on different datasets. In practice, hard queries tend to be out-of-distribution queries, queries in sparse regions of the collection, and queries that are ambiguous, sitting between several clusters. Identifying them in an evaluation set lets an engineer ask the right question: whether these queries matter to the application. If they are rare and the application tolerates occasional misses, average recall is the right target. If they are the important queries, such as rare-item lookups in a catalogue where common items are easy to find by other means, the index should be tuned for them specifically, possibly with adaptive search that spends more effort on queries detected as hard. Adaptive early termination is an active research topic. Rather than using a fixed ef or nprobe for every query, the search monitors its own progress and stops when further exploration is unlikely to change the results, or continues when the results are still changing. Learned approaches train a small model to predict, from features gathered during the search, how much more work is needed. These methods can reduce average cost at a fixed recall, but they make cost per query less predictable, which interacts with the tail-latency concerns of Chapter 8. Reading public benchmarks Public benchmarks are valuable and are frequently misread. The most widely cited, ANN-Benchmarks, runs a large number of algorithms on a set of standard datasets on a single machine and publishes recall-throughput curves. Its methodology is careful and its results are reproducible. Its limitations are those of any fixed benchmark. Several of its datasets are old and small. SIFT and GIST, image descriptors from the 2000s, have 128 and 960 dimensions and a million vectors each. GloVe word vectors are a few hundred dimensions and around a million vectors. These are different in character from modern transformer embeddings, which are typically higher-dimensional, more anisotropic, and much more numerous. Newer datasets built from modern text embeddings have been added to benchmarks over time, and results on them should be preferred where available. The default ANN-Benchmarks configuration measures single-threaded throughput with queries issued one at a time. This favours algorithms with low per-query latency and does not reflect behaviour under concurrency or batching. An algorithm that wins under these conditions may lose on a multi-core server under load, or on a GPU with large batches. Benchmarks measure what they measure and nothing else. Most report search performance on a static index. They say little about build time, update behaviour, memory under realistic conditions, filtered search, or behaviour at a hundred times the dataset size. The Big-ANN benchmarks address several of these, with billion-scale datasets, hardware-constrained tracks, and dedicated filtered, streaming, out-of-distribution, and sparse tracks. Vendor-produced benchmarks, such as the VectorDBBench suite released by Zilliz, test full database systems rather than libraries, including ingestion and filtering, which makes them more realistic for system selection, though readers should remember who produced them and check whether competing systems were configured with equal care. The right use of public benchmarks is to narrow the field and to calibrate expectations. They indicate which families of method are competitive for data of a given character and roughly what recall-throughput trade-offs to expect. They cannot substitute for measurement on one's own data, queries, and hardware. Index recall versus end-to-end quality Index recall measures fidelity to exact search. The application cares about something else: whether users find what they need, whether the assistant answers correctly, whether recommendations are engaging. The relationship between the two is the most practically important measurement of all, and it is often not made. Measuring end-to-end quality requires relevance judgements: for a set of queries, which items are actually relevant. These come from human annotation, from user behaviour such as clicks and purchases, or, increasingly, from language models acting as judges, which are cheaper but must themselves be validated against human judgement on a sample. Standard retrieval metrics then summarize quality: recall of relevant items in the top k, precision, mean reciprocal rank, and normalized discounted cumulative gain, which rewards placing highly relevant items near the top. For retrieval-augmented generation, the final measure is answer quality, assessed on a question set with reference answers. With relevance judgements in hand, the key experiment is a sensitivity analysis. Run the end-to-end evaluation with exact brute-force retrieval, then with the index at several recall levels, and observe how end-to-end quality changes. Typically the curve is flat at high index recall and falls off below some point. That point is the recall the application needs. Tuning the index for recall above it buys nothing measurable; tuning it below costs quality. The results are often surprising in both directions. Some applications show no measurable quality loss at index recall of ninety per cent, because the relevant items are well separated from the irrelevant ones and appear in the approximate results even when some near neighbours are missed, or because a re-ranker and a generous candidate list absorb the loss. Others show sharp quality loss at ninety-eight per cent, because the relevant item is frequently a single specific document that is only marginally closer than many near-duplicates. The same experiment should be run with alternatives to index tuning: a better embedding model, hybrid retrieval, a re-ranker, better chunking. Very often, the largest gains in end-to-end quality come from these, not from the index. An honest accounting allocates engineering effort to whichever lever moves end-to-end quality most per unit cost. A tuning sequence With measurement in place, tuning follows a sensible order. The steps below assume an in-memory graph index with optional compression, the most common configuration, and adapt straightforwardly to others. Fix the target first. Decide the recall the application needs, using the sensitivity analysis above if possible, and the latency percentile and throughput the service must meet. Without a target, tuning has no end. Establish the baseline with exact search. Measure brute-force latency on the full collection. If it already meets the latency target, which is the case more often than people expect for collections up to a few million vectors, especially with batching or a GPU, no approximate index is needed. Choose the build parameters with moderate defaults: M around 16 to 32, efConstruction around 200, adjusted upward for high-dimensional or difficult data. Build the index. Sweep the search parameter. Measure recall and latency over a range of ef values. Find the smallest ef that meets the recall target. If even large ef values cannot reach it, the graph quality is insufficient; rebuild with larger M or efConstruction, and sweep again. Add compression if memory is a constraint. Rebuild with int8 scalar quantization and full-precision re-ranking, sweep again, and check whether the recall target is still met at acceptable latency. If more compression is needed, try binary or product quantization with a larger candidate list for re-ranking. Measure memory as well as recall. Test under load, with filters, and after updates. Repeat the key measurements at the target throughput, with the production filter distribution, and after a realistic volume of inserts and deletes. Adjust parameters for whichever condition is binding. Deploy with monitoring. Set up sampled recall measurement as described in Chapter 8. Revisit the configuration whenever the collection grows substantially, the query distribution changes, or the embedding model is replaced. The sequence is unglamorous. Its value lies in its order: each step is measured against a target, each parameter is set by evidence, and the most expensive changes, rebuilding with different structural parameters, are made only when cheaper ones have been exhausted. The final chapter applies this reasoning to the decisions an engineer actually faces when designing a system: which family of index, on which hardware, at what cost. Chapter 10. Choosing an Architecture: Sizing, Trade-offs, and Worked Cases The preceding chapters described the components. This one assembles them into decisions. It begins with the questions that determine an architecture, in the order they should be asked, then works through four realistic scenarios with explicit arithmetic, and closes with the choice between libraries, database extensions, and dedicated vector databases. The arithmetic throughout uses only the definitions given earlier: four bytes per dimension for float32, one for int8, one bit for binary, m bytes for product quantization with m sub-spaces of 8 bits, and four bytes per graph edge. Real systems add overhead for identifiers, metadata, allocator slack, and buffers, so these figures are lower bounds, and a margin of thirty to fifty per cent is prudent in capacity planning. The questions, in order The first question is how many vectors there are, now and in two years, and of what dimension. This determines raw size: N times d times four bytes in float32. Everything else is constrained by it. Growth matters because rebuilding an architecture is expensive, and an index that fits comfortably today may not in a year. The second question is what recall the application needs, ideally established by the sensitivity analysis in Chapter 9. This determines how much approximation is affordable, and therefore how much compression and how aggressive a partition. The third is the latency objective, at a stated percentile, and the query rate. Together these determine whether the working set must be in memory, whether SSD latency is acceptable, and how many replicas are needed for throughput. The fourth is the update pattern: how many inserts, updates, and deletes per second, and how fresh results must be. This constrains the index family, since some handle updates much better than others, and determines the segment and compaction design. The fifth is the query shape: what filters are used, how selective they are, whether hybrid retrieval and re-ranking are involved, and whether there is a tenancy boundary. This affects index choice, partitioning, and the size of candidate lists. The sixth is the budget, which in the end decides among architectures that all meet the technical requirements. Answering these in order narrows the design rapidly. A collection of a few million vectors with a latency objective of tens of milliseconds and modest traffic can be served almost any way, and the decision should be driven by operational simplicity. A collection of billions with a millisecond objective and high traffic admits only a few designs, and the decision is driven by cost. Case one: an internal knowledge assistant An organization builds a retrieval-augmented assistant over its internal documents: wikis, policies, tickets, and reports. After chunking, there are five million passages, embedded with a 1,024-dimensional model. Traffic is modest, a few tens of queries per second at peak. The latency objective for retrieval is fifty milliseconds at the ninety-ninth percentile, since the language model call that follows takes seconds. Documents change daily. Every query carries an access-control filter: users may only retrieve passages from documents they are permitted to read. In float32, five million vectors of 1,024 dimensions occupy about 20.5 gigabytes. In int8, about 5.1 gigabytes. An HNSW graph with M of 16 adds about 128 bytes per vector at layer zero, about 0.64 gigabytes. Everything fits in the memory of a single ordinary server, even uncompressed, and an int8 index with full-precision re-ranking would fit on a small one. Brute force deserves a moment's consideration. Scanning 20.5 gigabytes at a few hundred gigabytes per second of memory bandwidth takes on the order of a hundred milliseconds per query on a CPU, too slow for the objective. On a GPU with a few terabytes per second of bandwidth it would take around ten milliseconds, within the objective, but a GPU for a few tens of queries per second is a poor use of hardware. An approximate index is warranted, and an HNSW index will answer in well under a millisecond at high recall. The hard part of this case is not the index. It is the access-control filter. Permissions in an organization are uneven: some users can see almost everything, others a narrow slice. For a user who can see one per cent of documents, in-traversal filtering on a standard HNSW graph risks the fragmentation described in Chapter 7. The appropriate design is a query planner that estimates the size of the user's permitted set and switches to brute force over that set when it is small enough, perhaps below tens of thousands of passages, where brute force over int8 vectors takes a few milliseconds, and uses in-traversal filtering otherwise. If the system supports a filter-robust graph method, that reduces the risk further. Permissions also change, and the filter must reflect the current state, which argues for storing permission metadata separately from the vector index and evaluating it at query time rather than baking it into the index. Given the scale, the operationally simplest option is likely the best: a vector extension to an existing database, such as pgvector in PostgreSQL, or vector fields in an existing search engine such as Elasticsearch or OpenSearch, which also provides BM25 for hybrid retrieval. Either keeps vectors, metadata, and permissions in one transactional system. A dedicated vector database is not needed at this size, though it would also work. The engineering effort in this case should go into chunking, the embedding model, hybrid retrieval, re-ranking, and evaluation, which together will affect answer quality far more than index parameters. Case two: product search with filters A retailer serves semantic search and visual similarity across two hundred million product listings, each embedded with a 768-dimensional model. Peak traffic is two thousand queries per second, the latency objective for retrieval is twenty milliseconds at the ninety-ninth percentile, and nearly every query carries filters: category, price band, availability, and shipping region. The catalogue changes constantly, with prices and availability updating many times per second and listings added and removed throughout the day. Raw size in float32 is about 614 gigabytes. In int8, about 154 gigabytes. With M of 32 to cope with high-dimensional, clustered product data, layer zero has up to 64 edges per node, about 256 bytes per vector, or about 51 gigabytes for the graph. An int8 HNSW index therefore needs a little over 200 gigabytes in memory, plus full-precision vectors for re-ranking, which can be held on local SSD and read only for candidates. A single large server could hold this, but it could not serve two thousand queries per second with headroom, and it would be a single point of failure. A reasonable design divides the collection into four to eight shards by random assignment, each holding twenty-five to fifty million vectors and roughly 25 to 50 gigabytes of index, and replicates each shard three or more times for throughput and availability. Random sharding is preferred here over content-based sharding because filters make content routing unreliable: the nearest shard by vector may hold none of the products in the requested category. Filters drive the rest of the design. Category is a natural candidate for the label-aware or partitioned approaches of Chapter 7, since there are a manageable number of top-level categories and most queries use one. Price and availability change too often to bake into the index and must be evaluated at query time from a fast metadata store. A query planner chooses between in-traversal filtering and brute force based on estimated selectivity per shard. Updates are split by type. Price and availability changes touch only metadata and need not affect the vector index at all. New and removed listings flow through a segment-based ingestion pipeline with a mutable buffer for freshness and background compaction to remove tombstones, with compaction triggered when deleted fractions grow large. The embedding of new listings, which requires running the image and text models, is a separate pipeline with its own GPU capacity. The tail-latency objective of twenty milliseconds across four to eight shards is achievable with hedged requests and with compaction isolated from serving. A sub-millisecond per-shard search leaves ample room for network, merge, and re-ranking. This is the scale at which a dedicated vector database, or a search engine with mature vector support, earns its keep, because the operational features it provides, sharding, replication, segment management, filtering, and hybrid search, would otherwise have to be built. Case three: web-scale passage retrieval A search engine or large AI provider wants dense retrieval over five billion web passages with 768-dimensional embeddings. Traffic is high, tens of thousands of queries per second. The latency budget for retrieval is around ten milliseconds at the ninety-ninth percentile. The collection is refreshed continuously but tolerates minutes of staleness. Raw size in float32 is about 15.4 terabytes. Holding this in memory, with an HNSW graph adding about 0.64 terabytes at M of 16, and replicated several times for throughput, means tens of terabytes of DRAM. It can be done, and some organizations do it, but the cost is large and every additional replica adds terabytes. Compression changes the picture. Product quantization at 64 bytes per vector reduces the collection to 320 gigabytes. Binary quantization at one bit per dimension gives 96 bytes per vector, 480 gigabytes. Either can be held in memory across a modest number of servers. With either, re-ranking is essential, which requires the full or partially compressed vectors to be accessible somewhere. Two architectures are natural. The first is an IVF index with product quantization in memory, sharded by content using the coarse partition, with full-precision or float16 vectors for re-ranking on local SSD. At a billion vectors per few machines in compressed form, five billion vectors fit in a cluster of tens of servers per replica. Content-based sharding reduces the number of machines each query touches, and the IVF scan is efficient and predictable. Recall depends on nprobe and on the candidate-list size for re-ranking. The second is a DiskANN-style design: compressed codes in memory, perhaps 32 bytes each, totalling 160 gigabytes, and the graph with full-precision vectors on SSD. Each node's on-disk record of about 3,072 bytes of float32 vector plus a neighbour list of 64 edges at four bytes each, 256 bytes, fits in a single four-kilobyte sector, so the on-disk index is about five billion sectors, roughly 20 terabytes across the cluster's SSDs. Storing vectors as float16 would roughly halve the per-node record, though the sector-aligned layout means the saving on disk depends on whether two records can share a sector. Each query costs a few dozen SSD reads in parallel batches and runs in low milliseconds per shard. Both architectures cut memory cost by an order of magnitude or more compared with an in-memory float32 graph, at the price of some latency and engineering complexity. Which is better depends on update rate, SSD capacity and throughput per server, and the recall target, and should be determined by measurement on the actual data. Organizations at this scale frequently run several tiers: a small, hot in-memory index for the most frequently retrieved passages, and a large compressed or disk-based index for the rest. At this scale, GPUs also enter the picture, for building indexes, where GPU-based graph construction and k-means clustering can cut build times from days to hours, and potentially for serving, if query batching can be arranged without breaking the latency objective. Case four: a multi-tenant service A software company offers semantic search as a feature to fifty thousand customer organizations. The total collection is two billion vectors of 1,024 dimensions, but the distribution is extremely skewed: a few hundred large tenants hold most of the data, and tens of thousands hold fewer than a hundred thousand vectors each. Most small tenants query rarely. Strict isolation between tenants is required. Isolation argues strongly for partitioning by tenant: each tenant's data in its own index, so that no query can ever touch another tenant's vectors and no filter selectivity problem arises at the tenancy boundary. But fifty thousand independent HNSW indexes, each kept in memory, would waste enormous amounts of memory on rarely used data. The architecture that has emerged for this pattern is tiered by both size and activity. Small tenants, with fewer than some threshold of vectors, perhaps tens of thousands, need no approximate index at all: brute force over their vectors takes well under a millisecond. Their vectors can be stored compactly and loaded on demand. Medium tenants get approximate indexes built when they cross the size threshold. Large tenants get their own dedicated shards. Indexes of inactive tenants are evicted from memory to SSD or object storage and reloaded on first query, accepting a slower first query after a period of inactivity. This is the cold-tier design described in Chapter 6, and its economics depend entirely on the activity distribution. If a tenth of tenants account for nearly all queries, then memory need only hold their indexes, and the remaining data can live on cheap storage. The engineering challenges are the cost of loading an index on demand, which argues for compact on-disk formats that can be memory-mapped and queried without full deserialization, and the management of many small indexes, which argues for a system designed for multi-tenancy rather than one adapted to it. Summary of the cases Table 5 brings the four cases together. The storage figures are computed from the dimensions and byte sizes stated in the text and exclude system overhead. Table 5. Sizing summary for the four worked cases. Case Vectors and dimension Float32 size Compressed in memory Likely architecture Knowledge assistant 5 million, 1,024 20.5 GB 5.1 GB int8 plus 0.6 GB graph Single node; database extension; planner for filters Product search 200 million, 768 614 GB 154 GB int8 plus 51 GB graph Random shards, replicated; label-aware filters Web-scale passages 5 billion, 768 15.4 TB 160 to 480 GB codes IVF-PQ or DiskANN-style, SSD re-ranking, tiered Multi-tenant service 2 billion, 1,024 8.2 TB Hot tenants only Per-tenant indexes; brute force for small; cold tier The progression in the table shows what this book has argued throughout. As scale increases, the share of the data kept in fast memory shrinks, the reliance on compression and re-ranking grows, and the engineering complexity of the system rises. At small scale the index is nearly irrelevant and everything else matters. At large scale the index design is the system design. Libraries, extensions, and databases The final decision is what to build on. There are three broad categories, and each suits different situations. Libraries provide index algorithms without a system around them. Faiss, from Meta, is the most comprehensive, covering flat, IVF, PQ and its variants, HNSW, and GPU implementations, with an index factory that composes them. hnswlib is the compact reference implementation of HNSW. ScaNN, from Google, offers its anisotropic quantization and partitioning. DiskANN, from Microsoft, offers the Vamana graph and its disk-based search. NVIDIA's cuVS offers GPU-accelerated IVF and CAGRA. Libraries are the right choice for offline jobs, for embedding search inside a larger custom system, and for teams with the expertise to build sharding, persistence, updates, and filtering themselves. They are a poor choice for teams that need a service and lack that expertise. Database extensions add vector indexes to systems that already exist. pgvector adds HNSW and IVF indexes to PostgreSQL. Lucene-based search engines, Elasticsearch and OpenSearch, offer HNSW vector fields alongside their mature full-text search. Several other relational and document databases have added vector types and indexes. The great advantage of extensions is that vectors live alongside the data they describe, with the same transactions, access controls, backups, and operational tooling. Hybrid search is natural in search engines. The disadvantages are that extensions are typically less optimized for vector workloads than dedicated systems, and their scaling characteristics are those of the host system, which may not suit billion-scale vector data. For collections in the millions to low hundreds of millions, extensions are often the best choice. Dedicated vector databases, such as Milvus, Qdrant, Weaviate, Vespa, Pinecone, and others, are built around vector search, with distribution, replication, segment management, filtering, hybrid search, and multi-tenancy designed in. They offer the widest range of index types and the most tuning control, and they scale furthest. Their cost is another system to operate, or another service to pay for, with its own consistency model and its own integration with the source of truth. They are the right choice when vector search is central to the product and its scale or requirements exceed what an extension provides. These categories blur. Dedicated databases are adding full-text search, search engines are improving vector performance, and libraries are gaining persistence features. The useful question is not which category is best but which system meets the requirements established by the six questions at the top of this chapter with the least operational burden. That question, and the measurements needed to answer it, are what an engineer brings to the choice. The specific products will change; the trade-offs they embody will not. Conclusion The argument of this book has been that every scalable vector index is a bet about where the nearest neighbours lie, that there are three basic kinds of bet, partitioning, compression, and navigation, and that the craft consists in choosing among them against the memory hierarchy and measuring how often they lose. Having worked through the pieces, it is possible to say several things that could not have been said at the outset. Recall is a budget, not a virtue The first is that recall should be treated as a budget to be spent, not a virtue to be maximized. Every technique in this book trades recall for something: memory, latency, throughput, update simplicity, or money. The question is never whether an index is accurate enough in the abstract but whether the recall it delivers is the recall the application needs, purchased at the lowest cost. That question has an empirical answer, found by measuring end-to-end quality at several levels of index recall. Most teams never make that measurement. They either accept a default and hope, or chase recall far beyond what their re-ranker and language model can use. Both errors are expensive, one in quality and the other in hardware. A consequence is that the most important number in a vector search system is not its latency or its recall but the recall at which end-to-end quality begins to fall. Knowing that number turns tuning from guesswork into engineering. It also redirects effort. If end-to-end quality is flat across a wide range of index recall, the index is not the bottleneck, and the work belongs in the embedding model, the chunking, the hybrid signals, or the re-ranker. The hierarchy decides The second conclusion is that the memory hierarchy, more than algorithmic ingenuity, determines which designs win. HNSW dominates in memory because random access to DRAM is cheap enough that its small number of distance computations translates into low latency. It loses on SSD because the same random access pattern becomes a sequence of slow reads, which is why DiskANN redesigned the graph to take fewer, larger, parallel steps. IVF survives despite inferior in-memory recall curves because sequential list scans suit caches, SIMD units, GPUs, and SSDs. Product quantization's table lookups work because the tables fit in the fastest caches, and fast-scan PQ works because sixteen-entry tables fit in a register. Binary quantization's revival rests on the population-count instruction. This suggests how to evaluate new techniques as they appear, as they will. Ask where the bytes live and how many of them each query touches at each level. A technique that reduces arithmetic but not bytes read will rarely help. A technique that converts random access into sequential access, or that moves the working set up a level, usually will. Hardware will continue to change. Memory capacity per server grows, high-bandwidth memory spreads beyond accelerators, storage-class memory and memory pooling over new interconnects may blur the line between DRAM and SSD. Each change will move the boundaries between the designs in this book, and the reasoning to follow will be the same. The query is the hard part The third conclusion is that the problems which remain genuinely hard are mostly not about the clean nearest-neighbour query that benchmarks measure. Within that clean problem, progress has been extraordinary: in-memory graph indexes reach high recall with a few hundred distance computations over collections of hundreds of millions, compressed indexes serve billions from modest memory, and disk-based indexes serve billions from a single node in milliseconds. What remains unsettled lies at the edges. Filtered search over arbitrary predicates still lacks a method that is robust across all selectivities without significant overhead, although filter-robust graphs and selectivity-aware planners have narrowed the gap considerably. Out-of-distribution queries, especially across modalities, still degrade indexes built from documents alone, and query-aware construction is young. Updates and deletions in graph indexes still often end in periodic rebuilds. Multi-vector representations, which address real limitations of single-vector retrieval, multiply storage by an order of magnitude or more and demand their own index engineering. The theoretical results on the limits of single-vector embeddings suggest that some retrieval tasks will always need more than one vector per item, or more than one stage, which means that the cost of these richer representations is a problem the field will have to solve rather than avoid. There is also a structural uncertainty about where retrieval is heading. Language models with very long context windows can read more candidate passages directly, which lowers the cost of missing a marginally relevant one and may relax recall requirements for the first stage. Agentic systems that issue many retrieval queries per task, refining as they go, raise the importance of throughput and per-query cost relative to single-query latency. Both trends change the shape of the budget without removing the need for an index. What to do differently For an engineer responsible for a vector search system, the practical consequences can be stated plainly. Measure recall in production, continuously, by comparing sampled live queries against exact search. Its absence is the most common gap in vector search operations, and the failures it would catch, from accumulating deletions, data drift, filter fragmentation, or configuration regressions, produce no errors otherwise. Evaluate with real queries. Held-out documents flatter every index, and the gap is largest in exactly the cases, asymmetric and cross-modal retrieval, where modern systems operate. Treat the embedding model as part of the schema. Tag every vector with the model that produced it, plan upgrades as migrations with shadow evaluation and rollback, and never let vectors from different models share an index. Start simpler than seems necessary. Brute force is exact and fast enough for more collections than people assume. A database extension is sufficient for more workloads than vendors suggest. Add approximation, compression, sharding, and tiers when measurement shows they are needed, and not before. Separate the two kinds of quality. Index recall measures fidelity to the embedding model; relevance measures whether the model was right. Diagnose problems by attributing them to the correct layer, and spend effort where the measured gains are largest. Finally, read the phrase "sub-millisecond search across billions of embeddings" with the precision it deserves. It describes something real: a single in-memory shard answering a well-formed query at a chosen recall. It does not describe a system's ninety-ninth percentile after fan-out, filters, and re-ranking, nor an index that has absorbed a month of deletions without maintenance, nor a query from a modality the index was not built for. The systems that live up to the phrase are the ones whose operators know exactly which bet they made, what it costs, and how often it loses. That knowledge, rather than any particular algorithm, is what this book has tried to provide. Glossary Approximate nearest-neighbour search (ANN). Search that returns items probably among the true nearest neighbours of a query, trading a controlled loss of accuracy for large gains in speed and memory. Asymmetric distance computation (ADC). Computing distances between a full-precision query and compressed database vectors, typically by summing entries from per-query lookup tables. Beam search. A graph search that keeps a bounded set of the best candidates found so far and expands the most promising unexplored one at each step; its width is controlled by ef in HNSW. Brute-force (flat) search. Exact search that computes the distance from the query to every stored vector. Coarse quantizer. The set of centroids, usually learned by k-means, that assigns each vector to a list in an inverted file index. efConstruction / ef (efSearch). HNSW beam widths used during index construction and during search respectively; larger values raise recall at higher cost. Embedding. A fixed-length vector of numbers produced by a learned model so that similar objects map to nearby points. Hubness. The tendency in high-dimensional data for a few points to appear among the nearest neighbours of disproportionately many others. Hybrid retrieval. Combining dense vector search with lexical or learned sparse search, typically through score or rank fusion. Inverted file index (IVF). An index that partitions vectors into lists by nearest centroid and searches only the lists nearest the query. M. The maximum number of neighbours per node in an HNSW layer; layer zero typically allows twice as many. Maximum inner product search (MIPS). Finding stored vectors with the largest dot product with the query, a problem related to but distinct from nearest-neighbour search under a metric. nlist / nprobe. The number of lists in an IVF index and the number of lists probed per query. Product quantization (PQ). Compression that splits a vector into sub-vectors and encodes each by the index of its nearest centroid in a per-sub-space codebook. Recall at k. The fraction of the true k nearest neighbours that an approximate search returns among its k results. Re-ranking. Recomputing scores for a candidate list with a more precise representation or model before selecting final results. Residual. The difference between a vector and its assigned centroid; encoding residuals improves compression accuracy in IVF. Scalar quantization. Compression that maps each coordinate independently to a lower-precision value, such as an 8-bit integer. Selectivity. The fraction of the collection that satisfies a query's metadata filter. Tombstone. A marker recording that a vector has been deleted, applied as a filter until the vector is physically removed during compaction. Further Reading Hervé Jégou, Matthijs Douze, and Cordelia Schmid, "Product Quantization for Nearest Neighbor Search," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011. The paper that introduced product quantization, asymmetric distance computation, and the IVFADC index. Still the clearest account of why compressed-domain search works. Yury Malkov and Dmitry Yashunin, "Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020 (preprint 2016). The original description of HNSW, including the neighbour-selection heuristic that is often omitted from secondary accounts. Suhas Jayaram Subramanya and colleagues, "DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node," NeurIPS 2019. Introduces the Vamana graph and the design that made SSD-resident billion-scale search practical. Qi Chen and colleagues, "SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Search," NeurIPS 2021. The partition-based alternative to DiskANN, with balanced clustering and boundary duplication for SSDs. Jeff Johnson, Matthijs Douze, and Hervé Jégou, "Billion-scale Similarity Search with GPUs," IEEE Transactions on Big Data, 2019 (preprint 2017). The GPU foundations of Faiss: brute force, IVF-PQ, and k-selection on graphics processors. Matthijs Douze and colleagues, "The Faiss Library," 2024 preprint. A survey of the library's design space by its authors, and a compact reference to most of the index types discussed in this book. Ruiqi Guo and colleagues, "Accelerating Large-Scale Inference with Anisotropic Vector Quantization," ICML 2020. The ScaNN paper, showing why quantization loss should be shaped for inner product search. Jianyang Gao and Cheng Long, "RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search," SIGMOD 2024. Binary quantization with random rotation and a calibrated distance estimator. Liana Patel and colleagues, "ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data," SIGMOD 2024. The most influential recent approach to filtered graph search. Martin Aumüller, Erik Bernhardsson, and Alexander Faithfull, "ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms," Information Systems, 2020. The methodology behind the recall-throughput comparisons that the field uses as a common language; the companion website publishes current results. Harsha Vardhan Simhadri and colleagues, "Results of the NeurIPS'21 Challenge on Billion-Scale Approximate Nearest Neighbor Search," PMLR, 2022, and the 2023 follow-up report. The best public evidence on how methods behave at billion scale, with filtered, streaming, out-of-distribution, and sparse tracks. Jeffrey Dean and Luiz André Barroso, "The Tail at Scale," Communications of the ACM, 2013. Not about vectors, but essential for anyone designing a sharded search service. Omar Khattab and Matei Zaharia, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT," SIGIR 2020. The starting point for multi-vector retrieval and its indexing problems. Hashtags: #VectorSearch #VectorSearchAtScale #SemanticSearch #SemanticRetrieval #VectorDatabases #VectorIndexing #VectorEmbeddings #ApproximateNearestNeighbor #ANN #SimilaritySearch #NearestNeighborSearch #HighDimensionalData #HNSW #HierarchicalNavigableSmallWorld #InvertedFileIndex #ProductQuantization #VectorCompression #ScalarQuantization #BinaryQuantization #DiskANN #FAISS #ScaNN #HybridSearch #DenseRetrieval #RetrievalAugmentedGeneration #RAG #InformationRetrieval #DistributedSearch #SearchOptimization #ScalableAI