Download the Book (PDF): Introduction In the floor cavities of a sandstone building on Macquarie Street in Sydney, archaeologists working in the early 1980s found the small possessions of people who had nowhere private to keep them. The Hyde Park Barracks had been built between 1817 and 1819 to house male convicts assigned to government labour gangs, and later sheltered orphaned Irish girls, newly arrived immigrant women and destitute elderly women. Over nearly seven decades, objects slipped or were pushed between the floorboards: scraps of cloth, buttons, clay pipes, pins, playing pieces, pages of letters. Rats carried more of it into their nests. What survived is now counted among the largest collections of nineteenth-century institutional material anywhere. None of it was ever meant to be seen. Much of it was deliberately hidden. That contrast, between a building designed to organise and discipline human labour and the tiny things its inmates hid inside it, is the subject of this book. Every regime that has forced people to work has left material traces of two kinds. There are the traces of the system itself: the walls, fences, watchtowers, leg irons, identity badges, ration bowls, ballast bars and quarry faces through which labour was extracted and bodies were controlled. And there are the traces of the people inside it: the beads worn against regulations, the engraved coins passed to a lover at a prison gate, the scratched name on a barrack wall, the wedding ring kept hidden until the last moment. Both kinds of evidence often lie in the same soil, sometimes in the same layer. Archaeology can recover both. What it chooses to do with them is the question that matters. The argument The book's argument is simple to state and demanding to practise. The material remains of forced labour are double-voiced: they record the regime's design for extracting work while erasing personhood, and they record the persistence of the persons it tried to erase. Every stage of archaeological practice, from the first survey of a landscape to the final label in a museum case, can amplify one voice or the other. Good practice is not neutral between them. It is judged by whether it restores the coerced person to view without reproducing, in a new medium, the logic of the system that coerced them. This is not a sentimental claim. It has practical consequences for which survey methods are chosen, how human remains are approached, what gets conserved and what is allowed to decay, how objects are arranged in a gallery, and who is consulted before any of it happens. A trench cut carelessly through a mass grave repeats a desecration. A museum wall of anonymous shoes, however moving, can turn murdered individuals into a quantity again. A heritage listing that celebrates a mine's engineering and omits the men conscripted to dig it re-performs their disappearance. In each case the archaeologist, conservator or curator is not only describing the past but acting within its unfinished consequences. What the book covers The field examined here is broad, so the book works through a deliberately mixed set of cases rather than a single region. Three families of coercive regime recur. The first is the penal colony and the convict labour system, above all Britain's transportation of convicts to Australia between 1788 and 1868, but also France's penal settlements in Guiana and New Caledonia and colonial prisons such as the Cellular Jail in the Andaman Islands. The second is chattel slavery in the Atlantic world, whose plantations, burial grounds, quarters and shipwrecks have been among the most intensively studied sites in historical archaeology since the 1970s. The third is the forced labour of the twentieth century's wars and dictatorships: the camps of Nazi Germany and occupied Europe, including those on the British Channel Island of Alderney; Japan's wartime mobilisation of Korean, Chinese and Allied prisoner labour; and the Soviet Gulag. These systems differed profoundly. Their legal foundations, their economic logic, the degree to which they intended their workers to die, and the kind of afterlife they have had in public memory are all distinct, and the book tries never to collapse them into a single category of generic suffering. A convict transported for theft who later received a ticket of leave and farmed his own land was not in the position of an enslaved child born into hereditary bondage, and neither was in the position of a Soviet prisoner of war worked to death on a fortification in the Channel Islands. What they share, and what justifies treating them together, is a material problem. In each case, the people whose labour built the site were denied control over their own record. The paperwork that survives was, overwhelmingly, written by those who owned, sentenced or guarded them. The physical remains are often the only evidence produced partly on the workers' own terms. How the book is organised The first chapter sets out what coercion leaves behind in general terms, and why the archive alone is not enough. The second turns to landscapes: how the layout of a camp, a plantation or a penal station encodes a theory of control, and how non-invasive survey methods, from wartime aerial photographs to airborne laser scanning and ground-penetrating radar, have transformed the ability to read those layouts without disturbing what lies beneath. The third chapter addresses excavation itself, especially where human remains are present, and the forensic, legal and religious constraints that govern digging at sites of atrocity. The next two chapters divide the finds. Chapter 4 examines the objects of the regime: restraints, marks of ownership and identification, the architecture and tools of work, and the equipment of feeding and punishment. Chapter 5 examines the objects of the coerced: the things people made, hid, adapted, traded and treasured, and the difficult question of how far such objects can be read as resistance. Chapter 6 turns to conservation, the least discussed and perhaps most consequential stage, where decisions about whether to stabilise, restore or let decay shape what future generations can know. Chapter 7 addresses display, and the ethics of placing instruments of oppression and possessions of the oppressed behind glass. Chapter 8 widens the lens to ask who has authority over these sites and their interpretation: descendant communities, survivor organisations, states, international heritage bodies. The conclusion draws out what follows from all of this for practitioners and for anyone who visits such places. A note on language and limits Words matter in this field, and they are contested. The book uses "enslaved people" rather than "slaves" where it can, because the phrase keeps the person in front of the condition, while recognising that historical sources and some descendant communities use other terms. It uses "forced labour" as an umbrella for work extracted under threat of punishment from people who had not freely offered it, following the broad sense of the International Labour Organization's definition, and it uses more specific terms, such as convict labour or chattel slavery, where the distinction matters. It refers to Nazi camps by function as far as possible, because a forced labour camp at an aircraft factory, a concentration camp and an extermination centre were different institutions even when they were linked. The book is also honest about the limits of its evidence. Archaeology does not replace survivor testimony, descendant memory or documentary history; it sits alongside them, and its findings are often ambiguous. A bead in a quarter may be a charm, an ornament, a trade good or something dropped by a visitor. A worn vertebra may reflect heavy labour or simply age. Much of this book is concerned with how practitioners handle that uncertainty responsibly: saying what the evidence supports, refusing to inflate it into a story it cannot carry, and remembering that the people behind the objects were more complicated than any interpretation of them. The things in the floor at Hyde Park Barracks were hidden because their owners had no right to privacy. Recovering them, and deciding how to show them, is a small restitution of that right. It is also a responsibility. The chapters that follow are an attempt to describe what that responsibility asks. Chapter 1: What Coercion Leaves Behind Every system of forced labour is, among other things, a building project. People who are compelled to work must be moved, housed, fed at the lowest tolerable cost, guarded, counted, and prevented from leaving. They must be given tools, or denied them. Their work must be organised in space: a cane field divided into pieces, a quarry face into allotted sections, a factory floor into stations, a railway embankment into daily lengths. And the whole arrangement must be made visible to those who run it and, as far as possible, invisible or illegible to everyone else. All of this leaves marks in the ground. The archaeology of forced labour begins from the recognition that these marks are evidence of a particular kind. They are not simply the residue of daily life that any settlement produces. They are the physical form of a relationship of domination, and they carry the imprint of both parties to it. A fence line records the intention to confine; a gap dug beneath it records an attempt to leave. A standardised ration bowl records a bureaucracy's calculation of the minimum needed to keep a worker productive; a name scratched into its base records someone insisting that the bowl was theirs. Why the archive is not enough Coercive regimes are, almost without exception, prolific producers of documents. The British convict system generated some of the most detailed records of individual bodies ever assembled before the twentieth century: indents listing each transported convict's height, complexion, eye colour, scars and tattoos, trade, offence and sentence; conduct registers recording every infraction and punishment; musters and assignment lists. Slaveholders kept plantation ledgers, inventories that listed people alongside livestock and tools, and probate records that valued them. The Nazi state and its industrial partners kept labour allocation files, camp registers, transport lists and invoices for prisoner labour hired out to firms. Japan's wartime mobilisation produced recruitment quotas and company payroll records. The Gulag administration kept statistics on output and mortality. These records are indispensable, and much of the best archaeology of forced labour is done by researchers who have mastered them. But they share three structural problems that no amount of archival skill can fully overcome. The first is authorship. The documents were written by the people who ran the system, for purposes of control, profit, or legal cover. They record what the administration needed to know and nothing else. A convict indent tells us the precise length of a man's scar and nothing about how he got it or what he thought about his sentence. A plantation inventory tells us an enslaved woman's assigned value and nothing about the songs she sang or the beads she wore. When enslaved or imprisoned people do appear as active subjects in these records, it is most often as the objects of punishment, which gives a distorted picture of their lives as a sequence of infractions. The second problem is silence by design. Some regimes deliberately avoided recording what they did. The operators of the Operation Reinhard extermination centres at Bełżec, Sobibor and Treblinka in occupied Poland, where small numbers of Jewish prisoners were forced to labour in the killing process itself, dismantled the camps in 1943 and planted over their sites. The German occupiers of Alderney appear to have destroyed many records before the island's liberation in 1945. Plantation records rarely described the internal organisation of the quarters, because the owners did not care how enslaved people arranged their own space so long as they appeared for work. Where there is no document, the ground is the record. The third problem is survival. Archives burn, are pulped, are lost in moves, or are held in countries whose governments restrict access. Records of Soviet forced labour were closed for decades and have become harder to consult again in recent years. Records of wartime Korean labour in Japanese mines and factories are scattered across company, municipal and national archives, many incomplete. Archaeology does not solve these problems, but it approaches them from a different angle. Material remains are not free of bias: what survives in the ground depends on soil chemistry, on later disturbance, and on who had the means to own durable things. But they were not produced primarily as a record. They are the unintended by-products of lives lived and work done, and they were shaped by the coerced as well as the coercers. That makes them a check on the archive and, sometimes, the only surviving voice of the people the archive reduced to entries. Three families of regime It helps to be precise about the systems under discussion, because their material signatures differ as much as their histories. The differences can be summarised along a few lines: the legal basis on which labour was compelled, how the work was organised, what the typical material evidence looks like, and where it has been studied, as Table 1 sets out. Table 1. Five coercive labour systems compared by legal basis, organisation and typical material evidence. System Legal basis of coercion Organisation of labour Typical material evidence Examples of studied sites British convict system (1788–1868) Criminal sentence; fixed term Government gangs, private assignment, probation stations Barracks, penitentiaries, road and bridge works, irons, marked equipment Hyde Park Barracks; Port Arthur; Cascades and Ross Female Factories; Darlington, Maria Island Atlantic chattel slavery (16th–19th c.) Hereditary property in persons Gang and task systems on plantations; urban and domestic labour Quarters, subfloor pits, burial grounds, restraints, ship ballast, local ceramics Newton Plantation, Barbados; New York African Burial Ground; Monticello; São José wreck Nazi forced and slave labour (1933–1945) Racial and political decree; wartime occupation Camps attached to factories, construction sites and SS enterprises Barrack footprints, fences, personal effects, industrial debris, mass graves Tempelhof, Berlin; Treblinka I; Alderney camps; Mittelbau-Dora Japanese war labour (1939–1945) Colonial conscription and recruitment; prisoner-of-war status Mines, factories, railway construction Mine workings, dormitories, railway cuttings, camp sites Burma–Thailand Railway; Hashima; Sado mines Soviet Gulag (1930–1950s) Administrative and criminal sentence Camp complexes tied to extraction and construction Camp perimeters, barracks, mines, burial areas Perm-36; camps of Kolyma and the far north The table flattens a great deal. Convict transportation, for example, was not one system but several, changing sharply after the introduction of the probation system in Van Diemen's Land in the 1840s. Chattel slavery in Virginia's tobacco economy differed materially from slavery on Barbadian sugar estates or in Brazilian cities. Nazi Germany operated tens of thousands of camps of different kinds, from small barracks housing foreign civilian workers near a village factory to vast complexes such as Auschwitz, where labour and murder were interwoven. What the comparison does show is that each system has a recognisable material grammar, and that the archaeologist's first task is to learn it. The two kinds of trace Within each system, the material record can be sorted, roughly, into two categories. The distinction is imperfect, but it organises much of what follows. The first category is the apparatus of the regime: everything built, issued or imposed in order to extract labour and maintain control. It includes architecture and site layout, perimeter and surveillance features, tools and equipment of work, restraints and instruments of punishment, identification marks and badges, and the standardised material of provisioning, such as rations and uniforms. These objects are often durable, because they were made of stone, brick and iron by institutions with capital. They are also often well documented, because the institution ordered and paid for them. Their archaeological value lies in showing how the system actually operated as against how its regulations said it should: whether the prescribed separation between groups of prisoners was maintained, whether rations were as described, whether a fence was where the plan says it was. The second category is the material life of the coerced: what people made, acquired, kept, modified, hid and discarded for their own purposes. It includes personal ornaments, religious and protective objects, food remains that go beyond the ration, tools and objects improvised from scrap, items traded illicitly, marks and inscriptions, and the arrangement of space within quarters. These objects tend to be small, fragile and undocumented. Their value lies precisely in their independence from the regime's record-keeping. The categories overlap constantly. An issued bowl becomes personal when someone scratches their initials into it. A regulation button reappears in a burial as an ornament. A tool of labour is turned into a weapon of escape. A tin can issued as a container becomes a cooking pot, a lamp, a cup. Much of the most interesting interpretive work in the field lies at this seam, where people took the material of their confinement and bent it to their own use. Why this matters now The archaeology of forced labour has grown from a marginal specialism into a substantial field over the past half-century, and several forces have driven that growth. In the Americas, the civil rights movement and the rise of social history in the 1960s and 1970s pushed archaeologists to study the sites of enslaved people rather than only the mansions of their owners. Charles Fairbanks's excavation of slave cabins at Kingsley Plantation in Florida in 1968, and his later work on the Georgia coast, are often cited as a starting point, and Jerome Handler and Frederick Lange's study of the Newton Plantation cemetery in Barbados, published in 1978, established what careful archaeology of enslaved communities could reveal. The rediscovery of the African Burial Ground in Lower Manhattan in 1991, during construction of a federal office building, became a turning point, because the descendant community's demand to control the research reshaped how such work is done. In Europe, the archaeology of the Nazi period was long resisted, partly from a sense that the period was sufficiently documented, partly from reluctance to disturb sites of mass death, and partly because archaeology in Germany had itself been compromised by the regime. That resistance began to give way from the 1990s. Andrzej Kola's survey of Bełżec in the late 1990s, the Sobibor excavations begun in 2007 by Yoram Haimi and Wojciech Mazurek, Caroline Sturdy Colls's work at Treblinka from 2007, and the excavation of forced labour camps at Tempelhof airfield in Berlin by Reinhard Bernbeck, Susan Pollock and their colleagues from 2012 demonstrated that archaeology could add substantially to what was known, even about the most studied events of the twentieth century. In Australia, the conservation movement of the 1970s and 1980s and the approach of the bicentenary of British settlement in 1988 brought convict sites into the heritage system. Large excavations at the Hyde Park Barracks and elsewhere were followed by the inscription of eleven Australian Convict Sites on the UNESCO World Heritage List in 2010. A further force is time itself. The last survivors of the Second World War's forced labour systems are dying. Direct descendants of people enslaved in the United States are now several generations removed from emancipation. As living memory fades, material evidence takes on a heavier burden, and sites become the main points of contact between the public and these histories. This is also why the sites are increasingly contested, and why decisions about excavation, conservation and display have become political in ways that earlier practitioners could sometimes avoid. The final force is denial. Every one of the regimes discussed here has its deniers or minimisers: those who claim convict life was a fair bargain, that slavery was benign, that the Holocaust did not happen or was exaggerated, that wartime labour in Japan's mines was voluntary employment. Physical evidence cannot by itself refute an ideologically committed denier, and archaeologists have learned that their findings are sometimes twisted by the people they might expect to rebut. But evidence in the ground, carefully recorded and publicly presented, sets limits on what can credibly be claimed. The foundations of the gas chambers at Sobibor, located in 2014, are not a matter of interpretation. This is the context in which the rest of the book should be read. Archaeologists working at these sites are not recovering a closed past. They are producing evidence that enters living disputes about responsibility, compensation, national identity and memory. That is why the ethical dimension is not an addition to the method but part of it. Beyond the familiar cases The field's best-known cases come from a small number of countries, above all the United States, Australia, Germany and Poland, and that concentration shapes its assumptions. It is worth remembering how much wider the phenomenon was. France operated penal colonies in French Guiana from the 1850s until the mid-twentieth century and in New Caledonia in the second half of the nineteenth century. Convicts sentenced to transportation, and later repeat offenders sentenced to relegation, were shipped across the world and put to work clearing forest, building roads and constructing the settlements that held them. The Camp de la Transportation at Saint-Laurent-du-Maroni in Guiana, where convicts were received and sorted, survives in part and has become a heritage site, and the remains of the penitentiary islands off the coast, including the notorious Devil's Island, are among the region's most visited places. Much of what has been written about them has come from memoirs and from the colony's own records; archaeological study has been far more limited than at Australian convict sites. Britain's colonial authorities in India built the Cellular Jail at Port Blair in the Andaman Islands, completed in 1906, to hold political prisoners in individual cells arranged in radiating wings, a design related to the separate system that shaped the Separate Prison at Port Arthur. Prisoners were set to labour such as oil pressing and coir work. The jail is now a national memorial in India, visited as a monument to the independence movement, and its meaning is framed almost entirely by that national story. In each of these cases, the physical form of the institution circulated between empires. A radial prison plan devised in Philadelphia and London was reproduced in Tasmania and the Andamans; a barrack design used for soldiers was adapted for prisoners; a regime of classification by conduct was borrowed from one colony by another. Part of the value of a comparative archaeology is that it can trace these borrowings, and show that forced labour systems were not isolated national aberrations but connected parts of a wider history of punishment, empire and labour. Reading doubled evidence One consequence runs through every later chapter. Because the evidence is double-voiced, interpretation needs a discipline that ordinary settlement archaeology sometimes lacks: the habit of asking, of every feature and object, whose intention it primarily records and whose intention it might also record. Take a simple example. At convict sites in New South Wales and Van Diemen's Land, government property was routinely stamped with the broad arrow, a mark of the British Crown's ownership that appeared on tools, buildings materials, clothing and equipment. A fragment of a spade with a broad arrow is, first, a record of the state's claim on the object and, by extension, on the labour of the person using it. But the same mark appears on objects found far from any government establishment, in contexts suggesting they were stolen, traded or kept. The mark of ownership then becomes a record of its own subversion. Or take the ration. At the Tempelhof forced labour camps, excavators found fragments of porcelain tableware bearing printed slogans and insignia of the regime. Such crockery tells us about the regime's attempt to saturate even eating with its symbols. Comparing the distributions of tableware and other finds across different barrack areas, however, can tell us something the regulations only hint at: how differently groups of workers were supplied and treated, and how far the hierarchy of the Nazi racial order, which ranked western European workers above those from Poland and the Soviet Union, reached down to the level of the soup plate. Reading doubled evidence also means resisting two temptations. One is to see resistance everywhere, turning every personal object into heroic defiance. People in forced labour systems did resist, often at great cost, but they also simply lived, and a clay pipe may be nothing more than evidence that someone smoked. The other temptation is to see only the regime, treating the coerced as passive material on which the system acted. That reading, however well meant, repeats the archive's reduction of people to the objects of power. The discipline lies between: taking the apparatus seriously as evidence of what was done, and taking the personal seriously as evidence of who it was done to, without forcing either into a story the material cannot support. The next chapter begins at the largest scale, with the landscapes in which these systems were built. Chapter 2: Landscapes of Control Before a single object is lifted from the ground, a site of forced labour can be read as a document. Its layout records decisions: where the workers slept relative to where their overseers slept, how far they had to walk to the place of work, which lines of sight were kept open and which were blocked, where the boundary ran and how it was guarded, where the dead were put. These decisions were made by people with a theory of how to extract labour from unwilling bodies, and the theory is legible in the plan. So, often, is the pushback: the paths worn where no path was intended, the gardens planted at the margins, the gaps under the wire. This chapter is about reading that landscape, and about the methods that now allow archaeologists to do so at a scale and with a gentleness that were not available a generation ago. Architecture as a theory of labour The clearest cases are those where a regime built from scratch according to an explicit plan. The Separate Prison at Port Arthur in Tasmania, built in the late 1840s and early 1850s and modelled on London's Pentonville, is an architecture of silence. Its wings radiate from a central hall from which a single officer could observe every corridor. Prisoners were confined to individual cells, forbidden to speak, and hooded when moved so that they could not recognise one another; in the chapel, each man stood in a separate stall that allowed him to see the chaplain but not his neighbours. The building is an argument, in stone, that isolation and silence would reform the criminal mind. It is also, now, the best evidence of how that argument was implemented, because the buildings survive even where the regulations are vague. The female factories of Van Diemen's Land and New South Wales, institutions where convict women were housed, put to work, and held between assignments, show a different logic. Eleanor Conlin Casella's excavations at the Ross Female Factory in Tasmania in the 1990s examined how the institution's division of women into classes, from those eligible for assignment to those undergoing punishment, was expressed in the separation of yards and wards. Her finds, which came to include small illicit goods traded across those boundaries, showed that the physical separation designed to sort women by conduct was also the setting for an internal economy that the administration did not control. On plantations, the logic of the plan was surveillance and proximity combined with social distance. At many Chesapeake and Carolina plantations, rows of quarters were placed within sight of the owner's house or the overseer's, so that the movements of enslaved people could be observed, while the house itself was positioned and ornamented to display the owner's mastery. At Monticello, Thomas Jefferson's plantation in Virginia, the row of workshops and dwellings known as Mulberry Row ran along the slope just below the main house, integrating enslaved artisans and domestic workers into the landscape of the mansion while keeping them in a clearly subordinate position. Archaeology along Mulberry Row since the 1950s, and more intensively since the 1980s, has recovered the footprints of buildings that had vanished above ground, turning an idealised landscape of the Enlightenment gentleman back into what it was: a working plantation. The forced labour camps of the twentieth century were often built from standardised, prefabricated components, and their regularity is itself evidence. Barracks built to a small number of stock designs, arranged in grids, fenced, and overlooked by towers, were erected across occupied Europe next to factories, quarries, airfields and construction sites. The standardisation made the camps cheap and quick to build and to dismantle, which is one reason so many vanished almost without trace after 1945. It also means that when an archaeologist finds a characteristic pattern of foundation posts or a regular sequence of rectangular footprints, the layout of a vanished camp can often be reconstructed from fragments. Landscapes of work are just as eloquent as landscapes of confinement. The limestone quarry on Robben Island, where political prisoners including Nelson Mandela were set to dig in glaring white rock, was a place of punishment through labour as much as of production. The Konyu cutting on the Burma–Thailand Railway, known to Allied prisoners as Hellfire Pass, was cut through rock largely by hand in 1943 by Australian, British and other prisoners and by Asian labourers during the period of intensified work the prisoners called the "Speedo". The cutting survives as a scar in the hillside, and walking through it communicates something about the scale of the labour demanded that no document can. The tunnel systems at Mittelbau-Dora in Germany's Harz mountains, where prisoners were forced to excavate and then work in underground factories producing V-2 rockets, are both workplace and tomb. Linear landscapes Not every site of forced labour is a bounded compound. Some of the largest are lines: roads, railways, canals and fortification belts, along which camps were strung like beads, each housing the workers for one section of the project and moving on as the work advanced. The Burma–Thailand Railway is the most notorious example. Built by the Japanese army in 1942 and 1943 to supply its forces in Burma, it ran for roughly 415 kilometres through mountainous jungle between Ban Pong in Thailand and Thanbyuzayat in Burma. Tens of thousands of Allied prisoners of war and a much larger number of Asian labourers recruited or coerced from across the region worked on it, housed in a chain of camps along the route. The death toll among the prisoners of war is usually put at around twelve to thirteen thousand; among the Asian labourers, whose deaths were far less carefully recorded, the numbers are uncertain but are generally estimated in the tens of thousands, far exceeding the prisoner deaths. Archaeologically, the railway is a long ribbon of features: cuttings, embankments, bridge footings, camp sites, cemeteries, and the scattered debris of work. Most of the camps were built of bamboo and thatch and have left little above ground, and many sections of the line have been absorbed into jungle, reservoirs or later railways. The linear form has consequences for research and memory. It is difficult to survey a site four hundred kilometres long, and so attention tends to concentrate at a few accessible points, above all Hellfire Pass and the bridge over the Khwae Yai at Kanchanaburi. The Asian labourers, whose camps were often separate from those of the Allied prisoners and whose experience was less documented, are correspondingly harder to find. A landscape approach that followed the whole line, rather than its most famous points, would almost certainly change the balance of the story. Road-building by convict gangs in Australia and by forced labourers and prisoners of war in German-occupied Finnish Lapland produced similar linear landscapes, with stockades, camps and quarries set at intervals along the route. In each case the road survives as infrastructure, still used, while the camps that built it have faded into the verge. The archaeologist's task is to restore the connection between the two. Rival geographies A landscape designed for control is never only that. The historian Stephanie Camp, writing about enslaved women in the American South, used the phrase "rival geography" to describe the alternative ways enslaved people knew and used the spaces around them: the paths through woods and swamps used to visit family on other plantations, to attend secret gatherings, to trade, or to hide for a time. Such geographies rarely leave dramatic features, but they can sometimes be traced in the distribution of artefacts away from sanctioned areas, in the location of provision grounds and gardens that enslaved people worked for their own subsistence, and in the placement of burial grounds at the edges of estates. At the Newton Plantation in Barbados, the enslaved people's burial ground lay in an uncultivated area away from the planter's domain. Burials there, investigated by Jerome Handler and Frederick Lange in the 1970s, showed practices that owed more to West African traditions than to the planter's church. The cemetery's location was, in part, a consequence of the owners' indifference; it was also a space that the enslaved community controlled. In twentieth-century camps, rival geographies appear at a smaller scale: the gap in a fence, the hiding place beneath a barrack floor, the garden plot, the area behind a latrine where forbidden conversations happened. Survivor testimony is often the key to recognising them, because an anomaly in the ground may mean nothing until someone who was there explains it. Seeing without digging The great methodological change in this field over the past three decades has been the rise of non-invasive survey. It matters here more than in most branches of archaeology, for three reasons. Many sites contain human remains that should not be disturbed, either because of religious law, the wishes of descendants, or simple respect. Many sites are large, overgrown or partly built over, so excavation can reach only a fraction of them. And many are contested, so that evidence gathered without disturbing the ground can be less controversial than evidence obtained by digging. The main tools, which are compared in the next chapter alongside excavation methods, work in complementary ways. Historical aerial photography is the foundation of much twentieth-century work. Allied and German reconnaissance aircraft photographed much of occupied Europe during the Second World War, and collections of these images, held in archives in Britain, the United States and Germany, record camps and work sites while they were in use or shortly after they were abandoned. Comparing wartime images with modern ones shows what has been lost, where structures stood, and where the ground has been disturbed. At Tempelhof, wartime air photographs helped locate the barracks of the forced labour camps that the excavators then investigated. Airborne laser scanning, generally known as LiDAR, sends pulses of light from an aircraft or drone and records their return. By filtering out the returns from vegetation, it produces a model of the ground surface that reveals subtle earthworks, foundations, pits and paths even beneath forest canopy. At Treblinka, where the site of the extermination camp and the adjoining penal labour camp known as Treblinka I were planted with trees after the war, Caroline Sturdy Colls's team combined LiDAR with walkover survey and archival research, and their work suggested that the camp had been considerably larger than the boundary marked by the memorial. On Alderney, similar methods have been used to map the remains of the island's four German labour and concentration camps, Helgoland, Norderney, Borkum and Sylt, and the fortifications their prisoners built. Ground-penetrating radar sends radio waves into the ground and records reflections from buried interfaces, such as the edges of pits, walls, floors and voids. It can suggest the size and depth of buried features, including graves, without breaking the surface, although its results depend heavily on soil conditions and are notoriously easy to over-interpret. Magnetometry detects small variations in the earth's magnetic field caused by buried iron, burnt material and disturbed soil, and is well suited to locating hearths, cremation deposits and metal-rich debris. Electrical resistance survey measures how easily current passes through the ground, which distinguishes, for example, compacted foundations from loose fill. None of these methods produces a picture of what is underground. Each produces a map of physical contrasts, which must be interpreted. A rectangular anomaly on a radar plot may be a grave, a cellar, a refuse pit, a bomb crater, or a natural feature. The strength of modern practice lies in integration: combining several survey methods, overlaying their results in a geographic information system with historical maps, aerial photographs and survivor accounts, and testing the interpretation with small, targeted interventions where these are acceptable. Landscapes of absence Some of the most important landscapes in this field are defined by what is missing. The Operation Reinhard camps were deliberately erased; so were many smaller forced labour sites whose operators, or whose post-war governments, had no wish to preserve them. After 1945, the barracks of countless labour camps in Germany were reused as housing for refugees and displaced persons, then demolished, and their sites built over. In the Soviet Union, the Gulag's camps were closed and allowed to decay in remote places where no one had reason to preserve them, and in some cases the state later discouraged attention to them. On many former plantations, the quarters were demolished while the main house was preserved and restored, producing a heritage landscape that shows the owners and hides the owned. Reading such landscapes requires attention to negative evidence: the unexplained clearing in a forest, the patch of different vegetation, the depression that marks a filled pit, the concrete footing without a building. It also requires attention to later human action, because the erasure was often itself a historical act. When the Nazi authorities dismantled Treblinka and built a farmhouse on the site to disguise it, they created a landscape that is evidence of a cover-up. When a post-war municipality built a housing estate over a labour camp, it made a decision about memory. These layers are part of the site's history and deserve recording as much as the wartime features beneath them. The landscape at Tempelhof illustrates the layering particularly well. The airfield had been the site of one of the earliest Nazi concentration camps, the Columbia-Haus, in the 1930s. During the war, aircraft manufacturers, including Weser Flugzeugbau, built labour camps for foreign workers and prisoners of war on the field's margins. After the war, the airport became a symbol of freedom through its role in the Berlin Airlift of 1948 and 1949, and later served as a commercial airport until its closure in 2008. The excavations from 2012 onward confronted a site where a celebrated story of liberty sat directly on top of a largely forgotten story of coercion, and where the wartime layer was literally beneath the post-war runways and fields. What landscape survey cannot do Landscape-scale methods have made it possible to document sites with minimal disturbance, and in some settings they are the only acceptable approach. They have limits that practitioners must be honest about. They are better at finding structures than people. A survey can locate a barrack's footprint but cannot tell who slept in it. They depend on physical contrasts that some features do not produce: a shallow grave in similar soil may be invisible to radar; a wooden structure that has rotted completely may leave no trace detectable from the surface. They generate large quantities of data that can be misread, and an anomaly announced prematurely as a mass grave can do serious damage if it turns out to be something else. Treblinka and other Holocaust sites have attracted denialists who seize on any gap between a survey interpretation and a later finding. Above all, landscape survey tends to reproduce the view from above, which was also the perspective of those who designed and ran these sites. The aerial photograph is literally a view from the reconnaissance aircraft; the site plan is the view of the architect and the commandant. The lived landscape of the prisoner, the enslaved worker or the convict was experienced on foot, at ground level, in cold and heat, under guard. Archaeologists working at these sites have increasingly tried to bring those experiences back into the interpretation, through walking survey, attention to what could and could not be seen from inside a barrack or a cell, and the integration of testimony. But the tension remains, and it is a first example of the book's larger theme: the methods available to archaeology are not neutral, and their default perspective is often that of power. The next step, when survey alone is insufficient and excavation is justified, carries heavier obligations still. Chapter 3: Excavating Under Moral Pressure Excavation is destruction. Every archaeologist learns this early: once a layer has been dug, it cannot be dug again, and the only thing that survives is the record made while digging. At most sites, this is an accepted trade-off between knowledge and preservation. At sites of forced labour, it acquires a moral weight that changes the whole calculation. The ground may contain the bodies of people who died because of the labour extracted from them. It may be sacred to a religious community or to descendants. It may be evidence in an unfinished legal or political dispute. And it may be the last place where anything of certain individuals survives. The question is not only how to dig well, but whether to dig at all, and for whom. The case for and against digging For much of the twentieth century, many specialists believed that the sites of Nazi atrocity did not need excavation. The crimes were documented by the perpetrators' own records, by the trials at Nuremberg and elsewhere, and by survivor testimony. Digging seemed at best redundant and at worst a desecration. Similar arguments were made, in different forms, about plantation burial grounds and convict cemeteries: that the dead should be left in peace, and that the documents told the story well enough. The argument has shifted, for reasons that the first two chapters have already indicated. The documents are partial, the sites were often deliberately erased, and denial feeds on gaps. Reinhard Bernbeck and Susan Pollock, reflecting on their work at Tempelhof, argued that archaeology's full potential is realised only when excavation is undertaken, because inconspicuous objects, studied systematically and in their spatial context, are a fundamental means of understanding both the conditions of suffering and the transgressive actions of camp inhabitants. Survey can locate a barrack; only excavation can recover the rosary, the wedding ring or the scrap of improvised jewellery that tells us who lived there. The case against digging remains strong wherever human remains are involved, and the resolution in most current practice is a hierarchy of intervention. Non-invasive methods come first and may be all that is done. Where excavation is justified, it is targeted at structures and activity areas rather than burials, designed to answer specific questions, and planned in consultation with the communities most concerned. Where human remains are encountered unexpectedly, work stops and the relevant authorities and communities are consulted before anything further happens. Jewish law and the Holocaust sites The most developed framework for excavation at sites of mass death has emerged from work at Holocaust sites, because Jewish religious law, Halacha, places strict limits on the disturbance of the dead. Graves are to be left undisturbed, and remains that are exposed should be reburied as quickly as possible. In practice, this has meant close cooperation between archaeologists and rabbinic authorities, above all the office of the Chief Rabbi of Poland, whose representatives have monitored work at several sites. At Treblinka, Caroline Sturdy Colls's team deliberately avoided the mass graves during their limited excavations, in order to comply with Halacha. Their excavations targeted structures such as the gas chambers, whose foundations were identified in part through distinctive ceramic tiles made by a Polish firm, consistent with survivor accounts that the chambers had been disguised as a bathhouse. Human remains that were encountered were reburied under rabbinic supervision. The earlier work at Bełżec shows the difficulty of these judgments. Between 1997 and 1999, a team led by Andrzej Kola of Nicolaus Copernicus University in Toruń used thousands of boreholes to map the extent of the burial pits ahead of the construction of a new memorial. The survey identified thirty-three pits interpreted as mass graves and gave the memorial's designers a basis for protecting them. But the technique itself, which involved drilling through burial deposits and bringing samples to the surface, has been criticised as incompatible with the principle of non-disturbance. The later preference for radar, LiDAR and other non-invasive methods at Holocaust sites is partly a response to that experience. At Sobibor, where Yoram Haimi and Wojciech Mazurek began excavations in 2007, joined later by Ivar Schute, the team worked over several seasons to locate the camp's structures, eventually identifying the foundations of the gas chambers in 2014. Among the thousands of personal objects recovered were a wedding ring inscribed in Hebrew with the traditional marriage formula and a pendant later linked to a young girl deported from Frankfurt. These finds came from the areas where victims undressed and walked to their deaths, and from a well used by the Jewish prisoners forced to labour in the camp's killing area. The excavation had to balance the scientific value of finding and documenting these areas against the certainty that parts of the site held human remains. Descendant authority and the African Burial Ground The most consequential case in the Americas began with a legal requirement and ended in a new model of practice. In 1991, archaeological testing required by federal law ahead of construction of a federal office building at 290 Broadway in Lower Manhattan revealed that part of an eighteenth-century burial ground for free and enslaved Africans survived beneath the city's streets. Colonial maps had marked it as the "Negros Burial Ground". Over the following months, the remains of more than four hundred individuals were excavated, under intense time pressure and with little consultation. The response from New York's African American community was forceful. Activists, clergy and politicians protested that the ancestors were being treated as an obstacle to construction, that the excavation was being run by firms with no expertise in African diaspora history, and that the descendant community had no say. Congress intervened, excavation of further burials was halted, and responsibility for the scientific study passed to a team based at Howard University, a historically Black institution, under the direction of the biological anthropologist Michael Blakey. The Howard team built its research design around questions posed by the descendant community: where did these people come from, what was their physical quality of life, what biocultural transformations did they undergo, and how did they resist enslavement. Blakey described the descendant community as the project's "ethical client", whose interests had priority over those of the government agency that funded the work or the scientific community that would read its reports. The studies documented skeletal evidence of hard, repetitive labour among both men and women, of nutritional stress and disease, and of the very high proportion of children among the dead, close to forty per cent. They also documented cultural continuities, including dental modification practised in parts of West and Central Africa, and burial goods such as beads. In 2003 the remains were returned to the site in carved wooden coffins and reinterred, and the burial ground is now a national monument. The African Burial Ground did not settle every question. Some descendants and scholars would have preferred no excavation at all; others valued the knowledge the research produced. What it established was a principle that has since shaped work at other burial grounds of enslaved people: those with a genealogical or cultural relationship to the dead have a right to shape whether, how and why they are studied. At the Anson Street burial ground in Charleston, South Carolina, found during construction in 2013, researchers working with a community group of Gullah Geechee descendants developed a genetic study whose design, sampling and interpretation were negotiated with the community before any laboratory work began; the results, published in 2023, traced the ancestries of the individuals to several regions of Africa. The forensic standard Archaeology at sites of coercion increasingly borrows the standards of forensic practice, developed in the investigation of recent crimes and human rights abuses. Since the founding of the Argentine Forensic Anthropology Team in 1984 to investigate the disappeared of Argentina's military dictatorship, forensic archaeologists have worked in the aftermath of conflicts from the Balkans to Guatemala to Iraq. Their methods are designed to produce evidence that can withstand hostile scrutiny in court. Several elements of that standard carry over to historical sites, even where no prosecution is possible. Recording is exhaustive. Every find is located precisely in three dimensions, every layer is photographed and described, and the relationship of every object to every other is documented. This is good archaeological practice anywhere, but at a site where the evidence may be contested, it must be done in a way that another specialist, or a sceptical reviewer, could check. Chain of custody is maintained. The movement of each find, from the ground to the laboratory to the archive, is logged, so that no one can later claim that material was introduced or altered. Interpretation is disciplined. Forensic reports distinguish between what the evidence shows, what it is consistent with, and what it cannot establish. At sites vulnerable to denial, an overstated claim can do more damage than an understated one, because a single error becomes a pretext for dismissing everything else. The review of the Nazi occupation of Alderney, commissioned by the United Kingdom's special envoy on post-Holocaust issues, Lord Pickles, and published in May 2024, illustrates this discipline. A panel of thirteen historians, archaeologists and other specialists, coordinated by the Cambridge archaeologist Gilly Carr, re-examined the evidence for the number of people who died on the island. The panel estimated that between 7,608 and 7,812 forced and slave labourers had been sent to Alderney, and that between 641 and 1,027 people had died there, well above the figure of 389 that had been widely cited. It also concluded that claims of tens of thousands of deaths, or of Alderney as a "mini-Auschwitz", were not supported by the evidence. The review's credibility rested on its readiness to say both things: that the toll was higher than the official figure, and lower than the most extreme claims. Archaeologists working on the island, including those who had mapped the camps and the fortifications their prisoners built, contributed to that assessment. Digging a labour camp Excavating a twentieth-century forced labour camp is technically different from excavating an ancient settlement, and the differences shape what can be learned. The deposits are shallow and compressed. A camp that was occupied for three or four years, then demolished and perhaps built over, may leave a layer only a few centimetres thick, lying directly beneath modern surfaces. Distinguishing the wartime material from later debris requires close attention to stratigraphy and to the manufacturing dates of objects, many of which carry makers' marks, patent numbers or datable designs. The quantity of material is large and repetitive. The Tempelhof excavations produced a database of some ninety thousand artefacts, a great many of them mass-produced: nails, glass, porcelain fragments, fittings, wire. Faced with such volumes, older practice sometimes recorded only the "interesting" pieces and discarded the rest. The Tempelhof team took the opposite approach, treating even the most inconspicuous finds as data and analysing their distribution statistically across different parts of the camps. The results allowed them to compare how different groups of workers lived, which would have been impossible if only the striking objects had been kept. The ground can be dangerous. Sites of wartime labour may contain unexploded ordnance, contaminated industrial waste, asbestos from demolished buildings and other hazards. Excavation at Tempelhof and at other German sites has had to be coordinated with specialists in explosive clearance. Similar precautions apply at military sites in the Channel Islands and elsewhere. The excavation is also a public event. Digging at a site of Nazi forced labour in the middle of a modern city attracts attention, and the Tempelhof team made a point of opening the excavation to visitors, explaining the work and the history it was uncovering. That openness is part of the ethical practice of such projects: the knowledge is produced in public, for a public that has a stake in it, rather than behind a fence. It also exposes the excavators to challenge, which is a healthy thing at a site where the past is contested. Finally, the objects themselves may carry obligations. Personal possessions recovered from a labour or concentration camp are, in a sense, the property of people who were robbed of everything. Where an owner or a family can be identified, some projects have returned objects or offered to do so. Where they cannot, the objects are held in trust, usually by a memorial museum, rather than treated as ordinary archaeological collections. Comparing methods Different techniques answer different questions at different costs to the site, and the choice among them is always a compromise. Table 2 sets out the main options side by side. Table 2. Principal survey and excavation techniques at sites of forced labour. Technique What it detects Level of disturbance Main limitation Archival aerial photography Structures, fences, pits and disturbance at the time of photography None Coverage and resolution vary; features may be obscured Airborne laser scanning (LiDAR) Surface earthworks, foundations and paths, including under trees None Cannot see features with no surface expression Ground-penetrating radar Buried interfaces, voids, walls, pits and graves None Highly dependent on soil; easily over-interpreted Magnetometry and resistance survey Burnt deposits, iron debris, foundations, disturbed soil None Detects contrasts, not identities; modern metal interferes Coring and augering Depth and content of buried deposits Low but direct; perforates deposits Unacceptable where graves may be pierced Targeted excavation Stratigraphy, artefacts in context, structures, remains High and irreversible Samples only small areas; ethical limits near burials The table understates one factor: time. Non-invasive survey can cover a large site in days, while careful excavation of a single barrack may take a season. At sites facing development, there is often no choice about whether to disturb the ground, only about how well it will be recorded. Much of the archaeology of forced labour in Germany, as in the United States, has taken place in this setting, triggered by construction rather than by research. Bodies of evidence Where human remains are studied, with appropriate consent, they can speak about the physical cost of coerced labour in ways that no document can. Bioarchaeologists read the skeleton for signs of heavy work, including changes where muscles and tendons attach to bone, degeneration of the spine and joints, and certain kinds of fracture. They read teeth and bones for episodes of childhood malnutrition or illness, which leave lines and defects in tooth enamel. Chemical analysis of the isotopes of strontium and oxygen in tooth enamel can indicate the geological region where a person grew up, and so distinguish people born in Africa from those born in the Americas. A study of individuals from the Newton Plantation cemetery in Barbados, published in 2009, used such analysis to identify several people who had probably been born in Africa and forcibly transported, confirming what dental modification had already suggested for some of them. These methods must be used with care. The skeletal markers associated with heavy labour are influenced by age, sex, genetics and body size, and a single skeleton rarely proves a particular occupation. Comparisons between populations are more reliable than readings of individuals, and even they depend on the composition of the sample. The ethical difficulty is also persistent: the bodies of enslaved people were exploited in life, sometimes dissected after death by medical schools, and displayed in museums as racial specimens. Any study that treats them primarily as data risks continuing that history. The best current work, like the Howard University project and the Anson Street study, tries to make the research serve the questions and interests of the communities connected to the dead. The same principle underlies the chapters that follow. What is recovered from the ground carries obligations. The next chapter turns to the first category of finds: the objects through which the regime imposed itself. Hashtags: #TheArchaeologyOfForcedLabor #ForcedLaborArchaeology #MaterialRemains #CoerciveRegimes #DoubleVoicedEvidence #ConvictLabor #AtlanticChattelSlavery #NaziForcedLabor #JapaneseWartimeLabor #SovietGulag #LandscapesOfControl #MaterialCultureOfCoercion #MaterialLifeOfTheCoerced #RivalGeographies #EthicsOfExcavation #NonInvasiveArchaeology #HistoricalAerialPhotography #LiDAR #GroundPenetratingRadar #ForensicArchaeology #DescendantCommunityAuthority #HumanRemainsEthics #AfricanBurialGround #HeritageAndMemory #FutureOfForcedLaborArchaeology

Download the Book (PDF): Introduction Data is the lifeblood of the modern enterprise, and Martin Kleppmann's textbook is the gold standard for understanding how large-scale data systems actually work. But at over 600 pages, it is heavily focused on the complex computer science of replication, partitioning, and distributed transactions. Expecting an IT management or business student to synthesize this immense volume of highly technical material into a concise strategic framework is a massive challenge. This book is your strategic shortcut. It is explicitly written to explain Designing Data-Intensive Applications by Martin Kleppmann, bridging the gap between hardcore software engineering and high-level enterprise architecture. We isolate the core concepts of scalability, consistency, and fault tolerance, translating them into the exact vocabulary you need for your exams. By focusing on the structural logic of data systems, this guide ensures your technological analyses are both accurate and academically rigorous. The book and its author Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems was published by O'Reilly Media in 2017. Its author, Martin Kleppmann, came to the subject from industry before turning to research. He co-founded the startup Rapportive, which LinkedIn acquired, and he then worked at LinkedIn on large-scale data infrastructure, including the stream-processing ecosystem around Apache Kafka and Apache Samza. He later moved into academic research on distributed systems, and he is now based at the University of Cambridge. That mixture of backgrounds shows on every page. The book is precise enough for researchers and practical enough for working engineers, and it cites hundreds of papers, blog posts and incident reports so that readers can trace each claim to its source. The book became something unusual for a technical title: a reference that working engineers, interviewers and university lecturers all treat as common ground. Its lasting appeal comes from a decision Kleppmann made early. Rather than teaching any single product, he explains the principles that products share. Databases, caches, search indexes, message queues and batch processors come and go; the ideas underneath them, such as how to copy data between machines or how to keep concurrent writes from corrupting each other, change far more slowly. A reader who understands those ideas can evaluate a new product in an afternoon, because the product is almost always a new combination of old trade-offs. In March 2026 O'Reilly published a second edition, written by Kleppmann together with Chris Riccomini, an engineer who also worked on data infrastructure at LinkedIn and co-created Apache Samza. The second edition updates the material for a world dominated by cloud services, data warehouses and data lakes, and it revises the organization and terminology in places. It grows to fourteen chapters, opens with a new chapter on trade-offs in data systems architecture, renames the chapter on partitioning to use the now-dominant term sharding, and closes with a chapter devoted to the ethics of data systems. This guide is based mainly on the first edition, whose three-part structure remains the clearest way to learn the subject and is the version most course reading lists still cite. Where the field has moved since 2017, the guide says so and marks the addition as its own. The first edition is organized in three parts. The first, Foundations of Data Systems, covers the ideas that apply even to a single machine: what reliability, scalability and maintainability mean; how data models shape what an application can express; how storage engines lay bytes on disk; and how data is encoded so that it can survive changes to the software that reads it. The second, Distributed Data, asks what happens when data lives on many machines: replication, partitioning, transactions, the unreliability of networks and clocks, and the hard problems of consistency and consensus. The third, Derived Data, looks at systems built by transforming data from one form into another, through batch processing and stream processing, and ends with Kleppmann's view of where data systems are heading, including a serious discussion of ethics and privacy. Why this matters to managers A student of IT management might reasonably ask why any of this is their business. Nobody expects a chief information officer to implement a consensus algorithm. But the choices those algorithms force are business choices. Whether a retailer's checkout should refuse orders during a network failure or accept them and risk overselling is not a question an engineer can answer alone; it depends on what an oversold order costs compared to a lost one. Whether a bank can let customers in two regions update the same account simultaneously depends on regulatory exposure, not just latency. Whether an analytics team should build on nightly batch jobs or real-time streams depends on how fast decisions need to be made and how much the organization is willing to spend on operational complexity. Kleppmann's book is, underneath its technical surface, a catalogue of such trade-offs. Each technique buys something and costs something. Replication buys availability and read capacity but introduces lag and the possibility of conflict. Partitioning buys scale but makes some queries expensive and transactions harder. Strong consistency buys simplicity for application developers but costs latency and availability during failures. The manager who can name these trade-offs, ask which one a proposed architecture has made, and connect that choice to business requirements is doing exactly the work that enterprise architecture exists to do. The controlling idea of this guide This guide argues one thing throughout: every data architecture is a bundle of explicit trade-offs among reliability, scalability and maintainability, and good architectural judgment consists of knowing which guarantees the business actually needs and what each guarantee costs. There is no best database, no correct consistency model and no universally right processing style. There are only systems whose guarantees match their purpose and systems whose guarantees do not. Holding that idea steady makes the technical material far easier to absorb. Every mechanism in the chapters that follow can be read with three questions in mind. What problem does this solve? What does it give up to solve it? When would a business prefer the opposite choice? A student who can answer those three questions for leader-based replication, hash partitioning, snapshot isolation or event sourcing understands the topic at the level an examiner, or an employer, is looking for. How the guide is organized The chapters follow the order of Kleppmann's argument, because that order is itself instructive: it builds from one machine to many, and from storing data to transforming it. The opening chapters establish the vocabulary of system goals, then examine how data is modeled, stored and encoded. The middle of the guide moves into distributed data, where most of the difficult ideas live: replication and its anomalies, partitioning and its hot spots, transactions and their isolation levels, the many ways a distributed system can fail, and the guarantees, such as linearizability and consensus, that engineers build to cope. The later chapters turn to derived data, treating batch and stream processing together because they share a single underlying model, and then to Kleppmann's closing arguments about the design of future systems and the responsibilities of the people who build them. Each chapter explains the relevant concepts in plain terms, then draws out the strategic lens: what decision a manager or architect faces and how to reason about it. Real incidents and published designs anchor the discussion. GitHub's October 2018 outage, in which a 43-second network interruption led to more than a day of degraded service, illustrates why replication and failover are business risks rather than purely technical details. Amazon's 2007 Dynamo paper and Google's 2012 Spanner paper show two of the world's largest companies making opposite choices on consistency, each for sound reasons. The leap second of June 2012, which disrupted websites and airline check-in systems, shows how something as apparently simple as time can undermine a distributed system. Every chapter closes with key takeaways and review questions. The takeaways condense the argument into statements worth remembering; the questions test whether the reasoning, not just the vocabulary, has landed. A short glossary, a list of further reading and notes on sources follow the conclusion. A note on faithfulness. This guide explains Kleppmann's book; it does not replace it. His arguments are paraphrased and attributed, and where the guide adds analysis, modern context or criticism of its own, it says so. Readers who find a topic here that matters to their work should go to the original, which rewards careful reading far more than any summary can. Chapter 1: Reliable, Scalable and Maintainable Systems Kleppmann opens his book by making a distinction that frames everything after it. Many modern applications are not compute-intensive, limited by how fast a processor can crunch numbers. They are data-intensive: their hardest problems are the amount of data, its complexity and the speed at which it changes. A social network, an online store or a banking app does relatively simple computation on each request. What makes it difficult is that millions of such requests arrive, each touching shared data that must be stored, retrieved, indexed, cached, searched and processed in the background. Such applications are rarely built on one piece of software. They are assembled from standard building blocks: a database to store data, a cache to speed up expensive reads, a search index to support keyword queries, a message queue or stream processor to handle work asynchronously, and batch jobs to crunch accumulated data periodically. Kleppmann's point is that once an application stitches these components together, the application developer has become a data system designer. The guarantees the whole system offers, such as whether a cache is correctly invalidated when the database changes, are no longer provided by any single product. They are the responsibility of whoever composed the pieces. To reason about such composite systems, Kleppmann proposes three concerns that matter in almost every case: reliability, scalability and maintainability. They are the vocabulary for the rest of the book, and for most exam questions on the subject. Reliability: working correctly when things go wrong Informally, a reliable system continues to work correctly even when things go wrong. Kleppmann sharpens this by distinguishing a fault from a failure. A fault is one component deviating from its specification: a disk dies, a process crashes, a network link drops packets. A failure is the system as a whole ceasing to provide the required service to the user. Because the probability of faults can never be reduced to zero, the goal is to design fault-tolerant (or resilient) systems, in which faults do not cascade into failures. Kleppmann notes, somewhat provocatively, that it can make sense to increase the rate of faults deliberately, by killing processes at random, so that fault-tolerance machinery is constantly exercised. Netflix's Chaos Monkey is the best-known example of this practice. He groups faults into three kinds. Hardware faults are the traditional concern. Hard disks have a mean time to failure often quoted at 10 to 50 years, which sounds reassuring until one remembers that a data center with ten thousand disks should then expect, on average, around one disk failure per day. The classic response is redundancy: RAID arrays, dual power supplies, backup generators. Kleppmann observes a shift here. As data volumes and computing demands have grown, applications run on far more machines, and cloud platforms are designed to prioritize flexibility over the reliability of any single machine, so virtual machines can disappear without warning. The consequence is a move toward systems that tolerate the loss of entire machines through software, which also brings an operational benefit: a machine can be patched and rebooted one node at a time without taking the whole system down. Software errors are more insidious because they are correlated. A disk failure on one machine says little about the disk on the next machine, but a bug in a shared piece of software can hit every node at once. Kleppmann gives the example of the leap second on 30 June 2012, when a bug in the Linux kernel caused many applications to hang simultaneously; the incident is examined more closely in Chapter 7. Other examples include a runaway process consuming shared resources, a dependency that slows down, and cascading failures in which a small fault triggers faults elsewhere. There is no quick fix, only many small disciplines: careful thinking about assumptions, thorough testing, process isolation, allowing processes to crash and restart, and measuring and monitoring behavior in production. Human errors are, by Kleppmann's reading of the evidence, a leading cause of outages; he cites a study of large internet services that found operator configuration errors to be a larger cause of outages than hardware faults. The response is not to blame operators but to design systems that make the right thing easy and the wrong thing hard: well-designed interfaces and abstractions, sandbox environments where people can experiment safely with real data, thorough automated testing, quick and easy rollback of configuration changes, gradual rollout of new code so that a bug affects a small subset of users first, and detailed monitoring, which Kleppmann calls telemetry by analogy with the instrumentation used to track rockets. Why should a business care? Kleppmann's answer is plain: bugs in business applications cause lost productivity, legal risk if figures are reported incorrectly, and outages of e-commerce sites carry direct costs in revenue and reputation. He also acknowledges that there are situations, such as a prototype for an unproven market, where one might consciously sacrifice reliability to reduce development cost. The strategic point is that this should be a conscious decision, not an accident. Scalability: describing load and performance Scalability is the term used to describe a system's ability to cope with increased load. Kleppmann insists that it is not a one-dimensional label. Saying "X is scalable" or "Y doesn't scale" is meaningless. The useful question is: if the system grows in a particular way, what are our options for coping with that growth? Load parameters and the Twitter example To answer that, one first has to describe the current load succinctly, using load parameters. The right parameters depend on the architecture: requests per second to a web server, the ratio of reads to writes in a database, the number of simultaneously active users, the hit rate on a cache. Sometimes the average case matters most; sometimes a small number of extreme cases dominate. Kleppmann illustrates this with Twitter's home timeline, drawing on a 2012 presentation by Twitter engineers. Two operations mattered. Posting a tweet averaged around 4,600 requests per second, with peaks above 12,000. Reading a home timeline, the stream of tweets from people a user follows, ran at around 300,000 requests per second. Handling 12,000 writes per second is not especially hard in itself. The difficulty was fan-out: each user follows many people and is followed by many people. Twitter tried two approaches. In the first, posting a tweet simply inserted it into a global collection. When a user loaded their home timeline, the system looked up everyone they followed, found all of those people's tweets, and merged them in time order. This is cheap on write and expensive on read, and the read load was far higher than the write load, so the system struggled. In the second approach, Twitter maintained a cache of each user's home timeline, like a mailbox. When someone posted a tweet, the system looked up all of their followers and inserted the new tweet into each follower's cached timeline. Reading a timeline became cheap, because the result had already been computed. This worked better because reads vastly outnumbered writes, so it made sense to do more work at write time and less at read time. The second approach has its own weakness. The average tweet reached about 75 followers, so 4,600 tweets per second became about 345,000 cache writes per second. But the average hides a long tail: some accounts had more than 30 million followers, so a single tweet from such an account triggered more than 30 million writes. Delivering tweets to followers within about five seconds, Twitter's target, became a significant challenge. Twitter settled on a hybrid. Most users' tweets continued to be fanned out at write time, while tweets from a small number of accounts with very large followings were fetched separately at read time and merged in. The lesson Kleppmann draws is that the distribution of followers per user, weighted by how often those users tweet, was the key load parameter for Twitter's scalability. A different application will have a different key parameter, and finding it is the first step of any serious scalability discussion. For a student, the example teaches a general pattern: work can be shifted between write time and read time, and the right balance depends on the shape of the load. Describing performance: percentiles and tail latency Once load is described, one can ask what happens to performance when load increases. Two framings are useful. If load increases and resources stay the same, how is performance affected? And if load increases, how much must resources increase to keep performance unchanged? For a batch system, performance is usually throughput: records processed per second, or the time to run a job over a dataset of a given size. For an online system, it is usually response time: the time between a client sending a request and receiving a response. Kleppmann distinguishes response time, which is what the client sees and includes network and queueing delays, from latency, the duration a request spends waiting to be handled. Response time is not a single number but a distribution, because the same request will take different amounts of time on each attempt. The average (the arithmetic mean) is a poor summary, because it says nothing about how many users actually experienced a given delay. Kleppmann recommends percentiles. Sort all response times from fastest to slowest; the median, or 50th percentile (p50), is the halfway point. If the median is 200 milliseconds, half of requests return in less than 200 milliseconds and half take longer. Higher percentiles, such as p95, p99 and p999, show how bad the outliers are. A p99 of 1.5 seconds means that 99 out of 100 requests are faster than 1.5 seconds and one in a hundred is slower. These high percentiles are called tail latencies, and Kleppmann argues they matter commercially. He notes that Amazon describes internal response time requirements in terms of the 99.9th percentile, even though that affects only one request in a thousand, because the customers with the slowest requests are often those with the most data in their accounts, who have made many purchases and are therefore among the most valuable. Amazon's published Dynamo paper gives an example of this style of requirement: a service guaranteeing a response within 300 milliseconds for 99.9 percent of requests at a peak load of 500 requests per second. Kleppmann also reports Amazon's observation that small increases in response time measurably reduce sales. Optimizing the 99.99th percentile, by contrast, was judged too expensive for the benefit. Percentiles are often used in service level objectives (SLOs) and service level agreements (SLAs), contracts that define the expected performance and availability of a service. A typical formulation might state that the service is considered up if its median response time is under 200 milliseconds and its p99 is under one second. Two further points make tail latency especially important. First, head-of-line blocking: a server can process only a few requests in parallel, so a few slow requests can hold up everything queued behind them. This is why response times should be measured on the client side. Second, tail latency amplification: when serving one user request requires calling several backend services, the chance that at least one of them is slow grows with the number of calls. Even if only a small fraction of backend calls are slow, a large fraction of end-user requests can end up slow. Approaches to coping with load Kleppmann distinguishes scaling up (vertical scaling, moving to a more powerful machine) from scaling out (horizontal scaling, distributing load across many smaller machines). The latter is also called a shared-nothing architecture. Systems that add resources automatically when load increases are called elastic. Kleppmann's practical advice is balanced: distributing stateless services is relatively simple, but taking stateful data systems from one node to many introduces a great deal of complexity, and until fairly recently it was common wisdom to keep a database on a single node until cost or availability forced otherwise. He expects distributed data systems to become the default as tools improve, which, looking back from today, is largely what happened with cloud-managed databases. The most important point is that there is no generic, one-size-fits-all scalable architecture. A system designed for 100,000 requests per second, each 1 kilobyte in size, looks very different from one designed for three requests per minute, each 2 gigabytes in size, even though the two have the same data throughput. An architecture that scales well is built around assumptions about which operations will be common and which rare. If those assumptions are wrong, the scaling effort is wasted. For an early-stage product, Kleppmann notes, it is usually more important to be able to iterate quickly on features than to scale to some hypothetical future load. Maintainability: the cost that lasts Kleppmann reminds readers that the majority of the cost of software is not its initial development but its ongoing maintenance: fixing bugs, keeping systems operational, investigating failures, adapting to new platforms and use cases, repaying technical debt and adding features. Many people dislike working on so-called legacy systems, but every system becomes legacy eventually, so it should be designed to minimize the pain. He proposes three design principles. Operability means making it easy for operations teams to keep the system running smoothly. Good operations teams monitor health, track down problems, keep software patched, anticipate future problems such as capacity limits, establish good practices for deployment and configuration, and preserve the organization's knowledge about the system as people come and go. Data systems help by providing visibility into their internal behavior, supporting automation, avoiding dependence on individual machines, offering good documentation and predictable behavior, and providing sensible defaults while allowing administrators to override them. Simplicity means making it easy for new engineers to understand the system, by removing as much complexity as possible. Kleppmann distinguishes essential complexity, inherent in the problem the software solves, from accidental complexity, which arises only from the implementation. The most powerful tool for removing accidental complexity is abstraction: a good abstraction hides implementation detail behind a clean, understandable interface and can be reused across many applications. High-level programming languages abstract away machine code; SQL abstracts away on-disk data structures and concurrency. Much of the book is about finding good abstractions for distributed data. Evolvability means making it easy to change the system in the future, as requirements, business priorities, regulations and platforms shift. Kleppmann connects this to agile practices but notes that most discussion of agility focuses on small scales, such as a few source files. His interest is in agility at the level of a whole data system, such as refactoring Twitter's architecture from one timeline approach to another. Simplicity and good abstractions make that kind of large change far easier. The strategic lens For a manager, Chapter 1 of Kleppmann's book supplies a checklist that can be applied to any proposed architecture. On reliability: which faults does the design tolerate, which does it not, and what is the business cost of each failure mode? On scalability: what are the load parameters, which one is expected to grow, and what happens to tail latency when it does? On maintainability: how will operators see what the system is doing, how much of its complexity is accidental, and how hard will it be to change when the business changes? This guide adds one observation of its own. These three properties compete. Redundancy improves reliability but adds components to operate. Scaling out handles load but adds the complexity of distribution. Aggressive simplicity may mean accepting a single-node database with a lower ceiling. Architectural judgment is the art of choosing which property to favor, for which part of the system, at which stage of the business's life. Key Takeaways · A fault is a component deviating from specification; a failure is the whole system ceasing to serve users. Reliable design aims to stop faults becoming failures. · Scalability is not a label but a question: if load grows along a specific parameter, what options exist to cope? · Twitter's home timeline shows that work can be shifted between write time and read time, and that the distribution of followers, not the average, was the critical load parameter. · Response times should be described with percentiles, not averages; tail latencies (p99, p999) often affect the most valuable users and are amplified when requests call many services. · Maintainability rests on operability, simplicity (removing accidental complexity through good abstractions) and evolvability. Review Questions 1. Distinguish a fault from a failure, and explain why deliberately inducing faults can improve reliability. 1. Why are software errors often more damaging than hardware faults in a large system? 2. Explain the two approaches Twitter used to build home timelines, and why the company moved to a hybrid of both. 3. A service reports an average response time of 150 milliseconds. Why might this figure mislead a manager, and what should be reported instead? 4. What is tail latency amplification, and what does it imply for an architecture composed of many microservices? 5. Give an example of accidental complexity and explain how abstraction can remove it. Chapter 2: Data Models and Query Languages Kleppmann argues that data models are perhaps the most important part of developing software, because they shape not only how software is written but how its authors think about the problem. Most applications are built by layering one data model on another. An application developer models the real world as objects and data structures; those structures are expressed in a general-purpose model such as tables, JSON documents or graphs; the database engineers decide how that model is represented as bytes in memory and on disk; and hardware engineers decide how those bytes become electrical currents and magnetic fields. Each layer hides the complexity of the layer below by providing a clean model. Choosing the model at the second level, the one application developers work with every day, is a decision with long consequences, because it determines which operations are easy and which are awkward or slow. The relational model and the document challenge The relational model, proposed by Edgar Codd in 1970, organizes data into relations (tables), each an unordered collection of tuples (rows). It was a theoretical proposal at first, and many doubted it could be implemented efficiently, but by the mid-1980s relational database management systems and SQL had become the standard tools for storing and querying data with some regular structure. Kleppmann notes the remarkable durability of this dominance. Competing approaches, including the network and hierarchical models of the 1970s and the object databases and XML databases of later decades, generated excitement and then faded, while relational databases went on to power a vast range of applications well beyond their origins in business data processing. In the 2010s the challenger was NoSQL, a term Kleppmann notes was originally just a catchy hashtag for a 2009 meetup on open source, distributed, non-relational databases. The drivers he identifies are worth remembering: a need for greater scalability than relational databases could easily achieve, including very large datasets and very high write throughput; a preference for free and open-source software; specialized query operations poorly supported by the relational model; and frustration with the restrictiveness of relational schemas. He predicts, correctly as it turned out, that relational and non-relational stores would be used side by side, an idea sometimes called polyglot persistence. The object-relational mismatch Most application code is written in object-oriented languages, and translating between objects in code and rows in tables requires an awkward layer of glue. This disconnect is called the impedance mismatch, a term borrowed from electronics. Object-relational mapping frameworks reduce the boilerplate but cannot hide the differences entirely. Kleppmann illustrates this with a résumé, such as a professional networking profile. A person has one name, but may have several jobs, several periods of education and several pieces of contact information. In a traditional relational schema, the one-to-many relationships would be split into separate tables, with foreign keys pointing back to the user. Fetching a full profile requires either several queries or a multi-way join. A document model represents the same profile as a single self-contained JSON document, in which the list of positions and the list of education entries are nested inside the user record. The document has better locality: everything about the profile is in one place, retrieved with one query. The tree structure of one-to-many relationships is made explicit in the document's nesting. Many-to-one and many-to-many relationships The document model's weakness appears with relationships that are not tree-shaped. Suppose each profile lists a region and an industry. It is better to store a standardized identifier, such as an ID for "Greater Seattle Area", than a free-text string, because a standardized list gives consistent spelling, avoids ambiguity, makes updates easy, supports localization and enables better search. Storing an ID rather than the human-readable text is a form of normalization: the meaningful information is stored in one place and referred to everywhere else, so there is no duplication to keep consistent. Normalization requires many-to-one relationships (many people live in one region), and these fit relational databases naturally because joins are easy. In document databases, joins were historically weak or absent, so the application had to emulate them with multiple queries. Moreover, data tends to become more interconnected as features are added. If organizations and schools become entities with their own pages, and users can recommend one another, the résumé acquires many-to-many relationships that a pure document tree cannot express cleanly. Kleppmann places this debate in history. The hierarchical model of IBM's IMS in the 1970s, which represented data as trees of records nested inside records, much like JSON, faced exactly the same difficulty with many-to-many relationships. Two solutions were proposed: the relational model, and the network model (CODASYL), in which records could have multiple parents and were accessed by following pointer-like access paths. The network model required programmers to know and navigate these paths manually, which made queries and schema changes painful. The relational model instead let a query optimizer decide automatically how to execute a query and which indexes to use. That separation, Kleppmann argues, is why the relational model won: the hard work of choosing access paths was done once, by the optimizer, rather than repeatedly by every application developer. Which model leads to simpler code? Kleppmann's comparison is even-handed. The document model offers schema flexibility, better performance due to locality, and a closer match to application data structures for some applications. The relational model offers better support for joins and for many-to-one and many-to-many relationships. If the application's data has a document-like structure, a tree of one-to-many relationships usually loaded all at once, the document model is a good fit. Shredding such a structure into many tables can make code unnecessarily complicated. If the application uses many-to-many relationships heavily, the document model becomes less appealing: joins can be emulated in application code, but that moves complexity into the application and is usually slower than a join performed by the database. For highly interconnected data, a graph model is the most natural. Kleppmann adds a caution about locality. The performance benefit of storing a whole document together applies only when the application needs large parts of the document at the same time. The database typically loads the entire document even if only a small portion is read, which is wasteful for large documents. On updates, the entire document usually has to be rewritten; only modifications that do not change the encoded size of a document can easily be performed in place. For these reasons, he recommends keeping documents fairly small and avoiding writes that increase their size. He also notes that the idea of grouping related data for locality is not unique to document databases: Google's Spanner offers the same property within a relational model, by allowing tables to be declared as nested within a parent table, and column-family stores in the Bigtable tradition achieve something similar. A related consideration is how the two models handle a field that is shared across many records. In a document database, a value such as a company name may be copied into every profile that mentions it. If the company is renamed, every copy must be found and updated, and any copy that is missed leaves the data inconsistent. In a normalized relational schema, the name is stored once and referenced by ID, so a rename is a single update. Denormalization is not wrong, and it is often a deliberate performance choice, but it transfers the job of keeping copies consistent from the database to the application. He also observes a convergence between the models. Relational databases such as PostgreSQL and MySQL added support for JSON documents, and some document databases added join-like features. A hybrid of the two is a sensible direction for databases to take. In the years since the book appeared, that convergence has continued, and it is now common for a relational database to store semi-structured JSON columns alongside normal tables. Schema-on-read versus schema-on-write Document databases are often called schemaless, but Kleppmann rejects the term as misleading. The code that reads the data usually assumes some structure; there is an implicit schema even if the database does not enforce it. A more accurate distinction is this. Schema-on-read means the structure of the data is implicit and interpreted only when the data is read. Schema-on-write means the schema is explicit and the database ensures all written data conforms to it, which is the traditional relational approach. He compares the difference to dynamic versus static type checking in programming languages, a debate with no single right answer. The practical difference shows up when the application changes the format of its data. Suppose names were stored as a single field and the application now wants first and last names separately. With schema-on-read, new code simply starts writing documents with the new fields and includes logic to handle old documents that lack them. With schema-on-write, one would typically run a migration that alters the table, adds the column and backfills existing rows. Schema changes have a reputation for being slow and requiring downtime. Kleppmann notes that this reputation is not entirely deserved: most relational databases execute an added column in milliseconds, although some older engines copied entire tables on certain alterations, and updating every row to backfill a value can be slow on a large table. Schema-on-read is advantageous when the items in a collection do not all share the same structure, for example when there are many different kinds of objects or when the structure is determined by external systems outside the organization's control. When all records are expected to share a structure, an explicit schema is a useful mechanism for documenting and enforcing it. From a management perspective, the choice is between flexibility now and discipline later. Schema-on-read allows a team to move quickly, but it moves the burden of handling historical data formats into every piece of code that reads the data. Schema-on-write centralizes that burden in migrations, which are more visible and easier to govern. Query languages: declarative and imperative When the relational model was introduced, it came with a new way of querying data. SQL is a declarative language: the user specifies the pattern of the data wanted, what conditions the results must meet and how they should be transformed, but not how to achieve that. An imperative language, by contrast, tells the computer to perform operations in a particular order, looping through records and testing each one. Kleppmann lists the advantages of declarative queries. They are more concise and easier to work with. They hide the implementation details of the database engine, which allows the engine to be improved without changing any queries. They lend themselves to parallel execution, because they specify only the pattern of results and not the algorithm, so the database is free to use a parallel implementation. He illustrates the idea with an analogy from the web: CSS selectors declaratively describe which elements should be styled, which is far simpler and more robust than writing imperative code to walk the document tree and change styles manually. He also discusses MapReduce querying, a programming model popularized by Google and supported in a limited form by some document databases. It sits between declarative and imperative styles: the query logic is expressed as snippets of code called repeatedly by the processing framework. Its usability problems led some databases to add more declarative query languages on top of it, which illustrates a recurring theme: declarative interfaces tend to win once they are available, because they separate what the user wants from how the system provides it. Graph-like data models When many-to-many relationships are very common, it becomes more natural to model data as a graph, consisting of vertices (nodes or entities) and edges (relationships or arcs). Social graphs, where vertices are people and edges indicate who knows whom, the web graph of pages and links, and road or rail networks are typical examples. Graphs are not limited to one kind of object: a single graph can store people, locations, events, check-ins and comments together, which Kleppmann notes is how Facebook maintains its data. He describes two main flavors. In the property graph model, used by systems such as Neo4j, each vertex and edge has a unique identifier, a label describing the relationship, and a collection of key-value properties. Any vertex can connect to any other, and one can efficiently traverse the graph forward and backward. This makes graphs highly evolvable: new kinds of relationships can be added without changing a schema. Kleppmann's example of querying such a graph uses Cypher, a declarative query language created for Neo4j, to find people who emigrated from the United States to Europe. Such a query follows chains of "within" edges of variable length, from a city to a state to a country to a continent, which is very awkward in SQL because the number of joins is not known in advance; SQL can express it only with recursive common table expressions, which are far more verbose. In the triple-store model, all information is stored as three-part statements of the form subject, predicate, object, such as (Jim, likes, bananas). Triple stores are closely associated with the semantic web and the RDF data model, which aimed to let websites publish machine-readable data. Kleppmann notes that the semantic web was overhyped in the early 2000s and has not been realized as its proponents imagined, but that the triple-store model remains a useful internal data model regardless. SPARQL is the query language for RDF triple stores. He also discusses Datalog, a much older logic-based query language that underlies several later systems and builds complex queries from small reusable rules. Kleppmann is careful to distinguish graph databases from the network model of the 1970s. In CODASYL, the schema specified which record types could be nested within which, access to records required following a path, and the ordering of children was fixed. Graph databases have none of those restrictions: any vertex can link to any other, vertices can be accessed directly by identifier or through an index, and query languages are declarative. Table 1 compresses the comparison of the three main models, which the discussion above has developed in detail. Table 1. Three general-purpose data models compared. Model Natural data shape Strengths Weaknesses Typical query style Relational Regular records with many-to-one and many-to-many links Joins, optimizer, mature tooling Mismatch with nested objects SQL (declarative) Document Self-contained trees of one-to-many data Locality, flexible schema Weak joins, duplication risk Document queries, aggregation pipelines Graph Highly interconnected entities Variable-length traversals, evolvability Specialized skills and tooling Cypher, SPARQL, Datalog The strategic lens The key managerial insight from this part of Kleppmann's book is that a data model is a bet about the future shape of the business's data. A startup that stores user profiles as documents because they are tree-shaped today may find, two years later, that its most valuable features depend on relationships between users, organizations and content. At that point it faces either an expensive migration or a growing mass of application code emulating joins. Conversely, forcing highly variable data, such as product catalogs with attributes that differ by category, into rigid relational tables can create accidental complexity. Three questions, drawn from the book's reasoning, help frame the decision. How interconnected will the data become as the product grows? Is the data usually read as a whole unit or combined in many different ways? Who controls the structure of incoming data, the organization or external parties? The answers do not dictate a product, but they narrow the choice considerably, and they explain why many enterprises end up running more than one kind of database. Key Takeaways · Data models shape how developers think about a problem; choosing one determines which operations are easy and which are hard. · Document models suit self-contained, tree-shaped data with one-to-many relationships; relational models suit data with many-to-one and many-to-many relationships. · The relational model won historically because its query optimizer freed developers from navigating access paths manually. · "Schemaless" is better described as schema-on-read; schema-on-write enforces structure at the database and pushes change into migrations. · Declarative query languages hide implementation details, allowing engines to optimize and parallelize without changing queries. · Graph models, in property-graph or triple-store form, are the natural choice for highly interconnected data. Review Questions 1. Explain the impedance mismatch and show how a document model reduces it for a résumé-style profile. 1. Why do many-to-many relationships weaken the case for a document database? 2. Compare schema-on-read and schema-on-write, and describe a business situation that favors each. 3. What advantages does a declarative query language offer over an imperative one? 4. Why is a variable-length traversal, such as finding everyone who lives somewhere within Europe, easier in a graph query language than in SQL? Chapter 3: Storage Engines and Data Encoding Application developers rarely write their own storage engines, but Kleppmann argues they still need a rough idea of what happens inside one. Choosing a storage engine appropriate to the workload, and tuning it, requires understanding what it is optimized for. The same is true of encoding, the question of how in-memory data is turned into bytes that can be written to a file or sent over a network. Both topics belong to the foundations of data systems because they determine performance on a single machine and, crucially, how easily a system can evolve over time. How databases store and find data Kleppmann begins with what he calls the world's simplest database: two shell functions, one of which appends a key and value to the end of a text file, and another which scans the file for the most recent value for a key. Writing is extremely efficient, because appending to a file is about the cheapest thing a disk can do. Many real databases internally use a log in the same sense: an append-only sequence of records. Reading, however, is terrible, because each lookup scans the entire file, so the cost grows in proportion to the amount of data. To find data efficiently, databases use an index, an additional structure derived from the primary data that acts as a signpost. Indexes speed up reads but slow down writes, because every write must also update the index. This is an important trade-off, and it is why databases do not index everything by default: the application developer or administrator chooses indexes based on the application's typical query patterns. Log-structured storage and LSM-trees The simplest indexing strategy for an append-only log is an in-memory hash map from each key to the byte offset of its latest value in the file. Kleppmann notes that this is essentially what Bitcask, the default storage engine in the Riak database, does. It works well when there are relatively few distinct keys, each updated frequently. To prevent the log from growing forever, it is broken into segments, and a background process performs compaction, discarding duplicate keys and keeping only the most recent value for each, then merging segments together. Append-only design has real advantages. Sequential writes are much faster than random writes on magnetic disks, and are also favorable on solid-state drives. Concurrency and crash recovery are simpler when segment files are immutable. Merging old segments avoids fragmentation. But a hash index must fit in memory, and it cannot efficiently answer range queries, such as all keys between two values. The next step is the Sorted String Table, or SSTable: a segment file in which the key-value pairs are sorted by key. Sorting makes merging segments efficient, similar to the merge step of the mergesort algorithm, and means the in-memory index can be sparse, holding only some keys, because the position of any other key can be found by scanning a small region between two known keys. How do records get sorted, when writes arrive in arbitrary order? Incoming writes are added to an in-memory balanced tree called a memtable. When the memtable grows beyond a threshold, it is written to disk as a new SSTable. To serve a read, the engine checks the memtable, then the most recent on-disk segment, then older ones. Background merging and compaction run periodically. To protect against losing the memtable in a crash, every write is also appended to a separate log on disk, which is used only for recovery. This design is known as a Log-Structured Merge-Tree, or LSM-tree, a name taken from a 1996 paper by Patrick O'Neil and colleagues. Kleppmann notes that storage engines based on this principle include LevelDB and RocksDB, and that similar engines are used in Cassandra and HBase, both inspired by Google's Bigtable paper. The same principle, keeping a sorted index and merging it in the background, also underlies the full-text search engine Lucene, used by Elasticsearch and Solr. Optimizations include Bloom filters, memory-efficient structures that can say with certainty that a key does not exist, avoiding needless disk reads, and different compaction strategies, such as size-tiered and leveled compaction. B-trees The most widely used indexing structure is quite different. B-trees were introduced in 1970 and, in Kleppmann's words, became ubiquitous within a decade; they remain the standard index in almost all relational databases and many non-relational ones. Like SSTables, they keep key-value pairs sorted by key, which supports efficient lookups and range queries. But rather than variable-size segments written sequentially, B-trees break the database into fixed-size pages, traditionally four kilobytes, and read or write one page at a time, a design that corresponds closely to the underlying hardware. Each page contains keys and references to child pages, with the keys marking the boundaries between ranges. Looking up a key starts at the root page and follows references down the tree until reaching a leaf page containing the value. The number of references per page, the branching factor, is typically several hundred, so the tree stays shallow: Kleppmann notes that a four-level tree of four-kilobyte pages with a branching factor of 500 can store up to 256 terabytes. Updating a value means finding its leaf page and overwriting it in place. Adding a key may require splitting a full page into two and updating the parent. Because B-trees overwrite pages in place, a crash in the middle of a page split could leave the tree corrupted. To guard against this, B-tree implementations use a write-ahead log (WAL), an append-only file to which every modification is written before it is applied to the pages. Concurrency also requires care, typically using lightweight locks called latches. Comparing the two Kleppmann offers rules of thumb while cautioning that benchmarks for a specific workload are the only reliable guide. LSM-trees are typically faster for writes, whereas B-trees are thought to be faster for reads, because an LSM-tree read may have to check several structures. LSM-trees usually achieve lower write amplification, meaning one logical write results in fewer physical writes over the database's lifetime, which matters especially on solid-state drives that can tolerate only a limited number of overwrites. They also compress better and produce smaller files on disk. On the downside, compaction can interfere with ongoing reads and writes, producing unpredictable high-percentile response times, and if compaction cannot keep up with a high write rate, unmerged segments accumulate until disk space runs out. B-trees offer a strong advantage for transactional semantics, because each key exists in exactly one place in the index, which makes it straightforward to attach locks to ranges of keys. He also covers other index structures briefly: secondary indexes, which are essential for joins and can be built with either approach; clustered indexes, which store the row data within the index itself, versus heap files that store rows elsewhere; multi-column indexes, including specialized spatial indexes; full-text and fuzzy indexes; and in-memory databases, whose performance advantage, he points out, comes less from avoiding disk reads than from avoiding the overhead of encoding in-memory structures into a form that can be written to disk. Transaction processing versus analytics A second major distinction in Kleppmann's treatment of storage is between two patterns of use. In the early days of business data processing, a write to the database typically corresponded to a commercial transaction, such as a sale or a payment. The term transaction stuck, even though databases came to be used for many other kinds of data. The access pattern of an interactive application, looking up a small number of records by key and inserting or updating records based on user input, is called online transaction processing (OLTP). Databases are also increasingly used for data analytics, which has a very different access pattern. An analytic query scans over a huge number of records, reads only a few columns per record, and calculates aggregate statistics such as counts, sums or averages. A question such as total revenue per store in January, or how many more bananas than usual were sold during a promotion, is analytic. This pattern is called online analytic processing (OLAP). The differences are summarized in Table 2. Table 2. OLTP and OLAP workloads compared. Property OLTP OLAP Main read pattern Few records, fetched by key Aggregates over many records Main write pattern Random-access, low-latency writes Bulk import (ETL) or event stream Primary users Customers via a web application Internal analysts for decision support Data represents Latest state of data History of events over time Typical dataset size Gigabytes to terabytes Terabytes to petabytes Typical storage layout Row-oriented with B-tree or LSM indexes Column-oriented and compressed Data warehousing In the 1980s and 1990s, companies increasingly stopped running analytics on their OLTP systems and moved them to a separate database called a data warehouse. The reasoning is straightforward. OLTP systems are expected to be highly available and to process transactions with low latency, because they are critical to running the business. Analysts running expensive ad hoc queries on them could harm performance for customers. A data warehouse contains a read-only copy of data from the various OLTP systems in the company. Data is extracted from those systems, transformed into an analysis-friendly schema, cleaned up and loaded into the warehouse. This process is called Extract-Transform-Load (ETL). Many warehouses use a star schema, also known as dimensional modeling. At its center is a fact table, each row representing an event, such as one customer's purchase of one product at a particular time. Fact tables can become extremely large. Some columns are attributes, such as the price; others are foreign keys referencing dimension tables, which describe the who, what, where, when, how and why of the event: the product, the store, the date, the customer, the promotion. When dimension tables are further normalized into sub-dimensions, the result is called a snowflake schema. Column-oriented storage Fact tables are often over 100 columns wide, yet a typical analytic query touches only four or five of them. In most OLTP databases, storage is laid out in a row-oriented fashion: all values from one row are stored next to each other. For an analytic query, the engine must load every row, with all its columns, from disk into memory, parse it, and filter out the rows that do not meet the conditions. That is slow. Column-oriented storage stores all the values from each column together instead. If each column is stored in a separate file, a query needs to read and parse only the columns it uses. The columns must all store rows in the same order, so that the twenty-third entry in each column file belongs to the same row. Kleppmann notes that the Parquet format, based on Google's Dremel, is a columnar storage format. Column storage lends itself to compression, because the values in one column are often repetitive. A technique called bitmap encoding can represent a column with a small number of distinct values as a set of bitmaps, one per value, which can then be compressed with run-length encoding. Such bitmaps also allow certain queries to be answered with fast bitwise operations. Beyond reducing disk reads, compact columns make efficient use of processor caches and allow vectorized processing, in which the engine operates on chunks of compressed column data in tight loops. Sorting rows in a column store can further help compression and queries, and some systems even store the same data sorted in several different ways on different replicas. Writes are harder in column stores, because inserting a row in the middle of a sorted, compressed column would require rewriting every column file. The solution is the LSM-tree approach: accumulate writes in memory and merge them into the column files in bulk. Warehouses also use materialized views and data cubes, precomputed aggregates that make certain queries very fast at the cost of flexibility. This guide adds a note on context. Since the book's publication, cloud data warehouses and open table formats have made columnar analytics widely available as managed services, and the separation of storage from compute has become common. The underlying principles Kleppmann describes, column orientation, compression and the separation of analytic from transactional workloads, remain the reason those services perform as they do. Encoding and evolution Applications change over time. Kleppmann connects this to the evolvability goal from the book's first chapter. When a feature changes, the data it stores usually changes too. In large applications, however, code changes cannot happen instantaneously. Server-side applications are updated through rolling upgrades, deploying the new version to a few nodes at a time, and client-side applications depend on users choosing to install updates. Old and new versions of code, and old and new data formats, therefore coexist. This requires compatibility in both directions. Backward compatibility means newer code can read data written by older code. Forward compatibility means older code can read data written by newer code. Backward compatibility is usually not hard, because the author of new code knows the old format. Forward compatibility is trickier, because it requires older code to ignore additions made by a newer version. Formats for encoding data Programs work with data in memory as objects, structures, lists and hash tables. To write data to a file or send it over a network, it must be converted into a self-contained sequence of bytes. That translation is called encoding (also serialization or marshalling), and the reverse is decoding. Kleppmann warns against language-specific formats such as Java's built-in serialization or Python's pickle. They tie data to one programming language, often create security problems because decoding can instantiate arbitrary classes, treat versioning as an afterthought, and are frequently inefficient. Textual formats such as JSON, XML and CSV are widely supported and somewhat human-readable, but have subtle problems: ambiguity around numbers, with JSON not distinguishing integers from floating-point values and losing precision on very large numbers; no native support for binary strings; optional and complex schema support; and, in the case of CSV, no schema at all. They remain good enough for many purposes, especially for data interchange between organizations, where the difficulty of getting everyone to agree on any format outweighs most other concerns. Binary formats with schemas For internal use, binary formats with schemas offer compactness and clearer evolution rules. Kleppmann examines three. Apache Thrift, originally developed at Facebook, and Protocol Buffers, developed at Google, both require a schema that defines each field with a name, a type and a numeric field tag. The encoded data refers to fields by tag number rather than by name, which is compact. Evolution follows from the tags. A new field can be added with a new tag number; old code that does not recognize the tag simply skips it, giving forward compatibility. New code can read old data as long as each tag keeps its meaning, giving backward compatibility, provided that any newly added field is optional or has a default value. Field names can change freely, but tag numbers cannot. Removing a field is the mirror image: only optional fields can be removed, and the tag number must never be reused. Apache Avro, which began as a subproject of Hadoop, takes a different approach. Its encoding contains no tag numbers or field identifiers at all; it is simply values concatenated together. Decoding therefore requires the exact schema the data was written with. Avro resolves this by distinguishing the writer's schema from the reader's schema. The two need not be identical, only compatible; the Avro library resolves differences by matching fields by name, ignoring fields the reader does not know, and filling in default values for fields the writer did not provide. The reader must somehow know the writer's schema: in a large file, it is included once at the beginning; in a database, each record can carry a schema version number that refers to a registry. Because Avro schemas contain no tag numbers, they are friendlier to dynamically generated schemas, such as dumping the contents of a relational database whose columns change over time. Kleppmann summarizes the value of schemas: they are more compact than textual formats, they serve as valuable documentation that is guaranteed to stay current, a database of schemas lets teams check compatibility before deploying, and for statically typed languages, code can be generated from them. Modes of dataflow Finally, Kleppmann considers the ways encoded data flows between processes. Through databases, the process writing encodes data and a process reading, possibly much later, decodes it; Kleppmann notes the principle that data outlives code, since a database may contain values written years ago in older formats. A subtle hazard arises when older code reads a record containing a new field, updates it and writes it back, discarding the field it did not understand. Through services, using REST or remote procedure calls, clients and servers must agree on an API, and since servers are usually updated before clients, requests need backward compatibility and responses need forward compatibility. He criticizes RPC frameworks for pretending that a network call is like a local function call, when in fact network calls can time out, be retried, have unpredictable latency and require encoding of all parameters. Through asynchronous message passing, via a message broker, the sender does not wait for the recipient; the broker acts as a buffer, can redeliver messages to a crashed process, decouples senders from recipients, and allows one message to reach several consumers. The actor model is a related approach. The strategic lens For a manager, this chapter carries two practical lessons. First, the separation of OLTP from OLAP is not a technology fashion but a structural consequence of different access patterns; systems built to serve customers in milliseconds and systems built to scan billions of rows are optimized in opposite ways. Proposals to run heavy analytics directly on production databases deserve scrutiny. Second, encoding decisions govern the organization's ability to change. A team that chooses a format with explicit schemas and clear compatibility rules can deploy independently and roll out changes gradually. A team that relies on ad hoc JSON with no compatibility checks will eventually break a consumer it did not know existed. Key Takeaways · An index speeds reads but slows writes; choosing indexes is a deliberate trade-off based on query patterns. · LSM-trees append and merge sorted segments, favoring write throughput and compression; B-trees update fixed-size pages in place, favoring predictable reads and simple locking. · OLTP serves many small, key-based requests from users; OLAP scans large volumes for aggregates, which is why warehouses are separated from operational databases. · Column-oriented storage reads only the columns a query needs and compresses well, making it the standard for analytics. · Backward compatibility lets new code read old data; forward compatibility lets old code read new data. Both are needed during rolling upgrades. · Thrift and Protocol Buffers rely on stable field tags; Avro resolves differences between writer's and reader's schemas by field name. Review Questions 1. Explain how an LSM-tree handles a write, a read and a crash. 1. Why do B-trees need a write-ahead log? 2. Using the distinctions in Table 2, explain why a retailer should not run quarterly revenue analysis on its checkout database. 3. Why is column-oriented storage faster than row-oriented storage for analytic queries? 4. Define backward and forward compatibility and explain why rolling upgrades require both. 5. How does Avro achieve schema evolution without field tag numbers? Hashtags: #TheArchitectureOfData #DesigningDataIntensiveApplications #MartinKleppmann #DataIntensiveSystems #Reliability #Scalability #Maintainability #FaultTolerance #TailLatency #DataModels #RelationalDatabases #DocumentDatabases #GraphDatabases #SchemaOnRead #SchemaOnWrite #LSMTrees #BTrees #OLTP #OLAP #ColumnOrientedStorage #DataEncoding #Replication #Partitioning #DistributedSystems #FutureOfDataArchitecture

Download the Book (PDF): Introduction Weighing in at over a thousand pages, Ross Anderson's encyclopedic textbook is the ultimate authority on cybersecurity. It covers everything from the mathematics of cryptography to the psychology of social engineering and the physics of hardware tampering. For an IT management student, attempting to memorize this massive, highly technical volume before a final exam is overwhelming and practically impossible. This book is your strategic shortcut. It is designed specifically to explain Security Engineering, synthesizing the exhaustive textbook into focused, exam-ready modules. It isolates the core principles of threat modeling, access control and distributed systems architecture, explaining the engineering intuition without the paralyzing depth of the advanced mathematics. It cuts out the technical filler, leaving you with the operational knowledge you need to master your security modules. The book and its author Security Engineering: A Guide to Building Dependable Distributed Systems was first published by Wiley in 2001. A second edition followed in 2008 and a third in 2020. The third edition was released chapter by chapter as drafts for comment from 2019 before appearing in print, and its author later made the full text freely available online. This guide is based mainly on that third edition, which reorganizes and substantially updates the earlier material to cover smartphones, cloud services, machine learning, the cybercrime economy and the security of cars and medical devices. Where the guide refers to the book's structure or arguments, it means the third edition unless it says otherwise. Ross Anderson was one of the most influential figures in computer security. He spent most of his career at the University of Cambridge Computer Laboratory, where he became Professor of Security Engineering in 2003, and from 2021 he also held a chair at the University of Edinburgh. He was a Fellow of the Royal Society and of the Royal Academy of Engineering. His research ranged across cryptographic protocols, hardware tamper resistance, payment systems, medical privacy and information hiding, and he is widely credited with founding the field of security economics: he helped launch the Workshop on the Economics of Information Security, first held in 2002, which remains the main meeting place for that discipline. He also founded the Foundation for Information Policy Research, a think tank on technology and civil liberties, and campaigned for decades on patient privacy, surveillance and the treatment of victims of bank fraud. Anderson died in March 2024 at the age of 67. His book is now his most widely read legacy. What made the book unusual, and what still makes it valuable, is its refusal to treat security as a branch of mathematics or a product category. Anderson wrote as an engineer who had spent years investigating how real systems failed: cash machines that paid out to fraudsters, smartcards whose secrets leaked through their power consumption, hospital systems that exposed patient records to anyone with a login, and protocols that looked sound on paper but collapsed when one message was replayed. His recurring observation was that systems usually fail not because the cryptography is broken but because of poor design, misplaced trust, confused requirements, careless implementation, or incentives that reward the wrong people. The book therefore draws on computer science, economics, psychology, law and history in equal measure, and it teaches mainly through cases. The controlling idea of this guide The single idea this guide keeps in view is that security is a property of whole systems, shaped by people and incentives, and it is engineered through explicit choices about policy, mechanism, assurance and incentive rather than bought as a feature. Every chapter returns to that idea from a different direction. A cipher is only as useful as the protocol around it; a protocol only as strong as the access controls that protect its keys; access controls only as good as the humans who configure and use them; and the whole arrangement only as durable as the economic incentives that decide whether anyone will maintain it. An IT manager who grasps this can reason about a system they have never seen before, which is what examiners and employers actually test. This is also why the guide keeps the mathematics intuitive. Anderson himself makes the point that most practitioners will never design a cipher and should not try; what they need is a clear picture of what each tool promises, what it does not promise, and how it tends to go wrong in practice. The guide gives that picture in words, and it avoids formulas and code entirely. How the guide is organized Anderson divides his book into three parts. The first lays the foundations: what security engineering is, who the opponents are, how human psychology shapes security, how protocols and cryptography work, how access control is enforced, how distributed systems behave under failure and attack, and why economics explains so much of what goes wrong. The second part applies those foundations to real systems, from military classification schemes and bank bookkeeping through physical locks, smartcards, biometrics, side channels, phones, networks and electronic warfare. The third part turns to policy and management: surveillance and privacy, how secure systems are developed, how assurance is gained and maintained, and what society should do about systems that increasingly make decisions about people. This guide follows the same arc but compresses it into ten chapters. The foundations receive the most space, because they are the concepts that transfer to every other topic and they are what security modules examine most heavily. The guide opens with the framework that organizes the whole discipline and then works through the opponent, the human user, protocols and cryptography, access control, distributed systems and economics, each in its own chapter. It then gives two chapters to the applied material, grouping Anderson's many case chapters around the lessons they share: first the security policies that govern information flows in military, medical and financial systems, and then the physical, hardware and network attacks that break systems in the field. The final chapter covers the management material that closes the book: privacy and surveillance, development methods, assurance, certification and the long-term sustainability of systems that must be patched for decades. Along the way the guide uses several well-documented real incidents to show the principles at work. Some come from Anderson's own research, such as the Cambridge team's demonstration that a card payment could be completed without the correct PIN. Others, such as the Post Office Horizon scandal, the WannaCry ransomware outbreak that disrupted the National Health Service, the SolarWinds supply-chain compromise and the Log4Shell vulnerability, are used as the guide's own illustrations of the book's arguments; where an example goes beyond what Anderson wrote, the text says so. Each chapter closes with a short list of key takeaways and a set of review questions. The takeaways state the points most worth remembering; the questions are written in the style of exam essay prompts and short-answer items, and the best way to use them is to answer them aloud or on paper before checking the chapter again. A glossary at the end defines the terms that recur most often, and a short annotated reading list points to the original book and to the classic papers behind its most important ideas. A final word on method. Anderson liked to compare security engineering to medicine: a doctor learns anatomy and physiology, but becomes competent by studying many cases and learning to recognize patterns of failure. Read this guide the same way. Learn the principles, but pay close attention to the stories, because in an exam, and in practice, the ability to say why a particular system failed and what would have prevented it is worth more than any list of definitions. Chapter 1: What Security Engineering Is Security engineering, in Anderson's definition, is about building systems that remain dependable in the face of malice, error or mischance. The phrase is deliberately broad. It covers the obvious threat of a hostile attacker, but also the clerk who makes a mistake, the software update that goes wrong, and the unlucky coincidence that no designer anticipated. As a discipline, it concentrates on the tools, processes and methods needed to design, implement and test complete systems, and to adapt existing systems as their environment changes. The emphasis on complete systems is the key. A student who has taken a course in cryptography may think of security as a matter of choosing strong algorithms. A student from a networking background may think of it as firewalls and intrusion detection. Anderson's argument is that neither view is sufficient, because real failures almost always happen at the joins: between the cryptography and the protocol, between the protocol and the user, between the technical design and the organization that runs it. Security engineering is the practice of looking at those joins deliberately. The four-part framework The organizing idea of the book is a framework of four interacting elements. Anderson presents them as the things a good security engineer needs to get right, and they recur, explicitly or implicitly, in almost every chapter. Policy is what you are supposed to achieve. It states who may do what, which information must be protected, from whom, and what counts as a breach. A policy may be a formal model, a short statement of rules, or a set of legal obligations. Without a clear policy, a system cannot be said to be secure or insecure, because there is nothing to measure it against. Mechanism is what you use to achieve the policy: the ciphers, access controls, hardware tamper resistance, audit logs, locks, guards and procedures that enforce the rules. Mechanisms are where most technical effort goes, and they are what most security products sell. Assurance is the amount of reliance you can place on each mechanism. A mechanism that exists on paper but has never been tested gives little assurance; one that has been independently evaluated, formally verified or battle-tested over years gives more. Assurance answers the question of how you know the mechanisms really do what they claim, and how well they will keep doing it. Incentive is the motive that the people guarding and maintaining the system have to do their job properly, and also the motive that attackers have to try to defeat it. Anderson places incentive alongside the technical elements because experience showed him that systems frequently fail when the people responsible for security do not bear the cost of its failure. These four elements interact. A strong mechanism with a vague policy protects the wrong thing. A clear policy with weak assurance cannot be relied upon. And any of them can be undermined by incentives. Anderson notes that much of the security industry is shaped by the incentive to be seen to be doing something rather than to reduce actual risk, a pattern Bruce Schneier named "security theatre". Airport screening, password rules that force frequent changes, and elaborate compliance checklists are the classic examples: they are visible, they reassure, and they may do little to stop a determined opponent. The framework gives the student a way to ask, of any control, whether it serves the policy, whether its assurance is real, and whose incentives it serves. A useful exam habit is to analyse any case study through these four lenses in turn. What was the policy, and was it written down? Which mechanisms enforced it? What evidence existed that they worked? And who gained or lost when they failed? Almost every failure in the book can be explained by a gap in at least one of the four. Four example settings Anderson grounds the framework in four familiar settings, chosen because each has a different dominant concern. They are worth knowing well, because they show that "security" means different things in different places. A bank. Banks worry above all about the integrity of their records and the prevention of fraud. The core bookkeeping system must ensure that money cannot be created or destroyed except through authorized, logged transactions. Around it sit cash machines, card networks, online banking and mobile apps, each with its own threats. Confidentiality matters, but a bank's deepest fear is that its ledger cannot be trusted. Banks also illustrate the incentive problem: where a bank can shift the cost of fraud onto customers or merchants, it has less reason to prevent it. A military base. Here the traditional concern is confidentiality: keeping classified information from the enemy. Military systems gave rise to the classic multilevel security models, in which documents and people carry labels such as Confidential, Secret and Top Secret, and information must not flow from a higher level to a lower one. Military settings also care intensely about availability, since communications that fail in combat can be fatal, and about authenticity, since a forged order is worse than no order. A hospital. Hospitals face a sharper conflict between security goals. Patient records must be kept confidential, but they must also be available immediately to clinicians in an emergency, and they must be accurate, because an error can kill. Safety and security interact closely: a drug-dispensing system or an infusion pump must resist tampering, but it must never lock out a nurse at a critical moment. Hospitals also involve many parties with different interests, including patients, clinicians, administrators, insurers and researchers, which makes their policies more complicated than a simple hierarchy. A home. The home now contains networked devices of every kind: routers, phones, smart speakers, televisions, thermostats, cars. The householder is not a security professional, has no security staff, and often cannot update or even see the software in the devices. Threats range from remote criminals who compromise devices in bulk to intimate threats such as an abusive partner who uses shared accounts or tracking apps. The home setting shows that security has to work for ordinary people without expert help, and that many of the most important threats are not technical at all. Taken together, the four settings teach that there is no single meaning of "secure". A system is secure relative to a policy, and the policy depends on the setting, the assets and the people involved. The vocabulary of the discipline Much confusion in security arises from loose terminology, and Anderson spends time pinning down terms. The most important distinctions are these. A system can mean a product or component, a collection of components including an operating system and communications, the same plus applications, or all of that plus the people who use and run it. Anderson's view is that the last, widest meaning is usually the right one for security analysis, because many failures happen when designers draw the boundary too narrowly and leave people out. A subject is a physical person or process acting in the system; a principal is an entity that participates in a security system and can be held responsible, such as a person, a role, a piece of equipment, or a group. A person is a human being; an identity is a correspondence between the names of two principals, such as between a person and a login name. Getting these right matters: many access-control and audit failures arise because a single login is shared by several people, so the system cannot tell who actually did something. Trust and trustworthiness are different. A trusted system or component is one whose failure can break the security policy. A trustworthy one is one that will not fail. The goal of good design is to minimize what must be trusted, and to make what remains trustworthy. This distinction explains why security engineers try to keep the "trusted computing base", the set of components whose correctness the security depends on, as small as possible. Secrecy, confidentiality and privacy overlap but differ. Secrecy is the broad engineering term for mechanisms that limit who can access information. Confidentiality is an obligation to protect someone else's secrets that you hold. Privacy is the ability and right of people to protect their own personal information, extending to protection from intrusion into their personal lives. Integrity means that information has not been altered in an unauthorized way, while authenticity can mean integrity combined with freshness, or can refer to whether a message really comes from the claimed source. Availability means the system works when needed. The trio of confidentiality, integrity and availability is often taught as the "CIA triad"; Anderson treats it as a starting point rather than a complete account, since many real policies care about accountability, non-repudiation, safety and fairness as well. Finally, the book distinguishes a vulnerability (a property of the system or its environment that, with an internal or external threat, can lead to a failure), a threat (a circumstance with potential to cause harm), an error (a design flaw or deviation from intended behaviour) and a failure (the actual departure from required behaviour). Keeping these apart helps with incident analysis: a vulnerability may exist for years without failure, until a threat actor finds it. Security and dependability The book's subtitle refers to dependable distributed systems, and the choice of word is significant. Dependability is a concept from safety and reliability engineering that covers the ability of a system to deliver a service that can justifiably be trusted. Anderson treats security as one aspect of dependability, alongside reliability, safety and availability. The practical consequence is that many techniques from safety engineering, such as hazard analysis, fault trees, redundancy and careful change control, carry over into security, and that the two fields increasingly need to be managed together. The difference between them is the adversary. A safety engineer worries about random failures and honest mistakes, which follow statistical patterns. A security engineer must assume that someone is actively searching for the weakest point and will exploit it deliberately, often in ways that are rare under normal conditions. A bridge does not redesign the wind to find its weakness; an attacker does exactly that to a system. This is why security testing cannot rely only on typical use cases, and why the book keeps returning to the need to think like an opponent. Cars, medical devices and industrial control systems have made this convergence urgent. A vehicle that can be hacked remotely is a safety problem as well as a security problem, and a regulator certifying its safety must now also ask about software patching and resistance to attack. Anderson's later work, discussed in the final chapter of this guide, focused on exactly this meeting point. Why systems fail: the recurring lessons The examples in the book yield several lessons that recur so often they are worth stating at the start. First, most failures are not cryptographic. Anderson's early research on cash machine fraud found that losses came overwhelmingly from implementation errors, poor procedures and insider abuse, not from anyone breaking the encryption. That finding, published in the early 1990s as a paper on why cryptosystems fail, shaped his whole approach. Second, designers tend to protect the wrong things. It is easy to spend money guarding against dramatic, well-publicized threats while ignoring mundane ones, such as the help-desk operator who resets a password for anyone who sounds convincing. Third, security and usability are not opposites but they must be designed together. Controls that are too burdensome get bypassed, and a bypassed control is worse than none because it creates a false sense of protection. Fourth, complexity is the enemy of security. Every added feature, interface or dependency is a new place for things to go wrong, and large systems accumulate them. Fifth, environments change. A system designed for a closed network becomes exposed when connected to the internet; a threat model that was reasonable ten years ago may be dangerously out of date. Security is therefore never finished, a point the book makes emphatically in its closing chapters. These lessons are why Anderson describes security engineering as needing breadth. The practitioner needs to understand enough cryptography to avoid misusing it, enough psychology to design for real users, enough economics to see where incentives point, and enough law and management to make changes stick. From framework to practice: threat models and risk The framework tells an engineer what to think about; a threat model is how that thinking is written down for a particular system. A threat model identifies the assets worth protecting, the opponents who might want them, the ways those opponents could get at them, and the consequences if they succeed. Anderson stresses that the threat model must come before the choice of mechanisms, and that the most common design error is to solve a problem different from the one the system actually faces. A payment system designed against outside hackers, for instance, may be wide open to its own staff. A practical way to build a threat model is to work backwards from the harm. Start with the outcome that would matter most, such as money leaving an account without authority or a patient record reaching a journalist, and ask what would have to happen for it to occur. Each answer becomes a branch, and each branch can be broken down further until it reaches concrete actions an attacker could take. This structure, often called an attack tree, makes it easier to see which single controls block many branches and which branches are left entirely undefended. It also reveals the cheapest path for the attacker, which is usually the one that will be used. Risk management then asks how much to spend. Traditional approaches multiply the likelihood of an event by its impact and compare the result with the cost of controls. Anderson treats such calculations with caution. Likelihoods for deliberate attacks are hard to estimate, because an intelligent opponent changes behaviour in response to defences, and rare catastrophic events are easy to underweight. Figures produced by risk-scoring exercises can give an air of precision that the underlying data do not support. The sensible use of risk assessment is to force a structured conversation about priorities, not to generate numbers that are then treated as facts. Two further ideas complete the practical toolkit. The first is defence in depth: arranging several independent controls so that the failure of one does not expose the asset. A bank vault has walls, a door, a time lock, alarms and guards; a well-designed network has perimeter controls, internal segmentation, strong authentication, monitoring and backups. The second is resilience: accepting that some attacks will succeed and designing so that they can be detected, contained and recovered from. The shift of emphasis from pure prevention towards detection and recovery runs through the modern practice the third edition describes, and it reflects the experience of an industry that has learned that perfect prevention is not achievable. Key Takeaways · Security engineering is about building systems that stay dependable despite malice, error or mischance, and it deals with whole systems including their people. · Anderson's framework has four interacting elements: policy (what you want), mechanism (how you enforce it), assurance (how much you can rely on it) and incentive (who is motivated to make it work or to break it). · The meaning of "secure" depends on the setting; banks, military bases, hospitals and homes each prioritize different properties. · Precise vocabulary matters: trusted is not the same as trustworthy, and confidentiality, privacy and secrecy are distinct ideas. · Security is one aspect of dependability; it differs from safety chiefly because an adaptive adversary seeks out the weakest point. · Most real failures come from design, implementation, procedure and incentive problems rather than broken cryptography. Review Questions 1. Explain Anderson's four-part framework and show how a failure in incentive can defeat a strong mechanism, using an example of your choice. 1. Compare the dominant security concerns of a bank and a hospital, and explain why a single generic security policy would not serve both. 2. Distinguish between a trusted component and a trustworthy one. Why do designers try to minimize the trusted computing base? 3. What is "security theatre"? Give two examples and assess what, if anything, they contribute to real security. 4. Why does Anderson treat security as an aspect of dependability, and what makes security engineering harder than conventional safety engineering? Chapter 2: Who Is the Opponent? A security design is only as good as its picture of the adversary. Engineers who imagine a lone teenage hacker will build different defences from those who imagine a national intelligence agency, and both will be wrong if the real threat is a disgruntled employee or an organized fraud gang. Anderson devotes an early chapter of the third edition to this question, and his answer is deliberately concrete. Rather than discussing abstract "threat actors", he describes the main communities of attackers as they actually operate, grouping them into four broad classes: spies, crooks, geeks and what he calls the swamp. The purpose of this classification is practical. Each class has different goals, resources, time horizons and tolerance for risk, and each therefore attacks in different ways. Knowing which of them you face tells you what to defend and how hard. The four classes are compared in Table 1, and each is discussed in turn below. Table 1. Anderson's four classes of opponent. Class Typical actors Main goals Characteristic methods Spies State intelligence and military agencies Intelligence, sabotage, influence Bulk collection, targeted intrusion, supply-chain compromise Crooks Organized cybercriminals, fraudsters, insiders Money Phishing, malware, ransomware, card fraud Geeks Researchers, hobbyists, academics Knowledge, reputation, fixing flaws Reverse engineering, vulnerability research, disclosure The swamp Harassers, extremists, abusers, hacktivists Harm, control, attention Doxxing, stalkerware, bullying, disinformation Spies The first class consists of state actors. Anderson discusses the Five Eyes alliance of the United States, the United Kingdom, Canada, Australia and New Zealand in some detail, drawing on the documents leaked by Edward Snowden in 2013, which revealed the scale of signals intelligence collection by the US National Security Agency and Britain's GCHQ. These programs included bulk collection of communications metadata, tapping of undersea cables, the compromise of network equipment, and efforts to weaken cryptographic standards. Anderson also describes the capabilities of other states, notably China, Russia, North Korea and Iran, each with its own style and priorities: large-scale theft of intellectual property, political interference, financially motivated operations to fund a sanctioned regime, and regional sabotage. What distinguishes spies is resources and patience. A state agency can spend years on a single target, develop custom tools, buy or discover previously unknown vulnerabilities (zero-days), and compromise suppliers in order to reach their customers. The practical lesson is that no ordinary organization can expect to keep out a determined, well-resourced state attacker indefinitely. The realistic aim is to make attacks expensive, to detect intrusions quickly, and to limit what an intruder can reach. Two widely documented cases, used here as illustrations of the book's argument rather than drawn from its text, show the spy class at work. Stuxnet, discovered in 2010, was a sophisticated worm that targeted the Siemens industrial controllers running uranium enrichment centrifuges at Iran's Natanz facility. It is widely reported to have been a joint American and Israeli operation, though neither government has formally confirmed this. Stuxnet crossed an "air gap" by spreading through removable media, used several zero-day vulnerabilities, and manipulated the centrifuges while showing operators normal readings. It demonstrated that software could cause physical destruction and that isolated networks are not immune. The SolarWinds compromise, disclosed in December 2020, showed the supply-chain route. Attackers inserted malicious code into updates for SolarWinds' Orion network-management software, which was then distributed through the vendor's normal update channel. The company said that up to around 18,000 customers had installed the affected updates, although the attackers went on to exploit only a much smaller number of high-value targets, including several US government departments. In April 2021 the US and UK governments attributed the operation to Russia's foreign intelligence service, the SVR. The lesson for security engineers is that trusting a supplier's update mechanism means trusting the supplier's own security, a dependency that most organizations had never modelled. Crooks The second class is criminal, and it is by far the most common threat most organizations face. Anderson describes cybercrime as an industry with a mature supply chain. Some criminals write malware; others rent out networks of compromised machines, known as botnets; others sell stolen card data, run phishing campaigns, launder money through "money mules", or provide hosting that ignores abuse complaints. This specialization matters because it lowers the skill needed to commit crime: an attacker can buy most of the components as services. Anderson's work at the Cambridge Cybercrime Centre, which he founded to collect and share data on online crime, informs this part of the book. Its central observation is that most online crime is volume crime, run for profit by people who will attack whoever is easiest. Criminals behave economically. They go where the returns are highest and the risks lowest, and they abandon methods that stop paying. This is good news for defenders, because it means that raising the cost of attack even modestly can push criminals elsewhere, whereas a spy will often simply spend more. Ransomware has become the most damaging form of this crime. Criminals encrypt a victim's data and demand payment for the key, often also threatening to publish stolen data. The WannaCry outbreak of May 2017 is a well-documented example. WannaCry spread automatically across networks by exploiting a vulnerability in older versions of Microsoft Windows, using an exploit known as EternalBlue that had been developed by the NSA and later leaked. Microsoft had released a patch two months earlier, but many systems had not applied it. The attack spread across the world within hours and was halted largely by chance, when a researcher registered a web domain that the malware checked and that acted as a "kill switch". The US and UK governments later attributed WannaCry to North Korea, which shows that the lines between spies and crooks can blur when a state uses criminal methods for revenue or disruption. The effect on the English National Health Service is examined in Chapter 9, where it serves as a case study in patching and assurance. Here the point is that WannaCry was not a targeted attack. The NHS was hit because it ran large numbers of unpatched machines, not because anyone chose it. Most victims of crime are selected by their weakness, not their identity. Insiders form an important subgroup of crooks. Employees and contractors already hold legitimate access, understand the procedures, and know where the valuable assets lie. Anderson's early research on cash machine fraud found that insider abuse was a major source of loss. Controls against insiders, such as separation of duties, dual control, audit trails and rotation of staff, are among the oldest security mechanisms in existence, and they remain among the most effective. Geeks The third class is the researchers and enthusiasts who probe systems out of curiosity, for reputation, or to improve security. Anderson counts himself and his students among them. Geeks reverse engineer products, find vulnerabilities and publish their findings. They are the reason many flaws come to light at all. The relationship between geeks and vendors has long been tense. Firms often prefer that flaws stay hidden, while researchers argue that public disclosure is the only reliable way to force fixes. The now-standard compromise is coordinated (or responsible) disclosure: the researcher reports the flaw privately, the vendor has a fixed period to fix it, and details are published afterwards. Bug bounty programs, in which firms pay for vulnerability reports, have made much of this relationship commercial. Anderson's own group was repeatedly involved in disputes over publication, including an attempt by the UK banking industry to suppress a student's thesis on payment system weaknesses in 2010, which the university refused. Geeks matter to the defender in two ways. They are a source of warnings that should be taken seriously, and the tools and techniques they publish are soon picked up by crooks. A vulnerability announced on Tuesday may be exploited at scale by Friday. The swamp The fourth class is the most varied. Anderson uses the term "the swamp" for a range of actors whose aims are neither espionage nor straightforward profit: hacktivists who attack organizations for political reasons, extremists who use online platforms to recruit and coordinate, people who distribute child abuse material, cyberbullies, and those who harass and stalk. Many of these harms are carried out with little technical skill. A stalker may need nothing more than access to a partner's phone or a shared cloud account. The swamp is important for two reasons. First, it shows that the most serious harms to individuals often come from people who know them. Intimate partner abuse using location tracking, stalkerware, smart home devices and shared accounts is a growing area of research, and it defeats threat models that assume the attacker is a stranger without physical access. Second, the swamp creates pressure for content moderation and surveillance, which in turn raises the privacy and civil liberties questions discussed later in the book. Anderson was consistently wary of proposals, such as mandated scanning of private messages, that justify broad surveillance by reference to these harms. How attacks unfold Whatever the class of attacker, serious intrusions tend to follow a recognizable sequence, and understanding it helps defenders decide where to place controls. The sequence is not Anderson's own formulation; it is the common industry model, often called the kill chain and elaborated in public catalogues of attacker techniques such as MITRE ATT&CK. It fits the cases the book describes closely. The attacker begins with reconnaissance: learning about the target's staff, suppliers, technology and exposed services. Much of this uses public information, such as company websites, job advertisements that reveal which software a firm uses, and professional networking profiles that identify who has administrator access. Next comes initial access, most often through a phishing email, stolen or guessed credentials, an unpatched internet-facing system, or a compromised supplier. Once inside, the attacker seeks persistence, installing a way back in that survives reboots and password changes, and then privilege escalation, moving from an ordinary user account to one with administrative rights. Lateral movement follows, as the attacker spreads from the first machine to others, looking for the valuable data or the systems that control them. Finally comes action on objectives: copying data out, encrypting files for ransom, manipulating a process, or simply waiting and watching. This sequence carries three lessons. First, every stage is an opportunity for detection. An attacker who gets in through phishing can still be caught when they try to escalate privileges or move sideways, provided someone is watching the logs. Second, the time an attacker spends inside a network before discovery, often called dwell time, is a major driver of damage; the SolarWinds intruders were inside some victims for months. Third, controls that block several stages at once are especially valuable. Strong multi-factor authentication hinders initial access and lateral movement; network segmentation limits lateral movement and the reach of ransomware; good logging supports detection at every stage. Measuring crime and its costs A recurring theme of Anderson's research is that discussions of cybercrime suffer from bad numbers. Estimates of global losses vary by orders of magnitude, and many come from firms that sell security products and have an interest in large figures. Anderson and colleagues published a study in 2012, updated in 2019, that tried to measure the costs of cybercrime more carefully by separating the direct losses from criminal acts, the indirect losses such as lost confidence and reduced online commerce, and the cost of defence. Their most striking finding was that for many types of online crime the indirect costs and defence costs far exceed the money criminals actually take. A spam campaign may earn its operators a modest sum while imposing large costs on everyone who has to filter, read and delete the messages. This asymmetry has a policy implication that Anderson pressed repeatedly: it can be far more efficient to spend money on catching and prosecuting the small number of people who run the major criminal operations than on ever-larger defensive products for the millions of potential victims. A second measurement problem is under-reporting. Many victims do not report fraud, firms often prefer not to disclose breaches, and police forces have historically lacked the skills and incentives to investigate. Crime that is not measured tends not to be policed. In England and Wales, the official Crime Survey began to include fraud and computer misuse only in the mid-2010s, and once it did, these offences turned out to account for a large share of all the crime that individuals experienced. Using the classification The four classes are not rigid. States hire criminals; criminals use techniques published by researchers; hacktivists may be fronts for state agencies. But the classification disciplines threat modeling in three ways. First, it forces a designer to ask who would realistically attack this system and why. A small retailer's website faces crooks almost exclusively. A defence contractor faces spies. A dating app faces the swamp. A hospital faces all of them. Second, it links the attacker to their economics. Crooks respond to cost and will move on; spies respond to value and will persist; the swamp often attacks individuals through personal relationships that no technical perimeter can separate. Defences should be designed accordingly. Third, it counters the tendency to design for the attacker who is most interesting rather than the one who is most likely. Media coverage favours dramatic state operations, but the losses most organizations actually suffer come from ordinary fraud, phishing and ransomware, often enabled by missing patches and weak authentication. A final point from the chapter is that the opponent is not always outside the organization or even hostile. Many breaches begin with a well-meaning employee who clicks a link, misconfigures a cloud storage bucket, or emails a spreadsheet to the wrong address. Good threat models include error alongside malice, which is why the next chapter turns to the psychology of the people who use systems. Key Takeaways · Anderson groups attackers into spies, crooks, geeks and the swamp, each with different goals, resources and methods. · State actors are patient and well funded; the realistic defence is to raise costs, detect intrusions quickly and limit the damage, as Stuxnet and SolarWinds illustrate. · Most organizations mainly face criminals, who run an industrialized, specialized economy and choose victims by weakness rather than identity. · Researchers expose flaws and publish tools; coordinated disclosure balances the interests of vendors and the public. · The swamp, including stalkers and abusive partners, shows that many serious harms come from people close to the victim and defeat perimeter-based threat models. · A good threat model names the realistic opponent, reflects that opponent's economics, and includes honest error as well as malice. Review Questions 1. Outline Anderson's four classes of opponent and explain how the likely attacker should change the design of a hospital information system compared with an online shop. 1. Why is a financially motivated criminal easier to deter than a state intelligence agency? Refer to the economics of each. 2. Using the SolarWinds case, explain what a supply-chain attack is and why it is hard to defend against. 3. What is coordinated disclosure, and what tensions does it try to resolve between researchers and vendors? 4. Why do intimate threats, such as a controlling partner, undermine conventional threat models? Suggest two design choices that would help. Chapter 3: The Human Element: Psychology and Usability For decades, security professionals described users as the weakest link and treated them as a problem to be managed. Anderson's third edition takes a different view. It argues that systems fail when they are designed without regard for how people actually think and behave, and that blaming users for predictable behaviour is a design failure in disguise. The chapter on psychology and usability draws on cognitive psychology, behavioural economics and a large body of usable-security research, much of it produced after the book's first edition, to explain why people are deceived, why they choose weak passwords, and what designers can do about it. The central lesson is simple to state and hard to apply: security mechanisms must fit the capabilities, goals and context of the people who use them. A control that users cannot understand or that obstructs their main task will be misused or bypassed, and the designer, not the user, should carry the responsibility. How minds get fooled Anderson summarizes several strands of psychology that explain why deception works. The first is the study of cognitive biases and heuristics, associated with Daniel Kahneman and Amos Tversky. People rely on quick, intuitive judgements that work well most of the time but fail in predictable ways. We judge risks by how easily examples come to mind, so dramatic, well-publicized threats seem more likely than mundane ones. We are more strongly moved by possible losses than by equivalent gains. We tend to be over-optimistic about our own chances and to discount future costs against present convenience. Each of these tendencies can be exploited by an attacker and must be allowed for by a designer. The second strand concerns social psychology, especially obedience to authority and conformity. Classic experiments showed how readily people follow instructions from someone who appears to be in charge. Fraudsters exploit this constantly, posing as bank officials, police officers, IT support staff or senior executives. The same research shows the power of social proof: people do what they see others doing, which is why fake reviews and fabricated urgency work. The third strand is the psychology of persuasion, including the well-known principles described by the psychologist Robert Cialdini: reciprocity, commitment and consistency, social proof, liking, authority and scarcity. Almost every scam uses several of them together. A fraudulent message that claims to come from the bank (authority), warns of an account being closed today (scarcity and urgency), and asks the victim to confirm details that "other customers have already updated" (social proof) is combining these levers deliberately. Anderson also discusses the psychology of the victim's situation. People are more easily deceived when they are busy, tired, stressed or distracted, and attackers time their approaches accordingly. Work email is read quickly between other tasks, and a convincing message arriving at the end of a long day may get only a second's thought. Research on why people fall for scams finds that intelligence and education offer surprisingly little protection; what matters more is whether the context triggers careful thought. Phishing and social engineering Social engineering is the manipulation of people into performing actions or divulging information. It is older than computers; confidence tricksters and "pretexters" who telephone an office pretending to be someone else have operated for as long as there have been offices. Phishing is its large-scale online form: messages designed to trick recipients into revealing credentials, paying money or installing malware. Anderson explains why phishing is so effective and so hard to stop by technical means alone. Email was designed without strong authentication of the sender, so forging a plausible sender address was long trivial; later standards for checking the sending domain helped but did not eliminate the problem, because attackers simply register look-alike domains. Web addresses are hard for ordinary people to read correctly, and the visual cues that browsers provide, such as padlock icons, have been widely misunderstood. A padlock shows that the connection is encrypted, not that the site is honest, and phishing sites routinely obtain valid certificates. The more dangerous variants are targeted. Spear phishing aims at particular individuals using personal details gathered from social media or earlier breaches. Business email compromise involves criminals impersonating a senior executive or a supplier to persuade finance staff to make payments to a fraudulent account. These attacks require little technical skill and have caused very large losses, because they exploit the ordinary business processes by which organizations move money. Anderson's analysis leads to several design principles. First, training users to spot phishing has limited value on its own. It can raise awareness, but a well-crafted message will still fool a proportion of recipients, and in a large organization that proportion is enough. Phishing simulations that punish staff who click can breed resentment and discourage reporting. Second, systems should be designed so that a single deceived user cannot cause a catastrophe. Payment changes should be confirmed through a separate channel; administrative access should need more than a password that can be typed into a fake page. Third, the most effective technical countermeasures are those that remove the user's judgement from the loop. Authentication methods bound to the genuine website, such as hardware security keys and the newer passkeys, will not release credentials to a look-alike domain however convincing it appears, because the check is made by the device rather than the person. A related category is authorized push payment fraud, where the victim is tricked into sending money themselves, for example by a caller claiming that their account is compromised and that funds must be moved to a "safe account". Because the victim authorizes the transfer, banks long argued that they were not liable. This became a significant policy issue in the United Kingdom, and it illustrates the incentive argument of the book: when banks bear the loss, they invest in prevention; when customers do, they tend not to. Passwords Passwords receive extended treatment in the book because they illustrate so clearly the collision between security theory and human capability. A password is supposed to be something only the user knows. Security theory wants it to be long, random, unique to each site and never written down. Human memory cannot meet these requirements for dozens or hundreds of accounts. The result is predictable: people choose memorable passwords, reuse them across sites, make small variations when forced to change them, and write them down. Research by Anne Adams and Angela Sasse in the late 1990s, which Anderson discusses, found that users were not careless by nature but were responding rationally to demands that conflicted with their work, and that security departments had failed to explain the reasons for the rules. Anderson examines several specific problems. Guessability. Large leaks of password databases have shown that people choose from a small set of common patterns: names, dates, dictionary words with digits appended, and keyboard sequences. An attacker who tries the most common few thousand choices will break a substantial fraction of accounts. Composition rules, such as requiring a digit and a symbol, tend to produce predictable variations rather than real strength. Reuse. When one site is breached, the leaked email and password pairs are tried automatically against other sites, an attack known as credential stuffing. Reuse turns a breach at a minor website into a compromise of a person's email or bank account. Forced changes. Requiring users to change passwords every few months was long standard practice. Research showed that it leads to weaker, more predictable passwords, and official guidance changed accordingly. The US National Institute of Standards and Technology revised its digital identity guidelines in 2017 to advise against arbitrary periodic changes, and Britain's National Cyber Security Centre gave similar advice. Recovery. The weakest part of many password systems is the process for resetting a forgotten password. Security questions ask for facts, such as a mother's maiden name or a first school, that are often public or guessable. Help desks can be talked into resets. Email-based recovery makes the email account the master key to everything else. Storage. On the system side, passwords must be stored in a form that is useless if stolen. The standard technique is to store not the password but the result of a deliberately slow one-way function applied to it, with a random value called a salt so that identical passwords produce different stored values. Chapter 4 explains the idea of one-way functions intuitively; the point here is that many breaches have exposed passwords stored carelessly, sometimes in plain text. The design response Anderson and the wider research community favour is to reduce reliance on memorized secrets. Password managers let users have long, unique passwords without remembering them. Multi-factor authentication adds a second element, such as a code from a phone or a hardware token, so that a stolen password is not enough on its own. Rate-limiting and monitoring make online guessing impractical. And risk-based approaches let systems ask for extra checks only when something looks unusual, reducing friction for normal use. The help desk and the recovery channel A pattern Anderson returns to throughout the book is that attackers go around strong mechanisms rather than through them, and the human-staffed recovery channel is the most common way around. Every system needs a way to help a legitimate user who has lost a password, a phone or a card. Whoever runs that process is under pressure to be helpful and quick, and has limited means of checking identity. An attacker who can sound plausible on the telephone therefore faces a much easier task than one who must break the authentication itself. The SIM swap attack shows the pattern clearly. Many online services send one-time codes by text message as a second factor or as a way to reset a password. A criminal who has gathered some personal details about a victim calls the victim's mobile operator, claims to have lost their phone, and asks for the number to be moved to a new SIM card in the criminal's possession. If the operator's staff are persuaded, every text message, including the security codes, now goes to the criminal, who can reset the victim's email and bank passwords within minutes. The strong mechanism, a second factor, has been defeated by attacking a different organization's customer-service process that its designers did not control. Regulators and operators have responded with extra checks, and security guidance now treats text-message codes as weaker than app-based or hardware factors, but the underlying lesson is general: the security of an account is the security of its weakest recovery path. The same logic applies inside organizations. Attackers who breached several large companies in recent years did so by telephoning the internal IT help desk, impersonating an employee, and asking for a password reset or a new multi-factor enrolment. The defence is procedural as much as technical: help-desk staff need a verification method that does not depend on facts an attacker can find, the authority to refuse without penalty, and a process that makes high-risk changes, such as re-enrolling an administrator's authenticator, subject to a second approval or a delay. Security culture in organizations Individual psychology operates within an organizational setting, and Anderson's broader argument about incentives applies to staff as much as to firms. Employees respond to what they are measured and rewarded on. A call-centre worker judged on the number of calls handled per hour has every reason to skip slow identity checks. A developer judged on features shipped has little reason to spend time on security testing. Unless security is built into the way work is measured, it will be sacrificed whenever it conflicts with the targets people are actually held to. Research on organizational security, including the work of Sasse and her colleagues, describes the "compliance budget": the limited amount of effort that staff are willing to spend on security tasks that do not help their main work. Once that budget is exhausted, further rules are ignored. Security managers should therefore spend the budget wisely, requiring effort only where it buys real protection and removing rules that exist for show. A healthy security culture also depends on how mistakes are treated. In safety-critical industries such as aviation, the practice of reporting errors without blame has greatly improved learning, because people report near-misses that would otherwise stay hidden. Security teams that punish staff who click on phishing links discourage the prompt reporting that allows an attack to be contained. The most useful response to a suspicious email is often a fast report from the person who received it, and that only happens if reporting is easy and safe. Designing for real people Beyond deception and passwords, the chapter sets out principles for usable security that apply across all systems. Make the secure way the easy way. Defaults matter enormously, because most people never change them. A system that ships with encryption turned on, updates installed automatically and administrator accounts disabled will be more secure in practice than one with better features that must be configured. Align security with the user's primary task. People use systems to get work done, not to be secure. Warnings that interrupt the task are dismissed; research on browser warnings has shown that people learn to click through them when most turn out to be false alarms. Warnings should be rare, specific and actionable. Avoid blaming the user. When a system can be defeated by a predictable human error, the fault lies in the design. This matters for liability as well as engineering. Anderson spent years challenging banks that told fraud victims they must have been negligent with their PINs, when the evidence often pointed to system weaknesses. Organizations that blame users also learn less from incidents, because people hide mistakes rather than report them. Consider the full range of users. Systems must work for older people, people with disabilities, people with limited literacy or technical confidence, and people under stress. Security that works only for young technical users excludes large parts of the population and pushes them towards less safe alternatives, such as asking a relative to manage their accounts. Remember that the attacker adapts. Any cue that users are taught to rely on, such as a padlock icon or a personal greeting in an email, will be copied by attackers once it becomes common. Defences that depend on the user noticing something will degrade over time; defences that are built into the system do not. The larger point of the chapter is that psychology is not an optional extra for security engineers. The same knowledge that lets marketers persuade people to buy things lets criminals persuade them to give away money, and a designer who ignores it will build systems that fail in the field no matter how sound their technical core. Key Takeaways · Systems fail when they ignore how people think; predictable user behaviour is a design constraint, not an excuse for blame. · Cognitive biases, obedience to authority and the standard levers of persuasion explain why social engineering and phishing work. · Training alone cannot stop phishing; design so that one deceived user cannot cause catastrophe, and prefer controls such as phishing-resistant authentication that do not rely on human judgement. · Password problems stem from impossible demands on memory; guessable choices, reuse, forced changes and weak recovery are the main failure points. · Password managers, multi-factor authentication, rate-limiting and sensible defaults reduce dependence on memorized secrets. · Make the secure option the default and the easiest path, keep warnings rare and meaningful, and design for the full range of users. Review Questions 1. Using two principles of persuasion, analyse how a business email compromise attack manipulates its victim, and propose process controls that would defeat it. 1. Why did official guidance move away from forcing regular password changes? What does this reveal about the relationship between security rules and user behaviour? 2. Explain credential stuffing and describe two measures a website operator can take against it. 3. Why is a padlock icon a weak basis for teaching users to avoid phishing sites? 4. "The user is the weakest link." Critically evaluate this claim with reference to Anderson's view of usability. Hashtags: #TheArchitectureOfDefense #SecurityEngineering #RossAnderson #DependableDistributedSystems #WholeSystemSecurity #SecurityPolicy #SecurityMechanisms #SecurityAssurance #SecurityIncentives #ThreatModeling #AttackTrees #DefenseInDepth #CyberResilience #AccessControl #TrustedComputingBase #SecurityEconomics #HumanFactorsSecurity #SocialEngineering #PhishingResistance #MultiFactorAuthentication #UsableSecurity #DistributedSystemsSecurity #SecurityAndDependability #AdversarialThinking #FutureOfSecurityEngineering

Download the Book (PDF): Introduction Conway's Law holds that organizations design systems that mirror their own communication structures. For students of IT and software management, this means organizational design is just as critical as software architecture. Matthew Skelton and Manuel Pais provide the definitive model for this relationship, but their frameworks are dense. Cognitive load theory, four team types and a precise vocabulary of interaction modes can easily overwhelm a student trying to build a coherent set of study notes. This companion provides the necessary structural clarity. It is written expressly to explain Team Topologies by Matthew Skelton and Manuel Pais, arranging their operational models into an accessible academic curriculum. It defines the four fundamental team types (stream-aligned, enabling, complicated-subsystem and platform) and the three core interaction modes (collaboration, X-as-a-Service and facilitating). With clear chapter summaries and review questions, it prepares you to write analytical essays on modern socio-technical systems. The book and its authors Team Topologies: Organizing Business and Technology Teams for Fast Flow was published by IT Revolution Press in 2019. Its authors came to the subject from practice rather than from the academy. Matthew Skelton, based in the United Kingdom, had worked for years on software delivery, operations and what the industry had begun to call DevOps. Manuel Pais, a Portuguese consultant and trainer, had a similar background in continuous delivery and the working relationships between development and operations. Before the book, Skelton had built an online catalogue of "DevOps Topologies": a set of recurring patterns and anti-patterns for how development and operations teams could be arranged. The book grew out of that catalogue, and out of the authors' observation that organizations adopting modern tools and practices kept stalling for reasons that had little to do with the tools. The book arrived at a good moment. By the late 2010s, many large organizations had adopted agile methods, cloud infrastructure and microservice architectures, yet they still found delivery slow and painful. The research programme later published as Accelerate (Nicole Forsgren, Jez Humble and Gene Kim, 2018) had shown statistically that loosely coupled architectures, and teams able to change and release their systems without heavy coordination, were strong predictors of software delivery performance. What organizations lacked was a practical way to design for that loose coupling on the human side. Team Topologies offered one: a small, memorable vocabulary for describing teams and the way they relate, grounded in a few well-established ideas about communication and human cognition. The book quickly became a standard reference in software engineering management. Its terms, especially "stream-aligned team", "platform team" and "cognitive load", now appear in job descriptions, conference talks and internal strategy documents at organizations of every size. The authors founded a company around the approach, and in September 2025 IT Revolution published a second edition. That edition keeps the core model unchanged. The four team types and three interaction modes stand as they were. It adds a reflective foreword, an appendix of case studies from organizations around the world and an afterword on what the authors learned from the community that had adopted their ideas. The second edition also carries a slightly revised subtitle, Organizing Business and Technology for Fast Flow of Value, which signals where the authors have moved their emphasis: from software delivery alone towards value delivered to customers. Why the book matters for students The central claim of Team Topologies is simple to state and difficult to act on: the structure of teams, and the way those teams interact, shapes the software an organization can build and how quickly it can change it. Most management writing treats organizational design as a question of reporting lines, spans of control and budgets. Most software architecture writing treats system design as a technical question of modules, interfaces and data. Skelton and Pais argue that these are the same question approached from two sides, and that treating them separately produces systems nobody intended. For a student, the book is valuable because it joins these two conversations. It draws on an old idea, Melvin Conway's 1968 observation about committees and the systems they design. It draws on a body of research from educational psychology, John Sweller's cognitive load theory. It draws on anthropology, Robin Dunbar's work on the limits of human social groups. And it gives all of this operational form. Essays on the book therefore reward careful attention to how the pieces connect. A student who can explain why cognitive load limits the size of a team's responsibilities, why that limit in turn shapes where software boundaries should fall, and why the interaction modes exist to keep those boundaries honest, understands the book. A student who can only list the four team types does not. The controlling idea of this guide This guide is organized around a single argument, which it takes to be the heart of Team Topologies: fast flow of change is an organizational design problem before it is a technical one, and the team, limited by what its members can hold in their heads, is the unit around which both the organization and its software should be designed. Everything else in the book follows from this. Conway's Law explains why team design and system design cannot be separated. Cognitive load explains how large a team's responsibilities can safely be. The four team types describe the few distinct jobs teams need to do in an organization built for flow. The three interaction modes describe the few distinct ways those teams should relate. Fracture planes describe where to draw boundaries. Organizational sensing explains why the whole design must keep changing. Read this way, the book is not a catalogue of labels to be stuck onto an existing org chart. It is a method for designing, and repeatedly redesigning, a socio-technical system. The guide returns to this point often, because the most common misreading of Team Topologies, in practice and in student essays alike, is to treat it as a static taxonomy. How the guide is organized The guide follows the logic of the argument rather than reproducing the book page by page, although it tracks the book's own progression closely. It opens with the problem: why conventional org charts mislead, and what the older tradition of socio-technical systems theory had already discovered about the fit between work, technology and social structure. It then turns to the forces that make team design consequential. Conway's Law, and the deliberate strategy known as the reverse Conway maneuver, are examined first. Then come the team-first principles of size, stability and ownership. Cognitive load is treated at length, because it is the constraint from which most of the book's practical guidance is derived. With those foundations in place, the guide sets out the model itself. The four fundamental team types each receive careful definition, and platforms, which are the most frequently misunderstood of them, receive extended treatment, including the idea of the thinnest viable platform. The guide then examines fracture planes, the natural seams along which a system and its teams can be divided, before turning to the three interaction modes and the way interactions should change as a piece of work moves from discovery to routine. The final chapter deals with evolution and practice: how organizations sense when their design needs to change, the static topologies and anti-patterns the authors warn against, the caution offered by the widely copied "Spotify model", and the developments that have followed the first edition, including Team Interaction Modeling and the second edition's new material. Where the guide reports what Skelton and Pais argue, it says so. Where it adds context, such as later research, publicly documented company examples or criticism of the model, it marks that material as its own. Each chapter closes with key takeaways and review questions designed to test understanding rather than recall. A glossary, an annotated list of further reading and a set of notes on sources complete the volume. Chapter 1: The Problem with Org Charts Every sizeable organization has an org chart. It shows who reports to whom, which departments exist and where budget authority sits. Team Topologies opens by questioning how much this familiar document can tell us about how work actually gets done. Skelton and Pais argue that it tells us surprisingly little, and that organizations which design themselves by rearranging boxes on a chart tend to damage the very flow of work they are trying to improve. This chapter explains that argument, places it within the older tradition of socio-technical systems thinking, and introduces the idea of fast flow that runs through the whole book. What an org chart shows and what it hides An org chart is a picture of formal authority. It is useful for questions of accountability: who approves spending, who conducts performance reviews, who can hire. It is much less useful for questions of communication and flow. Work in a software organization does not travel up and down reporting lines. A customer request, a defect report or a new product idea moves sideways across the organization, passing between product managers, designers, developers, testers, security specialists, operations engineers and support staff. Many of these people report to different managers. The path a piece of work follows is therefore largely invisible on the chart. Skelton and Pais make a stronger point than this. If people communicate only along the lines an org chart shows, the organization will be slow, because every cross-departmental question has to travel up one hierarchy and down another. In practice, healthy organizations work in spite of their charts. People build informal networks and talk directly to colleagues in other departments. Management literature has long recognized this gap. The authors draw on the work of the organizational consultant Niels Pflaeging, who distinguishes between an organization's formal structure, which exists mainly for compliance and legal purposes, its informal structure of personal influence, and its value creation structure, which is how work is actually accomplished through the interplay of teams. The org chart captures only the first of these. The consequence is that decisions based on the chart alone tend to optimize locally. A manager responsible for a testing department may make testing more efficient by batching requests, standardizing forms and prioritizing a queue. Within the box on the chart, that is a sensible improvement. Across the organization, it may lengthen the time every change takes to reach customers, because work now waits longer between development and release. The chart gives no hint of this, because the delay lives in the connections between boxes, not in the boxes themselves. A related problem is that org charts encourage thinking about individuals rather than teams. Each box represents a manager and the people who report to that manager. Whether those people actually work together as a team, share goals and depend on one another, is not shown. Skelton and Pais argue that the team, not the individual and not the department, is the most effective unit for delivering and running software. An org chart, by its nature, obscures teams. Restructuring as a recurring disappointment Many organizations respond to slow delivery by reorganizing. Departments merge, split or change reporting lines. New leaders arrive and draw new charts. Team Topologies is sceptical of this habit. The authors observe that reorganizations are often undertaken for reasons unrelated to the flow of work: cost cutting, the preferences of a new executive or a desire to be seen to act. Such reorganizations tend to disrupt the working relationships and shared understanding that made existing teams effective, while leaving the underlying causes of slowness untouched. The book's alternative is not to stop changing structure. Skelton and Pais are clear that organizations should evolve their team design continually. Their point is that structural change should be driven by an understanding of how work flows and how software is built, and that it should treat teams as durable units whose members stay together long enough to become effective. A reorganization that reshuffles individuals across teams every few months throws away the most valuable asset a team has: its members' accumulated trust and knowledge of one another and of their systems. This guide adds a practical observation. Students writing about organizational change often assume that a structural change is a single event with a before and an after. Team Topologies treats structure more like software: something released in small increments, observed in use and adjusted. That shift in framing is one of the book's most important contributions, and later chapters return to it. The socio-technical tradition Although Team Topologies is written for technology leaders of the twenty-first century, its central insight has a long pedigree. The authors acknowledge the tradition of socio-technical systems thinking, and a student essay will be stronger for understanding where that tradition came from. In the years after the Second World War, researchers at the Tavistock Institute of Human Relations in London studied the British coal industry. Eric Trist and Ken Bamforth published a now classic paper in 1951 on the social and psychological consequences of the longwall method of coal mining. Before mechanization, miners had worked in small, self-organizing groups. Each group handled the whole cycle of cutting, loading and securing the coal face. Members were skilled in several tasks and responsible to one another. The mechanized longwall method broke this pattern. It introduced a long face worked by specialized shifts, each performing one fragment of the cycle, with the work of one shift depending on the completion of the previous one. The technology was more advanced, but productivity disappointed, absenteeism rose and conflict between shifts grew. Trist and Bamforth concluded that the technical system and the social system could not be designed independently. A technology that fragmented work also fragmented responsibility and relationships, and the losses on the social side cancelled the gains on the technical side. Later studies in the same programme found that some pits had arrived at a different arrangement, sometimes called composite working. There, multi-skilled groups took collective responsibility for the whole cycle while using the new equipment. These groups performed better. The lesson, developed by Trist, Fred Emery and others over the following decades, was that organizations are socio-technical systems. Their performance depends on the joint optimization of technology and social structure. In 1976 Albert Cherns set out a list of principles of socio-technical design. Among them were the ideas that work should be specified only as far as necessary, that boundaries between units should not be drawn in ways that impede the sharing of information and learning, and that design is never finished. The echoes in Team Topologies are strong. A stream-aligned team that owns a piece of software from idea to operation resembles the composite work group that owned the whole mining cycle. A delivery process that passes work from a development team to a testing team to an operations team resembles the fragmented shifts of the longwall. And the idea that organizational design must evolve continually echoes Cherns's principle of incompletion. Skelton and Pais did not invent the socio-technical perspective. What they did was translate it into a concrete vocabulary suited to modern software delivery, where the technology in question is not a coal-cutting machine but a changing, interconnected set of services. Fast flow as the organizing goal The goal that ties the book together is what the authors call fast flow: the ability to deliver a steady stream of changes to customers rapidly, safely and sustainably. The idea draws on lean thinking and on the DevOps movement, both of which emphasize the end-to-end movement of work from request to value rather than the efficiency of individual steps. Flow matters because software products are never finished. Customer needs change, competitors act, regulations shift and security threats appear. An organization's advantage depends less on any single release than on how quickly and safely it can respond. The Accelerate research, which Skelton and Pais draw on, measured delivery performance through indicators such as deployment frequency, lead time for changes, time to restore service and change failure rate. It found that high performers did well on speed and stability together, rather than trading one for the other. It also found that architectures allowing teams to test and deploy independently were among the strongest predictors of that performance. Fast flow places particular demands on organizational design. Work flows fastest when it does not have to wait for other teams. It needs few handoffs, the team doing the work has the skills and authority to complete it, and the people involved understand the system well enough to change it confidently. Each of these conditions is organizational as much as technical. Handoffs arise where team boundaries fall. Skills and authority depend on how teams are composed. Understanding depends on how much a team is asked to hold in its head. Team Topologies is, at bottom, a set of design rules for meeting these conditions. The book also insists that flow is not the same as speed at any cost. A team that delivers quickly but leaves behind an unmaintainable system, or that burns out its members, is not achieving fast flow in the authors' sense. Sustainability is part of the definition, which is one reason the book pays such close attention to the limits of human attention and to the stability of teams. An illustration: the path of a single change A simple, hypothetical example makes the argument concrete. Imagine a mid-sized retailer whose online shop is run by a technology department organized along functional lines. There is a front-end development group, a back-end development group, a database administration team, a quality assurance team, an infrastructure team and a service desk. Each group has its own manager and its own queue of work. The org chart is tidy, and each manager can show that the group is busy and productive. Now suppose the marketing department wants customers to be able to save items to a wish list. The request starts with a product manager, who writes a specification and passes it to the front-end group. The front-end developers need a new back-end service, so they raise a ticket with the back-end group, which is working on other priorities and schedules the change for its next cycle. The back-end developers need a new database table, so they raise a ticket with the database administrators. When the code is written, it waits for a testing slot with the quality assurance team. When testing passes, it waits for a release window managed by the infrastructure team. After release, a customer reports a bug to the service desk, which routes the report back through the same chain. No single group in this story is doing anything wrong. Each is responding sensibly to its own queue. Yet the elapsed time from request to working feature may be many weeks, even though the hands-on effort was a few days. Most of the time was spent waiting at the handoffs between groups. The people who understand the wish list feature best, the developers who wrote it, may never hear about the bug report directly. Nobody owns the feature from end to end. The org chart gives no warning of any of this. It shows six well-run groups. What it does not show is that a customer-facing change must cross five team boundaries before it reaches a customer, and that every boundary is a queue. Skelton and Pais would ask a different question from the one the chart invites. Instead of asking how each group can become more efficient, they would ask how teams could be arranged so that a change like this can be made by one team with minimal dependence on others. That question leads directly to the idea of the stream-aligned team, examined later in this guide. It also explains why the book pays so much attention to the interfaces between teams. Wherever a boundary cannot be avoided, the way the teams on either side interact determines whether the boundary becomes a queue. The example also shows why technology leaders cannot hand organizational design to a separate human resources function and treat it as none of their business. The decision to create a separate database administration team, for instance, is a decision about where software boundaries will fall and how changes to data will be made. Skelton and Pais argue that whoever decides team structure is, in effect, making architectural decisions, whether or not they realize it. Organizational design therefore needs technical insight, just as software architecture needs an understanding of the teams who will build and run it. The example points to a funding issue as well. Many organizations finance technology work as a series of projects, each with a start date, an end date and a budget. People are gathered for the project and dispersed when it ends. This model fits the org chart well, because it treats teams as temporary assemblies of individuals drawn from functional departments. It fits the idea of fast flow badly, because the software a project creates lives on after the project ends, and nobody is left who owns it. Skelton and Pais favour persistent teams that own software over its whole life, which in turn implies funding teams or product areas over time rather than funding projects. The book does not develop a full theory of technology funding, but it makes clear that organizational design and financial practice cannot be separated for long. From problem to model Having established the problem, the book builds its model in stages, and this guide follows the same path. The first stage explains why team design has such influence over system design, which is the subject of Conway's Law. The second establishes the team as the basic unit and examines what makes a team effective. The third introduces cognitive load as the constraint that limits what any team can own. Only then does the book introduce its four team types and three interaction modes, which are best understood as the design choices that follow once those constraints are taken seriously. For students, the key analytical move is to see the model as a response to a diagnosis. The org chart fails because it describes authority rather than flow, and individuals rather than teams. The socio-technical tradition explains why that failure matters: social and technical structures shape each other, and designing one while ignoring the other produces poor results. Team Topologies proposes to design both together, using the team as the unit and flow as the measure. Key Takeaways · Skelton and Pais argue that org charts describe formal authority but not how work flows, so decisions made from the chart alone tend to optimize locally at the expense of the whole. · Drawing on Niels Pflaeging, the book distinguishes formal structure from informal influence and from the value creation structure through which work is actually done. · The authors are sceptical of frequent reorganizations that reshuffle individuals, and favour deliberate, incremental evolution of durable teams. · The book's perspective continues the socio-technical systems tradition begun by Trist and Bamforth's 1951 study of coal mining, which showed that technical and social systems must be designed together. · Fast flow, meaning rapid, safe and sustainable delivery of change, is the goal against which the book judges every design choice. Review Questions 1. Why does an org chart give a poor picture of how work moves through a software organization? Illustrate your answer with an example of local optimization. 1. What did Trist and Bamforth observe about the longwall method, and how does their finding anticipate the argument of Team Topologies? 2. Explain the distinction between formal structure and value creation structure, and say which one the book asks leaders to design for. 3. In what sense is fast flow an organizational problem as well as a technical one? 4. Why are Skelton and Pais wary of reorganizations that frequently move individuals between teams? Chapter 2: Conway's Law and the Reverse Conway Maneuver If one idea underpins Team Topologies, it is Conway's Law. Skelton and Pais treat it as the basic reason why organizational design and software architecture cannot be separated. This chapter traces the law from its origin in 1968, examines the evidence for it, and explains the deliberate strategy the book recommends in response: shaping team structures first so that the desired architecture emerges, an approach known as the reverse Conway maneuver. The chapter closes with a publicly documented case, Uber's domain-oriented microservice architecture, that shows the relationship between teams and systems in practice. Melvin Conway's observation In 1968 the computer scientist Melvin Conway published an article titled "How Do Committees Invent?" in the magazine Datamation. Its central claim was that any organization that designs a system will produce a design whose structure copies the organization's communication structure. Conway's reasoning was straightforward. Designing a system requires dividing the design work among people or groups. The way the work is divided determines which parts of the system are designed by which groups. The interfaces between parts of the system must be negotiated between the groups responsible for them. The system's structure therefore ends up mirroring the structure of the design organization, because the organization can only produce interfaces where its own members communicate. Conway gave examples, the most often repeated of which concerns compilers. If a compiler is designed by several groups, it will tend to have as many major stages, or passes, as there are groups. The number of passes reflects the organization rather than any technical necessity. Conway also drew a sobering conclusion about large projects. Because the initial design shapes the organization that is set up to build it, and that organization then constrains every later design choice, early decisions about team structure can lock a project into an architecture that is no longer appropriate once more is learned. The observation acquired its name a few years later. Fred Brooks, in The Mythical Man-Month (1975), cited Conway's argument and called it Conway's Law. Brooks's book, a study of what went wrong in the development of IBM's OS/360 operating system, was widely read, and the name stuck. It is worth being precise about what the law says. It is not a claim that organizations deliberately design systems to look like themselves. It is a claim about constraint. Communication paths make certain designs easy and others hard. Where two groups communicate frequently and richly, the parts they build can be tightly integrated. Where two groups rarely communicate, the parts they build will meet at narrow, formal interfaces, or will not meet at all. Over time, the system settles into a shape that the communication structure can support. One further point of precision matters for students. The communication structure Conway described is not the same as the formal reporting structure. Two teams in different departments that talk every day form a close communication link, whatever the org chart says. Two teams under the same manager that never talk do not. Conway's Law therefore operates on the real pattern of communication, which is exactly the pattern that org charts fail to show, as Chapter 1 of this guide explained. This is why Skelton and Pais focus on designing team interactions rather than reporting lines. Evidence for the mirroring effect Conway's Law began as an observation from experience, and for many years it was treated as folklore. Later research gave it firmer grounding. This section summarizes that research. It is context this guide adds, and it is useful for essays that need to show the law is more than an anecdote. Researchers at Harvard Business School, including Alan MacCormack, John Rusnak and Carliss Baldwin, studied what they called the mirroring hypothesis. In a study published in Research Policy in 2012, they compared pairs of software products that performed similar functions but were developed by different kinds of organization. Some were built by tightly knit commercial teams, others by distributed open-source communities whose members rarely met. Analysing the dependency structure of the code, they found that the products built by loosely coupled, distributed organizations tended to be more modular, while those built by tightly coupled organizations tended to have more interdependencies between components. The structure of the organization was reflected in the structure of the code. Lyra Colfer and Carliss Baldwin later reviewed a large body of empirical studies across industries in a 2016 article in Industrial and Corporate Change. They found broad support for the mirroring hypothesis, while also documenting exceptions. The exceptions are instructive. In some cases, organizations that were formally separate managed to build tightly integrated products because they invested heavily in communication tools, shared knowledge and common standards. In other cases, companies deliberately broke the mirror for strategic reasons. The lesson is that mirroring is a strong tendency rather than an iron rule, and that departures from it come at a cost in coordination effort. Studies at Microsoft and elsewhere have also linked organizational structure to software quality, finding that measures of how the development organization was divided could predict where defects would cluster. The general finding across this literature is consistent with Conway: where communication is weak, integration suffers, and where the organization is fragmented, the system tends to fragment in the same places. How Skelton and Pais use the law Team Topologies does not treat Conway's Law as a curiosity. The authors treat it as a design force to be used deliberately. Their argument has several parts. First, if system architecture tends to mirror team structure, then any decision about team structure is also a decision about architecture. A manager who creates a separate database team has made it likely that the system will have a central, shared database with changes controlled by that team. A manager who creates separate front-end and back-end teams has made it likely that the system will be layered along those lines, with every user-facing feature requiring coordinated changes in both layers. Skelton and Pais conclude that organizational design requires technical expertise. The people making structural decisions must understand the architectural consequences, and technical leaders must be involved in them. Second, the authors argue that unrestricted communication is not the answer. It might seem that if communication paths shape architecture, the best course is to encourage everyone to talk to everyone. Skelton and Pais argue the opposite. If every team communicates with every other team, the system will tend towards a design in which every part depends on every other part. That is, the system becomes a monolith in which nothing can change independently. The authors suggest that organizations should be deliberate about which teams need to communicate and how much. Where teams build parts of the system that should be loosely coupled, the communication between those teams should also be limited and structured. This is a counterintuitive message for many managers, who have been taught that more collaboration is always better. It is also one of the book's most important ideas, because it underlies the interaction modes discussed later in this guide. Third, the authors note that the tools an organization uses can shape communication, and therefore architecture, in unintended ways. A single shared ticketing system or chat channel used by every team can create communication paths that encourage coupling. Separate tools for teams that should collaborate closely can create barriers where none are wanted. Tool choices, in other words, are part of organizational design. Fourth, the book observes that the constraint runs in both directions. An existing architecture also shapes the organization. If a system is a large monolith, the teams that work on it will find themselves forced into constant coordination, regardless of what the org chart says. Conway's Law therefore explains why simply drawing new team boundaries is not enough if the software does not allow those teams to work independently. The reverse Conway maneuver If team structure shapes system architecture, an organization can use that fact deliberately. It can decide what architecture it wants, and then arrange its teams so that their communication structure matches that architecture. Given time, the system will tend to take the intended shape. This strategy is known as the reverse Conway maneuver, or sometimes the inverse Conway maneuver. The term circulated among consultants at the technology firm ThoughtWorks around 2010 and was popularized through their publications, including the ThoughtWorks Technology Radar. Skelton and Pais adopted it as a central technique. The maneuver works as follows. An organization first identifies the architecture it needs in order to achieve fast flow: typically, a set of loosely coupled services or components, each aligned to a coherent area of business capability, each changeable and deployable independently. It then forms teams whose boundaries correspond to those components, gives each team ownership of its component and limits the need for communication between teams to the interfaces between components. The teams, working mostly independently, will naturally build and evolve components with clear boundaries, because that is what their communication structure supports. The reverse Conway maneuver is not magic, and the authors do not present it as such. An existing system does not reshape itself overnight because the teams have been rearranged. If the current software is tightly coupled, newly formed teams will still find themselves needing to coordinate, and some technical work will be needed to separate components. What the maneuver does is align the organizational force with the desired technical direction, so that the two push the same way rather than against each other. Without it, a team trying to split a monolith while organized to support that monolith is working against the grain. The maneuver also demands clarity about the target architecture, and that is where many attempts falter. An organization that forms teams around vaguely defined areas will produce vaguely bounded software. The book's later guidance on fracture planes, discussed in Chapter 7 of this guide, is essentially a method for finding good boundaries to design towards. Monoliths hidden in the organization Skelton and Pais extend the idea of a monolith beyond the obvious case of a single large application. They describe several kinds of hidden monolith that can undermine fast flow even when the software appears to be divided. As Table 1 summarizes, each kind creates a coupling that forces teams to coordinate. Table 1. Kinds of monolith described in Team Topologies. Kind of monolith What is coupled Effect on teams Application monolith One large deployable system All teams release together Joined-at-the-database Separate services sharing one schema Schema changes need cross-team agreement Monolithic build One build for many components A change anywhere triggers a full rebuild Coupled release Components released as one bundle Teams wait for a shared release date Monolithic model One domain model or language forced everywhere Teams cannot fit models to their context Monolithic thinking One standard technology or approach for all Teams lose freedom to choose fitting tools Monolithic workplace One uniform office layout Space does not suit team collaboration needs The last three items are the most interesting for students of organization. They show that the authors see monoliths not only in code but in habits of mind and physical arrangements. A policy that every team must use the same technology stack, however well intended, couples teams to a central decision and can prevent them from choosing what suits their domain. An open-plan office designed to maximize communication may, by Conway's reasoning, encourage communication patterns that do not match the intended architecture, while failing to give teams the space they need to work closely together. Case study: Uber's domain-oriented microservice architecture The ride-hailing company Uber provides a well-documented public example of the relationship between organizational scale and system architecture. The following summary draws on a July 2020 article on the Uber engineering blog by the engineer Adam Gluck, introducing what the company called Domain-Oriented Microservice Architecture, or DOMA. The analysis connecting it to Team Topologies is this guide's own. Uber had adopted microservices during a period of very rapid growth, as a way of letting many teams work and release independently. By the time of the article, the company had around 2,200 critical microservices. The article explains that this degree of fragmentation had created its own problems. Understanding how a request travelled through the system could require tracing calls across many services owned by many teams. A single feature could require changes to services owned by several teams, which meant coordination, negotiation and waiting. The independence that microservices were meant to provide had been partly lost, because the boundaries between services did not correspond to coherent areas of business function. Uber's response was to group related microservices into domains, each representing a coherent area of functionality, such as map search, fares or matching riders with drivers. The article describes around 70 domains, organized into layers, with each layer depending only on the layers beneath it. Each domain presented a single entry point, which Uber called a gateway, so that other teams could use the domain's capabilities without needing to understand its internal services. Uber also introduced extension mechanisms that allowed teams to add behaviour to another domain's services without modifying them directly. The article reports improvements, including a large reduction in the cost of supporting platforms and faster integration of new features. Two features of the case are relevant to Team Topologies. First, the gateway is an explicit, narrow interface between a domain and the rest of the organization. It reduces the need for other teams to communicate with the domain's owners about internal details, which is exactly the kind of limited, structured interaction the book recommends. In the book's vocabulary, the domain is offered to other teams as a service. Second, the article notes that domains did not always follow the company's reporting structure. For example, it describes one engineering organization whose services were divided across three separate domains. This is a useful reminder that the unit of architectural design and the unit of formal management need not coincide. What matters, from the Conway perspective, is that the teams responsible for each domain can work within it with minimal coordination with others. The case also illustrates a point Skelton and Pais make repeatedly: breaking a system into many small pieces does not by itself produce fast flow. If boundaries are drawn without regard to business domains and team cognition, the result can be a distributed system that is just as coupled as the monolith it replaced, and harder to understand. Good boundaries, in the book's view, are those that allow a team to own a coherent piece of the business and deliver changes to it largely on its own. Key Takeaways · Melvin Conway argued in 1968 that organizations produce systems whose structure copies their communication structure; Fred Brooks later named this Conway's Law. · Empirical work on the mirroring hypothesis, including studies by MacCormack, Rusnak and Baldwin, supports the law as a strong tendency, with costly exceptions. · Skelton and Pais conclude that decisions about team structure are architectural decisions, and that unrestricted communication between all teams tends to produce monolithic systems. · The reverse Conway maneuver shapes team boundaries to match a desired architecture, so that organizational and technical forces push in the same direction. · The book identifies hidden monoliths in databases, builds, releases, models, standards and even office layouts. · Uber's domain-oriented architecture shows how grouping services into domains with narrow gateways can restore the independence that fragmented microservices had lost. Review Questions 1. State Conway's Law in your own words and explain the reasoning Conway gave for it. 1. Why do Skelton and Pais argue that encouraging every team to communicate with every other team can harm architecture? 2. Describe the reverse Conway maneuver and identify one reason it may fail in practice. 3. Choose two of the hidden monoliths in Table 1 and explain how each constrains team independence. 4. How does Uber's use of domains and gateways illustrate the principle of limiting communication between teams to well-defined interfaces? 5. What do the exceptions documented by Colfer and Baldwin suggest about the limits of Conway's Law? Chapter 3: Team-First Thinking Having shown why team structure shapes software, Team Topologies turns to the team itself. Skelton and Pais argue that the team, rather than the individual, should be the fundamental unit of delivery in a software organization. This chapter examines what they mean by a team, why they place so much weight on team size and stability, how they think about ownership, and what they mean by a team's API. Together these ideas make up what the authors call a team-first approach, and they set the stage for the discussion of cognitive load in the next chapter. Why the team is the unit of delivery Modern software systems are too large and complex for any single person to understand in full. They must be built and run by groups. The question is what kind of group works best. Skelton and Pais argue for small, stable teams whose members work closely together towards a shared goal. Such a team, they suggest, can achieve more than the sum of its members because its members share context, cover for one another and build on one another's knowledge. The book defines a team in a demanding way. A team is not simply a set of people who report to the same manager or who happen to be assigned to the same project. It is a stable group of people working together towards a shared goal, whose members depend on one another and are collectively responsible for results. By this definition, many so-called teams in large organizations are not teams at all. A pool of developers from which individuals are allocated to projects is not a team. Nor is a department of specialists who each take tickets from a queue. The authors support their emphasis on teams by pointing to research on group performance. This guide adds that the idea has broad support. Google's internal study of team effectiveness, known as Project Aristotle and publicly described by the company in 2015 and 2016, found that who was on a team mattered less than how the team worked together. The most important factor it identified was psychological safety, a concept developed by the Harvard researcher Amy Edmondson: a shared belief that team members can take interpersonal risks, such as admitting mistakes or asking questions, without being punished. Psychological safety takes time to build, and it depends on stable relationships. This connects directly to the book's insistence on long-lived teams. Team size and Dunbar's number How big should a team be? Skelton and Pais answer with a mix of industry experience and anthropology. From experience, they note that effective software teams are typically small. They suggest a size of around five to eight people, with an upper bound of around fifteen for teams with unusually high trust. Beyond this, the members can no longer maintain close working relationships with everyone else on the team. The best-known industry expression of this idea is Amazon's two-pizza team. The phrase is associated with Jeff Bezos and refers to the principle that a team should be small enough to be fed by two pizzas. Skelton and Pais cite the practice approvingly. The point is not the pizza but the limit: a team that grows past a certain size starts to fragment into sub-groups, communication overhead rises, and the team loses the shared understanding that made it effective. The anthropological grounding comes from the work of the British anthropologist Robin Dunbar. In research published in the early 1990s, Dunbar related the size of the neocortex in primates to the size of their social groups, and predicted that humans could maintain stable social relationships with roughly 150 people. This figure became known as Dunbar's number. Dunbar and colleagues later proposed that human social networks form a series of layers, each roughly three times the size of the one inside it: around 5 people with whom we are most intimate, around 15 close friends, around 50 good friends and around 150 meaningful contacts, with further layers of acquaintances beyond. Skelton and Pais use these layers as a guide for organizational design, as Table 2 sets out. The smallest layers suggest the natural size of a team in which members can trust one another deeply. The larger layers suggest natural limits for groupings of teams, such as a department or a business unit, within which people can still know one another. Beyond about 150 people, the authors suggest, an organization should expect relationships to become more formal and should design its interactions accordingly. Table 2. Dunbar's social layers as applied to team design. Approximate size Social meaning (Dunbar) Organizational use (Skelton and Pais) 5 Closest personal relationships Core of a small, high-trust team 15 Close friends Upper limit for a single high-trust team 50 Good friends Grouping of several teams with mutual trust 150 Stable social relationships Upper limit for a unit whose members know each other 500 and above Acquaintances Larger divisions needing formal interfaces The authors draw a further practical conclusion. Because these limits are real, organizations should watch what happens when a unit crosses them. A department that grows from 40 to 60 people, or a business unit that grows beyond 150, will find that the informal trust that held it together no longer reaches everyone. That is the moment to consider splitting the unit or making its internal interactions more explicit. The same logic applies to software. When a system becomes too large for a single team, the team should not simply grow. The system should be divided, and new teams formed around the parts. This guide notes that Dunbar's number has attracted scholarly criticism. Some researchers have questioned the statistical basis of the original estimate and argued that the range of plausible group sizes is very wide. For the purposes of Team Topologies, however, the precise figures matter less than the underlying principle. Human capacity for close working relationships is limited, and organizational design should respect that limit rather than assume it away. Stable, long-lived teams The second pillar of team-first thinking is stability. Skelton and Pais argue that teams should be kept together over long periods, and that work should flow to teams rather than people being moved to work. This reverses a common practice. In many organizations, especially those that fund technology through projects, people are assembled into a project team, disbanded when the project ends and reassigned to new projects. The authors see this as wasteful. Their reasoning draws on the psychology of group formation. It takes time for a group of people to become an effective team. They must learn one another's strengths and habits, develop shared norms and build trust. The psychologist Bruce Tuckman described this process in 1965 in his well-known sequence of forming, storming, norming and performing. Skelton and Pais refer to this model, while also noting that later research suggests teams do not move through the stages in a neat order. Conflict and renegotiation can recur throughout a team's life. Either way, the time needed to become effective is lost every time a team is broken up. A team that is dissolved just as it reaches high performance throws away its most valuable asset. Stability does not mean that team membership never changes. People leave, join and move for good reasons, including their own development. The authors' point is that change should be occasional and careful, so that the team's core remains intact. They suggest that a healthy team can absorb the loss or addition of a member without losing its identity, whereas a team whose membership turns over constantly never becomes a team at all. Stable teams also require a change in how work is funded and planned. If teams are long-lived, then organizations must direct a flow of work to them over time rather than creating teams for each new initiative. This is one reason why the book aligns naturally with product-based and value-stream-based funding models, in which money is allocated to a persistent team or product area rather than to a time-limited project. Ownership and the team-first mindset The third pillar is ownership. Skelton and Pais argue that every piece of software in an organization should be owned by exactly one team. Ownership here does not mean exclusive access. Other teams may read the code, suggest changes or even contribute changes under agreed rules. It means that one team is responsible for the software's health, its evolution and its behaviour in production. The value of single-team ownership can be explained with an analogy from farming, close in spirit to one the authors use. A farmer who holds land for a single season has little incentive to care about the long-term health of the soil. A farmer who holds the same land for many years will care about it. Software is similar. A team that owns a service over the long term will care about its maintainability, test coverage and operational health, because it will live with the consequences. A service that is touched by many teams but owned by none will decay, because nobody is responsible for its long-term condition. Single-team ownership also has a Conway dimension. If a piece of software is owned by one team, its internal design reflects that team's communication, which is dense and coherent. If it is shared among several teams, its design reflects their communication, which is likely to be thinner and more fragmented. Shared ownership therefore tends to produce software with unclear internal boundaries. Ownership requires that teams have the capability to look after what they own. A team that owns a service in production needs to be able to deploy it, monitor it and fix it. This is the principle often summarized by the phrase "you build it, you run it", which the Amazon chief technology officer Werner Vogels used in a 2006 interview to describe how Amazon's teams took operational responsibility for their own services. Skelton and Pais embrace the principle, while recognizing that teams need support, in the form of platforms and specialist help, to carry it without being overwhelmed. The authors also describe the behaviour they expect from members of a team. A team-first mindset means that members put the needs of the team above their own individual preferences. They attend team events, help one another, share knowledge and accept collective responsibility. The authors are frank that some individuals, however talented, find this difficult, and that organizations may need to address such cases directly. They also argue that diversity within teams helps, because people with different backgrounds and perspectives tend to generate more ideas and catch more problems. Finally, they recommend that organizations reward teams rather than individuals where possible, so that incentives support collaboration rather than competition within the team. Stable teams bring one further benefit that is easy to overlook: they make the organization predictable to itself. When teams persist, other teams learn whom to ask about what, and the relationships between teams can be designed and improved over time. When teams are constantly re-formed, every relationship must be rebuilt from scratch, and the organization's map of who owns what is always out of date. Stability inside teams is therefore a precondition for designing the interactions between them, which is the subject of the team API below and of the interaction modes later in this guide. The team API The most distinctive idea in this part of the book is the team API. In software, an application programming interface, or API, is the defined set of ways in which other programs can interact with a component. It hides the component's internal details and exposes only what others need. Skelton and Pais apply the same idea to teams. A team's API is the set of ways in which other teams can interact with it. It includes the obvious technical elements, such as the code the team owns and the service interfaces it offers, along with their versioning and documentation. It also includes less technical elements. These are the team's working practices and principles; its roadmap and current priorities; its preferred communication channels, such as a particular chat channel or regular office hours; and the way it wants other teams to request changes. A well-designed team API makes it easy for other teams to work with the team without needing to understand its internal workings or to interrupt its members constantly. The concept is useful because it turns vague notions of good collaboration into something that can be designed and improved. A team can ask itself whether its API is clear, whether other teams know how to reach it, whether its documentation is current and whether its interfaces are stable. Other teams can give feedback on the API, just as users of a software API would. The authors suggest that teams should think explicitly about which other teams they need to interact with and how, and should design their API to support those interactions. The team API also serves the book's broader goal of limiting unnecessary communication. If every interaction with a team requires a meeting or a conversation, communication overhead grows quickly as the organization grows. If most interactions can happen through clear, self-service interfaces, teams can work together without constant coordination. This is the organizational counterpart of loose coupling in software. Designing the environment for teams Skelton and Pais close their account of team-first thinking by turning to the physical and virtual environment. They argue that office layouts and digital tools should be designed around teams. Members of the same team should be able to work closely together, with space to discuss and collaborate. Teams that need to interact should be placed near each other, and teams that do not need to interact need not be. In a remote or hybrid setting, the same principles apply to chat channels, video tools and shared documents, which should be organized to support team boundaries and the interactions between teams. This point was made before the widespread shift to remote work in 2020. In 2022 the authors published the Remote Team Interactions Workbook, which applies the Team Topologies ideas to distributed teams. The underlying message is unchanged: the environment, whether physical or digital, shapes communication, and therefore, by Conway's Law, shapes the software. Key Takeaways · Skelton and Pais treat the stable, small team as the fundamental unit of software delivery, and define a team demandingly as a group with a shared goal and collective responsibility. · Team size should typically be around five to eight people, with Dunbar's social layers of roughly 5, 15, 50 and 150 offering guidance for teams and groupings of teams. · Teams should be long-lived, with work flowing to teams rather than people being shuffled between projects, because effectiveness takes time to build. · Every piece of software should be owned by exactly one team, which is responsible for its long-term health, in the spirit of "you build it, you run it". · The team API describes how other teams interact with a team, covering code, documentation, practices, roadmap and communication preferences. · Physical and digital workspaces should be designed around team boundaries and intended interactions. Review Questions 1. How do Skelton and Pais define a team, and why do many groups in large organizations fail to meet that definition? 1. Explain how Dunbar's research informs the book's guidance on team size and on groupings of teams. What criticism has Dunbar's number attracted? 2. Why do the authors argue that work should flow to stable teams rather than people moving between projects? 3. What does single-team ownership mean, and why does it tend to improve the long-term health of software? 4. List the elements of a team API and explain how a well-designed team API reduces communication overhead. Hashtags: #TheArchitectureOfFlow #TeamTopologies #FastFlow #SocioTechnicalSystems #ConwaysLaw #ReverseConwayManeuver #CognitiveLoad #TeamFirstThinking #StreamAlignedTeams #EnablingTeams #ComplicatedSubsystemTeams #PlatformTeams #TeamInteractionModes #Collaboration #XAsAService #Facilitating #TeamAPIs #StableTeams #SingleTeamOwnership #FracturePlanes #LooseCoupling #ValueStreams #OrganizationalDesign #SoftwareArchitecture #FutureOfTeamTopologies

Download the Book (PDF): Introduction Somewhere on every campus, right now, someone is composing an apology and deleting it. A student has read the message from her roommate three times, typed "I'm sorry but," looked at the "but," and put the phone face-down on the desk. A second-year who missed his turn presenting for a group project is drafting an email to the professor that opens with a paragraph about his grandmother's health, a paragraph about his laptop, and a paragraph about the bus. Two friends who have not spoken since a party a fortnight ago are each waiting for the other to go first, and each is privately rehearsing a version of events in which they were mostly right. None of this is dramatic. That is the point. The conflicts that fill a university year are rarely the stuff of tribunals. They are dishes left in a shared sink, a joke that landed badly in a seminar, a screenshot forwarded to the wrong chat, a deadline missed without warning, a comment about someone's accent, a borrowed jacket returned with a stain, a promise to split the rent on time that was not kept. They are small, and they are constant, and they are inevitable wherever several thousand young adults are placed in close quarters, handed demanding work, and asked to cooperate with people they did not choose. What decides whether such episodes pass or fester is, surprisingly often, a single conversation: whether someone says, clearly and without evasion, that they got something wrong, and what they intend to do about it. This booklet is about that conversation. Its argument is simple to state and harder to practise. An apology is not a ritual of self-abasement, a legal admission, or a social lubricant to be dispensed as freely as "no worries." It is a repair tool with a specific structure. It works when it names the harm accurately, accepts responsibility without defence, and commits to changed behaviour, and it fails when any of those parts is missing or counterfeit. On campus, where relationships are dense, repeated, and often unequal in power, learning to do this well and in proportion is one of the most practically useful things a student can learn, and very few are ever taught it. Why the campus is a special case People apologise everywhere, so it is fair to ask why the university setting deserves its own treatment. There are four reasons, and they recur throughout the chapters that follow. The first is density. A student may live, eat, study, work, and socialise inside a few square kilometres, with the same few hundred people appearing in every one of those settings. The person you snapped at in the kitchen is in your nine o'clock lecture, is dating your lab partner, and will be at the same party on Saturday. In most of adult life we can quietly let a damaged acquaintance drift. On campus that option is often unavailable, which makes unrepaired conflict unusually expensive: you pay for it every time you see the person, and you see them constantly. The second is repetition. Campus relationships run on cycles of a term, a year, a degree. A roommate is someone you will share a bathroom with for nine months. A doctoral supervisor is someone whose signature you will need for five years and whose reference you may need for fifteen. When a relationship has a long future, the way a single breach is handled shapes everything after it. Researchers who study trust describe this in terms of the "shadow of the future": the longer a relationship is expected to continue, the more each party has to gain from repairing it and the more each has to lose from letting a grievance harden. The third is asymmetry. Students deal daily with people who hold real power over them: lecturers who mark their work, teaching assistants who grade their labs, resident advisers who enforce housing rules, employers in campus jobs, admissions and funding panels. Apologising to a friend and apologising to the person who decides your grade are different acts with different risks, and the same words can land very differently depending on which direction they travel. The fourth is that students are, for the most part, still learning how to do this. Many are living without family around them for the first time. Many are negotiating shared space with a non-relative for the first time. Many arrive from school cultures in which conflict was managed for them by teachers or parents, and they find that at university nobody steps in. That is not a criticism. It is simply the situation in which the skill has to be acquired, often through painful trial and error. What this booklet does and does not cover The chapters move from principles to practice. The first two set out what an apology is for and what its working parts are, drawing on a body of research in social psychology, negotiation, and moral philosophy that has become quite specific about what makes apologies succeed. The third looks at why apologising is so hard, and at the familiar forms of the non-apology that people reach for instead. The fourth deals with timing, channel, and proportion: when to apologise, whether to do it in person or in writing, and how to avoid both the grudging under-apology and the anxious habit of apologising for everything. The next three chapters apply those principles to the three relationships the title names. Roommates and housemates present a distinctive problem: harms that are small, repeated, and embedded in routine, where an apology that does not change the routine is worthless. Friendships, and the group chats and group projects that surround them, raise questions about public and private repair, about digital permanence, and about harms done to someone who was not even present. Apologising to professors and teaching staff requires a different register altogether, one that respects the asymmetry of power without collapsing into grovelling or over-explanation, and that handles the most delicate case of all, the apology connected to academic misconduct. The final chapter turns the question around. It considers what to do when you are the one who is owed an apology, why forgiveness is not a duty, and what apology cannot fix. Some harms that occur on campus, including harassment, discrimination, and sexual misconduct, are not conflicts to be resolved between two people over coffee. They belong to formal processes, and an apology in those cases can be at best a small part of a much larger response and at worst a form of further pressure on the person harmed. Knowing where the tool stops is part of knowing how to use it. Throughout, the examples are composites: typical situations drawn from the ordinary run of university life, with invented names, rather than accounts of real individuals. Where research is cited, it is real, and the studies are described as they were actually conducted, including their limits. Much of the experimental work on apology uses scenarios, vignettes, and student samples, and it should be read as strong evidence about how people tend to respond rather than as a formula that guarantees any particular outcome. People are not vending machines. A perfectly constructed apology can still be refused, and a clumsy one offered with evident sincerity can still be accepted. What the research offers is a way of stacking the odds. A note on the word itself In ordinary speech "sorry" does many jobs. It expresses sympathy ("I'm sorry your exam went badly"), it softens a request ("Sorry, could you pass that?"), it signals embarrassment, it fills silence. None of those uses is wrong, and one later chapter looks at the surprisingly useful social work that such "superfluous" apologies do. But the apology this booklet is mostly concerned with is narrower. It is the act of acknowledging to another person that you did something that wronged or harmed them, that you were responsible for it, and that you regret it, in a way that aims to restore something the offence damaged. It is an act of communication with a purpose, and like any such act it can be done skilfully or badly. Done skilfully, it is also one of the more generous things one person can do for another. It tells the other person that their experience was real, that their standing matters, and that the relationship is worth the discomfort of admitting fault. Those are not small messages, and in a place as crowded and consequential as a university, they are worth learning to send well. Chapter 1: What an Apology Is For Consider two students who share a flat. Jonah borrows Priya's bicycle without asking, leaves it unlocked outside the library, and it is stolen. When she finds out, he says, immediately and with real distress, "I'm so sorry." She says it is fine. A week later she is still cold with him, and he cannot understand why. He apologised. She accepted. What more is there? The answer is that Jonah said the word without doing the work the word stands for. He expressed regret, which is real, but he said nothing about what he had actually done wrong, which was not losing the bicycle but taking it without permission in the first place. He did not acknowledge that she now had no way to get to her placement across town. He did not offer to replace it or pay toward a new one. He did not say anything about whether he would take her things again. Priya's "it's fine" was not acceptance; it was the reflexive reply that social life demands when someone says sorry, and her continued coldness was her honest verdict that nothing had been repaired. This small episode contains most of the problem this booklet addresses. People commonly treat an apology as a single speech act, a password that unlocks forgiveness. In fact it is a compound act that does several distinct jobs, and when some of those jobs are left undone, the word "sorry" does not cover the gap. To apologise well, it helps first to understand what an apology is actually for. Three things an offence damages When one person wrongs another, even in a small way, three things are typically damaged at once. An effective apology has to address all three, though not always with equal weight. The first is the injured person's standing. Being wronged carries an implicit message: you did not count for enough to be considered. If a housemate eats the food you bought for the week, the practical loss may be a few pounds or dollars, but the message is that your claims on shared space were not taken seriously. The psychiatrist Aaron Lazare, whose book On Apology remains one of the most careful treatments of the subject, argued that the restoration of the offended party's self-respect and dignity is among the most important needs an apology meets. An apology says, in effect: you did matter, and I was wrong to act as if you did not. This is why apologies that focus entirely on the offender's feelings ("I feel terrible, I'm such an idiot") often leave the recipient cold. They are about the wrong person. The second is the shared rules. Every relationship rests on a set of expectations, many of them never spoken: that friends do not share each other's secrets, that roommates ask before having overnight guests, that group members do their assigned part, that students submit their own work. An offence breaks one of those rules, and in doing so it raises a question about whether the rule still holds. If it is never acknowledged, the rule has in effect been rewritten. The philosopher Nick Smith, in I Was Wrong, treats this as one of the core functions of what he calls a "categorical" apology: the offender explicitly affirms the moral principle that was violated, and so reinstates it. When Jonah fails to say "I shouldn't have taken your bike without asking," he leaves open the possibility that he thinks it was acceptable, and Priya is left not knowing whether her belongings are safe. The third is trust, meaning the injured person's expectation about how the offender will behave in future. An offence is evidence. It tells the victim something about what the offender is willing to do, and a rational victim updates accordingly. An apology is, among other things, an attempt to supply counter-evidence: a signal that the offence was not representative, that the offender understands why it was wrong, and that it will not recur. This is why promises about future behaviour, and concrete offers of repair, carry so much weight. Words of regret cost little. Commitments that can be checked cost more, and people know the difference. Seeing apology as the repair of standing, rules, and trust explains a great deal about why some apologies work and others do not. An apology that restores standing but says nothing about the future leaves trust damaged. An apology that promises to do better but never names the rule that was broken leaves the recipient uncertain what exactly will be different. An apology that affirms the rule but is delivered with irritation or haste fails to restore standing, because its manner contradicts its content. What an apology is not Several things that look like apologies are something else, and it is worth separating them early because they account for most of the frustration people feel when "sorry" fails. An apology is not an expression of sympathy. "I'm sorry you feel that way" and "I'm sorry that happened to you" are statements of regret about a state of affairs, not acceptances of responsibility for causing it. There is nothing wrong with sympathy, but when it is offered in place of an apology it is usually heard, correctly, as a refusal to accept fault. An apology is not an explanation. Explanations have a place, and a later chapter looks at when an account of what happened helps and when it hurts. But an explanation answers the question "why did this happen?" whereas an apology answers "whose fault was it and what now?" People who have been wronged often experience a long explanation as a defence brief, and they are frequently right. An apology is not a transaction that obliges forgiveness. Offering an apology does not create a debt the other person must repay by forgiving, and apologies that come with an implied demand ("I said sorry, what more do you want?") tend to reopen the wound. The recipient is entitled to take time, to accept partially, or not to accept at all. The final chapter returns to this at length. An apology is not self-punishment. Some people, especially conscientious and anxious people, apologise in a way that turns the conversation into a display of their own suffering. They berate themselves, catalogue their failings, and may end up needing reassurance from the very person they wronged. This is a real and understandable response to guilt, but it reverses the direction of care. The injured party ends up comforting the offender. And an apology is not a formula. The research described in the next chapter identifies components that reliably make apologies more effective, and it is useful to know them. But an apology assembled from correct parts and delivered without understanding will usually be detected. What people respond to, underneath the structure, is evidence that the offender has genuinely grasped what they did and why it mattered. Guilt, shame, and the direction of attention Why do some people find it natural to focus on the other person when they apologise, while others slide into self-focus or defensiveness? One useful answer comes from the psychology of moral emotions, and in particular from the distinction between guilt and shame developed by June Price Tangney and her colleagues and set out in Shame and Guilt, which she wrote with Ronda Dearing. In Tangney's account, guilt and shame are both painful self-conscious emotions that follow wrongdoing, but they differ in what they are about. Guilt is focused on a behaviour: "I did a bad thing." Shame is focused on the self: "I am a bad person." The distinction sounds subtle, but it has consequences. Across a large body of research, guilt has been associated with a tendency to take reparative action, to apologise, confess, and try to put things right, because a bad action can be undone or compensated. Shame, by contrast, has been associated with hiding, withdrawal, and defensive anger, because if the self is the problem, there is nothing to repair; the only options are to escape or to deny. This helps explain both of the failure modes described above. The defensive non-apology, in which the offender minimises, justifies, or counter-attacks, is often shame in disguise: admitting the offence feels like admitting to being a bad person, so the offence must be denied. And the self-flagellating apology is often shame expressed directly: the offender cannot separate what they did from who they are, so the conversation becomes about their worth rather than about the harm. The practical upshot is that the most useful internal stance for apologising is a kind of focused guilt. You did a specific thing, it had a specific effect on a specific person, and you can do specific things about it. That frame is both more accurate and more bearable than the global judgement that shame delivers, and it points attention outward, toward the person who was harmed and what they need. One line of research lends this some support. Juliana Breines and Serena Chen, in a set of studies published in 2012, found that participants who were prompted to think about a past moral transgression with self-compassion, rather than with self-esteem boosting or no instruction, reported greater motivation to make amends and to avoid repeating the behaviour. Treating yourself decently after a mistake, it seems, does not let you off the hook. It makes it easier to stay on it. Small offences, large meanings One consequence of this three-part view is that the size of an offence and the size of the damage it does are only loosely related. A trivial act can carry a heavy message, and a costly accident can carry almost none. Take a seminar. In a discussion of a set reading, a student named Tomás makes an argument, and another student, Ellie, cuts across him with "that's not really what the text says, though," then goes on to make almost the same point in different words. The tutor praises Ellie's contribution. Nothing material has been lost. Tomás's grade is not affected. By any external measure the offence is tiny. But Tomás, who is the only first-generation student in the group and who spent an hour preparing that point, leaves the room convinced that he is seen as someone whose ideas do not count. The damage to his standing is real and out of all proportion to the act, and it will shape whether he speaks in the next seminar. Now take a costly accident. Two students are carrying a housemate's television down the stairs during a move, and one of them stumbles and it cracks. The loss is several hundred pounds or dollars. But if the stumbling student says at once, "I'm so sorry, that was me, I'll sort out the cost," the damage to standing and to the shared rules is close to zero. Nobody's claims were ignored and no rule was broken. What remains is a practical problem about money, which an offer of repair addresses directly. The lesson is that an apology should be calibrated to the damage as the injured person experiences it, not to the objective size of the act. This is harder than it sounds, because the offender's natural instinct is to measure the offence by their own intentions ("I didn't mean anything by it") and by its material consequences ("but it didn't actually affect you"). Neither measure captures what the other person lost. Ellie, if she noticed Tomás's silence afterwards, would do far better with a short, specific acknowledgement ("I cut you off in the seminar and then basically repeated your point; that wasn't fair to you") than with either a lengthy apology that makes the moment awkward or a decision that it was too minor to mention. The same principle works in reverse. Where the damage is almost entirely practical, a long emotional apology is often unwelcome. The housemate with the cracked television does not need a speech about how terrible the carrier feels; they need to know the cost will be covered. Matching the apology to the actual damage, standing, rules, trust, or simply material loss, is one of the recurring skills this booklet returns to. It also explains why the same act can require quite different apologies to different people. Reading someone's private messages over their shoulder might be a minor lapse to one friend and a serious breach to another, depending on what those messages were and what the friend's history with privacy has been. There is no universal price list for offences. The only reliable way to know what was damaged is to pay attention to the person, and, where it is unclear, to ask. Chapter 4 looks at how to do that without making the conversation about yourself. Why the campus raises the stakes The general functions of apology apply everywhere, but the university setting intensifies each of them. Standing matters acutely to people in their late teens and early twenties, who are in the process of establishing adult identities among peers they have only recently met. A slight that would be shrugged off by a settled forty-year-old can feel, to a first-year student eight weeks into a new city, like a verdict on whether they belong. This is not over-sensitivity. It is an accurate reading of a situation in which social position really is being worked out, and in which there is often no established network of old friends to fall back on. Shared rules are especially unclear on campus. A residence hall brings together people from different family cultures, different countries, and different assumptions about noise, cleanliness, guests, borrowing, privacy, and money. Many conflicts arise not because anyone intended harm but because two people were operating under different rules without knowing it. An apology in that situation does double duty: it repairs the breach and, if done well, makes the rule explicit for the first time. Trust has a long future. As the introduction noted, students are locked into relationships for terms and years. A friendship that survives a serious breach in the first year may become one of the defining relationships of a person's life. A working relationship with a supervisor that is damaged early and never repaired can shadow a whole degree. And the audience is often larger than two. On campus, conflicts are rarely private. Housemates take sides. Friend groups divide. Group chats relay events to people who were not present. Course cohorts notice who is not speaking to whom. That means apology on campus often has a social dimension that apologies in more dispersed settings lack: it repairs not only a relationship between two people but also a small public understanding of what happened. The case for learning to do it well It would be easy to treat everything in this chapter as common sense. Most people know, in principle, that they should own their mistakes. But the gap between knowing and doing is wide, for reasons Chapter 3 explores, and the cost of the gap on campus is high. A housemate conflict that nobody repairs can make a student dread going home for the rest of the year. Friendship ruptures in the first year can leave a student isolated at exactly the point when isolation is most harmful. Poorly handled lapses with teaching staff can turn a recoverable mistake into a lasting reputation. Against those costs, the skill of apologising well is cheap to acquire and pays off immediately. It does not require a change of personality. It requires understanding a handful of things: what the injured person needs, what the working parts of an apology are, what reliably goes wrong, and how the answers shift across the different relationships of campus life. The next chapter begins with the working parts. Before that, return briefly to Jonah and Priya. What would a better apology have sounded like? Something like this: "I took your bike without asking, and that was wrong; it's yours and I should have checked. Then I left it unlocked and it got stolen, which means you can't get to your placement. That's on me. I want to pay for a replacement, and until it arrives I'll cover your bus fares. And I won't borrow your things again without asking first." That apology is not eloquent. It takes about twenty seconds to say. It does not dwell on Jonah's feelings, though they are evident. But it restores Priya's standing by naming her ownership, it reaffirms the rule about asking, and it gives her concrete evidence about the future. It is the kind of apology after which "it's fine" might actually mean it. Chapter 2: The Working Parts If the previous chapter described what an apology is for, this one describes what it is made of. The question sounds almost too simple to research, but over the past three decades psychologists, negotiation scholars, and philosophers have taken it seriously, and they have arrived at a fairly consistent picture. Apologies are not all-or-nothing. They are built from recognisable elements, the elements do not carry equal weight, and which ones matter most depends in part on who is receiving the apology and what was damaged. Six elements, ranked The most frequently cited study of apology structure was published in 2016 by Roy Lewicki, Beth Polin, and Robert Lount in the journal Negotiation and Conflict Management Research. Drawing on earlier work, they identified six elements that commonly appear in apologies: an expression of regret, an explanation of what went wrong, an acknowledgement of responsibility, a declaration of repentance, an offer of repair, and a request for forgiveness. They then tested how much each element contributed to an apology's perceived effectiveness. Across two experiments involving 755 participants, 333 adults recruited online and 422 undergraduates, people read a scenario about a job applicant who had made an error on a previous employer's tax filing and then evaluated apologies containing different combinations of the six elements. Two findings stood out. First, apologies containing more of the elements were rated as more effective, credible, and adequate than apologies containing fewer. Second, the elements were not equally valuable. The most important by a clear margin was acknowledgement of responsibility. The second was an offer of repair. Expression of regret, explanation, and declaration of repentance sat in a middle tier. The least important was the request for forgiveness. Lewicki's own summary of the practical lesson was blunt: say it is your fault, that you made a mistake, and if you have to leave something out, leave out the request for forgiveness. The study has limits worth stating. It used a written workplace scenario rather than a real relationship, and participants were evaluating a stranger rather than someone they cared about. Real apologies are delivered with faces, voices, and histories, and those can outweigh any structural feature. But the ranking fits well with what other research and ordinary experience suggest, and it provides a sensible starting point. Table 1 sets out the six elements with what each one does and how it tends to sound when done well and badly in a campus setting. Table 1. The six elements of an apology, as identified by Lewicki, Polin and Lount (2016). Element Relative weight What it does Weak version Stronger version Acknowledging responsibility Highest Places fault clearly with the speaker "Mistakes were made with the rota" "I skipped my cleaning week twice" Offer of repair High Converts regret into action "Let me know if I can help" "I'll do the next two weeks" Expression of regret Middle Shows the speaker cares about the harm "My bad" "I'm genuinely sorry" Explanation Middle Makes the lapse intelligible A long list of pressures One sentence, no excuses Declaration of repentance Middle Commits to future change "It won't happen again" "I've set a reminder for my weeks" Request for forgiveness Lowest Invites the relationship to resume "So we're good, right?" "I hope you can forgive me" The rest of this chapter takes the elements in order of importance, because the order matters for how an apology is built. Acknowledgement of responsibility The core of an apology is a clear statement that you did something and that it was wrong. Everything else is supporting material. The reason this element carries so much weight is that it is the one most directly aimed at the injured person's standing and at the shared rules. By saying "I did this, and I shouldn't have," the offender confirms that the victim's grievance is legitimate and that the rule they relied on still holds. Three features distinguish a strong acknowledgement from a weak one. The first is agency. Grammar matters here. "The deadline got missed" and "I missed the deadline" describe the same event, but only the second has a person in it. The passive voice, which appears with suspicious frequency in institutional and political apologies, removes the actor from the sentence. In personal apologies, the equivalent move is to attribute the offence to circumstances ("things got hectic"), to a state of mind ("I wasn't myself"), or to a vague shared failure ("we both let things slide"). Sometimes those are partly true. But if they appear in place of a first-person admission rather than after it, they are heard as evasions. The second is specificity. "I'm sorry for everything" or "I'm sorry for how I've been" is almost always weaker than "I'm sorry I told Sam what you said about your parents." Specificity does two things. It shows the offender has actually understood what happened from the victim's point of view, rather than grasping only that the victim is upset. And it bounds the admission, which is reassuring to both parties: it makes clear what is being owned and what is not. Vague apologies can feel oddly threatening to recipients precisely because they are unbounded; the victim cannot tell whether the offender understands the real problem or is simply trying to make the discomfort stop. The third is naming the impact, not only the act. "I shared your message in the group chat" is an admission. "I shared your message in the group chat, and now people you barely know have read something you meant for me" shows that the offender has followed the act through to its consequence for the other person. That second step is often where apologies become genuinely restorative, because it is the step that tells the victim they have been seen. Offer of repair The second most important element is an offer to make things right. Its power lies in the fact that it costs something. Anyone can say sorry; offering to pay, to redo, to replace, to take on an extra share, or to put right a public account of events requires the offender to give up time, money, or face. That cost is what makes it credible. In the language of economics, it is a costly signal, and costly signals are believed in a way that cheap talk is not. A good offer of repair has three properties. It is concrete: not "let me know if there's anything I can do," which places the burden on the injured person to invent a remedy, but a specific proposal. It is proportionate: roughly matched to the harm, neither a token gesture for a serious loss nor an extravagant gesture for a small one, which can feel like an attempt to buy one's way out. And it is responsive: offered in a way that leaves room for the other person to say what would actually help. "I'd like to replace the book I damaged; would you rather I buy a new copy or give you the money?" is concrete and responsive at once. Not every offence admits of material repair. You cannot un-say a hurtful remark. But even then, repair is often possible in another currency. If you embarrassed someone in public, repair might mean correcting the record in front of the same audience. If you broke a confidence, it might mean going back to the person you told and explaining that you were wrong to pass it on. If you let a group down, it might mean taking on the next difficult task. The question to ask is: what did my action cost this person, and what can I give back that addresses that cost? The middle tier: regret, explanation, and repentance The three elements in the middle tier matter, but they are more easily overdone and their value is more dependent on delivery. Regret is the emotional register of the apology, the words "I'm sorry" or "I really regret it." It matters because it signals that the offender cares about the harm, not merely that they are conceding a point. Its absence is conspicuous: an apology that acknowledges responsibility in a flat, procedural tone can feel like a legal admission rather than a human one. But regret is also the element most prone to inflation. Excessive expressions of regret, particularly when they drift toward self-punishment, shift the focus to the offender. A sentence or two, sincerely meant, is usually enough. Explanation is the most double-edged element. On one hand, people who have been wronged often want to understand why. An unexplained offence is frightening because it is unpredictable; knowing that a friend forgot your birthday because their grandfather was in hospital that week makes the lapse intelligible and non-threatening. On the other hand, explanations slide easily into excuses. The difference lies partly in length and partly in placement. An explanation that comes after a clear acknowledgement of responsibility, and that is brief, is heard as context. An explanation that comes first, or that runs longer than the acknowledgement, is heard as defence. A useful rule is that an explanation should never contain the word "but" in a way that qualifies the admission: "I missed the meeting because my train was cancelled, but I should have messaged you" works; "I'm sorry I missed the meeting, but my train was cancelled" undoes the apology in its second half. Declaration of repentance is the promise not to do it again. It is aimed at trust, and it is strongest when it is specific and checkable. "It won't happen again" is a promise almost everyone has heard broken, and on its own it carries little weight. "I've put your birthday in my calendar with a week's warning" or "I'm going to stop borrowing things from the kitchen without asking, and if I do need something I'll text first" are declarations the other person can actually observe being kept. The more specific the commitment, the more it functions as evidence rather than aspiration. The request for forgiveness The least valuable element, according to Lewicki and his colleagues, is the request for forgiveness. This can seem surprising, since forgiveness is in a sense the point of the exercise. But the finding makes sense once you notice what the request does. It shifts the burden to the injured person, who now has to respond, and it can feel like pressure. A request for forgiveness offered too soon, or too insistently, turns a gift into a demand. This does not mean the request should never be made. In close relationships, saying "I hope you can forgive me" can be an honest expression of how much the relationship matters. The key is that it should be phrased as a hope rather than a question requiring an immediate answer, and that the offender should be prepared to accept that the answer may be "not yet." The request is also best left to the end, after everything else has been said, and it is the element most safely dropped when time or circumstance is short. Matching the apology to the person One further finding refines the picture. People differ in what they most need from an apology, and some of those differences are systematic. In a 2010 study published in Organizational Behavior and Human Decision Processes, Ryan Fehr and Michele Gelfand examined how the effectiveness of different apology components depended on the recipient's "self-construal," the extent to which people define themselves as independent individuals, in terms of close relationships, or in terms of larger groups. They found that offers of compensation were especially effective for people with a strongly independent self-construal, expressions of empathy for those with a strongly relational one, and acknowledgements that a rule or norm had been violated for those who defined themselves by group membership. For students this has a practical implication. A campus contains people from many cultural backgrounds and many family styles, and the apology that feels complete to you may feel incomplete to the person receiving it. Research on cultural differences in apology supports the same caution. William Maddux and colleagues, comparing American and Japanese participants in work published in 2011, found that Japanese participants were more likely to see apology as an expression of remorse and a way of repairing a relationship even when they were not personally at fault, while American participants were more likely to associate apology with an admission of personal blame. Neither view is wrong. But a student who apologises by offering to pay for damage, when the other person was mainly hurt by what the damage said about the friendship, will find the apology falls flat, and so will one who offers a heartfelt emotional apology to someone who simply wants the broken thing replaced. The practical answer is not to memorise cultural profiles, which is a good way to stereotype people. It is to include more than one component, since the research consistently finds that fuller apologies outperform thinner ones, and to pay attention to what the other person responds to. If a housemate brightens when you offer to replace something but remains distant when you talk about how bad you feel, that is information. If a friend seems more hurt by the fact that you did not ask how they were than by the practical consequences, that is information too. Putting the parts together In practice, a sound apology often follows a natural order: acknowledge what you did and its effect; express regret; offer repair; say what will be different; and, if appropriate, express the hope of forgiveness. Explanation, if needed, is best slotted in briefly after the acknowledgement. The whole thing can be short. The components are not a checklist to be recited but a way of checking, before you speak, whether anything essential is missing. Consider how this works in a real case. Aisha is one of four students on a marketing group project. She agreed to write the competitor analysis by Thursday so the group could assemble the slides over the weekend. She did not do it. She did not tell anyone until Friday night, when a groupmate messaged to ask where it was. The others then spent Saturday writing her section in a hurry, and the final presentation was weaker for it. Her first draft of an apology, typed into the group chat, reads: "Sorry guys, this week has been insane with everything going on, I had two other deadlines and I've been really stressed. I feel awful. Hope it's ok!" It contains regret and a long explanation. It contains no acknowledgement of what she actually did, which was not simply failing to write the section but failing to warn anyone. It offers nothing. It says nothing about the future. And its closing "Hope it's ok!" functions as a request for forgiveness that also asks the group to declare the matter closed. A rebuilt version might read: "I owe you all an apology. I said I'd have the competitor analysis done by Thursday, and I didn't, and worse, I didn't tell anyone until Sam asked on Friday night. That meant you three lost your Saturday redoing my part, and the presentation suffered for it. I'm really sorry. I had other deadlines, but I should have flagged it on Tuesday when I knew I was behind. For the next stage I'll take the write-up of the final report, which is the biggest remaining piece, and I'll post drafts in the shared folder by Wednesday each week so nobody has to chase me." The second version is not much longer than the first, but every element is different. The acknowledgement is specific, identifying both the missed work and the more damaging failure to warn. The impact on the group is named. The explanation is present but compressed to a single clause and placed after the admission, where it cannot be mistaken for a defence. The repair is concrete and proportionate: she takes on a larger share of the remaining work. The commitment about the future is checkable, since the others will see whether drafts appear on Wednesdays. And there is no request for forgiveness at all, which in a group-work context is probably right; the group's trust will return, or not, depending on what happens over the following weeks. Notice too what the rebuilt version does not do. It does not dwell on how awful Aisha feels. It does not ask anyone to reassure her. It does not over-promise, for example by offering to redo the whole presentation. It treats the other three as colleagues who were let down and who are owed a straight account and a practical remedy, which is exactly what they are. One final point concerns authenticity. Students who learn about the structure of apologies sometimes worry that using it will make them sound scripted. The worry is reasonable but mostly misplaced. The structure describes what sincere apologies already tend to contain. Knowing it does not make an apology less genuine; it makes it less likely that a genuine apology will fail because a key part was left out under stress. What makes an apology sound scripted is not structure but the absence of specific detail, the kind of detail only someone who has actually thought about what they did can supply. An apology that names the particular act, the particular effect, and the particular remedy cannot sound like a template, because no template contains those things. Chapter 3: Why Sorry Is Hard to Say If apologising well were only a matter of knowing the components, the previous chapter would be enough. It is not. Most people who fail to apologise, or who apologise badly, know perfectly well what they ought to say. The difficulty lies elsewhere: in the pull of self-justification, in the fear of what an admission will cost, and in a set of mental habits that make one's own offences look smaller than they are. Understanding these forces does not make them disappear, but it makes them recognisable, and a force you can recognise is one you can work around. Two stories about the same event One of the most illuminating studies in this area was published in 1990 by Roy Baumeister, Arlene Stillwell, and Sara Wotman in the Journal of Personality and Social Psychology. They asked people to write two autobiographical accounts: one of a time someone had angered them, and one of a time they had angered someone else. Because the same people wrote both kinds of story, differences between them could not be explained by personality. They reflected the position the writer occupied. The differences were systematic. When people wrote as perpetrators, they tended to describe the offence as a single, bounded incident that was over and done with, provoked by circumstances or by the other person, understandable given the context, and of limited lasting consequence. When they wrote as victims, they described offences as part of a longer pattern, as arbitrary or incomprehensible, and as having lasting effects. Each side's story contained some truth. But neither contained the whole of it, and each was shaped in a direction that suited the teller. This finding, sometimes called the magnitude gap, explains a great deal of campus conflict. The student who leaves dishes in the sink sees an isolated lapse during a busy week. The housemate who finds them sees the fifth time this month. The friend who made a joke at a party sees a moment of banter in a noisy room. The person the joke was about sees it as confirmation of something they have suspected for weeks. Both are reporting honestly. They are simply standing in different places. The practical consequence for apology is that the offender's sense of how big a deal something was is systematically unreliable. It is biased downward. This does not mean the victim's sense is always accurate, since victims too can inflate, but it means that when you are the one who caused harm, your first estimate of its seriousness should be treated with suspicion. If you find yourself thinking "this is a lot of fuss over nothing," that is precisely the moment to consider that the other person may be seeing a pattern you are not. The machinery of self-justification Why do perpetrators' accounts tilt so consistently in their own favour? One influential answer comes from the theory of cognitive dissonance, which the social psychologists Carol Tavris and Elliot Aronson apply to everyday wrongdoing in their book Mistakes Were Made (But Not by Me). Most people hold a view of themselves as reasonably decent, competent, and fair. When they do something that conflicts with that view, such as treating a friend unkindly, they experience discomfort. There are two ways to resolve it. One is to revise the self-image: "I am the kind of person who does unkind things." The other is to revise the account of the act: "It wasn't really unkind, given what she did first." The second is far less painful, and so the mind reaches for it automatically. Tavris and Aronson's point is that this process is not deliberate lying. It happens below awareness, and it is strengthened by each step. The first small justification makes the next one easier, until a person who began by feeling a flicker of guilt has constructed a complete and sincere narrative in which they were the wronged party. This is why arguments between housemates or friends can escalate so quickly and why, a week later, both parties are genuinely convinced they are owed the apology. The antidote is not to distrust yourself completely, which is paralysing, but to notice the characteristic signs of justification at work. A few are especially common. One is the sudden relevance of the other person's past faults: "Well, she never cleans the bathroom either." Another is the emergence of a technicality: "I never actually promised to be there." A third is an intense focus on intention: "I didn't mean it that way." Each may be true. But when they arrive quickly and with a feeling of relief, they are often doing the work of dissonance reduction rather than honest analysis. Three barriers The psychologist Karina Schumann, who has spent much of her career studying why people do and do not apologise, reviewed the evidence in a 2018 article in Current Directions in Psychological Science. She grouped the obstacles into three broad barriers, and they provide a useful map. The first is low concern for the victim or the relationship. People who care less about the person they have harmed, or who are less inclined to take others' perspectives, are less likely to apologise. This is the least surprising barrier, but it has a practical implication: when you find yourself reluctant to apologise, it is worth asking honestly whether the relationship matters to you. If it does, remembering that fact can be enough to get past the reluctance. If it does not, that is also worth knowing, though on a campus where you will keep seeing people, even relationships you do not value may be worth maintaining at a civil level. The second is perceived threat to the self. Apologising means admitting fault, and admitting fault can feel like conceding that you are incompetent, unkind, or untrustworthy. Schumann cites evidence that transgressors tend to overestimate how unpleasant apologising will be. They also tend to feel it as a loss of power. In a 2013 study with a memorable subtitle, "and we issue no mea culpa for this research finding," Tyler Okimoto, Michael Wenzel, and Kyli Hedrick found that people who were asked to refuse to apologise for a transgression reported greater feelings of power, control, and self-worth than those who apologised. Refusing, in other words, can feel good in the short term. That is part of why it is tempting, and it is worth naming, because the boost is bought at the expense of the relationship. The third is perceived ineffectiveness: the belief that apologising will not help, that the other person will not forgive anyway, so there is no point. Research by Marret Leunissen and colleagues on what they called the "apology mismatch" found that offenders tend to underestimate how much victims value apologies and how likely victims are to respond positively. People who have been wronged often want an apology more than the offender realises, and they are frequently more willing to move on after one than the offender expects. Fixed selves and the fear of being a bad person One of the more hopeful findings in this area concerns what people believe about personality itself. Carol Dweck's research on "mindsets," which is widely known in education, distinguishes between people who see traits as fixed and those who see them as capable of growth. In a 2014 paper in Personality and Social Psychology Bulletin, Schumann and Dweck applied this distinction to wrongdoing. They found that people who believed personality could change were more likely to accept responsibility for their transgressions and to offer fuller apologies, and that this was partly because they found the admission less threatening. If a mistake reveals a fixed flaw, admitting it is a verdict on who you are forever. If it reveals something you can change, admitting it is simply the first step in changing it. The researchers also found that encouraging people to adopt the growth view increased their willingness to take responsibility. Schumann has found a related effect with self-affirmation. In other work, people who first had a chance to reflect on values important to them, affirming their sense of themselves as fundamentally decent, went on to offer more comprehensive apologies. The logic is the same as with the Breines and Chen findings on self-compassion discussed in Chapter 1. When the self feels secure, it can afford to admit error. When it feels under siege, it defends. The personal implication is straightforward, if not easy. Before apologising for something that stings, it can help to remind yourself of two things: that you are more than this one action, and that the point of admitting it is to change it. Neither thought lets you off the hook. Both make it easier to get on it. The anatomy of the non-apology When the barriers win, people rarely refuse outright. They reach instead for one of several familiar forms that look like apologies but lack the essential element of responsibility. These forms are so common that most people can recognise them instantly when they are on the receiving end, and fail to recognise them when they are the ones speaking. The most common forms, what each signals to the listener, and how each can be repaired are set out in Table 2. Table 2. Common forms of non-apology and how to repair them. Form Typical wording What the listener hears Repair The conditional "I'm sorry if I upset you" Doubt that harm occurred Name the act: "I'm sorry I said that" The feelings apology "I'm sorry you feel that way" The problem is your reaction Own the cause, not their response The "but" "I'm sorry, but you started it" The apology is withdrawn Apologise fully; raise grievances separately The passive "Things got out of hand" Nobody is responsible Put yourself in the sentence The minimiser "It was just a joke" Your hurt is excessive Acknowledge the effect, not the intent The deflection "I'm a terrible friend" Now comfort me Keep attention on their experience The demand "I said sorry, what else?" Your response is on a clock Let them take their time Each of these forms fails for a slightly different reason, but they share a common structure: they allow the speaker to say the word "sorry" while withholding the substance. The conditional apology ("if I upset you") treats the harm as hypothetical. The feelings apology locates the problem in the other person's reaction. The "but" apology offers an admission and then retracts it. The passive and minimising forms remove the actor or shrink the act. The deflection turns the conversation toward the speaker's own distress. The demand converts the apology into a transaction. It is worth saying that not every sentence containing "if" or "but" is a non-apology. "I'm sorry I snapped; if there's anything else I did that bothered you, I'd like to know" uses "if" in a way that invites more honesty, not less. And there are situations in which an offender genuinely did not cause the harm attributed to them, and where a feelings-based response ("I'm sorry you've had such a hard time with this") is an honest expression of sympathy rather than an evasion. The test is not the presence of particular words but whether the speaker has clearly owned what they actually did. Attachment and defensiveness A final factor worth mentioning concerns the patterns people bring from earlier relationships. Research on adult attachment, which examines how people approach closeness and dependence, has linked attachment styles to apology quality. Schumann and Edward Orehek reported in 2019 that people higher in avoidant attachment, who tend to be uncomfortable with emotional closeness, offered lower-quality apologies, in part because apologising requires a vulnerability they find aversive. This is not a reason to diagnose yourself or anyone else. But it is a reminder that the difficulty some people have with apology is not simply stubbornness. For someone who learned early that admitting weakness invited criticism, or that emotional exposure was unsafe, saying "I was wrong" can feel genuinely dangerous. Recognising that in yourself can be the beginning of changing it. Recognising it in others can make their clumsy or partial apologies easier to receive generously. Barriers peculiar to campus The general barriers apply everywhere, but university life adds three of its own. The first is the audience. In a close community, an apology is rarely a purely private act. If you apologise to a housemate, the other housemates will probably hear about it. If you apologise in a friend group, the apology itself becomes part of the story that circulates. For students who are still working out their social position, this can make apology feel like a public loss of face, a concession that everyone will know about. The fear is understandable, but it usually gets the social arithmetic wrong. In most groups, people who own their mistakes cleanly are respected for it, and the person whose standing suffers over time is the one who never does. The student who says "I got that wrong, sorry" in front of the group is rarely the one people talk about afterwards. The one who digs in usually is. The second is mutual fault. Many campus conflicts are not cases of a clear offender and a clear victim. Two housemates have both been passive-aggressive about the cleaning. Two friends both said things they regret during an argument. In these situations each party is waiting for the other to go first, on the reasoning that apologising first would amount to accepting all the blame. The result is often a standoff that lasts far longer than the original conflict. The way out is to recognise that an apology for your own part does not concede anything about the other person's part. "I'm sorry for what I said about your family; that was out of line" is a complete apology for one act. It does not say "and everything else was my fault too." Going first, in these situations, is not surrender. It opens the door for the other person to do the same, and in practice they often walk through it. The third is the fear of formal consequences. Some students hesitate to apologise because they worry that an admission might be used against them, for example in a housing dispute, an academic integrity case, or a disciplinary process. In a minority of situations this concern is legitimate, and Chapter 7 discusses how to handle apologies when a formal process is under way. But in the great majority of everyday conflicts, no formal process is involved or likely, and the fear is simply the threat-to-self barrier wearing an official-looking disguise. A student who will not apologise to a roommate for breaking a lamp because "it might count as admitting liability" is usually protecting pride, not legal position. It is also worth noticing one barrier that is often more acute for students than for older adults: inexperience with repair itself. Many people reach university having rarely seen an adult apologise well. They may have grown up in families where conflicts were resolved by silence and a gradual return to normal, or where apologies were demanded as rituals of submission rather than offered as acts of repair. If the only apologies you have witnessed were forced out of children by parents ("say sorry to your sister"), it is natural to associate apology with humiliation. Part of what university offers, if you let it, is the chance to learn a different model, one in which apology is something competent adults do routinely and without drama. What to do with all this The lesson of this chapter is not that people are bad at apologising because they are bad people. It is that apologising cuts against several deep and ordinary features of human psychology: the tendency to see one's own offences as smaller than others see them, the urge to protect self-image, the short-term reward of refusing, and the pessimistic belief that apology will not help anyway. Each of these forces is strongest in exactly the moments when apology is most needed. Knowing this suggests a few practical habits. When you feel the rush of self-justification, slow down and ask what the other person's account of events would sound like. When you feel the threat to your self-image, remind yourself that the admission concerns an act, not your worth. When you feel that apologising is pointless, remember that the evidence suggests others value apologies more, and respond to them more generously, than offenders tend to expect. And when you hear yourself reaching for "if," "but," or "I'm a terrible person," take that as a signal to start the sentence again. Hashtags: #TheArtOfTheApologyOnCampus #CampusCommunication #RelationshipMaintenance #EffectiveApologies #ConflictRepair #AcknowledgementOfResponsibility #OfferOfRepair #ExpressionOfRegret #DeclarationOfRepentance #RequestForForgiveness #TrustRepair #RestoringStanding #SharedRules #InterpersonalConflict #RoommateConflict #FriendshipRepair #GroupProjectCommunication #ProfessorStudentCommunication #NonApologies #SelfJustification #CognitiveDissonance #GuiltAndShame #Forgiveness #CommunicationSkills #FutureOfCampusRelationships

Download the Book (PDF): Introduction At three in the morning a fourteen-year-old with type 1 diabetes is asleep in a bedroom down the hall from her parents. Her glucose, which ran high after a late slice of pizza, has been drifting down for an hour. A sensor under the skin of her arm reports a new value every five minutes. A small pump on her abdomen receives each value by radio, and software inside it, or on a phone beside the bed, runs a calculation: where will her glucose be in half an hour, or an hour, if nothing changes? The answer comes back lower than the system likes, so it trims the insulin it was about to deliver. Five minutes later it asks again. Nobody wakes. By breakfast her overnight trace is a nearly flat line in the target range, and her mother, who for three years set an alarm to check her daughter's glucose at two and four every night, has slept through. That scene, now ordinary in many countries, is what people mean when they speak of an artificial pancreas. The clinical term is automated insulin delivery, or a closed-loop system: a glucose sensor, an insulin pump and a control algorithm that connects them, so that insulin dosing responds to glucose without a person making every decision. Since the first commercial hybrid system reached the United States market in 2016, these devices have moved from research wards into routine care. In the 2026 Standards of Care, the American Diabetes Association describes automated insulin delivery as the preferred insulin delivery method for all people with type 1 diabetes who can use it [33]. In England, the National Institute for Health and Care Excellence recommended hybrid closed-loop systems in December 2023 for all children and young people with type 1 diabetes, for those who are pregnant or planning pregnancy, and for adults whose glucose remains above target or who have disabling hypoglycemia [34]. The trial evidence behind this shift is strong and consistent. Across children, adults, older adults and pregnant women, closed-loop systems add roughly two to three and a half hours per day in the target glucose range compared with the best non-automated care, while holding or reducing time spent low. The gains are largest overnight, and they are largest of all for people whose control was worst to begin with. And yet the phrase "artificial pancreas" oversells what these machines are. People using them still count or estimate carbohydrates, still announce meals, still change infusion sets every two or three days and sensors every week or two, still respond to alarms, still manage exercise, illness and alcohol, and still end up in hospital with ketoacidosis when a cannula kinks unnoticed. In large real-world datasets, users of current systems typically spend about three quarters of the day in range, not all of it [18]. A healthy pancreas, by comparison, keeps glucose between roughly 70 and 140 mg/dL (3.9 to 7.8 mmol/L) almost all of the time, through meals, sprints, fevers and sleep, without being told anything. The argument of this book Why the gap? The answer usually offered is that the algorithms are still immature, or that sensors are not yet accurate enough. Both are partly true, but neither is the main reason. The central argument of this book is that the artificial pancreas is, at bottom, a machine for managing delay, and that almost everything about it, from the design of its algorithms to what it still asks of the person wearing it, follows from that one fact. A pancreatic beta cell senses glucose in the blood that bathes it and releases insulin within a minute or two directly into the portal vein, which carries it straight to the liver, the organ that matters most for glucose. A closed-loop system senses glucose in the fluid between cells under the skin, several minutes behind the blood. It then delivers insulin into fatty tissue under the skin, from which it is absorbed slowly into the general circulation, reaching peak effect an hour or more later and continuing to act for several hours after that. The controller is always acting on old information, and whatever it decides will not take full effect until the situation has changed again. A thermostat in a house with a furnace that took an hour to warm up, and kept blowing heat for four hours after it was switched off, would face the same problem. Once that is understood, the rest of the field comes into focus. Every commercial algorithm is a different strategy for acting under delay: by predicting ahead, by damping its own enthusiasm with estimates of insulin already on board, by being cautious on the way down and aggressive on the way up. The mealtime announcement that users find so burdensome is a way of buying back the time the insulin needs. The residual hypoglycemia after exercise, the hyperglycemia after a large unannounced meal, and the danger of an infusion site failure all trace back to the same slow pharmacology. The ways forward, too, sort neatly by which delay they attack: faster insulins, additional hormones such as glucagon and amylin analogs, smarter meal detection, or delivery routes that reach the liver more directly. How the book is organized The first chapter describes what a healthy pancreas actually does and why subcutaneous insulin, however it is dosed, cannot fully copy it. The second traces the long history of closing the loop, from bedside machines of the 1960s and 1970s to the pocket systems of today, and shows that the obstacles were sensing and safety rather than computing power. The third examines the glucose sensor, the system's only window on the body, and the metrics by which its data are judged. The fourth explains how the main families of control algorithms work and how the commercial systems differ. The fifth takes up the problem of delayed insulin absorption directly and shows why meals are the hardest part of the day for any controller. The second half of the book turns from mechanism to consequence. The sixth chapter weighs the trial evidence and real-world data, including the populations in which it is strongest and the one important question on which it proved disappointing. The seventh is about the lived experience of handing part of one's physiology to a machine: trust, alarms, workload and who gets access. The eighth deals with what happens when the loop fails, and with situations such as illness, pregnancy, exercise and hospital admission that stretch it. The ninth looks at the attempts to build a fully closed loop that needs no meal announcements, and at the competing prospect of biological replacement. The conclusion draws out what the argument implies for how clinicians should offer these systems, how users should be taught, and what remains genuinely unsettled. The book is written for clinicians and students who want to understand how these systems work rather than merely which buttons to press, and for people living with type 1 diabetes and their families who want the same. It is educational and does not replace local protocols, device instructions or the judgment of a diabetes team who knows the individual. Glucose values are given in mg/dL, with mmol/L alongside where precision matters; to convert, divide mg/dL by 18. A final note on words. "Artificial pancreas" is the popular name and a useful one, provided it is taken as an aspiration rather than a description. "Hybrid closed loop" is the more honest term for today's commercial systems: closed for background insulin and corrections, open at mealtimes, where the user still supplies the information the machine cannot get quickly enough on its own. "Automated insulin delivery" is the umbrella term now used in guidelines. These pages use all three, and the difference between them is, in a sense, the subject of the whole book. Chapter 1: What the Pancreas Does, and Why Injected Insulin Cannot Copy It Any machine built to replace an organ is judged against the organ, so the place to begin is with what the endocrine pancreas does when it works. The comparison is humbling, and it is meant to be. The islets of Langerhans make up only a small fraction of the pancreas by weight, a scattering of perhaps a million cell clusters through an organ otherwise devoted to digestive enzymes. Yet they hold blood glucose in a narrow band across a range of challenges that would defeat any engineered system now on the market. Understanding precisely how they do it explains why an artificial pancreas, built from a skin sensor and a skin pump, faces limits that no amount of clever software can fully remove. The islet at work The beta cell as sensor, computer and pump A beta cell is a sensor, a controller and an actuator folded into one cell. Glucose enters it through transporter proteins in proportion to its concentration in the blood. Inside, the enzyme glucokinase acts as the glucose sensor proper: it phosphorylates glucose at a rate that rises steeply across the physiological range, so that the cell's metabolic activity tracks the blood glucose concentration closely. As glucose is metabolized, the ratio of ATP to ADP rises, potassium channels in the cell membrane close, the membrane depolarizes, calcium floods in, and granules of stored insulin fuse with the cell surface and empty into the capillary blood. From a rise in glucose to the release of insulin takes about a minute. That is the first advantage the natural system has over any artificial one. It measures glucose in arterial blood, the fluid whose concentration actually determines what the brain and muscles receive, and it measures it continuously, not every five minutes. There is no interstitial fluid to diffuse through, no enzyme electrode to drift, no radio link to drop. The second advantage is the shape of the response. When glucose rises suddenly, insulin secretion comes in two phases. The first phase is a sharp spike within the first few minutes, drawn from granules already docked at the membrane and ready to go. The second phase is a slower, sustained rise that continues as long as glucose stays elevated. The first phase matters more than its brief duration suggests: it is largely responsible for switching off the liver's production of glucose early in a meal, which prevents the postprandial rise from ever getting large. Loss of first-phase secretion is one of the earliest detectable defects in people progressing toward both type 1 and type 2 diabetes, and when it is lost, glucose after a meal climbs higher and stays up longer even if total insulin output is normal. The beta cell also anticipates. Before a single carbohydrate molecule reaches the bloodstream, the sight, smell and taste of food trigger a small cephalic-phase insulin release through the vagus nerve. As food reaches the small intestine, cells in the gut wall release the incretin hormones GLP-1 and GIP, which amplify the beta cell's response to glucose. By the time glucose from the meal begins to appear in the blood, insulin is already rising. The natural pancreas, in other words, doses for the meal slightly before the meal arrives, using information that no glucose sensor can see. Finally, insulin is not secreted as a smooth stream but in pulses every few minutes. The pulsatility appears to keep the liver's insulin receptors responsive, and it disappears early in diabetes. Nothing in current closed-loop practice tries to reproduce it; the point is simply that the natural signal is richer than a single number rising and falling. The liver comes first The most underappreciated feature of normal insulin secretion is where it goes. Insulin from the pancreas enters the portal vein, which drains the gut and pancreas and runs directly to the liver. The liver is therefore exposed to insulin concentrations considerably higher than the rest of the body, and on this first pass it extracts and degrades roughly half of the insulin it receives. The rest of the body, muscle and fat included, sees a lower, filtered concentration. This arrangement matters because the liver is the central organ of glucose control. Between meals it releases glucose, from stored glycogen and from new glucose made out of lactate, amino acids and glycerol, at a rate that closely matches the brain's steady consumption. After a meal, rising portal insulin swiftly suppresses that output and tells the liver to store incoming glucose as glycogen. A large part of the normal postprandial response is not muscle soaking up glucose but the liver simply stopping its own production. And because it happens under high portal insulin, the rest of the body does not need to be flooded with insulin to achieve it. Subcutaneous insulin reverses this geometry. Insulin injected or infused under the skin is absorbed into peripheral capillaries and lymphatics and enters the general circulation. The liver receives only what reaches it through the hepatic artery and portal vein after mixing with the whole blood volume. To suppress hepatic glucose output adequately, peripheral insulin concentrations must be pushed well above normal. People with type 1 diabetes on any subcutaneous regimen therefore live with a relative insulin deficiency at the liver and a relative excess in the periphery. That peripheral excess contributes to weight gain and to the tendency of muscle, especially during exercise, to draw glucose out of the blood faster than the liver can replace it. No algorithm, however well tuned, can change where the insulin goes. It can only change how much and when. The counter-regulatory half of the system The beta cell is only half of the islet's work. Alpha cells in the same islets secrete glucagon, which acts on the liver in the opposite direction: it stimulates glycogen breakdown and new glucose production. In a healthy person, glucagon and insulin are regulated reciprocally. As glucose falls, insulin secretion drops first, then glucagon rises, then, if the fall continues, adrenaline, cortisol and growth hormone join in. The first defense against hypoglycemia is simply that the pancreas stops making insulin, and the insulin already in the portal circulation is cleared within minutes. In type 1 diabetes, all three layers of this defense are compromised. The first is lost outright: insulin given under the skin cannot be withdrawn once delivered, and it continues to act for hours regardless of what glucose does. The second is lost within a few years of diagnosis in most people: the glucagon response to falling glucose becomes blunted or absent, even though alpha cells remain present and can still secrete glucagon in response to other stimuli, sometimes inappropriately after meals, which worsens post-meal highs. The third is often degraded by repeated hypoglycemia itself. Each low episode resets the glucose threshold at which adrenaline and warning symptoms appear, so that people with frequent lows lose their early warning, a condition called impaired awareness of hypoglycemia. These people are at the greatest risk of severe hypoglycemia, in which they need another person's help to recover. This is why the artificial pancreas problem is asymmetric. Too little insulin produces high glucose, which is harmful over months and years and dangerous over hours only when insulin is almost completely absent. Too much insulin produces low glucose, which can cause confusion, seizures or death within an hour, and which the body can no longer reliably correct. A controller for type 1 diabetes has an accelerator and no brake. When it overshoots on the way down, the only remedies are to stop delivery and wait for the insulin already in the tissue to wear off, or for the person to eat. A third hormone: amylin Beta cells co-secrete a second peptide with insulin, called amylin, in a fixed ratio. Amylin slows the emptying of the stomach, suppresses the inappropriate rise in glucagon after meals, and contributes to fullness. In type 1 diabetes amylin is lost along with insulin. The consequence is that food leaves the stomach faster than it would in someone without diabetes, glucose from a meal appears in the blood sooner, and glucagon rises when it should fall. The postprandial problem in type 1 diabetes is thus not only that insulin arrives late but also that glucose arrives early. An injectable amylin analog, pramlintide, has existed for two decades, and the idea of combining it with insulin in a closed loop, discussed in the final chapter, rests directly on this piece of physiology. What is lost, and what replaces it What goes wrong in type 1 diabetes Type 1 diabetes is an autoimmune disease in which T cells destroy the beta cells, usually over months to years. By the time of diagnosis most people have lost the majority of their functional beta-cell mass, and in most people the remaining secretion dwindles over the following years, though small amounts of residual insulin production, measurable as C-peptide, persist in a surprising number of people with long-standing disease. Residual C-peptide matters clinically: people who retain some have fewer severe hypoglycemic episodes and more stable glucose. That observation motivated one of the most interesting closed-loop trials, discussed in the sixth chapter, which asked whether tight glucose control from diagnosis could protect the surviving beta cells. It could not. Without insulin, cells cannot take up glucose efficiently and the liver produces it unchecked, so glucose rises. At the same time, fat is broken down, and the liver converts the resulting fatty acids into ketone bodies. Ketones accumulate, the blood becomes acidic, and the person develops diabetic ketoacidosis, a medical emergency. This happens quickly when insulin is absent entirely. In a person using an insulin pump, who has no long-acting insulin depot and relies wholly on a continuous trickle of rapid-acting insulin, a blocked or dislodged cannula can lead to rising ketones within a few hours. That vulnerability, inherent in pump therapy and therefore in every current closed-loop system, is the subject of the eighth chapter. The long-term cost of hyperglycemia was established decisively by the Diabetes Control and Complications Trial, published in 1993 [1]. Among 1,441 people with type 1 diabetes followed for an average of six and a half years, intensive therapy that lowered average glucose substantially reduced the development and progression of retinopathy, kidney disease and neuropathy. Follow-up of the same cohort later showed lasting reductions in cardiovascular events. But intensive therapy in that trial also roughly tripled the rate of severe hypoglycemia. The DCCT defined the dilemma that the artificial pancreas exists to resolve: lower glucose protects the eyes, kidneys, nerves and heart, but with the tools of the time, lowering glucose meant more frequent and more dangerous lows. For thirty years, clinicians and patients have negotiated a compromise between these two harms. The promise of closed-loop control is to shift that compromise, to lower average glucose without paying for it in hypoglycemia. Why subcutaneous insulin is a different drug It helps to be precise about why insulin under the skin behaves so differently from insulin secreted by the pancreas. Human insulin in solution naturally associates into hexamers, clusters of six molecules held together with zinc. A hexamer is too large to pass quickly through capillary walls, so after an injection it must first dissociate into dimers and then monomers before it can be absorbed. This is why regular human insulin, injected under the skin, takes thirty to sixty minutes to begin working and reaches its peak two to three hours later. The rapid-acting analogs introduced from the mid-1990s, lispro, aspart and glulisine, were engineered with small changes in amino acid sequence that weaken self-association, so they dissociate faster and are absorbed sooner. According to the American Diabetes Association's summary, they begin to act in about fifteen minutes, peak at about one to two hours, and continue to act for about two to four hours. In practice, pharmacodynamic studies show a long tail: a meaningful fraction of the glucose-lowering effect of a bolus is still present four or five hours after it was given, which is why pump settings for "active insulin time" are usually set longer than the label's nominal duration. Newer formulations, faster aspart (with added niacinamide) and ultra-rapid lispro (with treprostinil and citrate to increase local blood flow and permeability), shave several minutes off the early absorption, but they do not change the basic shape. Insulin under the skin still takes the better part of an hour to do most of its work and hours more to finish. Absorption is also variable. The same dose given into the same person on different days can be absorbed at noticeably different rates. Site matters: the abdomen tends to absorb faster than the thigh. Local blood flow matters: heat, massage and exercise of the underlying muscle speed absorption, and cold slows it. Tissue quality matters: repeated use of the same area produces lipohypertrophy, rubbery thickened fat in which insulin absorption is erratic and slower. Even the age of the infusion site matters; absorption from a pump cannula changes over the two to three days of its wear. A controller can learn a person's average insulin action, but it cannot see this day-to-day scatter until it shows up, too late, in the glucose trace. A moving target One further complication deserves mention because every controller must cope with it: the amount of insulin a person needs is not constant. Insulin sensitivity changes across the day. Many people with type 1 diabetes need more insulin in the early morning, when growth hormone and cortisol released during the night raise glucose, the so-called dawn phenomenon, and less in the afternoon. Sensitivity rises for many hours after exercise, as muscles replenish their glycogen, and falls during infection, with corticosteroid treatment and under psychological stress. In women it can shift across the menstrual cycle, with many needing more insulin in the days before menstruation. During puberty, growth hormone and sex steroids can raise insulin needs by a third or more, and needs change again through pregnancy, rising steeply in the second and third trimesters and falling abruptly after delivery. A healthy beta cell does not need to know any of this. It responds to glucose, and whatever the body's current sensitivity, it secretes as much insulin as glucose demands. A dosing system that works from fixed settings, a basal rate, a carbohydrate ratio, a correction factor, will be wrong whenever sensitivity changes, and it will stay wrong until someone notices and adjusts the settings. Much of the tedium of manual pump therapy was exactly this: noticing patterns over days, guessing at their causes and changing settings in small steps. A closed loop that responds to glucose rather than to fixed settings handles slow changes in sensitivity naturally, which is one of its main advantages. But because it acts through a delay, it handles abrupt changes, such as the sudden sensitivity of the hours after a long hike, less well. What the machine is up against Set side by side, the differences look like this. The natural system senses glucose in blood, instantly, and continuously; the artificial one senses it in interstitial fluid, several minutes behind, and every five minutes. The natural system anticipates meals through nervous and gut signals; the artificial one learns about meals only when glucose starts to rise, unless the person tells it. The natural system delivers insulin to the liver first, in pulses, within a minute or two of deciding to; the artificial one delivers it to fat under the skin, from which it reaches the general circulation over an hour and lingers for several more. The natural system can switch insulin off and see it cleared within minutes; the artificial one can only stop adding more. The natural system has glucagon and amylin to shape the response in both directions; nearly all artificial systems have only insulin. Engineers describe a system like this as having large "transport delays" and "actuator saturation": the controller's actions take effect late, and it cannot push in both directions. They know that such systems cannot be controlled tightly by feedback alone. Any controller that reacts strongly to current error will overshoot, because by the time its action takes effect, the situation it was reacting to has changed. Any controller that reacts gently will be too slow to stop large disturbances, such as a meal. The only ways to do better are to predict disturbances before they happen, to act before they are seen, or to shorten the delays themselves. That is the frame for the rest of this book. The sensor chapter examines the delay and error on the input side. The algorithm chapter examines how controllers predict and hedge. The chapter on insulin absorption examines the delay on the output side, and why meals remain the weak point. What the trials show, how people experience these systems and where the field is headed can then be read against the same frame. The artificial pancreas is not failing to be a pancreas because its engineers lack ambition. It is doing something genuinely hard: controlling a fast, dangerous, asymmetric variable through a slow, one-directional actuator with a delayed sensor. That it succeeds as well as it does is the real achievement, and the remaining gap is a map of exactly where the next advances must come from. Chapter 2: Closing the Loop: From the Bedside Machine to the Pocket The idea of an artificial pancreas is almost as old as the ability to measure glucose automatically. The engineering principle, measure a variable, compare it with a target, adjust an input, is the same one that runs a thermostat or a ship's autopilot, and it was well understood by the 1950s. Yet more than half a century separated the first working closed-loop glucose controller from the first one a person could wear home. The history is worth telling because it shows that the bottlenecks were never really about computing. They were about getting a trustworthy glucose signal from a living body, delivering insulin by a route that did not require a needle in a vein, and proving that an automated system could be allowed to dose a drug capable of killing the person wearing it. Early machines Intravenous beginnings In 1964 Arnold Kadish, a Los Angeles physician, described a "servomechanism for glucose monitoring and control" [2]. Blood was drawn continuously from a vein and analyzed by an automated chemistry instrument; when glucose went above a set point, the device infused insulin intravenously, and when it went below, it infused glucagon. It worked in principle, but the apparatus was the size of a large backpack and its control logic was simply on or off. It was a proof that the loop could be closed, not a treatment. A decade later, a team in Toronto led by Michael Albisser and Bernard Leibel reported what they called an artificial endocrine pancreas [3]. It too sampled venous blood continuously and infused insulin or dextrose intravenously, but its dosing was governed by mathematical algorithms, run on a small computer, designed to imitate the beta cell's response to both the level of glucose and its rate of change. It was tested first in pancreatectomized dogs and then in people. The Toronto group's conceptual contribution endured: they framed the task not as switching insulin on and off but as reproducing a physiological dose-response relationship that included anticipation of where glucose was heading. In the late 1970s the Biostator, a commercial bedside instrument made by Miles Laboratories, brought this design into hospitals and research units. The Biostator withdrew blood continuously through an intravenous line, measured glucose every minute with a glucose oxidase sensor, and infused insulin or dextrose intravenously according to algorithms derived from the Toronto work. It could hold glucose in the normal range for hours or days, and it became a workhorse of metabolic research, used for decades in glucose clamp studies of insulin action. But it required a double intravenous line, a trained operator, frequent recalibration, and a machine the size of a small refrigerator. It showed that a computer could control glucose if it had fast, accurate blood measurements and a fast intravenous route for insulin. The problem was that neither of those could be carried into daily life. The long detour through open-loop therapy For the next three decades, practical diabetes care moved in a different direction. Portable insulin pumps appeared in the late 1970s and became progressively smaller and more reliable through the 1980s and 1990s. Blood glucose meters for home use made fingerstick testing routine. The DCCT, completed in 1993, proved that intensive therapy, whether by multiple daily injections or pump, prevented complications, and it made tight control the standard of care [1]. But these were open-loop tools. The patient was the controller. Every dose was a human decision informed by a few fingerstick readings a day, carbohydrate estimates, and rules of thumb for correction. The problem with the human as controller is not intelligence; it is bandwidth. A person who tests four times a day samples glucose at intervals of several hours, in a system whose important dynamics unfold over tens of minutes. Overnight, most people are not sampling at all. Nocturnal hypoglycemia was, and remains, one of the most feared aspects of type 1 diabetes, partly because it can occur undetected during sleep. The "dead in bed" syndrome, sudden unexplained death of young people with type 1 diabetes found in an undisturbed bed, is rare, but it haunts parents and has been linked in some cases to nocturnal hypoglycemia. Any real advance had to address the night. Building the modern loop The sensor arrives The decisive technical change was the continuous glucose monitor. The first device approved in the United States, in 1999, was a "professional" monitor worn for a few days whose data were downloaded and reviewed afterward; the wearer could not see readings in real time. Real-time monitors, displaying a new reading every few minutes and able to alarm at high and low thresholds, reached the market in the mid-2000s. Early sensors were bulky, lasted three days, needed several fingerstick calibrations a day, and were inaccurate enough that many people abandoned them. But for the first time a stream of glucose data was available from a person going about ordinary life, and the stream could be fed into an algorithm. Around the same time, the research foundation now called Breakthrough T1D (then the Juvenile Diabetes Research Foundation) launched an artificial pancreas project that funded academic groups in the United States, the United Kingdom and elsewhere and pushed regulators to develop a pathway for approval. Three developments followed that made the modern field possible. The first was a simulator. Testing a new control algorithm had traditionally required animal studies before any human work. In 2007 Claudio Cobelli's group in Padova and Boris Kovatchev's group at the University of Virginia built a detailed mathematical model of glucose-insulin dynamics and populated it with 300 virtual subjects representing the variability of real people with type 1 diabetes. In January 2008 the United States Food and Drug Administration accepted this simulator as a substitute for animal trials in the preclinical testing of insulin-dosing algorithms [4]. That decision was quietly revolutionary. Algorithms could now be designed, tuned and stress-tested against thousands of simulated days, including extreme scenarios no ethics committee would allow, before any person wore them. Much of the subsequent progress in algorithm design was made possible by this ability to fail cheaply in silico. The second was a series of carefully controlled inpatient studies. In Cambridge, Roman Hovorka's group had developed a model predictive controller in the early 2000s [10]. In a randomized crossover trial in children and adolescents published in 2010, they showed that overnight closed-loop control, with glucose readings from a subcutaneous sensor and insulin by a subcutaneous pump, increased time in target and reduced time spent low compared with the children's usual pump therapy [5]. The loop in that study was closed manually, by a nurse transcribing each sensor reading into a laptop and each dosing recommendation into the pump. But the principle, fully subcutaneous closed loop, had been proven in the setting where it mattered most. The third was the move from the clinical research unit to the real world. Groups at Virginia, Cambridge, Boston, Montpellier, Padova and elsewhere built portable platforms, often smartphones running control algorithms and communicating wirelessly with sensor and pump, and tested them first in hotel rooms and diabetes camps, then at home. The Boston group led by Edward Damiano and Steven Russell, working with Firas El-Khatib, reported in 2014 an outpatient bihormonal "bionic pancreas" delivering both insulin and glucagon, tested in adults and adolescents over five days of normal life [6]. Across these studies, a consistent pattern emerged: closed-loop control was safe, it improved time in range, and its benefit was greatest overnight. From simulation to home: a staged path The research of this period established a staged path that every new algorithm still follows, and it is worth describing because it explains both the caution of the field and its speed. An algorithm is first developed and tested in simulation, where it can be run through thousands of virtual days, including missed meals, sensor failures, sudden exercise and extreme insulin sensitivity. It then moves to closely supervised inpatient studies in a clinical research facility, with frequent blood sampling and staff ready to intervene. Next come supervised outpatient studies, in hotels, camps or research houses, where participants eat real meals and walk around town but remain close to the study team. Only then does the system go home, first for days or weeks with remote monitoring and then for months in a randomized trial. Each stage answers a different question. Simulation shows whether the logic is sound and where it breaks. Inpatient studies show whether the system behaves in real bodies as it did in virtual ones. Supervised outpatient studies show whether it copes with ordinary life. Home studies show whether people will use it and whether it stays safe without a researcher at hand. Simulation has limits that the later stages expose. Virtual patients are only as varied as the data used to build them, and early simulators represented exercise, illness, stress and the changing absorption of an aging infusion site poorly or not at all. An algorithm that looked excellent in silico could perform disappointingly when a teenager played football after lunch. Part of the history of the field since 2008 has been the gradual enrichment of the simulators with these real-world effects, informed by the very trials they were used to design. This staged approach also shaped the regulatory conversation. Regulators could see a chain of evidence from mathematical model to home use, with each link tested separately, and it became possible to argue that an automated dosing system could be at least as safe as the human decisions it replaced. That argument, more than any single trial, is what eventually allowed closed-loop systems to be approved for home use. Safety first: suspending insulin Before regulators would allow a device to increase insulin automatically, they approved devices that could only decrease it. In 2013 the FDA approved a Medtronic pump with a "threshold suspend" feature, which stopped insulin delivery when the sensor glucose reached a preset low and the user did not respond to the alarm. The next generation, "predictive low glucose suspend," stopped insulin when glucose was forecast to go low within the next half hour and resumed it automatically when glucose recovered. These systems had modest effects on average glucose but meaningfully reduced the time spent low, especially overnight. These early systems are easy to overlook now, but they were an essential step. They demonstrated that a device could act on sensor data without a human in the loop, and they generated a regulatory precedent. They also illustrated the asymmetry described in the previous chapter. Suspending insulin is the one action a controller can take to prevent a low, and its effect is delayed by the insulin already on board. Predictive suspension works better than threshold suspension precisely because it acts earlier. Commercial systems and the community The first hybrid closed loop In September 2016 the FDA approved the Medtronic MiniMed 670G, the first commercial system that both increased and decreased insulin automatically. It was a hybrid closed loop: in its "Auto Mode" it adjusted basal insulin every five minutes according to sensor glucose, aiming at a fixed target of 120 mg/dL, while the user still had to bolus for meals by entering carbohydrate amounts. The pivotal trial enrolled 124 adolescents and adults, who used the system at home for three months [7, 8]. Glycated hemoglobin (HbA1c) fell from 7.7% to 7.1% in adolescents and from 7.3% to 6.8% in adults, and time in range rose, with no severe hypoglycemia or ketoacidosis. The trial had no control group, and its population was already well controlled, but it was enough for approval. The 670G proved that closed-loop control could be commercialized. It also exposed how much the human experience of a device matters. Its sensor required at least two fingerstick calibrations a day. Its algorithm was conservative and, when it lost confidence in the sensor or had delivered at its maximum or minimum rate for too long, it exited Auto Mode and required the user to intervene, often at night. In a prospective study of patients starting the system in routine care at Stanford, a third had stopped using Auto Mode within a year [28]. The most common reason was problems with the sensor. The lesson, which later systems took to heart, was that a closed loop is only as usable as its least usable component. We are not waiting While commercial systems moved through trials and regulatory review, a community of people with type 1 diabetes and their families decided not to wait. Beginning around 2013, under the banner #WeAreNotWaiting, programmers with diabetes reverse-engineered the communication protocols of their own pumps and sensors so that they could view glucose data remotely and, eventually, send commands to their pumps. Dana Lewis, a Seattle woman with type 1 diabetes, and Scott Leibrand built a system they later released as OpenAPS, the Open Artificial Pancreas System, running on a small computer that talked to an older model of pump [9]. Other projects followed: Loop, running on an iPhone; AndroidAPS, running on Android phones. The code was published openly. Anyone who could obtain compatible hardware and follow detailed documentation could build and run a closed loop, and thousands did. These open-source systems were not approved by any regulator, and many clinicians were initially uneasy about supporting them. But they were built by people with a direct stake in their safety, they were transparent in a way commercial algorithms were not, and they were highly configurable. They introduced features, such as small automatic correction boluses and algorithms that detected unannounced meals from the shape of the glucose rise, that commercial systems later adopted in their own forms. In 2022 a randomized trial in New Zealand, discussed in the sixth chapter, showed that an open-source system was as effective as the commercial ones [21]. In January 2023 the FDA cleared a version of Loop developed by the nonprofit Tidepool, bringing the community's work into the regulated world. The 2026 ADA Standards now explicitly support people choosing open-source systems and drop the older "do-it-yourself" label [33]. Interoperability and the modern generation A final regulatory development shaped the present landscape. Rather than approving each closed-loop system as a single fixed package of sensor, pump and algorithm, the FDA created categories for components that could be combined: in 2018 the integrated continuous glucose monitor (iCGM), a sensor meeting accuracy standards strict enough to feed an automated system; in 2019 the alternate controller enabled (ACE) infusion pump, a pump designed to take commands from an external algorithm; and, also in 2019, the interoperable automated glycemic controller, the algorithm itself. The first algorithm authorized in that category was Tandem's Control-IQ, derived from the University of Virginia's research controller, approved in December 2019 on the strength of a randomized trial published the same year [17]. The years since have brought a steady stream of systems. Insulet's Omnipod 5, the first tubeless patch-pump closed loop, was authorized in the United States in early 2022. Medtronic's MiniMed 780G, an advanced hybrid system with automatic correction boluses and a lower target than the 670G, reached the United States in 2023 after several years in Europe. Beta Bionics' iLet, the commercial descendant of the Boston bionic pancreas, cleared in May 2023, needs only the user's body weight to start and asks only that meals be announced as smaller, typical or larger than usual. CamAPS FX, the Cambridge algorithm running as a smartphone app, was already in use across Europe and the United Kingdom, including in pregnancy, and received FDA clearance in 2024. Indications have widened to include children as young as two, and since 2024 the leading systems have been authorized for adults with type 2 diabetes as well. The pattern across this history is consistent. The mathematics of control was ready decades before the hardware. What held the field back was a glucose signal good enough to act on, a delivery route convenient enough to live with, and a regulatory framework able to trust software with a potentially lethal drug. Each of those was solved, imperfectly, in the fifteen years after 2005. What was not solved, because it could not be solved by engineering alone, is the slowness of subcutaneous insulin. That is the physiological ceiling under which every system described in the following chapters still operates. Chapter 3: What the Machine Sees: Continuous Glucose Sensing A closed-loop controller knows nothing about the body except what its glucose sensor tells it. It cannot see the meal on the plate, the stairs being climbed, the fever developing, or the dose absorbed faster than usual from a warm abdomen. It sees a number every five minutes, together with whatever the user chooses to tell it. The quality of that number, how accurate it is, how noisy, how late, and how it fails, therefore sets the upper limit on what any algorithm can achieve. Seen from the controller's side, the sensor is an imperfect, delayed and occasionally misleading witness, and it is worth understanding exactly how. The sensor and its errors How a sensor measures glucose Nearly all current continuous glucose monitors work on the same electrochemical principle. A thin, flexible filament a few millimeters long is inserted under the skin, usually of the upper arm or abdomen, where it sits in the interstitial fluid that bathes the cells. The filament is coated with the enzyme glucose oxidase, held in place by membranes that control how much glucose and oxygen reach it. Glucose oxidase converts glucose and oxygen into gluconolactone and hydrogen peroxide; at the electrode, the hydrogen peroxide, or in some designs an intermediary molecule, is oxidized, generating a tiny electrical current proportional to the local glucose concentration. A transmitter on the skin measures the current, converts it to a glucose estimate using an algorithm, and sends the result by Bluetooth to a phone, a receiver or a pump. One important exception exists. A fully implantable sensor, placed under the skin of the upper arm in a minor office procedure, uses a fluorescent chemical that binds glucose and changes its light emission; an adhesive transmitter worn over the implant reads it. Its newest version lasts a year. It is compatible with some closed-loop systems, but the great majority of users wear the enzymatic type, typically lasting ten to fifteen days. Each design must deal with the same problems. The enzyme and membranes change over time. The body reacts to the filament as a foreign object, surrounding it over days with inflammatory cells that consume glucose and oxygen locally and alter the signal. The electrical signal is noisy. And the thing measured is not blood glucose but interstitial glucose, a related but distinct quantity. The lag between blood and tissue Glucose reaches the interstitial fluid by diffusing out of capillaries. When blood glucose is stable, interstitial glucose is close to it. When blood glucose is changing, interstitial glucose follows behind. Direct measurements using glucose tracers, injected into a vein and then sampled in the subcutaneous tissue of the abdomen, have quantified the physiological delay. In healthy fasting adults it was about five to six minutes [13]; in fasting adults with type 1 diabetes it was a median of about seven minutes, with all participants' tracers detectable in the tissue within ten minutes [14]. To the physiological lag, the device adds its own. Sensors filter their raw signal to remove noise, and filtering introduces a small additional delay. Most sensors used in closed loops report a value every five minutes, so on average the latest reading is a couple of minutes old even before the controller acts on it. Manufacturers have reduced the effective delay over time, in part by using algorithms that project slightly forward. But the controller should be thought of as seeing, at any moment, where glucose was about five to fifteen minutes ago. Is that a large delay? In steady conditions, no. Overnight, when glucose changes slowly, a ten-minute delay introduces an error of a few mg/dL. But after a meal, glucose can rise at 2 to 3 mg/dL per minute, and during vigorous exercise it can fall at similar speeds. At those rates, a ten-minute lag means the sensor reads 20 to 30 mg/dL behind the blood. That matters in two ways. A rising glucose is recognized late, so the controller starts adding insulin late. A falling glucose is recognized late, so the controller suspends insulin late, and the insulin it has already delivered will continue to act for hours. The researchers who measured the lag concluded, reasonably, that it should not be an obstacle to closed-loop control in fasting or overnight conditions. It is the fast transitions, meals and exercise, where it bites. Accuracy and its measurement Sensor accuracy is usually summarized by the mean absolute relative difference, or MARD: the average percentage difference between the sensor reading and a simultaneous laboratory reference measurement. Early sensors in the mid-2000s had MARDs in the mid to high teens; current sensors used with closed-loop systems report figures in the high single digits to low double digits under clinical study conditions. A lower MARD is better, but it is an average that conceals what the controller most cares about: how the sensor performs in the low range, how it behaves on the first day of wear, and how often it produces large errors. For this reason, the FDA's iCGM standard, introduced in 2018, set requirements not only for average accuracy but for accuracy within specific glucose ranges, for the proportion of readings far from the reference, and for the rarity of readings that would suggest the opposite of the truth, such as a normal reading when glucose is actually low. A sensor qualified as iCGM is certified to be accurate enough to drive automated dosing, and modern closed-loop systems do not require fingerstick calibrations, although most allow them. Several predictable sources of error persist, and users and clinicians need to know them because the controller does not. The first day. Sensor accuracy is usually worst in the first twelve to twenty-four hours after insertion, while the tissue settles around the filament. Some users insert a new sensor a day before the old one expires, so the new one has bedded in before the loop depends on it. Compression. When a person lies on the sensor, pressure reduces blood flow to the tissue around the filament; local glucose falls as cells continue to consume it, and the sensor reports a false low, often a steep drop followed by a sudden recovery when the person rolls over. For a closed loop this is not harmless. The controller suspends or reduces insulin in response to the false low, and the person may wake a few hours later with glucose that is genuinely high. Interfering substances. Certain drugs and supplements interfere with the electrochemistry of particular sensors. Reported examples include hydroxyurea and, for some sensor designs, paracetamol (acetaminophen) at higher doses, which can falsely raise readings, and large doses of vitamin C, which falsely raise readings on others. The specific list differs by device and is in each sensor's labeling. A falsely high reading is more dangerous for a closed loop than for a person reading a monitor, because the loop may respond by delivering more insulin. Physiology at the extremes. At very low glucose, and during rapid change, the relationship between interstitial and blood glucose is least stable. Sensors tend to be less accurate exactly where accuracy matters most. Dehydration, poor perfusion and critical illness. When peripheral circulation is compromised, as in shock, severe dehydration or during vasopressor treatment, interstitial glucose may not reflect blood glucose at all. This is one reason why closed-loop systems are not currently recommended in critical illness. A worked example: the lag during a run To see what the lag means in practice, consider a man who sets out for a forty-minute run in the late afternoon with glucose at 160 mg/dL and a moderate amount of insulin on board from a snack bolus an hour earlier. Twenty minutes in, his blood glucose is falling at about 2 mg/dL per minute and has reached 120. His sensor, reading ten minutes behind, still shows about 140, and it shows the fall as gentler than it is, because the sensor's own filtering smooths the change. His closed loop, seeing 140 and a modest downward trend, reduces insulin somewhat but does not suspend. At thirty minutes his blood glucose is about 100 and the sensor shows 120. By now the controller's forecast predicts a low and it suspends insulin, but the insulin delivered over the previous two hours is still being absorbed, and exercising muscle is taking up glucose at a high rate. At the end of the run his blood glucose is near 80, still falling, and the sensor reads around 100. The low that the controller is trying to prevent arrives ten or twenty minutes after the run ends, and it arrives because of decisions made before he laced up his shoes. None of the numbers in this illustration is extreme, and no component has failed. The sensor is accurate by its specifications and the controller has responded exactly as designed. The low is produced by the combination of sensor lag, the controller's reasonable reluctance to act on a single downward reading, and the long tail of insulin already delivered. The remedy, discussed in the fifth chapter, lies not in the sensor but in acting earlier, by setting an exercise target an hour or two before the run so that the insulin on board has already fallen by the time the fall begins. What fingersticks are still for Most current systems neither require nor encourage routine fingerstick checks, and for most of the day the sensor is the better guide, because it shows trend as well as level. But fingersticks retain three roles that every closed-loop user should know. The first is to check a reading that does not fit symptoms or circumstances, for example a normal sensor reading when the person feels clearly hypoglycemic, or a sudden unexplained high. The second is to confirm glucose before treating a low or giving a large manual correction when the sensor is in its first day or has been behaving erratically. The third is as a backup when the sensor fails or is out of supply. A meter with unexpired strips remains part of the essential kit. Noise, signal loss and what the controller does about them Besides systematic errors, sensors produce random noise: small fluctuations from one reading to the next that do not reflect real changes. Controllers must distinguish noise from signal. A controller that reacted fully to every five-minute fluctuation would oscillate, adding and withdrawing insulin in response to nothing. So every algorithm filters its input, whether explicitly, by smoothing recent readings, or implicitly, by using a model that makes implausible jumps unlikely. Filtering adds delay, and the designer must balance responsiveness against stability. This is an early example of a theme that recurs throughout the book: in a delayed system, every protection against error costs some speed. Sensors also sometimes stop reporting: the Bluetooth connection drops, the phone is left in another room, or the sensor signals that its readings are unreliable. Controllers handle missing data conservatively. A short gap may be bridged by predicting from recent trend; a longer one typically returns the pump to a preprogrammed basal rate or a "safe" basal amount, and eventually to manual mode. How often this happens, and how gracefully a system handles it, is a large part of what users experience as reliability. Time in range: how the loop's output is measured Continuous sensing changed not only how insulin is dosed but how glucose control is judged. For decades, HbA1c, which reflects average glucose over roughly the preceding two to three months, was the single measure of control. It predicts complications well, but it says nothing about variability or hypoglycemia. Two people can have identical HbA1c values, one with steady readings near 150 mg/dL and the other swinging between 50 and 300. Continuous monitoring allows a richer description. In 2019 an international consensus panel convened at the Advanced Technologies and Treatments for Diabetes congress agreed on standard metrics and targets [15]. The headline measure is time in range: the percentage of sensor readings between 70 and 180 mg/dL (3.9 to 10.0 mmol/L). Alongside it are time below range, divided into levels below 70 mg/dL and below 54 mg/dL (3.0 mmol/L), and time above range, divided into levels above 180 and above 250 mg/dL (13.9 mmol/L). Different targets apply to older or high-risk people, for whom avoiding hypoglycemia takes priority, and to pregnancy, where a tighter range of 63 to 140 mg/dL (3.5 to 7.8 mmol/L) is used, as Table 1 sets out. Table 1. Consensus targets for continuous glucose monitoring metrics. Population Time in range Time below range Time above range Most adults with type 1 or type 2 diabetes 70–180 mg/dL: more than 70% Below 70: less than 4%; below 54: less than 1% Above 180: less than 25%; above 250: less than 5% Older or high-risk adults 70–180 mg/dL: more than 50% Below 70: less than 1% Above 250: less than 10% Pregnancy with type 1 diabetes 63–140 mg/dL: more than 70% Below 63: less than 4%; below 54: less than 1% Above 140: less than 25% Source: Battelino et al., International Consensus on Time in Range, Diabetes Care 2019 [15]. Time in range relates to HbA1c, but loosely. In an analysis of 545 adults with type 1 diabetes from four trials, a time in range of 70% corresponded on average to an HbA1c of about 7%, and 50% to about 8%; each 10-percentage-point increase in time in range corresponded to an average decrease in HbA1c of about 0.6 percentage points, though with wide scatter between individuals [16]. A 10-point increase in time in range is 2.4 hours a day. This conversion is useful for reading closed-loop trials, most of which report time in range as their primary outcome. When a trial reports a gain of 11 percentage points, it means about two and a half more hours each day spent in the target range. The consensus also recommended that at least 70% of possible readings be captured over fourteen days for the metrics to be reliable, and that glucose variability, expressed as the coefficient of variation, be kept at or below 36%. Standardized reports, the ambulatory glucose profile, now summarize these metrics and overlay days to show the typical daily pattern. For closed-loop users these reports are the primary way clinicians review therapy, and they show the time spent in automated mode as well, a figure that says a good deal about how well a person is getting on with a system. What the sensor cannot tell the controller The most important limitation of the sensor is not an error but a blind spot. The sensor reports glucose, and only glucose. It tells the controller nothing about what is about to happen to glucose. A meal eaten now will show up in the sensor fifteen or twenty minutes later, as glucose from the gut appears in the blood and then diffuses into the tissue. A run started now will begin to lower glucose within minutes, but the sensor will show it only when the fall has begun, and the insulin delivered in the hour before the run is already committed. A healthy beta cell uses additional channels to see these events coming: nerve signals from the head and gut, incretin hormones, adrenaline. The closed loop has only what the user tells it, whether a meal announcement, a carbohydrate estimate, or an exercise mode. Experiments have tested additional inputs, such as heart rate and accelerometry from wearable devices, to recognize exercise automatically. They help somewhat, but none is yet part of a mainstream commercial system's core algorithm. The sensor therefore imposes a double constraint: its own lag, measured in minutes, and its blindness to the future, which, combined with the slowness of subcutaneous insulin, is measured in hours. How the algorithms try to see around that blindness is the subject of the next chapter. Hashtags: #TheArtificialPancreas #Type1Diabetes #AutomatedInsulinDelivery #ClosedLoopSystems #HybridClosedLoop #ContinuousGlucoseMonitoring #InsulinPumpTherapy #ControlAlgorithms #GlucosePrediction #InsulinOnBoard #ModelPredictiveControl #InterstitialGlucose #SensorLag #SubcutaneousInsulin #InsulinAbsorption #TimeInRange #HypoglycemiaPrevention #PredictiveLowGlucoseSuspend #MealAnnouncement #ExerciseManagement #BihormonalSystems #Glucagon #Amylin #OpenSourceArtificialPancreas #FutureOfAutomatedDiabetesCare

Download the Book (PDF): Introduction Exponential technologies are fundamentally altering global labour markets and macroeconomic structures. Brynjolfsson and McAfee's definitive text charts this disruption, but its massive scope — covering everything from the digitization of goods to the decoupling of productivity and employment — can easily overwhelm students attempting to synthesise a cohesive capstone thesis on digital transformation. This companion acts as your macroeconomic filter. It is explicitly written to explain The Second Machine Age, isolating the authors' complex economic models and translating them into clear, citable academic prose. We rigorously define the mechanics of "bounty" (the wealth generated by digital innovation) and "spread" (the resulting inequality). This guide gives you the exact terminology needed to critically analyse the future of corporate strategy and global labour. The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies was published by W. W. Norton in January 2014. Its authors are colleagues at the Massachusetts Institute of Technology: Erik Brynjolfsson, then director of the MIT Initiative on the Digital Economy and now a professor at Stanford and director of the Stanford Digital Economy Lab, and Andrew McAfee, a principal research scientist at MIT Sloan. Two years earlier they had self-published a short, urgent book called Race Against the Machine, which argued that the weak employment recovery after the 2008 financial crisis was not only cyclical but structural — that technology was destroying certain kinds of work faster than it was creating replacements. The Second Machine Age is the fuller, more ambitious treatment of that claim. Three years after it, in Machine, Platform, Crowd (2017), the same authors turned to the managerial implications of the same forces. The middle book is the one that entered the economics curriculum, the policy debate, and the vocabulary of practitioners. Its argument can be stated compactly. The Industrial Revolution was a rupture in human history because the steam engine overcame the limits of muscle power — human and animal — and in doing so bent the curve of population, income and welfare sharply upward after millennia of near-flatness. Digital technology, Brynjolfsson and McAfee argue, is a rupture of comparable magnitude, because it does for mental power what steam did for physical power. That is the first machine age and the second. The second is driven by three interacting forces. Computing capacity improves exponentially rather than incrementally, and because exponential growth compounds, its most dramatic effects arrive late — the authors borrow the ancient parable of the chessboard and the doubling grains of rice to make the point. Digital goods are non-rival and cost almost nothing to reproduce, so more and more of what the economy produces behaves unlike wheat or steel. And innovation is recombinant: new ideas are built by assembling existing ideas, so the stock of available building blocks grows faster than it is consumed. The consequences split into two streams, and the book's central analytical contribution is insisting that they are genuinely separate. The bounty is the growth in the volume, variety and quality of what the economy produces — a good thing, and larger than conventional statistics register, because free digital goods generate consumer surplus that never enters gross domestic product. The spread is the widening distance between economic winners and losers: between skilled and unskilled labour, between capital and labour, between superstars and everyone else in the same occupation. Technology drives both. The bounty is not compensation for the spread, and the spread is not evidence that the bounty is illusory. The book's diagnosis is that societies are getting the bounty and failing to manage the spread, and its prescriptions follow: not to slow the machines, but to reorganise education, taxation, immigration, infrastructure and research so that more people share in what the machines produce. Two features of the book explain why it has proved durable and also why it needs a companion. The first is its use of vivid technological demonstrations — a self-driving car on a public highway, a question-answering computer beating human champions on a quiz show, a robot that could be taught by being shown — to make an economic argument. Those demonstrations are memorable, but they are illustrations of a mechanism, not the mechanism itself, and students routinely mistake the one for the other. The second is that the book is a forecast. It was written in 2013 and published in 2014, before the deep learning boom, and eight years before large language models became a consumer product. A companion written now can do something the original could not: check the forecast against the record. That check is the controlling idea of this guide. Brynjolfsson and McAfee's framework — three forces producing two divergent consequences, requiring a policy response aimed at the distribution rather than the direction of technical change — has held up far better as an analytical apparatus than as a set of specific predictions. Measured productivity did not take off in the way the book's optimism implied; the great decoupling of productivity from median income, which the book treated as a durable structural fact, looks less clean in the data now available; and the automation of driving arrived a decade later and far more narrowly than the Google car demonstration suggested. Meanwhile the mechanism the book identified — that machines take over tasks, not occupations, and that the value of the tasks left to humans determines what happens to wages — has become the standard language of the entire field, including among the authors' sharpest critics. Understanding The Second Machine Age well means understanding both halves of that verdict. The guide is organised to build the argument the way the book builds it, then to test it. It begins with the historical claim about steam and the shape of human progress, and with the technological demonstrations the authors use to argue that the second machine age has arrived. It then takes each of the three forces in turn, giving each a full treatment, because students who can define exponential improvement, non-rivalry and recombinant innovation precisely can reconstruct almost everything else in the book from first principles. From there it follows the two consequences: the bounty, together with the deep measurement problem that makes the bounty hard to see in official statistics, and the spread, together with the task-based model of technical change that explains why the middle of the labour market hollowed out. A separate treatment of superstar economics explains why digital markets concentrate rewards so severely. The guide then sets out the book's recommendations — for individuals, for near-term policy, and for the long term, including the argument over a negative income tax versus a universal basic income — and the objections raised by Robert Gordon, Daron Acemoglu and others. A clearly marked final chapter brings the argument up to 2026: what generative artificial intelligence has done to the labour market and to measured output since 2022, what the best empirical studies show, and where the framework has earned its keep. Throughout, the distinction between what Brynjolfsson and McAfee argue and what this guide adds is kept explicit. Their claims are attributed to them. Evidence published after 2014, critiques of their position, and assessments of how the forecast has aged belong to the guide and are marked as such. The aim is not to summarise the book but to make you able to use it: to deploy its terminology accurately, to see where its mechanisms apply, and to know where an examiner or a reader will expect you to push back. Chapter 1: The First Machine Age and Why the Second One Matters Brynjolfsson and McAfee open with a question that sounds like a parlour game and is not: what, in the whole record of human existence, actually bent the curve? Their answer, borrowed from the historian Ian Morris, is that if you plot a composite index of human social development — population, energy capture, urbanisation, information technology, war-making capacity — across thousands of years, the line is essentially flat until the late eighteenth century, and then it turns almost vertically upward. Agriculture did not do that. Nor did writing, the wheel, bronze, the Roman road network, gunpowder, the printing press, the discovery of the Americas, or the founding of the great religions. All of these matter enormously to human affairs. None of them shows up as a break in the trend. What shows up is the steam engine, and specifically James Watt's separate-condenser improvement of the 1760s and 1770s, which turned Newcomen's inefficient pumping device into a general-purpose source of mechanical power. That is the first machine age: the moment when human and animal muscle stopped being the binding constraint on what could be produced, moved and built. The authors are careful about the causal claim. They do not say the steam engine alone caused modernity; they say it is the technology that coincides with the inflection and that is the plausible candidate, because power is upstream of nearly everything else. Factories, railways, steamships, mechanised agriculture and eventually electrification are all consequences of having abundant, portable, controllable power. The historical framing does a specific job in the argument, and it is worth being precise about what that job is. It sets a bar. If you want to claim that some new technology constitutes a genuine discontinuity rather than an ordinary improvement, you have to claim it does something to a fundamental constraint — not that it is impressive, or fast-growing, or widely adopted. The first machine age lifted the constraint on physical power. The second machine age, Brynjolfsson and McAfee argue, lifts the constraint on cognitive power: on our ability to process information, recognise patterns, generate and test hypotheses, communicate and coordinate. That is the whole basis for the analogy, and every subsequent chapter of the book depends on it holding. What the analogy commits the authors to Taking the steam analogy seriously has consequences the authors accept openly, and students should be able to name them. The first is that the effects will be broad rather than sectoral. A general-purpose technology, in the terminology economists use for this category, is one that spreads across almost all industries, improves continuously over decades, and spawns complementary innovations. Steam, electricity and the internal combustion engine qualify. If digital computing is a general-purpose technology of that class — and Brynjolfsson and McAfee are emphatic that it is — then its economic footprint should eventually show up everywhere, not merely in the technology sector. The second is that the effects will be slow to appear in the aggregate statistics, and then large. Electrification is the canonical case, studied in detail by the economic historian Paul David: electric motors were available in factories from the 1880s, but the productivity payoff did not arrive until the 1920s, because capturing it required rebuilding factories around the new technology rather than bolting it onto the old layout. The old design put a single massive steam engine in the basement driving overhead shafts, which forced every machine to be positioned by its access to the drive shaft. Small electric motors allowed a machine to be powered wherever it made sense to put it, which meant single-storey plants organised around workflow — but only for firms willing to scrap the buildings they had. That lag is not a failure of the technology. It is a feature of general-purpose technologies, and it becomes central to the book's treatment of productivity measurement. The third consequence is the uncomfortable one. The first machine age was not a smooth or painless transition for the people living through it. British real wages stagnated for decades during early industrialisation even as output per worker climbed — the "Engels pause" that economic historians still argue about. Handloom weavers were destroyed as a trade. The Luddites, whom the authors treat with more respect than the popular caricature allows, were not confused about technology; they were skilled workers correctly perceiving that a particular machine would eliminate the specific competence that gave them bargaining power. If the second machine age is really analogous to the first, then the analogy carries the disruption along with the abundance. Brynjolfsson and McAfee do not dodge this. Their entire treatment of the spread follows from taking it seriously. The claim that the discontinuity has already begun The book's more contentious move is not the historical analogy but the timing. The authors argue that we are not approaching the second machine age but are already some way into it, and that the mid-2000s to early 2010s mark the point where digital technologies crossed from disappointing to startling. Their evidence for this is deliberately anecdotal in form and systematic in intent. In 2004, the economists Frank Levy and Richard Murnane published The New Division of Labor, a careful and influential analysis of which human tasks computers could and could not perform. They used driving in traffic as their example of a task that resisted codification: executing a left turn against oncoming traffic requires integrating an enormous volume of visual information and applying judgement that the driver cannot articulate as rules. Their conclusion was reasonable, well argued, and — Brynjolfsson and McAfee point out — overtaken within a few years. The first Defense Advanced Research Projects Agency Grand Challenge, held in the Mojave Desert in March 2004, was a humiliation: no vehicle completed the course, and the best performer covered roughly 7.3 miles of a route of about 150 miles before failing. In October 2005, five vehicles finished a 132-mile desert course, with Stanford's entry, Stanley, winning in just under seven hours. By 2010 Google had autonomous vehicles logging public road miles in California. The authors' point is not that the Levy and Murnane analysis was sloppy. It is that a task can look permanently beyond machines right up until it is not, and that the interval between "clearly impossible" and "commercially demonstrated" had compressed to a handful of years. If that compression is general, then judgements about which human capabilities are safe from automation have a much shorter shelf life than anyone's career. The book assembles several such demonstrations, and the next chapter examines each in detail. What matters at this stage is the structure of the inference. Brynjolfsson and McAfee are not predicting from a trend line in the labour data. They are arguing from technological capability: here is a set of things machines could not do, here is evidence they now can, and the economic consequences follow from the capability. That is a legitimate method, and it is also the place where the book is most vulnerable, because a demonstration is not a deployment and a deployment is not an economy-wide effect. Google's self-driving car was operating on public roads in 2010; paid driverless taxi services reached meaningful scale only in the 2020s, and even in 2026 they operate in a limited set of cities. The capability was real. The economic transmission was slower than the demonstration implied. Where the argument came from The book did not appear from nowhere, and knowing its immediate prehistory helps explain both its urgency and its blind spots. In 2011 Brynjolfsson and McAfee published Race Against the Machine, a short self-issued volume written in the aftermath of the financial crisis. The economic puzzle of that moment was the jobless recovery: American output regained its pre-crisis level well before American employment did, and the employment-to-population ratio, which had already been sliding since 2000, fell sharply and then refused to climb back. The standard explanations were cyclical — weak demand, damaged balance sheets, a housing market that had trapped workers in place. A second camp argued the problem was structural in the sense of mismatch: workers with the wrong skills in the wrong places. Brynjolfsson and McAfee proposed a third reading. Firms had used the downturn to substitute capital, specifically information technology, for labour on a large scale, and much of that substitution would not be reversed when demand returned. The jobs were not waiting to come back because the tasks that constituted them had been reassigned to software. That argument was contested at the time and remains contested. What matters for reading The Second Machine Age is that the later book extends it in two directions. It pushes backwards to establish the historical scale of the claim, and forwards to separate the distributional problem from the growth problem. Race Against the Machine is mostly a book about employment. The Second Machine Age is a book about welfare, in which employment is one channel among several. The second piece of background is a chart the authors popularised, which they called the great decoupling. Plot United States labour productivity and real median household income on the same axes from 1947. Through the post-war decades the two lines rise together, which is what the textbook predicts: workers get paid roughly what they produce, so when output per hour rises, the typical household's income rises with it. From some point in the 1970s or 1980s — the precise date depends on which income series and which price deflator you use, and this matters more than it sounds — the lines separate. Productivity keeps climbing. Median income flattens. Whatever the extra output is buying, it is not going to the household in the middle of the distribution. That chart is the empirical heart of the book's case that something has changed, and it deserves a careful health warning that the book itself does not fully supply. The gap between the two lines is partly a measurement artefact. Productivity is computed from output deflated by a producer price index; household income is deflated by a consumer price index; the two indices diverge, and a share of the apparent decoupling is that divergence rather than any economic event. Productivity is measured per hour while income is measured per household, and household size has fallen. Compensation includes employer-paid health insurance, which grew rapidly, while money income does not. Careful reconstructions that correct for these factors shrink the gap without eliminating it. And the subsequent data complicates the story further: real median household income in the United States reached a record of $87,460 in 2025, 2.6 percent above the previous year, on the Census Bureau's measure — the strongest reading in the series since it began in 1967. A guide written in 2014 would have described a decade and a half of stagnation. A guide written now has to acknowledge that median households have made real gains since, which does not refute the decoupling thesis but does qualify it. The third strand of background is intellectual rather than empirical. The authors are writing within a tradition of economics that takes technology seriously as an endogenous force rather than treating it as an unexplained residual. Robert Solow's growth accounting in the 1950s established that most of the growth in output per worker could not be explained by adding capital or labour, and the unexplained portion — total factor productivity — was attributed to technical change. Paul Romer's work in the late 1980s and 1990s made the production of ideas itself the object of analysis, emphasising that ideas are non-rival and that this non-rivalry is what permits sustained growth rather than diminishing returns. Brynjolfsson and McAfee's second and third forces are applied Romer: non-rivalry as a property of digital goods, and the combinatorial production of new ideas. Students who can place the book in this lineage write far better about it than those who treat it as journalism about robots. Three forces, two consequences, one policy problem The architecture of the book is simple enough to hold in your head, and holding it there is most of what a student needs. Three forces produce the second machine age. Two consequences follow. One policy problem results from the gap between them. The three forces are exponential improvement in digital hardware, the peculiar economics of digital goods, and the recombinant character of innovation. They are analytically distinct — one is about hardware trends, one about the properties of information as a commodity, one about how ideas are produced — but they reinforce one another. Cheap computation makes it feasible to digitize more; digitized things become raw material for recombination; recombination produces new applications that generate demand for more computation. Table 1 sets out the three forces, the mechanism behind each, and the economic property each one generates, and it is worth returning to as the argument develops. Table 1. The three forces of the second machine age. Force Core mechanism Book's leading example Economic property generated Exponential Sustained doubling of computing capacity per unit cost Moore's Law; the chessboard parable Constraints fall away abruptly, late in the sequence Digital Information goods are non-rival and near-costless to copy Digitized maps, text, music, sensor data Near-zero marginal cost; vast consumer surplus Recombinant New ideas are combinations of existing ideas The self-driving car as assembled components Innovation prospects grow rather than exhaust The two consequences are the bounty and the spread. The bounty is the increase in the quantity, quality and variety of goods and services available, much of it delivered at zero price and therefore substantially invisible to gross domestic product. The spread is the increase in dispersion: between high-skill and low-skill workers, between owners of capital and suppliers of labour, and between the top handful of performers in a field and everyone else in it. Brynjolfsson and McAfee insist that both are consequences of the same technological forces, and that treating them as a trade-off — accepting more inequality as the price of more growth — misunderstands the mechanism. The spread is not a payment for the bounty. It is a separate output of the same process, and it is the one that responds to policy. The policy problem falls out of this. If technology is generating enormous value and distributing it narrowly, the sensible response is neither to obstruct the technology nor to wait for the market to redistribute on its own. It is to change the institutions that determine who captures the value: education systems that determine who can work alongside machines, tax systems that determine how returns to capital and labour are treated, immigration rules that determine where innovators locate, public investment in the research and infrastructure that private firms underinvest in. The book's recommendations are unusually concrete for a work of this kind, and they are examined in full later in this guide. One clarification is worth making now, because it prevents a common misreading. The Second Machine Age is not a book about robots taking all the jobs. The authors explicitly reject the claim that technological unemployment is inevitable or imminent, and they spend considerable effort on why humans and machines are complements in a great many settings. Their worry is compositional and distributional: which tasks remain valuable, who can perform them, and what happens to the earnings of people whose particular bundle of skills has just been rendered cheap. That is a different and more defensible claim than the one the book is often credited with, and getting it right is the first test of whether you have read it carefully. Key Takeaways · Brynjolfsson and McAfee argue that the Industrial Revolution is the only event in the historical record that visibly bends the curve of human development, and that the steam engine is the technology responsible. · The second machine age is defined by analogy: digital technology relieves the constraint on cognitive power as steam relieved the constraint on physical power. · Treating computing as a general-purpose technology commits the authors to broad effects, long lags before those effects appear in the statistics, and substantial disruption to existing workers. · The book argues the discontinuity has already begun, using technological demonstrations rather than labour-market trends as evidence — a method that is powerful but conflates capability with economic transmission. · The architecture is three forces (exponential, digital, recombinant) producing two consequences (bounty and spread) and one policy problem (who captures the value). Review Questions 1. What specific claim does the historical comparison with the steam engine commit Brynjolfsson and McAfee to, and what would count as evidence against it? 1. Why does the electrification of factories serve as the authors' template for how a general-purpose technology affects measured productivity? 2. Explain why the authors treat Levy and Murnane's 2004 analysis of driving as instructive rather than as an error. 3. Distinguish between a technological demonstration, a commercial deployment, and an economy-wide effect. Where does the book's argument rely on eliding these? 4. Why do the authors insist that the bounty and the spread are separate consequences rather than a trade-off? Chapter 2: The Demonstrations — What the Machines Could Suddenly Do Brynjolfsson and McAfee build their case for the second machine age out of a small number of technological events, each chosen because it broke a barrier that competent observers had recently judged durable. The demonstrations are the most quoted part of the book and the most frequently misremembered. Students who get them slightly wrong — who say Watson understood natural language, or that Baxter was a commercial success — undermine everything they write afterwards. This chapter states what actually happened in each case, and then examines what the examples are and are not evidence for. Driving, and the collapse of a reasonable prediction The self-driving car is the book's anchor example because its timeline is so compressed. Frank Levy and Richard Murnane's The New Division of Labor, published in 2004, is a serious piece of labour economics, not a casual prediction. Their framework distinguished tasks that can be expressed as explicit rules from tasks that rely on tacit knowledge — pattern recognition and judgement that the practitioner cannot fully articulate. Computers, they argued, are excellent at the former and hopeless at the latter, and driving in traffic sat firmly in the latter category. Executing an unprotected left turn requires reading the speed and intention of oncoming vehicles, anticipating pedestrians, and making a continuous risk judgement, none of which the driver could write down as a procedure. In March 2004, the Defense Advanced Research Projects Agency ran the first Grand Challenge, offering a prize for an autonomous vehicle that could complete a roughly 150-mile desert route. Every entrant failed; the furthest travelled about 7.3 miles. This looked like confirmation. Nineteen months later, in October 2005, five vehicles finished a 132-mile course. Stanford's Stanley won in six hours and fifty-four minutes at an average of about 19 miles per hour, ahead of two Carnegie Mellon entries. By 2010 Google had a programme running autonomous vehicles on Californian public roads with safety drivers, and the authors report riding in one. The lesson Brynjolfsson and McAfee draw is about the shape of technological progress rather than about cars. Capabilities that improve exponentially do not approach a threshold gradually; they are far from it and then past it, and the interval in which they are visibly "almost there" is short. Any judgement of the form "machines will never be able to do X" is therefore a judgement about the present state of the art dressed up as a claim about the nature of X. This guide would add a qualification the book could not have made. The gap between demonstration and economic effect turned out to be long. Public-road autonomy was demonstrated in 2010; driverless commercial services began carrying paying passengers without a safety driver only in the late 2010s, and scale arrived in the 2020s. By March 2026 Waymo was providing roughly 500,000 paid rides a week across about ten United States cities, having grown tenfold from around 50,000 weekly rides in May 2024. That is a genuine industry and a genuine displacement risk for drivers in those cities, and it is also, after sixteen years, a small fraction of American road transport. The capability claim was vindicated. The implied speed of economic transmission was not. Watson, and the difference between answering and understanding In February 2011, IBM's Watson system played three televised games of Jeopardy! against Ken Jennings and Brad Rutter, the show's two most successful human contestants, and won a first prize of one million dollars. The result is impressive for reasons that are easy to state imprecisely, so it is worth being careful. Jeopardy! is hard for a machine for three separate reasons. The clues are in natural language and are frequently oblique, relying on puns, cultural allusion and misdirection. The domain is unbounded: a clue can concern anything. And the game is adversarial in real time, requiring the system to estimate its own confidence and decide whether to buzz in, since a wrong answer costs money. Watson handled all three by running many candidate-answer generation strategies in parallel across a large corpus of unstructured text, scoring the candidates by multiple independent pieces of evidence, and combining those scores into a confidence estimate learned from past games. It was not connected to the internet during the match; it had already ingested its sources. What Watson did not do is understand anything, in the sense a person means by that word. It was a statistical evidence-aggregation machine of considerable sophistication. Brynjolfsson and McAfee are reasonably careful about this, and their point is economic rather than philosophical: for the purposes of doing valuable work, it may not matter whether the machine understands. If a system can produce reliably correct answers to open-ended natural-language questions, the tasks that consist of producing such answers become automatable regardless of what is happening inside. The subsequent history is instructive and should temper any triumphalism. IBM spent the following decade attempting to commercialise Watson, most prominently in oncology decision support, and the results fell well short of the promises. Divisions of the Watson Health business were sold off in 2022. Winning a quiz show is not the same problem as advising on cancer treatment, where the cost of error is asymmetric, the evidence base is contested, and the data is fragmented across incompatible institutional systems. This is a useful corrective to the book's method: a demonstration proves that a capability exists under demonstration conditions. Machine translation and the arrival of statistical methods The book's third capability example is language translation, and it is the one that has aged best. For four decades machine translation was attempted primarily by encoding grammatical rules and bilingual dictionaries, and it produced output that was usable only as a rough gist. The shift that changed the field was statistical: rather than teaching the machine grammar, feed it enormous quantities of text that has already been translated by humans — parliamentary proceedings, United Nations documents, multilingual corporate websites — and let it learn which sequences of words in one language correspond to which in another. The approach required no linguistic theory and improved automatically as more data became available. Brynjolfsson and McAfee use the business application of this to make their point, citing services that translate customer-support conversations in real time so that agents and customers who share no language can transact. The relevant economic property is that the quality improves as a function of data volume and computing power, both of which were growing exponentially, rather than as a function of linguists' effort, which was not. Since 2014 the field has changed again, twice: first to neural machine translation around 2016, which improved fluency dramatically, and then to large language models, which handle translation as one capability among many. The direction the authors identified was right, and the improvement has been larger than they suggested. Translation is also a cautionary case for the labour argument, because the profession did not disappear. Demand for translated content expanded, and the human role shifted towards post-editing, certification, and the categories where error is expensive — legal, medical, literary. This is exactly the pattern of task reallocation the book's own framework predicts, and it is the pattern its rhetoric sometimes obscures. Baxter and the physical world Robotics supplies the book's counterpoint to its software examples, because robots must contend with physical reality, which is unforgiving. Rethink Robotics, founded by the roboticist Rodney Brooks, introduced Baxter in 2012: a two-armed industrial robot with a screen displaying a cartoon face, priced at around twenty-two thousand dollars — roughly an order of magnitude below the installed cost of a conventional industrial robot arm at the time. Baxter's design was significant on three counts. It was safe to work next to, using compliant joints and force sensing rather than being caged, which meant it could be placed on a production line among people. It was programmed by demonstration: an operator physically moved its arms through a task and Baxter learned the motion, so deploying it did not require a robotics integrator. And it was cheap enough that a small manufacturer could plausibly buy one. Together these attributes pointed at a category that barely existed — the collaborative robot, or cobot — aimed at small-batch, high-variety work that had never justified conventional automation. The honest postscript is that Rethink Robotics failed. The company shut down in October 2018, having raised around $150 million, after sales fell far short of expectations; its intellectual property was acquired by a German firm. The collaborative robot category it helped create did succeed, and is dominated commercially by other manufacturers. Baxter itself was not accurate or fast enough for most production work, and the tasks it was ideally suited to were often tasks where a human was simply cheaper. The book is not wrong that Baxter mattered — it demonstrated a design philosophy that became standard — but a student citing it as evidence of automation's advance should know that this particular machine did not survive. Kiva, Amazon, and what large-scale deployment actually looks like The most economically consequential example in the book is also the least glamorous. Kiva Systems built warehouse robots that invert the logic of order fulfilment: instead of sending a human to walk to the shelf, the robot drives the entire mobile shelving unit to a stationary human picker. The saving is in walking time, which in a conventional fulfilment centre can consume the majority of a picker's shift. Amazon acquired Kiva in March 2012 for $775 million in cash and stopped selling the systems to anyone else, which tells you how valuable it considered the technology. Amazon's subsequent trajectory is the best available natural experiment on the book's central question, and it is worth following past the book's publication. By July 2025 Amazon reported having deployed more than one million robots across its operations, beginning from a single robot type in 2012. Over the same period the company's global workforce grew enormously; it employs well over a million people. Amazon states that its next-generation robotic fulfilment centre in Shreveport, Louisiana requires about 30 percent more employees in reliability, maintenance and engineering roles than a conventional site, and that it has put more than 700,000 employees through upskilling programmes since 2019. Three readings of this are possible, and a good answer holds all three. The optimistic reading is that automation and employment grew together, exactly as the complementarity argument predicts. The sceptical reading is that Amazon's employment grew because e-commerce volume exploded, and the relevant counterfactual is how many people the same volume would have required without robots — a number nobody knows, but certainly larger. The compositional reading, which is the one the book's own framework recommends, is that the content of the work changed: fewer hours walking, more hours at fixed stations working at machine-set pace, plus a smaller number of much better-paid technical roles maintaining the fleet. The aggregate headcount is the least interesting statistic in the story. Instagram, Kodak, and the disappearing payroll One further comparison appears in the book and belongs with the demonstrations, because it makes a point about capital and labour that none of the others do. In April 2012 Facebook agreed to acquire Instagram for about a billion dollars. Instagram was a photo-sharing service roughly eighteen months old with a staff of around a dozen people. Three months earlier, on 19 January 2012, Eastman Kodak had filed for Chapter 11 bankruptcy protection. Kodak had defined the consumer photography industry for a century, had employed well over a hundred thousand people at its height, and had — this is the part that makes the story sting — invented the digital camera internally in the 1970s and declined to let it cannibalise the film business. The authors use the juxtaposition to illustrate two things at once. The first is the digitization force: photography went from a physical good with per-unit costs to an information good with essentially none, and the value migrated with it. The second is distributional. A firm can now capture an enormous market with almost no payroll, which means the returns flow overwhelmingly to a small number of founders, engineers and equity holders rather than to a large workforce. The industry did not shrink — far more photographs are taken now than ever were on film — but the labour required to serve it collapsed. The comparison is rhetorically effective and somewhat unfair as accounting, and a careful student should say so. Instagram ran on infrastructure it did not own and did not have to build: smartphone cameras manufactured by hundreds of thousands of workers, cellular networks, data centres, payment systems. Its thirteen employees sat on top of a very large amount of embodied labour. What is true, and is the point worth keeping, is that the marginal labour required to add another user to a digital service is close to zero, and that this property determines who gets paid. Why expert predictions about machine capability fail Running underneath all these examples is a claim about forecasting that deserves separate statement, because it is the part of the chapter most useful outside the book. Predictions about what machines cannot do are usually generated by introspection. The forecaster imagines performing the task, notices that their own performance depends on something they cannot articulate, and concludes that the task requires a faculty machines lack. This is the structure of Levy and Murnane's argument about driving, and it is also the structure of most confident claims about the limits of artificial intelligence made since. The trouble is that the inference is invalid. That a human accomplishes a task by means of inarticulable judgement does not imply that the task can only be accomplished that way. Machines frequently solve problems by a completely different route — brute-force search, statistical pattern-matching over volumes of data no person could read — and arrive at comparable output. This is sometimes called Moravec's paradox, after the roboticist Hans Moravec, who observed in the 1980s that the tasks humans find hardest, such as playing chess at a high level or performing complex arithmetic, are comparatively easy to mechanise, while the tasks a small child performs effortlessly — recognising a face, walking over uneven ground, picking up an unfamiliar object — are extraordinarily hard. The explanation is evolutionary: perception and locomotion have been under selection for hundreds of millions of years and are implemented in enormously refined but entirely unconscious machinery, whereas abstract reasoning is recent, thin, and deliberate. Our intuitions about difficulty track how much conscious effort a task costs us, which is almost exactly the wrong signal. Brynjolfsson and McAfee's demonstrations are all cases where the paradox bit: driving and object manipulation are Moravec-hard, and progress on them was slow and then sudden because it depended on data and compute rather than on insight. The practical implication for anyone reasoning about automation is to stop asking whether a job seems to require human judgement and start asking two narrower questions. Can the task be decomposed into steps with observable inputs and outputs? Does a large volume of examples of the task being performed correctly already exist, or could it be generated? Where the answer to both is yes, the task is a candidate regardless of how sophisticated it feels to perform. What the demonstrations do and do not prove The examples establish, collectively and persuasively, that the boundary of machine capability moved substantially and unexpectedly in the decade before the book was written, and moved in directions that expert judgement had recently ruled out. That is a real finding and the book is right to insist on it. They do not establish three things that readers frequently take from them. They do not establish a rate of diffusion; every one of these capabilities took far longer to reach economic scale than the demonstration suggested, and two of the five specific commercial vehicles — Watson Health and Rethink Robotics — failed outright. They do not establish aggregate effects, because a capability that is deployed in a few thousand firms does not move national statistics. And they do not, by themselves, establish anything about employment, since whether a capability displaces or complements workers depends on the elasticity of demand for the output, on what the freed-up labour does next, and on the cost of the machine relative to the wage. The book supplies those arguments separately and properly. The demonstrations are the premise, not the conclusion. Key Takeaways · The Grand Challenge sequence — total failure in 2004, five finishers in 2005 — is the book's evidence that machine capability crosses thresholds abruptly rather than gradually. · Watson's 2011 Jeopardy! victory demonstrated open-domain question answering by evidence aggregation and confidence estimation, not understanding; IBM's later failure to commercialise it in medicine shows the gap between demonstration and application. · Machine translation improved through statistical rather than rule-based methods, making quality a function of data and computation; the profession reallocated towards post-editing rather than disappearing. · Baxter demonstrated safe, cheap, demonstration-programmed robotics and defined the collaborative robot category, but Rethink Robotics shut down in 2018. · Amazon's acquisition of Kiva in 2012 and its subsequent growth to over a million robots alongside a growing workforce is the book's most consequential case, and supports a compositional rather than a headcount reading. Review Questions 1. Reconstruct Levy and Murnane's reasoning about driving. Which part of their framework was wrong, and which part survives? 1. What three distinct technical problems did Watson have to solve, and why does the distinction between answering and understanding matter economically rather than philosophically? 2. Why did the shift to statistical methods change the growth path of machine translation quality? 3. Rethink Robotics failed commercially. Does that weaken the book's use of Baxter as evidence? Give the strongest argument on each side. 4. Amazon's robot fleet and its workforce both grew. Explain why headcount is a poor test of the automation hypothesis, and state what a better test would measure. Chapter 3: The Exponential Force — Moore's Law and the Second Half of the Chessboard The first of the three forces is the simplest to state and the hardest to feel. Digital technology improves at a rate that is roughly constant in percentage terms, which means the absolute increments grow without limit. Brynjolfsson and McAfee argue that most of the confusion about the second machine age — including the confusion of very intelligent people — comes from minds calibrated to linear change trying to reason about a process that is not linear. What Moore's Law actually says In April 1965, Gordon Moore, then director of research at Fairchild Semiconductor and later a co-founder of Intel, published a short article in Electronics magazine under the title "Cramming more components onto integrated circuits." He had four data points. He observed that the number of components that could be placed on an integrated circuit at minimum cost per component had been roughly doubling each year, and he projected the trend forward for a decade. In 1975 he revised the doubling interval to about two years. The regularity has been called Moore's Law ever since. Several clarifications are essential, because the term is used loosely and the looseness causes real errors in student writing. Moore's Law is not a law of physics. It is an observed empirical regularity that became, in the semiconductor industry, a self-fulfilling coordination device: firms, equipment suppliers and materials producers planned their capital investment against a published roadmap of expected improvement, and the roadmap made the improvement happen. When the industry's collective expectation shifts, so does the trend. Moore's original statement is about transistor density and cost, not about speed, and certainly not about the vaguely defined "power" of computers. Related but distinct regularities cover clock speed, cost per operation, storage density, memory bandwidth and network capacity, and they have different rates and different histories. Clock-speed scaling essentially stopped around 2004 when power dissipation made further frequency increases impractical; performance gains after that came from putting multiple cores on a chip, from specialised architectures, and from software. Conflating these is the commonest error in discussing the topic. Brynjolfsson and McAfee's actual claim is broader and more defensible than the narrow semiconductor one. They argue that sustained exponential improvement has been observed across a family of digital technologies over decades, that it has not stopped, and that its economic consequences follow from the compounding rather than from any particular mechanism. Whether the next factor of two comes from smaller transistors, from three-dimensional stacking, from purpose-built accelerators or from better algorithms is, for their purposes, immaterial. The parable of the chessboard The device the book uses to convey compounding is an old story, retold by the inventor Ray Kurzweil, about the invention of chess. The inventor presents the game to his emperor, who offers a reward. The inventor asks for rice: one grain on the first square of the board, two on the second, four on the third, doubling on each of the sixty-four squares. The emperor, thinking this modest, agrees. The arithmetic is where the lesson lives. Doubling through the first half of the board — thirty-two squares — yields a total of about four billion grains, which is a large heap of rice, roughly a field's worth. It is a quantity an emperor could deliver. But the total on the sixty-fourth square alone is two to the sixty-third power, which is more than nine quintillion grains: a pile that would dwarf a mountain. Crossing from the first half to the second half of the board is the moment at which the quantities stop being comprehensible in ordinary terms. The point is not that the numbers get big. It is that within an exponential process, every doubling adds more than the entire history that preceded it, and therefore the process looks manageable and then abruptly does not. Brynjolfsson and McAfee argue that digital technology entered the second half of its chessboard sometime in the 2000s. Their arithmetic for this is a rough count: if you date the start of sustained doubling to the era of commercial business computing around 1958 and assume doublings roughly every eighteen months, thirty-two doublings brings you to the mid-2000s. The dating is a heuristic rather than a measurement, and the authors present it as such, but the underlying claim is serious: the absolute increments in computing capacity delivered each year now dwarf anything delivered in earlier decades, even though the percentage rate is unchanged. The practical consequence is a rule for interpreting stalled technologies. A capability that has been "ten years away" for thirty years may be genuinely impossible, or it may be on an exponential curve that has not yet crossed the threshold. From inside the curve, the two look identical until suddenly they do not. Speech recognition spent decades as a perpetual near-miss and then, within a few years in the 2010s, became good enough to be embedded in every phone. The same profile applies to image recognition, to machine translation, and later to text generation. Why linear intuition fails The book spends real effort on the psychology of the thing, and the effort is justified, because the analytical error it describes is committed constantly by people who could do the arithmetic if asked. Human beings reason well about processes that add a constant amount per period and badly about processes that multiply by a constant factor. The reason is not innumeracy. It is that almost every quantity we have evolved to track — distance walked, food gathered, a child's height — behaves additively over the horizons that matter. Exponential processes in nature are typically short-lived, because they run into a resource ceiling and become logistic curves. Sustained exponentiality over decades is a peculiarity of certain human-engineered systems, and we have no trained intuition for it. The practical symptom is systematic mis-forecasting in two opposite directions at once. Over short horizons, people overestimate what an exponential process will deliver, because they extrapolate from the excitement of the moment rather than from the base. Over long horizons, they wildly underestimate, because they cannot hold in mind what thirty more doublings does. Both errors are visible in the discourse around every digital technology, often in the same commentator within a single paragraph. A useful discipline for writing about this is to work in doubling times rather than in percentages. A quantity growing at 7 percent a year doubles in roughly ten years; at 14 percent, in five; at 40 percent, in about two. The rule of thumb — divide seventy by the annual percentage growth rate to get the doubling time in years — converts a rate into an interval the mind can handle. If a capability is currently at one percent of what would be required to displace some category of work, and it is doubling every two years, the honest statement is not that it is far away. It is that it is about fourteen years away, and that it will look far away for twelve of them. The corollary the authors emphasise is about the timing of surprise. In an exponential process the last doubling before a threshold takes the system from half of what is needed to all of it. The period during which the technology is visibly nearly adequate is therefore always brief, regardless of how long the process has been running. This is why capabilities appear to arrive suddenly even when the underlying trend has been public and stable for decades, and why "nobody saw this coming" is so often false as a matter of record and true as a matter of experience. Cost, not just capability The economically important form of the exponential claim concerns price rather than performance, and the book makes this vivid with a comparison that repays careful handling. ASCI Red, built for the United States Department of Energy at Sandia National Laboratories, was the first computer to sustain a trillion floating-point operations per second, crossing the teraflop threshold in December 1996. It occupied roughly the floor area of a tennis court, consumed power on the scale of a large building, and cost tens of millions of dollars. It was decommissioned in June 2006, having held the record as the world's fastest computer for four consecutive years, longer than any other machine. In the year ASCI Red was retired, Sony released the PlayStation 3, a games console priced in the hundreds of dollars whose graphics and processing hardware delivered performance in the same teraflop territory on the specific kinds of parallel arithmetic it was built for. The comparison is not exactly like for like — a supercomputer's sustained general-purpose throughput on scientific workloads is a different quantity from a console's peak parallel arithmetic, and any serious writer should flag that — but the order-of-magnitude point survives the caveat. A capability that cost tens of millions of dollars and a dedicated building in 1996 cost a few hundred dollars and sat under a television a decade later. That collapse in price is what makes exponential improvement economically transformative rather than merely technically interesting. When something gets cheap enough, it stops being a capital project to be justified and becomes an input to be used casually, and the applications multiply in ways nobody planned. Cheap computation is why every vehicle, appliance, machine tool and doorbell now contains a processor; why a start-up can rent supercomputer-class resources by the hour; and why training a large machine learning model, an activity that would once have required a national laboratory, is within reach of a few thousand organisations. Where exponential improvement does not reach A discipline the book applies less consistently than it should is asking which parts of the economy actually sit on an exponential curve. Getting this right is essential, because the answer determines how large the aggregate effect can be. Computation, storage, bandwidth and — through them — anything that can be reduced to information have improved exponentially for decades. Many other inputs have not. The cost of constructing a mile of urban rail, a hospital bed, a housing unit or a kilometre of transmission line has risen rather than fallen in real terms in most rich countries. The cost of a university education has risen. The cost of a medical procedure has risen. Energy storage and photovoltaic generation are genuine exceptions, having followed steep and sustained learning curves, and they are worth noting precisely because they are rare. The economist William Baumol's account of cost disease explains the pattern. In sectors where output per worker cannot easily be raised — a string quartet requires four players for the same duration it did in 1800 — wages nonetheless rise, because employers must compete for labour with the sectors where productivity is climbing. The relative price of the stagnant sectors therefore rises without end, and they absorb a growing share of spending and employment. Over time, an economy where one sector improves exponentially and the rest do not comes to be dominated, in expenditure terms, by the sectors that did not improve. The exponential sector shrinks as a share of the economy precisely because it is getting cheaper. This is not an objection to the book's technological claim; it is a constraint on how much that claim can move aggregate statistics, and it is the mechanism underlying much of the productivity puzzle examined later in this guide. If computation gets a thousand times cheaper and computation is three percent of costs, the effect on total costs is bounded at roughly three percent no matter how spectacular the improvement. Growth comes from the exponential technology making the non-exponential sectors more productive — from software changing how hospitals, construction firms and schools work — and that transmission depends on organisational change, regulation and complementary investment, none of which is exponential. Brynjolfsson and McAfee are aware of this and address it through their treatment of complementary innovation, but the emphasis of the writing points the other way, and students who quote only the chessboard passage end up asserting more than the book can support. The stronger formulation, and the one worth adopting, is this: exponential improvement in a general-purpose input creates the potential for broad transformation, and the realisation of that potential is governed by slow, non-exponential processes of adoption and reorganisation. What has happened since, and how to write about it honestly The Second Machine Age asserts that Moore's Law shows no sign of stopping. That assertion requires qualification a decade on, and handling the qualification well is a good test of whether a student understands the argument or is repeating it. Transistor density scaling has slowed and become far more expensive. The economic form of Moore's Law — cost per transistor falling with each node — weakened substantially in the 2010s, as the capital cost of leading-edge fabrication plants rose into the tens of billions of dollars and the number of firms capable of operating at the frontier fell to a handful. Dennard scaling, the related principle that shrinking transistors also reduced their power consumption proportionally, broke down around 2005, which is why clock speeds stopped rising. And yet the delivered capability relevant to the book's argument has continued to improve at an extraordinary rate, by routes Moore did not describe. Specialised processors designed for the matrix arithmetic that machine learning requires deliver performance per dollar far beyond what general-purpose chips achieve. Parallelism at data-centre scale substitutes coordination for single-chip speed. Algorithmic improvement has contributed at least as much as hardware in several important domains: the compute required to train a neural network to a given standard of image classification fell by orders of magnitude over the 2010s through better methods alone. And the total quantity of computation applied to frontier machine learning models grew at a rate far exceeding Moore's Law during the 2010s and 2020s, driven not by cheaper chips but by vastly larger capital expenditure on buying them. That last point matters for the book's logic and is worth stating plainly, because it is the kind of distinction that separates a strong answer from an adequate one. Brynjolfsson and McAfee's exponential force is an argument about falling cost: capability arrives because the price of computation collapses, so anyone can have it. A great deal of the recent frontier progress has instead come from rising expenditure: capability arrives because a small number of very large firms are spending unprecedented sums. Investment in data centres, computing hardware and networking reached roughly 0.8 percent of United States gross domestic product by the first quarter of 2026, with total computing infrastructure investment at around 1.5 percent of output against a 2015–2022 average near 0.7 percent, according to analysis by Epoch AI. Those are different mechanisms with different distributional implications. Cheap computation democratises; expensive computation concentrates. The second machine age the book describes is a story about the first. Much of what has happened since 2020 is a story about the second. Key Takeaways · Moore's 1965 article observed doubling of components per integrated circuit at minimum cost, revised in 1975 to roughly every two years; it is an empirical regularity and industry coordination device, not a physical law. · The claim concerns transistor density and cost, not clock speed; Dennard scaling and clock-speed growth ended around 2004–2005. · The chessboard parable illustrates that in an exponential process each doubling exceeds the entire accumulated total before it, so change appears manageable and then abruptly is not. · The economically decisive version of the exponential claim is about collapsing price: ASCI Red's teraflop capability of 1996 was matched a decade later by a consumer games console. · Density scaling has slowed and grown costly since 2014, but delivered capability has continued to rise through specialised hardware, parallelism, algorithmic improvement, and sharply increased capital expenditure — the last of which concentrates rather than democratises access. Review Questions 1. State Moore's original claim precisely. Name two distinct technological trends that are often mistakenly labelled Moore's Law. 1. Explain why the second half of the chessboard is a claim about the size of increments rather than about rates of growth. 2. Why do the authors treat falling cost, rather than rising performance, as the economically decisive form of exponential improvement? 3. The ASCI Red and PlayStation 3 comparison is not strictly like for like. State the objection and assess how much it damages the argument. Distinguish capability gains that come from cheaper computation from those that come from greater spending on computation. What different distributional consequences follow? Hashtags: #TheAutomationEconomy #TheSecondMachineAge #DigitalTransformation #Automation #TechnologicalChange #ExponentialTechnologies #MooresLaw #SecondHalfOfTheChessboard #DigitalGoods #NonRivalGoods #NearZeroMarginalCost #RecombinantInnovation #GeneralPurposeTechnology #BountyAndSpread #ConsumerSurplus #GreatDecoupling #TaskBasedAutomation #HumanMachineComplementarity #LabourMarketPolarization #SkillBiasedTechnologicalChange #SuperstarEconomics #ProductivityGrowth #DigitalCapital #EconomicInequality #FutureOfTheAutomationEconomy

Download the Book (PDF): Introduction Writing code that works is easy; writing code that survives is incredibly difficult. "Uncle Bob" Martin's classic text outlines the universal rules of software architecture. Yet the book's deep dives into object-oriented programming paradigms, dependency inversion and component cohesion can quickly become a confusing blur of engineering jargon for students studying the macro-level management of software teams. This guide structures the engineering logic into managerial theory. It is designed specifically to explain Clean Architecture by Robert C. Martin, separating the high-level architectural philosophy from the granular code examples. We meticulously break down the SOLID principles and the concept of separating business rules from delivery mechanisms. Equipped with simplified structural models and clear takeaways, this guide gives you a massive advantage in understanding how software structure dictates business flexibility. The Book and Its Author Robert C. Martin, known across the software industry as "Uncle Bob", has been programming professionally since the early 1970s. He was one of the seventeen signatories of the Manifesto for Agile Software Development in 2001, ran a long-standing consultancy and training firm, and edited the C++ Report in the 1990s. His reputation outside specialist circles rests mainly on three books. Clean Code (2008) is about the craft of writing readable functions and classes. The Clean Coder (2011) is about professional conduct: saying no, estimating honestly, and practising the discipline of testing. Clean Architecture: A Craftsman's Guide to Software Structure and Design, published by Prentice Hall in 2017, completes the arc by moving from the line of code and the individual programmer up to the shape of whole systems. The book is short by the standards of software engineering texts, and it is organised as a climb in scale. It opens with an argument about why structure matters at all, then works upward from programming paradigms, through principles for designing modules and classes, through principles for grouping those modules into deployable components, to architecture proper, and finally to the parts of a system Martin insists are not architecture at all: databases, the web and frameworks. Several chapters draw on long professional experience, and two contributors add material of their own. James Grenning writes on embedded systems, and Simon Brown supplies a closing chapter that questions how code should be packaged. Much of what the book says was not new in 2017. The SOLID principles had circulated since the early 2000s, and Martin's component metrics had appeared in his earlier writing on agile development. What the book did was assemble these ideas into a single argument and give that argument a memorable image: a set of concentric circles with business rules at the centre and every technical detail pushed to the outside. That image, and the rule that governs it, became one of the most widely cited reference models in modern software design. Teams building mobile apps, banking platforms and cloud services describe their systems in its vocabulary, whether or not they have read the book. Why a Manager Should Care A student of management might reasonably ask why any of this belongs outside the engineering department. The answer is that software structure is an economic variable. It determines how much a change costs, how long a new feature takes, how many people must coordinate to ship anything, and whether an organisation can switch suppliers, databases or delivery channels without rewriting its core systems. Martin states the goal of architecture in explicitly economic terms: the aim is to minimise the human effort required to build and maintain a system over its life. A well-structured system keeps the cost of each change roughly flat as the system grows. A badly structured one makes every change more expensive than the last, until the organisation is paying more and more engineers to deliver less and less. Those outcomes are rarely visible on a project plan. Structure is invisible to customers and usually to executives. Its effects arrive late and indirectly, as slipping schedules, rising headcount, and a creeping sense that the software has become a constraint on the business rather than an instrument of it. A manager who understands architecture can recognise those symptoms early, ask the right questions of technical staff, and make better decisions about when to invest in structure and when to accept a shortcut. The Controlling Idea The argument of this guide can be put in one sentence. Good architecture is the deliberate arrangement of dependencies so that the parts of a system most valuable to the business, its rules and policies, never depend on the parts most likely to change, its technologies and delivery mechanisms, and this arrangement is what keeps the cost of change low and the organisation's options open. Almost every principle in Martin's book is a restatement of that idea at a different scale. The Single Responsibility Principle separates code by the people who ask for changes. The Dependency Inversion Principle lets high-level policy stay ignorant of low-level detail. The component principles decide which way whole deployable units should depend on each other. The Dependency Rule of the Clean Architecture applies the same logic to the system as a whole. Once a reader sees the common thread, the jargon becomes much less intimidating: each term names a particular way of controlling who depends on whom. How This Guide Is Organised The guide follows the rising scale of Martin's book, with a closing turn toward evaluation. It begins with the economic case: the difference between design and architecture, the two kinds of value software provides, and why structure is always important but rarely urgent. It then treats the three programming paradigms as a history of disciplines, each of which removed a dangerous freedom from programmers and, in doing so, created the tools architecture relies on. The middle of the guide covers the principles. The five SOLID principles are explained without code, through small everyday illustrations that preserve the reasoning. The component principles follow in two stages, first the rules for deciding what belongs together, then the rules for how components should depend on one another, including the numerical measures of stability and abstractness that Martin proposes, worked through with a concrete example. The guide then turns to architecture itself: what it means to keep options open, how boundaries are drawn, why business rules deserve their own protected core, and how the Clean Architecture arranges a system in layers. A further stretch deals with the practical questions that arise when the model meets real systems, such as partial boundaries, the role of the startup code, the place of microservices, testing and embedded software. Martin's argument that databases, the web and frameworks are details gets its own treatment, because it is the claim most likely to surprise newcomers. Finally, the guide steps outside the book. It sets Martin's model beside close relatives, notably Alistair Cockburn's hexagonal architecture and Jeffrey Palermo's onion architecture, considers the most serious criticisms of the approach, and examines documented cases from Netflix, Shopify and Amazon Prime Video that show both the value of clean boundaries and the limits of fashionable alternatives. Throughout, the text distinguishes Martin's own arguments from the guide's commentary, so that readers can tell what the book says from what others have said about it. Each chapter closes with key takeaways and review questions. Several tables compress comparisons that would otherwise sprawl across pages, and a glossary at the end collects the terms that recur most often. None of this requires the ability to program. It requires only the willingness to follow an argument about how the shape of a thing determines what can be done with it, which is, in the end, a management question as much as an engineering one. Chapter 1: Why Structure Is a Business Problem Most people who commission software judge it by what it does. Does the checkout accept payment? Does the report show the right totals? Does the app load quickly on a phone? These are reasonable questions, and they are the ones customers ask. Martin's book begins by insisting on a second question that customers never ask and that managers often forget: how hard will it be to change this system tomorrow? His answer to why that question matters is not aesthetic. It is economic, and it frames everything that follows. Design and Architecture Are the Same Thing Software people often talk as if "architecture" were the grand, high-level shape of a system and "design" were the fine-grained arrangement of its parts. Architecture, on this view, is what senior people draw on whiteboards; design is what junior people do in the code. Martin rejects the distinction. He compares a software system to a house. The architect of a house produces drawings that show the overall shape, the elevations and the placement of rooms, but the same drawings also show where every power outlet, light switch and water pipe goes. The low-level details and the high-level structure are part of one continuous description. You cannot understand the house by looking at only one layer of it. The same holds for software. Decisions about how a single class is organised feed into decisions about how modules relate, which feed into decisions about how services and deployable units are arranged. There is no clean line where design ends and architecture begins. For a manager, the practical consequence is that architecture cannot be delegated to a single meeting at the start of a project and then forgotten. It is the cumulative result of thousands of small decisions made by everyone who writes code, and it keeps evolving for as long as the system lives. The Goal of Architecture If design and architecture are one thing, what is that thing for? Martin gives a definition that sounds modest but has sharp edges: the goal of software architecture is to minimise the human resources required to build and maintain the required system. Architecture is measured by effort. When the effort needed to meet each new requirement stays low across the life of the system, the design is good. When the effort grows with every release, the design is bad, regardless of how elegant it looked at the start. This definition turns an abstract engineering topic into something a finance department can recognise. Human effort is the dominant cost in software. Servers and licences matter, but salaries usually dwarf them. An architecture that doubles the effort needed per feature doubles the largest line in the budget. The signature of a mess Martin illustrates the point with data from a company he does not name. Over a series of releases, the company's engineering staff grew steadily, from a small team to a much larger one. Over the same releases, the number of lines of code the organisation produced barely moved, and the cost of each line rose steeply. Engineers were working as hard as ever; indeed, they were working harder. But more and more of their effort went into fighting the existing code rather than adding new capability. By the later releases, the company was paying many times as much for each increment of functionality as it had at the start. The shape of those numbers is what matters, not the precise figures. Headcount climbs. Output flattens. Cost per unit of output rises. Martin calls this the signature of a mess. Managers see it as a sequence of apparently unrelated problems: schedules slip, estimates grow, new hires take longer to become productive, and fixes in one place cause failures in another. Each symptom has a local explanation. The common cause is structural, and it is invisible unless someone is looking for it. The hare and the tortoise Why do capable teams produce messes? Martin's explanation is overconfidence. Developers under deadline pressure tell themselves that they will ship now and clean up later. Later never arrives, because the next deadline comes, and the mess from the last round makes the next round slower, which increases the pressure to cut corners again. The cycle is self-reinforcing. He retells Aesop's fable of the hare and the tortoise to make the point. The hare is so confident of its speed that it rests; the steady tortoise wins. In software, the team that sprints without structure feels fast in the first weeks and then slows dramatically. The team that keeps its code clean feels slower at first and wins over any horizon longer than a few months. Martin's slogan for this is that the only way to go fast is to go well. He supports it with a small informal experiment in which the same programming exercise was completed repeatedly with and without disciplined test-first practice. The disciplined runs finished sooner. The experiment is anecdotal rather than scientific, and the guide treats it as an illustration, not proof, but the direction it points in matches what many engineering leaders report from experience. The managerial lesson is that a mess is not a technical problem that engineers have inflicted on the business. It is a business decision, usually made implicitly, to borrow time from the future at a very high rate of interest. Technical debt and the cost of change The borrowing metaphor has a history that predates Martin's book. Ward Cunningham introduced the phrase "technical debt" in a 1992 conference report, describing how shipping code that reflects an incomplete understanding is like taking out a loan: useful in the short run, but interest accrues until the code is reworked. Martin does not rely heavily on the term, but his argument is the same. Structure is the principal; the extra effort each change requires is the interest. It helps to picture the relationship between time and the cost of change as a curve, described here in words. In the classic account associated with Barry Boehm's work on software economics in the early 1980s, the cost of fixing a problem rises sharply the later in a project it is discovered: a mistake caught during requirements is cheap, one caught in production is very expensive. The curve climbs steeply to the right. In the late 1990s Kent Beck, writing about Extreme Programming, argued that disciplined practices such as continuous testing, simple design and relentless refactoring could flatten that curve, so that a change late in a system's life costs roughly what it would have cost early on. Martin's architecture is, in effect, a programme for flattening the curve. A system with good boundaries keeps each change local. A system without them makes every change ripple outward, so the cost of change grows with the size of the codebase. Students should be able to describe the two shapes: a curve that rises ever more steeply, where each year's changes cost more than the last year's, and a curve that stays close to horizontal, where the tenth year's change costs about what the first year's did. The gap between the two lines, summed over the life of a system, is the economic value of architecture. The Two Values of Software Martin next argues that every software system offers its owners two distinct kinds of value, and that organisations routinely mistake which is more important. The first value is behaviour. Software exists to make machines do something that saves or earns money for its users. Programmers are hired to write code that behaves as the requirements say, and to fix it when it does not. Many programmers, Martin observes, believe that this is the whole of their job. The second value is structure, or what Martin also calls architecture. The word "software" is built from "soft" and "ware", and the point of software, as opposed to hardware, is that it should be easy to change. When stakeholders change their minds about a feature, the change ought to be simple to make, and its difficulty ought to be proportional to the scope of the change, not to the shape the system happens to be in. When a small change requires a huge effort because it collides with an awkward structure, the software has lost the property that justifies its name. Which value is greater Business stakeholders, if asked, will almost always say that working behaviour matters more. Martin disagrees, and his argument is worth stating carefully, because it is the kind of reasoning examiners like to test. Consider two extreme programs. The first works perfectly today but is impossible to change. The second does not work at all but is easy to change. Which is worth more? The first program will become useless as soon as requirements shift, and requirements always shift. Nobody can adapt it, so its value decays to zero. The second program can be made to work, because making it work is simply another change, and it can keep working as requirements evolve. The easy-to-change program is therefore the more valuable one. Martin concedes that a truly unchangeable program is rare, but he notes that many systems become so expensive to change that the cost of a change exceeds the benefit it would bring, which amounts to the same thing in practice. The argument does not say that behaviour is unimportant. It says that behaviour is fragile without structure, and that an organisation which trades structure away for short-term behaviour is consuming the thing that lets it keep delivering behaviour. The Eisenhower matrix To explain why organisations still get the priority wrong, Martin borrows a device associated with President Dwight D. Eisenhower, who is reputed to have said that he had two kinds of problems, the urgent and the important, and that the urgent ones were rarely important and the important ones rarely urgent. Management trainers turned this into a two-by-two grid. Picture two questions asked of every task: is it important, and is it urgent? The answers produce four quadrants, which are conventionally ranked like this: 1. Urgent and important. 1. Important but not urgent. 2. Urgent but not important. 3. Neither urgent nor important. Martin places the two values of software onto this grid. Behaviour is usually urgent, because a customer or a deadline is waiting, but it is not always important in the long run. Architecture is important but almost never urgent, because no customer is shouting for a cleaner dependency structure this week. So behaviour sits in the first and third quadrants, and architecture sits in the first and second. The classic failure is to treat everything in the third quadrant, urgent but unimportant, as if it belonged in the first. A feature request arrives with a deadline attached, and because it is loud, it displaces the quieter work of the second quadrant. Repeat this for a year and the second quadrant is never visited. The architecture decays, and the signature of a mess appears. Martin's conclusion is that business managers are often not equipped to judge the importance of architecture, because it is invisible to them. That is precisely why software developers are hired: part of their professional responsibility is to assert the importance of structure over the urgency of features. He describes this as a struggle, and he means it literally. The development team is one of the stakeholders of the system, and it must fight for the long-term health of what it builds, just as sales fights for features and finance fights for cost control. If the architecture comes last, the system will become ever more costly to develop, and eventually change will become practically impossible. When that happens, Martin says, the development team did not fight hard enough. What This Means for Managers The first two ideas of the book translate into a small set of managerial habits. The guide draws them out here because they recur in every later chapter. The first habit is to treat changeability as a requirement. When a project is scoped, the question "how will we change this later?" deserves the same seriousness as "what will this do?". A system that meets every stated feature but hard-wires a particular database, vendor or user interface into its core has quietly failed a requirement nobody wrote down. The second habit is to watch the right indicators. Rising headcount paired with flat throughput, lengthening estimates for changes of similar size, and frequent regressions in unrelated areas all suggest structural decay. Several engineering metrics used in industry, such as the lead time from an idea to production and the rate at which changes fail, capture some of this, though none measures structure directly. The third habit is to budget for the second quadrant. Important but non-urgent work does not happen by accident. Teams that keep their systems healthy usually protect time for it explicitly, whether as a share of each iteration, as refactoring folded into every feature, or as dedicated improvement work tied to a business goal. The fourth habit is to take engineers seriously when they argue for structure. Martin's point about the development team as a stakeholder cuts both ways. Engineers must make the case in terms the business understands, such as cost, speed and risk, rather than in terms of purity or taste. Managers, in turn, must accept that an argument about structure is an argument about money. The guide adds one qualification. Not every system deserves heavy investment in structure. A prototype meant to test a market hypothesis and then be thrown away, or a script that will run once, has a short life, and the cost-of-change curve never has time to bend. Martin's argument has most force for systems expected to live and grow for years, which, in practice, is most systems that reach production, including many that were described as prototypes when they were built. Who owns the architecture A final question follows from Martin's argument: if architecture emerges from everyone's decisions, who is responsible for it? Martin's answer has two parts. Architects must remain programmers, close enough to the code to feel the effects of their decisions. And every developer carries part of the responsibility, because every change either respects the structure or erodes it. The guide adds the managerial corollary. Accountability for structure cannot be assigned to a single role and then forgotten. It has to be built into how work is planned, reviewed and measured, so that the long-term health of the system is someone's explicit concern at every level, from the individual pull request to the portfolio budget. Key Takeaways · Martin treats design and architecture as one continuous activity; high-level shape and low-level detail cannot be separated. · The goal of architecture is economic: to minimise the human effort needed to build and maintain a system over its life. · The signature of a mess is growing headcount, flat output and rising cost per change; it arises from repeatedly trading structure for speed. · Software has two values, behaviour and structure; Martin argues structure is the greater, because an easy-to-change system can always be made to work. · On the Eisenhower matrix, architecture is important but rarely urgent, so it is chronically neglected unless developers and managers deliberately defend it. Review Questions 1. Explain Martin's reason for refusing to distinguish between design and architecture, and state one consequence for how architecture is managed. 1. Describe the pattern Martin calls the signature of a mess and list three symptoms a manager might observe. 2. Reconstruct Martin's argument that a program that does not work but is easy to change is more valuable than one that works but cannot be changed. What objection could be raised? 3. Using the Eisenhower matrix, explain why architectural work is routinely displaced by feature work. 4. In words, contrast a steep cost-of-change curve with a flat one, and explain how architecture relates to the difference. Chapter 2: Paradigms as Disciplines Before Martin discusses principles or architecture, he takes a detour through history. He asks what the major programming paradigms actually contributed to the craft. His answer is counterintuitive and, once grasped, illuminating: each paradigm took something away from programmers. None of them added a new capability that machines lacked. Each imposed a discipline, a restriction on what programmers are allowed to do, and those restrictions turn out to be the raw materials of architecture. A paradigm, in Martin's usage, is a way of programming that tells you which structures to use and when. He identifies three: structured programming, object-oriented programming and functional programming. He adds the provocative claim that there are probably no more to come, because each of the three removes a fundamental freedom and there are few freedoms left to remove. Whether that claim holds is debatable, but the framework is useful for anyone trying to understand why software engineers argue so fiercely about seemingly small matters of style. Structured Programming: Discipline on Direct Transfer of Control In the early decades of computing, programs routinely used an instruction usually called "goto", which told the machine to jump unconditionally to some other point in the program. Jumps made it possible to write anything, but they also made it possible to write programs whose flow was impossible to follow, with control leaping backward and forward across hundreds of lines. Edsger Dijkstra, the Dutch computer scientist, saw this as a barrier to reasoning. He wanted programmers to be able to prove, in something like the way mathematicians prove theorems, that their programs were correct. He found that unrestrained jumps made such proofs impractical, whereas programs built from a small set of control structures could be decomposed into pieces small enough to reason about. His short 1968 letter to the Communications of the ACM, published under the famous title "Go To Statement Considered Harmful", set off a long controversy. Earlier theoretical work by Corrado Böhm and Giuseppe Jacopini had already shown that any program can be written using just three structures: sequence (do this, then that), selection (if this, do that, otherwise the other) and iteration (repeat this while a condition holds). Over the following years the programming world adopted this discipline, and modern languages either lack an unrestricted jump or discourage it. Proof gave way to testing Martin makes an interesting observation about how the story ended. Dijkstra's dream of mathematical proof never became everyday practice; formal proofs of ordinary programs proved too laborious. What survived was a different idea borrowed from science. A scientific theory cannot be proved true, but it can be shown to be false by a single counterexample. Theories that resist repeated attempts at falsification are accepted as good enough to rely on. Dijkstra himself had remarked that testing shows the presence of bugs, not their absence. Software, Martin argues, works the same way. We cannot prove a program correct, but we can write tests that try to show it is incorrect, and a program that survives thorough testing is accepted as correct enough. This only works if the program can be broken into small units that can each be tested. Structured programming makes that decomposition possible. It is therefore, in Martin's view, the foundation of functional decomposition, and the reason architects still care about breaking systems into small, provable, testable parts. The managerial reading For a non-programmer, the lesson is that constraints enable verification. A team that restricts itself to a predictable set of forms can check its work. A team with unlimited freedom produces work that nobody, including its authors, can confidently assess. The same principle appears in finance, where standard reporting formats allow audit, and in manufacturing, where standard parts allow inspection. Object-Oriented Programming: Discipline on Indirect Transfer of Control Object-oriented programming is usually explained with three words: encapsulation, inheritance and polymorphism. Martin examines each and concludes that only the third is a genuine contribution to architecture. Not a model of the real world Martin first sets aside a popular definition. Object orientation is often described as a way of modelling the real world, with software objects standing for things such as customers, invoices and vehicles. Martin regards this as too vague to be useful. Plenty of good object-oriented code has no obvious real-world counterpart, and plenty of real-world modelling can be done without objects. He looks instead for something more precise: a mechanism that object-oriented languages provide and that earlier approaches did not provide safely. Encapsulation and inheritance were not new Encapsulation means drawing a line around data and the functions that operate on it, so that outsiders see only what they need. Martin points out that older languages such as C already supported a strong form of this: a programmer could publish a list of available functions while keeping the underlying data entirely hidden. Some later object-oriented languages actually weakened this separation, because their conventions required details of a class's internal data to be visible to other parts of the program even when they were not meant to be used. Inheritance means defining a new kind of thing as an extension of an existing kind. Martin shows that careful programmers achieved much the same effect before object-oriented languages existed, by arranging data so that one structure could be treated as a variant of another. Object-oriented languages made this more convenient and safer, but they did not invent it. Polymorphism is the real contribution Polymorphism, from the Greek for "many forms", is the ability of a single instruction to produce different behaviour depending on what kind of thing receives it. An everyday example helps. A library issues the instruction "return this item". A printed book goes to a shelf, a laptop goes to a charging cabinet, and an e-book simply expires. The person issuing the instruction does not need to know which kind of item is involved. Each item knows how to handle "return" in its own way. Programmers had achieved polymorphism before, by hand, using tables of pointers to functions. Operating systems, for instance, allowed a program to write output without knowing whether the output was going to a screen, a printer or a file. But hand-built polymorphism was fiddly and dangerous, because a single mistake in the table could cause baffling failures. Object-oriented languages, beginning with Simula, designed by Ole-Johan Dahl and Kristen Nygaard in the 1960s, made polymorphism safe and routine. That, Martin says, is the discipline: object orientation imposes order on indirect transfers of control, just as structured programming imposed order on direct ones. Dependency inversion: the architectural payoff Why does this matter so much? Because polymorphism lets programmers reverse the direction of dependencies in source code. In a traditional program, high-level functions call lower-level functions, and to do so they must name them. The high-level code therefore depends on the low-level code: if the low-level code changes, the high-level code may have to be recompiled, retested and redeployed. The direction in which control flows at runtime and the direction of source-code dependency are the same, and both run downward, from policy toward detail. With polymorphism, the high-level code can instead depend on an abstract description of a service, often called an interface, which says what operations exist without saying how they are performed. The low-level code then promises to fulfil that description. At runtime, control still flows from high level to low level. But in the source code, the arrow now points the other way: the low-level module depends on the interface, which belongs with the high-level module. This reversal is called dependency inversion. Martin regards this as the single most important capability object-oriented programming offers. It gives architects absolute control over the direction of every source-code dependency in a system. That control makes two things possible. First, a system can be arranged so that business rules do not depend on the user interface or the database; instead, the user interface and the database depend on the business rules. Second, the parts can be compiled and deployed separately. A change to the database code need not force the business rules to be rebuilt. Martin calls the resulting style a plugin architecture: details plug into the core, much as a peripheral plugs into a computer through a standard port. For a manager, dependency inversion is the technical basis of vendor independence and parallel development. If the core business logic defines the port and the suppliers build adaptors to fit it, then suppliers can be replaced, and separate teams can work on the core and on the adaptors without constantly blocking one another. Functional Programming: Discipline on Assignment The third paradigm is, in terms of its ideas, the oldest. It grows out of the lambda calculus developed by Alonzo Church in the 1930s, and one of its best-known early languages, LISP, was created by John McCarthy in the late 1950s. Its defining discipline, in Martin's account, is a restriction on assignment. In a purely functional style, once a value has been given to a variable, it is never changed. Programs compute new values from old ones rather than altering existing values in place. Why immutability matters to architects The practical consequence concerns concurrency, meaning situations where several parts of a program, or several programs, run at the same time. Martin observes that nearly all the hardest bugs in concurrent systems, such as race conditions, deadlocks and inconsistent updates, arise because two activities try to change the same data at once. If data is never changed, those problems cannot arise. Immutability removes a whole family of failure modes at a stroke. Complete immutability is rarely practical, because real systems must record that things happen. Martin's architectural advice is therefore to segregate. Divide a system into components that never change data and a small number of components that do. Push as much logic as possible into the immutable components, and protect the mutable ones carefully with mechanisms designed to manage concurrent updates. Event sourcing He also describes a technique that shows how far the idea can go. Imagine a bank that, instead of storing each customer's current balance and overwriting it after every transaction, stores only the transactions themselves. The balance is never saved; it is calculated whenever needed by adding up the history. Nothing is ever updated or deleted, only appended. This approach is known as event sourcing. It trades storage and computation for a complete, trustworthy record and freedom from update conflicts. Martin notes that source-code version control systems work in essentially this way, and that as storage and processing become cheaper, the approach becomes more practical for business systems. A manager will recognise the accounting parallel. A general ledger records entries; balances are derived. The history is the source of truth, and corrections are made by adding new entries rather than erasing old ones. That design, centuries old in accounting, is exactly what functional thinking recommends for software. Paradigms in Today's Languages Martin's framing is historical, but it applies directly to the languages organisations use now. Most mainstream languages are no longer pure examples of any one paradigm. Java, first released in the mid-1990s as a thoroughly object-oriented language, added lambda expressions, a functional feature, in 2014. C#, JavaScript, Python, Kotlin and Swift all mix structured control flow, objects and functional techniques. Several newer languages make immutability the easy default: Kotlin and Swift, for example, distinguish between values that can be reassigned and values that cannot, and nudge programmers toward the latter. Popular user-interface libraries on the web encourage developers to treat application state as immutable and to produce new state rather than altering the old. The practical consequence is that the disciplines Martin describes are rarely enforced by the language alone. A modern language usually allows a programmer to write tangled control flow, to bypass polymorphism with long chains of type checks, and to change shared data freely. Whether a team works within the disciplines depends on its conventions, code review, automated checks and the example set by senior engineers. For managers, this shifts attention from language selection to engineering practice. Choosing a fashionable language does not, by itself, confer the benefits of any paradigm. What confers them is a team that understands which freedoms to give up and why. Coding standards, review checklists and static analysis tools are, in Martin's terms, instruments for imposing discipline, and their value lies in the classes of error they prevent. When a team proposes adopting a new language or library, a useful question is which mistakes it makes harder to commit, and whether those are the mistakes that have actually been costing the organisation money. There is also a staffing implication. Because the disciplines are portable across languages, engineers who understand them can move between technologies with relatively little loss of effectiveness. Engineers who know only the surface features of one framework cannot. Martin's argument that the fundamentals of software have changed little over half a century supports investing in those fundamentals when hiring and training. What the Three Paradigms Teach Together Martin draws the threads together in a way that serves the rest of the book. The three paradigms map onto the three great concerns of architecture. Structured programming underpins the algorithmic decomposition of a system into small, testable units. Object-oriented programming, through polymorphism, provides the means of controlling dependencies across boundaries. Functional programming provides a discipline for managing where and how data changes. Table 1 sets the three side by side. Table 1. The three paradigms as disciplines. Paradigm Freedom removed What it enables Architectural concern Structured Unrestricted jumps Decomposition into testable units Function and testing Object-oriented Unsafe indirect calls Safe polymorphism, dependency inversion Separation across boundaries Functional Changing values in place Freedom from concurrency conflicts Control of data and state Martin also makes a broader, almost philosophical point. In roughly half a century since these paradigms appeared, the basic materials of software have changed remarkably little. Programs are still made of sequence, selection, iteration and indirection, applied with discipline. Hardware has become enormously faster and cheaper, and languages and tools have improved, but the rules that decide whether a program is well structured are much the same. This continuity is why Martin believes that architectural principles can be stated in general terms and expected to last. The guide adds a note of caution. Martin's claim that no further paradigms remain is a prediction, not a finding, and many practitioners would point to logic programming, dataflow and reactive styles as candidates he sets aside. His narrower point, that each paradigm's value lies in a discipline rather than a new power, is the part worth keeping. It shifts the conversation about technology choices away from "what can this language do?" toward "what mistakes does this approach prevent?", which is usually the more useful question for an organisation. Key Takeaways · Martin presents structured, object-oriented and functional programming as disciplines that each remove a freedom rather than add a capability. · Structured programming restricts jumps, enabling decomposition into small units that can be tested; testing, like science, shows the presence of defects, not their absence. · Object orientation's decisive contribution is safe polymorphism, which lets architects invert source-code dependencies so that details depend on policy. · Dependency inversion underpins plugin architectures, independent deployment and freedom from particular vendors. · Functional programming restricts assignment; segregating immutable from mutable components, and techniques such as event sourcing, remove whole classes of concurrency failures. Review Questions 1. Explain what Martin means by calling each paradigm a discipline. Why does he think this is a more accurate description than calling them new capabilities? 1. How did the idea of scientific falsification come to replace mathematical proof in Martin's account of structured programming? 2. Using a non-technical example, explain polymorphism and describe how it allows the direction of a source-code dependency to be reversed. 3. Why does Martin regard dependency inversion as the most important contribution of object orientation to architecture? Give one business benefit. 4. Describe event sourcing and explain its connection to both immutability and traditional bookkeeping. Chapter 3: The SOLID Principles If programming paradigms are the bricks of software, Martin says, then the SOLID principles tell us how to arrange the bricks into walls and rooms. They operate at what he calls the mid-level of software structure: the level of modules and classes, where functions and the data they use are grouped together. The principles emerged in Martin's writing and in discussions among software practitioners from the late 1980s onward, and they acquired their acronym in the early 2000s, when Michael Feathers pointed out that the initial letters of the five principles could be rearranged to spell SOLID. Martin states the purpose of the principles plainly. They aim to create structures that tolerate change, are easy to understand, and can serve as the basis of components used in many systems. Each principle is a rule for managing change: who causes it, where it lands, and how far it spreads. This chapter explains each principle without any program code, using small illustrations from ordinary organisational life. The reasoning is preserved; the syntax is not needed. S: The Single Responsibility Principle The Single Responsibility Principle is the most misunderstood of the five, and Martin devotes care to correcting the misunderstanding. Many programmers take it to mean that a module should do only one thing. That is a sound rule for individual functions, but it is not what the principle says. Martin's final formulation is that a module should be responsible to one, and only one, actor. An actor here is a group of people, one or more users or stakeholders, who want the system changed for the same reasons. The principle is therefore about people and organisations, not about technical tasks. It asks: whose requirements drive changes to this piece of code? If the answer is more than one group, the code is at risk. The payroll illustration Martin's example concerns an employee record in a business system that offers three operations: calculating pay, reporting hours worked, and saving the record to the database. Each operation serves a different part of the organisation. Pay calculation is specified by the finance function, reporting hours by the operations function, and saving by the technical function that manages the database. Suppose the calculations for pay and for hours reporting both rely on a shared routine that computes regular, non-overtime hours. Now the finance team asks for a change in how regular hours are counted for pay purposes. A developer makes the change in the shared routine, the tests for payroll pass, and the change is deployed. But the operations team's hours report also used that routine, and nobody told them. Their figures are now wrong, and the error may go unnoticed for months while decisions are made on bad numbers. Martin calls this accidental duplication. Two groups appeared to need the same thing, so the code served both from one place. In truth their needs were only coincidentally similar and were free to diverge. By tying them together, the design allowed one actor's change to damage another actor's results. A second symptom is merge conflict. If the finance team and the technical team both request changes to the same module at the same time, two developers edit the same file independently, and their changes must be reconciled. Reconciliation is error-prone and puts both groups at risk. The remedy The cure is to separate code that different actors depend on. Martin describes several ways to do this, including keeping the shared data in a simple structure and giving each actor's logic its own module, with a thin front door, sometimes called a facade, that directs requests to the right place. The details matter less than the principle: each actor's rules live in their own place, so that changes requested by one group cannot quietly break another group's work. The organisational reading Students of management will recognise the logic. A job description that reports to three bosses creates conflicts; a process owned by three departments gets changed inconsistently. The Single Responsibility Principle is the software version of clear accountability. It also has a striking connection to Conway's law, the observation by Melvin Conway in 1968 that systems tend to mirror the communication structures of the organisations that build them. Martin's principle turns that observation into a design rule: align module boundaries with the boundaries between the groups who request changes. O: The Open-Closed Principle The Open-Closed Principle was formulated by Bertrand Meyer in his 1988 book on object-oriented software construction. It says that a software artefact should be open for extension but closed for modification. In plain terms, you should be able to change what a system does by adding new code rather than by altering existing code. Martin considers this the most fundamental reason we study software architecture at all. If simple extensions to requirements force massive changes to existing code, the architects have failed. The financial report illustration His illustration is a system that displays a financial summary on a web page, with negative numbers shown in red. The stakeholders then ask for the same information as a printed report, paginated, with proper headers and with negative numbers shown in brackets. How much existing code should have to change to add the printed version? In a well-designed system, very little. Martin describes separating the work into distinct responsibilities: one part calculates the figures, another prepares them for presentation, and separate parts render them for the web and for print. Crucially, he arranges the dependencies so that they all point toward the part that calculates the figures. The calculation, which embodies the business rules, depends on nothing that could change because of a presentation decision. The web and print renderers depend on the calculation, not the other way round. The result is a hierarchy of protection. The business rules sit at the top, protected from changes in everything else. Presentation logic depends on them. The specific renderers depend on the presentation logic. Adding a new output format, such as a spreadsheet export or a mobile view, means adding a new renderer. Nothing above it changes. The managerial reading The Open-Closed Principle is the software equivalent of a product platform. A car manufacturer that designs a common chassis can add new body styles without redesigning the drivetrain. A bank that defines a stable interface for payment providers can add a new provider without rewriting its ledger. The strategic value is that growth in variety does not produce growth in risk to the core. Martin adds a subtle point about direction. The principle is achieved by arranging components so that higher-level ones, which represent more important policy, are protected from changes in lower-level ones. This idea returns later as the organising logic of the whole Clean Architecture. L: The Liskov Substitution Principle The Liskov Substitution Principle takes its name from Barbara Liskov, the computer scientist who stated a version of it in a 1988 paper on data abstraction and hierarchy. In informal terms, it says that if a system is written to work with a certain kind of thing, it should keep working correctly when given any subtype of that thing. Substitutes must honour the contract of what they replace. The square and the rectangle Martin uses a classic puzzle. In geometry, a square is a special kind of rectangle. It seems natural, therefore, to define a square in software as a kind of rectangle. But a rectangle in software typically allows its width and height to be set independently. A square cannot allow that; changing one side must change the other. Now imagine a piece of code that takes a rectangle, sets its width to five and its height to two, and expects the area to be ten. Give it a square instead and the area comes out wrong, because setting the height also changed the width. The square is not a valid substitute for the rectangle, whatever geometry says. The code using it would have to check which kind of shape it had received, and every such check is a place where the design has broken down. The taxi dispatch illustration Martin's more architectural example concerns a taxi aggregation service that dispatches rides to many independent taxi companies. Each company exposes a web address through which the aggregator can send a booking, and all companies are supposed to follow the same format, so the aggregator can treat them interchangeably. Then one large company, in Martin's hypothetical, implements the format slightly differently, abbreviating one of the fields. The aggregator must now add a special case: if the company is this one, use the abbreviation. Every such exception clutters the core dispatch logic, invites errors, and creates a security and maintenance burden. Martin suggests that, if exceptions cannot be avoided, they should be handled through a configuration mechanism kept away from the core, but the real lesson is that a violation of substitutability pollutes the architecture. The managerial reading The Liskov principle is about honest contracts. A supplier who claims to conform to an industry standard but deviates in small ways imposes a hidden cost on every customer, who must write special handling. Procurement teams deal with this constantly: two parts with the same specification should be interchangeable, and if they are not, the specification is lying. In software, the principle has expanded from its origins in class hierarchies to cover any interface, including web services and data feeds. Wherever a system promises that implementations are interchangeable, they must truly be so. I: The Interface Segregation Principle The Interface Segregation Principle says that no module should be forced to depend on things it does not use. The operations illustration Martin describes a single module, which he calls an operations class, that offers three operations used by three different users. Each user needs only one of the operations. But because all three are bundled together, each user depends, in the source code, on all three. When the operation used only by the first user changes, the other two users may have to be rebuilt and redeployed, even though nothing they use has changed. In languages that require recompilation, this is directly costly; in others, it is a risk and a source of confusion. The remedy is to give each user a narrow interface containing only what it needs. The underlying module can still provide all three operations, but each user sees, and depends on, just its own slice. Beyond classes Martin generalises the principle to architecture. Depending on something that carries baggage you do not need can cause trouble you did not expect. He imagines a system that uses a framework, which in turn is tied to a particular database. The system needs none of the database's features, yet a change in the database can force the framework to be redeployed, which forces the system to be redeployed. Worse, a failure in the unused database feature can break the system. The managerial reading The organisational analogy is the all-in-one contract or the monolithic software suite. An organisation that buys a vast platform to use a single feature inherits every upgrade, outage and licence change affecting features it never touches. Narrow, purpose-specific dependencies reduce exposure. The guide notes that this is also an argument for small, well-defined service agreements between internal teams: a team should commit to what its consumers actually need, not expose its entire internal surface. D: The Dependency Inversion Principle The final principle generalises the mechanism introduced in the previous chapter. The most flexible systems, Martin says, are those in which source-code dependencies refer only to abstractions, not to concrete implementations. Stable abstractions, volatile details Taken literally, this would be impossible; every system must eventually use concrete things. Martin therefore refines the rule. It is not a problem to depend on concrete things that are stable, meaning they rarely change. The standard text-handling facilities of a programming language, for example, are concrete but change so rarely that depending on them is harmless. The danger lies in depending on volatile concrete things: the modules the team is actively developing and that will change frequently. Abstractions, by contrast, tend to be stable. An interface that describes what a payment gateway must do changes far less often than any particular gateway's implementation. So the principle becomes practical guidance with four parts, which Martin states roughly as follows. Refer to volatile things through abstract interfaces rather than directly. Do not build new modules by extending volatile concrete ones. Do not override concrete behaviour that other code relies on. And never mention the name of anything concrete and volatile. The last rule raises a practical question: if high-level code may not name a concrete class, how does it ever create one? Martin's answer is a pattern called an abstract factory. The high-level code asks an abstract factory to produce the object it needs. The concrete factory, which does know the specific class, lives on the low-level side of the boundary. Martin describes the boundary between the two sides as a curved line separating the abstract from the concrete. All source-code dependencies cross that line pointing toward the abstract side. Control, at runtime, flows the other way. That is the inversion. Some violation is unavoidable. Somewhere, a concrete object must be created and wired to the abstractions. Martin says such violations should be gathered into a small number of concrete components, typically the main component that starts the system, which is discussed in a later chapter. An office illustration Consider a head office that needs documents printed. If its procedures say "send the file to the Model X printer on the third floor", every printer upgrade requires rewriting procedures. If its procedures say "send the file to the print service", and the facilities department is responsible for making some printer satisfy that service, then printers can change freely. The head office depends on an abstraction it defined; facilities depends on the same abstraction. The dependency of policy on equipment has been inverted. The managerial reading Dependency inversion is the principle that most directly protects strategic choices. It is how an organisation keeps the option to switch cloud providers, payment processors, messaging tools or databases without rewriting its core logic. It also clarifies who owns an interface. In an inverted design, the interface belongs to the high-level policy that uses it, not to the supplier that implements it. That ownership reflects bargaining power: the business defines what it needs, and suppliers adapt. The Principles Side by Side The five principles work together. The Single Responsibility Principle tells us where to cut. The Open-Closed Principle tells us to arrange the pieces so that extension does not disturb the core. The Liskov principle ensures that the pieces we plug in keep their promises. Interface segregation keeps each dependency narrow. Dependency inversion ensures that dependencies point toward stable policy rather than volatile detail. Table 2 condenses them for revision. Table 2. The SOLID principles for managers. Principle Core rule Everyday analogue Warning sign Single Responsibility Answer to one actor One boss per role One team's change breaks another's report Open-Closed Extend without modifying Common product platform Every new variant edits the core Liskov Substitution Substitutes honour the contract Truly interchangeable parts Special cases for particular suppliers Interface Segregation Depend only on what you use Narrow service agreements Redeploying for changes you do not use Dependency Inversion Depend on stable abstractions "Print service", not "Model X" Business logic names specific technologies A final point is worth making about the status of these principles. Martin presents them as principles, not laws. Applied mechanically, each can produce clutter: interfaces with a single implementation that will never have a second, or modules split so finely that nobody can find anything. Martin's own framing ties each principle to a specific kind of change. Where that kind of change is plausible, the principle earns its cost. Where it is not, the principle can be relaxed. This theme of judgement, rather than rule-following, returns in the critical discussion toward the end of this guide. Key Takeaways · The SOLID principles govern the mid-level arrangement of modules and classes, and each is a rule for controlling the impact of change. · The Single Responsibility Principle means a module should answer to one actor, a group of people who request changes for the same reasons; it is not "do one thing". · The Open-Closed Principle protects high-level policy by arranging dependencies so that new behaviour is added rather than existing code edited. · The Liskov and Interface Segregation principles demand honest, narrow contracts, so that substitutes behave as promised and nothing depends on what it does not use. · The Dependency Inversion Principle directs dependencies toward stable abstractions owned by business policy, which is the technical basis of vendor and technology independence. Review Questions 1. Why does Martin reject the reading of the Single Responsibility Principle as "a module should do only one thing"? Illustrate accidental duplication with an example of your own. 1. Explain how the Open-Closed Principle creates a hierarchy of protection, using the financial report illustration. 2. Why is a square not a valid substitute for a rectangle in software, and what broader lesson does the taxi dispatch example teach? 3. Give an organisational example of a violation of the Interface Segregation Principle and describe its costs. Distinguish stable from volatile concrete dependencies. Why does Martin allow the former but not the latter, and where should unavoidable violations be collected? Hashtags: #TheBlueprintOfCode #CleanArchitecture #RobertCMartin #SoftwareArchitecture #SoftwareDesign #ArchitectureEconomics #CostOfChange #TechnicalDebt #DependencyManagement #DependencyRule #BusinessRules #SeparationOfConcerns #SOLIDPrinciples #SingleResponsibilityPrinciple #OpenClosedPrinciple #LiskovSubstitutionPrinciple #InterfaceSegregationPrinciple #DependencyInversionPrinciple #Polymorphism #PluginArchitecture #StructuredProgramming #ObjectOrientedProgramming #FunctionalProgramming #EventSourcing #FutureOfSoftwareArchitecture

Download the Book (PDF): Introduction On 23 August 2026, in an exhibition hall at Porte de Versailles on the southern edge of Paris, a French Trackmania driver named Gwendal Duparc won a race that was worth far more than the race itself. His victory settled the Club Championship of the Esports World Cup, a seven-week festival of twenty-five competitions across two dozen video games, and it handed his organisation, the Chinese club AG.AL, a first prize of $7 million. The club that finished second, Team Falcons, had won the same title in each of the two previous years. The whole event carried a prize pool of $75 million, the largest in the history of competitive gaming. It was funded by Saudi Arabia's Public Investment Fund, and it was meant to be held in Riyadh. It moved to Paris only in May, after war with Iran made the Gulf an unsafe place to fly two thousand competitors and their audiences. Almost everything about that afternoon illustrates how the business of e-sports actually works in the middle of the 2020s. The money came from a sovereign wealth fund rather than from ticket buyers or broadcasters. The prize was paid not to a player but to a club, and the club was rewarded for breadth, for fielding teams across many unrelated games, rather than for excellence in one. The games themselves belonged to other companies: Trackmania to Ubisoft, League of Legends to Riot Games, Counter-Strike to Valve, and so on through the list. None of those publishers could be told what to do by the organiser, and each had licensed its game to the event on its own terms. And the venue could be moved across a continent in a matter of weeks, because nothing about the competition was tied to a place in the way that a football club is tied to a city. This booklet is about the commercial machinery behind scenes like that one: how sponsorships are sold, how players are hired and paid, how leagues are organised and why so many of them have been reorganised, who has invested in teams and what they thought they were buying, how the broadcasts are distributed, and how the sale of virtual items inside games has become the single most important source of money that professional competition can draw on. It is written for intelligent readers who are not specialists: investors wondering whether e-sports is a real asset class, marketers deciding whether to sponsor a team, lawyers and agents drafting player contracts, students of sport business, and fans who want to understand why the teams they follow keep changing leagues, owners and names. The argument The booklet has one argument, and every chapter serves it. In e-sports, the publisher owns the field. That sentence sounds like a technicality. It is not. In traditional sport, nobody owns football. The rules of association football belong in practice to a body, the International Football Association Board, but no company can switch the game off, change its physics overnight, or decide that a particular tournament may not use it. A club that buys a stadium, signs players and builds a fan base owns something durable. Its league may be governed badly, but the underlying sport is a public good. Competitive video games are private property. League of Legends is a product owned by Riot Games, which is itself owned by the Chinese technology group Tencent. Counter-Strike 2 is owned by Valve. Overwatch and Call of Duty belong to Activision Blizzard, which Microsoft bought in 2023. Every tournament, every broadcast of every tournament, and every professional contract in those games exists because the owner permits it. The owner can change the game with a patch, license or refuse to license a tournament, ban a player, cap the prize money of community events, or stop supporting the game altogether. When the owner decides to reorganise its competitive ecosystem, everyone who invested in the old arrangement has to live with the new one. From that single structural fact follows most of what is distinctive about e-sports as a business. Franchise fees that looked like the price of a permanent asset turned out to be the price of a licence that the licensor could redraw. Sponsorship deals depend on audiences that the publisher, not the team, largely controls. Streaming rights are sold by the publisher, while the teams that provide the talent often receive nothing directly from them. Player contracts are shaped by publisher rules on salaries, transfers and conduct. Investors who valued teams as if they were young versions of the Dallas Cowboys discovered that a team's franchise could lose most of its value when a publisher changed course. And the virtual items that now fund much of the ecosystem, from weapon skins to team-branded bundles, are sold by publishers who decide how much of the revenue, if any, flows back to the competitors. The thesis of this booklet is that the first great wave of e-sports investment, roughly from 2017 to 2022, mispriced that dependency, and that the arrangements surviving in 2026 are those that face it squarely. Three kinds of arrangement have endured. The first plugs teams directly into the publisher's own revenue from digital goods, so that the publisher's interest in selling items becomes the team's interest too. The second draws on patient capital from a sovereign investor, above all Saudi Arabia, for whom e-sports serves national purposes beyond profit. The third builds on audiences of creators and fans whom the publisher cannot easily replace, so that a team or personality has bargaining power of its own. Organisations that relied on none of these, and that tried to behave as independent sports franchises with a durable asset, have mostly shrunk, merged, or disappeared. The state of play in 2026 The past three years have tested that thesis hard. The Overwatch League, the most ambitious attempt to transplant North American franchise sport into gaming, was folded in January 2024, six years after teams paid $20 million each to join. Riot Games merged its North American and Brazilian League of Legends leagues into a single League of the Americas for 2025, then reversed the decision within a year and restored both the LCS and Brazil's CBLOL for 2026. In April 2026 Riot announced that its Valorant Champions Tour, whose partnership leagues had begun only in 2023, would move to open regional qualifiers from 2027, with partner teams receiving support payments but no guaranteed places. Valve had already banned exclusive partnerships between teams and tournament organisers in Counter-Strike from 2025. The closed, permanent league is in retreat almost everywhere except Call of Duty. Meanwhile the money has shifted. Riot reported sharing more than $100 million with its Valorant partner teams in 2025, most of it from sales of in-game items. Saudi Arabia's gaming group Savvy owns ESL FACEIT Group, the largest independent tournament operator, agreed in March 2026 to buy the Mobile Legends publisher Moonton for $6 billion, and sits within a state fund that completed the $55 billion buyout of Electronic Arts in August 2026. Riot opened its leagues in the Americas and Europe to betting sponsors in 2025, reversing a long-standing refusal. And the market for Counter-Strike weapon skins, often described as the largest secondary market for virtual items in the world, lost billions of dollars of notional value in a single day in October 2025 because Valve changed one crafting rule. Each of those developments is a variation on the same theme. Where a publisher changed its mind, value moved. How the booklet is organised Chapter 1 sets out the landlord problem in detail: what it means for a publisher to own a sport, how licensing works, and how publishers have used their power in cases from Blizzard's punishment of a Hong Kong player in 2019 to Nintendo's refusal to license the largest grassroots Smash Bros. circuit in 2022. Chapter 2 examines the franchise bet: what investors paid to join the Overwatch League, the LCS, the LEC and the Call of Duty League, what they thought they were buying, and why the numbers did not work. Chapter 3 follows the models that replaced pure franchising, from Riot's revenue-sharing pools to Valve's open circuit, and asks what a team now owns. The middle of the booklet turns to the three commercial relationships that fund a team. Chapter 4 deals with sponsorship: what is actually sold, why the crypto boom of 2021 ended badly, and what the arrival of betting sponsors means. Chapter 5 deals with streaming rights and the rise of co-streaming, which has shifted value from the owners of broadcasts to the personalities who re-broadcast them. Chapter 6 deals with players: how contracts are structured, how salaries and buyouts work, what the LCS walkout of 2023 revealed, and why player bargaining power remains weak in most titles. The final chapters deal with capital. Chapter 7 tells the story of venture investment and public listings, from FaZe Clan's $725 million stock-market debut to its sale for about $17 million little more than a year later. Chapter 8 examines Saudi Arabia's role as the sector's most important patron. Chapter 9 explains the economics of virtual items, the revenue source on which the whole structure increasingly rests, and the regulatory and market risks that come with it. The conclusion draws out what follows for anyone who wants to invest in, sponsor, work in, or regulate the sector. A note on scope. The booklet concentrates on the large publisher-run ecosystems in North America, Europe and the Middle East, because that is where the business models discussed here were invented and tested. Chinese and Korean e-sports have their own histories, their own broadcasters and their own labour markets, and they appear where they illuminate the argument rather than as subjects in their own right. Mobile e-sports in South-East Asia, now a major market, appears mainly through its effect on investment flows. Figures are given in United States dollars unless otherwise stated, and they are reported figures unless the text says they are official. In a business where most companies are private and many deals are confidential, it is worth saying plainly when a number is an estimate. Chapter 1: The Landlord Problem Every sport has an owner of some kind. Football has its governing bodies, Formula One its commercial rights holder, the National Football League its thirty-two clubs acting together. But in all of those cases the thing being governed, the game itself, existed before the governors and would survive them. Nobody can revoke the right to kick a ball. In e-sports the order is reversed. The game is a piece of software, written and owned by a company, protected by copyright, and licensed to its users under terms the company writes. Competition happens inside that property, on the owner's servers or using the owner's client, and every broadcast of it is a public performance of the owner's copyrighted work. The professional scene is a tenant. The publisher is the landlord. This chapter sets out what that ownership means in practice. It identifies four distinct powers that a publisher holds, shows how each has been used, and explains why the rest of the industry's economics, from franchise fees to sponsorship rates, have to be read in their light. Four powers of the owner The first power is the power to license. A tournament that uses a commercial game, charges for tickets, sells sponsorship or streams the action is using the publisher's intellectual property for commercial gain, and it needs permission to do so. Most publishers grant that permission in two tiers. Small community events are allowed under a general licence, usually published online, which sets limits on prize money, sponsorship and scale. Larger events require a specific licence, negotiated with the publisher, which can carry fees, revenue shares, branding rules and a right of approval over the schedule. Riot Games, Blizzard, Epic Games and Nintendo all operate some version of this model. Valve, the owner of Counter-Strike and Dota 2, was for many years the most permissive, allowing almost anyone to run events. Even Valve, however, now publishes a formal rulebook for licensed Counter-Strike competition, hosted on its public code repository, which tells organisers how teams must be invited, how compensation must be disclosed and which business relationships are forbidden. The second power is the power to design. A publisher can change the game at any time. Balance patches alter which characters, weapons or strategies are strong, and a team that has built its roster around a particular style can find it obsolete overnight. More drastic changes replace the game entirely. In 2022 Blizzard replaced Overwatch with Overwatch 2, which cut team sizes from six players to five and forced every professional roster to shed a player. In 2023 Valve replaced Counter-Strike: Global Offensive with Counter-Strike 2. Activision releases a new Call of Duty title almost every year, and its professional league switches to each new game, so that players and coaches must relearn maps, weapons and movement annually. No traditional sport has ever had to cope with the ball changing shape every autumn. The third power is the power to govern conduct. Because professional play depends on the publisher's licence and often takes place in the publisher's own league, the publisher can discipline players and teams directly. It writes the rulebook, runs the integrity investigations and imposes the bans. In most traditional sports, discipline is shared between independent governing bodies, leagues and clubs, and it is usually subject to collectively bargained procedures. In e-sports, the company that sells the game is also the judge. The fourth power is the power to exit. A publisher can decide that its competitive scene no longer serves its interests and simply end it. This has happened repeatedly. In December 2018 Blizzard announced that it would end the Heroes Global Championship, the professional circuit for its game Heroes of the Storm, from the following year. Teams and players learned that their careers in the game were over in the same announcement as the public. In 2023 Blizzard and the owners of the Overwatch League teams agreed revised terms under which the league could be wound up, the owners voted in November to leave, and in January 2024 it was replaced with a smaller circuit run by a third-party operator. In both cases, investment in rosters, facilities and fan bases lost most of its value because a single company chose to stop. The powers in use These powers are not theoretical. Three episodes show how they are exercised and what they cost. The first is the Blitzchung affair of October 2019. During a post-match interview at a Hearthstone Grandmasters event broadcast from Taiwan, the Hong Kong player Ng Wai Chung, who competed as Blitzchung, put on a protest mask and called for the liberation of Hong Kong. The following day Blizzard banned him from Grandmasters for a year, took back roughly $4,000 in prize money and ended its work with the two broadcasters who had been interviewing him. Blizzard pointed to a rule that allowed it to remove any player who brought the publisher into disrepute or offended a portion of the public. The reaction was immediate. American senators and members of Congress from both parties wrote to the company, the hashtag #BoycottBlizzard trended globally, some of the company's own employees walked out, and Mitsubishi Motors withdrew its sponsorship of Hearthstone e-sports in Taiwan. Within days Blizzard's president halved the ban and restored the prize money. The episode showed how far the conduct power reaches: a publisher can penalise a player for political speech made on its broadcast, and there is no independent tribunal to appeal to. It also showed the limits of that power. Blizzard backed down because of outside pressure, not because of any procedural protection the player had. The second episode is the collapse of the Smash World Tour in November 2022. Super Smash Bros., Nintendo's fighting series, has one of the most dedicated grassroots competitive communities in gaming, and for years Nintendo tolerated it rather than supported it. In 2013 the company tried to stop the Evolution Championship Series from streaming its Super Smash Bros. Melee tournament, then relented after a public outcry. In 2020 it sent a cease-and-desist letter to The Big House, a major Smash tournament, over plans to run an online event using a community-made modification that enabled internet play. The Smash World Tour of 2022 was the most ambitious community effort yet. According to its organisers it drew together more than 6,400 offline tournaments and more than 325,000 players, culminating in a championship with a $250,000 prize pool, the largest in the game's history. Less than two weeks before the championship, the organisers announced that it was cancelled, saying that Nintendo had told them it would not grant a licence for that event or for the 2023 circuit. Nintendo later said that it had not required the finals to be cancelled. The dispute turned on the details of licensing conversations that are not public, but the lesson for everyone involved was plain: a community can build a sport for years and still have no standing if the owner of the game decides it will not license the result. In October 2024 Nintendo published community tournament guidelines under which unlicensed events are restricted to modest scale, with reported caps of around two hundred in-person participants and $5,000 in prizes. Anything larger needs Nintendo's explicit approval. The third episode is quieter but more consequential for money. In August 2023 Valve announced that, from 2025, Counter-Strike tournament organisers would no longer be allowed to have unique business relationships with the teams that played in their events. Invitations would be determined by Valve's ranking system or by open qualifiers, and all compensation paid to teams would be public and based on objective criteria. The announcement ended the partner-league arrangements that ESL and BLAST, the two dominant Counter-Strike operators, had built with selected teams, in which partners received guaranteed invitations and a share of the organisers' revenues. Those arrangements had been negotiated between private companies in good faith, and some had years left to run. Valve did not need anyone's agreement to end them. It simply wrote a new condition into its licence. Valve's stated rationale was to restore open competition, and there is a good argument that it succeeded. The point here is structural: a business model that two large operators and dozens of teams had invested in was abolished by the owner of the game with a two-year notice period. The broadcast right as the root of control The licensing power ultimately rests on copyright, and the clearest early test of it came not in America or Europe but in South Korea, the country where professional gaming first became a mass spectator sport. Through the 2000s, Korean television channels built an entire industry around StarCraft: Brood War, Blizzard's strategy game. The Korea e-Sports Association organised professional leagues, corporate-sponsored teams paid salaries, and two cable channels broadcast matches to large audiences. For years all of this happened without any formal licence from Blizzard, whose game was the product being shown. In 2010, as it prepared to launch StarCraft II, Blizzard sued the broadcasters for infringing its copyright. The Korean e-sports establishment argued that it had built the competitive scene, that the matches were created by the players and organisers rather than by the publisher, and that the broadcasts had helped sell Blizzard's game. Blizzard's position was that none of that mattered: the game was its property, and showing it to the public required its permission. The dispute was resolved through licensing agreements in 2011, and it established, in the market where e-sports had first become an industry, the principle that has governed the business ever since. However much value a community creates around a game, the publisher controls the right to broadcast it. That principle is why publishers can dictate terms to tournament organisers, why they can capture the most valuable media rights, and why a team's own content, when it uses game footage, is published at the owner's tolerance. It is also why the publisher's power extends well beyond its own leagues. A publisher that does not run any competition itself still controls who may run one and on what terms, as Nintendo's treatment of Smash and Valve's rulebook for Counter-Strike both show. Why publishers behave as they do It would be easy to read these episodes as stories of corporate caprice. They are better understood as the rational behaviour of a company whose e-sports operation is not its main business. For almost every publisher, the competitive scene is a marketing and retention tool. Its purpose is to keep players engaged with the game, to give them aspirational figures to watch and emulate, to generate content between game updates and to sell items. Riot Games has described its leagues as an investment in the long-term health of its games rather than a standalone profit centre, and in 2024 it said openly that the sponsorship-led model it had built with teams was not sustainable. When Riot laid off about 530 employees, around 11 percent of its staff, in January 2024, it also cancelled a planned e-sports streaming network of its own, which tells us something about where competition ranks when budgets tighten. A publisher will support e-sports as long as it believes the scene sells the game and its items. When that belief weakens, or when the scene becomes a source of reputational risk, it will cut back. This produces a particular pattern of incentives. Publishers want the competitive scene to be large, visible and exciting, because that is what drives engagement. They do not especially care whether independent teams make a profit, except insofar as failing teams create bad publicity or leave gaps in the league. They want control over messaging, because the league is a marketing channel. And they want flexibility, because the game itself will change and the competitive structure has to change with it. None of these incentives is malicious. All of them make the publisher an awkward partner for anyone trying to build a durable, independent business on top of the game. There is also a timing problem. Traditional sports franchises are valued on the assumption that the sport will last indefinitely. A video game has a commercial life. League of Legends, released in 2009, and Counter-Strike, whose lineage goes back to 1999, have shown that some games can last decades, but they are exceptions. Most competitive titles have a peak of a few years. An investor in a team is therefore making two bets at once: a bet on the team's ability to win and attract fans, and a bet on the game's longevity and the publisher's continued commitment. The second bet is usually the larger one, and it is the one the investor can do least to influence. The tenant's bargaining position If publishers hold such power, why do teams, sponsors and players deal with them at all? The answer is that the relationship is not entirely one-sided. Publishers need competitors. A league without recognisable teams is less watchable, and some teams bring audiences that exceed anything the league could build alone. When the League of Legends Championship Series players' association threatened to walk out in 2023, Riot had to delay the start of its summer season and ultimately negotiated an agreement, which Chapter 6 examines in detail. When the owners of Overwatch League teams collectively pushed for the league's wind-down, Blizzard had to pay attention. Popular creators and personalities, a subject for Chapter 5, can command viewing figures that rival the official broadcast. Third-party organisers such as the Esports World Cup can offer publishers large licence fees and global exposure, giving them reason to cooperate. Moreover, the publisher's power to change the rules cuts both ways. A publisher that behaves erratically will find it harder to attract sponsors, teams and investors to its next league. Blizzard's reputation in e-sports never fully recovered from the combination of the Heroes of the Storm shutdown, the Blitzchung affair and the Overwatch League's collapse. Riot's decision to open the Valorant Champions Tour to open qualifiers from 2027 was accompanied by a promise of guaranteed payments to partner teams precisely because the company understood that abrupt changes without compensation would damage trust. Still, the structural fact remains. In every negotiation, the publisher holds the one asset that cannot be replicated: the game. Teams can be replaced, as leagues have shown by selling slots from departing owners to new ones. Tournament organisers can be replaced; publishers have moved their licences from one operator to another many times. Broadcasters can be replaced. Sponsors can be replaced. The game cannot be. Every other participant in the ecosystem is, in economic terms, providing a service to the owner of that asset, and the price of their service depends on how badly the owner needs it. Reading the rest of the industry through this lens Once the landlord problem is understood, a great deal of the industry's history makes sense. The franchise fees of 2017 to 2019, examined in the next chapter, were payments for a licence to participate in a publisher's league, and their value depended on the publisher's continued commitment to that league. Sponsorship value depends on audience, but audience depends on the game's popularity and on the publisher's broadcast choices. Player contracts are constrained by publisher rules, and the best players' market value can collapse if their game declines. Investor valuations of teams must account for the chance that the publisher will reorganise or abandon the scene. And the most promising new revenue sources for teams, digital items sold inside the game, exist only because the publisher chooses to share revenue it could keep for itself. The practical question for anyone doing business in e-sports is therefore not whether the publisher has power. It does. The question is what makes the publisher want to share value with you. The rest of this booklet examines the different answers the industry has found, from franchise fees and revenue pools to sovereign sponsorship and creator audiences, and assesses which of them have held up. A useful discipline for readers is to ask of every e-sports arrangement a single question: what happens to it if the publisher changes its mind? An asset that survives a change of mind by the publisher has real value. An asset that exists only because of the publisher's current policy is a licence, and should be priced like one. Chapter 2: The Franchise Bet Between 2017 and 2020, four of the largest publishers in the world tried to turn their competitive scenes into something that looked like North American professional sport. Riot Games franchised its North American League of Legends Championship Series for 2018 and its European league the following year. Activision Blizzard launched the Overwatch League in 2018, organised around teams named after cities, and followed it with a franchised Call of Duty League in 2020. The idea was simple and, at the time, persuasive. Investors would pay a large entry fee for a permanent place in a closed league, free from the risk of relegation. In return they would receive a share of league revenues, the ability to build long-term brands and, eventually, an asset whose value would rise as e-sports grew, just as the value of a National Basketball Association team had risen over the previous three decades. This chapter examines that bet: what buyers paid, what they thought they were buying, what they actually bought and why, in most cases, it did not pay off. It is the central cautionary tale of e-sports business, and understanding it is the precondition for understanding what came after. Why franchising looked attractive Before franchising, most e-sports leagues ran on the European model of promotion and relegation. A team that finished at the bottom of the top league could be dropped to a lower tier, where sponsorship and viewership were a fraction of what they had been. For a team owner, this was an existential risk. One bad season could wipe out the value of years of investment. Sponsors were reluctant to sign long deals with teams that might not be in the top league the following year. Players could not be offered long-term security. And outside investors, particularly the owners of traditional sports teams who had begun to take an interest, found the model alien. American sports investors were used to closed leagues in which a franchise was a permanent asset. Publishers had their own reasons to change. Riot's North American league had been dogged by financial instability among teams, with owners complaining that they could not make money and that the risk of relegation made long-term planning impossible. A closed league, in which each team had paid a substantial sum to join, promised financial commitment and professional management. It also offered the publisher a large, immediate cash injection. And it allowed publishers to sell a new story to sponsors and media partners: e-sports was growing up and becoming a real sport, with stable teams, permanent rivalries and, in Blizzard's case, local fan bases in real cities. For investors, the case was built on analogy. The value of franchises in American sports had risen spectacularly. The cheapest National Basketball Association franchises had changed hands for tens of millions of dollars in the 1980s and were worth billions by the late 2010s. E-sports audiences were young, digitally native and hard to reach through television. If e-sports became even a fraction as large as traditional sport, the early buyers of permanent franchises would own the equivalent of early basketball teams. The price of entry, however high it looked, would seem cheap in retrospect. What was paid Table 1 sets out the reported entry prices and later fate of the four leading franchised leagues. The figures are as reported at the time by the leagues or by the specialist press; publishers did not publish the full terms, and fees were often payable in instalments. Table 1. The four major franchised leagues: first franchised season, reported entry costs, teams at launch and status in 2026. League Year Entry fee (reported) Teams Status in 2026 LCS 2018 $10m; $13m for new teams 10 Restored after LTA; 8 teams Overwatch League 2018 $20m; later $30m to $60m 12, then 20 Folded in 2024 LEC 2019 About 8m to 10.5m euros 10 10 partner teams Call of Duty League 2020 $25m 12 12 teams Sources: Riot Games and Activision Blizzard announcements; reporting by ESPN and others; league records. The buyers were a revealing mix. The Overwatch League's first owners included Robert Kraft, owner of the New England Patriots, whose group took the Boston franchise; Stan Kroenke's company, owner of the Los Angeles Rams and Arsenal, which took a Los Angeles franchise; and Sterling.VC, linked to the family that owned the New York Mets, which took New York. The LCS attracted the Golden State Warriors, the Houston Rockets and Dan Gilbert, owner of the Cleveland Cavaliers, who backed 100 Thieves. Alongside them came established e-sports organisations such as Team Liquid, Cloud9 and TSM, which raised venture funding in part to pay the fees. The buyers brought credibility, business discipline and, above all, the conviction that they knew how sports franchises gained value. What was bought The central flaw in the franchise bet is visible once we ask what exactly the buyers acquired. In American sport, a franchise is a share in a joint venture. The teams together own the league, negotiate its media rights, set its rules through collective governance and share its revenues. The commissioner works for the owners. No outside company can abolish the NFL, and no one owns American football. An e-sports franchise was something different. It was a licence, granted by the publisher, to participate in a league the publisher owned, run on the publisher's game, under rules the publisher wrote and could change. Teams typically received a share of league revenues, from media rights and league-level sponsorship, and they had some say through advisory councils. But ultimate control stayed with the publisher. The publisher could change the format, alter the schedule, merge the league with another, open it to new entrants, change the revenue-sharing formula or, in the extreme case, close it down. The franchise was only as permanent as the publisher's commitment. This mattered in two ways. First, it limited the upside. In American sport, franchise values rise partly because the teams collectively capture the growth of the league's media rights. In e-sports, the publisher captured much of that growth, and the most valuable revenue streams tied to the game, including sales of in-game items, stayed with the publisher unless it chose to share them. Second, it exposed the buyers to a risk that American sports owners never face: the risk that the owner of the underlying product would lose interest, change strategy or decide that the league was a cost to be cut. The difference can be put as a simple test. A traditional sports franchise is worth something even if the league office disappears, because the teams could form a new league around the same sport. An e-sports franchise is worth nothing if the publisher withdraws its licence, because the teams have no right to use the game. The arithmetic Set aside the question of permanence for a moment, and consider whether the franchise could have paid for itself even on favourable assumptions. The following worked example is illustrative, not a reconstruction of any real team's accounts. Suppose a team paid $20 million for an Overwatch League slot. Suppose its investors expected a return of 10 percent a year, modest by venture standards. The franchise would need to generate $2 million a year of profit, after all costs, just to justify its purchase price, before any return on the additional capital needed to run the team. Now consider the costs. A roster of eight to twelve players at the league's minimum salary of $50,000, plus health insurance, retirement contributions and housing that the league required, would cost well over half a million dollars a year before any star player was paid above the minimum. Coaches, analysts, managers, content staff, travel, facilities and marketing would add several million more. A realistic total for a competitive team in a top league was several million dollars a year. Against this, a team had four sources of revenue. It received a share of league revenues from media rights and central sponsorship. It could sell its own sponsorships. It could sell merchandise. And, in the Overwatch League's model, it could host home matches in its city and sell tickets. The league's media deals were large by e-sports standards: Twitch reportedly paid $90 million for two years of exclusive streaming rights from 2018, and YouTube reportedly paid about $160 million for three years of rights to Activision Blizzard's leagues from 2020. But divided among twenty teams, even after the league's share, such sums did not go far. Team sponsorship depended on audience, and audiences were concentrated in a few popular teams. Merchandise was a minor contributor for most organisations. Home matches, the distinctive feature of the city-based model, were barely tested before the COVID-19 pandemic ended live events in 2020. The result was that most teams operated at a loss. They were not recovering their entry fees through operating profit; they were betting on capital appreciation, the rise in franchise values that would let them sell to a new owner at a higher price. That is a legitimate strategy, and it worked for early owners of American sports teams. But it only works if new buyers keep arriving, willing to pay more. When the growth story faltered, the only exit was a sale at a lower price, or no sale at all. The Overwatch League's collapse The Overwatch League was the most complete test of the franchise model, and its failure was the most complete. Blizzard sold the city-based idea hard. Each team would represent a city, play home matches there and build a local fan base, just like a baseball team. The first season in 2018 had twelve teams, and eight more joined in 2019 at reportedly higher prices, with slots in Asia and Europe as well as North America. Several problems emerged quickly. The city-based model required expensive local infrastructure, venues and staff, but e-sports audiences are global and online rather than local. A fan in Texas was just as likely to support the London team as the Houston one. Home matches, when they happened, drew real crowds but at a cost that the revenues could not justify. The pandemic then made live events impossible for most of 2020 and 2021, destroying the central premise of the model before it had been properly tested. The game itself became a problem. Overwatch's player base declined after its early peak, and the long wait for Overwatch 2, released in October 2022, left the competitive scene playing an ageing game. When Overwatch 2 arrived, it changed the team size from six players to five, forcing rosters to cut a player. Viewership, never as strong as Blizzard had projected, fell further. Activision Blizzard was meanwhile engulfed by lawsuits over its workplace culture from 2021 and by Microsoft's acquisition, completed in October 2023. The owners, locked into a league that was losing money, pressed Blizzard for relief. In early 2023 Blizzard agreed to waive outstanding franchise fee payments and put in place revised operating terms. Those terms included a mechanism under which the teams could vote to leave the league, in which case each would receive a termination payment of $6 million. In November 2023 the required two-thirds majority of owners voted to leave. The total payout was reported at around $114 million. In January 2024 Blizzard formally ended the league and announced its replacement, the Overwatch Champions Series, run in partnership with ESL FACEIT Group and the Asian operator WDG Esports. The new circuit was open, regional and far smaller. The arithmetic of the Overwatch League from an owner's point of view is stark. An original owner paid $20 million, spent millions more each year operating a team and received $6 million on exit. The expansion owners, who paid more, lost more. The city-based sport that was supposed to create the next generation of billion-dollar franchises lasted six seasons. The partial survivors Not every franchised league collapsed. The Call of Duty League, launched in 2020 on similar city-based principles, has survived, though in altered form. Its home-and-away model gave way to centralised events, and many of the original slots have changed hands. By the 2026 season its twelve teams included the Saudi-owned Riyadh Falcons, a French entrant playing as Paris Gentle Mates and teams from established brands such as OpTic, FaZe, Cloud9 and G2, several of which operate slots with city names attached rather than genuinely local identities. The Call of Duty League has benefited from the game's enormous player base and from Activision's annual release cycle, which gives each season a fresh game to market. But its survival owes more to the importance of Call of Duty to Activision than to the success of the franchise model itself. Riot's leagues have survived by changing their terms. In 2024 Riot announced that it would move League of Legends e-sports from a model in which teams relied mainly on sponsorship and league revenue shares to one in which they received a fixed stipend and a share of sales of in-game digital content, as Chapter 3 describes. That change acknowledged that the original franchise economics had not worked. In North America, several original owners left. The Houston Rockets sold their slot within two years; the Golden State Warriors' Golden Guardians left after the 2023 season; TSM, once the region's most popular team, sold its place in 2023, and 100 Thieves announced its withdrawal after the 2025 season. Riot merged the LCS with Brazil's league into a single League of the Americas for 2025, then reversed the decision and restored both leagues for 2026, telling fans that it had learned they felt the deepest attachment to their original leagues. The restored LCS has eight teams, not ten, including one guest slot for a team promoted from the lower tier, a partial return of the promotion and relegation that franchising was supposed to abolish. The European league has been the most stable, with ten partner teams throughout. But even there, slots have turned over through sales and mergers. Karmine Corp, a French organisation built around a large and passionate online community, bought Astralis's place in 2023. Natus Vincere acquired Rogue's place in June 2025. The organisations that have thrived are those with strong fan bases, not those that relied on the franchise itself as a store of value. The case for the defence It would be unfair to leave the franchise era without stating the best argument in its favour, because it contains some truth. Franchising professionalised e-sports. Before 2018, many teams in top leagues were run informally, players were sometimes paid late or not at all, and contracts were often rudimentary. The franchised leagues imposed minimum salaries, standard contracts, benefits and operating requirements, and they attracted executives from traditional sport and media who brought accounting, marketing and legal discipline. The minimum salaries of $75,000 in the LCS and $50,000 in the Overwatch League, with health insurance and retirement savings, were a significant improvement for players, and they set expectations that have outlasted the leagues that introduced them. Franchising also brought in capital at a time when the industry needed it. The entry fees financed publisher investment in production, broadcasting and events, which raised the quality of e-sports as entertainment. And the credibility of owners such as Robert Kraft and the Golden State Warriors helped persuade mainstream sponsors that e-sports was a serious marketing channel. Some of the sponsor relationships formed in those years, such as Mastercard's with League of Legends, have lasted. Finally, the defenders argue, the failure was one of timing and execution rather than principle. The pandemic destroyed the Overwatch League's live-event model before it could be tested. Rising interest rates in 2022 hit every growth industry. Blizzard's own troubles, from its delayed sequel to its legal crises, would have harmed any league it ran. With better luck, the argument goes, franchising might have matured into something like the stable structures of traditional sport. There is force in these points, and the professional standards of the franchise era deserve credit. But the defence does not answer the central problem. Even the best-run franchised league, the European LEC, has seen its slots change hands at prices below the original fees, and Riot felt obliged to restructure its finances in 2024. The Call of Duty League survives because Activision values it, not because its owners have seen their investment appreciate. Bad luck explains why the Overwatch League failed so quickly; it does not explain why no franchised league has delivered the appreciation its buyers paid for. That failure is structural, and it follows from the fact that the franchise was a licence rather than a share in a jointly owned sport. What the franchise bet teaches The failure of the franchise bet was not simply a matter of bad luck, although the pandemic and the rise in interest rates from 2022 certainly did not help. It was a failure to price the landlord problem. Buyers paid for permanence and appreciation, the two things that make American sports franchises valuable, in a structure that could offer neither. The franchise was a licence to participate in a publisher's marketing programme, and it was worth what the publisher's continued commitment to that programme was worth. The lesson has been widely absorbed. Apart from Korea's LCK, which adopted franchising in 2021 on the back of a uniquely strong domestic market, no major publisher has launched a new franchised league with large entry fees since the Call of Duty League in 2020. The models that have replaced franchising, examined in the next chapter, are designed around the opposite principle: the publisher does not charge teams for entry, and instead pays them, directly or through a share of its own revenues, to take part. That reversal marks the end of the idea that an e-sports slot is an appreciating asset. It is better understood as a service contract, under which teams provide competitive content and audiences in return for a share of the value the publisher creates. Hashtags: #TheBusinessOfESports #EsportsBusiness #PublisherOwnedEcosystems #PublisherPower #GameLicensing #EsportsSponsorships #PlayerContracts #LeagueFranchising #RevenueSharing #DigitalGoodsRevenue #VirtualItems #MediaRights #CoStreaming #TournamentLicensing #EsportsTeams #FranchiseFees #OverwatchLeague #LeagueOfLegendsEsports #ValorantChampionsTour #CounterStrikeEsports #EsportsInvestment #SovereignCapital #PlayerBargainingPower #EsportsCommercialModels #FutureOfEsportsBusiness

Download the Book (PDF): Introduction In the summer of 1956 a small group of mathematicians, engineers and psychologists gathered at Dartmouth College in New Hampshire to work on a problem they believed could be substantially advanced in a few weeks. Their proposal, written the previous autumn, conjectured that every aspect of learning or any other feature of intelligence could in principle be described so precisely that a machine could be made to simulate it. They asked for two months, ten people and a modest grant from the Rockefeller Foundation. They named the field they were founding "artificial intelligence." Seventy years later, systems descended from one strand of that ambition write fluent prose, translate between hundreds of languages, predict the folded shapes of proteins and pass professional examinations. The field has received two Nobel Prizes in a single week. It also absorbs a share of global capital spending that would have astonished the Dartmouth group, who were grateful for a few thousand dollars. Between the founding summer and the present lie two long periods of disillusion, known in the trade as the AI winters, a decades-long quarrel between two incompatible theories of what intelligence is, and a hardware revolution that came, almost by accident, from the market for video games. The usual way to tell this story is as a contest of ideas. In one corner stands symbolic AI, which holds that thinking is the manipulation of symbols according to rules, and that intelligence can be built by writing those rules down. In the other stands connectionism, which holds that intelligence emerges from large networks of simple units whose connections are adjusted by experience. The symbolists dominated for thirty years; the connectionists were marginalised, revived, marginalised again, and finally triumphed in the 2010s with deep learning. Told this way, the history is a morality tale about a good idea suppressed and eventually vindicated. That version is not wrong, but it misses what actually decided each turn of the story. This book argues something more specific: the history of AI has been governed less by which ideas were correct than by whether the computation, data and money available at the time could carry them. The field's booms came when an approach found a match between its demands and what the world could supply. Its winters came when promises outran that supply, and when the people paying for the work discovered the gap. And the approach that finally prevailed, the training of very large neural networks on very large datasets, prevailed because it was the one method whose performance kept improving as computation became cheaper, at a time when computation was becoming cheaper faster than anything else in the economy. Why the substrate matters It is easy to read this argument as a dismissal of ideas, as if the scientists were passengers on a train driven by semiconductor fabrication. That is not the claim. Many of the ideas that power modern AI are old, some very old. The artificial neuron dates to 1943; the method used to train today's networks, backpropagation, was worked out in various forms between 1970 and 1986; the convolutional network that would win the decisive image-recognition contest of 2012 was reading handwritten digits at Bell Labs in 1989. What was missing for decades was not cleverness but the conditions under which those ideas could show what they were worth. A method that needs a billion examples and a trillion arithmetic operations cannot be evaluated by people who have neither. The converse also holds. The symbolic programmes of the 1960s and the expert systems of the 1980s were not foolish. They succeeded at the problems that fitted the machines and the knowledge of their time, sometimes spectacularly. They failed when they were asked to scale to the open-ended messiness of the real world, where the number of rules required, or the number of possibilities to be searched, grew faster than any machine or any team of knowledge engineers could follow. Their ceiling was set by an economics they could not escape: every additional unit of competence had to be paid for by human labour, written by hand. Seeing the history through this lens explains several things that the contest-of-ideas version leaves obscure. It explains why the same idea could fail in 1969 and succeed in 2012 with only modest technical changes. It explains why the winters hit funders and companies before they hit researchers, and why they were sharpest where expectations had been set highest. It explains why the decisive breakthroughs of the deep learning era came as often from the people who built datasets and wrote software for graphics cards as from those who invented new theories. And it gives a more useful way of thinking about the present, when the field is again making large promises underwritten by enormous expenditure on computation. What this book covers and what it leaves out The narrative runs from the founding ideas of the 1940s and 1950s to the early 2020s, with a closing look at how the present boom fits the pattern. It follows the main line of development in the United States and Britain, with attention to Japan and Europe where national programmes shaped events. It does not attempt a full history of robotics, of AI in fiction, or of the many applied fields, from operations research to computer vision in manufacturing, that share techniques with AI without sharing its ambitions. Nor is it a technical textbook. Where a mechanism matters to the argument, such as why a single-layer perceptron cannot learn certain functions, or why graphics processors happen to suit neural networks, it is explained in plain terms, because the mechanism is often where the economics hides. The chapters run in rough chronological order. The first describes the founding bet: what the pioneers believed, what they built in the first few years, and why they expected rapid success. The second follows the symbolic golden age of the 1960s, when programs that proved theorems and parsed sentences in toy worlds seemed to be the first steps on a short road. The third tells the parallel story of the perceptron, the first serious attempt to build learning machines from artificial neurons, and of how it was eclipsed. The fourth examines the first winter of the 1970s and the specific reports, budget decisions and technical walls that brought it on. The fifth covers the expert-systems boom of the 1980s, the national programmes it inspired, and the second winter that followed its collapse. The sixth describes the quiet decades in which neural networks were revived, stalled again, and survived largely through a handful of stubborn researchers, while statistical methods took over much of the field. The seventh explains how graphics hardware, a large labelled dataset and a single competition result in 2012 turned that minority position into the field's centre. The eighth follows the scaling era, from deep learning's early victories through the transformer architecture to the large language models of the 2020s. The conclusion asks what the whole pattern implies. A note on dates, numbers and names The history of AI is unusually well supplied with legends: the conference that "founded" the field in a single summer, the book that "killed" neural networks, the competition that "started" the deep learning revolution. Each contains truth and each simplifies. Where this book uses a specific figure, it comes from the original paper, the official record of a programme, or a careful secondary source listed in the notes. Where the record is uncertain, the text says so. Credit in this field is also contested, sometimes bitterly; the convention here is to name the people most associated with a result while acknowledging, where it matters, those who reached similar ideas earlier or independently. The reader needs no mathematics beyond arithmetic and no prior knowledge of computing. What the book asks is a willingness to think of intelligence, for the length of a few chapters, as something that has a price, and to follow how that price was paid, deferred, misjudged and finally, in part, met. Chapter 1: The Founding Bet Artificial intelligence did not begin at Dartmouth. By the time John McCarthy wrote the funding proposal for the 1956 summer project, most of its intellectual ingredients had been on the table for at least a decade. What Dartmouth did was give those ingredients a name, a community and a confident timetable. To understand why the founders were so sure of quick progress, and why that confidence would later cost the field dearly, it helps to see what they were building on and what they assumed it would cost. Machines that compute, neurons that switch The first ingredient was the idea that thinking might be a kind of calculation. Alan Turing's 1936 paper on computable numbers described an abstract machine that could carry out any procedure that could be specified step by step. In 1950, in a paper in the philosophy journal Mind, Turing turned directly to the question of whether machines could think. He replaced that unanswerable question with a practical test, an "imitation game" in which a judge converses by text with a hidden human and a hidden machine and tries to tell them apart. He also predicted that by the end of the century machines with a storage capacity of about a billion binary digits, roughly 125 megabytes as we would now put it, would play the game well enough to fool an average interrogator a good share of the time. The prediction about storage was roughly right; the prediction about conversation took a further two decades and a different kind of machine. The second ingredient came from neuroscience and logic. In 1943 Warren McCulloch, a neurophysiologist, and Walter Pitts, a self-taught logician, published a paper showing that idealised neurons, each of which either fires or does not depending on whether its inputs pass a threshold, could be wired together to compute any logical function. Their model was crude as biology but powerful as an idea: it suggested that the brain could be understood as a logic machine, and that a logic machine could, in principle, be built to do what brains do. Six years later the psychologist Donald Hebb proposed that learning occurs when connections between neurons that fire together are strengthened. Between them, McCulloch, Pitts and Hebb supplied the basic vocabulary of what would later be called connectionism: simple units, weighted connections, and learning as the adjustment of weights. The third ingredient was the electronic computer itself. The machines that appeared in the late 1940s and early 1950s, such as the Manchester Mark 1, the IAS machine at Princeton and IBM's 701, were built for numerical work in physics, ballistics and cryptography. But their designers understood that a stored-program computer manipulated symbols, not just numbers, and that the symbols could stand for anything. Claude Shannon, who would co-sign the Dartmouth proposal, published an analysis of how a computer might be programmed to play chess in 1950. At IBM, Arthur Samuel began work on a checkers program that would, over the following years, learn to improve its play by adjusting the weights it gave to features of board positions and by remembering positions it had seen. Samuel's work is often cited as the origin of the phrase "machine learning," which he used in a 1959 paper. The fourth ingredient was cybernetics, the interdisciplinary movement led by Norbert Wiener that studied control and communication in animals and machines. Cybernetics treated feedback, adaptation and purpose as engineering concepts, and it brought together mathematicians, physiologists and engineers in a series of meetings through the late 1940s and early 1950s. Many of the people who would later work on AI passed through its orbit. But cybernetics was broad, analogue in its instincts and philosophically sprawling, and part of what McCarthy wanted from a new name was distance from it. The British road and the machines that wandered Not every early attempt at mechanical intelligence ran through symbolic logic or through American institutions. In Britain, a loose community of physiologists, engineers and mathematicians met between 1949 and 1958 as the Ratio Club, a dining club whose members included Turing, the neurologist W. Grey Walter, the psychiatrist and cybernetician W. Ross Ashby, and the physiologist Horace Barlow. Their interests ran toward brains, feedback and adaptation rather than toward logic and proof. Grey Walter, working at the Burden Neurological Institute in Bristol, built a pair of small battery-powered robots in the late 1940s, which he named Elmer and Elsie and described as tortoises. Each had a light sensor, a touch sensor, two motors and a pair of vacuum tubes acting as something like two neurons. With nothing more than this, the tortoises wandered, sought out moderate light, avoided bright light and obstacles, and returned to a hutch to recharge when their batteries ran low. Walter's point was that apparently purposeful behaviour could emerge from a very small number of interacting elements coupled to an environment, without any internal representation of goals or plans. Ashby built a device he called the Homeostat, a set of interconnected electromechanical units that adjusted themselves until the system as a whole returned to a stable state after being disturbed, and he argued that adaptation of this kind was the essence of intelligent behaviour. Turing himself, in the 1950 paper, proposed that rather than trying to program an adult mind directly, one might build a machine resembling a child's mind and educate it. He imagined a process of teaching through rewards and punishments, and he anticipated that the machine might reach conclusions its teacher could not foresee. This proposal for machine learning sat at the end of a paper better remembered for its test of conversation, and it was not followed up in any serious way for decades. It is striking, in hindsight, how much of the eventual answer was sketched by the man who posed the question. These British and cybernetic lines of work shared an intuition that would return in the 1980s and again in the 2010s: that intelligence might be better grown than written. What they lacked was any way to grow it at a useful scale. A tortoise with two vacuum tubes could display an elegant behaviour; it could not be enlarged into something that recognised faces or understood sentences, because nobody knew how to train a system with thousands of adaptive parts. The symbolic programme, by contrast, offered a method that could be extended immediately, by writing more program. For a field that needed results to justify its funding, that difference was decisive. The proposal and the summer The Dartmouth proposal, dated August 1955 and signed by McCarthy, Marvin Minsky, Nathaniel Rochester of IBM and Shannon of Bell Labs, is a short document with an enormous claim at its centre. It proposed a two-month study by ten people, built on the conjecture that every aspect of learning or any other feature of intelligence could in principle be so precisely described that a machine could be made to simulate it. It listed topics for study: automatic computers, how a computer could be programmed to use a language, neuron nets, the theory of the size of a calculation, self-improvement, abstraction, and randomness and creativity. And it expressed the belief that a significant advance could be made on one or more of these problems if a carefully selected group worked on it together for a summer. The Rockefeller Foundation funded the project at a lower level than requested, and the summer of 1956 did not go as planned. Participants came and went over roughly two months; only a handful stayed throughout. There was no shared programme of research and no collective result. Ray Solomonoff, who attended for the whole period, kept notes that later historians have used to reconstruct who was there and when. The attendees included, at various points, Oliver Selfridge, who was working on pattern recognition; Arthur Samuel; Trenchard More; and two researchers from the RAND Corporation and Carnegie Institute of Technology, Allen Newell and Herbert Simon. Newell and Simon arrived with something the others lacked: a working program. Their Logic Theorist, written with the programmer Cliff Shaw, could prove theorems from the early chapters of Whitehead and Russell's Principia Mathematica, the monumental attempt to ground mathematics in logic. It did so not by trying every possible sequence of deductions, which would have been hopeless, but by using heuristics, rules of thumb that guided the search toward promising lines of proof, as a human mathematician might. It eventually proved thirty-eight of the first fifty-two theorems in the relevant chapter, and for one of them it found a proof more elegant than the original. Simon later recalled telling a class early in 1956 that over the Christmas holiday he and Newell had invented a thinking machine. The Logic Theorist mattered less for what it proved than for what it demonstrated about method. It showed that a computer could do something that, in a person, would be called reasoning, and that it could do so by searching through a space of possible symbolic manipulations, guided by heuristics. That template, intelligence as heuristic search through a space of symbol structures, became the dominant paradigm of AI for the next three decades. Why the founders expected speed Reading the predictions of the late 1950s now, the striking thing is not that they were wrong but how short the timescales were. In 1958 Simon and Newell wrote that within ten years a digital computer would be the world's chess champion, would discover and prove an important new mathematical theorem, and would write music that critics would regard as having considerable aesthetic value. They added that most theories in psychology would take the form of computer programs. In 1965 Simon predicted that within twenty years machines would be capable of doing any work a man could do. In 1967 Minsky wrote that within a generation the problem of creating artificial intelligence would be substantially solved. These were not careless men. Their optimism followed from a specific theory of what intelligence was and therefore of what it would cost to build. If intelligence consists of manipulating symbols according to rules, and if a few general-purpose methods of search and inference underlie the rules, then building an intelligent machine is a matter of finding those general methods and implementing them. The early programs seemed to confirm this. The Logic Theorist proved theorems; Samuel's program learned checkers; Newell, Shaw and Simon's next system, the General Problem Solver, applied a single method, called means-ends analysis, to puzzles as different as the Tower of Hanoi and logic exercises. Each success suggested that the general method had been found and only needed refinement and faster machines. What the founders underestimated was the cost of everything around the general method. The problems they chose to work on were problems that are hard for people but structurally simple: a game of chess or a logic proof has a small, precisely defined set of objects and legal moves. The things that people find easy, such as recognising a face, understanding a casual sentence, or picking up an unfamiliar object, turned out to depend on vast amounts of unstated knowledge and on perceptual processing of enormous complexity. This inversion, later called Moravec's paradox after the roboticist Hans Moravec, was not visible in 1956 because nobody had yet tried seriously to build the easy things. The early successes were drawn from exactly the class of problems where the symbolic approach was cheapest. There was also an underestimate of a more literal kind of cost. The computers available in the 1950s were extraordinarily slow and small by any later standard. The IBM 704, on which much early AI work ran, could perform on the order of tens of thousands of floating-point operations per second and had a main memory measured in kilobytes. Search-based methods are sensitive to exactly these limits, because the number of possibilities to be searched typically grows exponentially with the depth of the search. A program that handles a five-step proof comfortably may need billions of times more work for a fifteen-step one. The founders knew this in principle; it was why they emphasised heuristics. But they assumed that good heuristics would keep the growth in check, and that the remaining shortfall would be closed by faster hardware. Neither assumption held as well as they hoped. Institutions and money Ideas in AI have always needed patrons, and the patrons of the first generation shaped the field as much as any theory. After Dartmouth, McCarthy moved to the Massachusetts Institute of Technology, where he and Minsky founded what became the MIT AI Laboratory; McCarthy later left for Stanford, where he founded the Stanford AI Laboratory in the early 1960s. Newell and Simon built a group at Carnegie Tech, later Carnegie Mellon University. These three centres, together with the Stanford Research Institute, dominated American AI for decades. The money came largely from the military. After the Soviet launch of Sputnik in 1957, the United States created the Advanced Research Projects Agency, ARPA, later renamed DARPA. In 1962 ARPA established its Information Processing Techniques Office under the psychologist J. C. R. Licklider, who believed in funding talented people and leaving them free to pursue their own interests. Through the 1960s ARPA money flowed to MIT, Carnegie and Stanford in large, flexible grants with few deliverables attached. The MIT AI group was funded as part of Project MAC, an ARPA-backed effort in time-sharing and computing, from 1963. This funding model had two consequences that matter for the rest of the story. First, it allowed the field to pursue long-range, speculative goals without having to show near-term applications, which was essential for basic research. Second, it tied the field's fortunes to the willingness of a single government patron to keep believing in those goals. When that willingness changed, as it did at the end of the 1960s, the effect would be abrupt and severe. The bet, stated plainly By 1960 the shape of the founding bet was clear. The field had wagered that intelligence was a matter of the right general methods for manipulating symbols; that those methods could be discovered by building programs that solved hard intellectual puzzles; that the knowledge needed to apply them to the real world could be supplied by hand; and that the computational cost of the whole enterprise would be met by heuristics and by the steady growth of hardware. It was a bet on cleverness over scale. Alongside it ran a quieter bet, made by a smaller group, that intelligence would emerge from learning in networks of simple units, and that the right approach was to build machines that could be trained rather than programmed. That bet was older in its roots and in some ways more ambitious, because it proposed to sidestep the need to write knowledge down at all. It was also far more expensive, in ways that nobody could yet measure. The next two chapters follow the two bets through their first decade, beginning with the one that seemed, for most of the 1960s, to be winning. Chapter 2: The Symbolic Golden Age For roughly fifteen years after Dartmouth, artificial intelligence looked like a field on a steep upward curve. Programs appeared that did algebra, answered questions in English, planned the movements of robots and held conversations that some users found disconcertingly human. Each was limited, and the researchers who built them knew it, but each seemed to push the frontier a little further in a direction that pointed toward general intelligence. This chapter describes what those programs actually did, why they worked, and the pattern in their limitations that only became obvious in retrospect. Search, heuristics and the problem of explosion The core technique of the period was heuristic search. A problem is represented as a space of states, with a starting state, a goal, and operators that move from one state to another. Solving the problem means finding a sequence of operators that leads from start to goal. For a puzzle like the Tower of Hanoi, the states are arrangements of discs and the operators are legal moves. For theorem proving, the states are sets of established statements and the operators are rules of inference. For a game like chess, the states are board positions and the operators are moves, with the added complication of an opponent. The difficulty is that the number of possible states usually grows exponentially. If each state offers ten possible moves, a search ten moves deep must consider up to ten billion sequences. In chess, with around thirty-five legal moves per position on average, the full tree of possibilities is beyond astronomical. The response of the symbolic programme was to prune the tree using heuristics: estimates of how promising a state is, rules for which moves to try first, and techniques for ignoring branches that cannot affect the result. Much of the most lasting work of the period, including the alpha-beta pruning method for game search and the A* algorithm developed at the Stanford Research Institute in 1968 for finding shortest paths, consisted of making search more economical. These methods were genuine advances and are still taught and used. But they shared a structural weakness. A heuristic reduces the effective branching of the search; it does not change the fact that the search is exponential. A program that works well on problems of a certain size will eventually encounter problems just a little larger on which it bogs down entirely. This phenomenon, which came to be called the combinatorial explosion, would become the central technical charge against AI in the early 1970s. Microworlds The answer of the MIT group under Minsky was to work in simplified domains where the explosion could be kept in check, and where the real problems of representation and understanding could be studied without the noise of the real world. These were called microworlds. The idea was that by solving intelligence completely in a small, well-defined world, one could learn the principles needed to scale it up. Several of the period's best-known programs were microworld systems. Daniel Bobrow's STUDENT, written for his 1964 MIT doctorate, solved algebra word problems of the kind found in school textbooks, translating English sentences such as statements about ages or distances into equations and solving them. Thomas Evans's ANALOGY program, from the same era, solved geometric analogy problems of the type used in intelligence tests. James Slagle's SAINT program solved calculus integration problems at roughly the level of a first-year university student. The most celebrated microworld program was SHRDLU, built by Terry Winograd at MIT between 1968 and 1970. SHRDLU operated in a simulated world of coloured blocks, pyramids and boxes on a table. A user could type commands and questions in ordinary English: pick up a big red block; find a block which is taller than the one you are holding and put it into the box; what does the box contain? SHRDLU would parse the sentence, resolve pronouns and references by keeping track of the conversation, plan the necessary actions, carry them out in the simulation, and answer questions about what it had done and why. Transcripts of SHRDLU dialogues are still impressive. The program seemed to understand language in a way that no previous system had. What SHRDLU actually demonstrated was that complete understanding was achievable when the world was small enough for every relevant fact to be written down. The program knew everything about its blocks world because Winograd had told it everything, in carefully structured code. Extending it to a kitchen, let alone to the open world of ordinary conversation, would have required writing down an unbounded amount of further knowledge, and the structures that worked for blocks gave no obvious guidance about how to organise it. Winograd himself later became one of the more thoughtful critics of the symbolic programme's assumptions. The conversation that fooled people In 1966 Joseph Weizenbaum at MIT published a description of ELIZA, a program that conducted typed conversations by matching patterns in the user's input and transforming them into responses according to a script. The best-known script, DOCTOR, imitated a non-directive psychotherapist, which allowed the program to reflect statements back as questions ("Why do you say your mother dislikes you?") without needing to know anything about the world. ELIZA had no model of meaning at all. It was, by design, a demonstration of how little was needed to create an impression of understanding. Yet Weizenbaum was disturbed to find that users, including his own secretary, became emotionally engaged with the program and attributed understanding to it. Some psychiatrists suggested that automated therapy might become practical. Weizenbaum spent much of the rest of his career warning about exactly this tendency to over-attribute intelligence to machines, most fully in his 1976 book Computer Power and Human Reason. The phenomenon he identified, now sometimes called the ELIZA effect, has recurred with every generation of conversational software since, and it is part of the reason that public expectations of AI have so often run ahead of the technology. Robots and planning At the Stanford Research Institute, a team led by Charles Rosen built Shakey, a mobile robot developed between 1966 and 1972 that could perceive its surroundings through a television camera and range finder, build a model of a set of rooms containing boxes and doorways, plan a sequence of actions to achieve a goal such as pushing a box from one room to another, and carry out the plan. Shakey was the first robot to combine perception, reasoning and action in this integrated way, and it produced several techniques of lasting importance, including the STRIPS planning system and the A* search algorithm. Shakey also illustrated the cost structure of symbolic AI with unusual clarity. The robot moved slowly, often pausing for long periods while its off-board computer worked out what to do next. Its world was carefully prepared, with uniformly coloured walls and simple, large objects, because its vision system could not cope with clutter or variable lighting. Every change in the environment required updating its internal model, and every action required reasoning about what had and had not changed as a result. This last difficulty became known as the frame problem, identified by McCarthy and Patrick Hayes in 1969: how does a reasoning system know which facts remain true after an action without explicitly checking every one? For a human the answer is obvious and effortless. For a logic-based system it was a deep and persistent technical puzzle. Two theories of symbols: logic and psychology Beneath the variety of golden-age programs lay two different views of what the symbols should be and how they should be manipulated, and the difference between them shaped the field's internal debates for decades. The first, championed by McCarthy, held that the right foundation for AI was formal logic. In a 1958 paper titled "Programs with Common Sense," McCarthy proposed a hypothetical system, the Advice Taker, which would represent its knowledge of the world as sentences in a logical language and would decide what to do by deducing consequences from those sentences. The appeal was clarity: if knowledge was expressed in logic, its meaning would be precise, its consequences could be derived by general proof procedures, and new knowledge could be added simply by adding new sentences, without reprogramming. In 1965 the logician John Alan Robinson published the resolution principle, a single inference rule that made automated proof in first-order logic far more mechanical, and for a few years it seemed that a general reasoning engine might be within reach. In practice, resolution provers were overwhelmed by the combinatorial explosion as soon as their knowledge bases grew beyond a modest size, because they had no way of knowing which of the vast number of possible deductions were relevant to the problem at hand. The second view, championed by Newell and Simon at Carnegie and by Minsky at MIT in a different form, held that AI should be modelled on human problem-solving rather than on formal logic. Newell and Simon studied people thinking aloud as they solved puzzles and tried to build programs that followed the same steps, an approach that made AI and cognitive psychology two sides of a single enterprise. Their General Problem Solver, first running in 1959, embodied a method they had observed in human subjects: compare the current situation with the goal, identify the most important difference, and look for an operator that reduces it, setting up sub-goals where necessary. Minsky, for his part, distrusted the tidiness of logic and favoured representations that were messier and more pragmatic, closer to the heuristics that people actually use. In their 1975 Turing Award lecture, published the following year, Newell and Simon stated the shared foundation of both views as the physical symbol system hypothesis: that a physical system which manipulates symbols according to rules has the necessary and sufficient means for general intelligent action. It was a bold and testable claim. It said, in effect, that there was nothing to intelligence beyond the right symbolic processing, and therefore that symbolic AI would succeed if it found the right symbols and the right processes. The hypothesis was an accurate summary of what the field had bet on. It was silent on how the right symbols for the real world were to be obtained, and that silence was where the cost would turn out to hide. Lisp, time-sharing and the culture of the labs The programs of the golden age were made possible by tools that the AI community built for itself. The most important was Lisp, the programming language McCarthy designed in 1958. Lisp treated programs and data as the same kind of structure, nested lists of symbols, which made it natural to write programs that built and manipulated other symbolic structures, including other programs. It introduced or popularised ideas such as recursion as a primary control structure and automatic memory management through garbage collection. For three decades Lisp was the language of AI research in the United States. AI researchers were also among the chief advocates and early users of time-sharing, the technique of letting many users interact with a single computer at once. Interactive computing suited the exploratory style of AI programming, in which a researcher would write a fragment of code, try it, revise it and try again. The culture that grew up in the AI labs at MIT and Stanford, with its late nights, shared machines and ethic of open access to code, fed directly into the later hacker and open-source movements. This culture had a further effect that is easy to overlook. Because the labs had generous ARPA funding and controlled their own computers, AI researchers in the 1960s were, by the standards of the time, rich in computation. A university group with a dedicated PDP-6 or PDP-10, both made by Digital Equipment Corporation and widely used in AI labs, had access to more interactive computing than most of the world's scientists. That relative wealth made the symbolic approach's computational demands seem manageable. It also meant that the gap between what worked in the lab and what could be deployed elsewhere was rarely tested. What the golden age taught, and what it hid By the late 1960s AI had a set of impressive demonstrations, a powerful toolkit, three or four well-funded centres, and a widespread belief, inside and outside the field, that general machine intelligence was perhaps two or three decades away. Stanley Kubrick's 2001: A Space Odyssey, released in 1968, featured a computer, HAL 9000, that conversed fluently, read lips and played chess; Minsky had advised the production. The film's assumption that such a machine would exist by the turn of the century was not considered outlandish by many AI researchers. What the demonstrations hid was a consistent pattern. Every successful system worked in a domain small and clean enough for its knowledge to be written down by hand and for its search to be kept within bounds by heuristics. The progress from one system to the next was real, but it was progress in the art of choosing and engineering such domains, not progress toward a method that would work without them. The researchers were, in effect, paying for intelligence with human design effort, and they could afford it only in small quantities. There were people who said this at the time. The philosopher Hubert Dreyfus, in a 1965 RAND Corporation paper later expanded into the book What Computers Can't Do (1972), argued that human intelligence depends on a background of embodied, tacit, context-sensitive understanding that cannot be captured in explicit rules. He was received with considerable hostility by the AI community, partly because his arguments were couched in the language of continental philosophy and partly because some of his specific predictions, such as that no computer would ever play good chess, proved wrong. But his central claim, that the hard part of intelligence lay in what could not easily be written down, was close to the mark. It would take two winters and a change of paradigm before the field fully absorbed it. The other path, running alongside Throughout the golden age, a different approach to machine intelligence was also being pursued, with a different theory of cost. Instead of writing knowledge down, its practitioners proposed to let machines acquire it from examples by adjusting the strengths of connections in networks of artificial neurons. For a few years at the end of the 1950s this approach attracted as much public attention as the symbolic programmes, and rather more press. By the end of the 1960s it had been almost entirely displaced. Why that happened, and why its displacement was more about the limits of its time than the limits of its idea, is the subject of the next chapter. Chapter 3: The Perceptron and the Road Not Taken On 8 July 1958 the New York Times reported that the United States Navy had revealed the embryo of an electronic computer that it expected would be able to walk, talk, see, write, reproduce itself and be conscious of its existence. The machine in question was the Perceptron, developed by the psychologist Frank Rosenblatt at the Cornell Aeronautical Laboratory with funding from the Office of Naval Research. The newspaper's claims went far beyond anything Rosenblatt had demonstrated, but they captured a real excitement. Here was a machine that was not programmed to recognise patterns but learned to do so from examples. Within little more than a decade the approach it represented had been pushed to the margins of the field. The reasons are worth understanding in detail, because they concern the cost of learning, and that cost would eventually become the central fact of AI. What the perceptron did A perceptron, in its simplest form, is a single artificial neuron of the McCulloch-Pitts type with adjustable connection strengths. It receives a set of inputs, for example the brightness values of the cells of a grid onto which an image is projected. Each input is multiplied by a weight. The weighted inputs are added together, and if the sum exceeds a threshold, the neuron outputs one; otherwise it outputs zero. The output can be read as a classification: this image is a triangle, or it is not. The crucial contribution was the learning rule. Rosenblatt showed that a perceptron could be trained by presenting it with examples whose correct classifications were known. Whenever the perceptron got an example wrong, each weight was nudged slightly in the direction that would have made the answer more correct. Whenever it got an example right, nothing changed. He proved, in what became known as the perceptron convergence theorem, that if there existed any setting of the weights that classified all the training examples correctly, this procedure would find one in a finite number of steps. Rosenblatt first simulated the perceptron on an IBM 704, then built a dedicated machine, the Mark I Perceptron, which was completed around 1960. It had a twenty-by-twenty grid of photocells as its retina, connected to a layer of association units, which fed into response units. The adjustable weights were implemented by motor-driven potentiometers, so that learning was literally a physical process of turning dials. The machine could learn to distinguish simple shapes and letters presented to its camera. Rosenblatt was an ambitious theorist as well as an engineer. His 1962 book Principles of Neurodynamics described a wide range of perceptron architectures, including networks with multiple layers and with connections that fed back on themselves. He saw the perceptron as a model of the brain, not merely as a pattern classifier, and he speculated freely about what larger networks might eventually achieve. This made him a gifted publicist for the approach and an inviting target for its critics. Adaline and the engineering tradition The perceptron was not the only learning machine of its era. At Stanford, Bernard Widrow and his student Ted Hoff developed the Adaline (adaptive linear neuron) around 1960, together with a learning procedure now called the least mean squares rule, or the Widrow-Hoff rule. Instead of correcting weights only when the output was wrong, the Widrow-Hoff rule adjusted them in proportion to the size of the error before thresholding, which amounts to performing a step of gradient descent on the squared error. This was a small change with large consequences: gradient descent on an error measure is the core of how neural networks are still trained. Adaline and its multi-unit extension, Madaline, found practical uses in signal processing, notably in adaptive filters that cancel echoes on telephone lines, an application that has been in continuous use for decades. This engineering tradition of adaptive signal processing carried on productively throughout the period when neural networks were out of fashion in AI. It is a useful reminder that the ideas did not die; they moved to fields where their costs could be paid. Other learning machines The perceptron was the most famous learning machine of its era, but it was not alone, and the variety of approaches is a reminder that learning was a live research programme rather than one man's enthusiasm. At the Lincoln Laboratory of MIT, Oliver Selfridge, a Dartmouth participant, described in 1959 an architecture he called Pandemonium. It consisted of layers of "demons," simple processing units, each of which looked for a particular feature in the input and "shouted" with a strength proportional to how strongly it found it. Higher-level demons listened to the shouting of those below them and combined their reports, and at the top a decision demon picked whichever category demon shouted loudest. Selfridge proposed that the demons' weightings could be adjusted by experience and that useless demons could be discarded and replaced by mutated versions of useful ones. Pandemonium was a sketch rather than a working system of any scale, but its picture of perception as a hierarchy of feature detectors, each combining the outputs of simpler ones, is recognisably the picture that deep convolutional networks would realise half a century later. In Britain, Donald Michie built a learning machine in 1961 out of matchboxes. MENACE, the Matchbox Educable Noughts And Crosses Engine, consisted of about three hundred matchboxes, one for each distinct position it might face in noughts and crosses, each containing coloured beads representing possible moves. To play, one drew a bead at random from the box for the current position. After a game, beads for the moves that had led to a win were added and those that led to a loss were removed, so that good moves became more likely over time. Over a few hundred games MENACE learned to play well. It was a vivid demonstration of reinforcement learning, learning from the outcome of actions rather than from labelled examples, and it showed the method's cost structure plainly: even for a game as small as noughts and crosses, it needed a separate store of experience for every situation, and many games to fill it. In the Soviet Union, Alexey Ivakhnenko and colleagues developed, from 1965, a method they called the Group Method of Data Handling, which built multi-layer networks of polynomial units by training and selecting them one layer at a time on data. Some historians of deep learning, notably Jürgen Schmidhuber, regard this as the first working method for training deep networks. It was used for modelling and prediction in engineering, but it remained largely unknown in Western AI. Samuel's checkers program, which by the early 1960s played a respectable game, also belongs in this company. It learned in two ways: by memorising positions it had evaluated before, so that it did not have to evaluate them again, and by adjusting the weights it gave to features of a position, such as piece advantage or mobility, according to how well its evaluations predicted later outcomes. Samuel's program was a hybrid of search and learning, and it ran on IBM's commercial machines, which gave it a public visibility that the more academic projects lacked. What all these machines shared was that their learning was shallow: they adjusted a single layer of weights, or a table of values, over features that a human had chosen. None of them could discover new features for itself. That limit, which would turn out to be the crux, was about to be made mathematically precise. The limitation of a single layer The fundamental limitation of a single-layer perceptron is geometric. Because its output depends on whether a weighted sum of inputs exceeds a threshold, it divides the space of possible inputs with a straight line, or in higher dimensions a flat plane. It can learn any classification in which the two classes can be separated in this way, which mathematicians call linear separability. It cannot learn any classification in which they cannot. The simplest example is the exclusive-or function, often written XOR. Take two binary inputs. The XOR of the two is one if exactly one of them is one, and zero otherwise. Plot the four possible input pairs as the corners of a square: the two corners where the output should be one lie on one diagonal, the two where it should be zero lie on the other. No single straight line can separate the diagonals. A single-layer perceptron therefore cannot compute XOR, however long it is trained. XOR itself is trivial, but the limitation it illustrates is general and serious. Many of the properties one would want a visual system to recognise, such as whether a figure is connected or whether the number of objects in a scene is odd or even, are not linearly separable in the raw input. A perceptron can only recognise them if someone has already designed input features that make them separable, which brings back the very human design effort that learning was supposed to replace. The obvious remedy is to add layers. A network with a hidden layer of units between input and output can compute XOR easily: one hidden unit detects "at least one input is on," another detects "both inputs are on," and the output unit fires when the first is active but not the second. In general, multi-layer networks can represent vastly more complex functions. Rosenblatt knew this. The problem was training them. The perceptron learning rule told you how to adjust the weights feeding into the output unit, because you could compare its output with the correct answer. It gave no guidance about how to adjust the weights feeding into hidden units, whose correct outputs were not specified by the training data. This was the credit assignment problem: when the network makes an error, which of its many internal connections is to blame, and by how much? Perceptrons, the book In 1969 Marvin Minsky and Seymour Papert of MIT published Perceptrons: An Introduction to Computational Geometry. It was a rigorous mathematical analysis of what single-layer perceptrons, and certain restricted variants, could and could not compute. Its results were correct and in places elegant. Among other things, it showed that recognising connectedness required a perceptron whose size grew unacceptably with the size of the image, and that certain apparently simple properties required units that looked at the entire input at once, which defeated the purpose of a distributed architecture. The book's influence went beyond its theorems. Minsky and Papert expressed scepticism that multi-layer networks would escape the limitations they had identified, describing the extension to multiple layers as sterile in their judgement, while acknowledging that this was a matter of intuition rather than proof. Coming from two of the most prominent figures in AI, this judgement carried enormous weight, and it was widely read as a general verdict against the neural network approach. The extent to which Perceptrons actually caused the decline of neural network research has been argued over ever since. The standard story says the book killed the field. Later historians, notably Mikel Olazaran in a careful study of the controversy, have presented a more complicated picture in which the book confirmed and accelerated a shift that was already under way, and in which funding agencies and graduate students simply drew the conclusion that the symbolic approach was the safer bet. Minsky and Papert themselves, in an expanded edition of 1988, defended the book against the charge, arguing that the field had stalled because it lacked theory, not because they had attacked it. What is not in dispute is the outcome. Funding for neural network research in the United States dwindled through the 1970s. Rosenblatt died in a boating accident in 1971, on his forty-third birthday, removing the approach's most energetic advocate. Researchers who continued to work on neural networks often did so under other labels, such as adaptive systems, pattern recognition or theoretical neuroscience, and frequently outside computer science departments. A matter of price, not principle With hindsight, the perceptron's eclipse looks less like the refutation of an idea than a correct reading of its price at the time. Consider what a multi-layer network capable of useful visual recognition would have needed in 1969. First, a training method for hidden layers. The key mathematical ingredient, the efficient computation of how the error depends on each weight by working backwards through the network, was published by the Finnish mathematician Seppo Linnainmaa in 1970 as a general technique for computing derivatives, and was applied to neural networks by Paul Werbos in his 1974 Harvard doctoral thesis. So the method was, in principle, only a few years away. Second, a large number of labelled examples. Useful visual recognition requires training on thousands to millions of images. In 1969 there was no practical way to collect, store or digitise such a dataset. Images had to be scanned laboriously and stored on media whose capacity was measured in megabytes at best. Third, enormous computation. Training a network requires repeatedly running every example through the network and adjusting every weight, typically over many passes through the data. Even a modest network with tens of thousands of weights, trained on tens of thousands of examples for dozens of passes, implies billions of arithmetic operations. The machines of the late 1960s could perform somewhere between hundreds of thousands and a few million operations per second, and time on them was expensive and shared. Taken together, these requirements put useful neural networks perhaps two orders of magnitude, and in some respects much more, beyond the resources of the period. The symbolic approach, by contrast, could produce impressive demonstrations with a single researcher's insight and a few hours of machine time, because the heavy lifting was done by the programmer's knowledge rather than by computation over data. When funders and students chose symbolic AI, they were choosing the approach whose costs fitted the available budget. This is the pattern that the rest of the history will repeat. A learning approach that shifts the burden of acquiring knowledge from people to machines is only viable when machines, data and storage are cheap enough to carry it. In 1969 they were not. The connectionist bet was not lost; it was postponed, at compound interest, until the price came down. Meanwhile the symbolic programme, freed of its rival, was about to discover its own limits, and to discover them in public. Hashtags: #TheHistoryOfAI #DartmouthConference #ArtificialIntelligenceHistory #SymbolicAI #Connectionism #ArtificialNeurons #MachineLearning #TuringTest #LogicTheorist #HeuristicSearch #Microworlds #SHRDLU #ELIZA #Shakey #Perceptron #FrankRosenblatt #Backpropagation #AIWinters #ExpertSystems #GraphicsProcessors #DeepLearning #ScalingEra #TransformerArchitecture #LargeLanguageModels #FutureOfArtificialIntelligence