Download the Book (PDF): Introduction A mouse is placed in the corner of a square, brightly lit box and left alone for ten minutes. Overhead, a camera records every frame. Afterwards, a piece of software reports that the animal travelled a certain number of metres, spent a certain percentage of its time more than a few centimetres from the walls, and reared on its hind legs a certain number of times. Those three numbers will be entered into a spreadsheet, averaged across a dozen animals, compared with a dozen animals of a different genotype, and, if the difference is large enough, described in a paper as an "anxiety-like phenotype." Almost everything interesting about behavioral phenotyping lives inside that paragraph. There is the paradigm, the open field, a test invented in the 1930s and still among the most widely used procedures in neuroscience. There is the automation, a camera and software that replaced a person with a stopwatch and a clipboard. There is the machine vision, which in the last decade has moved from locating a blob of dark pixels against a white floor to estimating the positions of a dozen body parts in every frame and classifying what the animal is doing. And there is the inference, the leap from "spent less time in the centre" to "is more anxious," which rests on assumptions about what the test measures, how the animal experienced the procedure, and whether anything other than genotype differed between the two groups. This book is about that whole chain, from the design of the test to the interpretation of the number. It is written for researchers who use or evaluate behavioral data without being specialists in it: the molecular biologist whose knockout line needs a behavioral characterisation, the electrophysiologist who wants to relate neural activity to movement, the pharmacologist reading a preclinical efficacy package, the computational scientist who has been asked to build a better tracker, the grant reviewer, the journal editor. Each of these readers will at some point have to decide whether a behavioral result is believable. That decision requires knowing what the major paradigms were designed to do, what they can and cannot tell you, how automated measurement actually works, and where error and bias enter. Why behavior is hard to measure Behavior is the final output of the nervous system, which is precisely why it matters and precisely why it is difficult. Almost every manipulation of interest to biomedical science, whether a gene deletion, a drug, an infection, a diet or an injury, eventually expresses itself in what an animal does. But behavior is also the most context-dependent phenotype in biology. A liver enzyme level does not change because the technician who drew the blood wore a different scent, yet, as later chapters describe, pain sensitivity in mice does. Body weight does not depend on whether the animal was lifted by the tail or scooped up in a tunnel, yet anxiety-related measures do. The phenotype is not a fixed property sitting inside the animal waiting to be read out. It is produced by an interaction between the animal, the apparatus, the procedure and the people who run it. The history of the field can be read as a series of attempts to manage that context-dependence. The early ethologists, Tinbergen and Lorenz among them, insisted that behavior be described carefully before it was explained, and that the ecological function of a behavior be kept in view. Laboratory psychologists took the opposite route: they built simplified apparatus, such as mazes, operant chambers and open arenas, precisely to strip away context and isolate a single process such as learning or fear. Behavioral geneticists in the 1990s, faced with thousands of new mutant mouse lines, needed standardised test batteries that could be run quickly on many animals. Each tradition brought something valuable and each left blind spots. A widely cited demonstration of the stakes came in 1999, when John Crabbe, Douglas Wahlsten and Bruce Dudek tested the same inbred and mutant mouse strains in three laboratories, in Portland, Edmonton and Albany, with an unusually rigorous effort to equalise apparatus, test protocols, housing, diet and even the day on which the animals arrived. Some strain differences replicated across all three sites. Others did not: for certain measures, a strain that looked strikingly different from the others in one laboratory looked unremarkable in another. The paper did not show that behavioral phenotyping is hopeless. It showed that the laboratory environment is part of the experiment, whether or not the experimenter records it, and it set much of the agenda that followed. What has changed Three developments have transformed the field since then, and they shape this book. The first is scale and standardisation. International efforts such as the International Mouse Phenotyping Consortium have run standardised pipelines on thousands of knockout lines, generating a public resource and, as a by-product, detailed evidence about which behavioral measures are reliable across centres and which are not. Automated home-cage systems now record animals continuously over days, reducing the need to remove them from their familiar environment and handle them. The second is machine vision. From around 2018, deep-learning pose estimation tools, beginning with DeepLabCut from the Mathis laboratories and quickly followed by LEAP, SLEAP, DeepPoseKit and others, made it possible to track the nose, ears, paws, tail base and other body parts of freely moving animals from ordinary video, with accuracy approaching that of a human labeller, after training on a few hundred hand-labelled frames. Downstream tools learned to turn those body-part trajectories into behavioral labels, either by imitating human annotations or by discovering recurring motifs without human supervision. The result is a shift from a handful of summary variables per test to thousands of features per second. The third is a reckoning with reproducibility. Meta-research over the past fifteen years has documented how often animal studies fail to report randomisation, blinding and sample size calculations, how small most behavioral experiments are, and how strongly unreported factors such as the experimenter, the time of day and the handling method can shift results. Reporting guidelines such as ARRIVE, first published in 2010 and revised in 2020, and a growing literature on experimental design, have changed what journals and funders expect. These three developments interact. Automation promises to remove the human observer from scoring, and it does remove a real source of bias and inconsistency. But it does not remove the human from the experiment. Someone still handles the animal, chooses the test, sets the lighting, labels the training frames, picks the classifier threshold and decides which of thousands of features to report. Automation relocates bias as much as it eliminates it. Understanding where it goes is one of the central aims of this book. The argument of this book The controlling idea can be stated simply. A behavioral phenotype is a measurement of an animal within a procedure, and its value depends on three things holding together: a paradigm whose relationship to the underlying process is understood, a measurement system whose accuracy has been checked against that relationship, and a design that prevents anything other than the manipulation of interest from producing the difference. Machine vision has dramatically improved the second of these. It has done little, by itself, for the first and third, and in some respects it has made them more demanding, because richer data offer more ways to find a difference that is not there. The chapters follow that logic. Chapter 1 sets out what a behavioral phenotype is and how its validity is judged, drawing on the distinctions among face, construct and predictive validity that have organised debates about animal models for four decades. Chapter 2 examines the open field and its relatives, including the elevated plus maze and the light-dark box, the paradigms most often used to infer anxiety and activity, and explains why their interpretation is less straightforward than their popularity suggests. Chapter 3 turns to cognition and the classic mazes: the Morris water maze, the Barnes maze, the radial arm maze, spontaneous alternation and object recognition. Chapter 4 moves beyond the maze to operant touchscreens, fear conditioning and the home cage, where automation has longest been established. Chapter 5 explains how machines learned to see animals, from background subtraction to deep pose estimation in two and three dimensions and with multiple animals. Chapter 6 follows the data from body-part coordinates to behavioral categories, comparing supervised classifiers with unsupervised methods for discovering behavioral structure. Chapter 7 is about the experimenter: the documented ways in which human presence, handling and expectation alter both the animal's behavior and the scoring of it. Chapter 8 addresses design and analysis, including the counterintuitive evidence that excessive standardisation can reduce reproducibility, and the particular statistical hazards of high-dimensional behavioral data. The conclusion draws out what follows for the way laboratories, reviewers and funders should treat behavioral evidence. A note on scope The focus is on laboratory mice and rats, which account for the great majority of behavioral phenotyping in biomedical research, although many of the principles and tools, especially in machine vision, apply equally to flies, fish, primates and other species. The book does not attempt to catalogue every test; there are hundreds, and good protocol collections exist. Instead it treats the major paradigms as case studies in how behavioral tests are built, how they are interpreted and how they go wrong. Where specific software is discussed, it is to illustrate a class of approach rather than to recommend a product, since tools in this area change quickly. Where the book reports findings, it names the study so that readers can check it. Finally, a word about welfare. The refinement of behavioral testing, reducing stress and distress, is often presented as an ethical obligation separate from scientific quality. The evidence reviewed here suggests they are the same project. Animals that are chronically stressed by aversive handling, barren housing or tests run during their inactive phase are not simply suffering more; they are producing noisier and less interpretable data. The best-run behavioral laboratories have discovered that treating the animal well and measuring it well usually turn out to be the same instruction. Chapter 1: What Counts as a Behavioral Phenotype A phenotype, in the strict sense, is any observable characteristic of an organism that results from the interaction of its genotype with its environment. Coat colour, body weight and blood glucose are phenotypes. So is the latency with which a mouse finds a hidden platform in a pool of water. The definition is the same, but the practical situation is very different. Coat colour can be read by anyone at any time of day without altering it. Swim latency exists only when the animal is placed in a particular apparatus under particular conditions, and the act of measuring it changes the animal: the second trial is not the same as the first because the animal has learned something. This chapter sets out the vocabulary needed to think clearly about behavioral phenotypes. It covers what exactly is being measured when a behavioral test is run, how the relationship between a test and the process it is supposed to capture is judged, and why the phenotype is better thought of as a property of an animal-in-a-procedure than of the animal alone. Measures, constructs and the gap between them Every behavioral test produces a set of measures: distances, durations, counts, latencies, proportions. Distance travelled in an open field, time in the open arms of an elevated plus maze, number of errors in a radial arm maze, time spent immobile when suspended by the tail. These are what the software or the observer actually records. Researchers are almost never interested in the measures for their own sake. They are interested in constructs: anxiety, spatial memory, working memory, despair, sociability, motivation, pain. A construct is a hypothesised psychological or biological process that cannot be observed directly but is inferred from patterns of measures. The whole enterprise of behavioral phenotyping depends on the bridge between the two. When a paper reports that a knockout mouse "shows increased anxiety," what it has usually observed is that the mouse spent less time in the centre of an open field, and the bridge from one to the other is an assumption about what centre time reflects. That bridge can fail in many ways. The mouse may spend less time in the centre because it is more fearful of open spaces, which is the intended interpretation. It may also spend less time there because it is less active overall and so crosses the centre less often, because it has a visual impairment and relies more on whiskers against the wall, because it is hypothermic and huddling, or because it was handled more roughly on the way into the arena. A phenotype that is real at the level of the measure can be wrong at the level of the construct. The classic way to organise these problems comes from the debate over animal models of psychiatric disorders. In 1984 the psychopharmacologist Paul Willner proposed three criteria for judging whether an animal model of depression was valid, and his framework has since been applied far more widely. Face validity asks whether the animal's behavior resembles the human condition on the surface: does a "depressed" rodent show reduced interest in rewards, altered sleep, reduced activity? Predictive validity asks whether the model responds to interventions the way humans do: do antidepressants that work in patients reverse the behavior in the animal, and do drugs that fail in patients fail in the animal? Construct validity asks whether the model shares the underlying cause or mechanism: is the behavior produced by the same biological process that produces the human condition? These three criteria pull in different directions. Several of the most widely used rodent tests owe their popularity to predictive validity, because they were calibrated against drugs already known to work in people. The forced swim test, introduced by Roger Porsolt and colleagues in 1977, is the clearest example. Rodents placed in an inescapable cylinder of water eventually stop struggling and float, and a wide range of clinically effective antidepressants delay that transition. For decades this made the test a standard screen for new antidepressants. But predictive validity calibrated against existing drugs tends to find more drugs of the same kind, and it says nothing about whether the behavior has anything to do with depression. Critics have argued, with increasing influence, that immobility in the forced swim test is better understood as a shift in coping strategy, from active escape attempts to passive energy conservation, than as "despair" or "depression-like behavior." A 2017 commentary by Kristen Commons and colleagues in ACS Chemical Neuroscience made the case directly in its title: the test measures stress-coping strategy, not depression-like behavior. In an inescapable situation, floating may be the adaptive response. The general lesson is that labels matter. Calling a measure "anxiety-like," "depression-like" or "memory" imports a construct. Honest phenotyping reports the measure precisely, states the construct it is taken to reflect, and acknowledges the assumptions linking them. The ethogram and the ethological tradition A useful counterweight to construct-first thinking comes from ethology. Before asking what a behavior means, ethologists ask what the animal actually does. The basic tool is the ethogram: a catalogue of the discrete behavioral acts a species performs, each defined precisely enough that two trained observers would agree on when it occurs. A rodent ethogram might include walking, running, rearing (supported against a wall or unsupported in open space), grooming (with its characteristic sequence from face to body to tail), sniffing, digging, freezing, stretched-attend posture, jumping, and dozens of social acts such as anogenital sniffing, following, mounting and attacking. Niko Tinbergen's 1963 essay "On aims and methods of ethology" argued that any behavior could be explained at four levels: its immediate causation (the mechanism), its development over the individual's life, its function (its survival value) and its evolutionary history. Laboratory phenotyping has tended to focus narrowly on mechanism. The ethological reminder is that behavior evolved to solve problems in natural environments, and that laboratory tests make most sense when they engage those evolved solutions rather than fight them. Several of the best rodent tests do exactly this. Mice and rats are prey animals that tend to avoid open, brightly lit spaces where predators can see them, and to stay close to vertical surfaces. This tendency, known as thigmotaxis, underlies the open field, the elevated plus maze and the light-dark box. Rodents are also driven to explore novelty, a tendency that supports object recognition and spontaneous alternation tasks without any training or reward. And they are excellent swimmers who nonetheless prefer to escape water, which the Morris water maze exploits. A test that works with the animal's natural repertoire usually produces cleaner data than one that requires the animal to do something unnatural. The ethological tradition also matters for machine vision. When deep-learning tools began producing continuous streams of body-part coordinates, one of the first questions was what behavioral categories to extract from them. Some approaches trained classifiers to recognise the acts in a human-defined ethogram. Others, discussed in Chapter 6, tried to let the data define the ethogram by finding recurring patterns of movement without human labels. Both are, in a sense, attempts to automate what ethologists have always done by eye. In 2014 David Anderson and Pietro Perona published an influential essay in Neuron titled "Toward a science of computational ethology," arguing that automated measurement could make behavioral description as quantitative and high-throughput as molecular biology. Three years later John Krakauer and colleagues argued in the same journal that "Neuroscience needs behavior," warning that the field's investment in recording and manipulating neurons had outrun its investment in describing what the animals were doing. Both essays reflected a shared recognition: the bottleneck in understanding brain function was increasingly behavioral measurement. Phenotype as interaction The phrase "genotype times environment" is familiar from quantitative genetics, but it has special force in behavior. The Crabbe, Wahlsten and Dudek study described in the introduction found that for some measures, the difference between strains depended heavily on which laboratory ran the test, even though the laboratories had gone to extraordinary lengths to standardise everything they could think of. The statistical term is a genotype-by-laboratory interaction: the effect of genotype is not a single number but varies with the setting. Follow-up work by Wahlsten and colleagues, and later by Neri Kafkafi, Yoav Benjamini, Ilan Golani and collaborators, examined this problem quantitatively. Kafkafi and colleagues argued in a 2005 paper in the Proceedings of the National Academy of Sciences that genotype-by-laboratory interaction should be treated as a real source of variation that any claim of a genotype effect must survive, not as a nuisance to be wished away. If a strain difference is smaller than the typical variation in that difference across laboratories, it is not a reliable finding, however small its p-value in a single laboratory. Later work by the same group, including a 2017 paper in Nature Methods, proposed ways for a single laboratory to estimate this interaction from shared multi-laboratory data and adjust its significance thresholds accordingly. Chapter 8 returns to these ideas. The interaction view has a practical corollary that many researchers find uncomfortable. It means there is no such thing as the phenotype of a genotype in the abstract. There is the phenotype of a genotype in a given test, in a given laboratory, run by given people at a given time of day, with animals raised and housed in a given way. Some phenotypes are robust enough that these factors barely matter; a mouse that cannot walk will show reduced locomotion in any laboratory. Many are not. The job of the experimenter is to know which kind of phenotype they are dealing with, and to design experiments that reveal which parts of a result are robust. This view also changes how one thinks about "negative" results. If a knockout mouse shows no difference from wild-type in the open field, the conclusion is not that the gene has no role in anxiety. It is that, under these conditions and with this sensitivity, no difference was detected in these measures. Behavior is compensatory and flexible; a gene loss that disrupts one route to a behavioral outcome may be buffered by another. Conversely, a positive result in one test does not establish a general trait. Good phenotyping uses convergent evidence from several tests that share a construct but differ in their demands, so that an effect that appears across them is less likely to reflect a peculiarity of any one apparatus. Levels of description and the problem of dimensionality A final distinction runs through the rest of this book. Behavior can be described at several levels of granularity. At the coarsest level are summary measures: total distance travelled, time in a zone, latency to a goal. At an intermediate level are behavioral acts: rearing, grooming, freezing, each with a start and end time. At the finest level are kinematics: the continuous trajectories of body parts, joint angles, velocities and accelerations, sampled many times per second. For most of the twentieth century, practical constraints determined the level. A human observer with a stopwatch could reliably score a few acts or a single latency. Early video tracking systems, which located the animal's centre of mass by distinguishing it from the background, made summary measures of position and movement trivial to compute but could say little about posture. Pose estimation, discussed in Chapter 5, suddenly made kinematics available at scale. More information is not automatically better information. A summary measure such as distance travelled is crude, but it is also robust, easy to interpret and comparable across laboratories. Kinematic data are rich, but they are high-dimensional, noisy, sensitive to camera placement and body size, and easy to overanalyse. The right level of description depends on the question. A study of whether a drug causes sedation may need nothing more than distance travelled. A study of how a basal ganglia lesion alters the fine structure of movement may need gait kinematics. A study of whether a knockout mouse has an anxiety phenotype may benefit from counting risk-assessment postures such as the stretched-attend posture, which older tracking could not detect but which pose-based classifiers can. There is a further subtlety. When many measures are taken from the same test, they are rarely independent. Distance travelled, number of centre entries and number of rearings in an open field tend to be correlated, because all depend partly on general activity. Early factor-analytic work on rodent tests tried to identify a small number of underlying dimensions, such as activity and emotionality, from a large number of measures. The same problem recurs, in a much larger form, with machine-vision data that can produce hundreds of correlated features per animal. Chapter 6 discusses how unsupervised methods address this by finding lower-dimensional structure, and Chapter 8 discusses the statistical danger of reporting whichever of many correlated measures happens to reach significance. A worked example: characterising a new line Consider a laboratory that has generated a mouse lacking a synaptic protein expressed in the amygdala and hippocampus, and wants to know whether the animals differ behaviorally. The naive plan is to run a standard battery (open field, elevated plus maze, a memory task) and report whatever differs. The framework above suggests a better sequence. The first step is to rule out basic sensory and motor problems, because nearly every higher-level test depends on the animal being able to see, hear, smell, walk and grip normally. A general health and neurological screen of the kind formalised in the Irwin observational battery and its mouse descendant SHIRPA, and popularised for mutant lines by Jacqueline Crawley, covering body weight, coat condition, reflexes, grip strength and simple visual placing, is unglamorous but decisive. A mouse with impaired vision will perform badly in a water maze for reasons that have nothing to do with memory. The mouse strain C3H, for example, carries a retinal degeneration mutation that makes it effectively blind in adulthood, and older literature that compared its "spatial learning" with other strains has had to be read with this in mind. The second step is to measure general activity, because it confounds so many other measures. If the knockout travels half as far as wild-type in a novel arena, reduced open-arm time in a plus maze may reflect hypoactivity rather than anxiety. Collecting activity across the light and dark phases in the home cage, where novelty and handling are minimised, helps distinguish a baseline difference in activity from a heightened response to a new environment. The third step is to test the construct of interest with several paradigms that share the construct but differ in their demands. If the hypothesis is increased anxiety, convergent evidence from the open field centre, the elevated plus maze, the light-dark box and perhaps a conflict test such as novelty-suppressed feeding is far more persuasive than a large effect in one. If the effects diverge, that divergence is itself informative: it suggests the phenotype is specific to some feature of one test rather than a general trait. The fourth step is to think about order and history. Animals tested in several procedures carry each experience into the next, so more stressful tests are generally placed later, and the order is kept constant or counterbalanced so that its effects can be estimated. A 2001 study by Kathryn McIlwain, Richard Paylor and colleagues, among others, documented that prior testing history can alter subsequent results in some tests, which is why test order is part of the reported method. The final step is to interpret with restraint. A pattern of reduced centre time, reduced open-arm time and reduced light-compartment time, in the absence of hypoactivity or sensory deficits, supports a claim of increased avoidance of exposed areas. Whether that deserves the word "anxiety," and whether it models any human disorder, is a further claim requiring further evidence. Reliability before validity One more concept completes the toolkit. A measure cannot be valid if it is not reliable, meaning that it gives consistent values when the same thing is measured again. Behavioral reliability has several facets. Inter-rater reliability is the agreement between two observers scoring the same video. Test-retest reliability is the consistency of an animal's score across repeated sessions, which is complicated in behavior because many tests change the animal. Between-laboratory reliability, or reproducibility, is the consistency of a group difference across sites. Automation directly addresses the first. A tracking algorithm run twice on the same video gives the same answer. That is a genuine improvement over human scoring, where two trained observers can differ systematically in how they judge the onset of freezing or the boundary between a grooming bout and a pause. The 2020 study by Oliver Sturman, Johannes Bohacek and colleagues at ETH Zurich illustrated both sides: human raters scoring behaviors such as supported and unsupported rearing in the open field disagreed with each other to a measurable degree, and a machine-learning classifier trained on their pooled annotations reached accuracy comparable to the humans while eliminating the variation between them. But automation says nothing, by itself, about test-retest or between-laboratory reliability, and nothing at all about validity. A perfectly consistent algorithm can consistently measure the wrong thing. It can also be consistently accurate on the videos it was trained on and inaccurate on videos from another laboratory with different lighting, a different arena floor or a different strain with a different coat colour. The next chapters look first at the paradigms themselves, because no amount of measurement precision rescues a test whose relationship to the construct is misunderstood, and then at the machinery of measurement. Chapter 2: The Open Field and Its Relatives No behavioral test has been run more often, or interpreted more variously, than the open field. Its basic form could hardly be simpler: an animal is placed in an enclosed arena, usually square or circular and larger than its home cage, and its behavior is recorded for a few minutes to an hour. There is no training, no reward, no punishment beyond the novelty of the place itself. From this minimal procedure, researchers have drawn conclusions about locomotor activity, exploration, habituation, anxiety, sedation, stereotypy, motor coordination and, more recently, gait and frailty. That versatility is both the strength and the danger of the test. Because the open field is easy to run and cheap to automate, it appears in a large share of papers that characterise a mutant line or a drug. Because it is interpreted so variously, the same measure can be read as evidence for quite different constructs. This chapter examines the open field and the two tests most often used alongside it to infer anxiety, the elevated plus maze and the light-dark box, and argues that each is best understood as a conflict between opposing motivations rather than as a direct readout of a single trait. Origins and the logic of approach and avoidance The open field is usually credited to the American psychologist Calvin Hall, who in 1934 described placing rats in a large circular enclosure and counting their fecal boli as an index of "emotionality." Hall's reasoning was physiological: fear activates the autonomic nervous system, and autonomic activation increases defecation. Rats that defecated more and moved less in the novel arena were considered more emotional, and Hall and his successors used this to breed rat lines selected for high and low emotionality, work that later produced lines such as the Maudsley reactive and non-reactive rats. Defecation counts have largely fallen out of use, but Hall's framing persists. The modern interpretation rests on a conflict. A rodent placed in a novel environment is pulled in two directions. It is motivated to explore, because information about a new place has value for finding food, shelter and mates. It is also motivated to avoid exposure, because open, brightly lit spaces are where predators find prey. The resulting behavior is a compromise. The animal typically stays close to the walls, a tendency called thigmotaxis, makes brief excursions into the centre, and gradually ventures further as the arena becomes familiar. Measures derived from the open field map onto this conflict. Total distance travelled and speed reflect overall activity. Time spent in, distance travelled in, and number of entries into a central zone are taken to reflect the balance between exploration and avoidance, with lower centre measures interpreted as higher anxiety. Rearing, standing on the hind legs either against the wall or unsupported in open space, is usually treated as an exploratory act. Grooming bouts, freezing and the stretched-attend posture, in which the animal extends its body forward while keeping its hind feet in place, are associated with stress or risk assessment. The decline in activity over the session is taken to reflect habituation, a simple form of learning. Pharmacological validation supports part of this reading. Thigmotaxis was proposed as an index of anxiety by Dallas Treit and Marek Fundytus in 1988, who showed that anxiolytic drugs increased the time rats spent away from the walls. A 2003 review by Laetitia Prut and Catherine Belzung in the European Journal of Pharmacology surveyed the pharmacological literature on the open field and found that benzodiazepines, the classic anxiolytics, fairly consistently increased centre measures, while the effects of other drug classes, including serotonergic compounds and antidepressants, were much less consistent. The test, in other words, is sensitive to the kind of anxiolysis that benzodiazepines produce, and less clearly to others. The hidden parameters The open field is standardised in name only. Arena size, shape and material, wall height and colour, floor texture, illumination, the definition of the centre zone, session length, the point at which the animal is introduced, whether the arena is cleaned between animals and with what, and whether the test is run in the light or dark phase of the animal's day all vary across laboratories, and all can alter the results. Illumination is a good example. Because the avoidance component of the conflict depends on exposure, brighter light generally reduces centre time and increases the sensitivity of the test to anxiolytics, while dim light favours exploration. Mouse strains differ in their sensitivity to light, and albino strains, which lack pigment in the eye, are more sensitive to bright illumination. A strain difference found under 400 lux may shrink or reverse under 50 lux. The centre zone is a surprisingly arbitrary construct. It is usually defined as a square occupying some fraction of the floor, but the fraction varies widely between studies. Because time in a zone depends on the zone's area, results obtained with a small centre are not directly comparable with those obtained with a large one. Some laboratories now report distance-from-wall distributions rather than a single zone to avoid the problem. Session length and time course matter too. Activity in a novel open field is highest in the first minutes and declines as the animal habituates. A genotype difference that appears only in the first five minutes may reflect a difference in reaction to novelty rather than in baseline activity. Reporting only the session total conceals this. Plotting activity in bins of a few minutes, and analysing the time course, is almost always more informative. The International Mouse Phenotyping Consortium and its European predecessor projects, including EUMORPHIA, confronted these issues directly because they needed the same test to give comparable results across multiple centres. Their standard operating procedures, collected in the IMPReSS resource, specify arena dimensions, illumination and duration in detail. A 2008 study by Sabrina Mandillo and colleagues in Physiological Genomics, run across several European centres within EUMORPHIA, found that harmonising protocols to this level of detail improved agreement between centres, but that some measures remained more sensitive to local conditions than others. Even among highly motivated professional phenotyping centres, identical procedures on paper did not guarantee identical results. Exploration has structure A different tradition of open-field research, associated with the Israeli ethologist Ilan Golani and his collaborators, argues that summary measures throw away most of what the test reveals. Golani and David Eilam showed in 1989 that rats in a large arena establish a home base, a place, often a corner, where they spend disproportionate time, groom and return after excursions. Exploration is organised as a series of round trips from the home base, each extending a little further, with outbound movement slower and more cautious than the faster return. Later work by Golani, Yoav Benjamini, Ehud Fonio and colleagues quantified this structure in mice. A 2009 paper in the Proceedings of the National Academy of Sciences showed that mice introduced to a large arena build up their exploration in a predictable sequence of gradually increasing freedom of movement, from one dimension along the wall, to two dimensions across the floor, to three dimensions with vertical jumping, and that the stability of this unfolding was characteristic of the animal. The practical argument, made in a 2010 review by Benjamini and colleagues listing ten ways to improve descriptions of whole-animal movement, was that measures derived from this structure, such as the speed profile of excursions or the number of stops, could be more robust across laboratories than conventional summary variables. Software such as SEE (Strategy for the Exploration of Exploration) was built to extract them. This tradition anticipated, by two decades, the argument that machine vision would make routine: that a ten-minute video of an animal in an arena contains far more information than three numbers. Recent work from Vivek Kumar's group at the Jackson Laboratory has taken this idea further. A 2022 paper in Cell Reports by Keith Sheppard and colleagues used pose estimation on open-field video to extract stride-by-stride gait measures from freely moving mice, without a treadmill or walkway, and applied it to many strains and mutant lines. The same group published a machine-vision frailty index in Nature Aging in 2022, led by Leinani Hession and Gautam Sabnis, in which morphometric, gait and behavioral features extracted from open-field video were used to predict a conventional, manually scored frailty index with a mean absolute error comparable to mis-scoring a single item by one point. The open field, in this view, is less a test of anxiety than a general-purpose window through which many phenotypes can be seen, provided the camera and the analysis are good enough. The elevated plus maze The elevated plus maze sharpens the conflict at the heart of the open field. It consists of four arms arranged in a plus shape, raised well above the floor. Two opposite arms are enclosed by high walls; the other two are open, with at most a small lip. The animal is placed in the central square facing an arm and allowed to explore for five minutes. Rodents strongly prefer the enclosed arms, and the proportion of time spent on and entries made into the open arms is taken as an inverse index of anxiety. The test was developed from earlier elevated Y-maze work by Montgomery in the 1950s and introduced in its plus form by Sandra Handley and Sudha Mithani in 1984. Its pharmacological validation in rats by Sharon Pellow, Sandra File and colleagues in 1985 in the Journal of Neuroscience Methods established the conventional measures: open-arm entries and time as a percentage of total, with total or closed-arm entries used as an index of general activity. Adaptations for mice followed quickly. Two features of the plus maze deserve special attention. The first is its sensitivity to prior experience. Sandra File and colleagues documented in the late 1980s and early 1990s that benzodiazepines, which increase open-arm exploration in naive animals, lose this effect when animals are tested a second time, a phenomenon called "one-trial tolerance." Having once experienced the maze, the animal appears to shift from approach-avoidance conflict to a learned avoidance that anxiolytics do not affect in the same way. The practical upshot is that the plus maze should generally be run only once per animal, and that prior exposure to similar apparatus can change what it measures. The second feature is that the standard spatial measures miss much of the behavior. Robert Rodgers and colleagues argued in the 1990s that an "ethological" analysis of the plus maze, scoring risk-assessment behaviors such as stretched-attend postures and head dips over the edge of open arms, together with where they occurred, captured drug effects that the traditional measures missed. These postures were laborious to score by hand, and early video tracking, which represented the animal as a single point, could not detect them. Pose estimation now can, and several recent studies have used it to detect head dips and stretched-attend postures automatically. The light-dark box and related tests The light-dark box, introduced by Jacqueline Crawley and Frederick Goodwin in 1980, divides an arena into a large, brightly lit compartment and a smaller, dark, enclosed compartment connected by an opening. Mice placed in the apparatus show a preference for the dark side, and anxiolytics increase time in the lit side and the number of transitions between compartments. The test relies on the same approach-avoidance conflict as the open field and plus maze, but with a different balance: the dark compartment offers a refuge, so the measure reflects willingness to leave shelter as well as avoidance of light. Several other paradigms belong to the same family. Novelty-suppressed feeding measures how long a food-deprived animal takes to begin eating in a novel, brightly lit arena, pitting hunger against avoidance. The elevated zero maze is a circular version of the plus maze with no central square, which removes the ambiguity of time spent in the centre. Conflict tests such as the Vogel lick test and the Geller-Seifter operant procedure use punishment of rewarded behavior to create the conflict explicitly. These tests are less often used in routine phenotyping because they require more training or deprivation, but they are useful for convergent evidence. As Table 1 sets out, the common anxiety-related paradigms share a logic but differ in their demands and pitfalls. Table 1. Common approach-avoidance paradigms used to infer anxiety-related behavior in rodents. Paradigm Conflict exploited Core measures Main confounds Open field Exploration vs exposed centre Centre time and entries, distance, rearing Activity level, zone definition, lighting Elevated plus maze Exploration vs open, elevated arms Percent open-arm time and entries One-trial tolerance, activity, handling Light-dark box Exploration vs bright compartment Time in light, transitions, latency Light intensity, visual function, activity Elevated zero maze Exploration vs open sectors Open-sector time and entries Activity, handling, arena height Novelty-suppressed feeding Hunger vs novel arena Latency to feed Appetite, deprivation level, body weight Reading the tests together The consistent message from decades of work on these tests is that none of them measures anxiety directly and none measures it alone. Each creates a conflict between drives, and each outcome depends on the strength of both sides and on the animal's ability to act on them. A drug that reduces open-arm avoidance may do so by reducing fear, by increasing exploratory drive, by impairing the perception of height, or by producing a disinhibited, hyperactive state. A mutation that reduces centre time may increase fear, or it may reduce activity, impair vision, or alter the response to handling on the way into the arena. Three practices address this. The first is to always report an activity measure alongside the anxiety measure, and to check whether the anxiety measure still differs once activity is accounted for, for instance by using ratios such as percent open-arm entries, or by analysing covariance. The second is convergent testing across paradigms, as described in Chapter 1. The third, made practical by machine vision, is to look beyond location to posture and movement: an animal that avoids the centre while showing frequent stretched-attend postures toward it is behaving differently from one that avoids it while resting in a corner. There is also a question about which direction of change counts. In many studies, both increases and decreases in centre time are reported as interesting, and both increases and decreases in immobility are interpreted. When the direction of an expected effect is not specified in advance, a test with several measures has many ways to produce a "finding." Chapter 8 discusses how prespecification and design discipline protect against this. Finally, these tests are acutely sensitive to what happens to the animal before it is placed in the apparatus. Transport from the housing room, the method of lifting the animal, the presence and odour of the experimenter, noise in the corridor and the time since the lights came on can all shift anxiety-related measures. Because these are the tests in which experimenter effects have been most thoroughly documented, they are the main examples in Chapter 7. Chapter 3: Mazes and the Measurement of Memory The maze is the oldest instrument of experimental animal cognition. At the turn of the twentieth century Willard Small, working at Clark University, built a rat maze modelled on the hedge maze at Hampton Court, and within a few decades mazes of every shape had become the standard tool for studying learning. Edward Tolman's work on "cognitive maps" in rats in the 1930s and 1940s, in which animals appeared to learn the layout of a maze rather than a sequence of turns, set the theoretical agenda that spatial memory research still follows. The discovery of place cells in the rat hippocampus by John O'Keefe and Jonathan Dostrovsky in 1971, and later of grid cells by May-Britt Moser and Edvard Moser, gave that agenda a neural foundation and made the hippocampus the most intensively studied memory structure in the brain. Because of this history, maze tasks carry enormous weight in behavioral phenotyping. When a mouse model of Alzheimer's disease is characterised, when a candidate cognitive enhancer is tested, or when a gene implicated in intellectual disability is knocked out, a maze task is almost always involved. This chapter describes the major tasks, what they are designed to isolate, and the recurring problem that runs through all of them: separating a failure of memory from a failure of everything else that performance requires. The Morris water maze The water maze, developed by Richard Morris in the early 1980s, remains the most widely used test of spatial learning in rodents. A circular pool, typically between one and two metres in diameter for rats and somewhat smaller for mice, is filled with water made opaque with non-toxic paint or powdered milk. A small escape platform is hidden just below the surface in a fixed location. The room around the pool contains distinctive visual cues on the walls. Animals are released from different starting points around the edge on successive trials and must find the platform to escape the water. Over days, they learn to swim directly to it. Morris's 1981 paper in Learning and Motivation showed that rats could locate the hidden platform using only distal cues, because there was nothing at the platform itself to guide them. In 1982 Morris and colleagues reported in Nature that rats with hippocampal lesions were severely impaired at this task, while still able to swim to a visible platform. The combination, a spatial deficit with intact sensorimotor ability, made the water maze a canonical hippocampal test. The standard protocol, described in detail by Charles Vorhees and Michael Williams in a 2006 Nature Protocols article, has three components. Acquisition trials over several days measure the decline in escape latency and path length as the animal learns. A probe trial, in which the platform is removed and the animal swims for a fixed period, measures how precisely it remembers the location: time spent in the target quadrant, number of crossings of the former platform position, and average distance to it. A cued or visible platform version, in which the platform is marked, tests whether the animal can see, swim and is motivated to escape, independent of spatial memory. Reversal learning, in which the platform is moved to a new location, tests the ability to update a memory. Each component guards against a particular misinterpretation. Escape latency alone is ambiguous because it depends on swim speed; a slower swimmer takes longer even with perfect memory. Path length is less sensitive to speed and is generally preferred. The probe trial distinguishes knowing where the platform is from finding it by chance or by a non-spatial strategy such as swimming in a ring at the right distance from the wall. The visible platform version is essential whenever a group is impaired, because without it a visual deficit or a motivational difference cannot be excluded. Automated tracking has been built into the water maze almost from the beginning, because trajectories in a pool are easy to extract from overhead video against a uniform background. This has allowed analyses richer than latency. A 2009 study by Alexander Garthe, Joachim Behr and Gerd Kempermann in PLoS ONE classified each trial's trajectory into search strategies, such as thigmotaxis (swimming along the wall), random search, scanning, chaining (swimming in a ring at the platform's distance from the wall), directed search, focal search and direct swimming, and showed that the progression of strategies across training revealed differences, for example after manipulations of adult hippocampal neurogenesis, that conventional latency measures missed. Strategy classification has since been automated with machine-learning classifiers in several tools. The water maze has known weaknesses, especially in mice. Mice are competent swimmers but, unlike rats, do not naturally forage in water, and they find swimming more stressful. Ian Whishaw and Jo-Anne Tomie compared mice and rats on dry-land and water mazes in 1996 and found that mice performed comparably to rats on dry-land tasks but markedly worse in the water. Mice often float, swim along the wall for extended periods or show strain-dependent patterns of thigmotaxis that complicate interpretation. The stress of immersion also raises corticosterone and lowers body temperature, and both stress hormones and temperature affect memory. For these reasons many mouse laboratories prefer, or add, a dry-land alternative. The Barnes maze The Barnes maze, introduced by Carol Barnes in 1979 to study age-related memory decline in rats, is the most common dry-land alternative. It consists of a circular, brightly lit elevated platform with a ring of holes around its perimeter. Beneath one hole is a dark escape box; the others are blocked or lead nowhere. Motivation comes from the rodent's aversion to open, bright spaces, sometimes augmented by a loud noise or fan that stops when the animal enters the escape box. Over trials, the animal learns to go directly to the escape hole using distal room cues. Measures parallel those of the water maze: latency to find or enter the escape hole, number of errors (investigations of incorrect holes), path length, and in a probe trial with the escape box removed, time spent near the target hole. Search strategies can also be classified, typically as random, serial (checking holes one after another around the perimeter) or spatial (heading directly to the target or a nearby hole). The Barnes maze is generally considered less stressful than the water maze, and it avoids the confound of swimming ability. Its weaknesses mirror its strengths. Because the aversive stimulus is mild, poorly motivated animals may wander without searching, and some strains are more inclined than others to adopt a serial strategy that solves the task without spatial memory. Latency measures can be long and variable. Careful protocols require habituation, standardised illumination and noise, and consistent cleaning between trials so that animals cannot follow odour trails to the escape hole, a precaution sometimes neglected. The radial arm maze and the distinction between working and reference memory David Olton and Robert Samuelson introduced the radial arm maze in 1976. In its classic form, eight arms radiate from a central platform, and the end of each arm contains a food reward. A food-restricted rat is placed in the centre and allowed to collect the rewards. Rats solve this efficiently, rarely re-entering an arm they have already emptied, and they do so without adopting a simple rule such as always turning right. They appear to remember which arms they have visited within the session. The radial maze made a theoretically important distinction operational. Working memory, in this usage, is memory for information that is useful only within a trial or session, such as which arms have already been visited today. Reference memory is memory for information that remains constant across sessions, such as which arms are ever baited. In a variant in which only some arms are ever baited, entries into never-baited arms are reference memory errors, while re-entries into baited arms already visited within the session are working memory errors. Lesion studies, notably by Leonard Jarrard and others, used this dissociation to argue about which structures support which kind of memory. The radial maze requires food restriction to motivate the animal and several days or weeks of training, which makes it slower than other tasks. Water-based versions exist, as do versions in which doors control access to the arms, allowing the experimenter to impose delays between choices and so test how long information is retained. Automated radial mazes with motorised doors and pellet dispensers reduce handling, a point returned to in Chapter 4. Spontaneous alternation and object recognition Two families of tasks have become popular because they require no training, no reward and no aversive motivation. They exploit the rodent's natural drive to explore what is new. Spontaneous alternation relies on the tendency, first described in the early twentieth century and studied extensively by William Dember and others in the 1950s, of rodents to enter a less recently visited arm of a maze. In a T-maze or Y-maze, an animal that has just visited one arm will tend, on its next choice, to visit another. In the continuous Y-maze version, the animal explores freely for several minutes, and the sequence of arm entries is recorded. Percent alternation is calculated from the proportion of overlapping sets of three consecutive entries that include all three arms. Reviews by Robert Lalonde in 2002 and by Richard Hughes in 2004 surveyed the neurobiology of the behavior and argued that it depends on memory for recently visited places, with hippocampal lesions reducing alternation toward chance. Robert Deacon and Nicholas Rawlins described T-maze alternation protocols for mice in Nature Protocols in 2006. Spontaneous alternation is quick and cheap, but its measure is coarse. Low alternation can result from memory impairment, from a strong side preference, from perseverative tendencies or from low activity, which reduces the number of entries and makes the percentage unstable. Studies should report the total number of arm entries alongside the alternation percentage and exclude animals with too few entries by a rule set in advance. Novel object recognition, introduced by Abdelkader Ennaceur and Jacques Delacour in 1988, exploits the rodent's preference for novelty. In the familiarisation phase, the animal explores two identical objects in an arena. After a delay ranging from minutes to a day, one object is replaced with a new one. An animal that remembers the familiar object will spend more time exploring the novel one. The measure is a discrimination index, the difference between time spent exploring the novel and familiar objects divided by the total. Variants include object location, in which one familiar object is moved to a new position, which places more emphasis on spatial memory, and object-in-context tasks. Protocols such as those described by Marianne Leger and colleagues in Nature Protocols in 2013 emphasise habituation, careful selection of objects of comparable intrinsic interest, counterbalancing of which object is novel and where it is placed, and minimum exploration criteria. Object recognition illustrates a problem that machine vision has transformed. The central measure, time spent exploring each object, depends on a judgment about what counts as exploration. The usual definition is that the animal's nose is directed toward the object within about two centimetres of it, and that climbing on the object or sitting beside it without orienting to it does not count. For decades this was scored by hand, and it is exactly the kind of judgment on which human scorers disagree and on which expectation can bias scoring. Early video tracking, which followed only the animal's centre, could not implement the definition, because the centre point says nothing about where the nose is pointing. Tracking the nose separately, and later full pose estimation, made it possible to implement a consistent operational definition, and pose-based tools for object exploration are now common. The judgment has not disappeared, since someone must still choose the distance threshold and the orientation rule, but it is now applied identically to every animal and every frame, and it can be reported. Learning is a curve, not a number Maze data are repeated-measures data. A water maze experiment with four trials a day for five days produces twenty observations per animal, correlated with one another, and the question of interest is usually about the shape of the learning curve: does one group start differently, learn faster, reach a different asymptote, or forget more between days? Collapsing these data into a single number, such as mean latency across all trials, discards most of this structure and can hide real effects or create spurious ones. A common analysis is repeated-measures analysis of variance with group and day as factors, looking for a group-by-day interaction as evidence of different learning rates. This works when data are complete and well behaved, but maze data often are not. Latencies are bounded by a ceiling, because trials are stopped when an animal fails to find the platform within a fixed time, typically sixty to ninety seconds, and assigned the maximum. The distribution of latencies is skewed. Some animals never learn. Mixed-effects models, which treat each animal as having its own learning trajectory drawn from a group distribution, and survival-analysis approaches that treat the ceiling as censoring rather than as a true latency, handle these features better and are increasingly used. Individual differences also deserve attention rather than averaging away. In many cohorts, particularly of aged animals, performance is not normally distributed but split between animals that learn well and animals that barely learn at all. Carol Barnes, Michela Gallagher and others working on cognitive ageing in rats made this heterogeneity central, classifying aged animals as impaired or unimpaired relative to young controls and then asking what distinguished their brains. A group mean that shows a modest deficit may conceal a subgroup with a severe deficit and a subgroup with none, which is a different biological story. Plotting individual learning curves, not just group means with error bars, is a simple practice that reveals such structure. Finally, the probe trial deserves careful handling. Because it is usually a single trial, it is noisier than acquisition data, and several of its measures are correlated. Deciding in advance which probe measure is primary, for example proximity to the former platform location, which Gallagher and colleagues argued is more sensitive than quadrant time, avoids the temptation to report whichever measure happens to differ. Separating memory from performance The recurring difficulty in every maze task is that performance depends on far more than memory. To find a hidden platform, an animal must see the distal cues, swim competently, be motivated to escape, not be too stressed to learn, and choose a search strategy that uses spatial information. To alternate in a Y-maze, it must be active enough to make many entries and not be dominated by a side bias. To show object recognition, it must explore the objects enough to learn them and be interested enough in novelty to prefer the new one. Any manipulation that affects vision, motor function, motivation, stress reactivity or exploratory drive can produce what looks like a memory deficit. Table 2 summarises how the major tasks differ in their motivation, the memory demand they emphasise and the performance factors they are most vulnerable to. Table 2. Major rodent maze and recognition tasks compared. Task Motivation Memory emphasised Training required Key performance confound Morris water maze Escape from water Spatial reference, reversal Several days Swim ability, vision, stress Barnes maze Escape from bright open space Spatial reference Several days Low motivation, serial strategy Radial arm maze Food reward Working and reference Days to weeks Food restriction, side bias Y- or T-maze alternation Exploration of novelty Short-term spatial None Activity level, side bias Object recognition Exploration of novelty Recognition Habituation only Object preference, low exploration The table points toward good practice. Every memory claim should be accompanied by controls for the relevant performance factors: visible platform trials and swim speed in the water maze, total exploration time in object recognition, number of entries in alternation, and wherever possible, independent evidence of intact vision and motor function. Deficits that appear across tasks with different motivations, for example in both the water maze and the Barnes maze, are much more likely to reflect memory than deficits confined to one. Machine vision contributes here in two ways. It enriches the controls: pose-based tracking can measure swimming kinematics or gait during a maze task, revealing motor differences that latency conceals. And it enriches the memory measures themselves: trajectory shape, heading toward the goal, hesitations at choice points and search strategy all carry information about what the animal knows. The next chapter turns to tasks in which automation has gone further still, taking the experimenter out of the room altogether. Hashtags: #BehavioralPhenotypingInAnimalModels #BehavioralPhenotyping #AnimalModels #RodentBehavior #BehavioralNeuroscience #BehavioralParadigms #OpenFieldTest #ElevatedPlusMaze #LightDarkBox #MorrisWaterMaze #BarnesMaze #RadialArmMaze #ObjectRecognition #HomeCageMonitoring #BehavioralAutomation #MachineVision #DeepPoseEstimation #DeepLabCut #ComputationalEthology #BehaviorClassification #UnsupervisedBehaviorAnalysis #AnimalTracking #Reproducibility #BehavioralValidity #FutureOfBehavioralPhenotyping

Download the Book (PDF): Introduction A pension fund changes one line on an enrolment form. Instead of asking new employees to tick a box if they wish to join the retirement plan, it enrols them automatically and asks them to tick a box if they wish to leave. Nothing else changes: the same fund, the same contribution rate, the same employer match, the same freedom to walk away. Participation among new hires jumps from a minority to a large majority. When Brigitte Madrian and Dennis Shea documented exactly this pattern at a large American employer in 2001, they gave behavioural economics one of its founding field results, and they gave policy makers a reason to believe that small changes in how choices are presented could do large amounts of good. Now consider a different scene. Twenty students sit in cubicles in a university laboratory. Each has been given ten euros and told that they may send any part of it to an anonymous partner in another room; whatever they send will be tripled, and the partner may send back whatever they like. The students make their choices on a screen, and at the end they are paid in cash, privately, according to what they and their partners decided. No one lies to them about the rules. No one watches them choose. The design is sparse to the point of austerity, and that austerity is deliberate: every feature has been chosen so that when the numbers come in, the experimenter can say with confidence what caused them. These two scenes represent the two halves of a single discipline. Behavioural economics grew up by showing that people depart from the textbook model of rational choice in systematic, predictable ways. The laboratory is where those departures were first isolated and measured; the field is where they were put to work. Both depend on the same underlying craft: the design of an experiment whose result means what the researcher says it means. That craft is the subject of this book. Why method is the argument It is easy to treat experimental method as a set of house rules, the kind of thing a referee checks before reading the interesting part. That view gets the relationship backwards. In experimental economics, method is where the scientific claim lives. When a paper reports that people cooperate more than self-interest predicts, the claim is only as good as the answers to a series of prior questions. Were the participants paid according to their decisions, so that cooperation cost them real money? Did they understand the game, or were they confused into apparent generosity? Did the instructions, the framing, or the presence of the experimenter tell them what kind of behaviour was hoped for? Did they believe the partner was real, and did they have any reason to doubt it? Each of those questions corresponds to a design decision, and a wrong decision at any one of them can manufacture a finding out of nothing. The controlling idea of this book follows from that observation. An experiment in the decision sciences is a controlled environment built to support one causal inference, and every design choice, from how people are paid to what they are told, should be judged by whether it protects or corrupts that inference. Incentives exist so that the choices observed are the choices people would make when something is at stake. The strategy method exists to recover behaviour at decision points that would otherwise be seen too rarely. Rules against demand effects exist because the participant's guess about the hypothesis is a confound. The ban on deception exists because a subject pool that has learned not to trust experimenters no longer responds to the environment the experimenter has built. These are not separate conventions but applications of one principle. That principle also explains why the laboratory and the field are complements rather than rivals. The laboratory buys control at the cost of realism; the field buys realism at the cost of control. A nudge that works in a laboratory choice task may vanish in the noise of a real enrolment process, and a nudge that works in the field may do so for reasons the field cannot reveal. Good research programmes move between the two, and good researchers know what each setting can and cannot license them to conclude. What this book covers The book is written for people who design, commission, or evaluate behavioural experiments: graduate students in economics, psychology, and management; analysts in government behavioural insights teams and corporate research groups; and researchers in neighbouring disciplines who have begun running choice experiments and want to know why economists insist on doing things in a particular way. It assumes some familiarity with the ideas of behavioural economics, but no specialist training in experimental design. Chapter 1 sets out what an economic experiment is for, beginning with Vernon Smith's account of experiments as controlled microeconomic systems and the conditions under which an experimenter can claim to have induced the preferences she intends to study. It also lays out the modern taxonomy running from the conventional laboratory to the natural field experiment, because the design decisions in later chapters depend on where a study sits on that spectrum. Chapter 2 deals with incentives: why economists pay participants according to their decisions, what the evidence says about when payment matters, and how the standard devices, from random lottery incentives to the Becker-DeGroot-Marschak mechanism and proper scoring rules, work and fail. Chapter 3 turns to choice architecture as an experimental treatment, showing how to translate a nudge into a manipulation with a clear mechanism, a credible control condition, and an outcome that measures behaviour rather than intention. Chapter 4 examines the strategy method, in which participants state what they would do at every decision point of a game before learning what their counterpart did. It is one of the most useful tools in the experimentalist's kit and one of the most debated, and the chapter sets out what the comparative evidence shows. Chapter 5 addresses experimenter demand effects: the ways in which participants infer and respond to what the researcher seems to want, the evidence on how large those effects are, and the design choices that contain them. Chapter 6 takes up deception, the sharpest methodological line between experimental economics and much of social psychology, and explains why the prohibition is enforced at the level of the whole subject pool rather than the individual study. Chapter 7 carries the argument into the field, where randomisation happens inside organisations, participants do not know they are in an experiment, and the ethical and practical problems multiply. Chapter 8 closes the design cycle with inference: power, pre-registration, multiple outcomes, heterogeneity, replication, and the question of whether an effect found once will survive being scaled. What this book leaves out A short book declines far more than it includes. There is no survey here of the substantive findings of behavioural economics, except where a finding illustrates a design problem. There is little on the econometrics of experimental data beyond what an experimenter must decide before collecting it. Neuroeconomic methods, online labour-market experiments as a subject in their own right, and the large literature on survey experiments in political science are touched on only where they bear on the central questions. Nor does the book offer a software tutorial; the platforms change faster than the principles. What it does offer is a way of thinking about a design decision so that it can be defended. A researcher who understands why incentives, the strategy method, demand effects, and deception matter in terms of the single inference an experiment is built to support will be able to work through cases this book never mentions. That is the point of learning method rather than memorising rules. A note on evidence Experimental methodology is itself an empirical subject. Whether paying subjects changes their behaviour, whether the strategy method elicits the same choices as direct play, whether participants who have been deceived behave differently afterwards: these are questions that have been studied with experiments, and the answers are not always what the conventions would suggest. Where the evidence is clear, the book says so. Where it is mixed, it says that too, and explains what a careful researcher should do in the face of uncertainty. The references in the notes are to real, published work that the reader can consult. The field's conventions are strong, but they are not sacred, and the best reason to follow them is that one understands what they protect. Chapter 1: What an Experiment Is For Every experiment is an argument with a single conclusion in mind. The researcher wants to say that a particular feature of the environment, and nothing else, caused a particular change in behaviour. Everything about the design follows from the need to make that sentence true. Before discussing incentives, games, demand effects, or deception, it is worth being precise about what kind of sentence an economic experiment is designed to support, and what it means for an environment to be controlled. Experiments as microeconomic systems The modern understanding of economic experiments owes most to Vernon Smith, whose 1982 article in the American Economic Review, "Microeconomic Systems as an Experimental Science", gave the field its working vocabulary. Smith described any economic situation as a system with two parts. The first is the environment: the preferences, endowments, and technology of the agents involved. The second is the institution: the rules that govern how agents communicate, what messages they may send, and how those messages translate into outcomes. An auction, a posted-price market, a bargaining protocol, and a public-goods contribution game are all institutions. Behaviour is what agents do within a given environment and institution, and outcomes are what results. In the world outside the laboratory, the environment is mostly hidden. We do not observe what a used car is worth to a buyer or what a seller would accept. Smith's insight was that the laboratory allows the experimenter to create the environment rather than infer it. If a participant is told that each unit of a fictitious commodity will be redeemed at the end of the session for eight euros, then eight euros is her value for that unit, whatever she may privately think of the commodity itself. The experimenter knows the demand and supply curves exactly, because she wrote them down. She can then vary the institution, say from a double auction to a posted-offer market, and observe whether prices converge to the competitive equilibrium that theory predicts. This technique is called induced value, and it rests on conditions that Smith stated as a set of precepts. Three matter most for what follows. Non-satiation. Participants must prefer more of the reward medium to less, and must not become satiated within the range of the experiment. Money serves this purpose well for most people at the stakes experiments use. Saliency. The reward a participant receives must depend on her actions in the way the institution specifies. If she is paid a flat fee regardless of what she does, the induced values are merely numbers on a page; she has no reason to act on them. Dominance. Changes in the participant's utility from the experiment must come predominantly from the reward medium, and other influences, such as boredom, curiosity, the wish to please the experimenter, or concern for other participants, must be small by comparison. Smith added privacy, the requirement that each participant receives information only about her own payoff, as a way of protecting dominance when interpersonal considerations would interfere. He also stated the parallelism precept: propositions about behaviour tested in the laboratory apply to non-laboratory settings where similar ceteris paribus conditions hold. Parallelism is not a condition the experimenter can guarantee. It is a working assumption that later chapters will return to repeatedly, because much of the debate about laboratory and field experiments is a debate about when it holds. The inferential target Induced-value experiments test theories of institutions. They ask whether a given set of rules, operating on a known environment, produces the outcome theory predicts. Behavioural economics, however, is often interested in something different: the preferences and cognitive processes of the participants themselves. When an experimenter runs a dictator game, in which one person decides how to split a sum of money with an anonymous other who has no say, she is not inducing a preference. She is trying to measure one: how much the dictator cares about the recipient's payoff, or about appearing fair. This distinction clarifies what control means in each kind of study. In an induced-value market experiment, control means that the participant's valuation of the commodity is the one the experimenter assigned. In a preference-elicitation experiment, control means that the only thing varying across conditions is the feature under study, and that the participant's choice reflects her own preferences rather than her confusion, her beliefs about what the experimenter wants, or her suspicion that the rules are not what they seem. The dominance precept still matters, but now it cuts differently. Concern for the other participant is not a nuisance to be suppressed; it is the object of measurement. The nuisances are everything else: demand, misunderstanding, distrust. A useful discipline is to write down, before designing anything, the single sentence the experiment is meant to support, in the form "Changing X from A to B causes behaviour Y to change by roughly Z, among people like P, in settings like S." Each part of that sentence is a design commitment. X must be manipulated cleanly, with nothing else changing alongside it. Y must be measured in a way that reflects real behaviour rather than stated intention. Z requires enough observations to detect an effect of plausible size. P and S define the claims the study will be able to make about generality, and they are where the choice between laboratory and field comes in. Internal and external validity Two kinds of validity organise the rest of this book. Internal validity concerns whether the effect observed in the study was caused by the manipulation. Randomisation is the foundation: if participants are assigned to conditions by chance, then in expectation the conditions differ only in the treatment, and any systematic difference in outcomes can be attributed to it. But randomisation alone is not sufficient. If the treatment changes both the feature of interest and the participant's beliefs about what the experimenter expects, the effect is confounded even though assignment was random. If participants in one condition misunderstand the instructions more than participants in another, the effect may reflect confusion rather than preference. Internal validity is lost not only through selection but through any channel by which the manipulation affects behaviour other than the one the researcher claims. External validity concerns whether the effect generalises: to other populations, other stakes, other contexts, other times. A finding in a laboratory with undergraduate participants may or may not describe the behaviour of retirees choosing pension funds. A finding in one country's tax authority may or may not transfer to another's. External validity cannot be secured by any single design, but it can be threatened or strengthened by design choices, and the experimenter who ignores it will make claims her data cannot support. The two kinds of validity trade off, but not in the simple way sometimes suggested. It is not true that laboratory experiments have high internal validity and low external validity while field experiments have the reverse. A badly designed laboratory study can have poor internal validity because of demand effects or confusion, and a well-designed field experiment can have excellent internal validity because randomisation is clean and participants are unaware of being studied. What is true is that the laboratory gives the experimenter more levers to pull in pursuit of internal validity, while the field removes some threats to external validity by studying real people making real decisions in their natural environment. A taxonomy from laboratory to field The most widely used classification of economic experiments comes from Glenn Harrison and John List, whose 2004 survey in the Journal of Economic Literature distinguished experiments along several dimensions at once: the nature of the subject pool, the information and experience participants bring to the task, the commodity or stakes involved, the task and its rules, and the environment in which participants act. From these they derived four broad categories, which Table 1 summarises. Table 1. The Harrison-List taxonomy of experiments. Type Participants Task and stakes Awareness of study Typical use Conventional laboratory Standard pool, usually students Abstract framing, induced or modest real stakes Full Testing theory, measuring preferences Artefactual field Non-standard pool, e.g. traders, farmers Laboratory-style task Full Checking whether results depend on the pool Framed field Non-standard pool Task, commodity or information from the participants' own context Full Adding realism while keeping control Natural field Non-standard pool Task participants undertake in daily life None Measuring behaviour in its natural setting The categories are not rungs on a ladder of quality. Each answers a different question. A conventional laboratory experiment asks whether a behavioural regularity exists under tightly controlled conditions. An artefactual field experiment asks whether that regularity depends on the peculiarities of students; running the same trust game with professional traders or with rural households tests the assumption that the pool does not matter. A framed field experiment asks whether the regularity survives when the task uses the goods, context, and information that participants deal with in their own lives. A natural field experiment asks whether the regularity appears when people do not know they are being studied and are making decisions they would have made anyway. Moving along this spectrum changes what the experimenter controls. In the conventional laboratory she controls almost everything: the information participants receive, the payoffs, the timing, the interaction structure, the physical setting. In a natural field experiment she typically controls only the treatment itself, perhaps the wording of a letter or the default option on a form, and the assignment of people to conditions. Everything else is the world as it is. Two further categories deserve mention. Online experiments, run on platforms that recruit participants over the internet, occupy an uneasy middle ground. They offer larger and more diverse samples than a university laboratory, but less control over attention, comprehension, and the physical conditions of choice, and they raise particular concerns about participants who complete many studies and learn to recognise standard paradigms. Natural experiments, in which some external event assigns people to conditions as if by chance, are not experiments in the sense used here, because the researcher does not control assignment. They are valuable, but the design decisions discussed in this book mostly do not apply to them. Why behavioural economics needs both settings The history of behavioural economics can be read as a conversation between the laboratory and the field. Many of the field's central phenomena were first identified in controlled settings. The endowment effect, the tendency to value a good more once one owns it, was demonstrated by Daniel Kahneman, Jack Knetsch, and Richard Thaler in a series of laboratory experiments with coffee mugs published in the Journal of Political Economy in 1990. Present bias, loss aversion, and social preferences were likewise measured first in the laboratory before researchers sought them in naturally occurring data. The laboratory findings then raised questions that only the field could answer. Does the endowment effect persist among experienced traders? List's work on sports-card markets, published in the Quarterly Journal of Economics in 2003, found that it diminished markedly among dealers and experienced collectors, suggesting that market experience may attenuate the anomaly. Do social preferences measured in dictator games predict charitable giving? Steven Levitt and John List, writing in the Journal of Economic Perspectives in 2007, argued that laboratory measures of pro-social behaviour may overstate what people do in the field, because the laboratory combines scrutiny by the experimenter, an unusual social context, and self-selected participants in ways that inflate generosity. Their article provoked a substantial literature, and the evidence since has been mixed: some laboratory measures predict field behaviour reasonably well and others do not. The lesson is not that one setting is right and the other wrong, but that the parallelism precept is an empirical claim to be tested case by case. The movement also runs in the other direction. Field results frequently send researchers back to the laboratory to understand mechanisms. The success of automatic enrolment in pension plans could reflect inertia, procrastination, an implicit endorsement conveyed by the default, or the cognitive cost of making an active choice. Field data alone struggle to separate these explanations, because the default embodies all of them at once. Laboratory experiments that vary one feature at a time, for instance comparing a default described as recommended with one described as randomly chosen, can isolate the channels in a way the field cannot. Designing for a single inference The practical consequence of this chapter is a habit of mind. Before deciding how to pay participants, what to tell them, or whether to use the strategy method, the experimenter should know which inference the study is designed to support and which threats to that inference are most serious. A study measuring how default options shape choices among inexperienced savers faces different threats from one testing whether a new auction format raises revenue. The first worries most about whether a laboratory choice task resembles the real enrolment decision closely enough to say anything about it. The second worries most about whether bidders understand the rules and have their values correctly induced. It is also worth being explicit about the counterfactual. Every treatment effect is a comparison, and the choice of control condition determines what the effect means. A reminder letter compared with no letter measures the effect of contact. A reminder letter compared with a neutral letter of the same length measures the effect of the reminder's content. A default of enrolment compared with a default of non-enrolment measures something different from a default of enrolment compared with a forced active choice. None of these comparisons is wrong, but each supports a different sentence, and designs that blur them produce results that cannot be interpreted. The chapters that follow take up the major design decisions one at a time. Each can be understood as a response to a particular way in which the link between manipulation and inference can break. Incentives protect against choices that cost nothing and therefore reveal little. The strategy method addresses the problem of observing enough behaviour at rare decision nodes. Attention to demand effects guards against participants responding to the experimenter rather than the environment. The prohibition on deception preserves the credibility on which all induced environments depend. Field implementation and statistical inference determine whether a well-designed treatment yields a result that can be trusted and used. In every case, the question to ask is the same: does this choice help or harm the one conclusion the experiment exists to reach? Chapter 2: Paying for Decisions The most visible difference between an economics experiment and a psychology experiment is the envelope of cash at the end. Economists pay participants according to what they decide; psychologists have traditionally paid a flat fee or course credit. The difference is not a matter of disciplinary taste. It reflects a view about what a choice in an experiment is supposed to reveal, and it generates a set of practical problems whose solutions are among the most carefully studied topics in experimental methodology. Why pay at all The argument for incentives follows directly from Smith's precepts of saliency and dominance. If a participant's earnings do not depend on her choices, then the choices cost her nothing. She may still choose carefully, out of conscientiousness or curiosity, but she may equally choose quickly, randomly, or in whatever way she thinks will make the session end sooner or please the experimenter. A hypothetical question about whether she would prefer a certain twenty euros or a coin flip for fifty elicits her beliefs about her own preferences, filtered through whatever she thinks is the sensible or admirable answer. An incentivised question elicits a decision whose consequences she will live with. Ralph Hertwig and Andreas Ortmann, in a much-cited 2001 article in Behavioral and Brain Sciences, set out the contrast between the two traditions. They identified four practices on which experimental economics and psychology differ: whether scripts and instructions are standardised, whether participants repeat tasks and receive feedback, whether financial incentives are used, and whether deception is permitted. On incentives, they argued that the psychological practice of hypothetical or flat payment introduces uncontrolled variance, because participants' motivation becomes an unobserved variable that differs across people and conditions. Paying for performance does not guarantee that participants are motivated in the way the experimenter wants, but it makes the motivation more uniform and more transparent. What the evidence says about when incentives matter Whether incentives change behaviour is an empirical question, and it has been studied directly. The most influential review is by Colin Camerer and Robin Hogarth, published in the Journal of Risk and Uncertainty in 1999, which examined experiments that varied financial incentives across conditions. Their conclusion was nuanced. Incentives rarely reverse the direction of behavioural anomalies; many biases persist when participants are paid. But incentives often reduce the variance of responses, improve performance on tasks where effort helps, such as memory and clerical work, and move behaviour in risky-choice tasks toward greater risk aversion. For tasks where effort does not help, because the difficulty lies in understanding rather than diligence, incentives made little difference. Camerer and Hogarth summarised this as a capital-labour-production framework: incentives raise effort, but effort only improves performance when the participant's cognitive capital allows it to. A second strand of evidence concerns the size of stakes. Charles Holt and Susan Laury, in a 2002 article in the American Economic Review, introduced what is now one of the most widely used risk-elicitation tasks: a list of ten paired choices between a safer and a riskier lottery, with the probability of the higher payoff rising down the list. The row at which a participant switches from the safer to the riskier option measures her risk attitude. Holt and Laury compared low real stakes, high real stakes of twenty, fifty, and ninety times the baseline, and hypothetical high stakes. Measured risk aversion increased sharply when real stakes were scaled up, but not when the high stakes were merely hypothetical. People said they would behave at high stakes much as they did at low stakes, but when the money was real, they became markedly more cautious. This is one of the cleaner demonstrations that hypothetical responses can mislead in exactly the direction that matters. A third strand shows that small incentives can sometimes be worse than none. Uri Gneezy and Aldo Rustichini, in a 2000 article in the Quarterly Journal of Economics titled "Pay Enough or Don't Pay at All", found that participants paid a small amount per correct answer on an IQ-style test performed worse than those paid nothing, while those paid larger amounts performed better. A small payment appears to replace intrinsic motivation with a transaction that is not worth much effort. For the experimenter, the lesson is that incentives should be large enough to be meaningful, and that a trivial per-decision payment may not produce the salience that the design assumes. The practical guidance that emerges is proportionate rather than absolute. For tasks measuring preferences over money, including risk, time, and social preferences, incentives are close to essential, because the hypothetical answers are known to differ systematically. For tasks measuring cognitive performance, incentives help where effort matters. For tasks measuring attitudes, perceptions, or beliefs about non-monetary matters, incentives may be impossible or irrelevant, and the experimenter must be clear that she is measuring stated rather than revealed responses. Paying for one decision or many Most experiments ask participants to make many decisions. Paying for all of them creates two problems. The first is budgetary: an experiment with fifty risky choices at meaningful stakes would be expensive. The second is more subtle. If every decision is paid, participants accumulate wealth as the session proceeds, and their later choices may reflect the earnings already banked, a so-called wealth or portfolio effect. A participant who has already won a large amount may become more willing to gamble, or may treat the sequence of choices as a single portfolio and diversify across them. The standard solution is the random lottery incentive system: at the end of the session, one decision (or a small number) is chosen at random and paid for real. Because each decision has some probability of being the one that counts, and because there is no accumulation within the session, each should be treated as if it were the only one. The logic is attractive, but it depends on a theoretical assumption. The random lottery incentive system is incentive compatible only if the participant's preferences satisfy a form of the independence axiom of expected utility theory, since she is in effect facing a compound lottery over which decision will be paid. Charles Holt pointed out in the American Economic Review in 1986 that if participants violate independence, as much of behavioural decision research suggests they do, the random lottery system may distort the very choices it is meant to elicit. Edi Karni and Zvi Safra, writing in Econometrica in 1987, showed a related problem for preference-elicitation mechanisms more generally. The empirical evidence has been more reassuring than the theory. Chris Starmer and Robert Sugden, in a 1991 article in the American Economic Review, tested whether choices made under the random lottery system differed from choices made when a single decision was paid for certain, and found little evidence of distortion in their setting. Subsequent studies have mostly confirmed that the system works reasonably well in practice, though with exceptions. Gary Charness, Uri Gneezy, and Brianna Halladay reviewed the question in a 2016 article in the Journal of Economic Behavior and Organization, "Experimental Methods: Pay One or Pay All", and concluded that paying for one randomly selected decision is generally a sound practice, with some evidence that it performs at least as well as paying for all decisions. The more recent theoretical treatment by Yaron Azrieli, Christopher Chambers, and Paul Healy, published in the Journal of Political Economy in 2018, clarified exactly which assumptions each payment mechanism requires. Their central result is that paying for one randomly chosen decision is essentially the only mechanism that is incentive compatible under a weak monotonicity assumption, one much weaker than the full independence axiom. That result gives the standard practice a firmer foundation than it previously had, while making clear that no mechanism can be incentive compatible for every conceivable preference. Eliciting valuations and beliefs Many experiments need to measure a number rather than a choice: how much a participant would pay for a good, what she believes the probability of an event to be, or what she expects others to do. Simply asking produces answers that may be strategic or careless. A family of mechanisms exists to make truthful reporting optimal, and each has its own failure modes. Table 2 compares the most widely used. Table 2. Common incentivised elicitation mechanisms. Mechanism What it measures How it works Known weaknesses Becker-DeGroot-Marschak Willingness to pay or accept Participant states a price; a random price is drawn; trade occurs at the random price if favourable Poorly understood; misconceptions shift reported values Multiple price list Risk, time or value thresholds Series of binary choices with one attribute varying; switch point measures preference Multiple switching; anchoring on middle of list Quadratic scoring rule Probabilistic beliefs Payment falls with squared distance between report and outcome Biased by risk aversion; hard to explain Binarised scoring rule Probabilistic beliefs Quadratic score converted into probability of winning a fixed prize Robust to risk attitudes in theory; still complex Second-price auction Valuations in groups Highest bidder wins and pays second-highest bid Overbidding common despite dominance of truthful bidding The Becker-DeGroot-Marschak mechanism, introduced by Gordon Becker, Morris DeGroot, and Jacob Marschak in Behavioral Science in 1964, is the classic device for eliciting a participant's value for a good. She states the most she would pay. A price is then drawn at random. If the price is below her stated value, she buys at the random price; if above, she does not buy. Because the price she pays does not depend on what she states, only whether she buys, stating her true value is a dominant strategy. In theory, the mechanism is flawless. In practice, participants often misunderstand it. The clearest evidence comes from Timothy Cason and Charles Plott, whose 2014 article in the Journal of Political Economy asked participants to state selling prices for a card with a known induced value. Since the card's value to the participant was fixed by the experimenter, the only correct answer was to state that value. Many participants did not. Their reports were systematically distorted in a way consistent with confusing the BDM mechanism with a first-price auction, a phenomenon Cason and Plott called game form misconception. The implication is uncomfortable: some of the gap between willingness to pay and willingness to accept that has been attributed to preferences may instead reflect participants' misunderstanding of the elicitation procedure. This concern was at the heart of the debate over the endowment effect. Charles Plott and Kathryn Zeiler, in a 2005 article in the American Economic Review, argued that the gap between buying and selling prices could be eliminated by careful procedures: extensive training in the elicitation mechanism, anonymity, practice rounds, and incentive-compatible elicitation. Their results suggested that at least part of the anomaly was a product of experimental procedures rather than preferences. The ensuing debate, which continued for several years, turned on precisely the questions this book is about: whether a procedural feature, in this case the elicitation mechanism and the training that accompanied it, protected or corrupted the inference about what participants actually valued. Multiple price lists, of which the Holt-Laury task is the best known, avoid some of the comprehension problems of the BDM mechanism by breaking the elicitation into simple binary choices. Their weakness is that participants sometimes switch back and forth more than once, which is inconsistent with any monotone preference, and that the range and midpoint of the list can anchor responses. Experimenters commonly impose a single switch point, for instance by asking participants to choose the row at which they switch, which eliminates inconsistency at the cost of hiding it. Belief elicitation raises its own problems. The quadratic scoring rule pays participants according to the accuracy of a probability report, and truthful reporting maximises expected payment for a risk-neutral participant. But participants are rarely risk neutral, and a risk-averse participant will shade her report toward fifty per cent to reduce the variance of her payment. The binarised scoring rule, developed by Tanjim Hossain and Ryo Okui and published in the Review of Economic Studies in 2013, addresses this by converting the score into a probability of winning a fixed prize, which makes truth-telling optimal regardless of risk attitude under weak conditions. Yet a growing body of work, including research by Danz, Vesterlund, and Wilson published in the American Economic Review in 2022, suggests that the complexity of these rules can itself distort reports: when participants are shown how the payment rule works in detail, they report less accurately than when they are simply told that truthful reporting is in their interest. The more transparent the incentive, the more room there appears to be for participants to try to game it incorrectly. Practical rules for incentive design Several principles follow from this evidence. First, pay for decisions when the object of study is a preference over money, and pay enough that the amounts at stake are meaningful relative to what participants could earn elsewhere in the same time. A useful benchmark in many laboratories is that average earnings should exceed the local hourly wage for student jobs. Second, when a study involves many decisions, pay for one or a small number chosen at random, and explain clearly that each decision might be the one that counts. The random lottery incentive is well supported by current theory and evidence, and it avoids wealth effects. Third, prefer simple elicitation mechanisms to clever ones. A mechanism that is incentive compatible in theory but misunderstood in practice elicits confusion. Where complex mechanisms are unavoidable, provide practice rounds, test comprehension before the paid part of the session, and consider telling participants directly that truthful reporting maximises their expected earnings rather than asking them to verify this from the formula. Fourth, be careful about the meaning of the show-up fee. Nearly all laboratory experiments pay a fixed amount for attendance, which is necessary for recruitment but can create a house-money effect if participants treat it as a windfall to gamble or give away. Where the design involves losses, the show-up fee is often used as a buffer so that participants do not leave owing money, but the buffer changes the psychological reference point. Some designs require participants to earn their endowment through a real-effort task, which appears in several studies to reduce generosity in allocation games relative to endowments that are simply given. Fifth, remember that incentives are not the only source of salience. A participant who does not understand how her choices map to her earnings is not facing salient incentives, however much money is on the table. Comprehension checks, clear instructions, and worked examples are part of incentive design, not an afterthought to it. The general principle behind all five rules is the one stated in the first chapter. Incentives exist to make the observed choice an expression of the participant's preferences over outcomes she cares about. Any incentive scheme that introduces new motives, such as hedging across decisions, gaming the scoring rule, or treating the show-up fee as found money, may corrupt rather than protect the inference. The goal is not to pay as much as possible, or in the most theoretically elegant way, but to pay in the way that makes the chosen action mean what the experimenter will say it means. Chapter 3: Choice Architecture as a Treatment Richard Thaler and Cass Sunstein gave the term choice architecture wide currency in their 2008 book Nudge, defining a nudge as any aspect of the choice architecture that alters people's behaviour in a predictable way without forbidding any options or significantly changing their economic incentives. The definition is useful for policy but awkward for experimental design, because it describes interventions by what they do not do rather than by what they do. An experimenter cannot randomise "an aspect of the choice architecture". She must randomise something specific, a default, a framing, an ordering, a reminder, a comparison with peers, and she must know which psychological mechanism she believes it engages. This chapter is about turning the idea of a nudge into a manipulation that supports a clear inference. From intervention to mechanism The difficulty with many nudge studies is that the treatment is a bundle. A redesigned letter from a tax authority may change the wording, the layout, the salience of the deadline, the tone, the inclusion of a social norm, and the length, all at once. If the new letter raises payment rates, the study has shown that the new letter works better than the old one. It has not shown why, and without knowing why, neither the researcher nor the policy maker can predict whether it will work somewhere else. Consider the default effect. Madrian and Shea's 2001 study in the Quarterly Journal of Economics compared employees hired before and after a firm switched to automatic enrolment in its retirement plan, and found a dramatic increase in participation among those hired afterwards. Eric Johnson and Daniel Goldstein, in a short 2003 article in Science titled "Do Defaults Save Lives?", compared countries with opt-in and opt-out regimes for organ donation and found very large differences in consent rates, supplemented by an online experiment in which participants chose under different defaults. These findings established that defaults matter. They did not establish why. At least four mechanisms have been proposed. Inertia and transaction costs: changing from the default requires effort, and people who would prefer another option may not bother. Procrastination: people intend to change but keep postponing it. Implied endorsement: the default is read as a recommendation from whoever designed the system. Reference dependence: the default becomes a reference point, and moving away from it is coded as a loss. Each mechanism predicts different things. If inertia drives the effect, reducing the cost of switching should shrink it. If endorsement drives it, telling people the default was chosen at random should shrink it. If reference dependence drives it, the effect should be stronger when options are described in terms of what is given up. These predictions are testable, and experiments testing them are more informative than experiments that simply show defaults matter. A laboratory study can assign participants to a default described as recommended by experts, a default described as randomly determined, and an active-choice condition with no default, while holding the options constant. A field study can vary the effort required to opt out, for instance by comparing a return envelope with an online form. The mechanism matters for policy as well as science. Gabriel Carroll, James Choi, David Laibson, Brigitte Madrian, and Andrew Metrick, in a 2009 article in the Quarterly Journal of Economics titled "Optimal Defaults and Active Decisions", showed that requiring employees to make an active choice about retirement saving, rather than giving them a default, produced participation rates close to those of automatic enrolment while allowing people to choose contribution rates more suited to their own circumstances. Whether active choice or a default is better depends on how heterogeneous people's optimal choices are, and whether the default effect reflects inertia or endorsement. Incentivised choice architecture in the laboratory Much evidence on nudges comes from the field, where choices have real consequences but mechanisms are hard to isolate. The laboratory offers a complementary route, but only if the choice task carries real consequences. A hypothetical choice about organ donation or retirement savings, made by a student who will never face the decision, measures what the student thinks she would do. That is informative about beliefs and attitudes but not about behaviour. The solution is to build choice tasks in which the architecture shapes decisions that cost participants something. Several designs are common. In real-good choice tasks, participants choose among goods they will actually receive, such as snacks, vouchers, or small consumer items, under different presentations, orderings, or defaults. In allocation tasks, participants choose how to divide money between themselves and a charity or another participant, and the experimenter varies the default allocation or the framing. In savings and time tasks, participants allocate an endowment between an earlier and a later payment date, with the architecture varying the default or the presentation of interest. In effort tasks, participants choose how much work to do, and the design varies commitment options or reminders. These tasks are what "incentivised choice architecture" means in practice. They preserve the core virtue of the laboratory, which is the ability to vary one element of the presentation while holding everything else constant, while ensuring that the choice has consequences. They also allow the experimenter to measure things the field cannot. Because the experimenter knows the participant's payoff from every option, she can calculate how much each participant lost or gained by following the default. Because she can elicit the participant's preferences in a separate unframed task, she can check whether the nudge moved people toward or away from their own considered choices. That second possibility is important for welfare analysis, which is the question behind every nudge: were the people who were nudged better off? Douglas Bernheim and Antonio Rangel, in a 2009 article in the Quarterly Journal of Economics, "Beyond Revealed Preference", set out a framework for welfare analysis when choices depend on the frame in which they are made. Their approach treats choices made in some frames as more informative about welfare than choices made in others, for instance treating choices made without a default, after careful deliberation, or with full information as more reliable guides. An experiment designed with this framework in mind will include a condition that approximates the privileged frame, so that nudged choices can be compared with it. Designing a credible control A treatment effect is always a comparison, and in choice architecture research the control condition is often the weakest part of the design. Several failures are common. The first is the status quo control in which the control group simply receives whatever the organisation was already doing. This is appropriate when the question is whether a new intervention outperforms current practice, which is often the policy question. But it bundles every difference between the new and old intervention, and the effect cannot be attributed to any single feature. The second is the no-contact control, in which the control group receives nothing at all. A study comparing a letter with a social-norm message against no letter measures the combined effect of the contact and the message. Any letter might have raised payment by reminding people of the obligation. The more informative design adds a third arm, a plain letter without the norm, which isolates the content from the contact. The third failure is differential salience, in which the treatment is noticed and the control is not. If the treatment email has an eye-catching subject line and the control email is routine, the effect may reflect whether the email was opened rather than what it said. Well-designed studies match the controls on everything except the feature of interest, including length, visual design, timing, and sender. The fourth failure is choosing the wrong outcome. Many nudge experiments measure intentions, clicks, or sign-ups rather than the behaviour that matters. An intervention that raises the number of people who click on a link to a savings plan has not shown that it raises saving. An intervention that raises stated intentions to get vaccinated has not shown that it raises vaccination. Katherine Milkman and a large team of collaborators, in a 2021 megastudy published in Nature, tested more than fifty interventions designed to encourage gym attendance among tens of thousands of members of a fitness chain, and measured actual visits recorded over the following weeks rather than stated plans. The effects they found were mostly modest, and both experts and lay forecasters predicted poorly which interventions would work, which is a strong argument for measuring behaviour directly rather than trusting intuitions about it. The outcome should be as close as possible to the behaviour of ultimate interest, and where it is not, the design should include a follow-up measure that is. Common nudges and their experimental problems A handful of intervention types account for most of the choice architecture literature, and each raises distinctive design problems. Defaults. Beyond the question of mechanism, defaults raise a problem of active versus passive outcomes. If a default enrols people in a savings plan, measured participation rises, but some of those enrolled may offset the saving by borrowing or by reducing saving elsewhere. Raj Chetty, John Friedman, Søren Leth-Petersen, Torben Nielsen, and Tore Olsen, in a 2014 article in the Quarterly Journal of Economics using Danish administrative data, found that automatic employer contributions to retirement accounts raised total saving substantially, while tax subsidies mainly shifted saving between accounts among a minority of active savers. The lesson for design is to measure the full set of outcomes the default might affect, not only the one it targets. Social norms. Messages that tell people how their behaviour compares with others' are among the most widely used nudges. P. Wesley Schultz and colleagues, in a 2007 article in Psychological Science, showed that telling households their energy use relative to neighbours reduced consumption among high users but increased it among low users, a boomerang effect, and that adding an approving or disapproving symbol eliminated the increase. Hunt Allcott's 2011 article in the Journal of Public Economics evaluated Opower's home energy reports across many utilities and found average reductions of around two per cent. His later work in the Quarterly Journal of Economics in 2015 found that the earliest sites to adopt the programme had systematically larger effects than later ones, because the utilities that chose to participate first differed from those that came later. This site selection bias is a design problem that no single experiment can solve, and it is a warning about generalising from early successes. Reminders and planning prompts. Reminders are cheap and often effective, but their effect depends on timing and on whether the target behaviour was already intended. Dean Karlan, Margaret McConnell, Sendhil Mullainathan, and Jonathan Zinman, in a 2016 article in Management Science, found that reminders to save increased saving in several countries, particularly when they mentioned a specific goal. The experimental problem is that reminders interact with other contact from the same organisation; a reminder that arrives alongside a bill may have a different effect from one that arrives alone. Commitment devices. Commitment devices let people bind their future selves. Richard Thaler and Shlomo Benartzi's Save More Tomorrow programme, reported in the Journal of Political Economy in 2004, let employees pre-commit to raising their savings rate with future pay rises, and participants' saving rates rose substantially over several years. Nava Ashraf, Dean Karlan, and Wesley Yin, in a 2006 article in the Quarterly Journal of Economics, offered a commitment savings account to bank clients in the Philippines and found that those offered it saved more. The design problem with commitment devices is selection: the people who take them up are those who know they need them, so the effect of the offer differs from the effect of the device on those who choose it. Well-designed studies randomise the offer and report the intent-to-treat effect alongside the effect on take-up. Framing and salience. Changing how information is presented, for instance whether a fee is shown as a percentage or a currency amount, or whether a price includes tax, can change choices. Raj Chetty, Adam Looney, and Kory Kroft, in a 2009 article in the American Economic Review, showed that posting tax-inclusive prices on supermarket shelves reduced demand for the affected products, implying that shoppers underreacted to taxes that were not displayed. Framing effects are easy to produce and hard to interpret, because any change in presentation can change what participants understand the choice to be. The experimenter must be confident that the frames are informationally equivalent, or be explicit about what information each conveys. The measurement of effect size The early literature on nudges often reported large effects, and some of the most prominent findings have since been revised. Stefano DellaVigna and Elizabeth Linos, in a 2022 article in Econometrica, analysed the complete set of randomised trials run by two large nudge units in the United States. They found that the average effect in these trials was an increase of about 1.4 percentage points in take-up, much smaller than the average of around 8.7 percentage points found in published academic nudge studies. The gap was largely explained by publication bias and by the smaller samples of academic studies: small studies that happened to find large effects were more likely to be published. A 2022 meta-analysis by Stephanie Mertens and colleagues in the Proceedings of the National Academy of Sciences reported a moderate average effect across many choice architecture interventions. A subsequent letter in the same journal by Maximilian Maier and colleagues argued that once publication bias was accounted for, the evidence for an average effect largely disappeared, though it noted that some intervention types, especially defaults, remained well supported. The debate is not resolved, but it carries a clear lesson for experimenters. Choice architecture effects are real but usually modest, heterogeneous across settings, and easily exaggerated by small studies. A design that can only detect a large effect is likely to produce either a null result or an inflated estimate. Designing a choice architecture experiment Putting these points together yields a sequence of decisions. Begin by naming the mechanism the intervention is meant to engage, and design at least one arm that would behave differently if a rival mechanism were at work. Define the control condition so that it differs from the treatment only in the feature of interest, adding a separate contact-only arm if the comparison with no contact matters. Choose an outcome that measures the behaviour of ultimate interest, and plan a follow-up if the immediate outcome is an intermediate step. In the laboratory, attach real consequences to choices, and include an unframed elicitation of preferences so that the welfare effect of the nudge can be judged. In the field, measure the outcomes that the intervention might affect indirectly, not only the one it targets. Size the study to detect an effect of the magnitude the recent evidence suggests is plausible, not the magnitude early studies reported. None of this makes choice architecture research harder than it needs to be. It makes it informative. A nudge experiment that shows only that a redesigned intervention outperformed the old one has told one organisation something useful about one population at one time. A nudge experiment designed around a mechanism has told everyone something about how people choose. Hashtags: #BehavioralEconomicsExperiments #BehavioralEconomics #ExperimentalEconomics #DecisionSciences #LaboratoryExperiments #FieldExperiments #NaturalFieldExperiments #ChoiceArchitecture #Nudges #CausalInference #ExperimentalDesign #IncentiveCompatibility #InducedValue #RealIncentives #StrategyMethod #ExperimenterDemandEffects #DeceptionInExperiments #RandomLotteryIncentives #BDMMechanism #BeliefElicitation #InternalValidity #ExternalValidity #PreRegistration #BehavioralInterventions #FutureOfBehavioralExperiments

Download the Book (PDF): Introduction In 2015 the Open Science Collaboration published the results of an enormous coordinated effort: 100 studies drawn from three leading psychology journals, replicated by independent teams working with the original materials and, wherever possible, the blessing of the original authors. Ninety-seven of the hundred original studies had reported a statistically significant effect. Thirty-six of the replications did. The average effect size fell by roughly half. That result has been argued over ever since — about sampling, about fidelity, about whether the replication teams got the conditions right. What has been argued over much less is the thing sitting underneath it. Every one of those original papers had passed the same gate. Each had produced a number below .05 and had been read, by authors, reviewers, editors and readers, as having established something. The gate turned out to be far more permeable than anyone using it believed. This booklet is not another postmortem of the replication crisis. It is about the statistical machinery that the crisis exposed, and about a different way of doing the work — one that has been available for two and a half centuries in principle, and for about fifteen years in practice, because only recently has the computing made it routine. The argument is simple enough to state in a sentence. Null hypothesis significance testing answers a question no experimental psychologist actually asks, and does so in a way that systematically misleads; Bayesian estimation answers the question you meant to ask, and modern sampling algorithms have made it something you can do on a laptop in the afternoon. Everything that follows unpacks that claim and then teaches you to act on it. What is actually wrong The standard complaint about p-values is that people misinterpret them. That is true, and it is the least interesting thing wrong with them. A p-value is the probability of obtaining data at least as extreme as what you observed, if the null hypothesis were exactly true and if you repeated the study indefinitely under identical conditions. Note what the probability is attached to: the data, conditional on a hypothesis. What a researcher wants is the reverse — the probability of a hypothesis, conditional on the data they actually collected. These are different quantities, and no amount of care in interpretation converts one into the other. You cannot get P(hypothesis | data) out of P(data | hypothesis) without a prior. That is not a philosophical quibble; it is arithmetic. There is a second problem, less discussed and arguably worse. The p-value is a statement about a long-run procedure, and the procedure includes everything you would have done under data you did not see. Whether you would have stopped collecting at n = 50 or carried on to n = 80; whether you would have analysed the third dependent measure had the first two come out flat; whether you would have run the interaction test if the main effect had been large. All of these change the sampling distribution, and therefore change the p-value, without changing the data one bit. The "researcher degrees of freedom" literature — Simmons, Nelson and Simonsohn's 2011 demonstration that a few innocuous-seeming analytic choices push the false positive rate above 60% — is not a story about fraud. It is a story about an inferential framework whose output depends on counterfactuals that are neither recorded nor recoverable. And there is a third problem, which is what filtering on significance does to the literature. When a field publishes only results that clear a threshold, and when studies are underpowered, the published effects are the ones that happened to come out large. Gelman and Carlin's work on Type S and Type M errors makes the arithmetic vivid: with low power, a statistically significant result is not merely uncertain in magnitude, it is guaranteed to overstate the effect, and carries a non-trivial chance of having the sign wrong. The 2015 replication project's halved effect sizes are exactly what that arithmetic predicts. What Bayesian analysis offers instead The Bayesian move is to treat unknown quantities — the effect of your manipulation, the variability between participants, the slope relating two measures — as things you have beliefs about, and to update those beliefs with data using the one rule that probability theory permits. The output is not a verdict but a distribution: a full description of which parameter values are consistent with what you observed and what you knew beforehand. From that distribution you can read off anything you want. The most probable value. An interval containing 95% of the posterior probability, which means exactly what students have always wrongly assumed a confidence interval means. The probability that the effect is positive. The probability that it is large enough to matter. The predicted distribution of a future participant's response. None of these requires a new test, a new table, or a new set of assumptions; they are all just summaries of one object. This is not a rhetorical advantage. It changes what you can conclude. A conventional analysis that fails to reach significance leaves you with nothing to say — "we found no evidence of an effect" is compatible with a study that ruled out anything of consequence and with a study that was hopelessly noisy. A posterior distribution distinguishes those cases immediately: in one, the credible interval is tight around zero; in the other, it stretches from a large negative effect to a large positive one. Same p-value, entirely different scientific situation. Why now Bayes' theorem dates to the 1760s. Laplace used it to estimate the mass of Saturn. The reason it did not become the working statistics of the experimental sciences is that for almost all interesting models, the posterior distribution has no closed-form expression and cannot be integrated by hand. For two hundred years, being a Bayesian in practice meant either restricting yourself to a handful of mathematically convenient cases or doing approximations you could not fully trust. What changed is simulation. If you cannot write down the posterior, you can instead draw samples from it, and then compute any summary you like from the samples. Markov chain Monte Carlo — the family of algorithms that does this — emerged from physics in the 1950s, entered mainstream statistics in the early 1990s, and became genuinely practical for applied researchers when Hamiltonian Monte Carlo and its self-tuning variant, the No-U-Turn Sampler, were packaged into Stan. A hierarchical model that would have been a doctoral thesis in 1995 now fits in ninety seconds. This booklet takes that tooling seriously. The code it shows is Stan and R — specifically `brms`, which writes Stan code for you from model formulas that look almost exactly like the `lme4` syntax most experimental psychologists already know. The examples are psychology examples: reaction times, accuracy, Likert responses, repeated measures on the same participants. What this booklet is and is not It is a working guide. By the end you should be able to specify a model for a typical psychology experiment, choose and defend priors, fit the model, tell whether the sampler did its job, interpret the posterior honestly, compare competing models, and write the whole thing up in a way a sceptical reviewer will accept. It is not a mathematical statistics text. There are no proofs. Where a derivation matters — and one or two genuinely do — it is walked through in arithmetic you could check on paper. Where the mathematics is only decoration, it is left out. It is also not a manifesto. Bayesian methods are not magic, and a badly specified Bayesian model is worse than a competently run t-test because it comes dressed in more authority. A substantial part of what follows is about failure modes: priors that do more work than you realised, samplers that quietly fail to explore the posterior, model comparison metrics that answer a question you did not ask. The discipline of checking is not an optional extra bolted onto the method; it is the method. One more thing, which some readers will want stated up front. Nothing here requires you to believe that frequentist statistics are wrong in some deep sense. They are answers to well-posed questions about long-run error rates, and there are settings — regulatory drug trials, quality control — where those are the right questions. The claim is narrower and more practical: for a psychologist trying to learn the size and reliability of an effect from a finite, expensive, noisy experiment, the Bayesian question is the one worth asking, and it is now no harder to ask than the other one. Chapter 1 begins by taking the old machinery apart carefully enough that you can see precisely which part fails, and why replacing it is not a matter of taste. Chapter 1: What the p-Value Cannot Tell You Start with a concrete experiment, because the abstractions only become sharp when there is something at stake. You are testing whether a brief mindfulness induction improves performance on a sustained attention task. Forty participants, randomly assigned: twenty do a ten-minute guided breathing exercise, twenty listen to a neutral audiobook passage. Then everyone performs a twenty-minute continuous performance task, and you record d-prime. The mindfulness group's mean d-prime is 2.31, the control group's is 1.94, pooled standard deviation 0.62. An independent-samples t-test gives t(38) = 1.89, p = .066. What do you now believe about the effect of mindfulness induction on sustained attention? The honest answer, within the framework you just used, is that you are not entitled to believe anything in particular. The test has told you that if the true difference were exactly zero, data this extreme or more so would occur about 6.6% of the time in indefinite replication of this exact procedure. That is a statement about a hypothetical infinite sequence of experiments under a hypothesis you have no reason to think is precisely true. It is not a statement about mindfulness. This chapter works through why that gap cannot be closed from inside the framework, why the failure is structural rather than a matter of user error, and what the alternative question looks like. The inversion nobody performs correctly Write the p-value formally. It is P(T ≥ t_observed | H₀ true, sampling plan S) and what you want is P(effect is in some range | data observed). These are conditional probabilities running in opposite directions, and the relationship between them is Bayes' theorem, which requires a prior. There is no route from one to the other that does not pass through a statement about how plausible the hypothesis was before you collected data. It is worth dwelling on how badly the two can diverge, because the intuition that "p = .066 means there's about a 6.6% chance the null is true" is not merely imprecise, it can be off by an order of magnitude. Consider a field where, say, one in ten tested hypotheses reflects a real effect of appreciable size — not an unreasonable estimate for exploratory social psychology. Suppose studies have 50% power, which was roughly the median in psychology for decades. Out of a thousand tested hypotheses: 100 are real, of which 50 produce significant results; 900 are null, of which 45 produce significant results at α = .05. So of 95 significant findings, 45 — nearly half — are false positives. The p-value on each of those 45 papers was below .05, and on some of them it was below .01. This calculation is not exotic. It is the same base-rate reasoning that every psychology undergraduate learns in the context of medical screening, and then unlearns the moment they open SPSS. The p-value does not, and cannot, incorporate the base rate. Bayesian inference does nothing but incorporate the base rate, which is what the prior is. Jacob Cohen made this argument with some force in his 1994 paper "The Earth Is Round (p < .05)", and Gerd Gigerenzer sharpened it in "Mindless Statistics" a decade later. Neither was a fringe figure, and neither was ignored. What happened instead is that the argument was accepted in the abstract and had no effect on practice, because there was no practical alternative available to the working researcher with a deadline. That, more than any philosophical disagreement, is the historical reason for the persistence of NHST. The sampling plan is part of the answer The second structural problem is subtler and, for experimental work, more damaging. A p-value depends on the sampling distribution of the test statistic — what values it would take across hypothetical repetitions. That distribution is determined not only by the data-generating process but by your stopping rule and your analysis plan. Two researchers can collect the identical dataset and be entitled to different p-values, because they intended different things. Suppose researcher A planned to collect exactly 40 participants and did. Researcher B planned to collect 20, look, and add 20 more if p fell between .05 and .15. B ends up with the same 40 participants and the same numbers. B's p-value is not .066, because B's procedure had two chances to declare significance and the sampling distribution must account for the first look. The correction, if B is honest about it, is real. Now notice what this means. The evidential content of the data is identical. The difference lies entirely in a counterfactual — what B would have done in a world that did not occur. And in practice that counterfactual is unobservable and unverifiable. This is precisely the crack that optional stopping, undisclosed exclusion rules, and flexible outcome selection slip through. Simmons, Nelson and Simonsohn's 2011 paper in Psychological Science showed that combining four commonplace flexibilities — two dependent measures, optional stopping, a covariate, dropping a condition — raises the probability of at least one significant result above 60% when nothing is going on. Preregistration, the field's main structural response, works by making the counterfactual auditable. It is a good response. But it is a patch on a framework whose output should not have depended on intentions in the first place. Bayesian inference does not have this property. The posterior depends on the likelihood of the data you actually observed, and the likelihood is unaffected by why you stopped collecting. If you look at your posterior after every participant and stop when it satisfies you, your posterior is still a correct summary of what the data say given your model and prior. This is a consequence of the likelihood principle, and it is not a small practical matter — it means that sequential analysis, which in the frequentist world requires specialised alpha-spending machinery, is simply the default Bayesian behaviour. (There is a caveat, which honest treatments state: if you stop at a self-serving threshold you will get a biased collection of published posteriors, and your own posterior can be made to look temporarily impressive by an unlucky run. Bayesian methods protect the internal coherence of your inference, not the literature from selective reporting.) What significance filtering does to effect sizes The third failure is the one with the clearest fingerprints on the replication record. Take the mindfulness example and suppose, for the sake of argument, that the true standardised effect is d = 0.25 — a real but modest benefit. With twenty participants per group, the power to detect that at α = .05 is about 13%. So in the rare event that your study reaches significance, what must have happened? The observed difference had to exceed roughly 0.64 standard deviations in the sample. Any significant result from this design therefore reports an effect at least 2.5 times the true one. Andrew Gelman and John Carlin named the two components of this: the Type M (magnitude) error, the factor by which significant estimates exaggerate, and the Type S (sign) error, the probability that a significant estimate points the wrong way. For small true effects and small samples, Type M ratios of 3 to 10 are routine, and Type S probabilities of 5–15% are not unusual. Neither quantity appears anywhere in a conventional results section. This gives you the mechanism behind the 2015 replication findings without needing any story about bad faith. If a literature publishes only significant results from underpowered designs, the published effect sizes are inflated as a mathematical necessity. A replication with adequate power then recovers something smaller. The original authors, looking at their own significant result, have no way of knowing whether they are looking at a real effect of the size they estimated or at the tail of a small one. The Bayesian response is not a different test. It is that the posterior, with a prior that reflects the honest fact that enormous effects are rare in psychology, does the shrinking automatically. A prior concentrated on plausible effect sizes pulls an implausibly large sample estimate back toward the range the field has reason to expect, and the amount of pulling depends on how much data you have. That is not a thumb on the scale; it is what regularisation means, and every statistician who has worked on prediction problems, Bayesian or otherwise, ends up doing something like it. The two frameworks side by side It helps to see exactly which quantities each approach produces, because the words overlap while the meanings do not. Table 1 sets out the correspondences that matter most in practice. Table 1. Corresponding quantities in frequentist and Bayesian analysis, and what each actually licenses. Question a researcher asks Frequentist output What it means Bayesian output What it means Is there an effect? p-value Tail probability of the data under an exact null and a fixed sampling plan Posterior distribution of the effect Relative plausibility of every effect size given data and prior How big is it? Point estimate + standard error Value minimising squared error in repeated sampling Posterior mean or median Centre of the updated belief distribution How sure am I? 95% confidence interval 95% of such intervals, across repetitions, contain the true value 95% credible interval 95% posterior probability the value lies inside Is it negligible? Non-significant p, or TOST equivalence test Failure to reject, or rejection of effects outside bounds Posterior probability inside a region of practical equivalence Direct probability the effect is trivially small Which model is better? Likelihood ratio test, AIC Nested test, or predictive penalty Bayes factor, or LOO expected log predictive density Evidence ratio, or estimated out-of-sample predictive accuracy The fourth row deserves attention because it is where the practical asymmetry bites hardest in experimental psychology. Frequentist testing has no natural way to accept a null. The best available tools — equivalence testing, the two one-sided tests procedure — work by rejecting a different null, and require you to name a smallest effect size of interest, which is the right discipline but an awkward fit with how psychologists think. Bayesian estimation answers the question directly: here is the posterior probability that the effect lies within the range you consider negligible. Nothing new is needed. The confidence interval does not rescue you A common intermediate position is that psychologists should abandon p-values in favour of estimation with confidence intervals — "the new statistics", in Geoff Cumming's phrase. This is an improvement, and the emphasis on effect sizes and precision was overdue. But it does not solve the interpretive problem, because the confidence interval is also a statement about a procedure rather than a parameter. Richard Morey and colleagues made the point sharply in a 2016 paper with the unambiguous title "The Fallacy of Placing Confidence in Confidence Intervals". A 95% confidence interval is a member of a class of intervals, 95% of which would contain the true value across repetitions. It does not follow that this particular interval has a 95% chance of containing it, and the paper constructs examples — a submarine hatch problem being the memorable one — where a valid confidence interval can be known with certainty to contain the parameter, or known with certainty not to, while retaining its 95% coverage property. Almost every researcher reads a confidence interval as a credible interval. The useful thing about actually computing a credible interval is that the reading becomes correct. For simple models with flat priors the two intervals often coincide numerically, which is why the misinterpretation persists without visible damage. For hierarchical models, for bounded parameters, for small samples, and for anything non-linear, they can differ a great deal, and then the misinterpretation starts costing something. What replaces the test Suppose we redo the mindfulness study Bayesianly. We model d-prime in each group as normally distributed, put a prior on the difference in means that says effects larger than about one standard deviation are unlikely in this literature, and compute the posterior. The output is not a verdict. It is a distribution over the difference. Suppose it has a median of 0.34 with a 95% credible interval from −0.03 to 0.71. You can now say several true things. The most probable effect is a benefit of about a third of a d-prime unit. There is roughly a 96% posterior probability the effect is positive. There is a 71% probability it exceeds 0.2, the smallest difference your lab considers practically meaningful. And the interval is wide, which tells you that forty participants was not enough to pin this down, and quantifies how much more you would need. Compare that with "p = .066, n.s." Same data. One of these is a research finding and the other is a shrug. Note also what did not happen. We did not reach a different conclusion by relaxing a standard. The Bayesian analysis is, if anything, more conservative than an uncorrected test, because the prior pulls the estimate toward zero. What changed is that the output is a description of our uncertainty rather than a binary licence to make a claim. The objection about subjectivity The standard objection arrives immediately: the prior is arbitrary, so the conclusion can be whatever you want it to be. This deserves a real answer, and Chapter 3 gives it at length. The short version has three parts. First, the choice is not between a subjective analysis and an objective one; it is between a subjective input that is written down and defended and one that is smuggled in through the choice of model, the choice of test, the exclusion criteria and the stopping rule. Second, with sample sizes typical of experimental psychology and priors of the mild, regularising kind recommended throughout this booklet, the prior's influence on the posterior is usually modest and always calculable — you can vary it and report what happens, which is called a sensitivity analysis and takes about four lines of code. Third, in the cases where the prior does dominate, that is information: it means the data were too weak to speak for themselves, which you would want to know. The frequentist framework is not free of assumptions; it is free of stated assumptions. Normality, homogeneity of variance, independence, a fixed sampling plan, and an exactly-zero null are all assumed, most of them known to be false, and none of them is subjected to the scrutiny that a prior attracts. Where this leaves us The case against NHST is not that it is difficult or that people misuse it. It is that it computes the wrong conditional probability, that its output depends on unobservable intentions, and that filtering on its threshold corrupts the effect sizes a field believes it knows. Each of those is a structural property, not an implementation defect. The case for Bayesian estimation is that it computes the quantity researchers have been mistakenly reading off frequentist output for a century, that it does so in a way that is indifferent to when you stopped collecting, and that it shrinks noisy estimates toward plausibility as a matter of course. What it costs you is that you must say what you believed beforehand, and you must learn a computational method that occasionally fails in ways you have to be able to detect. The next chapter takes the first of those on: what Bayes' theorem actually does, worked through by hand on a problem small enough to see all the way through. A note on what is not being claimed Two misreadings of this chapter are worth heading off. The first is that p-values are meaningless. They are not; they are a well-defined property of a testing procedure, and in settings with genuine repetition and a genuine need to control long-run error — screening thousands of genetic markers, monitoring a production line, running a regulated clinical trial with a preset stopping boundary — they answer the operative question. The complaint is that a single psychology experiment is not that setting. The second is that the problems described here are solved by lowering the threshold. The proposal to redefine statistical significance at .005 would reduce false positives and would also multiply the sample sizes required, and it leaves every structural feature intact: the conditional probability still runs the wrong way, the sampling plan still enters the calculation, and significance filtering still inflates published effects — by more, since a stricter threshold selects more extreme results. Lowering the threshold treats a problem of interpretation as a problem of calibration. What is being claimed is narrower: that the quantity a psychologist wants is the posterior, that no manipulation of the p-value produces it, and that the posterior is now cheap to compute. Chapter 2: The Machinery of Updating Bayes' theorem is one line of algebra that follows immediately from the definition of conditional probability. The reason it takes a chapter is not that the algebra is hard but that using it changes what the symbols mean, and the change is easy to nod along to without absorbing. Probability attached to parameters In the framework you were trained in, a population parameter — the true mean difference between your two conditions, the true correlation between working memory span and reading comprehension — is a fixed unknown constant. Probability statements about it are meaningless, because it either equals 0.3 or it does not. Probability belongs only to the data, which are random because sampling is random. The Bayesian move is to let probability describe uncertainty, whatever its source. The parameter is still a fixed unknown constant; you simply do not know which constant it is, and a probability distribution is the natural language for describing degrees of belief across the possibilities. This is sometimes called epistemic probability, and it is how everyone reasons outside the lab. You do not think there is a 70% chance of rain in the sense that a long run of identical Tuesdays produces rain on seven in ten of them; you mean you are fairly but not entirely convinced. Once you accept parameters as objects of probability, everything else is bookkeeping. The theorem For a parameter θ and observed data y: p(θ | y) = p(y | θ) × p(θ) / p(y) Four objects, each with a name and a job. p(θ) is the prior: what you believed about θ before seeing these data. It is a distribution over the whole parameter space, not a single guess. p(y | θ) is the likelihood: for each possible value of θ, how probable the data you actually got would be if that value were true. Note carefully that once y is fixed at its observed value, this is a function of θ, not a probability distribution over θ — it need not integrate to one, and it is not a belief about θ. It is the channel through which data speak. p(θ | y) is the posterior: your updated belief. This is the output, and in a well-conducted Bayesian analysis it is essentially the entire result. p(y) is the marginal likelihood or evidence: the probability of the data averaged over all values of θ weighted by the prior. It does not depend on θ, so for the purpose of finding the shape of the posterior it is just a normalising constant that makes the distribution integrate to one. It returns with a vengeance in Chapter 9, where it turns out to be exactly what a Bayes factor is built from. Because the denominator does not depend on θ, the working form of the theorem is usually written as a proportionality: posterior ∝ likelihood × prior That is the whole engine. Belief before, multiplied point by point by how well each parameter value predicted what you saw, renormalised. Everything in this booklet is an elaboration of that sentence. Doing it by hand, once Abstractions about updating become concrete the moment you compute one, so here is a complete worked example with a parameter that takes only a few values. It is artificially coarse, which is the point: you can follow every multiplication. A participant in a forced-choice recognition task gets 7 out of 10 trials correct. Chance is 50%. What is their true accuracy rate θ? Suppose, for tractability, that θ can only be one of five values: 0.5, 0.6, 0.7, 0.8, 0.9. Before the data, you think a participant is most likely somewhere in the middle of that range — the task is moderately hard — so you assign prior probabilities 0.15, 0.25, 0.30, 0.20, 0.10. These sum to one. Now the likelihood. For each candidate θ, the probability of exactly 7 successes in 10 binomial trials is C(10,7) θ⁷ (1−θ)³, and C(10,7) = 120. Compute: · θ = 0.5: 120 × 0.0078125 × 0.125 = 0.1172 · θ = 0.6: 120 × 0.0279936 × 0.064 = 0.2150 · θ = 0.7: 120 × 0.0823543 × 0.027 = 0.2668 · θ = 0.8: 120 × 0.2097152 × 0.008 = 0.2013 · θ = 0.9: 120 × 0.4782969 × 0.001 = 0.0574 Multiply each likelihood by its prior: · 0.5: 0.1172 × 0.15 = 0.01758 · 0.6: 0.2150 × 0.25 = 0.05375 · 0.7: 0.2668 × 0.30 = 0.08004 · 0.8: 0.2013 × 0.20 = 0.04026 · 0.9: 0.0574 × 0.10 = 0.00574 These five numbers sum to 0.19737 — that is p(y), the marginal likelihood, the average probability of seeing 7/10 across all the θ values you thought plausible. Divide through: · θ = 0.5: 0.089 · θ = 0.6: 0.272 · θ = 0.7: 0.406 · θ = 0.8: 0.204 · θ = 0.9: 0.029 That is the posterior. Several things are worth noticing. First, the prior probability that θ = 0.5 was 0.15 and the posterior is 0.089: the data pushed belief away from chance performance but did not eliminate it. Seven out of ten is not much evidence. A conventional binomial test on these data returns p = .17 and would be reported as no evidence of above-chance performance, which is a cruder rendering of the same fact. Second, the posterior is not the likelihood. The likelihood peaks at θ = 0.7 and falls off slowly toward 0.9; the posterior falls off much faster on the upper side, because the prior said 0.9 was unlikely to begin with. With only ten trials, the prior still matters. Third — and this is the part worth internalising — if you now ran ten more trials, you would use this posterior as the prior for the next update, and you would arrive at exactly the same place as if you had analysed all twenty trials at once with the original prior. Bayesian updating is associative. That property is what makes the framework indifferent to when you look at your data. From five values to a continuum Real parameters are continuous, so the sums become integrals and the probabilities become densities. The structure is identical. In one special family of cases the integral has a closed-form answer, and it is worth seeing one because it builds intuition about what data do to a prior. If your prior for a probability θ is a Beta(a, b) distribution, and you observe s successes and f failures, the posterior is Beta(a + s, b + f). That is the entire calculation. The Beta distribution is conjugate to the binomial likelihood: the family is closed under updating. The arithmetic is transparent enough to reason with directly. A Beta(1, 1) prior is flat on [0, 1] — every accuracy rate equally plausible. Observe 7 successes and 3 failures and the posterior is Beta(8, 4), with mean 8/12 = 0.667 and a 95% credible interval of roughly [0.39, 0.89]. A Beta(10, 10) prior, by contrast, says you are fairly confident accuracy is near 0.5 — it has the weight of twenty prior observations, ten of each. Add the same data and you get Beta(17, 13), mean 0.567, interval roughly [0.39, 0.74]. The parameters a and b behave like pseudo-counts: prior successes and prior failures. This gives you an immediately interpretable handle on prior strength. A Beta(2, 2) prior is worth two observations and will be swamped by any real dataset. A Beta(50, 50) prior is worth a hundred, and if your experiment has forty trials, the prior is doing more work than the data. Being able to state your prior's influence in units of data is a useful discipline, and where an analogous statement is available for other models you should make it. Conjugate families exist for a handful of combinations — normal-normal, gamma-Poisson, Dirichlet-multinomial — and they are pedagogically valuable. They are also almost useless for real psychology, because the moment you have two groups, a covariate, participant-level variation and a non-normal outcome, no conjugate form exists. That is precisely the wall that stopped Bayesian methods from spreading for two centuries, and it is what sampling gets you over. Sequential updating and the irrelevance of the stopping rule Return to the associativity property, because its practical consequences are large. Imagine testing participants one at a time in a perceptual learning study and recomputing the posterior after each. The posterior after participant 30 is identical whether you computed it in thirty steps or one. It is also identical whether you had planned to stop at 30, planned to stop at 100 and were interrupted, or decided to stop because the interval had got narrow enough. Nothing in the calculation refers to your intentions, because nothing in p(y | θ) p(θ) refers to them. This is the likelihood principle in action: all the evidence in the data about θ is contained in the likelihood function. Frequentist inference violates it — necessarily, because error rates are properties of procedures — and Bayesian inference respects it. The practical upshot for experimental work is substantial. You can run until your credible interval reaches a target width, which is a far more sensible stopping rule than running until a threshold is crossed, and requires no sequential-testing corrections. You can add participants after a first look without penalty. You can pool across a series of studies by using one study's posterior as the next one's prior, which is what a meta-analysis should be. Two cautions. Deciding to stop when the posterior looks good, and then publishing only that, produces a biased literature exactly as selective publication of significant results does. The framework protects the coherence of your inference, not the field's. And a posterior that has been watched continuously will wander; if you stop at the first moment it crosses a threshold you have picked a high-water mark, and reporting it as if it were a fixed-n result overstates your case. Report the stopping rule anyway, not because the mathematics needs it but because the reader does. What the likelihood is, in practice Building the likelihood is where most of the modelling work lives, and it is the same work you already do when you choose a statistical test — just made explicit. When you run a t-test, you have implicitly asserted that observations within each group are independent draws from a normal distribution with a common variance. That assertion is the likelihood. When you run a repeated-measures ANOVA, you have asserted a rather elaborate structure about participant effects and sphericity. When you run logistic regression, you have asserted a Bernoulli likelihood with a logit link. The Bayesian workflow does not add an assumption here; it removes the pretence that the assumption is a technicality. Writing yi ~ Normal(μgroup[i], σ) in a Stan model makes you say out loud what the t-test assumed silently. And because you are writing it out, you can change it. If reaction times are right-skewed, you can write a lognormal or shifted-lognormal likelihood instead of pretending normality and log-transforming. If accuracy is bounded, you can use a binomial. If Likert responses are ordinal, you can use an ordered-logit likelihood rather than treating "agree" as one unit more than "neutral". None of these requires a new test to exist; you just write down what you think generated the data. This is the deepest practical change the Bayesian approach brings to experimental psychology, and it is not really about Bayes at all. It is that model-building replaces test-selection. The question stops being "which test applies to this design?" and becomes "what process plausibly produced these numbers?" A vocabulary for the rest of the book A few terms will recur, and it is worth fixing them now. The prior predictive distribution is what your model, before seeing data, thinks the data will look like. You get it by drawing a parameter value from the prior, generating a fake dataset from the likelihood at that value, and repeating. If your prior predictive generates reaction times of negative eight seconds or d-prime values of 40, your priors are wrong in a way you can see without any data. Chapter 3 makes this a routine step. The posterior predictive distribution is the same thing after updating: draw θ from the posterior, generate fake data, repeat. Comparing those simulated datasets with the real one is the main model-checking tool in Bayesian practice, and Chapter 7 uses it heavily. The logic is simple — if your fitted model cannot generate data that resemble what you observed, the model is wrong regardless of how clean the posterior looks. Marginalisation means integrating out parameters you are not interested in. If you fit a model with a mean difference, a residual standard deviation and twenty participant offsets, and you want to talk about the mean difference alone, you look at its marginal posterior — the distribution of that one parameter with everything else averaged over. With samples this is trivially easy: you just ignore the columns you do not care about. In frequentist practice, dealing with nuisance parameters requires specific machinery for each case. In Bayesian practice it is free, and this is one of the underrated practical advantages. The problem that remains You now have the whole logic. Prior times likelihood, renormalised, read off whatever summary you want. For the five-point example, arithmetic sufficed. For the Beta-binomial, a formula sufficed. For a hierarchical model of trial-level reaction times across 48 participants and 3 conditions, with participant-specific intercepts and slopes and a correlation between them, the posterior is a distribution over something like a hundred parameters, and computing its normalising constant means integrating a complicated function over a hundred-dimensional space. There is no formula. Numerical integration on a grid fails catastrophically — a grid with ten points per dimension in a hundred dimensions has 10¹⁰⁰ cells, and essentially all of them contain negligible probability. The solution is not to compute the posterior but to sample from it: to generate a large collection of parameter vectors whose frequency distribution matches the posterior, and then answer every question by counting. Want the posterior mean? Average the samples. Want a 95% credible interval? Take the 2.5th and 97.5th percentiles. Want the probability the effect exceeds 0.2? Count the fraction of samples above 0.2. How one generates such samples for a distribution one cannot even evaluate the normalising constant of is the subject of Chapter 5. First, though, the input that makes the whole thing possible and that attracts all the objections: the prior. An aside on what "probability" is allowed to mean One more conceptual point, because it is the hinge on which every objection to this framework turns, and it is usually argued badly on both sides. The frequentist definition of probability is a limiting relative frequency in a reference class of repeatable events. It is a good definition for coin flips and manufacturing defects. It is a poor one for most scientific questions, because the events are not repeatable and the reference class is not defined. What is the reference class for "the effect of this particular manipulation on this particular population"? There is only one such quantity and it does not have a frequency. The Bayesian definition treats probability as a measure of uncertainty that obeys the axioms of probability theory. Its justification is not that it feels intuitive but that it is forced. Several independent arguments — Cox's derivation from requirements of logical consistency, de Finetti's Dutch book argument showing that anyone whose degrees of belief violate the probability axioms will accept a set of bets guaranteeing a loss, and Savage's axiomatisation of rational decision under uncertainty — all conclude that coherent reasoning under uncertainty must be probabilistic. They start from different premises and arrive at the same place. That is a stronger foundation than most applied researchers realise. The claim is not that Bayesian reasoning is a reasonable option among several. It is that if you want your uncertainty judgements to be mutually consistent, you have no choice about the rules; the only freedom is in the inputs. None of which means a Bayesian analysis of a particular dataset is correct. Coherence is a property of the reasoning, not a guarantee about the model. You can be perfectly coherent and badly wrong, if your likelihood misdescribes the process or your prior is confidently mistaken. Coherence rules out contradiction; it does not rule out error, and the checking machinery in later chapters exists precisely because it does not. Hashtags: #BayesianDataAnalysisForExperimentalPsychologists #BayesianDataAnalysis #BayesianInference #ExperimentalPsychology #Priors #Likelihood #PosteriorDistribution #BayesianEstimation #CredibleIntervals #PosteriorProbability #BayesTheorem #PriorPredictiveChecks #PosteriorPredictiveChecks #MarkovChainMonteCarlo #MCMC #HamiltonianMonteCarlo #NoUTurnSampler #Stan #BRMS #HierarchicalModels #BayesianModelComparison #BayesFactors #SensitivityAnalysis #SequentialUpdating #FutureOfBayesianPsychology

Download the Book (PDF): Introduction In the spring of 2025, a group at Imperial College London had spent the better part of a decade working out how a family of mobile genetic elements in bacteria manages to spread between species. The elements — capsid-forming phage-inducible chromosomal islands — hijack the machinery of bacterial viruses to package themselves and travel. The puzzle was how they cross the species barrier, since a virus that infects one species of bacterium usually cannot infect another. The group had an answer, arrived at through years of experiment, and it was unpublished. José Penadés gave the problem, in the form of a short research goal, to a multi-agent system built on Google's Gemini models. Within two days it returned a ranked list of hypotheses. Its top-ranked proposal was the answer his group had spent years reaching: that the islands hijack tails from other phages, mixing and matching capsid and tail to change host range. Nothing in the published literature stated this. The system had assembled it from scattered pieces. That story, which was reported widely and later published in Cell and in a Nature paper describing the system, is the strongest single anecdote the field of automated hypothesis generation possesses. It is also, read carefully, an anecdote about a hypothesis that a human team had already generated, tested, and confirmed. The machine did not find something no one knew. It found something one laboratory knew and had not yet told anyone. That is a real and useful capability. It is not the capability the headlines described. This booklet is about the distance between those two readings, and about what it takes to close it. Generation is not the bottleneck The first thing to understand about automated hypothesis generation is that generating hypotheses is easy. It has always been easy. A hypothesis is a proposition about the world that could be false; producing propositions that could be false is a task at which a moderately capable language model is essentially unbounded. Ask one for a hundred reasons a particular protein might be implicated in a particular disease and it will give you a hundred, most of them grammatical, many of them mechanistically coherent, several of them consistent with everything in the literature, and — this is the difficulty — a small and unknown number of them true. The same is true of the graph-based methods that predate the current wave. Build a network in which nodes are genes, drugs, diseases, and phenotypes, and edges are relationships extracted from databases and papers, then train a model to predict which edges are missing. It will predict millions. Every one of those predictions is a hypothesis. Most are wrong. Some are trivially right in ways that were already known but absent from your particular graph. A few are the ones you want. So the interesting quantity in this field is never the number of hypotheses a system produces. It is the precision of the system's ranking — its ability to put the few worth testing near the top — and, behind that, the honesty of its estimate of its own precision. A generator without a critic is a random number generator with better grammar. This is the controlling claim of what follows: the entire value of an automated discovery system lies in how well it prices its own proposals against reality. Every genuine advance in this field over forty years has come from tightening the loop between a proposer and something that can say no — prior art, mechanism, statistics, or an experiment. Every disappointment has come from improving the proposer and calling it progress. The claim sounds deflationary. It is not meant to be. Pricing proposals well is a hard, deep, and largely unsolved technical problem, and the systems that have made real headway on it are genuinely impressive. But it reframes what the field is doing. The question is not "can a machine have an original idea?" — a question that turns out to be both unanswerable and uninteresting. The question is "can a machine tell a good idea from a plausible one?", and that question has measurable answers, some of which are encouraging and some of which are not. What the field actually contains Three technical lineages have converged on this problem, and they are more different from one another than the common vocabulary suggests. The oldest is literature-based discovery, which begins with Don Swanson's observation in the 1980s that the scientific literature contains propositions nobody has ever put next to each other. If one body of work establishes that A affects B, and an unconnected body of work establishes that B affects C, then A may affect C — and if no paper anywhere has said so, that is a hypothesis sitting in public, undiscovered. Swanson's method was almost embarrassingly simple and it worked at least twice. Its weakness was equally simple: the number of A-B-C chains in a large literature is astronomical, and Swanson had no principled way to rank them. The second lineage is representation learning from text. If you train a model to predict which words occur near which other words across millions of papers, the geometry of the resulting space encodes something about the concepts those words denote. A 2019 Nature paper showed that word embeddings trained on materials science abstracts, with no supervision and no chemical knowledge built in, placed compounds near the word "thermoelectric" years before those compounds were reported as thermoelectrics. The literature contained knowledge its authors had not stated. Large language models are the enormous descendants of this idea, and they inherit both its power and its central limitation: they learn the structure of what has been written, which is not the same as the structure of what is true. The third is graph learning. Encode biomedical or chemical knowledge as a network and use a graph neural network to reason over its topology: a drug and a disease that are connected by many short, semantically coherent paths are candidates for a direct relationship. This lineage has produced the most legible successes, because a path through a graph is an explanation, and an explanation is something a domain expert can reject on sight. What has happened since roughly 2023 is that these lineages have been stitched together inside agent architectures: a language model proposes, a second instance criticises, a retrieval system checks prior art, a graph or a simulator supplies structure, a tournament ranks survivors, and the whole thing iterates. The systems described in this booklet — Co-Scientist, Robin, Kosmos, Coscientist, ChemCrow, and their relatives — are all variations on that pattern. Their architectural differences matter less than the thing they have in common, which is that every one of them is an attempt to build a critic good enough to make a fluent generator useful. Three domains, three different hard parts The booklet moves through biomedicine, chemistry, and materials science, because the hypothesis-pricing problem takes a different shape in each. In biomedicine, the bottleneck is causal identifiability. There is an enormous amount of data and a rich literature, so generating mechanistically coherent proposals is cheap; determining which of them describe causation rather than correlation is expensive and slow, and the validation experiments cost months. In chemistry, the bottleneck is synthesizability. A generative model can propose a molecule with any property profile you specify. Whether anyone can actually make it, at what cost, in how many steps, from what starting materials, is a separate and much harder prediction — and a molecule nobody can synthesise is not a hypothesis, it is a picture. In materials, the bottleneck is novelty verification, and this is where the field has embarrassed itself most publicly. When DeepMind announced 2.2 million new crystal structures and a Berkeley group announced forty-one autonomously synthesised new compounds, specialists went through the claims and found that a large fraction were not new: they were ordered idealisations of compounds already known in disordered form, or duplicates already sitting in structural databases. A 2026 analysis found the same problem in Microsoft's MatterGen, whose flagship "novel" compound turned out to have been reported in 1971 and to be present in the model's own training set. The lesson is not that these systems are worthless. It is that knowing whether something is new is a research problem in its own right, and one that generative models are constitutionally bad at. What you should expect to take away By the end, you should be able to look at any announcement of an AI-discovered hypothesis and ask the three questions that determine whether it means anything: Against what was novelty checked? What said no, and how often? Was the hypothesis executed, or only scored? That last question has become the sharpest instrument in the field. In 2024 a Stanford group ran a large study in which more than a hundred researchers blindly reviewed research ideas, some written by experts and some by a language model, and found the machine's ideas rated more novel. In 2025 the same group had forty-three researchers actually implement a subset of those ideas — over a hundred hours each — and re-reviewed the resulting papers blind. The machine's ideas fell further than the humans' on every metric, and the ranking reversed. The gap between an idea that reads well and an idea that survives contact with an experiment is the central empirical finding of this field so far, and no amount of additional fluency closes it. What closes it is better critics, cheaper experiments, and honest accounting. Those are the three things worth building, and the chapters that follow are largely about how far each has got. Chapter 1: Undiscovered Public Knowledge Don Swanson was a librarian by profession and a physicist by training, and he spent the last third of his career on a single idea that almost nobody in either field wanted. The idea was that the scientific literature contains facts nobody knows. Not facts nobody has written down — facts that are written down, in public, in indexed journals, and that nonetheless no human being has ever assembled into a single thought. Swanson called this undiscovered public knowledge, and in a 1986 paper he gave it a precise structure. Suppose literature A establishes that some intervention affects some intermediate quantity. Suppose literature C establishes that the same intermediate quantity affects some outcome. Suppose, further, that the two literatures do not cite each other, share no authors, appear in journals read by different communities, and use partly different vocabulary. Then the inference from the intervention to the outcome exists in public knowledge, is entailed by published results, and has never been stated by anyone. This is not a claim about clever reasoning. The inference is one step long. It is a claim about the social structure of science: that specialisation has fragmented the literature into communities that cannot see each other, and that the fragmentation itself creates a systematic and harvestable class of missed connections. Fish oil and Raynaud's Swanson's demonstration case has been retold often enough to have become slightly mythologised, so it is worth stating what actually happened. He was interested in Raynaud's syndrome, a circulatory disorder in which blood flow to the extremities is episodically restricted. Reading the clinical literature, he extracted a set of physiological factors associated with the condition: elevated blood viscosity, high platelet aggregability, and vasoconstriction. He then searched, separately, for literatures about anything that reduced those quantities. Dietary fish oil — specifically eicosapentaenoic acid — came up repeatedly. Papers in nutrition and lipid biochemistry reported that fish oil lowered blood viscosity, reduced platelet aggregation, and had vasodilatory effects. Those papers were not about Raynaud's. The Raynaud's papers were not about fish oil. Swanson checked: in 1986, no publication linked the two. He published the hypothesis, in a medical journal, explicitly as a hypothesis generated by a literature method rather than by laboratory work. Three years later a clinical trial reported that fish oil supplementation improved symptoms in patients with primary Raynaud's. Swanson repeated the trick with magnesium and migraine, again finding a large, scattered set of published results that jointly implied a deficiency relationship nobody had asserted, and again the connection was subsequently supported. Two successes is not a track record. But the two successes were enough to establish that the phenomenon was real — that the fragmentation of the literature genuinely hides inferences — and they created a small research programme, now generally called literature-based discovery, which has run continuously ever since. The ABC model and its arithmetic problem Swanson's structure was formalised as the ABC model. A is a starting concept, C is a target concept, and B is any intermediate that links them. In closed discovery you fix A and C and search for plausible Bs that would explain a suspected relationship — this is essentially mechanism-hunting. In open discovery you fix only A and let the method propose Cs. Open discovery is where the hypotheses come from, and it is where the trouble starts. The trouble is arithmetic. Consider a biomedical vocabulary of roughly a hundred thousand concepts — genes, proteins, drugs, diseases, phenotypes, chemicals, processes. A concept of any importance co-occurs in the literature with hundreds or thousands of others. Fix an A and enumerate its Bs: you have, say, five hundred. Enumerate each B's partners: another few hundred each. The candidate set of Cs is now in the hundreds of thousands, with many reachable by multiple paths. Filter out the ones where A and C already co-occur, and you are still left with tens of thousands of "novel" A-C pairs, every one of which is, formally, undiscovered public knowledge. Swanson knew this. His actual method involved a great deal of manual reading and domain judgement; the automated part narrowed the field, and Swanson's own expertise did the rest. What the field inherited was the narrowing, and for two decades it tried to automate the judgement with progressively more elaborate ranking statistics. Those statistics were mostly variations on the same theme: a measure of how surprising a co-occurrence is relative to chance. Mutual information. Log-likelihood ratios. Term frequency weighting borrowed from information retrieval. Later, semantic filtering using the relationship types extracted by systems like SemRep, which parses biomedical text into subject-predicate-object triples, so that you could require B to be, say, inhibited by A and causally associated with C rather than merely mentioned in the same abstract. Each refinement helped. None solved the problem, and the reason is instructive. All of these methods rank candidates by how strongly the literature supports the connection. But the literature's support for a connection is, roughly, a measure of how obvious it is. The best-supported novel connections tend to be the ones a competent domain expert would have produced in ten seconds of thought, and the genuinely surprising ones sit in the long tail alongside an enormous quantity of noise. Ranking by literature support is therefore self-defeating at exactly the point where you want it to work. A worked chain, and what goes wrong in it It helps to walk through a single chain in the detail an actual analyst would. Take A to be a well-studied kinase inhibitor. The literature reports, across perhaps two hundred papers, that it suppresses the activity of a particular signalling pathway in several cell types. That is the A-B link, and it is solid: multiple groups, multiple assays, dose-response curves, a structural rationale. Now take B, the pathway, and find its downstream literature. Here the picture is messier. Several hundred papers report associations between that pathway's activity and a range of outcomes: fibrotic remodelling in one organ, immune cell differentiation, a class of neurodegenerative phenotypes. Pick the fibrosis literature as C. The B-C link is supported, but it is supported by a different kind of evidence — mostly genetic knockouts in mice and expression correlations in human tissue, not pharmacological intervention. The ABC inference is now available: the inhibitor may reduce fibrosis. Check the literature: nobody has said so. You have a hypothesis. Four things can be wrong with it, and only one of them is visible from the text. The first is scope mismatch. The A-B evidence may come entirely from cancer cell lines at micromolar concentrations while the B-C evidence comes from hepatic stellate cells at physiological signalling levels. The pathway is the same word in both literatures and a different biological object. The second is directionality collapse. The relationship extracted from the B-C literature may be "associated with" rather than "causes", and the association may run the other way — fibrosis driving pathway activity rather than the reverse. Automated relation extraction handles this poorly, because scientific writing hedges, and hedged causal language is precisely what parsers flatten. The third is pharmacological impossibility. The inhibitor may not reach the relevant tissue at the relevant concentration, may be metabolised too fast, or may hit a second target whose effect on fibrosis is the opposite and larger. None of this is in the A-B or B-C literature because neither literature was asking. The fourth is silent prior art. Somebody may have tried it. The result may be in a thesis, a conference abstract, a failed trial that was never written up, or a patent. The negative search that made the hypothesis interesting covers indexed peer-reviewed literature and essentially nothing else, and the parts of science that do not get indexed are disproportionately the parts where something did not work. Only the fourth is in principle fixable by better search. The first three require knowledge of the world that the text does not contain in retrievable form. A human expert supplies that knowledge implicitly, which is why Swanson's method worked in Swanson's hands and degraded when it was automated end to end. Two decades of refinement Between Swanson's papers and the arrival of neural methods, literature-based discovery developed into a small but serious subfield, and its history is a catalogue of attempts to automate exactly the judgement described above. The first generation was co-occurrence-based and vocabulary-driven. Arrowsmith, developed by Swanson with Neil Smalheiser, took two literatures and computed the list of title words they shared, which the user then triaged. It was deliberately a tool for a reader rather than an oracle, and it remained in academic use for years. The second generation added semantic structure. The National Library of Medicine's SemRep system parses biomedical sentences into predications — subject, relation, object — using the UMLS Metathesaurus for concept normalisation and a hand-built relation ontology. Instead of "these terms appear together", you could now require "X INHIBITS Y" and "Y CAUSES Z", which removes a great deal of chaff. Systems like BITOLA and, later, the SemMedDB-based discovery tools of Trevor Cohen and colleagues, built ranking schemes over these predication graphs. Cohen's work in particular introduced distributional methods — vector representations of predications that allowed reasoning by analogy rather than exact pattern matching — and is the clearest ancestor of the embedding-based work described in the next chapter. The third generation, arriving in the 2010s, added network science. Treat the predication graph as a network and the discovery problem becomes link prediction: score candidate edges by topological features such as the number of short paths, the degree of the intermediates, or the similarity of the endpoints' neighbourhoods. This was a genuine improvement, because it replaced one-step ABC reasoning with something that could weigh multiple converging routes, and it set up the graph neural network methods that now dominate biomedical hypothesis generation. Across all three generations, the same evaluation pathology persisted and deepened. Because there were only a few canonical discoveries to replicate, the field developed benchmarks in which a system is asked to predict, from a literature snapshot taken at year T, relationships that were first published in years after T. These time-sliced benchmarks are the right idea, and they exposed something uncomfortable: the relationships that get published next are, overwhelmingly, the ones that were easiest to guess from the state of the field at time T. A system that scores well on a time-sliced benchmark has demonstrated that it can predict the direction of ordinary scientific progress. Whether it can predict anything else is not tested, and arguably cannot be tested by any benchmark constructed from what was actually published. Why this matters for everything that came after It is tempting to treat Swanson as prehistory — an ingenious manual method, superseded by machine learning. That reading misses the point, because every problem the modern systems face was present in the ABC model in miniature, and in some cases the modern systems are worse off. The combinatorial explosion did not go away; it got larger. A language model's hypothesis space is not bounded by co-occurrence at all. It can propose relationships between concepts that share no literature whatsoever, which is both the source of its apparent creativity and the reason its precision is difficult to characterise. Swanson could at least enumerate his candidate set and say how big it was. Novelty checking was always the load-bearing step. The entire force of Swanson's fish oil claim rested on the negative search: no paper connects these. That search was over a well-curated index, MEDLINE, using controlled vocabulary, and it was still hard to do correctly, because a connection can be stated in language the index does not capture. Modern systems perform the same negative search over a vastly larger and messier corpus, frequently by asking a language model whether it has seen something before — which is not a search, it is a recollection, and an unreliable one. Plausibility and novelty pull in opposite directions. This tension, visible in the ABC ranking statistics, is now the central design problem of every generate-and-rank architecture. A system tuned for plausibility returns the obvious. A system tuned for novelty returns the unhinged. The useful region is narrow and there is no principled way to find it from text alone, because text does not contain the information needed to distinguish "surprising and true" from "surprising and false." Only the world contains that. Evaluation was already broken. How did the field assess literature-based discovery systems? Overwhelmingly by replication of known discoveries: take the literature as it stood before 1986, run your system, and see whether it surfaces fish oil for Raynaud's. This is a reasonable sanity check and a terrible benchmark, because there are only a handful of such cases, everyone tunes against the same ones, and a method that recovers Swanson's discoveries has demonstrated that it recovers Swanson's discoveries. The modern version of this error is evaluating a language model on whether it can propose a finding published after its training cutoff — better, but vulnerable to contamination, and still a test of recall rather than of precision. What the early systems did get right Three things, all of which the current generation has partly forgotten. First, they were explicit about their inference. An ABC chain is a stated argument: here is the intermediate, here are the papers on each side, here is why the conclusion follows. A domain expert can read it and identify the flaw in thirty seconds — that the two literatures use the same word for different things, that the effect sizes are incompatible, that the B-C relationship only holds in a cell type irrelevant to the A-B one. Modern generative systems often produce a fluent paragraph of justification that is not the actual basis of the proposal, and cannot be checked in the same way. Second, they were honest about the human in the loop. Swanson never claimed the computer had made a discovery. He claimed the computer had narrowed a search space and that he had then done the reading. The most productive current systems operate the same way and describe themselves less clearly. Third, they took the negative case seriously. Arrowsmith, the tool Swanson and Neil Smalheiser eventually built and made publicly available, was designed around the constraint of not already known, with the interface organised to make the researcher confront the prior-art question directly rather than to produce impressive-looking output. The shape of the problem, restated Strip away the specifics and literature-based discovery poses the problem that this booklet is about, in its simplest form. You have a corpus. The corpus entails more than it states. You want the entailments that are (i) not already stated, (ii) true, and (iii) worth knowing. Property (i) is a search problem over the corpus, and it is hard but tractable. Property (iii) is a value judgement that depends on who is asking and why, and no system has made much progress on it. Property (ii) is the one that cannot be obtained from the corpus at all. That last point deserves emphasis because it is the permanent constraint on this entire enterprise. A corpus of text is a record of what people have claimed. Its statistical structure reflects, in varying proportions: the structure of the world, the structure of what was funded, the structure of what was publishable, the structure of what was fashionable, and the accumulated errors of everyone who wrote it down. A model trained on that corpus learns all five at once and has no internal means of separating them. When such a model proposes that drug X might treat disease Y, the proposal's grounding is that the literature's geometry makes the relationship look like the relationships that are usually reported — and the literature reports positive results far more often than negative ones, reports results from well-funded areas far more often than neglected ones, and contains a substantial minority of findings that do not replicate. Nothing in the last forty years has changed this. What has changed is the sophistication of the machinery for working within it, and the emergence of something Swanson never had: the ability to attach a generator to an experiment and let the world do the arbitration. That is the thread the rest of the booklet follows. But the first move is to understand what the generators are actually doing when they read, which is the subject of the next chapter. The answer is stranger, and more limited, than the phrase "the model understands the literature" suggests. Chapter 2: Meaning Without Supervision The word "unsupervised" does a great deal of work in this field and it is worth being precise about what it means, because the precision is where the interesting constraints live. A supervised model learns a mapping from inputs to labels that somebody provided. Show it ten thousand molecules annotated with their measured solubility and it learns to predict solubility. Its knowledge is bounded by the annotation: it knows what the labellers knew, and it cannot tell you about a property nobody measured. An unsupervised model learns structure in data without labels. In the case that concerns us, the data is text, and the structure is co-occurrence: which words appear near which other words, in what contexts, with what frequency. Nobody tells the model what a thermoelectric is. Nobody tells it that lithium is an alkali metal. It infers a geometry from usage, and the geometry turns out to encode a surprising amount of the semantics. The reason this matters for hypothesis generation is specific. A supervised model can only propose things within the schema of its labels. An unsupervised model trained on a whole literature has, in principle, absorbed everything that literature's usage patterns encode — including regularities that no author ever stated, because the regularity is in the aggregate rather than in any single document. That is the modern version of Swanson's insight, and it is the mechanism by which a model can "know" something that is not written down. The materials science demonstration The cleanest evidence that this works came from a 2019 Nature paper by Vahe Tshitoyan and colleagues at Berkeley, which remains the reference experiment for the whole idea. They trained a word2vec model — a simple, by 2019 standards already old, method that learns a few-hundred-dimensional vector for each word by predicting its neighbours — on roughly 3.3 million abstracts from the materials science literature. No chemical knowledge was built in. The tokenisation handled chemical formulae, and that was essentially the only domain engineering. The resulting space behaved as if it contained chemistry. Vector arithmetic on element names recovered relationships from the periodic table. Compounds clustered by application. Most importantly, the researchers ran a retrospective test: they trained the model on abstracts up to a given year, then ranked all known compounds by the cosine similarity of their vectors to the word "thermoelectric", and checked which of the top-ranked compounds were subsequently reported as thermoelectric materials in later years. Several of the top predictions were later confirmed in the literature — compounds the model had placed near the concept before anyone published them as such. The paper's framing was careful and correct: the literature contained latent knowledge, and an unsupervised model surfaced it. The mechanism is not mysterious. Materials with similar electronic structure, similar constituent elements, and similar synthesis routes get written about in similar linguistic contexts. A compound that has never been called a thermoelectric but that is discussed in the way thermoelectrics are discussed will sit near the word. Some of those compounds are thermoelectrics that nobody has tested. Two caveats attach to this result and both are important. The first is that the confirmation criterion was publication, not measurement: a prediction "came true" when someone later published the compound as a thermoelectric. That is a real test — the model predicted the literature's future — but it is a test of anticipating the field, which is not quite the same as anticipating nature. The second is reproducibility. A 2023 analysis of reproducibility in materials science NLP found that re-running this class of experiment is harder than the published descriptions suggest, with results sensitive to corpus composition, tokenisation choices, and hyperparameters that papers do not always report. The finding holds; its effect size is less firmly pinned down than the citation count implies. From static vectors to contextual ones Word2vec assigns one vector per word, which is a problem in science, where the same string means different things in different subfields. "Expression" is a genetics term and a mathematics term. "Resolution" is a microscopy parameter and a chemical separation. "Ni" is nickel and an abbreviation for something else entirely in a clinical paper. Contextual models solved this. BERT and its descendants compute a representation for each token given its sentence, so the same word gets different vectors in different contexts. Domain-adapted variants followed quickly: SciBERT, trained on a large corpus of scientific papers; BioBERT and PubMedBERT for biomedicine; MatBERT and similar for materials. These models are the workhorses of scientific information extraction, and they are what turns a corpus of prose into a structured resource: named entity recognition to find the genes and compounds, relation extraction to find the claims connecting them, coreference resolution to stitch the mentions together. This is the unglamorous plumbing of automated hypothesis generation and it deserves more credit than it gets. The knowledge graphs described in the next chapter are built largely by these extractors. Their error rates propagate into everything downstream. A relation extractor operating at ninety per cent precision — which is respectable — introduces one wrong edge in ten, and a graph reasoning model has no way to distinguish a wrong edge from a surprising one. Some of the most confident, most novel-looking predictions a graph model produces are artefacts of extraction errors, and tracing them back is tedious work that most pipelines do not do. The geometry, examined It is worth being sceptical about what the embedding geometry actually encodes, because the popular account overstates it. The famous demonstrations are analogies: the vector for "king" minus "man" plus "woman" lands near "queen"; in the materials case, arithmetic on element vectors recovers periodic-table relationships. These are real, but subsequent analysis of analogy benchmarks in general-purpose embeddings showed that much of the apparent reasoning is an artefact of how the nearest-neighbour search is conducted — in particular, the standard procedure excludes the input terms from the answer set, and without that exclusion a large share of analogies return one of their own inputs. The geometry encodes similarity robustly and relational structure much more weakly. For hypothesis generation the practical consequence is that an embedding-based system is best understood as a recommender. It is doing what a retail recommender does: users who bought X also bought Y, therefore a user who bought X may want Y. Substitute papers for users and concepts for products and the analogy is exact. Recommenders are extremely useful and they have a characteristic failure mode — they recommend the popular. A compound discussed in ten thousand abstracts has a well-estimated vector; a compound discussed in six has a noisy one that sits wherever its few contexts happened to put it. The system's confidence is therefore highest exactly where the literature is densest, which is exactly where the remaining discoveries are fewest. This has a measurable signature. If you take an embedding-based discovery system and plot its hit rate against the literature density of the entities involved, the curve slopes the wrong way for a discovery tool: it does best on well-studied things. Several groups have proposed correcting for this by explicitly down-weighting popularity, or by sampling candidates from sparse regions of the graph, and this helps — but it trades precision for coverage, and the resulting candidate lists are much harder to triage. Extraction: turning prose into claims Between raw text and any structured reasoning sits a pipeline that most descriptions skip, and its properties determine a great deal. The first step is entity recognition and normalisation. A paper mentions "TNF-α", "tumour necrosis factor alpha", and "TNF"; all three must be mapped to one identifier, and must not be mapped to the gene of the same abbreviation in a different organism. This is done with a combination of dictionary lookup against controlled vocabularies — UMLS, MeSH, ChEBI, the Gene Ontology — and neural taggers fine-tuned on annotated corpora. Performance is good in well-curated domains and markedly worse for chemicals, materials, and anything with productive naming conventions, where new strings appear constantly. The second step is relation extraction: deciding that a sentence asserts a relationship, of what type, between which entities, and with what polarity and certainty. This is where most of the information loss happens. Scientific prose is full of constructions that a classifier flattens: "may contribute to", "was associated with in a subset of patients", "did not reach significance but trended toward". A pipeline that records these as plain assertions manufactures a literature far more confident than the real one. The third step is aggregation. Twenty papers assert a relationship, three deny it, and the graph stores one edge. What weight should that edge carry? Practice varies from simple counts to elaborate confidence models, and the choice propagates into every downstream prediction. A model reasoning over a graph in which contested and settled facts look identical will treat them identically. None of this is exotic engineering, but it explains a persistent observation: two research groups building hypothesis generators over the same literature, with comparable models, routinely produce non-overlapping candidate lists. The divergence usually comes from the extraction layer rather than the reasoning layer. What large language models changed, and what they did not Generative language models are continuous with this lineage in their training objective — predict the next token — and discontinuous in almost every practical respect. Three differences matter for hypothesis generation. They compose. A word embedding can tell you that two concepts are close. A generative model can produce a paragraph explaining why they might be related, invoking a mechanism, naming intermediate steps, anticipating an objection. Whether the explanation is the true basis of the model's output is a separate and largely unanswered question, but the explanation is legible to a domain expert in a way a cosine similarity is not, and legibility is what allows a human to reject it quickly. They follow instructions. You can constrain the hypothesis space by specification: propose only mechanisms involving a named pathway, only compounds synthesisable in under five steps, only interventions testable in a specific model organism. This is a genuinely large practical gain, because most of the combinatorial explosion in earlier methods came from candidates that were irrelevant to what the researcher actually wanted. They use tools. A model that can call a search engine, a structural database, a docking program, or a Python interpreter is no longer reasoning purely from its own weights. This is the single most important development for the pricing problem, because tools are how a system gets access to something outside the literature's statistics. What did not change is the epistemic status of what the model learned from text. Three limits recur throughout the rest of this booklet. The literature is a biased sample. Positive results are published; most negative ones are not. Well-funded topics generate orders of magnitude more text than neglected ones. A model trained on this corpus learns that the well-studied relationships are the important ones, which is partly true and partly an artefact of funding history. Ask it for hypotheses about a rare disease and it will reach for mechanisms borrowed from common ones, because that is where the density is. The literature contains errors at a meaningful rate. Large replication studies in psychology and in preclinical biology have found that a substantial fraction of published findings do not reproduce, with the most-cited estimates in cancer biology and social psychology landing well below fifty per cent replication for the samples examined. A model has no signal distinguishing the robust findings from the fragile ones except citation-weighted repetition, which correlates with robustness weakly and with attention strongly. Fluency is not calibration. A model produces its hundredth-ranked hypothesis in the same confident register as its first. Extracting a usable probability from a generative model remains difficult; verbalised confidence scores are poorly calibrated, and token-level likelihoods measure the probability of the wording, not of the claim. Every practical system therefore builds its ranking machinery outside the generator, which is the subject of Chapter 4. Recall is not search One further distinction runs through everything that follows, and it is easy to lose. When a language model answers a question about the literature from its weights, it is recalling. When it issues a query to an index and reads what comes back, it is searching. These feel similar from the outside and they have opposite error profiles. Recall degrades gracefully into confabulation. A model asked whether any paper has linked two concepts will produce an answer in the same register whether or not it has any relevant information, and the answer will be shaped by the plausibility of such a paper existing rather than by its existence. This is the mechanism behind fabricated citations, and it is not a bug that scale fixes: the model is doing what it was trained to do, which is to produce likely continuations. Search fails differently. It returns nothing when the query terms are wrong, returns irrelevancies when the index is poor, and — critically — it can be audited. You can look at what was retrieved. Retrieval-augmented architectures exist to move work from the first mode to the second, and the good scientific agents are aggressive about it. FutureHouse's literature agents were built around the principle that every claim should be traceable to a retrieved passage; Kosmos was designed so that each conclusion in its reports links to the specific lines of analysis code or the specific passages that produced it. This traceability is not a user-interface nicety. It is the only practical way to distinguish a system's genuine findings from its confabulations at the volume these systems operate at, and a system that cannot offer it cannot be checked at all. The corollary is uncomfortable for the field's marketing. The most important engineering in a modern hypothesis generator is usually not the model. It is the indexes, the citation graphs, the structured databases, and the discipline of forcing every assertion through them. Contamination: the problem that eats evaluations There is one more consequence of unsupervised training on the whole literature, and it undermines a large fraction of published evaluations in this field. The standard way to test a hypothesis generator is temporal: train or cut off the model at time T, ask it to propose relationships, and check how many were subsequently confirmed after T. If the model proposes a drug-disease pair that was validated in a trial reported in year T+3, that looks like a successful prediction. The difficulty is that "cut off at time T" is very hard to enforce for a model trained on a web-scale corpus. Documents are undated, republished, quoted, summarised, and aggregated. A review article from T+5 discussing the trial may appear in a scrape whose nominal date is earlier. Preprints predate publications by a year or more. Clinical trial registries announce what is being tested before the results exist. A model asked to predict a post-cutoff finding may simply be recalling a post-cutoff document. The field's better evaluations now go to considerable lengths to control this: constructing corpora with hard date filters, using held-out findings from unpublished laboratory work, or — the strongest design — running the proposed experiment. The Penadés case described in the introduction is valuable precisely because the answer was unpublished and therefore uncontaminated; the same is true of the Kosmos runs whose conclusions were checked against manuscripts in preparation. These are expensive evaluations, which is why there are so few of them, and why the literature is full of cheap ones whose numbers should be discounted heavily. The honest summary of what unsupervised text learning provides It provides a prior. A very good one, learned from more reading than any human could do, encoding real regularities about how chemistry, biology, and materials behave, and capable of surfacing associations no author stated. It does not provide truth, calibration, or novelty verification. Those have to come from elsewhere: from structured knowledge that can be checked, from tools that compute rather than recall, and ultimately from experiments. The next chapter deals with the first of those — the attempt to give machine reasoning something firmer than prose to stand on. Chapter 3: Science as a Graph There is a moment in most conversations about automated discovery when someone draws a network on a whiteboard. Circles for genes, drugs, diseases; lines between them; an arrow to a gap where a line should be. The gesture is so natural that it is worth asking why the graph representation is so persuasive, and what it costs. The answer to the first question is that a graph makes the inferential structure explicit. Text asserts relationships in a form that must be parsed; a graph stores them in a form that can be computed over. If a drug binds a protein, and that protein is part of a pathway, and that pathway is dysregulated in a disease, the chain is three hops and can be found by traversal. More importantly, the absence of an edge is representable. In a corpus of text, "no paper says X treats Y" requires an exhaustive negative search. In a graph, it is a missing edge, and predicting missing edges is a standard machine learning problem with standard machinery. The cost is that everything not in the schema disappears. A graph edge labelled "treats" carries no dose, no population, no effect size, no evidence quality, no context. Reducing a hedged, contested, context-dependent literature to typed edges is lossy in exactly the places where scientific judgement lives. The graph's clarity is purchased by throwing away the caveats. What the graphs contain The biomedical knowledge graphs in serious use are large, heterogeneous, and assembled from curated databases rather than primarily from text. Hetionet, built by Daniel Himmelstein and Sergio Baranzini and released in 2017, is the reference example for the open literature: roughly 47,000 nodes of eleven types and 2.25 million edges of twenty-four types, integrated from twenty-nine public resources. Its defining feature is that it is small enough to reason about and fully documented, which is why it became the standard testbed. Its companion project, Rephetio, used it for drug repurposing and, in a much-cited demonstration, predicted a set of indications that were subsequently supported. SPOKE, the Scalable Precision Medicine Open Knowledge Engine from the same lineage, scaled the idea to tens of millions of nodes and hundreds of millions of edges, and connected it to electronic health record data — which introduces observational patient evidence alongside curated biology. PrimeKG, from the Zitnik lab at Harvard, assembles a disease-centric graph designed specifically for precision medicine tasks. In chemistry and materials, the equivalents are structural and reaction databases — the Materials Project, the Inorganic Crystal Structure Database, Reaxys and its relatives — which are graphs in a looser sense but serve the same function of providing checkable structure. The distinction between database-derived and text-derived graphs matters enormously and is often elided. A database edge has provenance, curation, and usually a defined evidence code. A text-derived edge has whatever the extraction pipeline believed. Hybrid graphs contain both and frequently do not distinguish them downstream, with predictable consequences for how much weight a prediction deserves. How machines reason over them Three broad families of method operate on these structures, and they have genuinely different properties. Path-based methods enumerate the routes between two nodes and score them. Rephetio's approach — degree-weighted path counts over typed paths, or metapaths — is the canonical version: a path such as Compound–binds–Gene–participates–Pathway–associated with–Disease is a metapath, and a model learns how much evidence each metapath type provides, correcting for the fact that paths through high-degree hubs are uninformative. The output is directly interpretable: here are the three metapaths that drove this prediction, and here are the specific nodes on them. Embedding methods learn a vector for each node and each relation type such that true triples score higher than false ones. TransE, the simplest, models a relation as a translation in vector space, so that the head's vector plus the relation's vector should land near the tail's. Successors — DistMult, ComplEx, RotatE — use richer algebra to capture symmetry, antisymmetry, and composition. These scale well and are fast to query, and they produce no explanation whatsoever. Graph neural networks compute node representations by message passing: each node repeatedly aggregates information from its neighbours, so that after k rounds a node's representation reflects its k-hop neighbourhood. Relational variants such as R-GCN and CompGCN handle typed edges; attention-based variants weight neighbours differentially. For link prediction, the two endpoint representations are combined and scored. GNNs generally outperform the other two families on benchmark metrics, and their explanations must be reconstructed post hoc by methods such as GNNExplainer, which identify the subgraph most responsible for a prediction. A fourth family has emerged more recently: language-model-augmented graph reasoning, in which textual descriptions of nodes are encoded by a language model and used as initial features, or in which a language model is asked to evaluate candidate edges surfaced by a graph method. This combines the graph's structure with the model's prior, and it is where most current systems sit. Table 1 sets out how these families differ on the properties that matter for hypothesis generation. Table 1. Four families of graph-based hypothesis generation compared. Approach Basis of prediction Explanation offered Characteristic failure Path-based (metapaths) Weighted counts of typed routes between nodes Explicit paths and their node identities Misses relationships with no short route; hand-specified path types Knowledge graph embedding Geometry of learned node and relation vectors None intrinsic Popularity bias; scores reflect node degree as much as evidence Graph neural network Message passing over local neighbourhoods Post hoc subgraph attribution only Over-smoothing; sensitive to graph construction choices Language-model augmented Graph structure plus textual node semantics Fluent rationale, not necessarily the true basis Rationale and mechanism can diverge; contamination from pretraining Message passing, and what it does to a scientific claim It is worth spelling out the mechanics of a graph neural network, because the intuition people carry about them is usually wrong in a way that matters. Each node starts with a feature vector — sometimes learned from scratch, sometimes initialised from a text embedding of the entity's description. In one round of message passing, every node collects the vectors of its neighbours, transforms them by a learned weight matrix (a different matrix per relation type, in the relational variants), aggregates them by sum or mean or attention-weighted average, and combines the result with its own previous vector. After two rounds, a node's vector reflects its two-hop neighbourhood. After three, three hops. Two consequences follow that are rarely discussed in the application literature. The first is over-smoothing. Each round mixes a node with its neighbours, and the mixing is contractive. After enough rounds, every node in a connected component converges toward the same representation and the model loses the ability to distinguish them. In practice this caps useful depth at two or three layers, which caps the reasoning horizon at two or three hops. A mechanistic chain of five steps — drug, target, phosphorylation event, transcription factor, gene programme, phenotype — is beyond what a standard architecture can represent in a single prediction. The systems that appear to handle longer chains are usually doing so through the graph's own shortcut edges, which were put there by a curator who had already made the inference. The second is structural determinism. A GNN's prediction is a function of neighbourhood topology. Two drugs with identical neighbourhoods receive identical predictions regardless of their chemistry. This is fine when the graph encodes everything relevant and pathological when it does not — and the graph never encodes everything relevant. It is the formal reason these models are so sensitive to graph construction. Graph construction is, accordingly, where most of the real modelling decisions are made, and almost none of them are neutral. Is a gene one node or one node per isoform? Is a disease the clinical entity, the ontology term, or the molecular subtype? Do you include the "similar to" edges computed from sequence homology, which densify the graph enormously and inject a strong and possibly circular prior? Do you keep edges that came from a single low-powered study? Each choice changes the neighbourhoods, and changing the neighbourhoods changes every prediction. This is why two groups reasoning over "the same" biomedical knowledge produce different answers, and why benchmark leaderboards move when the underlying graph is rebuilt. The model is a small part of the system. The ontology is most of it. Explanation, in practice The explainability story for these methods deserves a harder look, because "interpretable" is doing a lot of promotional work. For metapath methods, the explanation is genuine: the model's score decomposes into contributions from specific typed paths, and you can list the highest-contributing ones with their intermediate nodes. A pharmacologist reading Compound–binds–GeneA–interacts–GeneB–associated–Disease can check each link. For GNNs, the explanation is reconstructed. Methods such as GNNExplainer search for the minimal subgraph and feature subset that preserves the prediction, effectively asking what the model would still need in order to say the same thing. This is useful, but it answers a counterfactual question about the model, not a mechanistic question about biology. A subgraph that suffices to reproduce a prediction may not be the reason the prediction is correct, and if the prediction is wrong the subgraph explains the error rather than exposing it. For language-model-augmented systems, the explanation is generated prose, and the gap between the stated rationale and the actual computation is widest of all. A 2026 commentary in the drug repurposing literature made the point sharply: benchmark accuracy has advanced much faster than pharmacological credibility, and a prediction that cannot be defended mechanistically will not survive contact with a medicinal chemist regardless of its rank. The practical upshot for anyone building these systems is that explanation should be treated as a filter, not a feature. Its value is that it lets an expert reject candidates fast. A system that produces fifty ranked predictions with checkable supporting paths, of which an expert discards forty-seven in an hour, has done its job. A system that produces fifty predictions with fluent paragraphs that take an hour each to evaluate has produced a week of work and no information. The evaluation trap Here the field has a problem that should be understood clearly, because it explains why benchmark performance and practical utility have diverged so far. The standard protocol for link prediction is to hide a random sample of the graph's edges, train on the remainder, and measure how highly the model ranks the hidden edges against sampled negatives. Report AUROC, mean reciprocal rank, Hits@10. Compare methods. Publish. Every step of this contains a problem for discovery. Random edge removal is not how discovery works. The edges that are missing from a real knowledge graph are not a random sample. They are systematically the relationships nobody has studied — in understudied diseases, for understudied genes, in populations that were not recruited. A model that excels at recovering randomly hidden edges has learned to interpolate within the well-studied core, which is precisely the region where no discovery remains. Negative sampling is fiction. Knowledge graphs record what is known to be true; they rarely record what is known to be false. Training and evaluation therefore treat random unobserved pairs as negatives. Some of those are true relationships nobody has recorded — which is to say, they are exactly the discoveries the system exists to find, being used as examples of what to score low. The metrics reward degree. Because high-degree nodes have more hidden edges and more plausible-looking neighbourhoods, a model that simply learns node popularity performs respectably. Several benchmarking studies have found that trivially simple degree-based baselines match sophisticated architectures on standard splits. A 2026 benchmarking effort in drug repurposing — assembling multiple graph neural network algorithms on common tasks and splits — found that relative rankings shift substantially with graph construction and split design, which is the polite way of saying that leaderboard positions are partly artefacts of the evaluation. The fix, insofar as there is one, is the same as in Chapter 2: temporal splits, in which the graph is built from evidence available before a date and evaluated on relationships established after it. This is much harder, because it requires provenance timestamps on every edge, which most integrated graphs do not carry. Where it has been done, absolute performance numbers drop sharply — often by a large factor relative to random splits — and the gap between methods narrows. What graphs give that text does not Despite all this, the graph representation earns its place for three reasons. Constraint. A graph reasoning system cannot invent a gene. It operates over a fixed node set with defined identifiers, which eliminates an entire class of hallucination. When a graph model outputs a prediction, the entities in it exist. Auditability. For path-based methods and, with more work, for GNNs, you can ask why. The answer is a subgraph, and a subgraph is something a domain expert can attack. In practice this is how good candidates get filtered: a biologist looks at the supporting paths, sees that two of the three run through a promiscuous hub protein that connects to everything, and discards the prediction in under a minute. Composability with computation. Nodes in a chemical graph can be handed to a docking program; nodes in a materials graph can be handed to a density functional theory calculation. The graph is a scaffold onto which non-textual evidence can be attached. This is the beginning of the loop that the rest of the booklet is about — the point at which a hypothesis stops being a statement about the literature and starts being a statement about the world, checkable by something other than more reading. Where this has actually worked Two cases are worth holding onto, one celebrated and one cautionary. The celebrated one is baricitinib. In early 2020, researchers at BenevolentAI used a knowledge graph assembled from the biomedical literature to search for approved drugs that might interrupt both the inflammatory cascade and the viral entry process in COVID-19. The graph surfaced baricitinib, a rheumatoid arthritis drug, on the grounds that it inhibits AAK1 and GAK — kinases implicated in clathrin-mediated endocytosis, the route the virus uses to enter cells — while also suppressing the JAK-STAT inflammatory signalling that drives severe disease. The hypothesis was published in The Lancet as a short correspondence in February 2020. It was subsequently tested in the ACTT-2 trial and in the RECOVERY platform trial, received emergency use authorisation, and is now part of standard practice for hospitalised patients. That is a genuine, consequential, computationally originated hypothesis, and it should be cited more carefully than it usually is. The graph did not autonomously discover a drug. It produced a shortlist; human pharmacologists recognised why one item on the shortlist was interesting; the mechanism was independently plausible and articulable in two sentences; and the world happened to contain the resources to run large randomised trials within months. Remove any of those conditions and the story ends at the shortlist. The cautionary case is the general one. Graph-based repurposing pipelines have produced many thousands of published predictions. The number that have been experimentally tested is small; the number validated in humans is smaller still. The bottleneck is not prediction. It never was. There is a useful discipline hidden in that observation. If a group can run, say, twenty cell-based validation assays a quarter, then a hypothesis generator producing ten thousand ranked candidates has given them precisely twenty decisions to make, and everything below rank twenty is decorative. The quantity that determines the system's value is its precision in the top twenty — a number that almost no published repurposing paper reports, because computing it requires running the assays. Reporting AUROC over a million candidate pairs instead is not merely a weaker measurement; it is a measurement of a different thing, and the two can move in opposite directions. A graph, then, is a good scaffold and a poor oracle. It constrains the hypothesis space to real entities, it makes the reasoning inspectable, and it provides attachment points for computation and experiment. What it cannot do is tell you which of its missing edges are missing because nobody looked and which are missing because they are not there. Answering that requires something the graph does not contain, and the next development in the field was an attempt to organise the search for it. Hashtags: #AutomatedHypothesisGeneration #ScientificDiscovery #MachineLearningForScience #UnsupervisedLearning #HypothesisGeneration #LiteratureBasedDiscovery #UndiscoveredPublicKnowledge #SwansonABCModel #RepresentationLearning #ScientificTextMining #WordEmbeddings #ScientificLanguageModels #KnowledgeGraphs #GraphNeuralNetworks #LinkPrediction #AgentArchitectures #MultiAgentSystems #NoveltyDetection #PriorArtChecking #HypothesisRanking #ScientificReasoning #ExperimentalValidation #HumanInTheLoop #MachineAssistedDiscovery #FutureOfAutomatedScience

Download the Book (PDF): Introduction In the summer of 2021, a database appeared on the internet that contained the predicted three-dimensional structure of nearly every protein encoded in the human genome. A year later it held more than two hundred million structures, covering most of the known protein universe. Before that database existed, determining a single protein structure was a project: months of expression and purification, crystallisation trials that failed more often than they worked, beam time at a synchrotron, and a graduate student's patience. Structural biologists had spent six decades accumulating roughly two hundred thousand experimentally determined structures. A neural network produced a thousand times that number in a few weeks of GPU time. It is worth being precise about what happened, because the event is routinely described in ways that are either too modest or wildly overstated. What happened was not that a machine learned biology. What happened was that a machine learned to approximate, from data, the output of a physical process that nobody knew how to compute directly. The protein folding problem had been framed since the 1960s as a question about physics: a chain of amino acids adopts the conformation that minimises its free energy, so in principle you should be able to compute the structure from the sequence and a good enough energy function. Half a century of effort showed that "in principle" was carrying almost unlimited weight. The energy functions were not good enough, and even if they had been, the search space was too large to explore. AlphaFold2 did not solve that physics problem. It sidestepped it, by learning a map from sequences to structures out of the accumulated evidence of evolution and the existing structural database. That move — replace an intractable computation with a learned approximation of its result — is the single idea underneath nearly everything this booklet describes. It is the reason neural networks have become central to molecular modeling, and it is also the reason their limits fall where they do. A learned approximation is only as good as the region of the world its training data covered. It is fast, sometimes by six or seven orders of magnitude, and that speed changes what questions are worth asking. But it does not know what it has not seen, it cannot distinguish an interpolation from an extrapolation without help, and it will produce a confident, physically plausible, entirely wrong answer when pushed past the edge of its experience. Everything that is thrilling about these methods and everything that is dangerous about them comes from the same source. The argument of this booklet is that the neural network revolution in molecular modeling is best understood as a systematic substitution of learned surrogates for expensive physical computations — and that tracing where each substitution works, where it fails, and what it costs tells you far more about the future of computational drug discovery than any headline about a model beating a benchmark. Follow the substitutions and you can see the bottleneck in pharmaceutical research moving. It used to sit in structure determination and in compute. It does not sit there any more. It has moved to the places where no surrogate is available: to the quality and coverage of experimental data, to the biology of whether a target matters at all, and to the slow, expensive, irreducibly physical business of putting a molecule into a living system and watching what it does. This is not a sceptical book. The gains are real and they are large. A virtual screen that once covered a few million compounds now routinely covers billions, because a neural network trained on a few hundred thousand docking scores can rank the rest at a thousandth of the cost. Free energy calculations that needed a week of supercomputer time for a handful of compounds are being run at speeds that make them usable inside a design cycle. Proteins that do not exist in nature are designed on a laptop, ordered as synthetic genes, expressed, and found to bind their intended targets — not always, not reliably, but often enough to have changed what protein engineering means. In 2026, molecules whose structures were proposed by machine learning systems entered human clinical trials. None of this was possible fifteen years ago, and most of it was not obviously possible five years ago. But it is a book that insists on the accounting. The pharmaceutical industry's problem was never primarily that it could not generate molecules. Medicinal chemists have been able to make more compounds than they could test for as long as there has been medicinal chemistry. The problem is attrition: nine out of ten compounds that enter clinical trials fail, most of them in Phase II, most of them because the biological hypothesis was wrong or the safety margin was too narrow. A tool that makes the early stages faster and cheaper improves the economics of the front end. Whether it improves the number of drugs that reach patients is a different question, and the honest answer in 2026 is that we have the first fragments of evidence and they are encouraging but thin. The reader this booklet is written for is someone who works at, or near, the boundary between machine learning and molecular science and wants the other side explained properly. That might be a computational chemist who can build a force field but has only a functional understanding of what an attention head does, or a machine learning engineer who can implement an equivariant graph network but has never had to explain why a scoring function's error of one kilocalorie per mole matters. Both audiences are served by the same thing: mechanism. What is the model actually computing? What does its input encode and what does it throw away? What physical quantity is being approximated, and what is the approximation's error budget? That commitment shapes the book. It spends a chapter on molecular representation before it spends a chapter on any architecture, because the choice of representation determines more about a model's behaviour than the choice of layers. It treats protein structure prediction and protein language models as separate developments with different logic, rather than folding them into one story. It gives binding prediction — the problem of whether a small molecule sticks to a protein and how tightly — more room than structure prediction, because binding is where drug discovery actually lives and where the methods are least settled. It takes neural network potentials seriously as a distinct branch, because learning an energy function is a fundamentally different task from learning a structure, with different failure modes and, arguably, a more durable long-term payoff. A note on what is left out. This is a booklet about molecular modeling: about the physics and geometry of proteins, ligands, and their interactions. It is not a survey of artificial intelligence in pharmaceutical research generally. Target identification from omics data, clinical trial optimisation, medical imaging, pathology, real-world evidence, and the use of large language models as research assistants are all substantial fields with their own literatures, and none of them appears here except where they bear directly on the molecular story. Nor does the book cover the drug discovery pipeline downstream of a chemical series: formulation, process chemistry, regulatory strategy. The subject is the molecule and the machine. A second note, on currency. Any book written about this field ages quickly, and one written in 2026 about methods that were mostly invented after 2018 will age faster than most. The specific model names in these pages will be superseded. What will not be superseded, or not as quickly, is the structure of the problem: which computations are expensive, which are learnable, what data exists and what does not, and where the physical constraints sit that no amount of training data will relax. The book is organised around those durable features, and uses the current generation of models as illustrations rather than as the point. The chapters proceed roughly in order of how far the methods sit from raw physics. The first establishes what molecular modeling was trying to compute before machine learning arrived, and why it was so expensive. The second examines how molecules get turned into something a neural network can consume, which is the central and underappreciated design decision in the field. The third and fourth treat protein structure prediction and protein language models — the two great successes, and the ones that established the pattern. The fifth turns to binding, where the problem is harder and the record more mixed. The sixth covers neural network potentials, where machine learning is used not to shortcut physics but to accelerate it. The seventh looks at screening, where the substitution is economic rather than scientific: the model is a cheap filter, not a source of truth. The eighth covers generative design, the inversion of the whole enterprise. The ninth is an accounting of failure: what the benchmarks do not measure, why prospective validation keeps disappointing, and what the honest error bars look like. The conclusion asks where the constraint sits now, and what that implies for anyone deciding where to put their effort. One warning, offered once and then assumed. Everything in this field is reported in terms of performance on benchmarks, and the benchmarks are, with few exceptions, retrospective: a model is trained on data up to a cutoff and evaluated on data after it. This is a much weaker test than it sounds, because chemical and biological data are not independent samples. Molecules cluster into series, proteins cluster into families, and a random split of a dataset almost always puts near-duplicates on both sides of the line. A great deal of reported progress in molecular machine learning is the measurement of memorisation. Distinguishing the real advances from the artefacts is the single most useful skill in reading this literature, and it is a theme that runs through every chapter that follows. Chapter 1: The Computation Underneath Before there were neural networks in molecular modeling there was a set of well-defined computational problems that nobody could solve at the accuracy and speed the problems demanded. Understanding what those problems are, and precisely why they are expensive, is the prerequisite for understanding what machine learning changed. Otherwise the field looks like magic, and the limits look arbitrary. Everything in molecular modeling reduces to two questions. Given a molecular system — a protein, a small molecule, a complex of the two, surrounded by water and ions — what arrangements in space can it adopt, and what is the energy of each? From those two things, everything else follows. Structure is the arrangement with the lowest free energy. Binding affinity is the free energy difference between the bound and unbound states. Reaction rates come from energy barriers between arrangements. Selectivity is a comparison of binding free energies across targets. The whole discipline is, in principle, an exercise in evaluating energies and searching over conformations. Both halves of that are hard, and they are hard in different ways. The energy problem The exact energy of a collection of nuclei and electrons is given by the Schrödinger equation, which is unsolvable analytically for anything more complicated than a hydrogen atom. Quantum chemistry is the century-long project of approximating it. The approximations form a ladder. At the top sit methods like coupled cluster with single, double, and perturbative triple excitations — CCSD(T), often called the gold standard — which for small organic molecules reaches accuracies of a fraction of a kilocalorie per mole. Its cost scales as roughly the seventh power of system size. Doubling the number of atoms multiplies the compute by about 128. A calculation on twenty atoms is routine; on two hundred it is a research project; on a protein it is out of the question forever. One rung down sits density functional theory, which replaces the many-electron wavefunction with the electron density and scales as something like the third or fourth power of system size. DFT is the workhorse of computational chemistry. It handles systems of a few hundred atoms comfortably and, with effort and approximations, a few thousand. Its errors on reaction energies are typically a few kilocalories per mole, depending heavily on the choice of functional, of which there are hundreds and no principled way to choose. Still: DFT on a protein-ligand complex in explicit water, sampled over the microseconds needed to get thermodynamics right, is not happening. So the field descends to molecular mechanics. A classical force field discards electrons altogether and models the molecule as balls connected by springs: bond lengths and angles have harmonic terms, rotations about bonds have periodic terms, and non-bonded atoms interact through a Lennard-Jones potential for shape complementarity and a Coulomb term between fixed partial charges. Evaluating this costs essentially nothing — it scales linearly with the number of atoms once you cut off or approximate the long-range electrostatics — which is why molecular dynamics simulations of hundreds of thousands of atoms over microseconds are routine. The price is accuracy, and more importantly, a particular kind of brittleness. A force field is a fitted model, parameterised against quantum calculations and experimental data for a set of chemical environments the developers anticipated. Standard biomolecular force fields such as AMBER, CHARMM, and OPLS handle proteins, nucleic acids, lipids, and water well, because those systems are chemically homogeneous and enormous effort has gone into them. They handle arbitrary drug-like small molecules much less well, because a drug-like molecule may contain a heterocycle, a halogen bond, a sulfonamide, or a tautomeric ambiguity that the parameter set never anticipated. Generalised small-molecule force fields exist — GAFF, CGenFF, OpenFF — and they assign parameters by analogy, which works until it does not. More fundamentally, a fixed-charge force field cannot represent polarisation. Real electron clouds deform in response to their environment; a fixed partial charge does not. In a protein binding site, where a ligand may sit between a charged residue and a metal ion, this matters. Polarisable force fields exist and are a factor of several more expensive, and they have never quite taken over. And no classical force field can represent bond breaking or formation at all, which rules out covalent inhibitors and any enzymatic mechanism without resorting to hybrid quantum-classical schemes. So the energy ladder presents a stark choice: accurate and unaffordable, or affordable and approximate in ways that are hard to bound. Table 1 lays out the tiers. Table 1. Approximate cost and accuracy of energy evaluation methods. Method class Scaling with system size Typical energy error Practical system size Coupled cluster, CCSD(T) ~N⁷ <1 kcal/mol tens of atoms Density functional theory ~N³–N⁴ 2–5 kcal/mol hundreds of atoms Semi-empirical quantum ~N³ 5–10 kcal/mol thousands of atoms Classical force field ~N highly variable; poor for novel chemistry millions of atoms Empirical docking score ~N 2–3 kcal/mol typical, weakly correlated millions of atoms Error figures are order-of-magnitude indications for organic molecules and depend strongly on the property and system; they are not benchmark results for any specific dataset. That table conceals the number that matters most for drug discovery. A tenfold change in binding affinity corresponds to about 1.4 kilocalories per mole at body temperature. Distinguishing a nanomolar compound from a micromolar one — the difference between a drug candidate and a discarded molecule — means resolving energy differences of four kilocalories per mole. Against methods whose errors are of the same magnitude or larger, that is a demanding target. The entire dynamic range that medicinal chemistry cares about is smaller than the error bar of the cheap methods. The sampling problem Even a perfect energy function would not be enough, because the quantity that determines observable behaviour is not energy but free energy, and free energy includes entropy. A molecule does not sit at the bottom of a single well. It moves among an enormous number of conformations, weighted by the Boltzmann distribution. The free energy of a state is a sum over all the microstates that belong to it, which means computing it requires visiting them in the right proportions. The scale of the problem is easy to state and hard to internalise. Consider a modest drug-like molecule with six rotatable bonds. If each bond has three preferred rotational states, the molecule has 3⁶ — 729 — distinct conformers before you account for ring puckering or the continuous nature of real rotations. A protein of three hundred residues has two backbone dihedral angles per residue plus side chain torsions; the number of conformations is astronomical in the literal sense. Cyrus Levinthal made the point in 1969: if a protein searched its conformational space randomly, it would take longer than the age of the universe to find its native state, and yet proteins fold in microseconds to seconds. The resolution of Levinthal's paradox is that folding is not a random search — the energy landscape is funnelled, biasing the chain toward its native structure — but the paradox correctly identifies the computational difficulty. Our algorithms do search something close to randomly. Molecular dynamics attacks this by integrating Newton's equations of motion with a timestep of one or two femtoseconds, limited by the frequency of bond vibrations involving hydrogen. A microsecond of simulated time therefore requires around a billion force evaluations. For decades that was out of reach; specialised hardware such as D. E. Shaw Research's Anton machines and the maturation of GPU codes have brought microsecond and even millisecond simulations within reach for well-resourced groups. But the events that matter in pharmacology often happen on timescales longer still. A ligand's residence time in a binding site may be minutes. A conformational change that opens a cryptic pocket may take milliseconds. A protein may fold in seconds. Brute-force simulation cannot reach these, and so an entire subfield of enhanced sampling methods — umbrella sampling, metadynamics, replica exchange, Markov state models — exists to bias the simulation toward rare events and then correct for the bias. The combination is the real difficulty. The methods that sample well are cheap and inaccurate; the methods that are accurate cannot sample. Free energy perturbation calculations, the most rigorous practical approach to computing relative binding affinities, sit at an uncomfortable compromise: a classical force field for the energy, heavy sampling along an artificial path connecting two ligands, and a typical accuracy of around one kilocalorie per mole on well-behaved congeneric series. That is genuinely useful — it is the difference between a compound worth making and one that is not — but each calculation costs hours of GPU time, works only for pairs of closely related molecules, and degrades badly when the perturbation involves a change in net charge, a significant conformational rearrangement, or a change in the water network in the binding site. The economics this creates Translate the computational picture into a discovery programme and the shape of the problem becomes clear. A typical small-molecule programme begins with a target: a protein whose modulation is believed to affect disease. If a structure is available, computational chemists can dock compound libraries into the binding site, which means generating plausible poses and scoring them. Docking is fast — a second or less per compound on one CPU core — because it uses a heavily simplified scoring function and a restricted conformational search, usually keeping the protein rigid. Its pose prediction is respectable when the binding site is well formed; its ranking of affinities is poor. Published enrichment studies consistently find that docking scores correlate weakly with measured binding, often with Pearson correlations below 0.5, and that hit rates from docking-driven screens sit in the low single-digit percentages. That is better than random — a random screen of a diverse library might hit at a tenth of a percent — but it means most of what the computation recommends is wrong. The expensive methods are reserved for late-stage optimisation, where a chemist has a series and wants to know which of six proposed analogues to synthesise. Free energy perturbation is used at that point by most large pharmaceutical companies, and it works, and it is confined to that narrow window because it costs too much and requires too much structural and chemical similarity to be used earlier. Everything in between — the vast middle where most of the work happens — was handled by human judgement plus cheap heuristics. Medicinal chemists carry intuitions about what makes a molecule bind, what makes it soluble, what makes it metabolically stable, and they apply them to prioritise a synthesis queue. Those intuitions are real expertise and they are also unexamined, inconsistent between chemists, and biased toward chemistry the chemist knows how to make. Meanwhile the cost structure of the industry sat on top of this. The headline figure most often cited for the capitalised cost of bringing a new drug to market is around $2.6 billion in 2013 dollars, from the Tufts Center for the Study of Drug Development, a number that has been extensively criticised on methodological grounds and that varies enormously by therapeutic area and study. What is not disputed is the attrition pattern behind it: of compounds entering Phase I clinical trials, historically around ten percent reach approval, with the largest single loss occurring in Phase II where efficacy in humans is tested for the first time. The failures are not mostly chemistry failures. Analyses of clinical attrition consistently attribute the majority to lack of efficacy — the target did not do what it was thought to do — followed by safety. This matters enormously for how one should read claims about AI in drug discovery, and it is worth stating bluntly at the outset. The computational problems described in this chapter — energy evaluation, conformational sampling, pose prediction, affinity ranking — all sit in the discovery phase, which accounts for a minority of the total cost and time of a programme and almost none of the attrition. Solving them perfectly would make discovery faster and cheaper. It would not, by itself, fix the reason drugs fail. A worked example: the water nobody models It is worth following one concrete case through, because the abstract statement that binding free energy is hard does not convey how it is hard. Consider a medicinal chemistry decision of the most ordinary kind. A team has a compound that binds its target at 100 nanomolar. A chemist proposes adding a hydroxyl group to a phenyl ring that points into a small subpocket. The question is whether to spend three weeks making it. The intuitive argument for the modification is that the hydroxyl can donate a hydrogen bond to a backbone carbonyl three angstroms away, and a hydrogen bond in a buried environment is conventionally worth one to two kilocalories per mole — a tenfold to thirtyfold improvement in affinity. The docking program agrees: the modified compound scores better, because the scoring function has a term that counts hydrogen bonds and this adds one. Now the complications. That subpocket is not empty in the unbound protein. It contains one or two ordered water molecules, visible in the crystal structure as discrete density. Those waters are making their own hydrogen bonds to the protein. If the new hydroxyl displaces a water that was well satisfied — making three or four good hydrogen bonds, sitting in a comfortable position — then the modification pays the full cost of removing a happy water and receives back only one hydrogen bond. The net can easily be unfavourable. If instead the displaced water was frustrated, unable to make its full complement of hydrogen bonds because the pocket geometry does not permit it, then displacing it releases that water to the bulk solvent where it is happier, and the modification gains both the new hydrogen bond and the entropy and enthalpy of the liberated water. The same chemical change, in two pockets that look similar, can be worth a hundredfold in either direction. Nothing in a docking score represents this. The scoring function does not know the waters are there — most docking protocols delete them — and has no term for their satisfaction. Methods that attempt to address it exist, most notably inhomogeneous solvation theory implementations such as WaterMap, which run a molecular dynamics simulation of the pocket and estimate the enthalpy and entropy of water in each hydration site relative to bulk. They are useful and they are expensive, and their accuracy depends on the same force field whose limitations opened this chapter. The hydroxyl also carries costs the score ignores. It adds a rotatable bond, which costs conformational entropy on binding. It must be desolvated, and a hydroxyl is well solvated in water, so stripping it costs several kilocalories that must be repaid by the interaction it makes. If the local geometry is slightly off — if the donor-acceptor distance is 3.4 angstroms rather than 2.9, or the angle is poor — the repayment is partial. The sum of these effects is a number between roughly minus two and plus two kilocalories per mole, and the methods available compute it with an uncertainty of comparable size. Which is to say: the calculation does not answer the question. This is the routine situation in lead optimisation, it recurs dozens of times in a programme, and it is the concrete form of the accuracy problem that everything in this book is trying to address. What a surrogate model offers Set against this background, the appeal of machine learning is easy to see, and it is not the appeal that popular accounts usually emphasise. The classical methods are all, in a sense, doing the same thing: computing a function from a molecular configuration to a number, using a theory that specifies the function explicitly. Quantum chemistry specifies it from first principles and pays the cost. Force fields specify a cheap functional form and fit its parameters. Docking scores specify an even cheaper form and fit it to known binders. A neural network is the limiting case of that progression. It abandons the requirement that the functional form be interpretable or physically motivated, and instead uses a flexible function approximator with millions or billions of parameters fitted to whatever data exists. If you have enough examples of configurations and their energies — computed at the DFT level, say — you can train a network to reproduce the DFT energy at force-field cost. If you have enough examples of sequences and their structures, you can train a network to map one to the other without ever evaluating an energy. If you have enough examples of molecules and their measured solubilities, you can predict solubility without modelling a single water molecule. That is the substitution. And its properties follow directly from its nature. It is fast, because evaluating a fixed network is cheap regardless of how expensive the quantity it approximates was to compute originally. It is accurate within the distribution of its training data, sometimes matching the reference method it was trained on almost exactly. It is unreliable outside that distribution, in a way that is difficult to detect from the model's own output, because a neural network has no notion of the boundary of its own competence unless one is explicitly engineered in. And it requires data, which in molecular science is the binding constraint: computed data can be generated at will but only at the level of theory you can afford, and experimental data is scarce, noisy, heterogeneously measured, and systematically biased toward the compounds and targets that people chose to study. The rest of this booklet is an examination of that trade in its various forms. It begins with the question that determines everything downstream: how do you put a molecule into a neural network in the first place? Chapter 2: How a Machine Sees a Molecule A neural network consumes tensors. A molecule is not a tensor. The translation between them is the most consequential design decision in molecular machine learning, and it is routinely treated as a preliminary detail before the interesting part about architectures. This is backwards. The representation determines what information the model can possibly use, what symmetries it must learn versus what it gets for free, and what kinds of generalisation are available to it. Two models with identical architectures and different representations behave like different species. Two models with the same representation and different architectures usually behave quite similarly. The reason representation is unusually hard for molecules is that a molecule is several different objects at once. It is a set of atoms with types. It is a graph, with bonds as edges. It is a geometric object in three-dimensional space, existing not as one conformation but as a thermally weighted ensemble of them. It is a quantum system whose electron density is smeared across the whole thing and does not respect the bond lines chemists draw. And in a biological context it is a participant in a much larger system of protein, solvent, and ions. Every representation picks some of this and discards the rest. Strings The oldest machine-readable representation is a string. SMILES — Simplified Molecular Input Line Entry System, devised by David Weininger in the late 1980s — writes a molecule as a linear sequence of characters by walking its graph: atoms become element symbols, branches become parentheses, ring closures become matching digits, aromaticity becomes lower case. Aspirin is `CC(=O)Oc1ccccc1C(=O)O`. It is compact, human-readable with practice, and universally supported. Its appeal to machine learning was obvious once sequence models matured: a molecule written as a string is exactly the kind of object that recurrent networks and then transformers were built for. Chemical language models — trained with the same objectives used for natural language, whether next-token prediction or masked-token recovery — learn to generate valid molecules, to complete partial structures, and to produce embeddings that transfer to property prediction. Models in this family have been trained on hundreds of millions of molecules from public databases, and they work better than their crudeness suggests. But SMILES has two problems that no amount of training fully repairs. The first is non-uniqueness: the same molecule has many valid SMILES depending on which atom you start from, so the representation imposes an arbitrary ordering that carries no chemical meaning. Canonicalisation algorithms pick one, at the cost of making trivially similar molecules receive dissimilar strings. Randomised SMILES augmentation — training on many string forms of the same molecule — is the usual mitigation and it helps, which tells you the model is spending capacity learning to undo an artefact of the encoding. The second problem is fragility. A single wrong character produces an invalid molecule, and a generative model sampling character by character will produce unclosed rings and pentavalent carbons at some rate. SELFIES, introduced by Krenn and colleagues in 2020, solves this by designing a grammar in which every string decodes to a valid molecule, which is elegant and removes an entire class of failure in generative work. The deeper limitation is that strings encode topology only. A SMILES string knows the bond graph and stereochemistry annotations but nothing about three-dimensional shape, and shape is what determines binding. For predicting properties that are largely topological — solubility, metabolic liability, some toxicity endpoints — this matters less than one might expect, because the bond graph constrains the shape considerably. For predicting how a ligand sits in a pocket, it is disqualifying. Graphs The representation that matches a chemist's mental model most closely is the molecular graph: atoms as nodes carrying features such as element, formal charge, hybridisation, and aromaticity; bonds as edges carrying order and stereochemistry. Graph neural networks operate on this by message passing. Each node holds a vector. At each layer, every node collects transformed vectors from its neighbours, aggregates them with a permutation-invariant operation such as a sum or mean, and updates its own state. After k layers, a node's representation reflects everything within k bonds of it. A final pooling step aggregates node vectors into a molecule-level vector for prediction. This framework, formalised as message passing neural networks by Gilmer and colleagues in 2017, subsumes most graph architectures in chemistry and remains the default for property prediction from topology alone. It is a good fit for chemistry because it respects the right symmetry automatically. A molecule has no canonical atom ordering, and permutation-invariant aggregation means a graph network's output does not depend on how the atoms were numbered. Nothing is wasted learning that the molecule is the same when you renumber it. The limitations are known precisely, which is unusual and useful. Standard message passing has expressive power bounded by the Weisfeiler-Lehman graph isomorphism test, which means there exist non-isomorphic molecules it provably cannot distinguish. In practice the harder limitation is over-squashing: information from distant parts of the graph must be compressed through a fixed-width vector along a narrow path, so long-range effects — a substituent's electronic influence across a conjugated system, an intramolecular hydrogen bond between groups far apart in bond distance — are captured poorly. Adding layers makes it worse, because repeated aggregation smooths node representations toward uniformity, a phenomenon called over-smoothing. Most production graph networks in chemistry are three to five layers deep. And, again, the graph is not three-dimensional. Two stereoisomers with the same connectivity differ in ways a plain graph network cannot see unless stereochemistry is explicitly encoded as node or edge features. Conformational flexibility is invisible entirely. Three dimensions, and the symmetry problem To model binding you need geometry: atoms at coordinates in space. The moment you introduce coordinates, you introduce a problem that dominates the design of every modern structural model. Physical properties do not depend on where you put the origin or how you orient the axes. A molecule's energy is unchanged if you translate it, rotate it, or reflect it. Its forces rotate with it. In the language of symmetry, scalar properties are invariant under the Euclidean group and vector properties are equivariant to it: rotate the input, and the output rotates the same way. A naive model taking raw coordinates as input has none of this. It must learn from data that a rotated molecule is the same molecule, which is an enormous waste of capacity and never fully succeeds. The early workaround was data augmentation — train on many random rotations — which helps, and is a symptom of a design error rather than a solution. The first clean fix is to use invariant features instead of coordinates. Interatomic distances are unchanged by rotation and translation; so are bond angles and torsions. A model built on distances and angles is automatically invariant. This is the approach of the first generation of neural network potentials: Behler-Parrinello symmetry functions, which describe each atom's environment through radial and angular distribution functions, and the ANI models built on them. It works well and is simple to implement. Its weakness is that a set of distances loses information — distinguishing a structure from its mirror image requires more than pairwise distances, and building richer invariant descriptors gets combinatorially awkward. The more powerful approach is to build equivariance into the network so that internal representations are geometric objects rather than scalars. In an equivariant network, each atom carries not just a scalar feature vector but also vector-valued and higher-order tensor features, and the operations that combine them are constrained to transform correctly under rotation. The mathematical machinery comes from representation theory: features are decomposed into irreducible representations of the rotation group, indexed by degree, and combined using tensor products with Clebsch-Gordan coefficients. This sounds forbidding and reduces, in practice, to a constrained form of message passing where the messages are geometric. The payoff is not just elegance. Equivariant models are dramatically more data-efficient. Because the symmetry is built in rather than learned, a training example teaches the model about the whole orbit of rotated copies at once. Benchmarks on molecular force prediction have repeatedly shown equivariant architectures reaching a given accuracy with an order of magnitude less training data than invariant or unconstrained ones. In a field where data is the binding constraint, that is the whole argument. Table 2 sets out the main representation families and what each commits you to. Table 2. Molecular representations and their properties. Representation Encodes Symmetry handling Typical models Main limitation SMILES / SELFIES string topology, stereo labels arbitrary atom ordering chemical language models no geometry; non-unique Fingerprint vector substructure presence order-invariant by construction random forest, gradient boosting fixed, hand-designed features Molecular graph topology, atom and bond types permutation-invariant message passing networks no geometry; over-squashing Invariant 3D descriptors distances, angles invariant by construction Behler-Parrinello, ANI information loss; no vector outputs Equivariant 3D point cloud full geometry equivariant by construction NequIP, MACE, e3nn-based computationally heavier Voxel grid geometry on a lattice none without augmentation 3D convolutional networks resolution and orientation artefacts Two families in that table deserve a note. Fingerprints — most commonly extended-connectivity fingerprints, ECFP, which hash circular substructures around each atom into a fixed-length bit vector — predate deep learning and remain shockingly competitive. In head-to-head comparisons on property prediction benchmarks, gradient boosted trees on ECFP frequently match or beat graph neural networks, particularly on small datasets, which describes most real medicinal chemistry datasets. Anyone building a molecular property model who has not run that baseline is not entitled to claim their deep model works. Voxel grids, which place the molecule on a three-dimensional lattice and apply convolutions as one would to an image, were popular around 2017 and have largely been abandoned. They discretise coordinates, lose rotational symmetry, and scale badly in memory. They remain occasionally useful for representing binding-site shape, where the object of interest genuinely is a field rather than a set of points. Proteins, which are a different problem A protein can be treated with any of the above, but its scale and its biology invite different choices. At the sequence level, a protein is a string over an alphabet of twenty amino acids, typically a few hundred characters long. This is a natural fit for language models, and Chapter 4 takes up what they learn. At the structural level, the useful abstraction is the residue frame. Rather than treating a protein as thousands of individual atoms, one treats it as a chain of rigid bodies: each residue's backbone nitrogen, alpha carbon, and carbonyl carbon define a local coordinate frame, a position and an orientation. The protein becomes a sequence of frames, and the modelling problem becomes predicting rigid transformations between them. This is the representation at the heart of AlphaFold2's structure module, where the protein is initialised as a "residue gas" of identical frames at the origin and iteratively refined. Side chains are handled separately as torsion angles off the backbone frame, which is legitimate because bond lengths and angles within a residue are essentially fixed. The frame representation is efficient — hundreds of frames instead of thousands of atoms — and it matches the physics, since backbone conformational freedom really does live in the torsion angles. Its cost is that anything not captured by idealised geometry, including unusual covalent modifications and strained conformations, needs special handling. There is one more representation that dominated structural bioinformatics before end-to-end models arrived and still matters: the pairwise distance matrix, or contact map. For a protein of N residues, this is an N×N matrix of distances or binary contacts. It is invariant to rotation and translation by construction, it is a natural output format for a two-dimensional convolutional network, and it can be converted back to coordinates by distance geometry. The generation of structure prediction methods immediately before AlphaFold2 — including the first AlphaFold — predicted distance distributions and then folded a structure to match them. The pair representation survives inside modern models as an internal state even though the final output is now coordinates directly. Attention as the connecting tissue The transformer deserves a short treatment here because it appears in nearly every model discussed later, in forms that look different from its use in language. The mechanism is simple. Each element of a set produces a query, a key, and a value vector. The compatibility between one element's query and another's key, normalised across all elements, determines how much of that element's value flows into the first element's update. It is a learned, content-dependent, all-to-all information exchange over a set. Three properties explain its ubiquity in molecular modeling. First, it is permutation-equivariant over the set it operates on, which suits collections of atoms or residues. Second, it is all-to-all, so it has none of the locality restrictions that cause over-squashing in message passing — a residue can attend directly to another five hundred positions away. Third, it is easily biased: the attention logits can be modified by additional terms encoding, for example, the predicted distance between two residues, which is how geometric information is injected into an otherwise geometry-free mechanism. A graph network is, viewed from a certain angle, attention restricted to bonded neighbours with uniform weights. A transformer over atoms is a graph network on a complete graph with learned edge weights. The distinction that matters is not architectural taxonomy but what the model is allowed to attend to and what inductive bias is imposed on the attention. The cost is quadratic in set size, which for a protein of a few hundred residues is fine and for a system of tens of thousands of atoms is not. The various approximations that address this — sparse attention, local attention windows, structure-aware neighbour lists — are engineering rather than science, but they determine what is feasible. The complex, which is two objects at once A protein-ligand complex forces a representational decision that has no clean answer, and the choices made here largely determine how a binding model behaves. The two components are chemically different in a way that matters. A protein is a polymer of twenty standard units, highly repetitive, with fixed local geometry and variation concentrated in torsions. A drug-like ligand is a small, chemically idiosyncratic object drawn from an enormous space of elements, ring systems, and functional groups, with no repeating structure. A representation tuned for one is poorly suited to the other. Three strategies are in use. The first treats everything as atoms: one point cloud containing both protein and ligand, one network, no distinction. This is the most physically honest choice and the most expensive, since a binding site plus ligand is several thousand atoms and an all-atom equivariant network over that is heavy. It is what neural network potentials do, and it is why they are applied to the pocket rather than the whole protein. The second uses separate encoders and a cross-attention interface: a protein encoder operating on residues, a ligand encoder operating on atoms, and attention between them producing interaction features. This is efficient and it introduces an asymmetry — the model's notion of the protein is coarser than its notion of the ligand — which is usually acceptable, since the ligand is where the chemistry varies. The third, used by the co-folding models, tokenises heterogeneously: a protein residue is one token, and a ligand atom is one token, in a single sequence processed by one network. This is what allows AlphaFold3 and its successors to handle proteins, nucleic acids, ions, and small molecules in one framework, and it means the model must learn to treat tokens of very different physical scale appropriately from data rather than by construction. A further decision is whether to represent the pocket at all. Many binding models take only the residues within some radius of the site, which is a hundredfold reduction in size and discards allosteric context and anything the rest of the protein contributes to the site's shape. The cutoff is almost always chosen by convention rather than by measurement. Conformations, and the ensemble the model does not see One assumption runs through nearly all three-dimensional molecular machine learning and is rarely stated: that a molecule can be represented by a single conformation. This is false for essentially every drug-like molecule. A compound with six rotatable bonds populates dozens of conformations at room temperature, and its properties are averages over that ensemble weighted by energy. Which conformation a model is given therefore matters, and the usual answer — generate conformers with a distance-geometry method such as ETKDG, minimise with a force field, take the lowest-energy one — introduces a systematic bias, because the lowest-energy conformation in vacuum or implicit solvent is frequently not the one that binds. Bound ligands routinely adopt conformations several kilocalories above their solution minimum, paying strain energy in exchange for better contacts. Some models sidestep this by taking a set of conformers and pooling over them, which is more honest and multiplies the cost. Others take the crystallographic conformation, which is correct for the training complex and unavailable at prediction time, and is a quiet source of leakage in retrospective evaluations. Generative models that produce conformer distributions rather than single structures — trained to sample from the Boltzmann distribution rather than to predict a mode — are the principled solution and are not yet reliable enough to be default practice. The honest position is that single-conformation representation is a known, unquantified approximation that most reported results depend on. It is one of the reasons three-dimensional models underperform their promise on flexible ligands, and one of the reasons topological representations remain competitive on many tasks: a fingerprint at least does not pretend to know a shape it has guessed. The choice as a statement about the problem None of these representations is correct in any absolute sense. Each is a claim about which information matters for the task. Using a string says: topology determines this property. Using a graph says: topology and local chemical environment determine it. Using a single three-dimensional conformation says: this property is determined by shape, and one conformation adequately represents it — a claim that is defensible for a rigid ligand in a crystal structure and increasingly indefensible for a flexible molecule in solution. Using an ensemble of conformations says: the thermally accessible ensemble matters, at the cost of needing to generate and weight that ensemble. The field's trajectory has been steadily toward richer representations as the data and compute allowed. But the direction is not monotone, and the most common error in applied work is reaching for the most sophisticated representation available rather than the one the data can support. A three-dimensional equivariant model trained on four hundred measured IC50 values will lose to a gradient boosted tree on fingerprints, reliably, every time. The representation must match the task, and it must also match the amount of evidence available to fit it. Chapter 3: The Folding Problem Falls In 1972 Christian Anfinsen accepted a Nobel Prize for work showing that a denatured ribonuclease, allowed to refold in solution with nothing but buffer, recovered its native structure and its enzymatic activity. The conclusion — that a protein's amino acid sequence contains all the information needed to specify its three-dimensional structure — became the founding premise of an entire field. If the sequence determines the structure, then in principle a sufficiently good computation can read one off the other. For the next half century, that principle stubbornly refused to become a method. The failure was not for lack of effort, and the structure of the failure is instructive. What the classical approaches got right and wrong Two broad strategies competed. Template-based modelling exploited the observation that proteins with similar sequences have similar structures: if a query sequence is more than about thirty percent identical to a protein of known structure, you can thread the query onto that template and refine, and you will usually get a model accurate to within a couple of angstroms. This works well and covered a useful fraction of new sequences, but it is fundamentally a lookup table with interpolation. It says nothing about proteins with no structural relatives. For those, the field attempted de novo prediction: search conformational space for the structure that minimises an energy function. Rosetta, developed in David Baker's laboratory from the late 1990s, was the most successful embodiment, assembling structures from short fragments taken from known proteins and scoring them with a physically motivated but heavily empirical energy function. It produced genuinely correct predictions for small proteins and, given enough compute — including the distributed effort of Rosetta@home — occasionally for larger ones. But it ran directly into both problems of Chapter 1 at once: the energy function was not accurate enough to make the native state reliably the global minimum, and the search could not cover the space well enough to find that minimum even when it existed. Progress was measured by CASP, the Critical Assessment of Structure Prediction, a biennial blind trial running since 1994 in which organisers collect sequences whose structures have been solved but not released, and participants submit predictions. CASP is unusually well designed as a benchmark: the answers genuinely do not exist in public at prediction time, which makes it immune to the leakage problems that plague most machine learning evaluation. Its principal metric, the global distance test total score, measures the percentage of residues placed within increasing distance thresholds of their true positions after optimal superposition, scaled from 0 to 100. Scores above about 90 are conventionally regarded as comparable to experimental accuracy. Through the 2000s and early 2010s, performance on the hardest category — free modelling targets, those with no usable template — improved slowly and plateaued. Typical top scores sat in the 30s and 40s: enough to get the fold roughly right sometimes, never enough to be useful for drug design. Coevolution: the signal that made learning possible The idea that broke the plateau did not come from machine learning at all. It came from evolutionary statistics, and it is worth understanding properly because it explains both the success and the specific limitations of what followed. Consider two residues that are far apart in the sequence but touch in the folded structure. If one mutates to a bulkier amino acid, the structure is destabilised unless its partner mutates to something smaller. Over evolutionary time, across the thousands of species that carry a homologue of the protein, such pairs will vary in a correlated way. Collect a multiple sequence alignment — all known homologues of your protein, aligned column by column — and the statistical couplings between columns carry information about which residues are in contact. The idea dates to the 1990s, but early attempts foundered on a confound: correlations propagate. If residue A is coupled to B and B to C, then A and C appear correlated even if they never touch. The breakthrough, achieved around 2011 by several groups working on direct coupling analysis, was to disentangle direct from indirect couplings by fitting a global statistical model — a Potts model over the alignment columns — rather than computing pairwise correlations. Suddenly, for proteins with deep alignments, predicted contacts became accurate enough to fold a structure from. Deep learning entered as a refinement of this. Instead of extracting couplings with an explicit statistical model, train a convolutional network to map from alignment statistics to a contact map. Methods in this family, including RaptorX and the original AlphaFold entered at CASP13 in 2018, improved substantially on direct coupling analysis, and AlphaFold's contact-and-distance-prediction approach won that round by a clear margin. It was a significant advance and it was still, structurally, the old pipeline: predict constraints, then fold to satisfy them. The essential point is this: the information that makes protein structure prediction tractable is evolutionary, not physical. The model is not solving the folding problem in Anfinsen's sense. It is reading out, from the pattern of variation across a protein family, the constraints that selection has preserved — and those constraints happen to encode the structure. AlphaFold2 At CASP14 in 2020 the picture changed categorically. AlphaFold2 achieved a median global distance test score of 92.4 across all targets, and on the free modelling targets where no template existed it scored in the high 80s. The assessors' judgement, widely quoted, was that the structure prediction problem was solved for single protein chains in the sense that had been asked since Anfinsen. The method was published in Nature in July 2021 along with its code, and shortly afterwards DeepMind and the European Bioinformatics Institute released the AlphaFold Protein Structure Database, which by mid-2022 held over two hundred million predicted structures. What changed architecturally is worth laying out, because the design choices have been influential far beyond this one model. The first was to make the pipeline end-to-end differentiable. Previous methods predicted an intermediate — contacts, distances — and then handed it to a separate folding procedure. AlphaFold2 outputs coordinates directly, with the loss computed on the final structure, so every component is trained against the thing that actually matters. The second was the Evoformer, a transformer-based trunk operating on two coupled representations simultaneously: a multiple sequence alignment representation, a matrix of sequences by residues, and a pair representation, a matrix of residue pairs. Attention runs over rows and columns of the alignment, extracting evolutionary signal; attention runs over the pair representation; and, crucially, information flows between the two. Alignment features update the pair representation through an outer product; pair features bias the alignment attention. The pair representation is additionally updated by triangle operations, which enforce a geometric consistency condition: if residues i and j are close, and j and k are close, that constrains the distance between i and k. This is the triangle inequality wired directly into the architecture as an inductive bias. The third was the structure module, which converts the refined representations into coordinates. It maintains each residue as a rigid frame and uses invariant point attention, an attention mechanism that operates on points in the local frames and produces updates that are equivariant to global rotation. Frames are updated iteratively, all residues at once, with no requirement that the chain remain physically connected during the process — a striking design choice that allows the structure to rearrange globally rather than being trapped by local geometry, with chain connectivity emerging by the end. The fourth was recycling: the entire network's output is fed back as input and the whole thing runs again, typically three times. This gives an effectively deeper network with shared weights and lets the model refine an initial guess. The fifth, and in practice one of the most useful, was the confidence prediction. AlphaFold2 outputs pLDDT, a per-residue predicted confidence score on a 0-100 scale, and PAE, a predicted aligned error matrix giving expected positional error between every pair of residues. These are calibrated remarkably well. A region with pLDDT above 90 is almost always right; a region below 50 is usually wrong, and — an unexpected and valuable discovery — such regions correlate strongly with intrinsic disorder, meaning the model's uncertainty is often reporting a genuine biological fact rather than a failure. What was not solved The achievement was real and the framing of "the protein folding problem is solved" obscures a great deal. AlphaFold2 predicts a single static structure. Proteins are dynamic, and a substantial fraction of biology lives in the transitions: a kinase moving between active and inactive conformations, a transporter alternating between inward- and outward-facing states, an allosteric site opening under ligand pressure. The model typically returns one state, usually the one most represented in the training data, with no indication that others exist. Work on eliciting alternative conformations — subsampling the alignment to increase diversity, clustering alignments by subfamily — produces useful results and is a workaround rather than a capability. It does not predict the effect of mutations on stability or function in any reliable way. A single point mutation that destabilises a protein enough to abolish its activity typically produces an essentially unchanged AlphaFold prediction, because the model is reading the evolutionary signal of the family, and one mutation barely perturbs the alignment. This is the most common misapplication in the literature: the model predicts what the family's structure is, not what this particular variant does. It depends on alignment depth. For proteins with few known homologues — orphan proteins, fast-evolving viral proteins, designed sequences with no evolutionary history — accuracy falls substantially. This is the direct consequence of the coevolution insight: no family, no signal. It says little about the parts of proteins that do not fold. Roughly a third of the human proteome is intrinsically disordered in isolation, and for those regions there is no single structure to predict. And it does not, on its own, tell you anything about ligands. The prediction is of apo protein structure; the binding site geometry it produces is the one implied by the family, not the one induced by a particular small molecule. This limitation is the subject of Chapter 5 and is the reason structure prediction did not translate immediately into better drug design. What happened when drug discovery tried to use it The reception in structural biology was immediate and unambiguous: predicted models are now routinely used for molecular replacement in crystallography, for fitting into cryo-electron microscopy density, and for generating hypotheses about domain architecture. In those applications the models are used as what they are — very good starting structures — and they have worked extremely well. The reception in structure-based drug design was more complicated, and the disappointment is instructive. The obvious hope was that predicted structures would open up targets for which no experimental structure existed. Several studies tested this directly by running virtual screens against AlphaFold models and comparing with screens against crystal structures of the same protein. The results were sobering: enrichment of known actives was consistently worse with predicted structures, sometimes substantially, even though the backbone accuracy was high. The reason is a mismatch between what the metric measures and what docking needs. A model with a global distance test score of 90 has its backbone essentially right. Docking depends on the side chain positions lining the pocket, on the precise width of a subpocket, on whether a rotamer points in or out. Errors of one to two angstroms in a handful of side chains, invisible in a global accuracy score, are enough to close a pocket that should be open or to create a clash that rejects the correct pose. Predicted structures are also, by construction, apo structures reflecting the family consensus, whereas a drug binds an induced-fit conformation that may not resemble the consensus at all. The workarounds are the obvious ones: ensemble docking against several predicted conformations, flexible side chain treatment, induced-fit protocols, and refining the model against whatever experimental data exists. They recover part of the gap and do not close it. The practical guidance that has emerged is that a predicted structure is a good basis for understanding a target, a reasonable basis for hypothesis generation, and a poor substitute for a co-crystal structure when the task is choosing which compound to make. This is not a criticism of the method. It is a statement about the difference between two accuracy requirements that happen to be measured on the same object. The one CASP measures was met. The one medicinal chemistry needs is roughly an order of magnitude tighter and was never what the benchmark asked for. Complexes, which are a harder problem than they look Predicting how two proteins assemble is a separate task, and the initial extension of AlphaFold2 to multimers — AlphaFold-Multimer, released in 2021 — made clear how much harder it is. Accuracy on protein-protein complexes is markedly lower than on single chains and varies enormously by complex type. Obligate complexes, whose subunits fold together and never exist separately, are predicted well, because they are effectively one structural unit and the coevolutionary signal across the interface is strong. Transient complexes — a signalling protein briefly contacting a receptor, an antibody binding an antigen — are predicted much less reliably, because the partners evolved largely independently and the interface leaves little coevolutionary trace. Antibody-antigen complexes, which are of enormous commercial importance, remain among the worst cases: the antibody's binding loops are hypervariable by design, produced by somatic recombination rather than by evolution of a conserved pair, so the signal the whole approach depends on is absent in principle rather than merely sparse. The confidence measures help here more than anywhere. The predicted aligned error matrix distinguishes, for a two-chain prediction, whether the model is confident about each chain's internal structure but not about their relative placement — a very common situation, and one a single global score would obscure. Interface-specific confidence measures derived from it are now the standard way to triage which complex predictions to believe. The second generation The years after 2021 saw the architecture generalised in two directions. RoseTTAFold, from the Baker laboratory, arrived almost simultaneously with a three-track architecture — sequence, pair, and coordinate representations updated together — and was quickly extended to protein-protein complexes and then to nucleic acids and small molecules. Its openness accelerated adoption enormously. AlphaFold3, published in Nature in May 2024, made a more fundamental change: it replaced the structure module with a diffusion model operating directly on atomic coordinates, and generalised the input to arbitrary molecular systems — proteins, nucleic acids, ions, small molecules, covalent modifications. The diffusion formulation means the model learns to denoise a random cloud of atoms into a structure, conditioned on the trunk representation, and it removes the need for the frame-based machinery and the stereochemical loss terms that AlphaFold2 required. The same change introduced a new failure mode: diffusion models can hallucinate plausible-looking structure in regions with no supporting evidence, and AlphaFold3's paper documents spurious ordered structure in disordered regions, mitigated with cross-distillation from AlphaFold2 outputs. Because AlphaFold3's code was initially unavailable, open reimplementations followed quickly — Boltz-1 from MIT and Chai-1 from Chai Discovery, both in late 2024, with weights released for research and in some cases commercial use. By 2025 the architecture class, usually called co-folding models, had become a commodity available to any group with a few GPUs. For structure prediction of single chains, the practical state of affairs is now stable. The problem is, for most purposes, done. What remains genuinely open — complexes, conformational ensembles, the effect of specific mutations, and the interaction with small molecules — is not what CASP was measuring, and it is where the rest of this book's difficulties live. The lesson the field took from this It is tempting to draw from AlphaFold2 the conclusion that sufficiently large models trained on sufficiently large datasets solve hard scientific problems. The more careful reading is different, and it explains why the pattern has been harder to repeat elsewhere. Protein structure prediction had three properties that are rare in combination. There was a large, curated, standardised body of ground-truth data — the Protein Data Bank, with over a hundred and fifty thousand experimentally determined structures deposited under common standards and validated by the community. There was an enormous auxiliary signal, the sequence databases, containing far more information than the labelled data and directly informative about the target quantity. And there was a rigorous blind evaluation, CASP, that made self-deception difficult. Binding affinity prediction has none of these. The measured affinity data is smaller by orders of magnitude, scattered across assay formats that do not agree, biased toward the compounds that industrial programmes happened to pursue, and enriched for the successes. There is no evolutionary database that tells you how tightly a small molecule binds. And there is no CASP for affinity — the community-run blind trials that exist are valuable but small. The next two chapters follow these two branches: what could be learned from sequence alone, where the data was abundant, and what happens at the interface with small molecules, where it was not. Hashtags: #AIInMolecularModeling #MolecularModeling #AIDrugDiscovery #NeuralNetworks #MolecularMachineLearning #ComputationalDrugDiscovery #ProteinStructurePrediction #AlphaFold2 #ProteinLanguageModels #MolecularRepresentations #GraphNeuralNetworks #EquivariantNeuralNetworks #MolecularGraphs #BindingAffinityPrediction #ProteinLigandInteractions #NeuralNetworkPotentials #MolecularDynamics #FreeEnergyCalculations #VirtualScreening #MolecularDocking #GenerativeMolecularDesign #DrugDesign #Benchmarking #ProspectiveValidation #FutureOfAIDrugDiscovery

Download the Book (PDF): Introduction In 1983, researchers publishing in the most prestigious basic-science journals in the world were confident about the clinical promise of their work. Two decades later, a team led by Despina Contopoulos-Ioannidis and John Ioannidis went back and checked. They identified 101 articles from leading journals, published between 1979 and 1983, whose authors had explicitly claimed that a discovery held real promise for prevention or treatment. By 2002, only 27 of those discoveries had been tested in a randomized trial. Five had led to a licensed product. One was in wide clinical use. The finding, published in the American Journal of Medicine in 2003, is not an indictment of basic science. Most of those papers were good science. It is a measurement of how much happens, or fails to happen, between a discovery and a patient. A second number circulates in almost every lecture on this subject. In 2000, Andrew Balas and Suzanne Boren estimated that it takes about seventeen years for research evidence to reach routine clinical practice, and that only about fourteen percent of original research ever does. The figure has been criticised, refined and occasionally mocked, most usefully by Zoë Morris, Steven Wooding and Jonathan Grant, whose 2011 review in the Journal of the Royal Society of Medicine was titled "The answer is 17 years, what is the question." Their point was that the time lag depends entirely on where one starts the clock and where one stops it, and that different studies measure different segments of a long and branching path. That point is the beginning of this book. What "translation" means Translational science is the study of how knowledge moves from one kind of work to another: from a laboratory observation to a first human dose, from a trial result to a clinical guideline, from a guideline to what actually happens in a clinic on a Tuesday afternoon, and from an effective intervention to a change in the health of a whole population. It is distinct from translational research, which is the work of moving any particular discovery along that path. The science asks why the path is so slow, so lossy and so uneven, and what can be done about it. Over the past two decades, the field has converged on a vocabulary of numbered stages. T0 is discovery: the identification of a mechanism, a target, a biomarker or a candidate intervention. T1 is the move into humans: first-in-human studies and the early trials that establish safety, dose and proof of mechanism. T2 establishes efficacy in patients and turns that evidence into guidance. T3 is the move from guidance into practice: implementation, dissemination and the study of why proven interventions are or are not used. T4 is the move from practice to population: the outcomes that follow at scale, and the policies that shape whether an intervention reaches everyone who could benefit. Different institutions draw these lines in slightly different places, and the history of how the lines came to be drawn is itself instructive. What matters more than the precise boundaries is the recognition that each stage asks a different question, demands a different kind of evidence, is carried out by different people in different institutions, and is funded and rewarded by different means. The argument of this book The central claim of what follows is simple to state and has consequences that take the rest of the book to work through. Discoveries do not stall in the pipeline so much as at its joints. The characteristic failure of translation is not that the science at any one stage is bad, though sometimes it is. It is that each stage is optimised for its own question and hands the next stage something it was never designed to use. A preclinical study optimised to publish a striking mechanism produces an effect size that no human trial will reproduce. A Phase I study optimised to find the highest tolerable dose hands forward a dose that may be wrong for the patients who will take it for years. An efficacy trial optimised for internal validity, with carefully selected participants and expert investigators, hands forward a result that community clinics cannot reproduce with their own patients and staff. An implementation programme optimised for uptake in one setting hands forward a model that no payer will fund and no legislature will mandate. At every joint, a question that the next stage needed answered was not asked, because asking it was nobody's job. The remedy, which the book develops stage by stage, is to design each stage with the next one's question in view. That means preclinical studies built to predict human outcomes rather than merely to demonstrate mechanism; first-in-human studies that collect the pharmacology later trials will need; efficacy trials that test interventions in forms that can actually be delivered; implementation studies that measure what policymakers will ask about; and policies designed so that their effects can be evaluated. It also means recognising that the pipeline is not a line but a loop. Observations at the bedside and in populations constantly send questions back towards the bench, and some of the most productive translation has run in that direction. How the book is organised Chapter 1 sets out the T0 to T4 framework, its origins and its limits, and argues for reading it as a map of handoffs rather than a sequence of stages. Chapter 2 examines the first and most notorious joint, between preclinical discovery and the decision to test in humans, where problems of reproducibility and model validity cause most candidate interventions to fail before or shortly after they reach a person. Chapter 3 is devoted to the first-in-human study itself: how starting doses are chosen, what went wrong in the most instructive Phase I disasters, and how dose-finding designs have evolved. Chapter 4 follows candidates through the long and costly work of establishing efficacy, where attrition is highest and where the choice of endpoints and trial designs determines whether a result will mean anything outside the trial. The second half of the book turns from products to people and systems. Chapter 5 addresses the gap between evidence and practice, and the implementation science that has grown up to study it. Chapter 6 follows interventions into communities, where the tension between fidelity and adaptation plays out and where sustainability is decided. Chapter 7 examines T4, the translation of evidence into public health policy and the population effects that follow. Chapter 8 considers the pipeline as a system: the flows that run backwards from practice to discovery, the de-implementation of practices that should never have been adopted, and the infrastructure that institutions have built to connect the stages. The conclusion draws out what the whole analysis implies for the people who fund, conduct and use translational research. A note on scope Translational science spans drugs, biologics, devices, diagnostics, behavioural interventions and policies. This book draws examples from all of them but leans on two families of cases: pharmaceutical development, where the early stages are best documented, and public health prevention, where the late stages are. It is written from a largely American and British vantage point, because those are the systems whose translational infrastructure is most thoroughly studied, but the problems it describes are general. Wherever a figure or a study is cited, it is a real one, named so that the reader can find it. Where the evidence is contested, the book says so. The reader need not be a scientist. Anyone who has wondered why a promising headline about a new treatment so rarely becomes a treatment, or why a proven intervention sits unused for a decade, is asking the questions this field exists to answer. The answers turn out to be less about any single failure of intelligence or effort than about the structure of the enterprise itself, and structures, unlike laws of nature, can be redesigned. Chapter 1. The Pipeline and Its Joints The word "translational" entered the vocabulary of biomedical policy in the early 2000s, and it arrived as a diagnosis. In 2003, Nancy Sung and colleagues, writing for the Clinical Research Roundtable convened by the US Institute of Medicine, published an analysis in JAMA of the central challenges facing the national clinical research enterprise. They identified two "translational blocks." The first lay between basic biomedical research and its application in human studies. The second lay between the results of clinical studies and their adoption in everyday practice and decision-making. The same year, Elias Zerhouni, then director of the US National Institutes of Health, launched the NIH Roadmap for Medical Research, which named "re-engineering the clinical research enterprise" as one of its three themes and set in motion the institutional changes that would later produce the Clinical and Translational Science Awards. The two-block model had the virtue of simplicity, and it captured something real. American biomedical research budgets had doubled between 1998 and 2003, and there was a growing unease that the flood of basic discovery was not producing a corresponding flow of new therapies or improvements in health. But the model also concealed a great deal. It grouped together, under "the second block," activities as different as running a randomised trial, writing a clinical guideline, persuading a clinician to follow it, and persuading a government to pay for it. As researchers from different disciplines began to look at the path in detail, the number of blocks began to multiply. From two blocks to five stages The first elaboration came from primary care. In 2007, John Westfall, James Mold and Lyle Fagnan published a short and influential piece in JAMA titled "Practice-based research—'Blue Highways' on the NIH Roadmap." Their argument was that the move from a successful trial to routine practice is not a single step. Trials are conducted largely in academic medical centres, with selected patients and expert staff. Most patients, however, are seen in community practices, and there is a distinct body of research, conducted in those practices, that asks whether and how trial findings work there. Westfall and colleagues labelled this T3 and called for practice-based research networks to carry it out. In 2008, Steven Woolf wrote in JAMA on "The meaning of translational research and why it matters." Woolf observed that the term was being used for two quite different enterprises. For laboratory scientists and the pharmaceutical industry, translation meant turning discoveries into drugs and devices, work that depended on molecular biology, animal models and early-phase trials. For health services researchers, translation meant ensuring that proven interventions actually reached patients, work that depended on epidemiology, behavioural science, organisational research and policy analysis. The two groups competed for the same money under the same label while needing entirely different expertise, infrastructure and timelines. Woolf's warning was that the first enterprise, being closer to the traditional interests of academic medicine, would tend to absorb the funds intended for both. The fullest version of the framework came from genomic medicine and epidemiology. In 2007, Muin Khoury and colleagues at the US Centers for Disease Control and Prevention, writing in Genetics in Medicine, described a continuum of translation research for genomic discoveries running from T1 through T4. In 2010, Khoury, Marta Gwinn and John Ioannidis extended it in the American Journal of Epidemiology in a paper titled "The emergence of translational epidemiology: from scientific discovery to population health impact." They added a T0 stage for the discovery research that precedes all application, and argued that epidemiology had roles to play at every stage, not just the last. The resulting five-stage scheme is now the most common reference point. As Table 1 sets out, each stage answers a distinct question, uses characteristic kinds of study, and ends with a handoff that the next stage depends on. Table 1. The five translational stages and their handoffs. Stage Central question Typical studies What it hands forward T0 Is there a mechanism or target worth pursuing? Laboratory, animal, genomic and observational discovery research A candidate intervention and a rationale T1 Is it safe and active in humans, and at what dose? First-in-human, Phase I, early Phase II, proof of mechanism A dose, a safety profile, early signals T2 Does it work in patients, and should it be recommended? Phase II and III trials, systematic reviews, guidelines Evidence of efficacy and a recommendation T3 Is it adopted and delivered well in real practice? Implementation, dissemination and effectiveness research A deliverable programme and its uptake T4 Does it improve population health, and for whom? Outcomes research, policy evaluation, surveillance Population impact and policy Other institutions draw the lines differently. The US National Center for Advancing Translational Sciences, established in December 2011, describes a spectrum from basic research through pre-clinical, clinical, clinical implementation and public health research, and deliberately depicts it without arrows so as to emphasise that each stage can feed any other. Some schemes fold Phase II and Phase III into T1; others place guideline development in T3. These are not trivial differences, because the labels determine which funders and which review panels consider an application. But they share the essential insight: the journey from discovery to population health has several distinct segments, and a problem that is solved in one segment can remain entirely unsolved in the next. Why the joints matter more than the stages It is tempting to read the T-stages as a production line, in which each station does its work and passes the product along. The metaphor misleads in two ways, and both matter for everything that follows. First, the stations are not run by the same organisation. On a factory line, a single management can see that the paint shop is producing parts that the assembly shop cannot use, and fix it. In translational science, T0 is conducted largely by academic laboratories funded by research councils and rewarded by publications in high-impact journals. T1 and T2 for drugs are conducted largely by companies, regulated by agencies such as the US Food and Drug Administration and the European Medicines Agency, and rewarded by market approval. T3 is conducted by health systems, professional societies and a relatively small community of implementation researchers, and is rewarded, if at all, by quality metrics and payment incentives. T4 is conducted by public health agencies and legislatures and is rewarded by political and electoral considerations. No single actor is responsible for the whole line, and none is rewarded for the quality of what it hands forward. Second, each stage defines success in its own terms, and those terms do not always serve the next stage. A preclinical paper succeeds if it is published and cited. A Phase I study succeeds if it identifies a dose that can be carried forward without unacceptable toxicity. An efficacy trial succeeds if it achieves statistical significance on its primary endpoint. An implementation programme succeeds if the intervention is adopted. A policy succeeds if it is enacted. Each of these definitions is reasonable in isolation. Each can be satisfied while failing the stage that follows. A published mechanism can be unreproducible. A tolerable dose can be far higher than necessary. A statistically significant result can depend on a population that looks nothing like the patients who will use the treatment. An adopted programme can be delivered so poorly that it has no effect. An enacted policy can go unevaluated. The consequence is that the most useful way to read the T-framework is as a map of handoffs, and to ask of each joint three questions. What does the receiving stage need to know? Who, at the sending stage, is responsible for finding it out? And what would it cost them to do so? When the answers are "a great deal," "nobody," and "more than they are rewarded for," the joint will leak. Much of the history of translational science since 2003 can be read as a series of attempts to change those answers, by building infrastructure that spans stages, by changing reporting and design standards, and by creating incentives for the sending stage to care about what happens downstream. The seventeen-year problem, measured properly The best-known summary statistic in the field, the seventeen-year lag estimated by Balas and Boren in 2000, was derived by adding together estimates of the time taken by successive steps: the delay from submission to publication, from publication to inclusion in reviews and textbooks, and from there to implementation. It is a composite of averages taken from different studies, which is why Morris, Wooding and Grant, in their 2011 review, found that estimates of time lags in the literature varied enormously and depended on how each study defined its start and end points. Their review is worth dwelling on because it demonstrates the value of the joint-by-joint view. Some lags are between stages, as when a positive trial result waits years for a guideline to incorporate it. Some are within stages, as when a trial takes years to recruit. Some are parallel, as when regulatory review and guideline development proceed simultaneously. Speeding up translation requires knowing which of these dominates for a given kind of intervention, and the answer is different for a cancer drug, a surgical technique, a vaccine and a behavioural programme. There is no single number because there is no single path. What the detailed studies do show, consistently, is that loss is at least as important as delay. The Contopoulos-Ioannidis study described in the introduction found that of 101 highly promising discoveries, only one reached wide clinical use in twenty years. That is not a story of slow progress along a path; it is a story of nearly everything falling off. At the other end of the pipeline, Elizabeth McGlynn and colleagues at RAND, in a 2003 study in the New England Journal of Medicine based on medical records and telephone interviews with adults in twelve US metropolitan areas, found that participants received about 55 percent of the care recommended for their conditions. The failure there is not that the evidence was slow to arrive. It had arrived, been summarised, and been turned into quality indicators. It was simply not being acted upon about half the time. One discovery, five joints The value of the handoff view is clearest when a single intervention is followed all the way along the path. The human papillomavirus vaccine is one of the few for which the whole journey, from laboratory discovery to measured population effect, is now documented, and each of its joints taught a different lesson. At T0, the decisive work was the identification of specific papillomavirus types in cervical cancer tissue. Harald zur Hausen's group in Germany reported HPV16 in 1983 and HPV18 in 1984, against a prevailing view that a herpesvirus was the more likely culprit; the work later earned him a share of the 2008 Nobel Prize in Physiology or Medicine. Establishing that a virus was a necessary cause of most cervical cancers turned a disease of uncertain origin into a potentially preventable infection. But a causal virus is not a vaccine. The critical technical step came in the early 1990s, when Ian Frazer and Jian Zhou at the University of Queensland, and independently groups at the US National Cancer Institute and elsewhere, showed that the viral L1 capsid protein could assemble itself into virus-like particles that carried no genetic material yet provoked a strong antibody response. At T1 and T2, the question became whether those particles were safe, immunogenic and protective in people. A problem arose that recurs in many prevention programmes: the outcome that mattered, invasive cancer, takes decades to develop. The trials therefore used high-grade precancerous lesions as their endpoint, which was defensible biologically but meant that the claim to prevent cancer rested, at licensure, on a surrogate. The quadrivalent vaccine was approved in the United States in June 2006 on the strength of trials such as FUTURE II, reported in the New England Journal of Medicine in 2007. At T3, the programme met a different kind of obstacle. The vaccine worked best if given before sexual debut, which meant vaccinating children and young adolescents against a sexually transmitted infection, an idea that provoked resistance in some countries. Delivery systems mattered enormously: Australia, which began a publicly funded school-based programme in 2007, achieved high coverage quickly, while countries relying on opportunistic vaccination in primary care lagged for years. The same vaccine, with the same efficacy, reached very different fractions of its intended population depending on how it was delivered. At T4, the long-awaited population evidence eventually arrived. A Swedish registry study by Jiayao Lei and colleagues in the New England Journal of Medicine in 2020, covering more than 1.6 million girls and women, found that those vaccinated before the age of 17 had an incidence of invasive cervical cancer close to 90 percent lower than those who were not vaccinated. Scottish data published in 2024 found no cases of invasive cervical cancer among women who had been fully immunised at 12 or 13 in the routine programme. These findings in turn fed policy: the World Health Organization launched a global strategy to eliminate cervical cancer as a public health problem in 2020, and in 2022 its advisory group concluded that a single dose could offer protection comparable to two or three, a change that dramatically lowers the cost of reaching low-income countries. Each joint in this story required a different kind of expertise and was owned by a different set of institutions. At each one, the intervention could have stalled, and in some countries it did. None of the difficulties at T3 or T4 could have been solved by better science at T0. That is the practical meaning of saying that translation fails at the joints. Two directions of travel One further limitation of the linear picture needs to be stated at the outset, because it will recur throughout the book. Knowledge does not flow only from bench to bedside. It also flows back. Some of the most consequential discoveries in medicine began with an observation in patients that sent researchers back to the laboratory. The recognition that a particular chromosomal abnormality was present in the white cells of patients with chronic myeloid leukaemia, first described by Peter Nowell and David Hungerford in Philadelphia in 1960, began at the microscope with patient samples. It took decades of laboratory work to identify the fusion gene responsible and the abnormal enzyme it produced, and then a targeted drug, before imatinib came back to patients at the end of the 1990s. Clinical trials that fail are another source of reverse flow: a well-designed negative trial can reveal that the mechanism studied in animals is not the one operating in humans, and send the question back to T0. Population surveillance at T4 can reveal an unexpected safety signal, a disparity in benefit, or an unanticipated effect that generates new hypotheses for the laboratory. This is why NCATS removed the arrows. The practical significance is that a good translational system does not merely push discoveries forward. It also has channels, and people, whose job is to carry questions backwards, and it treats the information in failures as seriously as the information in successes. Chapter 8 returns to this in detail. What the framework is for A framework of this kind earns its keep if it helps people make better decisions. For a laboratory scientist, the T-framework is a reminder that a discovery's value depends on questions that will be asked at later stages, and that a study designed with those questions in mind is worth more than one designed only to be published. For a clinical investigator, it is a reminder that the trial population, the endpoints and the form of the intervention all determine whether the result will be usable in practice. For a funder, it is a way of seeing where the pipeline is thinnest: historically, far more money has gone to T0 than to T3 and T4 combined, although the precise balance is hard to measure because funders classify their portfolios differently. For a policymaker, it is a way of recognising that evidence of efficacy is not evidence of population effect, and that the absence of population evidence may reflect the absence of anyone paid to generate it. The chapters that follow take the joints in order. The first, between discovery and the decision to test in humans, is where the greatest number of candidates are lost and where the scientific problems are, in some ways, the most fundamental. It is also the joint where the incentives of the sending stage, academic discovery science, are least aligned with the needs of the receiving one. Chapter 2. The First Joint: Why Promising Discoveries Fail to Travel In 2012, C. Glenn Begley, who had spent a decade as head of global cancer research at the biotechnology company Amgen, and Lee Ellis of the MD Anderson Cancer Center published a short comment in Nature that has been cited thousands of times since. Over the preceding decade, Begley's team had attempted to confirm the findings of 53 papers they considered landmark studies in preclinical cancer research, before committing resources to drug programmes built on them. They were able to confirm the scientific findings in only six, about eleven percent. A year earlier, Florian Prinz and colleagues at Bayer had reported in Nature Reviews Drug Discovery that in-house attempts to reproduce published data on potential drug targets had matched the published results in only about a fifth to a quarter of projects. These industry reports had limitations. The companies did not publish which papers they had tested or exactly how, so the claims themselves could not be checked, which is an irony the critics did not fail to note. But they were soon joined by more transparent evidence. The Reproducibility Project: Cancer Biology, a collaboration between the Center for Open Science and Science Exchange, set out to repeat key experiments from high-profile cancer biology papers published between 2010 and 2012. It reported its final results in eLife in 2021. Of the 193 experiments originally selected, the team was able to complete only 50, from 23 papers, in large part because the original papers did not report enough methodological detail and the original authors could not or would not supply it. Among the experiments that were completed, the replication effect sizes were, at the median, about 85 percent smaller than those originally reported. For translational science, these findings describe the condition of the first joint. When a company, a funder or an academic team decides to take a discovery towards human testing, it is relying on a body of preclinical evidence that is, on this evidence, often considerably weaker than it appears. The decision to move to T1 is expensive, and it is taken on the basis of information that the sending stage was not designed to make reliable. Where the weakness comes from It would be comforting to attribute the problem to misconduct, but fraud accounts for a small part of it. Most of the weakness has ordinary causes that operate on honest researchers. The first is design. Preclinical experiments, especially in animals, have historically been small, often unrandomised, and rarely blinded. A series of systematic reviews led by Malcolm Macleod, Emily Sena and colleagues in the CAMARADES collaboration, looking across animal studies of stroke and other conditions, found that studies which did not report randomisation or blinded outcome assessment tended to report larger treatment effects than those which did. That pattern is exactly what one would expect if unconscious bias in allocating animals or scoring outcomes inflates apparent effects. The second is publication bias. Sena and colleagues, in a 2010 paper in PLoS Biology, analysed data from animal studies of stroke and estimated that publication bias alone accounted for roughly a third of the apparent efficacy reported in that literature. Studies with null results are less likely to be written up, less likely to be accepted, and less likely to be cited, so the visible evidence is a selected sample of the more favourable results. The third is analytic flexibility. When an experiment can be analysed in several defensible ways, and the researcher chooses among them after seeing the data, the chance of finding a statistically significant result by chance rises well above the nominal five percent. John Ioannidis's much-discussed 2005 essay in PLoS Medicine, "Why most published research findings are false," made the general argument that small studies, small effects, flexible designs and fields with many competing teams all reduce the probability that a published positive finding is true. Preclinical biology has historically combined all of these. The fourth is the reward structure. A laboratory scientist is rewarded for novelty. A striking result in a high-impact journal can secure grants, promotions and a career. A careful confirmation of someone else's finding, or a well-conducted negative result, rarely does. The sending stage at the first joint is therefore systematically incentivised to produce exactly the kind of evidence the receiving stage finds least reliable: novel, dramatic, underpowered and unconfirmed. Leonard Freedman, Iain Cockburn and Timothy Simcoe attempted to put a price on the consequences. Their 2015 paper in PLoS Biology, drawing on published estimates of irreproducibility rates, suggested that about half of US preclinical research might be irreproducible and that the cost of that irreproducible research was on the order of 28 billion dollars a year. The authors were explicit that the estimate was rough, and the underlying irreproducibility rates are themselves contested. The order of magnitude, however, gives a sense of what is at stake at this joint. The validity problem Even perfectly reproducible preclinical findings can fail to translate, because a reproducible result in a model is not the same as a true result in humans. This is the problem of external or predictive validity, and it is at its most acute with animal models of complex human diseases. Stroke provides the most studied example. Victoria O'Collins and colleagues, in a 2006 systematic review in the Annals of Neurology, catalogued 1,026 experimental treatments for acute ischaemic stroke that had been tested in animals. Of these, 114 had been tested in patients. Only thrombolysis with tissue plasminogen activator had established itself as effective, and aspirin had a modest role. One compound, the free-radical trapping agent NXY-059, became a cautionary tale: it met the field's own recommended criteria for preclinical evidence, produced a marginally positive result in the SAINT I trial, and then showed no benefit in the larger SAINT II trial reported in the New England Journal of Medicine in 2007. Subsequent analyses of its preclinical record found that the animal studies had been small, had often used young, otherwise healthy animals rather than the elderly, hypertensive and diabetic population that suffers most strokes, and had often given treatment soon after the stroke rather than hours later as happens in clinical practice. Sepsis is another. In 2013, Junhee Seok and colleagues, in a large collaborative study in the Proceedings of the National Academy of Sciences, compared genomic responses to trauma, burns and endotoxin in humans and in the mouse models commonly used to study them, and reported that the mouse responses correlated poorly with the human ones. The paper was widely read as an indictment of mouse models of inflammation. A reanalysis of the same data by Keizo Takao and Tsuyoshi Miyakawa, published in the same journal in 2015, reached the opposite conclusion, finding substantial correspondence when the analysis focused on genes that changed significantly in both species. The dispute is instructive in itself. Whether a model is "valid" depends on which features of the disease one needs it to reproduce, and that can only be judged in relation to the specific therapeutic question being asked. Alzheimer's disease offers perhaps the starkest record. Jeffrey Cummings, Travis Morstorf and Kate Zhong, in a 2014 analysis in Alzheimer's Research & Therapy, examined drug-development programmes registered between 2002 and 2012 and calculated a failure rate of 99.6 percent. Many candidates had cleared amyloid or improved cognition in transgenic mice that carried human familial mutations, a form of the disease accounting for a small minority of patients. The eventual approval of antibodies such as lecanemab, which produced modest slowing of decline in large trials, came after two decades of failures, and the debate over how much benefit those drugs provide is still active. The general lesson is that a model's validity has several components that are easy to conflate. Face validity is whether the model looks like the disease. Construct validity is whether it is produced by the same causal mechanism. Predictive validity is whether interventions that work in the model work in patients. A model can have high face validity and poor predictive validity, as some stroke models appear to, or can be useful for one question and useless for another. The first joint leaks badly when models are chosen for convenience and familiarity rather than for their demonstrated ability to predict the human outcome that the next stage will measure. The valley of death To the scientific problems at the first joint must be added an economic one. A discovery made in an academic laboratory is typically far from being a product. Before it can be tested in humans, a candidate drug must be optimised for potency and selectivity, formulated, manufactured to a standard fit for human use, and put through a package of safety pharmacology and toxicology studies designed to satisfy regulators. These activities are expensive, unglamorous, and poorly suited to academic funding, which rewards hypotheses and publications rather than development. Yet they are too risky, at this early stage, to attract most commercial investment. The resulting gap has been called the valley of death since at least the early 2000s, and Declan Butler's 2008 feature in Nature, "Translational research: crossing the valley of death," helped to fix the phrase. Many public programmes since have been designed specifically to bridge it. In the United States, NCATS runs programmes that provide academic investigators with access to industrial-style drug-development expertise, and its Therapeutics for Rare and Neglected Diseases programme was created to advance candidates that no company was likely to pursue. Several universities have established their own drug-discovery units. Venture philanthropy, in which patient foundations fund early development directly, has also played an important role, most famously when the Cystic Fibrosis Foundation invested in the research that led to the drug ivacaftor. The valley of death is not only about money. It is also about knowledge. A laboratory scientist who has discovered a promising target may not know what regulators will require before a first-in-human study, what properties will make a molecule a viable drug, or what the clinical community needs to see before it will enrol patients. The bridging programmes that have worked best supply that knowledge as much as they supply money. Designing the first joint to hold Over the past fifteen years, the response to these problems has taken several forms, and they share a common logic: they make the sending stage accountable for what the receiving stage needs. The first is reporting standards. The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments), first published by Carol Kilkenny and colleagues in PLoS Biology in 2010 and revised as ARRIVE 2.0 by Nathalie Percie du Sert and colleagues in 2020, set out what information an animal study needs to report for readers to judge its reliability, including sample-size calculation, randomisation, blinding and the handling of excluded animals. A 2012 paper in Nature by Story Landis and colleagues, arising from a US National Institute of Neurological Disorders and Stroke workshop, called for a core set of transparent-reporting standards along similar lines. Journals and funders have adopted these standards unevenly, and adherence has lagged endorsement, but the direction is clear. The second is funder requirements. From 2016, the NIH began requiring grant applicants to address the rigour of the prior research on which their proposals rested, the rigour of their own designs, the consideration of sex as a biological variable, and the authentication of key resources such as cell lines and antibodies. The sex-as-a-biological-variable policy, announced by Janine Clayton and Francis Collins in Nature in 2014, responded to evidence that preclinical research had relied disproportionately on male animals and cells, so that effects in females were often simply unknown when a product entered human testing. The third is preregistration and confirmatory design. Borrowing from clinical trials, some preclinical researchers now register their hypotheses and analysis plans before collecting data, which removes the analytic flexibility that inflates effect sizes. A more ambitious approach is the multicentre preclinical randomised trial, in which a promising intervention is tested in parallel in several independent laboratories using a common protocol. A 2015 study by Gemma Llovera and colleagues in Science Translational Medicine, which tested an anti-CD49d antibody in experimental stroke across six European centres, showed both that such trials are feasible and that they can overturn findings from single laboratories: the effect was seen in one stroke model but not in another. The fourth is a change in the models themselves. Human-derived systems, including organoids grown from patient stem cells, microphysiological "organ-on-a-chip" devices, and computational models, promise in some applications to predict human responses better than animals do. Regulators have begun to accommodate them. The FDA Modernization Act 2.0, signed into US law in December 2022, removed the statutory requirement that a drug be tested in animals before human trials, allowing sponsors to use other methods where they are adequate. In April 2025, the FDA announced a plan to phase out animal testing requirements for monoclonal antibodies and other drugs, in favour of what it called new approach methodologies. How quickly this changes practice remains to be seen, and for many questions, especially those involving the whole-body interaction of a drug with multiple organs over time, no current alternative fully substitutes for an animal study. One further development deserves separate mention because it reverses the usual direction of evidence at this joint. Instead of starting with a mechanism in a model and hoping it applies to people, researchers increasingly start with evidence from people. Large genetic studies can show that individuals who naturally carry variants disabling a particular gene have lower rates of a disease, which is a kind of natural experiment on what would happen if a drug inhibited that gene's product. The development of PCSK9 inhibitors for lowering cholesterol followed this path: families with gain-of-function mutations in PCSK9 had very high cholesterol, and people with loss-of-function variants had low cholesterol and fewer heart attacks, before any drug had been tested. Matthew Nelson and colleagues at GlaxoSmithKline, in a 2015 analysis in Nature Genetics, found that drug targets with this kind of human genetic support were roughly twice as likely to succeed in development as those without it, and a later analysis by Eric Minikel and colleagues in Nature in 2024 reached a similar conclusion with more data. Human genetic evidence does not replace laboratory work, but it answers, before the first joint is crossed, a question that animal models answer poorly: does modulating this target matter in humans at all? Finally, some of the most important changes are in how decisions are made at the joint. Several pharmaceutical companies now routinely attempt to reproduce key academic findings internally before investing in a programme. AstraZeneca, after a review of its own pipeline published by David Cook and colleagues in Nature Reviews Drug Discovery in 2014, adopted what it called the "5R" framework, asking whether a project had the right target, right tissue, right safety, right patient and right commercial potential, and reported that its later success rates improved. The right patient criterion is especially significant: it asks, at the earliest stage, which people the drug will eventually be tested in, and whether the preclinical evidence speaks to them. That is exactly the kind of downstream question that the first joint has historically neglected. What the first joint hands forward When the first joint works, what passes across it is not merely a molecule and a hypothesis. It is a candidate with reproducible evidence of activity in models chosen for their relevance, an understanding of how it is absorbed, distributed and eliminated, a toxicology package that identifies the organs most at risk, and a biomarker or other measure that can show, in the first human studies, whether the drug is engaging its target. It is also a clear statement of uncertainty: what the preclinical evidence does not show, and therefore what the first human studies will need to find out. Most candidates, even after all this, will fail in humans. But the goal at the first joint is not to eliminate failure; it is to make failure informative and to make it happen early, when it is cheap. A candidate that fails in Phase I because it does not engage its target in humans has taught the field something. A candidate that fails in Phase III because the animal effect was an artefact of unblinded scoring has wasted years and, more seriously, exposed patients to risk for no scientific return. The next chapter follows the candidate across the joint, into the first human beings who will receive it. Chapter 3. Crossing Into Humans: The Logic and Risk of Phase I On the morning of 13 March 2006, eight healthy young men were dosed in a private clinical research unit at Northwick Park Hospital in north-west London. They were the first human beings to receive TGN1412, a monoclonal antibody designed to activate a subset of immune cells by binding to a receptor called CD28. Six received the drug and two received a placebo. The doses were given at intervals of about ten minutes. Within ninety minutes, the six who had received the active drug began to experience severe headache, back pain and fever. Within hours, they had developed what Ganesh Suntharalingam and colleagues, reporting the case in the New England Journal of Medicine later that year, described as a cytokine storm: a massive release of inflammatory signalling molecules leading to multi-organ failure. All six were admitted to intensive care. All survived, but some suffered lasting harm. The dose they had received, 0.1 milligrams per kilogram of body weight, was about five hundred times lower than the highest dose that had caused no adverse effects in cynomolgus monkeys. By the conventional rules for selecting a first human dose, it was cautious. The problem was that the conventional rules assumed that toxicity in animals would predict toxicity in humans, and for this drug, acting on this target, it did not. Subsequent investigation suggested that the immune cells most responsible for the reaction in humans did not express CD28 in the same way in the monkeys used for testing. TGN1412 is the single most influential event in the modern history of first-in-human research, and it illustrates precisely what Phase I is for. It is the stage at which the entire preclinical case is put to its first real test, and where the gap between models and humans becomes, for the first time, directly visible in a living person. What Phase I is trying to learn The classical purpose of a Phase I study is to establish the safety and tolerability of a new intervention in humans, to characterise how the body handles it (its pharmacokinetics) and what it does to the body (its pharmacodynamics), and to identify a dose or range of doses for further study. For most drugs outside oncology, these studies are conducted in healthy volunteers, usually in specialised units, with single ascending doses given to successive small cohorts, followed by multiple ascending doses given over several days. In oncology, the logic is different. Cancer drugs have historically been cytotoxic, damaging healthy cells as well as tumours, and it would be unethical to expose healthy people to them. Oncology Phase I trials are therefore conducted in patients with advanced cancer who have usually exhausted standard options. That changes the ethical calculus: participants may hope for benefit, and the studies have a therapeutic as well as a scientific character. It also changes the design logic, because for cytotoxic drugs the working assumption was that more drug meant more effect, so the goal was to find the highest dose patients could tolerate, known as the maximum tolerated dose. The regulatory scaffolding for Phase I is broadly similar across major jurisdictions. In the United States, a sponsor must submit an Investigational New Drug application to the FDA, including preclinical pharmacology and toxicology data, manufacturing information and a clinical protocol, and may proceed after thirty days unless the agency places the study on hold. In the European Union, clinical trials are now authorised through a single application under the Clinical Trials Regulation, which became applicable in January 2022. In both systems, an independent ethics committee or institutional review board must also approve the protocol, and participants must give informed consent. Choosing the first dose The most consequential single decision in a first-in-human study is the starting dose. Too low, and many cohorts of volunteers will be exposed to a drug at doses that cannot possibly do anything, wasting time and money and raising its own ethical questions. Too high, and the first cohort may be harmed. The traditional approach, codified by the FDA in a 2005 guidance on estimating the maximum safe starting dose in healthy volunteers, begins with the no observed adverse effect level (NOAEL) in the most appropriate animal species, converts it into a human equivalent dose using scaling factors based on body surface area, and then divides by a safety factor, by default ten, to give a maximum recommended starting dose. This approach rests on toxicity: it asks how much drug can be given before something goes wrong in animals. After TGN1412, the UK government convened an Expert Scientific Group, chaired by Gordon Duff, whose report in December 2006 recommended that for high-risk agents, particularly those acting on the immune system with a novel mechanism, the starting dose should instead be based on the minimal anticipated biological effect level, or MABEL. This approach asks not how much drug causes toxicity but how much drug is needed to produce any pharmacological effect at all, using all available data on how the drug binds to its target and what fraction of receptors it occupies at a given concentration. For TGN1412, a MABEL-based starting dose would have been very much lower than the one used. The European Medicines Agency issued a guideline on first-in-human trials for potential high-risk medicinal products in 2007, incorporating MABEL, and substantially revised it in 2017. The other lessons of TGN1412 concerned the conduct of the study. All six active participants had been dosed before any had shown symptoms, because the dosing interval was shorter than the time the reaction took to develop. The revised guidance now emphasises sentinel dosing, in which one or two participants receive the drug first and are observed for an appropriate period before the rest of the cohort is dosed, and dosing intervals chosen on the basis of the drug's expected pharmacology. It also emphasises that studies of high-risk agents be conducted in units with immediate access to intensive care. The second warning: BIA 10-2474 In January 2016, a Phase I trial in Rennes, France, of BIA 10-2474, a drug that inhibits the enzyme fatty acid amide hydrolase, was stopped after one participant was declared brain dead and several others were hospitalised with neurological damage. The affected participants were in a multiple-ascending-dose cohort receiving 50 milligrams a day, the highest dose tested. Unlike TGN1412, the drug had not caused any comparable effect in animals at the doses tested, and other drugs in the same class had been given to humans without serious harm. Investigations by the French medicines agency and an expert committee identified several contributing concerns: the steep escalation between the previous cohort and the one in which harm occurred, doses far above those needed to fully inhibit the target enzyme, and the possibility that the drug had off-target effects on other enzymes in the brain. The episode reinforced a lesson that TGN1412 had already taught. The data that matter for dose escalation are not just whether any adverse effects have been seen, but whether each further increase in dose is actually needed. If the drug has already fully engaged its target at a lower dose, going higher adds risk without adding any therapeutic information. The 2017 revision of the EMA guideline, which was prepared with the Rennes events in mind, emphasises integrating pharmacokinetic, pharmacodynamic and target-engagement data into escalation decisions. Rules for climbing the dose ladder In oncology, where Phase I studies are conducted in patients and the aim has traditionally been to find the maximum tolerated dose, the design of dose escalation has been a subject of intense statistical debate. The approaches in widest use differ in how they decide whether to increase, hold or reduce the dose for the next cohort, and in how efficiently they identify the right dose, as Table 2 summarises. Table 2. Common dose-escalation designs in early oncology trials. Design How escalation is decided Main strength Main weakness 3+3 Fixed rules based on toxicities in cohorts of three Simple, familiar, needs no statistician at the bedside Often misidentifies the target dose; treats many patients at low doses Continual reassessment method (CRM) A statistical model of dose and toxicity, updated after each cohort Uses all accumulated data; identifies the target dose more accurately Requires modelling expertise; perceived as opaque Bayesian optimal interval (BOIN) Compares the observed toxicity rate at the current dose with pre-set boundaries Nearly as accurate as model-based designs; simple to run Still focused on toxicity rather than benefit The 3+3 design, which dates in essence from the 1970s and 1980s, treats three patients at a dose; if none experiences a dose-limiting toxicity, the next three receive a higher dose; if one does, three more are added at the same dose; if two or more do, escalation stops and the dose below is usually declared the maximum tolerated dose. It is transparent and easy to run. It is also, according to a large body of simulation studies, poor at identifying the dose it aims to find, and it treats a large fraction of patients at doses well below any likely to be effective. The continual reassessment method, introduced by John O'Quigley, Margaret Pepe and Lloyd Fisher in Biometrics in 1990, fits a statistical model relating dose to the probability of toxicity and updates it after each patient or cohort, recommending the dose whose estimated toxicity is closest to a target rate. It uses information more efficiently and more patients are treated near the target dose. Its adoption was slowed by concerns, some justified in its earliest versions, that the model could escalate too aggressively, and by the practical need for statistical support throughout the trial. The Bayesian optimal interval design, described by Suyu Liu and Ying Yuan in 2015, occupies a middle ground: it uses pre-calculated decision boundaries that can be written down in a table at the start of the trial, so that it is as easy to run as the 3+3 design while performing much closer to model-based methods. When the maximum is not the optimum All three designs share an assumption inherited from cytotoxic chemotherapy: that the right dose is the highest dose patients can tolerate. For many modern cancer drugs, that assumption is wrong. Targeted therapies that inhibit a specific enzyme may achieve full target inhibition at doses well below those that cause intolerable side effects. Immunotherapies may have flat dose-response relationships across a wide range. For such drugs, pushing to the maximum tolerated dose adds toxicity without adding benefit, and patients who take the drug for months or years may suffer side effects that lead them to reduce doses or stop treatment altogether. The FDA's Oncology Center of Excellence launched Project Optimus in 2021 to address this, and in 2024 published final guidance on optimising the dosage of oncology drugs. It asks sponsors to compare multiple doses, often in randomised fashion, before committing to a registrational trial, and to characterise the relationships between dose, exposure, activity and safety. A frequently cited illustration is sotorasib, a drug targeting a mutated form of the KRAS protein, which received accelerated approval in May 2021 at a dose of 960 milligrams daily. As a condition of approval, the FDA required a post-marketing trial comparing that dose with a quarter of it, reflecting uncertainty about whether the lower dose would be as effective with fewer side effects. This is a case where the handoff at the second joint, from Phase I to later development, had been designed around the wrong question. A Phase I design optimised to find the highest tolerable dose was handing forward a dose that later stages, and patients, did not need. The correction required changing what Phase I was asked to deliver. The ethics of being first The ethical issues in Phase I differ between healthy-volunteer studies and patient studies, but both concern the relationship between risk and benefit when the participant is the first human being to take a drug. For healthy volunteers, there is no prospect of direct benefit, and participation is typically paid. The concern is whether payment induces people to accept risks they would otherwise refuse, and whether the population of habitual paid volunteers, often economically marginal, bears a disproportionate share of the burden of drug development. The empirical record on harm is, on the whole, reassuring. Ezekiel Emanuel and colleagues, analysing data from healthy-volunteer Phase I studies conducted by a large pharmaceutical company in a 2015 paper in the BMJ, found that serious adverse events were rare and most adverse events were mild. The rarity of disasters such as TGN1412 and BIA 10-2474 is part of what makes them so instructive. For patients in oncology Phase I trials, the concern is different: whether patients understand that the primary purpose of the trial is to find a dose, not to treat them, and whether the prospect of benefit is realistic. Elizabeth Horstmann and colleagues, in a 2005 analysis in the New England Journal of Medicine of 460 Phase I oncology trials conducted between 1991 and 2002, found an overall response rate of 10.6 percent and a rate of death attributed to toxicity of 0.49 percent, with response rates higher in trials that included an established anticancer agent. More recent analyses have reported higher response rates in the era of targeted therapy. These data have been used to argue that Phase I participation offers more potential benefit than had often been assumed, and that describing these trials as purely non-therapeutic in consent discussions is inaccurate. They have also been used to argue that the true chance of benefit varies so widely by drug and setting that honest consent requires trial-specific information, not general reassurance. A different response to the ethics of first-in-human exposure is to reduce what needs to be learned at full dose. The FDA's 2006 guidance on exploratory Investigational New Drug studies allowed very limited early human studies, sometimes called Phase 0, in which subtherapeutic microdoses are given to a small number of people to learn about how a drug is distributed or whether it reaches its target. Such studies cannot establish safety or efficacy, but they can end a programme early, at low risk, if the drug behaves in humans very differently from how it behaved in animals. What Phase I should hand forward The standard output of Phase I, a recommended dose and a list of observed adverse events, is necessary but not sufficient. The next stage needs to know not just what dose can be tolerated but whether the drug reached its target, whether it engaged it, and whether that engagement produced the expected biological effect. These three questions, sometimes called the three pillars of survival in drug development, were articulated in a 2012 analysis by Paul Morgan and colleagues at Pfizer in Drug Discovery Today. Reviewing the company's Phase II programmes, they found that many failures were in programmes where it had never been established that the drug had reached and engaged its target at the doses tested. Without that information, a negative Phase II result cannot distinguish between a drug that does not work and a drug that was never given a fair chance. A Phase I study designed with the next stage in view therefore measures pharmacodynamic biomarkers alongside safety, explores more than one candidate dose, and characterises how exposure varies between people. It hands forward not just a number but an understanding of the dose-exposure-response relationship on which every later decision will depend. That understanding becomes indispensable in the stage the next chapter examines, where most candidates that survive Phase I will fail. Hashtags: #TranslationalScienceFundamentals #TranslationalScience #BenchToBedside #TranslationalResearch #T0ToT4 #PreclinicalResearch #FirstInHumanStudies #PhaseITrials #ClinicalTrials #ImplementationScience #DisseminationScience #PopulationHealth #Reproducibility #PredictiveValidity #ValleyOfDeath #DrugDevelopment #DoseOptimization #Pharmacokinetics #Pharmacodynamics #TargetEngagement #EvidenceToPractice #ImplementationResearch #TranslationalMedicine #ClinicalResearch #FutureOfTranslationalScience

Download the Book (PDF): Introduction The first review most scientists write arrives without ceremony. An email from an editor they have never met, a manuscript title that sits somewhere near their own work, a deadline two or three weeks away, and a link to a submission system with a text box and a drop-down menu of recommendations. There is rarely any training attached. The graduate student or postdoctoral researcher who clicks "accept invitation" is expected to know, somehow, what the editor wants, what the authors deserve, what counts as a fatal flaw and what counts as a matter of taste, and how to say all of it in prose that will be read by strangers who have spent years on the work in question. Most early-career reviewers fill this gap by imitation. They recall the reviews they have received themselves, the sharp ones that stung and the lazy ones that missed the point, and they write something in between. Some become harsh, on the theory that rigour and severity are the same thing. Some become timid, afraid that a junior person has no standing to criticise a senior laboratory. Many produce long lists of small objections, because small objections are easy to find and feel like diligence. Very few are ever told whether their reviews were any good. This book is written for that reviewer. It treats peer review not as a single act of judgement but as an ecosystem: a set of roles, each with distinct obligations, that together produce the decisions which shape what enters the scientific record. Authors, reviewers, handling editors, editors-in-chief, publishers, research integrity officers, readers who comment after publication, and increasingly the platforms that publish reviews openly all occupy positions in this system. A reviewer who understands only their own position will misjudge what their report is for. A reviewer who understands the whole system can write a report that does its job. The argument The controlling claim of this book is simple to state and harder to practise. The ethics of peer review and the craft of peer review are the same subject. Fairness, confidentiality, disclosure of conflicts, and honesty about the limits of one's own expertise are usually taught, when they are taught at all, as a compliance layer: rules to observe before the real intellectual work begins. That framing is wrong. A review distorted by an undeclared rivalry is not an ethical failure sitting beside a technically excellent assessment; it is a bad assessment. A review that hides its uncertainty behind confident language misleads the editor about the evidence. A review that is contemptuous in tone gets its substantive points ignored, which means the flaws it found are more likely to survive into print. Constructive critique is not critique with the edges sanded off. It is critique that is accurate about the work, accurate about the reviewer's own position, and aimed at a decision someone else must make. Three commitments follow from that claim, and they run through every chapter. The first is that a review is written for two audiences with different needs. The editor needs help making a decision: is this work sound, is it important enough for this venue, and what would it take to fix? The authors need help improving the work, whatever the decision. A report that serves only one of these audiences is half a report. The second is that the reviewer is a witness, not a judge. Reviewers advise; editors decide. This matters practically, because it changes how a recommendation should be framed, and ethically, because it limits what a reviewer is entitled to do when something looks wrong. A reviewer who suspects manipulated data is not the investigator, the prosecutor, or the jury. They are the person who noticed, and their obligation is to report what they noticed clearly and to the right person. The third is that peer review is a human process with known and measurable failure modes. Reviewers disagree with each other far more than most scientists assume. They miss errors deliberately planted in manuscripts. They are swayed by the prestige of authors and institutions. They are, at times, cruel. None of this is a reason for cynicism, but all of it is a reason for method. A reviewer who knows where the process tends to fail can build habits that guard against those failures in their own work. What the chapters do The first two chapters establish the system. Chapter 1 asks what peer review is for, tracing how a practice that most scientists assume is ancient became standard only in the second half of the twentieth century, and summarising what the evidence says about its reliability. Chapter 2 maps the editorial system: who does what between submission and decision, what editors actually need from reviewers, and how a reviewer's report is used once it leaves their hands. Chapters 3 and 4 are about the core craft. Chapter 3 sets out a method for reading a manuscript critically, from the first pass that establishes what the paper claims to the detailed interrogation of design, analysis, and reporting. Chapter 4 turns to writing: how to structure a report, how to separate essential problems from preferences, how to phrase criticism so that it is heard, and how to make a recommendation that respects the editor's role. Chapters 5, 6, and 7 address the ethical terrain directly, though each treats ethics as part of accurate assessment. Chapter 5 covers conflicts of interest and confidentiality, including the newer question of whether a reviewer may feed a confidential manuscript to a generative AI tool. Chapter 6 examines implicit bias: what the experimental evidence shows about status, gender, and confirmation effects in review, and what an individual reviewer can do about biases they cannot see directly. Chapter 7 deals with disputes over data, from suspected image manipulation to authors who refuse to share the data behind their claims, and sets out what a reviewer should and should not do when something looks wrong. Chapter 8 looks at the changing shape of the system itself. Open identities, published reports, reviewed preprints, and portable reviews that travel between journals are no longer experiments at the margins. Several prominent journals now publish reviewer reports routinely, and at least one major life-sciences journal has abandoned accept-or-reject decisions after review altogether. These models change what a review is, who reads it, and what it is worth to the person who wrote it. The conclusion draws these threads into an argument about what an early-career reviewer should actually do differently, and about which problems in the system remain open. What this book leaves out The book concentrates on manuscript review for journals and preprint review platforms, because that is where most early-career scientists begin. Grant review shares many of the same ethical principles, and the discussion of conflicts, bias, and confidentiality applies there with little change, but the mechanics of study sections and funding panels differ enough that they are treated only in passing. The book also does not attempt to cover every discipline's conventions. Its examples come mostly from the life, physical, and social sciences, where the journal article remains the main unit of communication. Reviewers in fields where conference proceedings dominate, such as much of computer science, will find the principles transferable, and some of the most instructive evidence about bias in review comes from exactly those venues. Finally, the book does not offer templates to be filled in. Checklists have a place, and several appear in the chapters where they help, but a review assembled from a checklist without judgement reads like one. The aim is to give a new reviewer a clear enough picture of the system, and of their own role within it, that they can make good judgements in situations no checklist anticipated. A note on standing Early-career scientists often ask whether they have the standing to review at all, particularly when the authors are senior. The answer is that standing in peer review comes from expertise in the specific question at hand, not from seniority. An editor who invites a postdoctoral researcher usually does so because that person has recently worked with the precise technique, dataset, or model system the manuscript depends on, and often knows its pitfalls better than anyone else in the field. The appropriate response to that invitation is neither deference nor bravado but candour: say what you can assess with confidence, say what lies outside your competence, and do the work carefully. That candour is the thread that connects everything that follows. It is what makes a review useful to an editor, fair to authors, and worth the hours it takes to write. Chapter 1. What Peer Review Is For Ask a room of scientists when peer review began and many will point to the seventeenth century. In 1665 Henry Oldenburg, secretary of the Royal Society of London, launched the Philosophical Transactions, and the journal is often described as the origin of the practice. The description is misleading in an instructive way. Oldenburg edited the Transactions as a private venture, and he selected material largely on his own judgement and through his correspondence. The Royal Society did not take formal responsibility for the journal until 1752, and the system of sending papers to members for written reports developed gradually through the nineteenth century. Even then, it was one practice among several. Many journals were run by editors who decided alone or with a small circle of trusted advisers. The historian Melinda Baldwin has shown how recent the modern expectation really is. Nature, for most of its history after its founding in 1869, relied heavily on editorial judgement, and it did not require external refereeing for all research papers until the 1970s. Across much of science, universal external review became standard only in the decades after the Second World War, driven by the explosive growth of research funding, the resulting flood of submissions, and a growing need for scientific institutions to demonstrate accountability to the governments that paid for them. Peer review, in Baldwin's account, became a marker of scientific legitimacy in public life at roughly the same time it became routine inside journals. A well-known episode from 1936 captures the transition. Albert Einstein and Nathan Rosen submitted a paper on gravitational waves to Physical Review, arguing that such waves might not exist. The editor, John Tate, sent it to a referee, whose anonymous report identified a serious error. Einstein, accustomed to the German journals where editors published the work of eminent authors without external review, was indignant that the manuscript had been shown to a colleague before publication. He withdrew it and published elsewhere. The historian Daniel Kennefick later established that the referee was the cosmologist Howard Percy Robertson, and that Robertson subsequently helped Einstein's assistant understand the problem. The published version, when it appeared in the Journal of the Franklin Institute, reached a quite different conclusion from the original. The referee had been right. The story is often told as a joke at Einstein's expense. It is better read as a reminder that the practice now taken for granted was contested within living memory, and that its authority rests less on tradition than on whether it actually improves the work that passes through it. The functions a review serves Peer review is asked to do several jobs at once, and much confusion about how to review well comes from failing to separate them. At least four can be distinguished. The first is quality control: checking that the methods are sound, the analyses appropriate, and the conclusions supported by the evidence. This is the function most scientists have in mind when they speak of peer review, and it is the one a reviewer is best placed to perform, because it draws directly on technical expertise. The second is selection: judging whether the work is important, novel, or interesting enough for the particular venue. A methodologically flawless study may still be a poor fit for a journal that publishes only work of broad significance. Selection judgements are more subjective than quality judgements, more dependent on the journal's editorial policy, and more vulnerable to bias. Many journals now ask reviewers to separate the two explicitly, and some, notably PLOS ONE since its launch in 2006 and a number of other "soundness-only" journals since, have removed the importance criterion altogether, asking reviewers to judge whether the work is rigorous rather than whether it is exciting. The third is improvement. Reviews routinely make manuscripts better: they catch errors, suggest additional controls or analyses, point to missing literature, and force authors to state their claims more precisely. Surveys of authors consistently find that most believe their published papers were improved by review, even when they resented the process. The improvement function operates regardless of the decision. A rejected paper, revised in light of thoughtful reports, often goes on to a better version at another journal. The fourth is certification. Publication in a peer-reviewed venue signals to readers, hiring committees, funders, journalists, and courts that the work has passed some form of expert scrutiny. This is the function that gives peer review its public weight, and it is also the one most often overstated. Review certifies that a small number of experts, working with limited time and without access to the raw data, found no fatal problem with the manuscript as presented. It does not certify that the findings are true. A reviewer's report can serve all four functions, but not equally. It serves quality control and improvement best when it is specific and technical. It serves selection best when the reviewer states their view of the work's significance separately from their view of its soundness, so that the editor can weigh each against the journal's own priorities. And it serves certification best when it is honest about what the reviewer could and could not check. What the evidence says about reliability For a practice so central to science, peer review was subjected to systematic study remarkably late. The International Congress on Peer Review and Scientific Publication, first held in Chicago in 1989 under the sponsorship of JAMA, began to change that, and a substantial body of research now exists. Its findings are sobering, and an early-career reviewer should know them. Reviewers agree with each other far less than intuition suggests. A 2010 meta-analysis by Lutz Bornmann, Rüdiger Mutz, and Hans-Dieter Daniel, published in PLOS ONE, pooled studies of inter-reviewer agreement on journal manuscripts and found levels of agreement that were low by any conventional standard: reviewers assessing the same manuscript agreed only modestly better than would be expected by chance. Earlier, Peter Rothwell and Christopher Martyn had examined reviews of submissions to two clinical neuroscience journals and conference abstracts, reporting in Brain in 2000 that agreement between reviewers on whether to publish was little better than chance. Grant review shows similar patterns. Low agreement is not in itself proof that review is broken. Editors deliberately choose reviewers with different expertise, and two reviewers who assess different aspects of a paper will naturally reach different overall views. But low agreement does mean that any single review is a noisy signal, and that the recommendation at the bottom of a report is far less informative than the reasoning above it. That is one of the most practical lessons in this book: editors can combine and weigh reasoning, but they cannot do much with a bare verdict. Reviewers also miss errors. In a study published in JAMA in 1998, Fiona Godlee, Catharine Gale, and Christopher Martyn took a paper that had already been accepted by the BMJ, introduced eight deliberate weaknesses in design, analysis, and interpretation, and sent it to hundreds of reviewers. On average, reviewers identified only about two of the eight. A later study led by Sara Schroter, published in the Journal of the Royal Society of Medicine in 2008, inserted nine major errors into three papers and sent them to more than six hundred BMJ reviewers; the average reviewer detected fewer than three of the major errors, and training produced only modest improvement. In emergency medicine, William Baxt and colleagues reported in 1998 on a fictitious manuscript seeded with errors and sent to all reviewers of the Annals of Emergency Medicine; many recommended acceptance despite fundamental flaws. These studies share a design that flatters no one: a manuscript deliberately constructed to be flawed, sent to reviewers who had no reason to expect it. They probably overstate how often errors survive in the ordinary course of review, where several reviewers and an editor scrutinise the work, and where authors are usually not trying to hide anything. But they establish beyond reasonable doubt that individual reviewers, working under normal conditions, miss a large share of serious problems. Systematic reviews of whether editorial peer review improves the quality of published research have reached cautious conclusions. A Cochrane review led by Tom Jefferson in 2007 found little rigorous evidence either way, largely because few studies had been designed to answer the question well. Later work has found that reporting quality tends to improve between submission and publication, but the size of the effect varies and the mechanism is not always clear. What review cannot do The limitations of peer review follow from its structure, and they are worth stating plainly, because a reviewer who expects the process to do something it cannot will review badly. Review cannot detect fraud reliably. A reviewer sees a manuscript, not a laboratory. If an author fabricates data competently, presents internally consistent results, and describes plausible methods, there is usually no way for a reviewer to know. The major fraud cases of the past quarter-century, from the fabricated organic transistor results of Jan Hendrik Schön at Bell Labs, exposed in 2002, to the fabricated stem-cell lines of Hwang Woo-suk, whose papers in Science were retracted in 2006, passed through review at the most selective journals in the world. They were uncovered by readers, colleagues, and whistleblowers after publication, often because someone noticed duplicated figures or impossible consistency across supposedly independent experiments. Chapter 7 returns to what a reviewer can do when something looks wrong. The honest starting point is that review is not designed as a fraud detector and should not be judged as one. Review cannot replicate. The reviewer's evidence is the manuscript and whatever supplementary material the authors provide. Even where data and code are available, reviewers rarely have the time to rerun analyses, and almost never have the resources to repeat experiments. Review assesses whether a claim is plausibly supported by the evidence presented, not whether the claim will hold up. Review cannot compensate for a poor question. A reviewer can point out that a study is underpowered, that a comparison group is inappropriate, or that a conclusion overreaches. They cannot make an uninteresting study interesting, and a report that tries to redesign the whole project is usually of little use to anyone. Review cannot be neutral. Reviewers are human, drawn from the same communities as the authors, with the same rivalries, loyalties, theoretical commitments, and unconscious associations. Chapter 6 examines the evidence on bias in detail. The point here is that a system built on human judgement inherits human failings, and the question is not whether bias exists but how it can be reduced and made visible. Who relies on the verdict The weight carried by the phrase "peer-reviewed" extends far beyond the journal. Systematic reviewers and meta-analysts often restrict their searches to peer-reviewed literature, so a study that passes review becomes a data point in a pooled estimate that may inform a clinical guideline or a regulatory decision. Science journalists use publication in a reviewed journal as a threshold for coverage. Hiring, promotion, and funding committees count reviewed publications, and frequently weigh them by the selectivity of the venue. The law relies on it too. In Daubert v. Merrell Dow Pharmaceuticals, decided by the United States Supreme Court in 1993, the Court set out factors that federal judges may consider when deciding whether expert scientific testimony is admissible. Whether a theory or technique has been subjected to peer review and publication is one of them. The Court was careful to say that publication is not a requirement and that review is an imperfect filter, but the decision nonetheless made peer review a formal consideration in legal proceedings affecting product liability, criminal forensics, and environmental regulation. The COVID-19 pandemic showed both how much the certification function matters and how easily it can be bypassed or undermined. Preprints, which appear without review, became a main channel of scientific communication during 2020, and many were covered by the press before any expert had examined them. At the same time, peer review itself failed conspicuously in at least one high-profile case. In June 2020 The Lancet and the New England Journal of Medicine retracted papers based on a hospital database supplied by the company Surgisphere, after independent researchers raised questions that the company could not answer about where its data had come from. The papers had passed review at two of the most selective medical journals in the world. The data behind them could not be verified. For a reviewer, the lesson of this chain of reliance is not that every report carries the fate of public health. It is that the verdict a reviewer contributes to does not stay inside the journal. Downstream users rarely read the reviews; they see only the outcome. A reviewer who waves a paper through with a careless "minor revision" has, in effect, lent their expertise to every later use of that paper, without the users having any way to know how much scrutiny was really applied. That is one reason, discussed in Chapter 8, why the movement to publish reviews alongside papers has gathered strength: it lets downstream readers see what the certification consisted of. Why it is still worth doing well Given all this, it would be easy to conclude that peer review is theatre, and some critics have said as much. The conclusion does not follow. The evidence shows that review is noisy and fallible, not that it is useless. Manuscripts are routinely improved by it; errors are routinely caught; overstated claims are routinely moderated. Most working scientists can point to a review that saved them from publishing something wrong. And the alternatives that have been proposed, from publishing everything and letting readers sort it out to relying on metrics of attention, have failure modes of their own. The more realistic reform agenda, explored in Chapter 8, keeps expert review but changes when it happens, who can read it, and how reviewers are credited. What the evidence does suggest is that the quality of peer review depends heavily on the quality of individual reviews, and that individual reviews vary enormously. A careful, specific, well-reasoned report can steer a paper and an editor toward a sound outcome. A careless or hostile one can do real damage, delaying good work, letting weak work through, or discouraging a young author from submitting again. The difference between the two lies largely in the habits of the person writing, and those habits can be learned. That is where this book begins its practical work. Before turning to how to read and write a review, however, a reviewer needs to understand the system their report enters: who reads it, who decides, and what they need. That is the subject of the next chapter. A working definition It helps to end with a definition that reflects what the evidence and history suggest, rather than what the ceremony of "peer-reviewed" implies. Peer review is a structured request for expert advice, made by an editor to a small number of specialists, about whether a piece of work is sound, whether it suits a particular venue, and how it might be improved, delivered on the basis of the evidence the authors have chosen to present, under time pressure, and with the knowledge that the final decision belongs to someone else. Each clause of that definition carries an obligation. Structured means the reviewer should organise their advice so it can be used. Expert means the reviewer should confine confident judgements to what they actually know. Advice means the reviewer is not the decision-maker. The evidence the authors have chosen to present means the reviewer should notice what is missing as well as what is there. Under time pressure means the reviewer should prioritise, spending their limited attention where it matters most. The chapters that follow are, in a sense, an extended commentary on these obligations. Chapter 2. The Editorial System: Who Decides What A reviewer's report is one input into a decision made by someone else, and it travels through a system that most reviewers never see. Understanding that system is not administrative trivia. It determines what a report should contain, how its recommendation will be read, which remarks the authors will see, and what happens when a reviewer raises a concern about something other than the science. This chapter follows a manuscript from submission to decision, describing the people it passes through and what each needs. The journey of a manuscript A manuscript arrives through an online submission system, usually one of a small number of commercial platforms that most journals license. Before any scientist looks at it, editorial staff typically perform technical checks: that the required files are present, that the word count and formatting are within limits, that ethics approvals and data availability statements have been provided, that author details and conflict of interest declarations are complete. Many publishers now also run automated screening at this stage, including similarity checks for text overlap with published work and, increasingly, tools designed to detect manipulated images or patterns associated with commercially produced fraudulent papers. The manuscript then reaches an editor. At large journals this may be a professional editor, a full-time employee with a doctorate who no longer runs a laboratory; at most society and specialist journals it is an academic editor, a working scientist who handles manuscripts alongside research and teaching. The first decision this editor makes is whether to send the paper for review at all. Desk rejection, the return of a manuscript without external review, is common at selective journals, where a large majority of submissions may be declined at this stage. The reasons are usually scope, perceived significance, or obvious problems of quality or presentation. Desk rejection spares authors weeks of waiting and spares reviewers from assessing work the journal would never publish, which is why, uncomfortable as it is, it is generally considered good editorial practice when done promptly and with a brief explanation. If the paper goes forward, an editor identifies reviewers. At many journals an editor-in-chief assigns the manuscript to a handling editor, sometimes called an associate or academic editor, who has relevant expertise and manages the rest of the process. The handling editor searches for reviewers using their own networks, the manuscript's reference list, databases of past reviewers, and, at some journals, algorithmic reviewer-suggestion tools. Authors are often invited to suggest or exclude reviewers. Editors vary in how they treat these suggestions; some avoid author-suggested reviewers entirely, particularly since investigations in the mid-2010s revealed peer-review rings in which authors supplied contact details for fake reviewer accounts they controlled, leading to the retraction of scores of papers across several publishers. Finding reviewers is frequently the slowest step. Editors routinely send many invitations for each acceptance, and reviewer fatigue, the concentration of reviewing burden on a relatively small pool of willing and reliable people, is a widely recognised problem. This is one reason editors are often glad to invite early-career researchers, whose expertise in current techniques is frequently more precise than that of senior investigators and whose availability is sometimes greater. Once reviews are in, the handling editor reads them alongside the manuscript and reaches a decision, or makes a recommendation to the editor-in-chief who makes it. The decision letter goes to the authors with the reviewers' comments attached. If revisions are requested, the revised manuscript may return to the same reviewers, to a subset of them, or only to the editor, depending on the extent of the changes and the journal's policy. The roles and their obligations Each position in this system carries distinct responsibilities, and several of the most common problems in peer review arise when someone acts outside their role: a reviewer who treats their report as a final verdict, an editor who forwards reports without reading them, an author who treats review as an adversarial negotiation. The main roles and their core duties are set out in Table 1. Table 1. Principal roles in journal peer review and their core obligations. Role Primary responsibility Key ethical duties Typical failure Author Report the work fully and accurately Honest data, disclosure of conflicts, credit to contributors Overclaiming; withholding data Handling editor Manage review and reach or recommend a decision Choose qualified, unconflicted reviewers; weigh reports critically Treating reviews as votes Editor-in-chief Set policy and take final responsibility Consistency, appeals, handling misconduct Favouring prominent authors Reviewer Advise on soundness, significance, and improvement Confidentiality, disclosure, fairness, candour about expertise Harshness; scope creep Publisher and staff Operate systems and integrity checks Screening, record-keeping, corrections Opaque processes The table compresses a good deal, and two rows deserve fuller comment. The handling editor is the person a reviewer is really writing for. They are usually an expert in the broad field, but not necessarily in the specific technique or subfield of the manuscript; that is why they sought reviewers. They will read two or three reports, often of very different lengths, tones, and recommendations, and must reconcile them. What helps them most is a report that makes its reasoning visible, distinguishes the problems that would change the conclusions from those that would merely improve the presentation, and is candid about the reviewer's own confidence. What helps them least is a verdict without reasons, or a list of forty objections of undifferentiated weight. Good editors do not treat reviews as votes. If two reviewers recommend minor revision and a third identifies a fundamental flaw that the others missed, a careful editor will weigh the substance of the third report, not count the recommendations. Equally, if one reviewer recommends rejection on grounds that are clearly a matter of taste or theoretical allegiance, the editor may discount that recommendation. This is why the reasoning in a review matters more than the recommendation. Reviewers sometimes feel frustrated when an editor's decision does not follow their advice, but the system is designed that way: the reviewer advises on the evidence they have, and the editor decides with the benefit of all the evidence. The editor-in-chief holds ultimate responsibility for what the journal publishes and for its policies, and is usually the person who handles appeals and serious concerns about misconduct. Reviewers rarely deal with the editor-in-chief directly, but it is useful to know that the handling editor is not the last line of escalation. If a reviewer's serious concern is ignored, the editor-in-chief, and beyond them the publisher's research integrity team, are the appropriate next contacts. What the editor's invitation is asking A review invitation is a specific request, and reading it carefully is the first act of good reviewing. Most invitations include the title and abstract, a deadline, and sometimes a note from the editor about what they would like the reviewer to focus on. That note matters. An editor who writes "we would particularly value your assessment of the single-cell analysis" is telling the reviewer that someone else will cover the physiology. A reviewer who ignores the request and writes a general report may duplicate another reviewer's work while leaving the question the editor actually needed answered. Three questions should be answered honestly before accepting. First, is this within my competence? A reviewer does not need to be expert in every aspect of a manuscript, but they should be able to assess a substantial part of it with confidence, and they should be willing to say which parts they cannot assess. Accepting a review for which one is not qualified and then writing confidently about it is a quiet form of misrepresentation. Second, do I have a conflict of interest? Chapter 5 treats this in detail. At this stage, the question is whether any relationship with the authors, the work, or its subject would lead a reasonable observer to doubt the reviewer's impartiality. If so, the reviewer should either decline or disclose the relationship to the editor and let them decide. Third, can I meet the deadline? Late reviews are one of the most common complaints from both editors and authors, and they cost authors real time at stages of their careers when time matters greatly. It is better to decline promptly, perhaps suggesting a qualified colleague, than to accept and then delay. If circumstances change after accepting, a short message to the editor asking for an extension is courteous and usually granted. Early-career researchers sometimes receive review requests indirectly, when a senior colleague asks them to help with a review the senior colleague has been invited to write. This practice, sometimes called ghostwriting of reviews, is widespread. A survey led by Gary McDowell and colleagues, published in eLife in 2019, found that many early-career researchers had co-reviewed with a principal investigator, and that a large share of those had not been named to the editor. The authors argued that this deprives junior researchers of credit and misleads editors about who actually assessed the work. Most journals permit co-reviewing, but they expect the invited reviewer to obtain permission before sharing a confidential manuscript and to name the co-reviewer. An early-career researcher asked to help should ask whether the editor has been told; a principal investigator who asks for help should tell the editor and ensure the junior colleague is credited. What happens to a report Once submitted, a report is usually divided into two parts: comments for the authors and confidential comments for the editor. The authors will see the first part verbatim, sometimes lightly edited by the editor to remove anything inappropriate. They will not see the confidential comments. The division is useful but frequently misused. Confidential comments are appropriate for information the editor needs but the authors should not see: a note about the reviewer's own limits of expertise, a concern about possible duplicate publication or data manipulation that needs investigation before the authors are approached, a candid view of whether the work reaches the journal's bar for significance, or an explanation that the reviewer has a relationship with the authors that they judged not to be disqualifying. They are not appropriate for delivering criticisms that the reviewer is unwilling to make openly. A reviewer who writes a mild report to the authors and a damning note to the editor leaves the authors unable to respond to the real reasons for rejection. The general principle is that any substantive criticism of the science should appear in the comments to authors, where it can be answered. Most journals send each reviewer the other reviewers' reports and the decision letter after a decision is made. Reading these is one of the best ways for a new reviewer to calibrate. It shows what others noticed that one missed, how different reviewers weighted the same problems, and how the editor combined the advice. Where a reviewer's view differed sharply from the decision, it is worth considering whether the editor saw something in the other reports that justified the difference. If the manuscript is revised, the reviewer may be asked to assess the revision. The task then changes. The question is whether the authors have adequately responded to the points raised, not whether the reviewer can find new objections. Raising entirely new issues at the second round, when they could have been raised at the first, is a common source of author frustration and prolongs review without clear benefit. New problems introduced by the revision itself are fair game, and so are serious problems that were genuinely missed the first time, but the reviewer should acknowledge that they are new. Decisions and their meaning Journals use a small set of decision categories, though the labels vary. Accept is rare after first review. Minor revision means the paper is fundamentally sound and needs only limited changes, often without further external review. Major revision means substantial problems exist but appear fixable; the revised paper will usually be re-reviewed. Reject and resubmit, used by some journals, means the problems are serious enough that a substantially new manuscript is required, though the journal is willing to consider it. Reject means the journal will not consider the work further, whether because of fundamental flaws or insufficient fit or significance. A reviewer's recommendation should correspond to these meanings, and the report should make clear which category the reviewer has in mind and why. It is particularly helpful to state what would be required to change the recommendation. "I would support publication if the authors can show that the effect survives correction for multiple comparisons and holds in the second cohort" gives an editor far more to work with than "major revision". Reviewers should also remember that selection criteria vary between journals and that their recommendation is specific to the venue. A paper that is too incremental for a broad-scope journal may be an excellent contribution to a specialist one. A good report makes this explicit, separating the reviewer's view of the work's soundness, which should not change from journal to journal, from their view of its fit, which should. Variations on the standard workflow Not every manuscript follows the path just described, and a reviewer should recognise the variants, because each changes what the report is for. Registered Reports move the main round of review before the results exist. Introduced at the journal Cortex in 2013, under the editorial leadership of the psychologist Chris Chambers, and now offered by hundreds of journals, the format asks authors to submit their introduction, hypotheses, methods, and analysis plan before collecting data. Reviewers assess this Stage 1 protocol on the importance of the question and the rigour of the design. If the protocol is accepted in principle, the journal commits to publishing the final paper regardless of whether the results support the hypotheses, provided the authors follow the approved plan. At Stage 2, reviewers check that the plan was followed and that the conclusions follow from the results. For a reviewer, the shift is substantial. At Stage 1 there is no result to be impressed or disappointed by, so the assessment rests entirely on whether the design can answer the question. At Stage 2 the reviewer's freedom is deliberately restricted: they are not invited to reject the paper because the findings are unexciting or unexpected. The format was designed to counter publication bias, and it works only if reviewers respect its rules. Cascading or transfer review allows a manuscript rejected by one journal to be passed, with its reviews, to another journal from the same publisher, or in some arrangements to a different publisher altogether. Reviewers are often asked at the time of review whether they consent to their report being transferred. Consent costs little and can save authors months, but it means a report should be written so that it remains useful to an editor at a different venue; a report whose substance is simply "not important enough for this journal" transfers poorly. Special issues and conference proceedings often run on compressed timelines, with guest editors who may be less experienced than a journal's regular editorial board. In recent years several publishers have retracted large numbers of papers from special issues after discovering that guest editors had been impersonated or that review had been compromised. A reviewer invited to a special issue should apply exactly the same standards as for a regular submission and should be alert, as any reviewer should, to signs that a manuscript has been produced by a paper mill, a topic returned to in Chapter 7. Appeals and disputes Authors who believe a decision was wrong may appeal. The Committee on Publication Ethics, known as COPE, an organisation founded in 1997 whose guidance most major publishers follow, recommends that journals have a clear appeals process. Appeals are usually handled by the editor-in-chief or a senior editor, who may seek a further opinion. Reviewers are occasionally asked to respond to an author's rebuttal. When that happens, the right approach is to read the rebuttal on its merits, concede points where the authors are right, and explain clearly where the original concern still stands. A reviewer who treats an appeal as a personal challenge has forgotten that their role was always advisory. The editorial system, then, is designed to distribute responsibility. Authors are responsible for the honesty and completeness of what they submit. Reviewers are responsible for accurate, candid advice within their competence. Editors are responsible for choosing reviewers well, weighing their advice critically, and making decisions they can defend. When each role is performed well, the system can reach good decisions despite the noise in any single review. When a reviewer understands that their report is one part of this structure, they can write it to be used, which is the subject of the next two chapters. Chapter 3. Reading a Manuscript Critically Most weak reviews fail before a word is written. The reviewer reads the manuscript once, from beginning to end, marking objections as they arise, and then assembles those objections into a report. The result is predictable: a long list of comments of wildly varying importance, heavily weighted toward the introduction and early methods where the reviewer's attention was freshest, often missing the one problem in the analysis that actually determines whether the conclusions hold. Linear reading is how people read papers for pleasure or for background. It is not how to assess one. A more reliable approach treats reading as a sequence of distinct passes, each with its own purpose. The first establishes what the paper claims. The second tests whether the design and analysis can support those claims. The third examines the details: reporting, figures, statistics, data, and the fit between what was done and what is said. The passes need not be rigid, and experienced reviewers blend them, but separating them at first builds the habit of asking the important questions before the easy ones. The first pass: what is being claimed? Before assessing whether a paper is right, a reviewer must know precisely what it says. This sounds trivial and is not. Many manuscripts contain several claims of different strength, and the abstract, the discussion, and the title frequently state them differently. A title may announce that a gene "controls" a behaviour while the results show a correlation in one mouse strain; an abstract may report a "significant reduction" while the discussion concedes that the effect was small and confined to a subgroup. The first pass should therefore be quick and focused on the architecture of the argument. Read the title, abstract, and final paragraphs of the introduction, where the authors usually state their aims. Look at every figure and table, with their legends, without reading the results text. Then read the discussion's opening and closing paragraphs. At the end of this pass, a reviewer should be able to write down, in two or three sentences, the central claim of the paper, the key evidence offered for it, and the type of study that produced that evidence. It is worth actually writing this summary. It becomes the opening of the review, where it serves an important function discussed in the next chapter: it shows the authors and the editor that the reviewer has understood the work, and it exposes any misunderstanding before it contaminates the rest of the assessment. It also serves the reviewer. If the central claim cannot be stated clearly after a careful first pass, that is itself a finding: the manuscript is unclear about what it is arguing, and saying so is one of the most useful things a report can do. The first pass should also identify the claim's type. Is it causal, or descriptive, or predictive? Is it a claim about a mechanism, a population, a method's performance, or the existence of a phenomenon? The type determines what evidence is required. A causal claim from observational data needs a credible strategy for addressing confounding; a claim about a new method's superiority needs a fair comparison against current alternatives on relevant benchmarks; a claim about a population needs a sample that represents it. Much of the substance of a good review consists of matching the claim to the evidence its type requires. The second pass: can the design support the claim? With the claim in hand, the reviewer reads the methods and results closely, asking a single overarching question: if everything reported here was done exactly as described, would it justify the conclusion? This question sets aside, for now, whether things were done as described. It concerns the logic of the study. Several recurring problems are worth checking for explicitly, because they are common and because they are easy to miss when reading in the authors' frame. Alternative explanations. For every main result, ask what else could have produced it. In experimental work this usually means asking whether the controls rule out the obvious alternatives: whether a knockdown effect might be off-target, whether a drug effect might be due to the vehicle, whether a behavioural difference might reflect a motor deficit rather than a cognitive one. In observational work it means asking about confounding, selection, and reverse causation. A reviewer who can name a specific, plausible alternative explanation that the design does not exclude has found something important. A reviewer who says only that "other factors may be involved" has not. Comparison and baseline. Every claim of an effect is a claim about a difference, and the choice of comparison determines what the difference means. Is the control group appropriate? Is a new method compared with the current best alternative or with a straw man? Is a change measured against a baseline that could itself have shifted? Sample and power. Is the sample large enough to detect effects of the size claimed, or to make a null result informative? Small studies that report large effects deserve particular scrutiny, because they are disproportionately likely to be false positives or inflated estimates, a point made forcefully by John Ioannidis and by Katherine Button and colleagues, whose 2013 analysis in Nature Reviews Neuroscience found the median statistical power of neuroscience studies to be low. Where sample sizes are not justified by a power calculation or equivalent reasoning, a reviewer may reasonably ask for one. Independence of observations. A frequent and consequential error is to treat non-independent measurements as independent: multiple cells from the same animal, repeated measures from the same participant, several samples from the same culture. Doing so inflates apparent sample sizes and produces spuriously small p-values. It is worth checking what the unit of analysis is and whether it matches the unit of replication. Analytical flexibility. Were the analyses planned in advance, or chosen after seeing the data? Were outcomes, covariates, or exclusion criteria selected in ways that might have favoured the reported result? In clinical trials, comparison with the registered protocol is often revealing; in other fields, preregistration is increasingly common and should be checked when it exists. Where there is no preregistration, the reviewer can still ask whether the results are robust to reasonable alternative analytical choices. Generalisation. Do the conclusions extend beyond the conditions actually tested? Findings in one cell line, one species, one population, or one dataset are frequently described as if they held generally. A reviewer should ask the authors to match the scope of their claims to the scope of their evidence. By the end of the second pass, a reviewer should know whether there are any problems that, if not addressed, would mean the central conclusion is unsupported. These are the major concerns. They may be few, and there may be none. Identifying them correctly is the most valuable thing a reviewer does. The third pass: details, reporting, and data The third pass is where most reviewers spend most of their time, which is why it belongs last. Here the reviewer checks whether things were done and reported as they should be: whether statistical tests are appropriate and correctly described, whether figures display what the text says, whether the numbers in different places agree, whether the methods are described in enough detail to be reproduced, and whether data and code are available as the journal requires. Reporting guidelines are a practical aid here. Over the past three decades, groups of methodologists and editors have produced checklists specifying what should be reported for common study designs, and many journals require authors to complete the relevant checklist on submission. A reviewer does not need to audit every item, but knowing the relevant guideline helps identify omissions quickly. The most widely used are summarised in Table 2; the EQUATOR Network, an international initiative founded in 2008 to improve the reliability of health research reporting, maintains a searchable library of several hundred more. Table 2. Widely used reporting guidelines and the study types they cover. Guideline Study type Issued by or associated with Useful for checking CONSORT Randomised controlled trials CONSORT Group Randomisation, allocation, flow of participants PRISMA Systematic reviews and meta-analyses PRISMA Group Search strategy, selection, risk of bias STROBE Observational epidemiology STROBE Initiative Confounding, selection, missing data ARRIVE Animal research NC3Rs (UK) Randomisation, blinding, sample size STARD Diagnostic accuracy studies STARD Group Reference standard, spectrum of patients The guidelines matter to review for a reason beyond completeness. Their items encode the specific ways each type of study tends to go wrong. ARRIVE asks whether animals were randomly allocated and whether outcome assessors were blinded because unblinded, unrandomised animal studies have repeatedly been shown to report larger effects. CONSORT asks for a participant flow diagram because attrition that differs between arms can create an apparent treatment effect. Reading a manuscript against the relevant guideline is, in effect, reading it with the accumulated experience of the methodologists who studied that design's failures. Figures deserve particular attention in the third pass. Check that axes are labelled and scaled honestly, that error bars are defined, that the sample size for each panel is stated, and that representative images are accompanied by quantification. Look for panels that seem too clean or that resemble each other in ways they should not; Chapter 7 discusses what to do if something appears duplicated or altered. Check that individual data points are shown where sample sizes are small, since bar charts of means can conceal very different distributions. Numbers should be checked for internal consistency. Do the sample sizes in the methods match those in the figure legends and tables? Do percentages add up? Are reported test statistics, degrees of freedom, and p-values consistent with one another? Tools exist to help with the last question. The program statcheck, developed by Michèle Nuijten and colleagues, recomputes p-values from reported test statistics in APA-formatted text, and the GRIM test, proposed by Nick Brown and James Heathers in 2016, checks whether reported means are arithmetically possible given the sample size and the granularity of the underlying data. Inconsistencies found this way are usually typographical, but they can indicate deeper problems, and in either case they should be corrected. Finally, the reviewer should check what the data and code availability statement says, and whether it is true. A statement that data are "available on reasonable request" is weaker than a deposit in a public repository, and studies have found that such requests are frequently unanswered. Many funders and journals now require deposit where ethically possible. If the journal's policy requires data to be available and it is not, or if a link leads nowhere, the reviewer should say so. If the data are available and the reviewer has the time and competence, looking at them, even briefly, can reveal problems invisible in the manuscript. The three passes in practice A hypothetical example shows how the passes change what a reviewer notices. Suppose a manuscript reports that a dietary supplement improves memory in older adults. Its title says the supplement "enhances cognitive function in ageing". The abstract reports a randomised trial of 120 participants over twelve weeks and a significant improvement on a word-recall task. A linear reader might begin by objecting to the introduction's selective citations, move on to request more detail about the supplement's formulation, question the choice of font in a figure, and eventually, somewhere in the results, notice that the primary outcome listed in the trial registry was a composite cognitive score, not word recall. That last observation, buried as comment seventeen of twenty-two, is the one that matters. The first pass, done properly, would have surfaced it early. Writing down the central claim forces the question of which outcome supports it, and the type of claim, a causal effect from a randomised trial, directs the reviewer immediately to CONSORT's concerns: was the primary outcome prespecified, and was it the one reported? The second pass would then ask whether word recall was among several secondary outcomes, whether the composite score showed any effect, whether multiple comparisons were accounted for, and whether a twelve-week improvement on one task justifies a title claiming enhanced "cognitive function" in general. It would also ask about blinding, since a supplement with a distinctive taste might allow participants to guess their allocation, and about attrition, since older participants who experience side effects may drop out unevenly between arms. Only in the third pass would the reviewer check the formulation details, the consistency of the numbers across tables, and the figure design. Those checks are still worth doing. But the report built from these passes leads with the outcome-switching question and the overreach in the title, which are the problems that determine whether the conclusion stands, and relegates the rest to minor comments. The editor reading it knows at once what the decision turns on. The authors know what they must address, and they may well have a good answer: perhaps the registry was amended before unblinding, for documented reasons. A review that asks the right question clearly allows that answer to come out. Reviewing within one's competence Few reviewers are expert in every method a modern manuscript uses. A single paper in cell biology may combine imaging, genomics, animal behaviour, and sophisticated statistics; a paper in ecology may combine field sampling, remote sensing, and Bayesian modelling. The reviewer's obligation is not to be omniscient but to be clear about where their competence ends. The practical rule is to review confidently what one knows, to review cautiously what one partly knows, and to state plainly what one cannot assess. "I am not able to evaluate the phylogenetic methods and would recommend that the editor seek a specialist opinion on them" is a useful sentence. It tells the editor precisely where a gap exists in the assessment, and editors frequently act on it. By contrast, a reviewer who comments confidently on methods they do not understand may mislead the editor, burden the authors with inappropriate requests, or miss real problems while inventing false ones. Statistics deserve a special word. Many reviewers who are expert in their experimental domain are less secure in statistical analysis, and statistical problems are among the most common serious flaws in published research. A reviewer who is uncertain whether an analysis is appropriate should say so and suggest statistical review. Several journals employ statistical reviewers or editors for exactly this reason, and an editor who is told that a statistical question needs specialist attention can obtain it. Time, attention, and proportion A thorough review of a substantial manuscript takes time, typically several hours spread over more than one sitting. There is no merit in taking longer than necessary, but there is real cost in taking less than the work requires. The distribution of time matters as much as its total. A useful rough allocation is to spend a modest share on the first pass, the largest share on the second, and the remainder on the third, adjusting for the paper's complexity. Reviewers who spend most of their time correcting typographical errors and requesting additional citations have allocated their attention poorly, however long they spent. It is also worth leaving time between reading and writing. First reactions to a manuscript are often stronger than considered ones, in both directions. A day's distance frequently turns an objection that seemed decisive into one that is real but minor, or reveals that an apparently convincing result depends on an assumption not stated. The reviewer who writes immediately after a single read is likely to produce a report that reflects their mood as much as the manuscript. Reading with charity The final principle of critical reading is one that sounds soft but is methodological: read the manuscript as the strongest version of what the authors are trying to say. When a passage is ambiguous, consider the most reasonable interpretation before assuming the worst one. When a control seems missing, check whether it appears in the supplementary material or whether a different control addresses the same concern. When a result seems implausible, ask what would have to be true for it to be correct. Charitable reading is not credulity. Its purpose is to ensure that criticisms, when they come, are aimed at what the authors actually did rather than at a misreading. A report that attacks a straw man wastes the authors' time, undermines the reviewer's credibility with the editor, and often leaves the real weaknesses untouched. A report that engages with the strongest version of the argument, and still finds problems, is one that the authors must take seriously. With the reading done, the reviewer knows what the paper claims, whether the design can support it, and where the details fall short. The next task is to turn that knowledge into a report that an editor can use and authors can act on. Hashtags: #ThePeerReviewEcosystem #PeerReview #ScholarlyPeerReview #EditorialEthics #ResearchIntegrity #ConstructiveCritique #EditorialRoles #ReviewerResponsibilities #HandlingEditors #EditorsInChief #ReviewerEthics #ConflictOfInterest #Confidentiality #ReviewerBias #ImplicitBias #PublicationEthics #COPE #CriticalManuscriptReview #ResearchQuality #ScientificPublishing #OpenPeerReview #RegisteredReports #EditorialDecisionMaking #ResearchTransparency #FutureOfPeerReview

Download the Book (PDF): Introduction In the spring of 2022 a neurologist at Vanderbilt University named Matthew Schrag, working on a separate question entirely, began looking closely at the Western blots in a group of papers on amyloid beta, the protein at the centre of the dominant theory of Alzheimer's disease. Among them was a paper published in Nature in 2006 that described a specific assembly of the protein, called Aβ56, and reported that it impaired memory in rats. The paper had been cited thousands of times. Schrag's concerns, set out in a dossier and later investigated by the journalist Charles Piller for Science, centred on bands in the blots that appeared to have been duplicated, spliced and erased. In June 2024 Nature* retracted the paper. The retraction notice referred to signs of excessive manipulation, including splicing, duplication and the use of an eraser tool, and recorded that the first author, Sylvain Lesné, did not agree with the retraction while his coauthors did. Three features of that episode are worth holding onto, because they recur throughout this book. The first is that the evidence was on the page for sixteen years. Nothing new had to be discovered in a laboratory; someone simply had to look at the published figures with trained eyes and the right software. The second is that the person who looked was not a colleague of the authors, not a reviewer, not an editor, and not an institutional official. He was an outsider who took on the work in his own time. The third is that the distance between the first credible alarm and the formal correction of the record was measured in years, and it was closed only after sustained attention from a major news outlet. That pattern, visible evidence, outsider detection, slow correction, is the subject of this book. Over the past fifteen years the detection of fabricated and falsified research has been transformed. What was once almost entirely a matter of chance, a junior researcher noticing something odd at the bench and deciding to risk a career by reporting it, has become in large part a forensic discipline practised on the published record itself. Image analysts can find a duplicated panel in a figure from a paper published two decades ago. Statisticians can show that a set of reported means could not have arisen from integer data, or that baseline variables in a randomised trial are too similar, or too different, to be the product of chance. Linguists and bibliometricians can flag the fingerprints of a paper mill in the phrasing of an abstract. None of this requires access to anyone's laboratory notebooks. It requires only the paper, the method, and the will to apply it. The controlling argument This booklet argues one thing. Detection of research fraud has outrun correction. The forensic methods for finding fabricated images, impossible numbers and manufactured papers are now good enough, cheap enough and scalable enough that the bottleneck in scientific integrity is no longer finding problems but acting on them. Journals, universities and funders were built for a world in which misconduct was rare, detected from the inside, and handled confidentially one case at a time. They are now facing a world in which problems are detected from the outside, in public, at industrial scale, and in which some of the misconduct is itself industrial. The institutions have not caught up, and the consequence is that the scientific record is being corrected largely by volunteers, at personal cost, on a timescale that bears no relation to the speed at which bad work enters it. It follows that the most important reforms are not better detection tools, though those help. They are changes to what happens after a problem is found: who is obliged to respond, how quickly, on what standard of evidence, with what protection for those who raise concerns, and with what separation between correcting a paper and judging a person. The final chapters make that case in detail. What this book covers The first chapter sets out what research misconduct is, in the legal and regulatory sense, and what the best evidence says about how common it is. It separates fabrication and falsification from the much larger territory of questionable research practice and honest error, because the forensic methods discussed later behave very differently across those categories. The next two chapters deal with images. Chapter 2 describes the forms that image manipulation takes in published figures, from the simple reuse of a panel to the splicing, cloning and erasing of bands in a blot, and explains why a visible duplication is evidence of a problem but not, on its own, evidence of intent. Chapter 3 turns to the tools: the software that now screens submissions at several major publishers, the older techniques of contrast adjustment and error-level analysis, and the looming difficulty posed by images generated or altered with machine learning, which leave few of the traces that current methods look for. Chapter 4 is about numbers. It explains in plain terms how a set of forensic statistical tests work, from the arithmetic consistency checks known as GRIM and SPRITE, through the analysis of baseline balance in randomised trials pioneered by the anaesthetist John Carlisle, to the examination of digit patterns and spreadsheet metadata. Each test is described with what it needs, what it can show and where it breaks down. Chapter 5 moves from individuals to organisations. Paper mills, businesses that manufacture and sell authorship of fraudulent manuscripts, have become the largest single source of fabricated research by volume. The chapter describes how they operate, how they were discovered, how they are detected, and why the scale of the problem, documented in 2025 in a large study in the Proceedings of the National Academy of Sciences, has changed what integrity work has to mean. Chapters 6 and 7 look at people and institutions. Chapter 6 is about whistleblowers, both the insiders who report colleagues and the outsiders, the so-called sleuths, who scrutinise the literature in public. It examines what they have achieved and what it has cost them. Chapter 7 follows an allegation through the formal machinery of institutional inquiry and investigation and journal correction, and identifies where that machinery stalls. Chapter 8 sets out a programme of reform grounded in the evidence of the previous chapters. A note on names and cases This is a book about real cases, and some of them involve named people. The standard applied throughout is simple. Where a finding of misconduct has been made by a competent body, whether a government office, a university investigation or a court, it is reported as that body's finding and attributed to it. Where a paper has been retracted, the retraction is reported along with, where relevant, the reasons the journal gave. Where allegations have been made but not adjudicated, or where the person concerned disputes them, that is said plainly. Several of the people discussed have denied wrongdoing, and some have litigation under way. A book about the discipline of evidence has no business running ahead of the evidence about individuals, and this one does not try. The same discipline applies in the other direction. A duplicated image or an impossible mean is a fact about a document. It is not, by itself, a fact about the character of any person who signed it. Much of what follows is an argument that institutions should learn to act firmly on the first kind of fact without waiting for the second, which is slower, harder and properly surrounded by due process. Keeping those two things apart is not a courtesy to the accused. It is the only way to correct the record at the speed the record now requires. Chapter 1. What Counts as Misconduct, and How Much of It There Is Every investigation of a published paper begins with a question that sounds simpler than it is: what, exactly, is wrong? A paper can be wrong because its authors made a mistake in a calculation. It can be wrong because they analysed their data in a dozen ways and reported the one that worked. It can be wrong because an image was assembled carelessly from the wrong folder. And it can be wrong because someone invented the numbers. Only the last of these, and a few close relatives, is research misconduct in the formal sense. The distinctions matter enormously, both for what the forensic methods described later in this book can show and for what institutions are entitled to do once they have shown it. The narrow definition and why it is narrow In the United States the working definition comes from a federal policy issued by the Office of Science and Technology Policy in 2000 and adopted across research agencies. Research misconduct is fabrication, falsification or plagiarism in proposing, performing or reviewing research, or in reporting research results. Fabrication is making up data or results and recording or reporting them. Falsification is manipulating research materials, equipment or processes, or changing or omitting data or results, such that the research is not accurately represented in the record. Plagiarism is the appropriation of another person's ideas, processes, results or words without giving appropriate credit. The policy states explicitly that research misconduct does not include honest error or differences of opinion. A finding of misconduct under this framework requires three things: a significant departure from the accepted practices of the relevant research community; that the misconduct was committed intentionally, knowingly or recklessly; and that the allegation is proven by a preponderance of the evidence. For research funded by the US Public Health Service, which includes the National Institutes of Health, these rules are codified in the federal regulations at 42 CFR Part 93 and overseen by the Office of Research Integrity, usually called ORI. Those regulations were substantially revised in a final rule published in September 2024, which took effect at the start of 2025 and gave institutions until the start of 2026 to bring their own policies into compliance. The revision clarified definitions, permitted institutions to handle multiple respondents and allegations more efficiently, and adjusted procedures around confidentiality and the scope of investigations, but it left the core triad of fabrication, falsification and plagiarism untouched. The narrowness is deliberate. A finding of misconduct can end a career, bar a scientist from federal funding and in rare cases lead to criminal prosecution. Regulators have therefore confined it to conduct that is plainly dishonest rather than merely poor. Other jurisdictions draw the line in slightly different places. Several European codes, including the European Code of Conduct for Research Integrity published by ALLEA, the federation of European academies, use the same fabrication, falsification and plagiarism core but surround it with a larger list of unacceptable practices, such as manipulating authorship, withholding data without justification, or misrepresenting the achievements of others. Some national systems, including those in Denmark and Sweden, have statutory definitions with their own nuances. But the idea of a hard core surrounded by a softer, larger periphery is almost universal. For the forensic investigator this has a practical consequence. The methods described in this book detect anomalies in documents. A duplicated band in a blot, a mean that cannot be produced by the stated sample, a set of baseline characteristics that are implausibly balanced: each is an observation about a paper. None of them, taken alone, establishes intent, which is a fact about a person's state of mind. The step from anomaly to misconduct is taken, when it is taken at all, by an institution that can interview people, collect raw data and notebooks, and weigh explanations. The investigator's job is to establish that the published record cannot be relied on. That is a different and much lower bar, and one of the central claims of this book is that institutions should be willing to act on it without waiting for the higher one. The larger territory: questionable practice and honest error Around the narrow core lies a large territory of conduct that distorts the record without falling within the legal definition. It is usually called questionable research practice. The canonical list includes failing to report all of a study's outcome measures, deciding whether to collect more data after looking at the results, selectively reporting studies or conditions that worked, excluding data points after seeing their effect on the result, rounding p-values down across a significance threshold, and presenting an unexpected finding as though it had been predicted all along. The distinction between these practices and falsification is sometimes a matter of degree rather than kind. Dropping an outlier with a documented rationale is standard practice. Dropping every observation that weakens an effect until the effect crosses a threshold, and then failing to mention that anything was dropped, begins to look like changing or omitting data such that the research is not accurately represented, which is the regulatory definition of falsification. Where the line falls in a given case depends on facts that the published paper usually does not reveal. Honest error is different again. Laboratories are busy places, figure assembly is tedious, and large projects generate thousands of image files with similar names. Many duplicated images in the literature are the result of someone dragging the wrong file into a figure panel. Many statistical inconsistencies are typographical. A mature integrity system needs to be able to correct these errors quickly and without stigma, precisely so that it can reserve its heavier machinery for the cases that warrant it. As later chapters show, the absence of a fast, stigma-free route to correction is one of the reasons errors and fraud are so often tangled together in practice. Authors resist correcting an honest mistake because a correction looks like an admission, and institutions hesitate to demand one because it might. The grey zones around the core Between the hard core of fabrication and falsification and the soft periphery of questionable practice lie several categories that matter to forensic investigators because they generate many of the anomalies found in the literature. The first is duplicate publication, in which substantially the same data are published more than once without disclosure, sometimes with different framing or in a different language. It inflates the apparent weight of evidence, which matters when trials are pooled in meta-analyses, since the same patients are counted twice. Closely related is the reuse of figures from one's own earlier papers to represent new experiments. When an image published in 2015 as showing one cell line reappears in 2019 as showing another, the second use is not self-plagiarism in any harmless sense. It is a misrepresentation of what the 2019 experiment produced, and so falls under falsification if done knowingly. The second is authorship manipulation: adding people who contributed nothing, omitting people who did, or, as Chapter 5 describes, selling places on the author list. Authorship matters for integrity not only because it misallocates credit but because it determines who is responsible when something goes wrong. A paper whose listed authors cannot explain how a figure was made has a responsibility gap that makes investigation harder. The third is manipulation of the review and citation system itself: fabricated reviewer identities, reviews written by the authors under false names, and coordinated citation of one another's work to inflate metrics. These practices do not falsify data directly, but they defeat the mechanisms meant to catch falsified data, and several publishers have retracted large batches of papers on the ground that peer review was compromised. What unites these grey zones is that each leaves a documentary trace, in duplicated text, repeated images, reviewer email addresses or citation patterns, that can be found without access to anyone's laboratory. That is why forensic reading of the literature has become so productive: much of what corrupts the record is visible in the record. How common is it? Estimating the prevalence of something people have strong reasons to conceal is inherently difficult. Three kinds of evidence exist: surveys in which researchers report their own behaviour and what they have witnessed in others, audits of samples of published papers, and the record of retractions. Each has predictable biases, and each tells part of the story. The most cited survey evidence is a meta-analysis by Daniele Fanelli, published in PLoS ONE in 2009, which pooled the results of eighteen surveys. On average about 2 percent of scientists admitted to having fabricated, falsified or modified data or results at least once, and up to about a third admitted to other questionable practices. When asked about colleagues rather than themselves, respondents reported much higher figures: around 14 percent said they had observed colleagues falsifying, and up to 72 percent reported questionable practices by others. The gap between self-report and report about others is itself informative. It may reflect underreporting of one's own behaviour, the fact that one bad actor can be observed by many colleagues, or both. A more recent and larger estimate came from the Dutch National Survey on Research Integrity, whose results were published in PLOS ONE in 2022 by Gowri Gopalakrishna and colleagues. The survey used a technique designed to encourage honest answers about sensitive behaviour and drew on nearly seven thousand respondents across all disciplines in the Netherlands. It found that 4.3 percent of respondents reported having fabricated data and 4.2 percent reported falsification in the previous three years, with 8.3 percent reporting at least one of the two. Just over half reported engaging frequently in at least one questionable research practice. Rates of fabrication and falsification were highest among PhD candidates and junior researchers and in the life and medical sciences. Earlier work in psychology by Leslie John, George Loewenstein and Drazen Prelec, published in Psychological Science in 2012, surveyed around two thousand academic psychologists in the United States and found that questionable practices such as failing to report all dependent measures and deciding to collect more data after checking significance were admitted by large proportions of respondents. That study helped trigger what became known as the replication crisis in psychology, and it is important to the argument here because it showed that the boundary between ordinary bad habit and falsification is heavily populated. The main surveys are summarised in Table 1, which is intended to convey the order of magnitude rather than a precise rate, since the samples, time windows and question wording differ. Table 1. Selected survey estimates of fabrication, falsification and questionable practice. Study Population Fabrication or falsification Questionable practices Fanelli (2009), PLoS ONE Meta-analysis of 18 surveys About 2% admitted at least once; about 14% observed in colleagues Up to about 34% admitted; up to 72% observed in colleagues John, Loewenstein and Prelec (2012), Psychological Science About 2,000 US psychologists Low admission rates High admission rates for several practices Gopalakrishna et al. (2022), PLOS ONE About 6,800 researchers in the Netherlands 8.3% reported at least one in previous three years 51.3% reported at least one frequently Audits of published papers give a different view. The most important for this book is the study by Elisabeth Bik, Arturo Casadevall and Ferric Fang, published in mBio in 2016, which examined images in more than twenty thousand papers from forty biomedical journals and found that about 3.8 percent contained a problematic duplication. Chapter 2 examines that study in detail. In clinical research, the anaesthetist John Carlisle's analysis of trials submitted to the journal Anaesthesia, discussed in Chapter 4, found false data in a striking proportion of trials for which individual patient data were available. Audits of this kind do not measure intent, but they measure something arguably more important: the fraction of the record that cannot be trusted as it stands. What retractions tell us, and what they do not The retraction record is the most visible measure and the most misleading. A retraction happens only when a problem has been detected, reported, investigated and acted on, and each of those steps filters out cases. Retraction rates therefore measure the efficiency of the correction system at least as much as the prevalence of the underlying problem. The best-known analysis of why papers are retracted is by Ferric Fang, R. Grant Steen and Arturo Casadevall, published in the Proceedings of the National Academy of Sciences in 2012. They examined more than two thousand retracted biomedical and life-science articles indexed in PubMed and, by consulting secondary sources such as ORI findings and news reports as well as retraction notices, found that about two-thirds were attributable to misconduct, including fraud or suspected fraud, duplicate publication and plagiarism. Only about a fifth were attributable to error. The study's broader lesson was that retraction notices themselves were often uninformative or misleading about the reason, so that reading notices alone understated the role of misconduct. The absolute numbers have grown sharply. The Retraction Watch database, founded by the journalists Ivan Oransky and Adam Marcus and acquired in 2023 by Crossref, the nonprofit that registers digital object identifiers for scholarly works, which then made it openly available, now records tens of thousands of retractions. In 2023 more than ten thousand papers were retracted in a single year, a record driven largely by mass retractions at the publisher Hindawi, then owned by Wiley, of papers linked to paper mills and compromised special issues. Chapter 5 examines that episode. Even so, retractions remain a very small fraction of the literature, on the order of a few papers in every ten thousand, far below any plausible estimate of the fraction that contains fabricated or falsified material. The gap between those two numbers is the space in which this book operates. Why it matters It is tempting to treat fraud as a problem for the careers of the people involved, and to assume that the scientific process will absorb the damage because false results fail to replicate. The second assumption is weaker than it sounds. Replication is expensive and rarely attempted directly. A fabricated result in a crowded field can shape grant priorities, animal experiments and drug development for years before anyone establishes that it cannot be reproduced, and the failure to reproduce it is often attributed to technical differences rather than to the original being false. The costs are concrete. An analysis by Andrew Stern, Arturo Casadevall, R. Grant Steen and Ferric Fang, published in eLife in 2014, examined papers retracted as a result of research misconduct identified by ORI between 1992 and 2012 and estimated that they accounted for roughly 58 million dollars in direct NIH funding, a small fraction of the agency's budget but a substantial sum in absolute terms. That figure does not include the downstream work built on retracted findings, the patients enrolled in trials justified by them, or the careers of junior colleagues whose names appeared on compromised papers. In clinical medicine the stakes are higher still. When fabricated trials enter systematic reviews, they shift pooled estimates of treatment effects and therefore clinical guidelines. The case of the Japanese anaesthetist Yoshitaka Fujii, whose trials were examined statistically by John Carlisle in 2012 and subsequently investigated by the Japanese Society of Anesthesiologists, led to the retraction of more than a hundred and seventy papers, many of them trials of drugs for postoperative nausea. Those trials had been included in meta-analyses. A fabricated trial is not an isolated lie; it is a contaminant that spreads through every synthesis that includes it. That, in the end, is why the rest of this book is concerned less with punishing individuals than with the integrity of the record. The forensic methods that follow are ways of reading a document and asking whether it could have been produced by the process it describes. When the answer is no, the most urgent question is not who is to blame but who is relying on it, and how quickly they can be told. Chapter 2. The Anatomy of a Manipulated Image Most of the evidence in experimental biology is ultimately visual. A Western blot shows whether a protein is present and in roughly what quantity. A micrograph shows the shape of cells, the location of a fluorescent marker, the architecture of a tissue. A gel shows fragments of DNA sorted by size. A flow cytometry plot shows how a population of cells divides across two measured properties. In each case the published figure is offered as a faithful record of what the instrument recorded, and readers are invited to see the result for themselves. That invitation is what makes images such powerful evidence, and it is also what makes their manipulation such a direct form of deception. A falsified number in a table can be hidden among other numbers. A falsified image is placed at the centre of the argument. What an honest image looks like To understand manipulation it helps first to understand what an unmanipulated scientific image contains. Every digital image produced by a camera or scanner carries, in addition to the signal of interest, a great deal of incidental information: electronic noise from the sensor, variations in background intensity, dust on the lens or the scanner glass, scratches on a membrane, uneven staining, the particular granularity of film if the image was scanned from an X-ray film exposure. In a Western blot, the background around each band has a texture that is effectively unique. In a micrograph of cells, the exact arrangement of every cell in the field is unique, as is the pattern of debris between them. This incidental information is the forensic analyst's friend. Two independent experiments will never produce identical noise, identical dust and identical arrangements of cells. If two panels that are supposed to show different samples share the same background blemishes in the same relative positions, they came from the same original image, whatever their labels say. The signal itself can be similar between experiments; the noise cannot. Honest image processing is permitted and often necessary. The guidelines that shaped modern practice were published in the Journal of Cell Biology in 2004 by Mike Rossner, then the journal's managing editor, and Kenneth Yamada, in an article titled "What's in a picture? The temptation of image manipulation." Their core rules have since been adopted, with local variations, by most life-science journals. No specific feature within an image may be enhanced, obscured, moved, removed or introduced. Adjustments of brightness, contrast or colour balance are acceptable if they are applied to the whole image and do not obscure, eliminate or misrepresent any information present in the original, including the background. Grouping images from different parts of the same gel, or from different gels, fields or exposures, must be made explicit by the arrangement of the figure, for example with dividing lines, and in the legend. Nonlinear adjustments, such as changes to gamma settings, must be disclosed. These rules are simple, and they draw a clear line. Cropping a blot to show the relevant bands is acceptable. Cutting out a lane and closing the gap so that the result looks like a continuous gel, without marking the join, is not, because it conceals that the lanes were not run side by side. Raising the contrast of a whole image so a faint band becomes visible is acceptable if the background is raised too. Painting over a band that should not be there is not. The Journal of Cell Biology did more than publish guidelines; it began screening every accepted manuscript's figures before publication. Rossner later reported that over several years roughly a quarter of accepted manuscripts contained at least one figure that had to be remade because it had been manipulated in a way that violated the guidelines, most of them for reasons that did not affect interpretation, and that about one percent were found to contain manipulations that did affect the conclusions, and so had their acceptance revoked. Those figures established two things early: that guideline violations were common, and that most of them were not fraud. The taxonomy of problems The most influential attempt to classify image problems in the published literature, rather than in submissions, is the study by Elisabeth Bik, Arturo Casadevall and Ferric Fang in mBio in 2016. Bik, then a microbiologist working in industry, examined by eye the images in 20,621 papers published between 1995 and 2014 in forty biomedical journals. She flagged papers that appeared to contain inappropriately duplicated images, and each flagged paper was reviewed by her two coauthors, with a paper counted only if all three agreed. The final tally was 782 papers, about 3.8 percent. The rate varied greatly between journals, from well under one percent at the Journal of Cell Biology, which had screened images for years, to more than twelve percent at the International Journal of Oncology. The proportion of problematic papers rose markedly after about 2003, around the time digital image handling became routine. The study sorted problems into three categories, which have since become the working vocabulary of the field. As Table 2 sets out, the categories differ in what is duplicated, how, and how readily an innocent explanation is available. Table 2. Categories of inappropriate image duplication, after Bik, Casadevall and Fang (2016). Category What is seen Typical innocent explanation Weight as evidence of intent Simple duplication The same image used twice to represent different conditions Wrong file selected during figure assembly Low on its own Duplication with repositioning Overlapping or identical images shifted, rotated, flipped or rescaled Rarely plausible; repositioning requires deliberate action Moderate to high Duplication with alteration Regions cloned, stamped, spliced or erased within or between images Very rarely plausible High The first category is the most common and the most forgiving. A paper that shows the same loading-control blot under two different experiments, or the same micrograph labelled as two different treatments, may well be the product of a mistake. Laboratories store thousands of images, often with sequential or nearly identical file names, and a tired postdoctoral researcher assembling a figure late at night can easily pick the wrong one. Such errors still invalidate the figure, because one of the two panels does not show what it claims, but they do not in themselves suggest dishonesty. They call for a correction backed by the original data. The second category involves an extra step. If a micrograph appears in two figures but one version has been rotated by ninety degrees, or mirrored, or cropped to a different region that overlaps the first, someone performed an action beyond selecting a file. A rotated duplicate is harder to explain as a slip of the mouse, because it makes the duplicate harder to recognise, which is the effect one would expect if concealment were the aim. That is not proof: people rotate images for layout reasons, and overlapping fields of view can arise legitimately if the same slide was imaged twice and both images were then mislabelled. But the innocent explanations are narrower. The third category is the most serious. Here the image has been altered internally: a region has been copied from one part of an image to another, a band has been pasted into a lane, a patch of background has been cloned over something that was there, or pieces from different images have been combined without disclosure. Bik and her coauthors judged that at least half of the problematic papers in their sample showed features in the second or third categories, suggestive of deliberate manipulation. Alteration of this kind is very difficult to produce by accident. A cloned region in a histology image, where the same cluster of cells appears twice in one field, or a Western blot in which the background around one band is identical to the background around another, requires a person to select a region and paste it. The characteristic forms of alteration Within the third category, certain patterns recur so often that experienced analysts recognise them almost at a glance. Band splicing without disclosure. A Western blot is assembled from lanes that were not run on the same gel, or not adjacent on the same gel, and the joins are hidden. Tell-tale signs include a sharp vertical discontinuity in the background, a change in background texture or intensity from one lane to the next, or bands that sit at slightly different heights from their neighbours in a way inconsistent with a single run. Splicing is sometimes legitimate if disclosed, which is why the rules require dividing lines; undisclosed, it conceals the conditions under which the comparison was made. Band duplication. A single band, or a set of bands, appears more than once, sometimes in the same blot to represent different samples, sometimes in different blots in the same or different papers. The shapes of bands vary from run to run in small, characteristic ways, so identical shapes with identical surrounding noise indicate a common source. Erasure and background cloning. Something is removed from an image and the gap is filled with a patch of background copied from elsewhere, or with a uniform tone. Erased areas can be detected when the patch has a noise pattern identical to another region, or when it has no noise at all, which real images never lack. Adjusting the contrast of a suspect image to an extreme setting often reveals rectangular or irregular areas whose texture does not match their surroundings. The retraction notice for the 2006 Nature paper on Aβ*56 referred specifically to the use of an eraser tool, alongside splicing and duplication. Cloned cells and tissue. In micrographs, clusters of cells or features of tissue are duplicated within a single field, sometimes to make a population look denser or a treatment effect look stronger. Because the arrangement of cells in any real field is unique, a repeated constellation is a strong signal. Recycled images across papers. The same image appears in papers by the same group published years apart, representing different cell lines, different treatments or even different organisms. This pattern is especially important because it can only be detected by looking across papers, which neither reviewers nor editors usually do. Duplicated plots and spectra. The same logic applies beyond photographs. Flow cytometry dot plots, in which each dot represents a single cell, contain thousands of randomly positioned points; two plots with identical scatter are the same data. Spectra, chromatograms and other traces carry noise that functions like the background in a blot. A worked examination It helps to see how an experienced analyst approaches a single suspect figure. Consider a composite of the kind common in cell biology papers: a Western blot showing a protein of interest across six lanes representing a control and five treatments, above a second blot showing a loading control, such as actin, for the same six lanes. A reader has noticed that the bands in lanes two and five look alike. The analyst begins by obtaining the highest-resolution version of the figure available, usually from the publisher's website or the supplementary files, since images embedded in a downloaded article file are often compressed. She crops the two lanes and places them side by side at high magnification. The bands are similar in shape, but that alone means little; bands of the same protein often look alike. She therefore looks away from the bands, at the background. In the lanes in question there is a small dark speck just above and to the right of each band, at the same distance, and a faint diagonal streak running through both. Those features are incidental, and their repetition is the first real evidence of common origin. She then adjusts contrast and gamma across the whole figure. At extreme settings the background of the blot, which looked uniform, resolves into a mottled texture. Between lanes four and five a sharp vertical line appears, with slightly different background intensity on either side, suggesting that lane five was placed there from elsewhere. She overlays lane two on lane five with partial transparency and finds that the speck, the streak and the mottled texture align exactly, with no rotation. She checks the loading-control blot beneath and finds that it shows no such join, which means that the protein-of-interest blot and its loading control do not share a common history, even though the figure presents them as a matched pair. Finally, she searches for the same image elsewhere: in the other figures of the same paper, and, using image-comparison software, in earlier papers by the same group. She finds nothing further. Her report records what she found and nothing more: that lanes two and five of the upper blot appear to derive from the same original image, that there is a discontinuity consistent with undisclosed splicing between lanes four and five, and that the upper blot and loading control appear not to derive from a single membrane as the layout implies. She does not say that anyone committed fraud. She asks the authors and journal for the original, uncropped images. What happens next depends on whether those images exist and what they show. From image to inference The fact that an image has been duplicated or altered tells the reader that the figure does not show what it claims. That is a sufficient reason to correct or retract the paper if the original data cannot be produced. It is not, on its own, a finding about who did it or why, and the investigators who find these problems are generally careful to say so. Several considerations shape the inference. The first is whether the problem affects the conclusions. A duplicated loading control in a supplementary figure is less consequential than a duplicated result panel that constitutes the main evidence for the paper's claim. The second is pattern. A single duplication in a large paper is more consistent with error than a series of duplications across several papers from the same group, particularly if they span different journals, years and first authors. The third is the response. Authors who can promptly produce the original, unprocessed images showing what the figure should have contained are demonstrating that the underlying experiment was performed. Authors who cannot, or who produce new images that were evidently generated after the question was raised, are demonstrating something else. Some of the most consequential episodes in the history of the field turned on images. In 2005 the South Korean researcher Hwang Woo-suk and colleagues published in Science a paper claiming to have derived patient-specific embryonic stem cell lines by cloning. Within months, anonymous users of a Korean online forum for biologists had pointed out duplicated photographs of cell colonies in the paper's supplementary material. An investigation by Seoul National University concluded that the data in the 2005 paper, and in an earlier 2004 paper in Science, had been fabricated, and both papers were retracted. The image duplications did not by themselves prove fabrication, but they were the thread that, once pulled, unravelled the rest. The Stanford case of 2022 and 2023 followed a similar logic. Concerns about images in papers on which Marc Tessier-Lavigne, then Stanford's president, was an author had been posted on PubPeer, a website on which readers can comment on published papers, some years earlier, and were brought to wide attention by reporting by Theo Baker in the student newspaper The Stanford Daily. A scientific review panel commissioned by the university's board of trustees reported in July 2023. According to the report as published, the panel found no evidence that Tessier-Lavigne had himself manipulated data or had known of the manipulation when it occurred, but it identified repeated instances of manipulation of research data or subpar scientific practices in papers from laboratories he led, and found that he had not taken adequate steps to correct problems when they were raised. He announced his resignation as president and said he would retract or correct several papers; retractions of papers in Cell and Science followed. Both cases show the same structure. The image problem is the entry point. It establishes that something in the record is wrong. What it cannot establish, and what only an investigation with access to people and original data can, is how the wrong thing came to be there. The next chapter turns to the tools that now allow image problems to be found at a scale no individual could manage by eye, and to the new difficulty of images that were never photographed at all. Chapter 3. Screening at Scale: Software, Source Data and the Synthetic Image For most of the history of image forensics the principal instrument was the human eye. Elisabeth Bik's survey of more than twenty thousand papers, described in the previous chapter, was done largely by looking. So was much of the work of the volunteers who scrutinise the literature on PubPeer. The eye is remarkably good at this task once trained: it notices a repeated cluster of cells, a familiar blemish, a band whose shape it has seen before. But it does not scale. A single analyst can examine perhaps a few dozen papers carefully in a day. The world's journals publish several million papers a year. Between those numbers lies the case for automation, and over the past decade software has begun to close part of the gap. This chapter describes what the tools do, how they work in principle, what they miss, and the arrival of a problem they were not built to solve. The analyst's manual toolkit Before automated screening became widespread, image forensics relied on a small set of techniques that remain useful and that explain much of what the software now does. The simplest is aggressive adjustment of brightness, contrast and gamma. An image that looks clean at normal settings can reveal its history when pushed to extremes. Pasted regions often carry a slightly different background level or noise texture from their surroundings; at high contrast those differences become visible as rectangles, sharp edges or areas of unnatural smoothness. Applying a false-colour lookup table, which maps small differences in grey level to strongly contrasting colours, achieves a similar effect. The US Office of Research Integrity has for many years distributed a set of simple scripted actions, known as forensic droplets, for Adobe Photoshop that apply these kinds of transformations to an image so that an investigator can inspect it quickly. The open-source scientific image program ImageJ, and its widely used distribution Fiji, provide equivalent capabilities. A second technique is direct comparison. Two suspect panels are overlaid, one made semi-transparent or subtracted from the other. If they share an origin, the subtraction leaves almost nothing, or the overlay reveals perfect alignment of every speck of noise. If one has been rotated, flipped or rescaled, the analyst applies the inverse transformation before comparing. A third is error level analysis, which exploits the way JPEG compression works. When a JPEG image is saved, it is divided into blocks and compressed with some loss of information. Resaving an image at a known quality level and measuring how much each region changes can reveal areas that were compressed a different number of times from the rest, as can happen when content from another image is pasted in. Error level analysis is available in free online tools such as FotoForensics and Forensically, the latter of which also offers clone detection and noise analysis in a browser. It must be used with care. Many innocent operations, including resizing, format conversion and the publisher's own processing of figures, produce patterns that look suspicious to an untrained eye, and the technique has a history of being misread in public disputes. It is a pointer for closer inspection, not evidence in itself. The common thread in all of these methods is that they look for inconsistency: between two images that should differ but do not, or within one image whose parts should share a history but do not. That principle carries over directly into automated tools. How automated screening works Commercial image-integrity software emerged in the late 2010s and has spread quickly. The two products most widely used by publishers are ImageTwin, developed in Austria, and Proofig, developed in Israel. Several large publishers have also built tools of their own. The details of the algorithms are proprietary, but the general approach is well understood. The software first extracts every image from a manuscript, separating composite figures into their component panels. It then looks for matching regions, both within a single panel and across all panels in the paper. The matching relies on techniques from computer vision that identify distinctive local features in an image, such as corners, edges and textured patches, describe them in a way that is robust to rotation, scaling and changes in brightness, and then search for the same features elsewhere. Where a large number of features match in a consistent geometric arrangement, the software reports a probable duplication and shows the analyst the two regions side by side, often with the transformation that links them. This is essentially copy-move detection, and it is well suited to finding the second and third categories of the Bik taxonomy, repositioned and altered duplicates, which are exactly the ones a tired human is most likely to miss. The more powerful capability, and the one that has changed the economics of detection, is comparison across papers. A duplication within one manuscript can be found by comparing the manuscript to itself. The reuse of an image from a paper published five years earlier, perhaps in another journal, can only be found by comparing the manuscript against a large database of previously published figures. Some vendors have assembled such databases from openly available literature, which allows a submission to be checked against millions of earlier images. The coverage of these databases is limited by what the vendor can legally obtain and index, and a figure published behind a paywall, or in a journal that has not shared its content, may be invisible to them. This is one of several places where the integrity of the record depends on cooperation between publishers that commercial incentives do not naturally produce. Adoption has moved from pilots to policy. In January 2024 the editor-in-chief of the Science family of journals, Holden Thorp, announced in an editorial titled "Genuine images in 2024" that the journals would use Proofig to screen images in papers under consideration, having piloted it for several months with, in his words, clear evidence that problematic figures could be detected before publication. Other publishers have announced screening of all or most submissions, and the STM association's Integrity Hub, a shared infrastructure launched by publishers to detect problems across submissions, has worked on duplicate-image and duplicate-submission detection across participating publishers. What software misses and what it overcalls Automated screening is valuable, but its limits are important, because they shape what the rest of the integrity system must do. It produces false positives. Many legitimate scientific images contain repeating structures: the regular pattern of wells in a plate, the repeated motif of a crystal lattice, the similar appearance of many cells of the same type, scale bars and labels that recur across panels. A loading control may legitimately be shown more than once if the same blot was probed for several proteins and the paper says so. The software flags matches; a trained human must decide whether they matter. Publishers that have adopted these tools generally describe them as triage, with every flag reviewed by an editor or integrity specialist before any action is taken. It produces false negatives. Small duplicated regions, heavily processed images, low-resolution figures and duplications involving images outside the reference database can all escape detection. So can manipulation that does not involve duplication at all: a band that has been darkened, a background that has been uniformly smoothed, a quantification that does not match the image it accompanies. Most fundamentally, duplication-based software can only find a manipulation that leaves a copy behind. It detects reuse. It does not detect invention. An image that was fabricated from nothing, or taken from an experiment different from the one described but never published anywhere, contains no duplicate to find. That limitation was always there, but until recently it mattered less, because inventing a convincing scientific image from nothing was hard. It is no longer hard. The arithmetic of screening A further limit is statistical rather than technical, and it is often overlooked by those who expect software to settle cases. It concerns base rates. Suppose, purely for illustration, that a journal receives a thousand manuscripts containing images, and that 4 percent of them, forty papers, contain a genuine problem, a figure close to the rate Bik and her colleagues found in the published literature. Suppose also that the screening software is good: it detects 90 percent of genuine problems and wrongly flags only 5 percent of clean papers. The software will correctly flag 36 of the 40 problematic papers. It will also flag 5 percent of the 960 clean ones, which is 48 papers. Of the 84 papers flagged in total, then, more than half are clean. And four genuinely problematic papers pass unflagged. The numbers are hypothetical, but the structure is general. Whenever the thing being screened for is rare, even an accurate test produces many false positives relative to true ones. This is why a flag from screening software cannot by itself justify a rejection, still less an accusation, and why every serious user of these tools treats a flag as the start of a human review. It is also why the most useful output of a screening tool is not a verdict but a precise pointer to the regions of an image that match, which a human can then examine with the methods described earlier in this chapter. The same arithmetic runs in the other direction for the published literature, where screening is being applied retrospectively to millions of papers. A tool with a small false-positive rate, applied at that scale, will flag very large numbers of innocent papers in absolute terms. If those flags are posted publicly without review, they can cause real harm to authors who did nothing wrong. The responsible practice among analysts who use these tools on published work is to verify every flag by eye before raising it, which returns part of the bottleneck to human attention. The synthetic image problem Generative machine-learning models, first generative adversarial networks and more recently diffusion models, can produce photographs of faces, landscapes and objects that are difficult to distinguish from real ones. The same techniques can be applied to scientific images. A model trained on a large collection of Western blots can generate new blots, with plausible bands, plausible backgrounds and entirely novel noise. Researchers demonstrated several years ago that such synthetic blots could be produced and that experienced scientists had difficulty distinguishing them from real ones. Similar results have been shown for histology and microscopy images. A synthetic image defeats duplication detection by construction. Its noise is unique, because it was generated rather than copied. It will not match any image in any database. Techniques for detecting generated images exist, based on statistical regularities that current generators leave in the frequency content of an image or on inconsistencies in fine structure, but they are in an arms race with generators that improve continually, and a detector trained on one generation of models may fail on the next. There is already evidence that fabricated images of a more primitive kind were being produced at scale before generative models became widely available. In 2020 Bik and several other analysts who publish under pseudonyms, including those known as Smut Clyde, Morty and Tiger BB8, described a group of more than four hundred papers that shared a distinctive style of Western blot, with bands of a characteristic tadpole-like shape on similar backgrounds. The papers came from different hospitals and research groups, mostly in China, and the blots were not exact duplicates of one another. What linked them was a common appearance, suggesting that they had been produced by the same source rather than by the independent experiments the papers described. The group became known as the Tadpole paper mill. The episode is instructive because detection depended not on finding identical pixels but on recognising a shared manufacturing style, which is a far harder task for software and one that synthetic images will make harder still. Chapter 5 returns to paper mills in detail. From detection to provenance If the forensic analysis of published images is becoming less reliable as generation improves, the natural response is to stop relying on the published image alone and to demand the evidence behind it. This is a shift from detection to provenance: rather than asking whether an image looks manipulated, asking whether it can be traced back to an original acquisition. Several journals and publishers already require or request source data. EMBO Press was an early proponent of publishing the source data behind figures alongside the paper. Many journals now require uncropped, unprocessed versions of all blots and gels as supplementary files, so that readers can see what surrounded the displayed bands. Some funders and institutions require raw data to be retained for a set period and produced on request. These requirements raise the cost of fabrication because they require the fabricator to produce not one convincing image but a consistent set of originals, with plausible file formats, metadata and surrounding context. Stronger forms of provenance are technically feasible. Microscopes, gel imagers and plate readers could sign their output files cryptographically at the moment of acquisition, embedding a verifiable record of which instrument produced the image and when. Industry standards for content provenance, developed largely with news photography in mind, provide a model. Laboratory data management systems could record a hash of every raw file as it is created, making later substitution detectable. None of this is yet widespread in academic research, and all of it faces practical obstacles: older instruments, proprietary file formats, the cost of storage, and the reluctance of researchers to accept systems that feel like surveillance. But the direction is clear. As synthetic images make the published figure less trustworthy as evidence in itself, the evidential weight has to move to the chain of custody behind it. What the tools have changed The practical effect of automated screening has been to move a substantial share of image detection from after publication to before it. A duplication caught at submission costs the journal an email and costs the authors an embarrassing request for original data. The same duplication caught ten years after publication may cost a university a year-long investigation, a journal a contested retraction and a field years of work built on a false premise. Screening at submission is therefore enormously cost-effective, and the most important argument for it is not that it catches fraudsters but that it catches errors while they are still cheap to fix. Screening has also changed the incentives, though more slowly than one might hope. A researcher who knows that every image will be checked against millions of others has less reason to reuse one. A paper mill that knows its standard blots will be recognised has an incentive to switch to generated ones. That second effect is a reminder that detection alone cannot end the problem. Each improvement in screening moves the most determined fraud to the next method that screening cannot see. The published literature, meanwhile, still contains everything that was published before screening became routine, and the tools are now good enough to find much of it. The result is a large and growing backlog of flagged papers awaiting action from journals and institutions. The existence of that backlog, and the reasons it is not being cleared, are among the central concerns of the second half of this book. Before turning to them, the next chapter examines the other great body of forensic method: the analysis of numbers. Hashtags: #ScientificIntegrityAndWhistleblowing #ScientificIntegrity #ResearchMisconduct #ResearchFraud #Whistleblowing #ResearchIntegrity #ImageManipulation #ImageForensics #DataFalsification #DataFabrication #WesternBlotManipulation #ImageDuplication #ResearchWhistleblowers #ScientificSleuthing #PaperMills #PublicationEthics #ResearchTransparency #ForensicStatistics #GRIMTest #DataProvenance #SourceData #Retractions #ResearchAccountability #FraudDetection #FutureOfResearchIntegrity

Download the Book (PDF): Introduction In the late summer of 1854, John Snow walked the streets of Soho with a notebook, recording where people had died of cholera. The map he later published, with its stacked bars of deaths clustered around the Broad Street pump, has become the founding image of spatial epidemiology. It is usually told as a story about a map. It is better told as a story about decisions. Snow chose which deaths to count, how to assign each one to a household address, which water sources to mark, and how to measure proximity. He knew that straight-line distance was the wrong measure, so he also traced walking routes through the street network. Every one of those choices shaped what the map could show. The pump handle came off because the choices were good ones. A modern analyst facing the same kind of question has tools Snow could not have imagined. A disease surveillance team can pull geocoded case records from a health information system, overlay them on a population grid at 100-metre resolution, attach land surface temperature from a satellite that passes overhead every day, compute a local cluster statistic in a few seconds, and fit a Bayesian spatial model that borrows strength from neighbouring districts. An ecologist can download a few hundred thousand occurrence records for a mosquito species, stack them against nineteen bioclimatic variables, and project the species' suitable range onto climate scenarios for the end of the century before lunch. What has not changed is that every result still rests on choices about representation, scale and dependence. The difference is that software now makes most of those choices silently. A spatial join assigns each case to a polygon using a rule the analyst may never have read. A raster reprojection resamples values with an interpolation method set by default. A regression treats observations as independent unless told otherwise. A species distribution model draws ten thousand background points from wherever the extent happens to reach. In each case the code runs, the output looks plausible, and the error, if there is one, is invisible in the result. This booklet is built on a single claim: the craft of geospatial analysis consists in making explicit the choices the software would otherwise make for you. The tools in R and Python are now mature, fast, and largely interchangeable. Both ecosystems sit on the same small set of open-source libraries that read file formats, transform coordinates and compute geometric relationships. What separates a defensible analysis from a misleading one is rarely the language or the package. It is whether the analyst knew which coordinate reference system was in play, what spatial support each variable had, how neighbours were defined, how the observations were sampled, and how the model was validated against data that were not simply its own spatial neighbours. Who this is for The intended reader already does some quantitative work in R or Python and has an environmental or health question with a place in it. That reader might be an epidemiologist who has mapped incidence rates by district and wants to know whether the clustering is real, an ecologist who has been handed occurrence records and asked for a distribution map, a public health analyst estimating how many people live within a given distance of a contaminated site, or a graduate student who has inherited a pipeline built on packages that no longer install. The booklet assumes familiarity with ordinary regression and with writing a script. It does not assume training in geography, cartography or remote sensing. Because the reader works in one language and often collaborates with people who work in the other, the booklet treats R and Python side by side. The aim is not to teach either syntax in full. Package documentation does that better, and it changes every year. The aim is to explain what the operations do, where they can go wrong, and what the equivalent tool is called in each ecosystem, so that a reader can move between them without mistaking a difference in naming for a difference in method. What the booklet covers, and what it leaves out The chapters move from the representation of spatial data to its analysis. The first chapter describes the shared software engine beneath both ecosystems and the two data models, vector and raster, that everything else builds on. The second deals with coordinate reference systems, the most common source of quiet error in the field. The third and fourth cover the operations that construct analytical datasets: vector overlays, spatial joins and aggregation for exposure assessment, and raster algebra, resampling and zonal statistics. The fifth turns to satellite imagery, from the physics of reflectance through cloud masking, spectral indices and compositing to the accuracy assessment of classified maps. The last three chapters deal with inference. The sixth covers spatial autocorrelation: how neighbours are defined, what Moran's I and its local variants measure, and how to test for clusters of disease without being fooled by small populations. The seventh covers spatial regression, from the classical lag and error models of spatial econometrics to geographically weighted regression and the hierarchical Bayesian models now standard in small-area disease mapping. The eighth covers ecological niche and species distribution modelling, which has become central to the epidemiology of vector-borne and zoonotic disease, and which has its own well-documented ways of producing confident maps from biased data. Several substantial topics are left out, deliberately. Cartographic design is not covered, because the booklet is concerned with analysis rather than presentation. Point process models receive only brief mention. Network analysis, trajectory analysis of movement data, and deep learning for image segmentation are each large enough to deserve their own treatment and are named only where they touch the main argument. Commercial desktop GIS is not discussed at all. Where a topic is omitted, the reason is that treating it at booklet length would mean treating everything else more thinly. A note on software The R spatial ecosystem went through a significant transition between 2020 and 2023. The packages that had carried it for fifteen years, rgdal, rgeos and maptools, were retired from CRAN in October 2023, and the older raster package has been superseded by terra. Current practice rests on sf for vector data, terra and stars for rasters and data cubes, and spdep, spatialreg and related packages for spatial statistics. In Python, geopandas reached version 1.0 in 2024 and is built on shapely 2.0, whose vectorised geometry operations made large vector workflows practical. Raster work runs through rasterio, xarray and rioxarray, and spatial statistics through the PySAL family of packages. Any booklet that named function signatures in detail would be out of date before it was printed. Package and function names are given where they help a reader find the right tool, and the reader should expect small details to drift. The concepts beneath them, which are the subject of this booklet, do not. How to read it The chapters are ordered so that each builds on the last, and a reader new to the field should take them in sequence. A reader with a specific problem can go directly to the relevant chapter, though the second chapter on coordinate systems is worth reading by everyone, because its errors propagate into every later step. Throughout, the emphasis falls on reasoning rather than recipes. The recurring question is not "which function do I call?" but "what has this operation assumed, and is that assumption true of my data?" An analyst who asks that question at every step will produce work that holds up. One who does not will, sooner or later, produce a beautiful map of an artefact. Chapter 1. One Engine, Two Languages A newcomer to geospatial analysis often spends the first weeks deciding whether to learn it in R or in Python, as though the choice would determine what analyses are possible. It mostly does not. Beneath the surface syntax, both ecosystems call the same compiled libraries to read files, transform coordinates and compute geometric relationships. When an R user reads a shapefile with sf and a Python user reads the same file with geopandas, the bytes pass through the same code. When each reprojects the data, the same transformation pipeline runs. When each asks whether two polygons intersect, the same algorithm answers. Understanding that shared foundation is the first step toward understanding why results agree across languages when they should, and why they occasionally do not. The three libraries underneath Three open-source C and C++ libraries do most of the work in both ecosystems. GDAL, the Geospatial Data Abstraction Library, reads and writes spatial file formats. Its raster side handles GeoTIFF, NetCDF, HDF, JPEG2000 and more than a hundred other formats; its vector side, historically called OGR, handles shapefiles, GeoPackage, GeoJSON, PostGIS connections, KML and many others. GDAL also provides virtual file systems that let software read a file directly from a web server or cloud bucket without downloading it first, a capability that has reshaped remote sensing workflows and is taken up again in Chapter 5. PROJ transforms coordinates between reference systems. Since its sixth major version, released in 2019, PROJ has used a formal description of coordinate reference systems known as WKT2 and a database of transformations maintained from the EPSG registry. The change mattered because the older approach, which described systems with short text strings, silently dropped information about datums. Transformations that had been accurate to metres could be off by a hundred metres or more without any warning. Chapter 2 deals with this at length. GEOS, the Geometry Engine Open Source, is a C++ port of the Java Topology Suite. It computes predicates such as intersects, contains and touches, and operations such as buffers, unions, intersections and convex hulls, on planar geometries. It treats coordinates as if they lay on a flat plane. That assumption is harmless for projected data in metres over a modest area and wrong for longitude and latitude over a large one. For that reason sf in R, from version 1.0 onward, sends geometric operations on geographic coordinates to a different library, s2, which Google developed to compute on the sphere. An intersection or buffer on longitude and latitude in sf is therefore computed on a spherical model by default, while the same operation in geopandas is computed on the plane by GEOS unless the analyst projects the data first. This is one of the few places where the two ecosystems give different answers to what looks like the same call, and it is a direct consequence of which engine each hands the work to. The vector data model Vector data represent the world as discrete objects with explicit boundaries: points, lines and polygons, each carrying attributes. The standard that governs them is the Simple Features specification of the Open Geospatial Consortium, also published as ISO 19125. It defines seven core geometry types: point, linestring, polygon, and their multi-part variants multipoint, multilinestring and multipolygon, plus a geometry collection that can hold a mixture. Both sf and geopandas implement it directly, which is why a GeoPackage written in one reads cleanly in the other. In both ecosystems a vector dataset is a data frame with one special column holding geometries. In sf it is an sf object whose geometry column is a list of geometries with an attached coordinate reference system; in geopandas it is a GeoDataFrame with an active GeoSeries. Everything an analyst knows about data frames carries over: filtering rows, joining tables, grouping and summarising. The spatial operations are additional verbs that act on the geometry column. Simple Features has rules about validity that matter more than they appear to. A polygon's exterior ring must not cross itself; interior rings, which represent holes, must lie inside the exterior ring and must not overlap each other. Administrative boundary files, especially those digitised by hand or simplified for web display, frequently break these rules. An invalid polygon can make an intersection fail outright, or worse, return a result that is wrong without raising an error. Both ecosystems provide validity checks and repair functions, st_is_valid and st_make_valid in sf, is_valid and make_valid in geopandas. Running the check on every boundary file before analysis costs seconds and prevents a class of error that is otherwise very hard to trace. A second subtlety concerns topology. Simple Features stores each polygon independently. Two neighbouring districts share a boundary, but each stores its own copy of the vertices along it. If the file has been simplified, the copies may no longer coincide, leaving slivers of overlap or gaps between districts. This becomes important in Chapter 6, where the definition of which areas are neighbours depends on whether their boundaries touch. A gap of a few metres between two districts that obviously share a border will make them non-neighbours under a strict contiguity rule, and the resulting spatial weights will be wrong. The raster data model Raster data represent the world as a regular grid of cells, each holding a value. A satellite image is a raster; so is a digital elevation model, a gridded climate surface, or a population density layer. The grid is defined by its extent, its cell size, its number of rows and columns, and its coordinate reference system. Values are stored in bands, so a single raster file might hold one band of elevation or thirteen bands of multispectral reflectance. The raster model has two conceptual traps. The first concerns what a cell value means. In a digital elevation model, the value may represent the elevation at the cell centre. In a satellite image, it represents a weighted average of reflected light over roughly the cell's footprint, blurred by the sensor's point spread function. In a population grid, it represents a count of people within the cell. In a climate surface, it is an interpolated estimate from station data, with uncertainty that the file does not record. These are different kinds of quantities, and they behave differently when the grid is resampled or aggregated. Averaging population counts to a coarser grid is wrong; they must be summed. Summing temperatures is meaningless; they must be averaged. The software does not know which kind of quantity a layer holds, so the analyst must. The second trap concerns alignment. Two rasters can have the same cell size and coordinate reference system and still be offset from each other by half a cell, because their grid origins differ. Map algebra between them, such as dividing one by the other, requires that cells coincide. If they do not, one layer must be resampled onto the other's grid, and resampling changes values. Chapter 4 treats this in detail. In R, terra is the main package for raster data. It was written by Robert Hijmans as the successor to his earlier raster package, with a C++ core that makes it much faster and able to process files larger than memory by working through them in blocks. The stars package, developed by Edzer Pebesma, takes a different approach: it represents spatiotemporal arrays with any number of dimensions, which suits data cubes of many images over time. In Python, rasterio provides a thin, explicit interface to GDAL for reading and writing raster files, while xarray provides labelled multidimensional arrays, and rioxarray connects the two so that an xarray object knows its coordinate reference system and geotransform. Data cubes and the blurring of the models The distinction between vector and raster is less absolute than it once was. Satellite archives now deliver thousands of images of the same place over time, and analysts increasingly treat them as a single four-dimensional array indexed by band, time, and two spatial coordinates. This is the data cube model. In Python, xarray has become the standard container for it, often backed by dask so that computations are split into chunks and run lazily across many processors. In R, stars fills the same role, and the gdalcubes package builds regular cubes from irregular collections of images. Vector data can also be cubed. A set of district polygons with weekly case counts over five years is a vector data cube: one spatial dimension indexed by district and one temporal dimension indexed by week. stars supports this directly, and the representation is natural for spatiotemporal disease surveillance. The practical consequence is that the analyst should think less about which data model a file uses and more about what the dimensions of the problem are. A question about how vegetation greenness in the three months before an outbreak relates to case counts per district involves a raster cube, a vector cube, and an aggregation from one to the other. The software can perform each step, but only the analyst can say which aggregation respects the meaning of the variables. Reading the same data in both languages The ecosystems have converged enough that most operations have a direct counterpart in the other language, as Table 1 sets out. The table lists the main package for each task. Names change over time, but the pairing has been stable for several years. Table 1. Principal packages for common geospatial tasks in R and Python. Task R Python Vector data frames sf geopandas, shapely Raster files terra rasterio, rioxarray Data cubes stars, gdalcubes xarray, dask Coordinate transformation sf, terra (via PROJ) pyproj Spatial weights and autocorrelation spdep libpysal, esda Spatial regression spatialreg, GWmodel, R-INLA spreg, mgwr Zonal statistics exactextractr rasterstats, exactextract Species distribution models dismo, predicts, biomod2, ENMeval elapid, scikit-learn Several points about the pairing are worth stating. First, the R side of spatial regression and Bayesian disease mapping is considerably deeper than the Python side. The R-INLA package, which fits latent Gaussian models by integrated nested Laplace approximation, has no mature Python equivalent, and much of the small-area estimation literature publishes code in R. Second, the Python side is stronger in large-scale remote sensing, machine learning and cloud-native processing, because xarray, dask and the scientific Python machine learning stack were built with large arrays in mind. Third, the SDM tooling in R is more mature and more thoroughly tested, though the elapid package, described in the Journal of Open Source Software in 2023, provides a Python implementation of Maxent and related tools. When the two disagree Because both ecosystems use GDAL, PROJ and GEOS, they usually agree to many decimal places. When they do not, the cause is almost always one of four things. The first is the spherical versus planar distinction already described. A buffer of one degree around a point on geographic coordinates means different things on a sphere and on a plane, and the two ecosystems default to different ones. The second is library versions. sf and geopandas may be linked against different versions of PROJ, and a newer PROJ may use a more accurate transformation grid that an older one lacked. Both ecosystems report the versions of their linked libraries, through sf_extSoftVersion in R and geopandas.show_versions in Python, and those versions belong in any methods section that reports transformed coordinates. The third is default parameters. Resampling methods, the treatment of cells that are partly inside a polygon during zonal statistics, and the handling of missing values all have defaults, and the defaults differ between packages. A mean temperature by district computed in one package may differ from another because one counted only cells whose centres fell inside the district while the other weighted every cell by the fraction of its area inside. The fourth is the definition of neighbours in spatial statistics. Queen and rook contiguity, distance thresholds, the number of nearest neighbours and the row standardisation of weights are all choices, and the defaults in spdep and libpysal are not always identical. None of these differences is a bug. Each is a choice that one package made on the analyst's behalf. When results from the two languages disagree, the disagreement is useful: it reveals a choice that had not been made explicit. The remainder of this booklet is, in a sense, a catalogue of such choices and of how to make each one deliberately. File formats and why they matter The choice of file format is usually treated as a matter of convenience, but several formats in common use carry limitations that shape analysis. The shapefile, introduced by Esri in the early 1990s, remains the most widely distributed vector format, and it is a poor one. A single dataset is spread across at least three files, and often six or seven, which travel separately and are easily separated. Attribute field names are limited to ten characters, so a column named population_2020 becomes truncated on writing, sometimes colliding with another truncated name. Text encoding is ambiguous, which corrupts place names with accented characters. Individual component files cannot exceed two gigabytes. There is no way to store a proper null value in numeric fields, so missing values are often written as zero, which is catastrophic for a column of case counts. GeoPackage, an OGC standard built on SQLite, avoids all of these problems in a single file, and it should be the default for exchanging vector data. For large analytical datasets, GeoParquet, which stores geometries in the columnar Apache Parquet format, has become the fastest option in both ecosystems; geopandas reads and writes it natively, and sf can do so through GDAL or the sfarrow and geoarrow packages. FlatGeobuf fills a similar niche for streaming vector data over the web. For rasters, the GeoTIFF remains standard, and its Cloud Optimized variant, the COG, organises the file internally in tiles with overviews so that software can request only the portion it needs over a network. NetCDF and HDF5 are common for climate and atmospheric data with many time steps. Zarr, a chunked array format designed for cloud storage, is increasingly used for large data cubes and is read natively by xarray. The format affects not only speed but correctness: a GeoTIFF records a nodata value in its header, and software that ignores it will treat the fill value, often minus 9999 or zero, as a real measurement. Reproducibility as a design constraint Geospatial workflows are harder to reproduce than most statistical work. They depend on compiled libraries whose versions matter, on large input files that may be revised by their publishers, and on web services that change. A climate surface downloaded in one year may be silently updated the next; a satellite archive may be reprocessed to a new collection, changing every reflectance value slightly. The practical defences are straightforward. Record the versions of GDAL, PROJ and GEOS alongside package versions. Use environment managers, renv in R and conda or pixi in Python, which can pin compiled libraries as well as packages. Record the exact version or collection identifier of every remote dataset, and where possible the date it was retrieved. Store intermediate products in open formats such as GeoPackage for vectors and Cloud Optimized GeoTIFF or Zarr for rasters, so that a reviewer can inspect them without the original software. Container images, built with Docker or Apptainer, provide the strongest guarantee when a workflow must be rerun years later. These habits matter more in spatial work than elsewhere because spatial errors are hard to see. A misaligned raster or a mistransformed coordinate produces numbers that look entirely reasonable. The only protection is a workflow whose every step can be inspected and rerun, and whose choices are recorded rather than inherited from defaults. Chapter 2. Coordinates, Projections and the Geometry of Error Every spatial dataset makes a claim about where things are, and that claim is only meaningful relative to a coordinate reference system. Two numbers such as 51.5 and minus 0.12 identify a point in central London only if the reader knows that they are latitude and longitude, in that order, in degrees, on a particular model of the Earth. Change any of those assumptions and the same two numbers identify somewhere else, or nowhere. Coordinate reference systems are the least glamorous part of geospatial analysis and the most frequent source of error that nobody notices, because a mistransformed dataset still draws on a map and still produces statistics. It just produces the wrong ones. Datums, ellipsoids and the moving Earth A geographic coordinate reference system has three components: an ellipsoid, a datum, and a prime meridian. The ellipsoid is a mathematical surface that approximates the shape of the Earth, slightly flattened at the poles. The datum ties that ellipsoid to the physical Earth by fixing its position and orientation. The prime meridian, almost always Greenwich, sets where longitude zero lies. Different datums place the same ellipsoid in slightly different positions, so the same point has different coordinates under each. The differences are not trivial. In Great Britain, the historical Ordnance Survey datum OSGB36 differs from the global WGS84 by up to about 120 metres. In North America, coordinates under NAD27 can differ from modern systems by tens of metres or more. Even between modern datums the differences are measurable: NAD83 and WGS84 were nearly identical when defined in the 1980s but now differ by one to two metres across much of the continent, because the North American plate has moved and the two systems treat that movement differently. Plate motion is not a pedantic concern. The Australian plate moves north-east at about seven centimetres a year, and by the 2010s the country's official datum, GDA94, was out of step with satellite positioning by around 1.8 metres. Australia adopted GDA2020 to correct this. For most environmental and epidemiological work, errors of a metre or two are well within the uncertainty of the data. For high-resolution imagery with ten-metre pixels, or for linking survey plots to individual trees, they are not. WGS84, identified by the code EPSG 4326, is the datum used by GPS and by most global datasets. It is not a single fixed system but a series of realisations that have been updated several times to track the International Terrestrial Reference Frame. When a dataset says it is in WGS84, the realisation is usually unstated, and for most purposes that ambiguity is harmless. PROJ, in recent versions, will warn when a transformation involves a datum ensemble whose members differ by more than a threshold, and the warning should be read rather than suppressed. Projections and what they preserve A projected coordinate reference system flattens the curved surface onto a plane so that positions can be expressed in linear units, usually metres. No flat map can preserve all geometric properties of a sphere. Every projection trades off among area, shape, distance and direction. Equal-area projections preserve area: a square kilometre anywhere on the map covers a square kilometre on the ground. Albers equal-area conic and Lambert azimuthal equal-area are common choices for continental analysis; the Mollweide and equal Earth projections serve for global work. Conformal projections preserve local angles and therefore local shape, at the cost of distorting area. Mercator is the famous example, as is Transverse Mercator, on which the Universal Transverse Mercator system is based. Equidistant projections preserve distances from one or two points, or along certain lines. The practical rule is simple: choose the projection according to the calculation. If the analysis computes areas, densities or rates per square kilometre, it needs an equal-area projection. If it computes distances over a small area, a local conformal projection such as the appropriate UTM zone keeps distance errors below about one part in a thousand within the zone. If it computes distances over a large area, it should compute them geodesically on the ellipsoid rather than on any projection. Web Mercator, EPSG 3857, deserves particular warning. It is the projection of most online map tiles, and data exported from web tools often arrive in it. It is conformal and wildly non-equal-area: at 60 degrees latitude, areas are inflated by a factor of four relative to the equator, and the inflation grows without bound towards the poles. On a Web Mercator map Greenland looks roughly the size of Africa, though Africa is about fourteen times larger. Any area, density or buffer computed in Web Mercator coordinates is distorted in proportion to latitude. It is a display projection and should never be used for measurement. Geographic coordinates as a trap Much environmental data arrives in longitude and latitude, and a great deal of analysis is carried out on it without projection. That is sometimes fine and sometimes badly wrong. A degree of latitude is about 111 kilometres everywhere. A degree of longitude is about 111 kilometres at the equator and shrinks with the cosine of latitude: about 78 kilometres at 45 degrees and 56 kilometres at 60 degrees. A buffer of 0.1 degrees around a point is therefore a circle only at the equator; everywhere else it is an ellipse stretched north and south. A nearest-neighbour search on raw degrees in Scandinavia will systematically prefer neighbours to the east and west over those to the north and south. The same geometry affects rasters. A global climate grid with 30 arc-second cells, the resolution of WorldClim, has cells of about 0.86 square kilometres at the equator but only about 0.43 square kilometres at 60 degrees latitude. Any procedure that treats cells as equal, such as sampling background points uniformly by cell for a species distribution model or counting cells to estimate the area of suitable habitat, over-represents high latitudes. The correction is to weight by true cell area, which terra computes with cellSize, or to work in an equal-area grid. As noted in Chapter 1, sf now avoids many of these errors for vector operations by computing on the sphere through the s2 library whenever data are in geographic coordinates. Distances, areas and buffers are then computed correctly in metres. geopandas does not do this: its geometric operations are planar. Python users must either project the data before computing areas and distances or use pyproj's Geod class, which computes geodesic distances and areas on the ellipsoid using the algorithms Charles Karney published in 2013, accurate to within a few nanometres. The axis-order problem A particularly irritating source of error concerns the order of coordinates. Mathematicians and most GIS software write coordinates as x then y, which for geographic data means longitude then latitude. The EPSG registry, however, defines EPSG 4326 with latitude first, following geodetic convention. For years most software ignored the official order. Since GDAL 3 and PROJ 6, the libraries respect the authority's axis order by default, while offering an option to override it. The consequence is that the same file can be read with swapped coordinates depending on software and settings. sf uses traditional GIS order, longitude first. pyproj requires an always_xy argument set to true to guarantee longitude-first order when creating a transformer; without it, a transformation from EPSG 4326 expects latitude first. Web services following the OGC's WMS 1.3 and WFS 2.0 standards also honour authority order. A dataset with swapped axes places points at the mirror position across the line where latitude equals longitude, which for data in Africa or Europe often lands in the ocean and is easy to spot, but for data near that line can produce plausible-looking errors. The defence is a set of simple checks run on every point dataset as it enters an analysis. Plot the points over a base map. Confirm that longitudes fall between minus 180 and 180 and latitudes between minus 90 and 90. Check that points fall on land if they should, within the expected country if they should, and not at exactly zero, zero, a location in the Gulf of Guinea that is the default value in many databases and has become known as Null Island. Occurrence data for species distribution models, discussed in Chapter 8, suffer from all of these problems, and the CoordinateCleaner package in R was written specifically to detect them. Reprojecting vectors and rasters Reprojecting vector data is conceptually clean. Each vertex is transformed independently, and the geometry is reconstructed from the transformed vertices. The only subtleties are that straight lines between vertices in one projection are not straight in another, so long edges may need densifying before transformation, and that geometries crossing the antimeridian at 180 degrees longitude can wrap incorrectly. Reprojecting rasters is not clean. The grid of the output raster does not align with the grid of the input, so every output cell must be assigned a value computed from input cells that only partly overlap it. This is resampling, and the choice of method changes the data. Nearest-neighbour resampling assigns each output cell the value of the input cell closest to its centre. It preserves original values exactly and is the only correct choice for categorical data such as land cover classes, where averaging class codes would produce nonsense. It produces blocky artefacts and can duplicate or drop cells. Bilinear interpolation takes a distance-weighted average of the four nearest input cells and cubic convolution uses sixteen; both produce smoother surfaces suited to continuous variables such as temperature or elevation, but both invent values that were never measured and smooth away local extremes. Average, sum, mode, minimum and maximum methods aggregate all input cells that fall within each output cell, and are appropriate when moving from fine to coarse resolution. Population counts must be resampled with a sum, or with an area-weighted redistribution, never with bilinear interpolation, which does not preserve totals. Each reprojection degrades a raster slightly. Two reprojections degrade it more. The rule that follows is to reproject rasters as few times as possible, ideally once, and to prefer transforming vector data into the raster's coordinate system rather than the reverse. When a study requires many raster layers on a common grid, the analyst should define that grid explicitly, with its projection, origin, cell size and extent, and resample every layer onto it once, choosing the method layer by layer according to what the values represent. In terra this is done with project using a template raster; in Python, rioxarray's reproject_match performs the same operation. Heights and discrete global grids Two further reference questions arise often enough in environmental work to need a word. The first concerns height. Elevations are measured relative to a vertical datum, and there are two broad kinds. Ellipsoidal heights, which GPS receivers report directly, are measured from the mathematical ellipsoid. Orthometric heights, which is what most people mean by elevation above sea level, are measured from the geoid, an irregular surface that approximates mean sea level and departs from the ellipsoid by as much as about a hundred metres in either direction. The Shuttle Radar Topography Mission elevation model, for instance, reports heights relative to the EGM96 geoid, while the Copernicus global digital elevation model uses the later EGM2008. Combining GPS heights from field plots with an elevation model without converting between the two can produce elevation errors of tens of metres. For studies where altitude is itself an exposure, as in the epidemiology of malaria transmission in East African highlands, where small differences in elevation shift temperature enough to change mosquito development rates, such errors matter. The second concerns alternatives to conventional projections for global or multi-scale aggregation. Discrete global grid systems partition the Earth's surface into cells of roughly equal area at several nested resolutions. The most widely used is H3, a hexagonal hierarchical index released as open source by Uber in 2018, with bindings in both R and Python. Hexagons have the useful property that every neighbour is at the same distance from the cell centre, unlike squares, whose diagonal neighbours are farther than edge neighbours. That makes hexagonal grids attractive for aggregating point data such as case locations or species occurrences to a uniform lattice before computing autocorrelation or density. H3 cells are not exactly equal in area, and their boundaries do not nest perfectly across resolutions, but for many purposes they avoid both the latitude distortion of geographic grids and the need to choose a projection. The price is that most raster data do not come in hexagons, so the aggregation step must be done carefully and with the same attention to what each variable represents that applies to any resampling. Choosing a working system A defensible choice of coordinate reference system for a study usually follows from three questions. What calculations will be performed? Areas and densities need equal area. Local distances need a conformal projection or geodesic computation. Neighbour relations for autocorrelation need a system in which distance means the same thing in every direction. How large is the study area? A single city or district fits comfortably in a UTM zone or a national grid. A country spanning several UTM zones, or a continent, needs a conic or azimuthal projection centred on the region. A global analysis should either use an equal-area global projection or work on the ellipsoid directly. What are the native systems of the key rasters? If the most important raster, such as a satellite scene, is in a particular UTM zone, it is often better to bring everything else into that zone than to reproject the scene. For health data, a fourth consideration applies. Many national statistical agencies publish boundaries in a national system, and many health information systems store facility locations in WGS84. Merging them without explicit transformation is one of the most common errors in applied work. The fix is to declare the coordinate reference system of every dataset as it is read, transform to the working system immediately, and check the result visually. Positional accuracy of health data Epidemiological data carry positional error beyond anything introduced by projection. Case addresses are converted to coordinates by geocoding, which matches street addresses against a reference database. Match rates vary widely, and addresses that fail to match are not missing at random: rural addresses, informal settlements, new developments and addresses written in non-standard formats fail more often, so geocoded datasets systematically under-represent certain populations. Studies in the geocoding literature have repeatedly found that positional errors for matched addresses are larger in rural areas, where an address may be interpolated along a long road segment, than in dense urban grids. Privacy adds a further, deliberate layer of error. The Demographic and Health Surveys programme, which runs household surveys in many low- and middle-income countries, publishes cluster locations with random displacement: up to two kilometres for urban clusters and up to five kilometres for rural clusters, with one percent of rural clusters displaced by up to ten kilometres. The displacement is constrained to stay within the correct administrative area. Analysts who link DHS clusters to environmental rasters, such as distance to a health facility or mean vegetation index, must account for it, typically by extracting values over a buffer that matches the displacement radius rather than at the published point. Geomasking of this kind protects confidentiality, but it also biases any analysis that relates outcomes to fine-scale exposures, generally toward the null, because the exposure assigned to each cluster is a noisy version of the true one. The lesson generalises. The coordinates in a health dataset are an estimate, with an error structure that depends on how they were produced. That error structure belongs in the analysis plan, not in a limitations paragraph added afterwards. Chapter 3. Vector Operations and the Construction of Exposure Most spatial analyses in environmental health begin with a question of linkage. Which people live near the smelter? Which villages lie within the flood extent? What proportion of each district's population is within an hour's travel of a clinic? Which survey households drink from surface water in an area where the river is contaminated? None of these questions is answered by a statistical model. Each is answered by a sequence of geometric operations that construct a dataset, and the statistical model then takes that dataset as given. If the construction is wrong, the model faithfully analyses the wrong thing. This chapter concerns those constructions: spatial joins, buffers, overlays, distance calculations and aggregations. They are simple to perform and deceptively hard to perform well. The same few pitfalls recur across studies, and each one is a choice the software makes unless the analyst makes it first. Spatial joins and their predicates A spatial join attaches attributes from one layer to another based on a geometric relationship rather than a shared key. Joining case locations to district polygons, to find which district each case falls in, is the canonical example. In sf it is st_join; in geopandas it is sjoin. Both default to the intersects predicate: a case is joined to every district its geometry intersects. For points strictly inside polygons, intersects behaves as expected. Points exactly on a shared boundary intersect both polygons and are joined twice, silently duplicating the case. With coordinates rounded to four or five decimal places, as health records often are, points on boundaries are more common than one might expect. The fix depends on intent. Using the within predicate excludes boundary points entirely; assigning each point to a single polygon, by keeping the first match or the largest overlap, keeps them once. In sf, st_join has a largest argument that assigns each feature to the polygon it overlaps most. Whichever rule is chosen, the analyst should count rows before and after every join, because a join that changes the number of rows has almost always done something unintended. Joins between polygon layers are subtler still. Joining health facility catchments to administrative districts will match every catchment to every district it touches, including districts it overlaps by a few square metres because of boundary imprecision. The largest-overlap rule, or an explicit intersection followed by area weighting, is almost always what is wanted. A further hazard is that spatial joins are left joins by default in sf and inner joins by default in geopandas. A case that falls outside every district, perhaps because it was geocoded to the wrong side of a coastline, is retained with missing district attributes in one and dropped silently in the other. The unmatched records are exactly the ones worth inspecting. Buffers and proximity as exposure Proximity is the most common proxy for environmental exposure. Residents within 500 metres of a major road are classed as exposed to traffic pollution; households within five kilometres of a mine are classed as exposed to its emissions; children within a certain distance of a malaria vector breeding site are classed as at elevated risk. Buffers are easy to compute and hard to justify. The first question is the metric. A buffer drawn in geographic coordinates without spherical computation is not circular, as Chapter 2 explained. A buffer drawn in a projected system is circular in that projection, which is close enough over short distances in an appropriate projection and wrong in Web Mercator. The second question is the cut-off. A binary buffer asserts that exposure is uniform inside and absent outside, which is rarely true. Traffic-related air pollutants such as ultrafine particles and nitrogen dioxide decline steeply over the first few hundred metres from a busy road, then more slowly. A study that compares residents within and beyond a fixed buffer will find different effects depending on where the cut-off falls, and the temptation to try several cut-offs and report the one that gives the clearest result is a well-known route to false findings. Where the decay of exposure with distance is known from measurement, it is better to model it directly, with a continuous distance term or a dispersion model. Where it is not known, the cut-off should be fixed in advance on physical grounds and sensitivity to it reported. The third question is whether straight-line distance is the right measure at all. Snow understood in 1854 that walking distance through streets, not straight-line distance, determined which pump a household used. Modern equivalents are everywhere. Access to health care depends on travel time along roads and paths, not on distance as the crow flies. A village separated from a clinic by a river without a bridge is far from it regardless of the straight-line distance. Network distance along a road graph can be computed with sfnetworks or dodgr in R and with osmnx and networkx in Python, using road data from OpenStreetMap. Where roads are sparse, as across much of rural Africa and Asia, a friction surface approach is more realistic. Each raster cell is assigned a travel speed according to its land cover, road class and terrain, and a least-cost path algorithm computes the minimum travel time from every cell to the nearest destination. The Malaria Atlas Project published a global friction surface of this kind, with a map of travel time to cities for 2015 by Daniel Weiss and colleagues in Nature in 2018 and a follow-up mapping travel time to health care facilities in Nature Medicine in 2020. Both surfaces are freely available and have become standard covariates for access-related outcomes. The gdistance package in R and the scikit-image graph routines in Python provide the cost-distance computations needed to build such surfaces locally. Overlay and the transfer of attributes between units Environmental and health data rarely share geographic units. Cases are reported by health district, population by census tract, pollution by monitoring station or model grid, land cover by pixel. Bringing them together requires moving attributes from one set of units to another, and the method of transfer matters. The simplest approach is areal weighting. The source polygons are intersected with the target polygons, and each piece receives a share of the source attribute in proportion to its area. If a census tract with 4,000 residents is split evenly between two health districts, each district receives 2,000. This assumes that the attribute is spread uniformly over the source polygon, which for population is almost never true. A rural tract whose residents are concentrated in one village at its northern edge will have its population misallocated in proportion to area. Dasymetric mapping improves on areal weighting by redistributing counts according to ancillary information about where they are likely to be. For population, the ancillary data are usually land cover, building footprints or nighttime lights: residents are allocated only to cells classed as built-up, or in proportion to building volume. Gridded population products such as WorldPop, developed at the University of Southampton, and the Global Human Settlement Layer produced by the European Commission's Joint Research Centre, are themselves the output of dasymetric models, redistributing census counts to cells of about 100 metres using satellite-derived covariates. For most health applications in low- and middle-income countries, these products provide better denominators than areal weighting of census polygons, with the caveat that their accuracy is lowest precisely where census data are oldest and settlement patterns are changing fastest. Intensive variables, those expressed per unit such as rates, densities and concentrations, must be transferred differently from extensive variables such as counts. A rate cannot be split by area; it must be averaged, and the average should be weighted by population rather than by area if the rate refers to people. Many analyses go wrong by applying areal weighting to a rate or area-weighted averaging to a count. The modifiable areal unit problem Once data are aggregated to areas, results depend on the areas chosen. This is the modifiable areal unit problem, a name given to it by the geographer Stan Openshaw in a 1984 monograph, though the phenomenon had been observed decades earlier. It has two parts. The scale effect is that statistics change as units are aggregated into larger units. Correlations between variables tend to strengthen with aggregation, because aggregation averages out individual-level noise. A modest association between deprivation and hospital admission at the level of small census areas can become a strong one at the level of large districts. The zoning effect is that statistics change when the same area is divided into the same number of units along different boundaries. Openshaw demonstrated that by redrawing the boundaries of a fixed number of zones, correlations between two census variables could be driven across a very wide range. Electoral gerrymandering exploits the same property. The modifiable areal unit problem is closely related to the ecological fallacy, which is the error of inferring individual-level relationships from group-level data. The classic demonstration is W. S. Robinson's 1950 paper in the American Sociological Review, which showed that the correlation between being foreign-born and being literate in the United States had opposite signs at the level of states and at the level of individuals: states with more immigrants had higher literacy, though immigrants themselves were less often literate, because immigrants had settled in states with high literacy. There is no general solution. The honest responses are to analyse data at the finest resolution available, to state the unit of analysis as part of the finding rather than as an incidental detail, to repeat key analyses at more than one scale, and to avoid individual-level interpretation of area-level associations. In disease mapping, the hierarchical models of Chapter 7 reduce the instability that small units produce without abandoning them, which is one reason they have displaced crude rate maps. From points to surfaces Some analyses keep cases as points rather than aggregating them. Kernel density estimation converts a set of points into a smooth surface by placing a small bump, the kernel, over each point and summing. The result depends much more on the bandwidth, the width of the bump, than on the shape of the kernel. Too narrow a bandwidth yields a surface with a spike at every case; too wide a bandwidth smears everything into a single hill. Automatic bandwidth selectors exist, and they disagree with one another. A kernel density surface of cases is almost never what an epidemiologist wants on its own, because it mostly reflects where people live. The informative quantity is relative risk: the density of cases divided by the density of the population at risk. When individual controls are available, as in a case-control study, the ratio of a case density to a control density gives a spatially varying relative risk surface. Julia Kelsall and Peter Diggle set out this approach in 1995, and the sparr package in R implements it with adaptive bandwidths and tolerance contours that mark regions where the risk is significantly elevated. The point-pattern package spatstat, developed over many years by Adrian Baddeley, Rolf Turner and Ege Rubak, provides the broader toolkit for point processes, including Ripley's K function for testing whether points cluster more than expected under a given background intensity. Aggregation with denominators and uncertainty Almost every health rate has a numerator and a denominator from different sources, often at different spatial resolutions and different dates. Cases come from surveillance for the current year; the population denominator comes from a census five or ten years old, projected forward. If the projection is wrong for a particular district, perhaps because of migration after a conflict or displacement after a flood, that district's rate is wrong, and in a small district the error can be larger than any true variation in risk. Denominator uncertainty is rarely propagated into analysis, and it should be at least acknowledged. WorldPop and some other gridded population products now publish uncertainty surfaces alongside their estimates. Where denominators are known to be poor, a sensitivity analysis using alternative population sources, such as comparing census projections against a gridded product, reveals which findings depend on the choice. A worked construction: population within reach of care A common planning question shows how these operations fit together and where the choices lie. Suppose a ministry of health wants to know, for each district, what share of the population lives within one hour's travel of a facility offering emergency obstetric care. The inputs are a table of facilities with coordinates, a district boundary file, a gridded population surface, and a friction surface giving travel time per metre across each cell. The first step is cleaning the facility list: checking that coordinates fall inside the country, that none sit at zero latitude and zero longitude, and that facilities flagged as offering the service actually do. Facility registries are notoriously out of date, and in practice this step often takes longer than all the others combined. The second step is computing travel time. The friction surface is converted to a graph in which each cell connects to its eight neighbours, with edge costs equal to the time to cross between cell centres. A least-cost algorithm propagates outward from every facility at once and records, for each cell, the minimum time to reach any facility. In R, gdistance builds the transition matrix and computes accumulated cost; in Python, the minimum-cost-path routines in scikit-image do the same on a NumPy array. The result is a raster of travel time on the friction surface's grid. The third step is aligning the population raster to that grid. If the population grid is finer, it must be aggregated with a sum. If it is coarser, it must be disaggregated with care, since dividing a cell's population evenly among its sub-cells assumes uniformity. If the grids are offset, one must be resampled, again with a method that preserves totals. The fourth step is a zonal computation: for each district, sum the population in cells with travel time under sixty minutes and divide by total population in the district. The next chapter discusses how cells that straddle district boundaries should be treated, which for small districts can change the answer by several percentage points. Each step embeds a choice that could be defended differently. Walking speeds in the friction surface assume a mode of travel; a woman in labour may be travelling by motorcycle, by donkey cart or on foot. The facility list assumes that listed facilities are functioning. The population grid assumes that its dasymetric model placed people correctly. The one-hour threshold is conventional, not physiological. A well-reported analysis states each assumption, and a careful one shows how the district rankings change when the most uncertain of them are varied. In published work of this kind, the ranking of districts is usually more robust than the absolute percentages, and that is worth saying explicitly to the people who will use it. Privacy, confidentiality and the construction of data Linking health records to places raises obvious concerns about confidentiality. A map of individual case locations, even without names, can identify people in sparsely populated areas. Standard responses include aggregation to areas large enough that each contains a minimum number of people, random displacement as in the Demographic and Health Surveys, and more elaborate geomasking methods such as donut masking, which displaces each point by at least a minimum and at most a maximum distance so that the original location cannot be recovered by averaging. Every such method trades confidentiality for analytical precision. From the perspective of this booklet's argument, the point is that masked coordinates are a data construction with known properties. An analyst who knows the masking rule can model it; one who treats masked coordinates as exact will mis-estimate exposure. The same holds for every other construction in this chapter. Joins, buffers, overlays and aggregations each embed a rule, and the rule should be chosen, recorded and, where it matters, varied to show that the conclusion survives. Hashtags: #GeospatialAnalysisInRAndPython #GeospatialAnalysis #SpatialDataScience #RemoteSensing #GeographicInformationSystems #GIS #SpatialEpidemiology #SpatialStatistics #CoordinateReferenceSystems #MapProjections #VectorData #RasterData #SpatialJoins #RasterAnalysis #SpatialAutocorrelation #MoransI #SpatialRegression #BayesianSpatialModels #SatelliteImagery #LandSurfaceAnalysis #EcologicalNicheModeling #SpeciesDistributionModels #Maxent #GeospatialReproducibility #FutureOfGeospatialScience

Download the Book (PDF): Introduction Every year, thousands of published documents announce that a group of experts has agreed. A panel of clinicians agrees on which outcomes every trial in a disease should measure. A group of public health specialists agrees on what governments should do next in a pandemic. A committee of speech therapists, psychologists and paediatricians agrees on what to call children whose language does not develop as expected. Very often, the phrase that carries the weight is short: "consensus was reached using a modified Delphi process." Readers are invited to treat that phrase as a guarantee. They usually should not, at least not until they know what lies behind it. The Delphi method is a structured way of asking a panel of knowledgeable people the same questions several times, in writing, without letting them see who said what, and showing them between rounds how the group as a whole answered. It was devised at the RAND Corporation in the early 1950s to extract usable judgements from military and technical specialists about questions that no data could settle, and it has since spread into medicine, nursing, public health, education, environmental management, technology foresight and public policy. It is cheap, it can be run by email or on a web platform, it can bring together people on different continents who would never sit in one room, and it produces an output that looks like evidence: percentages, medians, lists of statements that crossed a threshold. That last property is both the method's great strength and its standing temptation. A Delphi study converts opinion into numbers. Those numbers can be honest summaries of a considered, independent judgement by the right people, or they can be the predictable result of a narrow panel, leading questions, feedback that nudged the waverers, a threshold chosen after the data came in, and the quiet departure of the people who disagreed. From the outside, the two can look identical. The published paper reports that 87 per cent of respondents agreed, and nothing on the page tells the reader which kind of 87 per cent it was. The argument of this book This book makes one argument and follows it through the life of a Delphi study. The consensus a Delphi study reports is not discovered; it is produced, by a chain of design decisions each of which could have been made differently. Who counts as an expert, how the questions are worded, which scale is used, what feedback panellists see, how many rounds are run, where the threshold sits, and how dropouts are handled all shape the final agreement at least as much as the underlying state of knowledge does. A Delphi result is therefore credible not because agreement was reached, but because the choices that produced the agreement were made in advance, for stated reasons, and disclosed in enough detail that a reader can tell agreement apart from conformity, attrition or design. This is not a counsel of despair. It is the reason the method can be done well. If consensus were simply out there waiting to be found, there would be little a researcher could do to improve a Delphi study beyond finding more experts. Because it is produced, every stage offers a lever, and the difference between a careful study and a careless one lies in how those levers are pulled. The chapters that follow are organised around those stages, and each asks the same two questions: what choices exist here, and how does each choice change the agreement that will eventually be reported? Why this matters now Delphi studies have never been more common or more consequential. Clinical guideline groups use them to fill the gaps where trials are absent. Core outcome sets, which determine what hundreds of future trials will measure, are routinely built on them. Reporting guidelines for research are developed through them, which gives the method a curious recursive authority: the rules that govern how evidence is reported are often themselves the product of expert consensus. During the COVID-19 pandemic, a Delphi study involving 386 experts from 112 countries and territories, published in Nature in November 2022, set out 41 consensus statements and 57 recommendations for ending the pandemic as a public health threat. Governments, professional bodies and funders now read Delphi outputs as a legitimate basis for action. At the same time, methodologists have spent two decades documenting how inconsistently the method is used. A systematic review by Ian Diamond and colleagues, published in the Journal of Clinical Epidemiology in 2014, examined 100 Delphi studies and found that only 72 of the 98 aiming at consensus defined what consensus meant; that the median percentage threshold, where one was given, was 75 per cent but ranged from 50 to 97; and that in 70 studies the process simply stopped after a fixed number of rounds rather than when consensus was reached. Reviews in health care quality indicators and in core outcome set development found similar variability. The response has been a wave of reporting guidance, including CREDES for palliative care in 2017, and, in 2024, both the ACCORD guideline for consensus methods in biomedicine and DELPHISTAR for Delphi studies across the social and health sciences. Reporting guidance helps readers see what was done. It does not tell researchers what to do, and it cannot by itself stop a well-reported study from being badly designed. That gap is where this book sits. What the book covers and what it leaves aside The first chapter sets out what the Delphi method is for, where it came from, and the family of variants that now shelter under the name, from the classical forecasting Delphi through the RAND/UCLA Appropriateness Method to the policy Delphi and the real-time online versions. It also argues, against a common habit, that Delphi is a method of last resort for questions that evidence cannot yet answer, not a shortcut around evidence that exists. The second chapter treats the protocol: the steering group, the evidence review that should precede the first questionnaire, and the decisions that must be written down before any panellist is contacted. The third addresses the panel, which is the single largest determinant of what a Delphi study will conclude: who qualifies as an expert, how different stakeholder groups should be represented, how large the panel needs to be, and how to recruit people who will stay. Chapters four and five follow the rounds. The fourth covers the first questionnaire, whether open or seeded, and the craft of writing items and choosing rating scales. The fifth covers what happens between rounds, where the method's distinctive mechanism, controlled feedback, does its work, and where it can do damage. Chapter six turns to the statistics of consensus: percentage agreement, medians and interquartile ranges, the RAND/UCLA definitions of disagreement, coefficients of variation, Kendall's W, and tests of stability between rounds. The chapter argues that no measure is correct in the abstract, and that the defensible choice is the one that matches the question and is fixed in advance. Chapter seven confronts the method's oldest criticism: that it manufactures agreement through social pressure, even without face-to-face contact. It examines groupthink, conformity, anchoring on feedback and selective attrition, and sets out the design choices that reduce each. Chapter eight takes the method into policy, where the goal is often not agreement at all but a clear map of disagreement, and where the legitimacy of the output depends on who was asked as much as on what they said. It ends with the reporting standards that allow outsiders to judge a finished study. The conclusion draws the argument together into a set of commitments for anyone who designs, commissions, or relies on a Delphi study. Some things are deliberately left out. The book does not survey every application domain, and it does not teach the statistics of forecasting accuracy in depth. It treats the nominal group technique and consensus conferences only where they illuminate a Delphi choice. It gives no software instructions, because platforms change faster than principles. Its examples are drawn mainly from health and public policy, because that is where the method is now most heavily used and most consequential, but the reasoning applies wherever a group of people with relevant knowledge must be asked to judge something the evidence cannot yet settle. A note on the reader The book is written for three kinds of reader. The first is the researcher about to run a Delphi study, often for the first time, often as part of a doctorate or a guideline project, and often with a steering group that has more enthusiasm than experience. The second is the commissioner: the guideline body, funder, professional society or ministry that will pay for a study and then act on its output. The third is the reader of Delphi results, including journal editors, reviewers, clinicians and officials, who needs to know which questions to ask of a paper before taking its conclusions seriously. All three share a need that the phrase "consensus was reached" does not meet. They need to know what kind of agreement they are looking at, how it was made, and how much weight it can bear. That is what the following chapters try to supply. Chapter 1. What Delphi Is For The Delphi method was born of a practical embarrassment. In the early years of the Cold War, the United States Air Force needed estimates about matters for which no data existed and none could be gathered: how an adversary would target American industry, how many weapons would be needed to achieve a given effect, when particular technologies would become feasible. The obvious source of such estimates was expert opinion. The obvious way to gather it was to put the experts in a room. And the obvious problem, familiar to anyone who has sat on a committee, was that the room distorted what the experts knew. The most senior or most voluble person set the terms of discussion. People defended positions they had stated publicly rather than revise them. Dissent was expensive. Agreement, when it came, often reflected the social dynamics of the meeting more than the balance of knowledge within it. Researchers at the RAND Corporation, most prominently Norman Dalkey and Olaf Helmer, set out to keep what was valuable about pooled judgement while removing what was corrosive about face-to-face deliberation. Their experiment, conducted for the Air Force in the early 1950s and published in the journal Management Science in 1963 under the title "An Experimental Application of the Delphi Method to the Use of Experts," asked a small panel to estimate, from the viewpoint of a Soviet strategic planner, the number of atomic bombs needed to reduce American munitions output by a specified amount. The experts never met. They answered questionnaires, received controlled information about the group's answers and about relevant factors raised by others, and answered again. Their estimates, initially spread across a very wide range, converged over the rounds. The name, a nod to the oracle at Delphi, was meant with some irony; the method's authors knew that nobody was consulting a god. Within a decade the technique had been turned to long-range forecasting of science and technology, most famously in the study reported by Theodore Gordon and Olaf Helmer in 1964, and had begun to spread into corporate planning, education and health. By 1975, when Harold Linstone and Murray Turoff edited the collection that became the field's standard reference, The Delphi Method: Techniques and Applications, the method had already diversified into forms its originators had not anticipated. Linstone and Turoff offered a definition broad enough to cover them all: Delphi is "a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem." The four defining features Beneath the variety, four features define the method, and each exists to solve a specific problem of face-to-face groups. The first is anonymity. Panellists do not know, at least during the rating process, who gave which answer. The point is not secrecy for its own sake but the removal of status cues. A junior clinician can disagree with a professor without the disagreement being noticed as such; a professor can change her mind without losing face. Anonymity in this sense is often called quasi-anonymity, because panellists may know who else is on the panel, and the organisers always know who said what. What matters is that responses are not attributed to individuals when fed back to the group. The second is iteration. The same questions, or refined versions of them, are put to the panel more than once. Iteration gives people the chance to reconsider in light of what others think and why, which is the mechanism by which Delphi hopes to improve on a single survey. A one-round expert survey is not a Delphi study, however carefully designed, because nobody has had a chance to respond to anyone else. The third is controlled feedback. Between rounds, the organisers tell panellists something about how the group answered: typically a measure of central tendency such as the median, some measure of spread, often the panellist's own previous answer for comparison, and sometimes the reasons others gave. Feedback is "controlled" because the organisers choose what to show, and those choices, as later chapters will argue, are among the most consequential in the entire design. The fourth is statistical aggregation of the group response. The output is not a negotiated form of words agreed by everyone in a room but a summary of individual ratings. Every panellist's judgement enters the final result, and the result can be expressed with a measure of how much the panel agreed. This is what allows a Delphi study to report that 82 per cent agreed with one statement and 54 per cent with another, and it is what gives Delphi outputs their appearance of quantitative rigour. Each of these features is a partial remedy, and each introduces its own difficulties. Anonymity removes status effects but also removes accountability; a panellist who need never defend a rating may give it less thought. Iteration allows reconsideration but also creates opportunities for fatigue and dropout. Controlled feedback provides information but also creates a reference point that can pull ratings towards it regardless of their merits. Statistical aggregation includes everyone but can conceal the difference between a panel that genuinely agrees and one whose members have simply stopped resisting. The rest of this book is, in effect, an extended account of how to use these features while guarding against their side effects. The family of variants Few Delphi studies today follow the procedure Dalkey and Helmer described. The name now covers a family of related designs that differ in their purpose, their first round, their use of meetings, and the kind of output they seek. The main members of the family differ on these points in ways worth setting side by side, as Table 1 does. Table 1. Main variants of the Delphi method and how they differ. Variant Primary purpose First round Face-to-face element Typical output Classical Delphi Forecasting or estimation Open questions None Converged estimates with spread Modified Delphi Agreement on statements or items Items drawn from evidence review Sometimes a final meeting List of items meeting a threshold RAND/UCLA Appropriateness Method Rating appropriateness of clinical indications Evidence summary plus rating Second-round panel meeting Indications classed as appropriate, uncertain or inappropriate Policy Delphi Exposing options and disagreement Open or seeded None or optional Map of positions and arguments Real-time or online Delphi Faster iteration at scale Seeded None Continuously updated group ratings The classical Delphi begins with open questions, uses the answers to build a structured questionnaire, and iterates until estimates stabilise. It remains common in technology foresight. The modified Delphi, now the dominant form in health research, replaces the open first round with a list of items derived from a systematic review, a qualitative study, or existing guidelines. The word "modified" is used so loosely that it tells a reader very little; it can mean a seeded first round, a final consensus meeting, a change of scale, or all three. A study that describes itself as a modified Delphi should always say what was modified and why. The RAND/UCLA Appropriateness Method, developed in the 1980s to study the overuse and underuse of medical procedures, is a particular and well-specified hybrid. Panels of seven to fifteen clinicians, with nine the traditional number, receive a review of the literature and rate hundreds of clinical scenarios on a nine-point scale of appropriateness. They then meet, discuss the areas of disagreement under a trained moderator, and rate again privately. Each scenario is classed as appropriate if the panel median falls between 7 and 9, uncertain between 4 and 6, and inappropriate between 1 and 3, with any scenario rated "with disagreement" treated as uncertain regardless of its median. The method's user's manual, published by RAND in 2001 under the lead authorship of Kathryn Fitch, remains one of the most precise descriptions of any consensus procedure. The policy Delphi, proposed by Murray Turoff in 1970, inverts the usual goal. Its purpose is not to reach agreement but to generate the strongest possible range of options and arguments on a contested policy question, and to identify where and why informed people disagree. Chapter eight returns to it in detail. Real-time Delphi, described by Theodore Gordon and Adam Pease in 2006, abandons discrete rounds. Panellists log in to a platform, see the current distribution of responses and the reasons given, enter or revise their own ratings, and can return as often as they like. The design trades the clean structure of rounds for speed and for a continuous exchange of reasons. The e-Delphi, now nearly universal, is simply a Delphi conducted through email or web survey software. It changes the logistics rather than the logic, although online administration makes very large panels feasible, and very large panels behave differently from small ones. When Delphi is the right tool The method's popularity has outrun its justification. Delphi exists to extract considered judgement where better sources of knowledge are unavailable. It is appropriate when three conditions hold together. The first condition is that the question cannot be answered adequately by existing evidence. If randomised trials have established that a treatment works, a panel of experts agreeing that it works adds nothing, and a panel disagreeing that it works should not override the trials. Delphi belongs in the gaps: questions of definition and terminology, choices about what to measure, judgements about appropriateness in populations the trials excluded, forecasts of future states, priorities among many plausible actions. The CATALISE study, which asked 59 experts from ten disciplines across six English-speaking countries to agree on how to identify children with language impairments, is a good example. The disagreement it addressed was not about data that could be collected; it was about categories, thresholds and words, which only the professional community could settle. The second condition is that relevant knowledge is distributed among people who cannot easily be brought together, or whose interaction would be distorted if they were. International panels, panels that mix patients with professionals, and panels on politically sensitive topics all benefit from written, anonymous exchange. The third condition is that the answer will be used for something that needs a defensible, transparent basis. A Delphi study is laborious. If a steering group could simply decide, and nobody would reasonably question its authority, a Delphi study may be ceremony rather than method. Delphi is the wrong tool when the question is empirical and answerable, when the aim is to legitimise a conclusion already reached, or when the relevant expertise is so narrow that the panel would consist of the steering group's collaborators. It is also the wrong tool when what is needed is deliberation in the full sense, a process in which people change their understanding through sustained argument. Written rounds allow the exchange of brief reasons; they are poor at the kind of back-and-forth that reveals a hidden assumption or produces a new synthesis. For some problems, a well-run meeting, or a Delphi combined with one, does better. Neighbouring methods Delphi is one of several formal consensus methods, and knowing its neighbours clarifies what it does and does not offer. The nominal group technique, described by André Delbecq and Andrew Van de Ven in the early 1970s and set out at length in their 1975 book with David Gustafson, Group Techniques for Program Planning, brings a small group of perhaps eight to twelve people into a room but constrains how they interact. Participants first write down ideas silently, then share them one at a time in round-robin fashion without debate, then discuss each idea for clarification, and finally vote or rank privately. The technique keeps face-to-face contact while blunting its worst effects, and it is fast: a single session can generate and prioritise a list. What it cannot do is include large or dispersed panels, and it cannot fully remove the influence of a forceful participant during the discussion phase. The consensus development conference, used for many years by the United States National Institutes of Health and adopted in various forms elsewhere, is a public event at which experts present evidence to a lay or mixed panel, which then deliberates in private and issues a statement. It resembles a jury more than a survey. Its strength is the depth of the evidence presentation and the visible, accountable nature of the panel's deliberation; its weakness is that it relies on a small number of panellists and on the dynamics of one closed meeting. Delphi differs from both in giving up deliberation almost entirely in exchange for scale, anonymity and a quantified output. That trade is often worth making, but it is a trade. A Delphi study cannot discover that its panel was using a key term in two different senses unless the free-text comments happen to reveal it. It cannot follow a promising line of argument in real time. Several modern designs therefore combine methods: a Delphi to establish where agreement and disagreement lie across a large panel, followed by a smaller meeting, sometimes run with nominal group rules, to resolve the items left in between. The RAND/UCLA method builds that combination into its core. When a study reports such a hybrid, the reader should ask which stage decided what, because a meeting at the end can overturn or quietly reframe what the Delphi rounds found. The evidence on whether it works It is reasonable to ask whether Delphi actually improves judgement. The best evidence comes from forecasting and estimation, where answers can later be checked. Gene Rowe and George Wright reviewed the experimental literature in the International Journal of Forecasting in 1999 and again in a 2001 chapter in Principles of Forecasting. Comparing Delphi with "staticised groups," in which individual judgements are simply averaged without feedback, they counted twelve studies favouring Delphi, two favouring staticised groups, and two ties. Comparing Delphi with unstructured interacting groups, they counted five studies favouring Delphi, one favouring the interacting group, and two ties. Their conclusion was cautiously positive: iteration with feedback tends to improve accuracy over a single round, and the structured process tends to beat free discussion. Two caveats matter. Many of those experiments used students answering almanac-style questions, which is some distance from senior clinicians judging the appropriateness of surgery. And accuracy is a meaningful criterion only when there is a true answer to be accurate about. Most health and policy Delphi studies ask questions about value, priority or definition, for which there is no later moment of truth. What the studies can be judged on is whether the process was fair, whether the panel was appropriate, and whether the reported consensus reflects considered judgement rather than artefact. These are the standards the rest of this book applies. Consensus as a product A final observation frames what follows. In the original RAND experiments, convergence of estimates was treated as a sign that the panel was approaching a better answer. That interpretation made some sense for estimation questions, where independent errors can cancel out and exchange of information can correct mistakes. It makes less sense for the kinds of questions modern Delphi studies ask. When a panel converges on the view that a particular outcome should be measured in every trial, the convergence may reflect shared recognition of the outcome's importance, or it may reflect the fact that the item was well worded, appeared early in the questionnaire, was supported by a high group median in the feedback, and was opposed mainly by panellists who dropped out after round one. The method does not itself distinguish these possibilities. Its designers must. That means treating every design decision as a decision about what kind of consensus the study is capable of producing, and making those decisions before the data can influence them. The next chapter begins at the point where that discipline starts: the protocol. Chapter 2. Deciding Before Asking: The Protocol A Delphi study is decided long before the first questionnaire goes out. By the time panellists are rating items, most of the determinants of the final result have already been fixed: the question being asked, the items on offer, the people doing the rating, the scale they will use, the feedback they will receive and the rule that will separate consensus from its absence. The protocol is where those determinants are chosen. It is also where they can be protected from the most insidious threat to a Delphi study's credibility, which is not bad faith but hindsight: the natural tendency, once results start to arrive, to adjust the rules until the results look sensible. This chapter treats the protocol as the study's first and most important act. It covers the steering group that designs the study, the definition of the question, the evidence review that should precede any rating, and the specific decisions that belong in writing before anyone is invited. The steering group and its power Every Delphi study has an organising group, whether it is called a steering committee, a management group, a working group or simply the research team. The label matters less than the recognition that this group holds more power over the outcome than any panellist. It decides what is asked and how. It interprets free-text comments and decides whether they justify new items. It chooses what feedback to show. It often has the final word in a closing meeting where items near the threshold are resolved. In many published studies, the steering group also writes the final statements, which means its members translate the panel's ratings into the form of words that will be cited. The composition of the steering group should therefore be treated with the same seriousness as the composition of the panel. A group drawn entirely from one institution, one discipline or one school of thought will tend, without any intention to mislead, to frame items in ways that reflect its own assumptions. A steering group for a core outcome set in a surgical condition that includes only surgeons is likely to produce an initial item list heavy with technical and complication-related outcomes and light on the outcomes patients care about, such as return to work or bodily appearance. Including a methodologist, at least one person from each major stakeholder group that will sit on the panel, and, where the topic concerns patients or the public, people with lived experience of the condition in the steering group itself, is the most effective single protection against framing bias. The steering group's conflicts of interest also matter. A Delphi study on the appropriate use of a device, organised by people who hold patents on it or receive consultancy payments from its manufacturer, may be methodologically impeccable and still not be trusted. The ACCORD reporting guideline for consensus methods, published in PLOS Medicine in January 2024, asks studies to report the roles of those who organised the process and their funding and conflicts of interest, precisely because the organisers' influence is so large. The best protection is to declare interests in the protocol and to design the process so that no single interested party controls a decision that matters. Defining the question A surprising number of Delphi studies are vague about what they are trying to decide. "To reach consensus on best practice in the management of X" is an aspiration, not a question. Before designing anything, the steering group needs to be able to state precisely what the panel is being asked to judge, for what purpose, and for which population or setting. A well-defined Delphi question has four elements. It specifies the object of judgement: outcomes to be measured, recommendations for practice, definitions of a condition, indicators of quality, priorities for research, forecasts of a future state. It specifies the criterion of judgement: importance, appropriateness, feasibility, desirability, likelihood, clarity. It specifies the scope: which population, which setting, which country or health system, which time horizon. And it specifies the use: what will happen to the output, and who will act on it. The criterion is where many studies go astray. Asking panellists whether a quality indicator is "important" conflates at least three distinct judgements: whether the thing it measures matters, whether the indicator validly measures it, and whether it is feasible to collect. A panellist might believe that timely pain relief in the emergency department matters enormously while doubting that "time from triage to first analgesic dose" measures it well and knowing that most departments cannot extract the data. A single rating of importance forces that panellist to average three different judgements in an unknown way. The RAND/UCLA tradition, which asks separately about validity and feasibility in some applications, is instructive here. Separate criteria mean separate ratings, which lengthens the questionnaire, but they produce results that can be interpreted. The scope needs equal care. An international panel asked about the appropriateness of an intervention without being told which health system to imagine will answer from its members' own contexts, and the resulting disagreement may reflect differences in resources rather than differences in judgement. If the question is meant to apply to high-income settings, say so. If it is meant to apply everywhere, consider asking panellists to rate for their own setting and analysing by region. The evidence comes first A Delphi study that asks experts to rate items without first showing them what is known is asking them to rely on memory, and memory is selective. Panellists will recall the studies they read, the cases they saw, and the conference presentations that impressed them. Different panellists will recall different things, and some of the resulting disagreement will be disagreement about facts that a literature review could have settled. For this reason most rigorous modified Delphi designs begin with a structured evidence review, and many distribute a summary of it to panellists. The RAND/UCLA method makes this explicit: panellists receive a literature review, often a substantial one, before they rate a single indication. In core outcome set development, the COMET Initiative's handbook, published by Paula Williamson and colleagues in Trials in 2017, recommends identifying candidate outcomes from a systematic review of the outcomes measured in existing trials, supplemented where possible by qualitative research with patients to capture outcomes that trials have ignored. The evidence review does two jobs. It generates candidate items, so that the first questionnaire reflects the full range of what has been proposed rather than what the steering group happened to think of. And it gives panellists a shared factual base, so that their ratings reflect judgement about what the evidence means rather than disagreement about what it says. The distinction matters because a Delphi study cannot resolve factual disputes; it can only record them. If one panellist believes a treatment reduces mortality and another believes it does not, iteration will at best produce a compromise between two readings of the literature, which is worse than either reading checked against the literature itself. The evidence summary is itself a source of influence. How it is written, what it emphasises and what it omits all shape the ratings that follow. A summary that describes one intervention as "well established" and another as "promising" has done some of the panel's work for it. The discipline is to present evidence neutrally, in a consistent format for each item, with the strength of evidence stated using an explicit system where one is available, and to have the summary checked by someone outside the steering group. What must be decided in advance The heart of the protocol is a list of decisions that must be made and written down before the data can influence them. Several reporting guidelines, including CREDES, published by Saskia Jünger and colleagues in Palliative Medicine in 2017, ACCORD, and DELPHISTAR, published by Marlen Niederberger and colleagues in PLOS ONE in August 2024, ask studies to report these decisions. The protocol is where they should first appear. Panel eligibility and structure. Who qualifies as an expert, by what criteria, and in which stakeholder groups. Whether groups will be analysed separately. The target size of each group and the minimum number of respondents required for a round to count. Recruitment. How potential panellists will be identified and approached, and what they will be told about the time commitment. The first round. Whether it is open, seeded, or both; where seeded items came from; whether panellists can propose new items; and the rule for adding them. Items and scale. The exact wording of the instructions and the rating scale, including the labels on each point and whether there is an "unable to judge" option. Feedback. What panellists will see between rounds: which statistics, whether their own previous rating, whether comments from others, and whether results will be split by stakeholder group. The consensus definition. The statistical rule for declaring consensus that an item should be included, consensus that it should be excluded, and the absence of consensus. The minimum number of respondents to whom the rule applies. Stopping. The maximum number of rounds, and whether the study stops when consensus is reached, when responses stabilise, or after a fixed number of rounds regardless. Item carry-forward. Which items return in later rounds. Some designs re-present every item; others drop items that have already reached consensus. The choice changes the dynamics of later rounds. Attrition. Whether panellists who miss a round are invited to the next; how non-response will be analysed; what response rate will be considered adequate. Any final meeting. Who attends, what it may decide, and how its decisions will be recorded and reported alongside the Delphi results. Analysis and reporting. How free-text comments will be analysed, and how items that did not reach consensus will be reported. Diamond and colleagues' review found that consensus was specified in advance, with a threshold, in only 42 of the studies that reported achieving it. Where the rule is chosen afterwards, the reader cannot tell whether a 70 per cent threshold was selected because it was principled or because it allowed a particular item through. The protocol removes that doubt, and it is only as good as the record of it. Publishing the protocol in a journal, registering it on a public platform such as the Open Science Framework, or, for core outcome sets, registering the study in the COMET database, creates a timestamped record that the study can later be checked against. A protocol in miniature It helps to see how these decisions fit together in a single, deliberately simple case. The following is a hypothetical design, constructed for illustration rather than drawn from any published study. Suppose a national nursing body wants to agree on a set of indicators for the quality of pressure-injury prevention in residential care homes. The steering group comprises two tissue-viability nurses, a care-home manager, a geriatrician, a health services researcher experienced in consensus methods, and two relatives of care-home residents. Its first act is to narrow the question. It is not asking which prevention practices work; that belongs to trials and systematic reviews. It is asking which measurable aspects of practice a regulator should monitor, judged on two separate criteria: whether the indicator reflects something that matters for residents, and whether a typical home could collect it without new systems. A scoping review of existing indicator sets and inspection frameworks produces an initial list of thirty-four candidate indicators, each with a one-paragraph evidence note written to a common template. The panel will comprise three groups: nurses working in care homes, specialists in tissue viability and wound care, and residents' relatives together with advocacy representatives. The protocol sets a target of at least twenty respondents per group and states that any group falling below twelve in a round will be reported descriptively but excluded from the consensus rule. The protocol fixes a nine-point scale for each criterion, labelled at the ends and middle, with an "unable to rate" option. It defines consensus for inclusion as at least 70 per cent of respondents in every group rating an indicator 7 to 9 on both criteria, and no more than 15 per cent rating it 1 to 3 on either. It defines consensus for exclusion symmetrically. It sets a maximum of three rounds, with every item not yet at consensus carried forward, and states that the process stops early if no item changes category between two consecutive rounds. Feedback will show the distribution of ratings for each group separately, the panellist's own previous rating, and a summary of reasons given, grouped by theme and stripped of identifying detail. Panellists who miss round two will still be invited to round three. A final online meeting, open to a sample of panellists from each group, may discuss items left without consensus but may not add or remove items that met either rule. None of these choices is uniquely correct. A different steering group might use a five-point scale, a higher threshold, or a single combined criterion. The point is that each choice has been made for a reason that can be stated, before any rating has been seen. When the results arrive, nobody will be tempted to wonder whether the threshold was chosen to include a favoured indicator, because the threshold was public before the indicator was rated. Allowing for the unforeseen Pre-specification is not rigidity. Delphi studies encounter surprises: an item that turns out to be ambiguous, a stakeholder group that responds far less than expected, a round-two comment that reveals a crucial missing option. A protocol can anticipate some of these by stating in advance how they will be handled. It might say, for example, that items flagged as unclear by more than a stated share of respondents will be reworded and re-presented, with the rewording reported; or that if a stakeholder group falls below a minimum number, its results will be reported descriptively but not used to determine consensus. When something happens that the protocol did not foresee, the honest course is to make the change, record why, and report it as a deviation. Deviations are normal. What damages credibility is the undisclosed deviation, the threshold quietly relaxed from 80 to 70 per cent in the final round, or the stakeholder group whose disagreement was reported only in a supplementary file. A reader who can see the deviation can judge it. A reader who cannot see it has been misled, even if the change was defensible. Ethics and consent Delphi panellists are research participants, and a protocol must address their interests. They need to know what they are agreeing to: how many rounds, how long each will take, how their responses will be used, whether they will be named in the publication, and whether they can withdraw. Many studies acknowledge panellists by name in the final paper, which rewards participation but has consequences. If panellists know they will be listed as contributors to a consensus statement, some may feel bound to its conclusions and reluctant to express dissent that might later be attributed to them, even though individual ratings remain anonymous. Offering panellists the choice to be acknowledged or not, and being clear that acknowledgement does not imply endorsement of every statement, helps. Patient and public panellists raise particular considerations. They may need plain-language versions of items, support with technical terms, and payment for their time on terms comparable to those offered to professionals. They may be rating items that touch on their own experience of illness. The protocol should state how these needs will be met. Failing to meet them is not just an ethical lapse; it is a methodological one, because a patient group that cannot understand the items or loses heart after round one will be underrepresented in the final result, and the consensus will tilt towards the professionals by default. The protocol as a public commitment It is tempting to think of the protocol as paperwork required by ethics committees and journals. That view misses its function. A Delphi study produces a consensus whose authority rests on a claim that the process was fair. The protocol is the evidence for that claim. It shows that the rules were set before anyone knew which items would pass them, that the panel was chosen by criteria rather than by acquaintance, and that the feedback and stopping rule were not adjusted to steer the outcome. A study that can point to a protocol written, registered and followed, with deviations disclosed, can say something that a study without one cannot: that its consensus was not engineered after the fact. Every other chapter in this book depends on that foundation. The most consequential decision the protocol records is who will be asked, and that is where the next chapter turns. Chapter 3. Who Counts as an Expert If a Delphi study's result could be predicted from a single piece of information, that information would be the composition of the panel. Items, scales and thresholds shape the edges of the outcome; the panel shapes its centre. A panel of surgeons and a panel of patients, asked the same questions about the same operation, will reach different conclusions, and both will be reported as consensus. A panel recruited through one professional society will tend to reproduce that society's orthodoxy. A panel that is ninety per cent from high-income countries will produce recommendations that assume high-income resources, however "international" the title claims it to be. This is not a flaw that better statistics can remove. It follows from what Delphi is: a structured way of aggregating the judgements of particular people. The judgements belong to those people, and the consensus is theirs. The question for the designer is therefore not how to find the objectively correct panel, since there is none, but how to choose a panel whose judgement the intended users of the result will have good reason to trust, and how to describe it so that readers can see whose consensus they are looking at. What expertise means in a Delphi study The word "expert" suggests a person with specialist knowledge, usually certified by qualifications, experience, and publication. That meaning is appropriate for some Delphi questions and misleading for others. For estimation and forecasting, the relevant expertise is knowledge that bears on the answer. A panel forecasting the adoption of a medical technology needs people who understand the technology, the regulatory pathway, health system purchasing and clinical practice. Formal credentials are a reasonable proxy, although the research of Philip Tetlock, summarised in his 2005 book Expert Political Judgment, is a standing reminder that credentialed experts are often poorly calibrated forecasters, and that breadth of perspective can matter more than depth of specialism for predicting complex events. For questions of value and priority, the relevant expertise is different. Deciding which outcomes matter most in a chronic disease is not a matter on which professors know more than patients. Patients have knowledge of what living with the condition is like, what they would trade for what, and which symptoms dominate daily life, that no amount of clinical training supplies. The core outcome set movement has taken this seriously for more than a decade, and the inclusion of patients as a distinct panel group is now standard in well-conducted core outcome set studies. The same logic applies to carers, to service users in social care, to affected communities in environmental policy, and to front-line staff whose knowledge of what is feasible is practical rather than academic. For questions of definition and terminology, expertise includes knowledge of how terms will be used. The CATALISE panel on children's language impairments deliberately spanned education, psychology, speech and language therapy, paediatrics, child psychiatry and other disciplines, because the terminology it proposed would have to work across all of those professions and in the services where they meet. A Delphi protocol should therefore define expertise in relation to the question, not in the abstract. For each stakeholder group, it should state what knowledge or experience qualifies a person for inclusion and how that will be checked. Typical criteria for professionals include a minimum number of years in relevant practice, current involvement in the relevant field, publications or leadership roles, or nomination by a professional body. Typical criteria for patients and carers include direct experience of the condition or service, sometimes within a stated time frame. The criteria should be specific enough that a reader could, in principle, apply them. Heterogeneity and its consequences Designers face a basic choice between homogeneous and heterogeneous panels. A homogeneous panel, drawn from one profession or perspective, will usually reach consensus faster and on more items, because its members share assumptions. A heterogeneous panel, spanning professions, sectors and lived experience, will agree on less, take longer, and produce a result that is more robust precisely because it has survived challenge from different angles. Rowe and Wright's review of forecasting evidence recommended heterogeneous panels, and the logic extends beyond forecasting. The value of pooling judgements comes from the partial independence of the errors in them. A panel of people who all trained in the same place, read the same journals and attend the same meetings shares its blind spots, and pooling their views multiplies confidence without adding information. A panel of people with different vantage points brings errors that partly cancel and knowledge that partly complements. Heterogeneity has costs. Items must be written so that all groups can understand them. Feedback must be designed so that the majority group does not swamp the minority. And the analysis must decide what consensus means when groups disagree. If 85 per cent of clinicians rate an outcome as critical and 40 per cent of patients do, has the panel reached consensus? Pooling the groups into a single percentage would hide the disagreement and let the result depend on how many of each were recruited. The better practice, now common in core outcome set work, is to analyse each group separately and to require consensus within every group, or at least to report group-level results alongside any pooled figure. Chapter six returns to the statistics. How big should a panel be? There is no statistical formula for Delphi panel size, because a Delphi panel is not a sample from which inferences are drawn about a population. It is the population whose judgement is being reported. The question of size is therefore a practical and substantive one, not a matter of power calculations. Published guidance and practice span a wide range. The RAND/UCLA method uses panels of seven to fifteen, with nine as its traditional number, reflecting its reliance on a face-to-face meeting in which everyone must be able to participate. Rowe and Wright suggested between five and twenty experts for forecasting panels. Core outcome set studies commonly recruit hundreds of panellists across several groups, and the COVID-19 study published in Nature in 2022 involved 386. The ACCORD guideline's own Delphi recruited 72. Several considerations bear on the choice. The first is the number of distinct perspectives that need representation. A panel with four stakeholder groups needs enough members in each to produce a stable percentage; with only six patients, one person changing their rating moves the patient agreement figure by more than sixteen percentage points, and a threshold of 70 per cent becomes a question of whether four or five people agree. The second is expected attrition, discussed below; a panel that begins at the minimum size will end below it. The third is the burden of the questionnaire; a very long item list with a very large panel produces an enormous analytic task and may reduce the quality of each rating. The fourth is credibility: a consensus statement meant to guide an international field may need to show broad participation to be taken seriously, even if a smaller panel would produce similar results. A useful rule is to decide, for each stakeholder group, the smallest number of respondents at which a percentage agreement would be meaningful, commonly around fifteen to twenty, then inflate the recruitment target to allow for the attrition the study expects. ACCORD's own design required agreement from at least 80 per cent of a minimum of twenty respondents, a helpful example of making the floor explicit. Finding and recruiting panellists How panellists are found determines who is found. Invitations sent through a single professional society reach its members, who are not all practitioners in the field and who may share its official positions. Invitations to authors of relevant papers reach researchers rather than practitioners. Snowball recruitment, in which initial panellists nominate others, reaches their networks, which may be homogeneous. Open calls through social media reach people who follow the organisers and are motivated enough to respond. A structured approach helps. Chitu Okoli and Suzanne Pawlowski, writing in Information & Management in 2004, described a "knowledge resource nomination worksheet" for Delphi recruitment. The designers first list the disciplines, organisations and types of knowledge the panel needs, then identify individuals in each category through literature, organisations and nominations, then rank and invite them category by category until each is filled. The value of such a procedure lies less in its details than in its effect: it forces the designer to decide what the panel should look like before seeing who agrees to join, and it makes gaps visible. Geographic and demographic balance deserve explicit attention. International panels in health research are routinely dominated by North America, Western Europe and Australasia, partly because recruitment runs through English-language journals and networks. If the result is meant to apply globally, recruitment should target low- and middle-income settings deliberately, and the final panel composition should be reported by region so that readers can judge how global the consensus is. The same applies to gender, career stage, practice setting and, for patient groups, to age, ethnicity and severity of condition. Conflicts of interest among panellists should be collected and reported. The aim is not usually to exclude people with interests, since in many fields those with the deepest knowledge also have commercial or professional stakes, but to allow readers to judge whether the panel as a whole was balanced. A panel on the appropriate use of a drug in which a majority receive funding from its manufacturer is not neutral, however anonymous its ratings. Attrition and why it matters so much Every Delphi study loses panellists between rounds. People are busy, questionnaires are long, and the second and third rounds offer less novelty than the first. Response rates of 70 to 80 per cent per round are often considered acceptable, and studies commonly report cumulative retention well below that. The ACCORD Delphi retained 58 of 72 panellists in its first round, 54 in its second and 51 in its third. DELPHISTAR's own Delphi study had 91 completers in its first round, 69 in its second and 56 in its third. Attrition would be merely an inconvenience if it were random. It rarely is. Those who drop out may differ systematically from those who stay: they may be less engaged with the topic, more pressed for time, less convinced of the study's value, or, most importantly for a consensus method, in disagreement with the emerging view. A panellist who sees in round two that her ratings are far from the group median has several options. She can hold her position, which requires explaining it; she can move toward the group; or she can stop responding. The third option is the least effortful. If dissenters leave at a higher rate than others, the remaining panel will show more agreement than the original one did, and the study will report rising consensus that is in fact rising homogeneity. The effect is hard to detect after the fact unless the study was designed to look for it. The protocol can require that the round-one ratings of those who later dropped out be compared with those of completers; if dropouts were systematically more likely to have rated against the eventual consensus, the reader should know. Studies can also report consensus figures using the original panel as denominator as well as the responding panel, which shows how much of the apparent agreement depends on who stayed. Retention can be improved. Clear expectations at recruitment, short questionnaires, prompt rounds with reasonable deadlines, personalised reminders, and feedback that shows panellists their contribution was read all help. So does inviting non-responders to later rounds rather than excluding them, which keeps the door open for those who missed a deadline without losing interest. Some studies offer modest payments or certificates of participation; for patient panellists, payment for time is increasingly regarded as a matter of fairness rather than incentive. Weighting, self-rated expertise and the organisers' own votes Once a heterogeneous panel is assembled, a question follows that many studies never ask aloud: should every panellist's rating count equally? Pooling ratings into a single percentage answers it implicitly, and the answer depends on recruitment rather than principle. If a study recruits 150 clinicians and 30 patients and pools them, clinicians carry five times the weight of patients, not because anyone decided they should but because they were easier to find. There are three defensible responses. The first is to analyse groups separately and require consensus in each, which gives each group a veto without weighting individuals. The second is to weight groups equally in a pooled figure, so that the patient percentage and the clinician percentage each count for half regardless of numbers. The third is to fix group sizes in advance and recruit to them, accepting pooled figures because the balance was designed. Each is reasonable. What is not reasonable is to let the ratio emerge from recruitment and then report a pooled percentage as if it were neutral. A related idea, explored since the early RAND work, is to weight panellists by their own assessment of their expertise on each item, or to exclude ratings from those who declare themselves unfamiliar with an item. The attraction is obvious: a cardiologist asked about a rehabilitation outcome may know less than a physiotherapist, and a self-rating allows the method to recognise that. The difficulty is that self-assessed expertise is only loosely related to actual knowledge, and confidence varies by personality, gender and professional culture as well as by competence. A more modest and more common practice is to offer an "unable to rate" or "outside my expertise" option and to exclude those responses from the denominator for that item, reporting how many were excluded. That lets panellists abstain honestly without inviting them to rank themselves. The steering group's own participation is a final issue. In many studies, members of the organising group also sit on the panel. There are arguments for this: they are often among the most knowledgeable people available, and excluding them may weaken the panel. But they have designed the items, they know the purpose of the study, and they may have views about what the result should be. At a minimum, a report should state whether organisers rated items and how many of them did. Where the panel is small, a steering group that makes up a fifth of it can move an item across a threshold on its own. Some studies exclude organisers from rating altogether, or analyse the results with and without them, which is the clearest way to show that the consensus did not depend on them. Reporting the panel Because the panel so largely determines the result, the description of the panel is among the most important parts of any Delphi report. Readers should be able to see how many were invited and how many responded in each round, broken down by stakeholder group; the qualifying criteria and how they were checked; the geographic distribution; the relevant demographic and professional characteristics; conflicts of interest; and the characteristics of those who dropped out compared with those who stayed. A reader armed with that information can make the judgement that the phrase "an international panel of experts" does not permit. She can ask whether these are the people whose agreement should guide her decision, and whether what they agreed was likely to have been shaped by who was in the room and who left it. That judgement cannot be delegated to the method. The next chapter turns to the other side of the encounter: what the panel is actually asked. Hashtags: #DelphiMethodologies #DelphiMethod #ExpertConsensus #ConsensusBuilding #ExpertPanels #PolicyDelphi #HealthDelphi #ModifiedDelphi #ClassicalDelphi #RANDUCLAAppropriatenessMethod #ExpertJudgement #ConsensusMethods #ControlledFeedback #IterativeRounds #PanelAnonymity #ConsensusThresholds #StakeholderRepresentation #ExpertSelection #PanelAttrition #ConsensusMeasurement #KendallsW #CoreOutcomeSets #ConsensusGuidelines #PolicyConsensus #FutureOfDelphiResearch